Government's Patch Management Challenges

From the Editors of CIOSC

Patch management has moved quickly to the forefront of IT challenges facing the federal government after many years as an afterthought for system administrators. It's easy to see why. Hackers have been quick to capitalize on product vulnerabilities as an attack vector. The primary driving force today is the heightened awareness of the impact of inadequate patch management: computers and systems left vulnerable to attacks and intrusions - even takeovers. In a world that is now so security-oriented, these threats must be dealt with swiftly and comprehensively.

That task is increasingly difficult, as the number of attacks and their severity increase at an exponential rate, threatening to overwhelm established methods of response. Recent events demonstrate how quickly the problem has escalated:

• The Code Red virus of 2001 achieved an infection rate of over 20,000 systems within 10 minutes. The projected damage from the two Code Red viruses was estimated at over $2 billion.
• In early 2003, the Slammer worm successfully created a new level of Internet chaos by infecting more than 90 percent of vulnerable systems within 10 minutes.
• At its peak, the MyDoom virus of 2004 infected one of every 12 messages, according to CNET.
• In May 2004, the initial strain of the Sasser worm was estimated to infect as many as a million computers within a week, and subsequent variations spread even more rapidly.

Federal awareness of the challenges

No enterprise takes the extent of these threats and their potential consequences more seriously than the United States government. Following the 9/11 attacks, the federal government took a series of steps to harden the nation's computer systems against attack, including creating US-CERT and initiating a Government Accountability Office (GAO) investigation into the problem of computer system vulnerabilities and how to improve the government's information technology security, particularly in the realm of patch management.

In a report issued last June, the GAO made clear to Congress what many IT professionals already knew. Federal agencies face several fundamental challenges to implementing patch management practices that meet the current and emerging environment, including:

• Prioritizing patches
• Quickly installing patches while maintaining effective patch management practices
• Patching heterogeneous systems
• Ensuring that mobile systems receive the latest patches
• Avoiding unacceptable downtime when patching high-availability systems
• Dedicating sufficient resources toward patch management

Comprehensive need, diverse responses

While there appears to be general agreement about the critical nature of effective patch management, the federal government is divided about how to best solve these challenges.

One school of thought is to build all applications internally using open source technology where possible. The idea here is that these unique systems will be more secure and stable because they won't have the vulnerabilities associated with the Windows family of operating systems. In summer 2005, Government Enterprise magazine reported that the Office of Personnel Management has taken this approach in building its public-facing E-government applications. These include USAJobs, a federal job-search portal, as well as an E-training program and a background checking system for new hires. The OPM reports no difficulties, but while these are government-wide systems, they are smaller and their missions more narrowly defined than many federal systems. It remains to be seen whether this approach could serve the broader needs of the federal government or even major departments.

Another approach places greater reliance on widely available hardware and software. The United States Air Force recently announced an agreement with Microsoft and Dell to combine all existing Air Force software and support contracts into one, affecting about 525,000 computers. While there will be cost benefits from this consolidation, the Air Force characterizes it as primarily an initiative to improve security on its network and to protect the integrity of the information that travels on it.

The Air Force has had as many as 38 separate contracts, each configured individually and managed by the local installations, resulting in thousands of configurations to manage. "It takes months, literally, in most cases [to install a security patch]," John M. Gilligan, the Air Force's chief information officer, said in the official announcement. That creates situations where installing patches is costly and time-consuming, an unacceptable situation when the Air Force - and its network - have to be ready around the clock.

The Air Force concluded that, in an ideal situation, every machine would be exactly the same, providing a central control for network managers to discover problems, devise solutions, and apply fixes. "The major driver for us is security," observed Gilligan." Our warfighters recognize that as we come to depend on this network, it has to be available." The Air Force is fighting attacks on its critical computer systems by creating an environment that allows simultaneous, real-time patch management under the running assumption that it will be more efficient to install the same patch across a large number of homogenous systems.

Roadblocks remain

While the federal government has recognized the importance of effective patch management as an essential part of maintaining network availability and ensuring security, the 2004 GAO report indicates that achieving those goals won't be easy. The major headings in the report tell the story:

• "Agencies Are Not Consistently Implementing Common Practices for Effective Patch Management"
• "Agencies' Degree of Centralization Varies"
• "Agencies Are Not Consistently Performing Risk Assessments"
• "Agencies Are Not Testing All Patches Before Deployment"
• "Significant Patch Management Challenges Remain"

A proactive approach

A key component to effective patch management is rapid response. Given today's environment in which the window between vulnerability discovery and attack has narrowed from months to days to sometimes hours, it's critical to install the necessary patches as quickly as possible to as many vulnerable computers as possible. The most effective way to address patch management is as part of a comprehensive solution, one that not only addresses vulnerabilities and viruses but spyware, malware, DoS attacks, and other intrusions. Another element in the reduction of vulnerabilities via an optimized patch management process is the need for advanced warning and intelligence concerning vulnerabilities and their exploitation.

Advanced customization capabilities allow organizations to receive only those alerts relevant to their specific environment. Alerts, status tracking, and reporting are available through a secure Web site. Alerting messages are dispatched by email, voice, fax, or SMS. Content is structured for integration into current security operations. The optional XML format ensures simple and efficient information reuse within an organization's IT support and Help Desk operations. By eliminating hours spent searching through Web sites and emails to gather information, distributing it, and then following up on the results, this approach enables a proactive attack for ensuring a secure environment while saving time and money.

A March 2005 report on cyber security by the President's Information Technology Advisory Committee characterized the country's IT infrastructure as "highly vulnerable to premeditated attacks with potentially catastrophic effects" and a "prime target for cyber terrorism as well as criminal acts." The federal government now has technologies available that can help stop cyberattacks before they strike.