Center for Arms Control and Non-Proliferation

Documents on the Department of Homeland Security 2006 Bioterrorism Risk Assessment

by Alan Pearson [contact information]

January 9, 2008

Homeland Security Presidential Directive 10 (HSPD-10, "Biodefense for the 21st Century, April 28, 2004) states that "the United States requires a continuous, formal process for conducting routine capabilities assessments to guide prioritization of our on-going investments in biodefense-related research, development, planning, and preparedness. These assessments will be tailored to meet the requirements in each of these areas. Second, the United States requires a periodic [in practice, every four years starting in 2008] senior-level policy net assessment that evaluates progress in implementing this policy, identifies continuing gaps or vulnerabilities in our biodefense posture, and makes recommendations for re-balancing and refining investments among the pillars of our overall biodefense policy."

The Department of Homeland Security (DHS) is responsible for conducting these assessments, the first one biannually beginning in 2006 and the second one every four years beginning in 2008. In 2005, Homeland Security Secretary Chertoff stated that DHS would use a risk-management approach to guide its strategies and activities, consistent with the intent of the Homeland Security Act of 2002 (Section 201(d)(2)). Risk is a function of likelihood and consequence. DHS has adopted this approach for the first of the two assessments mandated by HSPD-10.

The first "Bioterrorism Risk Assessment," prepared by the DHS National Biodefense Analysis and Countermeasures Center (NBACC) using a methodology developed by Battelle Memorial Institute, was completed on January 31, 2006 and a report on the assessment was published on October 1, 2006. The assessment used threat scenarios and consequence modeling to rank 28 biological agents (primarily the "CDC Category A, B, and C" agents) according to their relative risk. For this purpose, the estimated likelihood of agent use in a range of different scenarios ("the probability that an adversary acquires, produces, and disseminates a biological weapon," based on intelligence community input and the judgment of subject matter experts) was multiplied by the projected consequences resulting from each scenario (using data vetted by the Department of Health and Human Services). The risk calculation was weighted towards high-consequence events.

In mid-2006, DHS requested the National Research Council (an entity of the National Academies of Science) to establish a Committee to review its first Bioterrorism Risk Assessment and to make recommendations for methodological improvements that would help guide its efforts in preparing a new risk analysis that is to be released in 2008. The documents here comprise some of the presentations made by DHS and others to meetings of this Committee on Methodological Improvements to the Department of Homeland Security's 2006 Bioterrorism Risk Assessment, as well as the Interim Report of the Committee. The final report of the Committee, due by September 30, 2007, has not yet been released.

Background on Bioterrorism Risk Assessment

A brief background on bioterrorism threat and risk assessments.

The National Biodefense Analysis and Countermeasures Center

Bernard C. Courtney, Ph.D., Scientific Director
28 August 2006

This PowerPoint presentation provides an overview of the mission of the National Biodefense Analysis and Countermeasures Center (NBACC) and its component parts. NBACC is a Federally Funded Research and Development Center operated by Battelle Memorial Institute. NBACC's Biological Threat Characterization Center (BTCC) is responsible for producing the biannual Bioterrorism Risk Assessment. In so doing, the BTCC identifies "knowledge gaps" that, if filled, could reduce uncertainties in the assessment. The BTCC is also responsible for conducting laboratory-based threat characterization research to fill these knowledge gaps. This second role of the BTCC is controversial because it can include investigations of offensive aspects of biological agents. Slide 6 of this presentation mentions 45 bioterror/biocrime cases for which the Bioforensics Analysis Center component of NBACC had processed samples between sometime in 2004 and the date of this presentation. According to DHS personnel, in all of these cases initial tests indicated the presence biological material (anthrax, ricin and corn starch all qualify as biological material). No further information was provided.

DHS Bioterrorism Risk Assessment

Steve Bennet, Ph.D., Risk Assessment Program Manager
28 August 2006

This PowerPoint presentation provides an overview of the 2006 DHS Bioterrorism Risk Assessment. DHS calculates risk as the product of the likelihood and the consequences of a biological attack. According to DHS, previous assessments such as those mentioned on slide 7, have focused almost entirely on consequences. Slide 10 states that the DHS methodology is dynamic rather than static - that is, the methodology allows a user "to adjust parameters to address different questions." This should enable decision makers to compare the relative impacts of different risk mitigation strategies on overall risk and to identify those strategies which would have the greatest impact per dollar. However, DHS personnel have indicated that the 2006 version of the methodology did not achieve this goal. Slide 10 also states that the risk assessment "is anchored by a PRA-based approach." PRA stands for Probabilistic Risk Assessment (see presentation by Denning). Note that slide 13 does NOT present the actual results of the 2006 risk assessment; it is simply an example of what the results of a bioterrorism risk assessment might look like.

2006 DHS Bioterrorism Risk Assessment: Methodology

Richard S. Denning, Ph.D.
28 August 2006

The key item in this selection of documents, this PowerPoint presentation describes the Probabilistic Risk Assessment (PRA) methodology used to produce the 2006 DHS Bioterrorism Risk Assessment. In this method, multiple independent scenarios are constructed from a series of possible events (discrete steps along an attack pathway, each with multiple options), the likelihoods (probabilities) and consequences of each scenario are estimated, and risk is then calculated as the product of these estimated likelihoods and consequences. For the 2006 Bioterrorism Risk Assessment, the PRA methodology was used to determine both absolute and relative risks of bioterrorist attacks from 28 different biological agents (largely drawn from the CDC Category A-C list) in terms of fatalities, illnesses, and direct economic impacts.

Slide11 lists the 17 events used to construct the scenarios in this risk assessment. Significant uncertainty surrounds the probabilities associated with the various options (branches) for these events; this uncertainty is captured as a probability distribution (analogous to a bell curve). Except for event 1 (initiation frequency), the branch probabilities were determined primarily by subject matter expert (SME) elicitation and must add to 1 for each event (that is, the method assumes that once action is taken to begin a process that could lead to an attack, the attack will occur unless interdicted). Interestingly, the SMEs assigned probability rankings for agent selection (event 3, slide 28) to only approximately five of the agents, stating that the rest simply accounted for a combined probability of 5%. Because SME elicitation is heavily dependent on subjective judgment, the assumptions and reasoning of SMEs must be captured in the risk assessment process.

Slides 33 – 36 describe the consequence calculations. The possibility of terrorist failure (as opposed to interdiction) at any step after agent acquisition is incorporated into the consequence calculation rather than in the scenario probability analysis. Depending on how this is done, it could bias the outcome of the risk assessment toward higher reported risk. Slide 34 states that the amount of agent produced (target mass) was calculated as a distribution based on a range of production times suggested. These times were suggested by the intelligence community. Note that the "factors" listed on slide 34 are "reduction factors" – they represent the fraction of starting material lost at each stage in the production, storage and transportation process.

Slides 40 – 45 describe how risk and uncertainty were determined and reported. The middle figure on slide 43 illustrates the type of output generated by the risk assessment: a graph showing the likelihood that a bioterrorist attack will generate a given level of consequence or greater for when various biological agents are used. This is one of the most useful ways to present the data as it shows the variation of both likelihood and uncertainty with consequence magnitude, allowing for a fuller appreciation of risk. It does not, however, reveal the sources of uncertainty or the confidence associated with the estimates.

In the 2006 Bioterrorism Risk Assessment report, DHS decided instead to present the data primarily as a relative (not absolute) risk ranking for the 28 biological agents analyzed –illustrated by the right hand figure on slide 43. This simpler graph does not show how likelihood and uncertainty vary with consequence. It also assigns greater weight to rare high consequence events than to frequent low consequence events. Sensitivity studies were also performed to show the impact of analysis assumptions and uncertainties on the risk results. The actual results of both the relative risk ranking and the sensitivity analyses are classified. Note that slide 45 does NOT present the actual results of the 2006 risk assessment; it is simply an example of what the results of a bioterrorism risk assessment might look like.

Interim Report on Methodological Improvements to the Department of Homeland Security's Biological Agent Risk Analysis

Committee on Methodlogical Improvements to the Department of Homeland Security's Biological Agent Risk Analysis
2007

In this Interim Report, the Committee finds the risk assessment methodology used by DHS for its 2006 Bioterrorism Risk Assessment to be "adequate but incomplete." It's "main concern" is that the current methodology "does not fully support any of the components of risk analysis" (risk assessment, risk perception, risk communication, and risk management). In particular, it "does not include consideration of the actions of an intelligent and reactive adversary," it makes "no provision for risk perception," and it "does not allow the explorations by decision makers of 'what-if' questions, which is needed for risk management."

The Committee states that DHS lacks and should establish a clear statement of the long-term purposes of its bioterrorism risk analysis, enunciating how it will support the risk assessment, risk perception, and especially risk management needs of its stakeholders and it should specify criteria and measures for determining how well these purposes are achieved. DHS should also determine how it will incorporate new information into its analyses. The Committee makes clear its opinion that the key sources of uncertainty in bioterrorism risk assessment are associated with likelihoods, not with consequences. It states that the reliability of probability assessments generated using expert opinion "will always be problematic, requiring careful attention to the elicitation methods as well as needing well-designed sensitivity analyses." It also states that the event-tree methodology used by DHS to determine probabilities was not developed to model the possible actions of intelligent adversaries. The Committee applauded DHS' plan to address these challenges by improving expert elicitation, but expressed serious concern about DHS' plan to pursue more fine-grained and detailed consequence modeling, which it said "could result in false precision that might be mistaken for accuracy in a model that is, by necessity, not particularly well validated, affecting both risk assessment and risk management." Instead, the Committee recommended that DHS improve its analysis of intelligent adversaries by supplementing or complementing the event-tree methodology with alternative methods for incorporating expert knowledge. Importantly, it also recommended that risk analyses should include a "qualitative assessment of sources and quality of the data and perhaps quantitative indications of the confidence and precision associated with the current estimates." Finally, the Committee found that the current methodology can not be used to "determine which portfolio of investments is most effective" for reducing risk, and it recommends that DHS increase the methodology's emphasis on risk management. To do so, the Committee argues that rather than increasing the fidelity of existing models, DHS should expand the models to take into account, for example, how different risk management strategies might affect terrorist decisions.

2006 DHS Bioterrorism Risk Assessment: Thoughts and Impressions from the NAS Interim Report

Department of Homeland Security
10 February 2007

This PowerPoint presentation provides initial responses to the recommendations made in the Interim Report of the NRC committee reviewing the DHS bioterrorism risk assessment methodology. DHS appears to accept most, though not all, of the recommendations made in the Interim Report and it indicates the steps it is taking to address those recommendations. DHS does not address the recommendation to establish a clear statement of the long-term purposes of the bioterrorism risk assessment, including criteria and measures for assessing how well these purposes are being achieved. DHS also disagrees with the Committee's contention that there is little value in pursuing more fine-grained and detailed consequence modeling, and it disagrees in part with the Committee's conclusions that the current methodology provides inadequate support for risk management decision making.

2006 DHS Bioterrorism Risk Assessment: Planned Improvements for the 2008 Risk Assessment

Traci Hale
29 August 2006

The first of two PowerPoint presentations detailing planned improvements to the DHS bioterrorism risk assessment methodology for the 2008 risk assessment. Presented to the NRC Committee at its first meeting.

2008 DHS Bioterrorism Risk Assessment: Planned Improvements

Traci Hale
10 February 2007

The second of two PowerPoint presentations detailing planned improvements to the DHS bioterrorism risk assessment for the 2008 iteration. This was presented after DHS received the Interim Report of the NRC Committee and reflects changes to account for some of the Committee's recommendations.