Architect of the Capitol: Incorporating All Leading Practices Could Improve Accuracy and Credibility of Projects' Cost Estimates

AOC is responsible for the maintenance, renovation, and new construction of the U.S. Capitol complex, which comprises more than four dozen facilities. Reliable cost estimates for projects are crucial to AOC's capital-planning and construction processes. The House Appropriations Committee report accompanying the fiscal year 2014 Legislative Branch Appropriations bill mandated that GAO review AOC's cost-estimating methodology.

This report addresses the extent to which AOC's policies and guidance for developing cost estimates adheres to leading practices. GAO analyzed AOC's cost-estimating guidance, interviewed AOC officials, and compared AOC's cost-estimating guidance and documentation and two projects' cost estimates to leading practices in GAO's Cost Guide . When most or all of the practices associated with each characteristic of a high-quality, reliable estimate are followed, GAO considers the characteristic to be “fully” or “substantially” met. When, in turn, all four characteristics are at least “fully” or “substantially” met, GAO considers a cost estimate to be reliable.

GAO's Cost Estimating and Assessment Guide ( Cost Guide ) defines 12 leading practices that are associated with four characteristics— c omprehensive, well documented, accurate , and credible —that are important to developing high-quality, reliable project-cost estimates. Using the Cost Guide , GAO determined that the Architect of the Capitol's (AOC) cost-estimating guidance conforms to leading practices for developing estimates that are, in general, comprehensive and well-documented . However, AOC's guidance does not substantially conform to leading practices related to developing cost estimates that are accurate and credible . For example, pertaining to the credible characteristic, AOC's guidance does not require determining the confidence level of estimates or quantifying the extent to which a project's costs could vary due to changes in key assumptions. GAO found the strengths and weaknesses of AOC's guidance generally reflected in the cost estimates for AOC's Cannon House Office Building's (Cannon Building) renewal project ($753 million) and Capitol Dome's restoration project ($125 million).

Cannon Building renewal—GAO found the estimate is substantially comprehensive, well documented, and accurate, but several factors that affect its credibility are lacking. For example, AOC's risk analysis does not allow for determination of which risks have the greatest influence on project costs and may overstate the effect of the risks.

Capitol Dome restoration—GAO found the estimate is substantially comprehensive and well documented, but lacking key analysis that support accurate and credible estimates. For example, AOC did not use actual costs from completed phases to update its estimates and did not complete a risk and uncertainty analysis.

Overall, AOC's cost-estimating guidance may not enable fully reliable estimates because it incorporates some, but not all, leading practices. Without reliable cost estimates that convey their confidence levels, AOC's projects risk experiencing cost overruns or budget surpluses, missed deadlines, and performance shortfalls. Potential limitations in the reliability of AOC's estimates may make it difficult for Congress to make well-informed funding decisions and affect how AOC allocates resources across competing projects in its capital portfolio.

Source: GAO analysis of AOC documents and data.

= Fully Meets =Substantially Meets =Partially Meets =Minimally Meets = Does Not Meet

Note: A characteristic is fully met when the associated tasks of underlying leading practices are completely satisfied; substantially met when a large portion of the associated tasks are satisfied; partially met when about half of the associated tasks are satisfied; minimally met when a small portion of the associated tasks are satisfied; and not met when none of the associated tasks are satisfied.

GAO recommends that AOC incorporate additional leading practices into its cost-estimating guidance and submit the confidence levels of project estimates to Congress. AOC concurred with the recommendations and provided context and clarification on its cost-estimating guidance and policies.

The Architect of the Capitol (AOC) is responsible for the maintenance,  renovation, and new construction of the U.S. Capitol complex in  Washington, D. C., which comprises more than four dozen facilities.  Among the major projects involving these facilities, AOC has begun work  restoring the dome atop the U.S. Capitol (Capitol Dome) and designing  the renewal of the Cannon House Office Building (Cannon Building). AOC  currently estimates the Capitol Dome and Cannon Building project costs  to be $125 million and $753 million, respectively.

Reliable cost estimates are crucial for supporting AOC’s capital planning  and construction processes. Without a reliable cost estimate, an AOC  project is at risk of experiencing cost overruns, budget surpluses, missed  deadlines, and performance shortfalls. Furthermore, AOC’s ability to  generate reliable cost estimates is important to informing Congress’  funding decisions and enabling AOC to effectively allocate resources  across competing projects in its capital program. The House  Appropriations Committee report accompanying the fiscal year 2014  Legislative Branch Appropriations Bill (H.R. 2792) mandated that we  review AOC’s cost-estimating methodology to ensure that AOC is  accounting for all of the variables that should contribute to a project’s cost  estimates. This report addresses the extent to which AOC’s policies and  guidance for developing cost estimates conform to leading practices  identified in GAO’s Cost Estimating and Assessment Guide and provide  a reliable basis to support funding and capital program decisions. This  report also examines whether the estimates for the Capitol Dome and the  Cannon Building projects reflect leading practices.

In conducting our study, we reviewed AOC’s policies and guidance for  developing cost estimates and interviewed AOC officials. To determine  the extent to which AOC’s policies and guidance comply with practices for  cost estimating, we compared them to the leading practices set forth in  GAO’s Cost Estimating and Assessment Guide (Cost Guide). The Cost Guide identifies 12 leading practices that represent work across the  federal government and private sector and that have been found to be the  basis for a high-quality, reliable project cost estimate. An estimate created  using the leading practices exhibits four broad characteristics: it is  comprehensive, well documented, accurate and credible. That is, each  characteristic is associated with a specific set of leading practices. In turn,  each leading practice is made up of a number of specific tasks. (See  appendix I for a listing of the tasks that make up each of the 12 leading  practices.) When the tasks associated with the leading practices that  define a characteristic are mostly or completely satisfied, we consider the  characteristic to be “substantially” or “fully” met. When all four  characteristics are at least “substantially met”—we consider a cost  estimate to be reliable.

We also examined AOC’s cost estimates for two major capital projects:  the Cannon Building renewal and the Capitol Dome restoration. We  selected these projects based on their significance to the Capitol  complex, comparatively high costs, and public visibility. In addition, our  review of the Cannon Building’s renewal estimate is responsive to  another mandate in the House Appropriations Committee report  accompanying the fiscal year 2010 Legislative Branch Appropriations Bill  (H.R. 2918) requiring us to monitor the progress of the project.selected versions of the estimates to review that had the most complete  information for our assessments. We assessed the reliability of data used  in developing the estimates and found them to be sufficiently reliable for  the purposes of this report. For example, while some source data were  unavailable to us, we were able to assess AOC’s process for building its  estimates with the data and check for errors. We also interviewed AOC  staff and its technical consultant about these projects and observed  existing conditions at the Cannon Building and Capitol Dome. While our  review of these projects’ cost estimates provides key insights and  illustrates recent products of AOC’s cost-estimating policies and  guidance, the results of our review should not be used to make  generalizations about all AOC project cost estimates.

We conducted our work from September 2013 to March 2014 in  accordance with all sections of GAO’s Quality Assurance Framework that  are relevant to our objectives. The framework requires that we plan and  perform the engagement to obtain sufficient and appropriate evidence to  meet our stated objectives and to discuss any limitations in our work. We  believe that the information and data obtained, and the analysis  conducted, provide a reasonable basis for any findings and conclusions in  this product. Appendix II provides a more detailed description of our  scope and methodology.

The GAO Cost Estimating and Assessment Guide explains how four  characteristics of a high-quality, reliable cost estimate can be understood  in relation to 12 leading practices. The extent to which an agency meets  the leading practices underlying each characteristic determines its   For example, we consider the  performance for that characteristic.comprehensive characteristic to be “substantially met” if the organization  substantially meets the underlying leading practices of (1) developing an  estimating plan and (2) determining an estimating structure. Because the  leading practices are separate and discrete, an agency’s performance in  each of the characteristics can vary. For example, an organization’s cost  estimating methodology could be found to be comprehensive and well  documented, but not accurate or credible, resulting in the organization  producing cost estimates of limited reliability. Table 1 illustrates the  relationship of the 12 leading practices to the four characteristics of a  high-quality, reliable estimate.

AOC project development consists of three stages: planning, design, and  construction. According to AOC’s guidance, AOC refines requirements  and updates cost estimates as projects progress through these stages.  AOC has guidance and requirements for cost estimates at each stage of  a project’s development. In general, as projects develop over time and  requirements are refined, the accuracy of cost estimates is expected to  increase.

Given the current fiscal environment and existing building conditions,  AOC must prioritize projects in its capital program and decide to either  request funding for a project or defer it while mitigating potential facility  issues. For example, in fiscal year 2014, AOC requested almost $155  million for 17 projects that AOC deemed urgent, while deferring 46  projects estimated to cost about $172 million.most recognizable projects that it considers urgent are the Cannon  Building renewal and the Capitol Dome restoration.

The Cannon Building, completed in 1908, is the oldest congressional  office building and is occupied by members of the House of  Representatives and their staffs. (See fig. 1.) The building houses 142  office suites, 5 conference rooms, 4 hearing rooms, and the Caucus  Room, which can accommodate large meetings. The building also  provides space for a library, food service, and a health unit.

According to AOC planning studies, the building is plagued with serious  safety, health, environmental, and operational issues that are worsening.  For example, without action, essential systems for heating and cooling  that are located behind walls and in mechanical rooms will continue to  deteriorate, potentially negatively affecting Members of Congress, staff,  and constituents. AOC has been developing the scope of the Cannon  Building’s renewal project since approximately 2004 when AOC’s  consultant conducted a facility condition assessment (FCA) that identified  the building’s deficiencies. In 2009, we reviewed AOC’s progress in  developing the Cannon Building project and found that while it had  followed a reasonable process to plan the building’s renewal such as  through updating the FCA, it was important that AOC continue, as  planned, to refine the project’s scope and cost estimate.review, AOC has proceeded with the design phase of the project, based   Since our 2009  upon its budget of $753 million for the planning, design, and construction  phases. The project is currently expected to be completed by 2025.

According to AOC officials and current design documents, AOC plans to  correct most of the Cannon Building’s identified deficiencies and to  address requirements such as energy conservation, physical security,  hazardous materials abatement, and historic preservation. The project is  to involve substantial reconfiguration of interior and exterior spaces to  include reconstructing the building’s top floor, which now partially consists  of storage space, and landscaping the courtyard. The project is also  expected to provide refurbished windows and a new roof. Additional work  is intended to preserve and repair the building’s stone exterior. The  project is also expected to allow for complete replacement of all plumbing,  heating and cooling, fire protection, electrical, and alarm systems;  refurbish restrooms and make them more accessible to people with  disabilities; and provide new wall and floor finishes in some areas. In  addition, the project includes removing asbestos that may be contained in  plaster ceilings and walls. AOC plans to conduct the work in phases  corresponding to the four sections of the building and including an initial  phase for utility work as shown in figure 2. Tenants displaced during  construction of each section are to move to temporary offices while other  occupants will remain in building sections not affected by construction.

The U.S. Capitol Dome, an important symbol of American democracy and  an architectural icon, was constructed of cast iron more than 150 years  ago. According to AOC, the dome has not undergone a complete  restoration since 1960, and due to age and weather is now plagued by  more than 1,000 cracks and deficiencies that are causing it to deteriorate.  Figure 3 shows cracks at the exterior column base and deteriorating  interior paint.

The project is intended to stop deterioration in the dome’s cast iron  structure as well as to ensure the protection of the interior of the dome  and rotunda. In the 1960 restoration, the dome was stripped of its paint so  the ironwork could be repaired, primed with a rust inhibitor, and then  repainted. As part of the current project, AOC is undertaking similar  restorative work to include removing old paint, repairing the cast iron, and  repainting.

The project has proceeded in phases as shown in table 2, with phase IIA  restoration work currently in progress. In earlier phases, AOC completed  interim painting, revalidated the project’s design, and restored the base— or skirt—of the dome. To complete the project, AOC is seeking  appropriations of about $20 million for construction and other costs for  phase IIC work in fiscal year 2015. As of February 2014, AOC estimated  the total cost of the Capitol Dome restoration project at about $125  million.

AOC’s Policies and Guidance Conform to Leading Practices for Two Characteristics of Reliable Cost Estimates, but Fall Short in Two Others

As previously discussed, the GAO Cost Estimating and Assessment  Guide defines 12 leading practices related to four characteristics— comprehensive, well documented, accurate, and credible—that are  important to developing high-quality, reliable estimates. Our analysis  determined how well AOC met a characteristic based on our assessment  of AOC’s conformance to the leading practices related to that  characteristic. We discuss characteristics using five rating categories— does not meet, minimally meets, partially meets, substantially meets, or  fully meets. As shown in table 3, our analysis found that AOC’s cost-  estimating policies and guidance contribute to estimates that are  comprehensive and well documented in that AOC fully meets most of the  tasks that underlie the leading practices associated with these two  characteristics. We found that AOC’s cost-estimating policies and  guidance partially met the accuracy characteristic and minimally met the  credible characteristic based on their conformance to the leading  practices associated with those characteristics. Table 3 also contains key  examples of our rationale for our assessment of each leading practice  and characteristic.

Characteristic  Leading Practice  Comprehensive  Develop the estimating plan  AOC has a formal process that develops the estimating plan,  Key examples of rationale for assessment   including describing responsible parties and defining specific tasks.  The estimating structure follows a work breakdown structure (WBS)

AOC uses different types of cost estimates and has defined when  each applies and what each entails.

AOC has formally defined program characteristics.

AOC points to supporting documents that, when developed, should  contain ground rules and assumptions specific to a given project.

AOC guidance is specific about sources of data. These include  commercially available construction cost databases as well as  market and industry knowledge, staff knowledge, and historical data.

AOC Conforms to the Leading Practices Associated with Comprehensive and Well Documented Cost Estimates

In reference to the leading practices underlying the characteristics of a  comprehensive cost estimate, we found that AOC has a formal process for developing estimating plans, and follows an estimating approach incorporating a work breakdown  structure (WBS) that is widely used in the construction industry.

In developing estimating plans, we found that AOC’s policies and  guidance that describe the project-planning process provide the  framework for meeting this leading practice. In the planning stage, AOC  project managers establish project scope and outline roles and  responsibilities for planning, design, and construction stages. To facilitate  these efforts, AOC project managers are to use particular documents.  One document, the Project Development Form, provides a basis for  preparing requirements studies that help to define scope and estimate  costs. AOC then is to use this information in developing Project  Management Plans that are to describe other key components of project  delivery. These components include defining the strategy for providing  design and construction services, identifying project team members,  establishing a communications plan, and setting project controls such for  managing project changes. AOC also considers cost reporting as a  component of the project delivery process. For example, AOC typically requires its design contractors to provide cost reports at the same time  incremental design submittals are made. Because AOC’s project  development process accounts for cost reporting, we determined that it  satisfies the intent of the leading practice encouraging use of a formal  process for developing estimating plans.

With regard to leading practices related to producing well-documented  estimates, we found that AOC  requires formal definitions of programprogram’s purpose;   characteristics, such as the develops ground rules and assumptions from supporting documents; requires the use of cost data sources that are specific to the  construction industry (i.e., obtain the data);  requires documenting the estimate and supporting documents; and  requires management approval of the cost estimate.

AOC fully meets the requirements of most of these leading practices.  For example, AOC’s policies and guidance for the project development  process provide the structure to enable an adequate understanding of  program characteristics—such as key design features, technical  definitions, and the acquisition strategy—that will comprise the cost  estimate. In addition, AOC guidance establishes requirements for use of  industry-accepted cost-data sources and allows for application of staff  knowledge and historical data in developing estimates. AOC guidance  also provides for estimates to be documented to show important  parameters, assumptions, descriptions, methods, and calculations used  to derive the estimate. However, while AOC requires management  approval of its cost estimates, it falls short of fully meeting the  requirements of this leading practice. In particular, AOC’s briefings do not  include the level of information defined by our leading practice,  particularly information related to risks associated with the underlying  data and methods. This is due, in part, to AOC’s policies and guidance  not requiring a risk and uncertainty analysis in developing estimates. We  discuss this leading practice in the following section of this report.

AOC Partially and Minimally Conforms to the Leading Practices Associated with Accurate and Credible Cost Estimates, Respectively

We found that AOC policies and guidance do not require enough detail to  suggest that resulting estimates would be accurate and credible.

In reference to accuracy, we found that AOC’s policies and guidance  partially meet the respective underlying leading practices pertaining to  this characteristic. According to AOC’s policies and guidance, AOC is to  develop estimates as projects progress through planning and design  phases that are based on sufficiently detailed documentation of  construction requirements. AOC is to then use these estimates to support  budget requests for construction funding. However, AOC’s guidance does  not require that cost estimates be updated with actual costs during a  project’s construction phase—a leading practice. According to the Cost  Guide, updating estimates to reflect actual costs as the project  progresses allows agencies to review variances between planned and  actual costs and provides insight as to how the project changed over  time.

In reference to credibility, we found that AOC’s policies and guidance  minimally meet the underlying leading practices pertaining to this  characteristic. For example, we found that AOC’s guidance does not  require  following all steps for conducting a “sensitivity analysis”determining the estimate’s reasonableness, and  conducting a risk and uncertainty analysis.

While AOC guidance does discuss conducting some sensitivity analysis,  it skips many associated tasks of leading practices we identified, such as  identifying key cost drivers, ground rules and assumptions for sensitivity  testing, and evaluating the results to determine which drivers most affect  the cost estimate. Similarly, while AOC guidance provides that  contingencies be added to estimates to account for risk and uncertainty,  AOC’s guidance does not provide documented reasons explaining how  the actual budgeted amounts for unforeseen costs were developed.  AOC officials told us that while their policies and guidance do not require  sensitivity or “quantitative-risk and uncertainty analyses,” they perform  such assessments qualitatively to establish budget contingency.  However, the leading practice to determine whether a program is  realistically budgeted is to perform a quantitative-risk and uncertainty  analysis, so that the probability associated with achieving its point  estimate can be determined. The results of a quantitative-risk and  uncertainty analysis—the range of costs around a point estimate—can be  useful to decision makers because it conveys the confidence level in  achieving the most likely cost and informs them about cost, schedule, and  technical risks. Not having an understanding of an estimate’s  confidence level limits AOC’s ability to determine the appropriate level of  contingency that is needed to address risks and uncertainty for a  particular project and could lead AOC to ineffectively allocate resources  across competing projects if contingency levels are either overstated or  understated. In addition, absent information on estimates’ confidence  levels, Congress will not have critical information for making well-informed  funding decisions.

In addition, pertaining to the leading practice of developing a point  estimate and comparing it to an independent cost estimate (ICE)—which  affects both accurate and credible characteristics—we found that the  independent estimates defined by AOC’s estimating process have a  limited degree of independence and focus solely on proposed contractor  costs rather than on the entire cost estimate, including both government  and contractor efforts. According to our leading practices, an ICE,  conducted by an organization outside the program office, provides an  objective and unbiased assessment of whether the agency’s program  estimate can be achieved. However, because AOC’s project  management and cost-estimating functions are performed within the  same organizational group, the estimates could potentially be influenced  by AOC’s project managers and therefore could be susceptible to bias.

Because of the weaknesses in AOC’s policies and guidance pertaining to  two of the four characteristics that are important to developing high-  quality, reliable estimates, the project cost estimates that AOC produces  may not always be reliable. Without reliable cost estimates, AOC’s  projects risk experiencing cost overruns or budget surpluses, missed  deadlines, and performance shortfalls. Furthermore, potential limitations  in the reliability of its estimates may impair Congress’s ability to make  well-informed funding decisions and affect how AOC allocates resources  across competing projects in its capital program.

Strengths and Weaknesses in AOC’s Policies and Guidance Are Reflected in the Cannon Building and Capitol Dome Projects

In comparing the Cannon Office Building’s renewal and Capitol Dome’s  restoration cost estimates to GAO’s leading practices, we found strengths  and weaknesses that generally correspond to our assessment of AOC’s  overall policies and guidance for developing cost estimates. We initially  determined that both estimates were comprehensive while lacking, to a  varying extent, in their documentation, accuracy, and credibility.  Following our initial determination, AOC provided further documentation  that resulted in improvements to each estimate’s assessment. Our final  assessment found that  the Cannon Building renewal estimate to be substantially  comprehensive, well documented, and accurate while lacking in  elements affecting its credibility, and  the Capitol Dome restoration estimate to be substantially  comprehensive and well documented while lacking in areas pertaining  to accuracy and credibility.

In aggregate, because of weaknesses pertaining to characteristics that  are important to the development of high-quality, reliable estimates,  AOC’s cost estimates for the Cannon Building’s renewal and Capitol  Dome’s restoration may not be fully reliable. Appendix III provides  additional details on our comparison of these two cost estimates to our  leading practices.

Appendix III provides additional information on our assessment of these estimates  compared to the characteristics of high-quality, reliable estimates. and allows for tracking of cost and schedule performance by defined  elements of work. In addition, AOC conducted a life-cycle-cost analysis of  the heating and cooling system alternatives it considered to include  analyzing their energy consumption for the purpose of comparing  operational costs. However, the estimate did not fully meet the  characteristic as it did not include full life-cycle costs, from inception  through design, construction, operation, and maintenance. Without a life- cycle cost analysis that captures the total cost of the project, AOC cannot  evaluate design alternatives on a total-cost basis.

In terms of being well documented, we found that the estimate  substantially met associated leading practices. In general, the information  provided by AOC describes how the estimate was built up from  engineering drawings, specifications, and design documents. In addition,  there was evidence of documented management approval. However, the  documentation AOC provided did not contain source data, such as from  contractor bids or cost estimating databases. While AOC officials said  they could obtain this information for our review, to the extent that it was  available, our leading practices indicate that an estimate’s  documentation should be detailed enough so that the derivation of each  cost element can be traced to all sources allowing for the estimate to be  easily replicated and updated. Because AOC could not readily provide the  actual source data for the estimate, we determined that it did not fully  meet the requirements of this characteristic.

For the accuracy characteristic, we found the estimate substantially met  leading practices. AOC’s acquisition approach to the building’s renewal  involves its contracting with an architect, construction manager, and  construction contractor, each of whom produced separate estimates.  These estimates have a limited degree of independence because they  were conducted for the same program office and focus only on the  proposed contractor cost. However, this array of estimates has enabled  AOC to make comparisons among them, determine similarities and  differences, and develop a reasonably accurate assessment of estimated  costs.

For the credibility characteristic, we found the estimate partially met  leading practices. During the course of our review, AOC conducted a risk  and uncertainty analysis in accordance with a key leading practice for this  characteristic.  However, we found several issues affecting the quality of  the analysis that AOC provided to us. For example, AOC’s analysis  concluded that the Cannon Building’s renewal estimate had a confidence  level that exceeded 90 percent—meaning there is a greater than a 90  percent probability that actual costs will be equal to or less than AOC’s  estimate—which may be unreasonably high. However, an AOC official  said that the agency will be reconsidering the confidence level once the  design progresses further. In addition, we found that the method AOC  used to model the project’s risks it identified (1) resulted in an unusually  narrow range of estimated costs across the confidence intervals and (2)  provides managers limited ability to understand the effects of individual  risks.opposed to modeling them separately, AOC cannot identify relationships  between risk elements and determine which risks have the greatest  influence on project costs. As a result, AOC is limited in its ability to  manage the risks. In addition, modeling the aggregated risks likely  contributes to the overly narrow range of estimated costs over the   Because AOC aggregated risks for the purposes of its analysis as  confidence levels, which implies that AOC’s analysis overstates the effect  of the risks.

In reference to the comprehensive characteristic, we found that the  estimate substantially met leading practices. In particular, the estimate  uses an industry-standard WBS format and contains sufficient detail of  the technical characteristics of the project. However, the estimate did not  fully meet leading practices, in part because we did not find a  consolidated list of ground rules and assumptions or descriptions of how  these affected the estimate. Contract documents provide indications of  ground rules and assumptions affecting the project. For example, contract  specifications and drawings for the recently-awarded phase II work  describe assumptions pertaining to availability of funding for contract  options and accessibility of the site during the workday. However, we  could not determine if these represented a comprehensive assessment of  all ground rules and assumptions. According to the Cost Guide, because  ground rules and assumptions can significantly affect cost and introduce  risk, it is important that they are clearly documented in the estimate to  enable areas of potential risk to be identified and resolved.

For the well documented characteristic, we determined the estimate as  having substantially met leading practices. We found, for example that  cost elements from the estimate can be compared to information in the  drawings and specifications that define the contract for the phase II  restoration. This linkage enables a good understanding of key  characteristics of the estimate. In addition, AOC attested that the estimate  had been briefed to and approved by management. However, because  records describing briefings of the estimate to management are not well  documented, it is difficult to trace management’s recommendations for  changes, feedback, and key decisions affecting the project. While AOC  officials told us that its project staff and management communicate  routinely as part of their normal business functions, documenting  management briefings is important because some key personnel have  changed over the life of the Capitol Dome restoration project, which  originated more than 15 years ago. By not having well documented  records of management briefings over the life of the project, AOC risks  loss of continuity in its oversight.

Considering the accuracy characteristic, we determined that the estimate  partially met leading practices. We found, for example, that the estimating  software and database used to construct the estimate included standard  information, such as costs for labor, materials, and equipment. However,  AOC did not consider the actual cost of previously completed restoration  phases in updates to its estimate for the phase II work. While AOC used  information on the price it paid for certain items in the phase I restoration  work that are also part of the phase II project, it did not determine the  contractor’s actual cost to perform the work. AOC officials said that  having information on actual contractor costs for items may be of limited  usefulness because market conditions and other factors determine what  contractors bid and AOC ultimately pays. However, not having  information on actual costs makes it difficult for AOC to assess the  difference between the price it paid and its contractors’ costs to determine  the estimate’s reasonableness.

For the credibility characteristic, we determined that the estimate  minimally met leading practices. We found, in contrast to the approach it  took in developing the Cannon Building renewal estimate, that AOC did  not conduct a risk and uncertainty analysis of the Capitol Dome  restoration estimate. While AOC officials told us they have not conducted  a quantitative-risk and uncertainty analysis of the estimate, they said they  have taken steps to qualitatively assess and mitigate project risks. For  example, AOC structured the phase II contract solicitation to have  contractors include a base bid (phase IIA) and pre-priced options for later  stages (phases IIB and IIC) of the work. According to AOC officials, by  structuring the contract this way, AOC is protected from price escalations  affecting the future phases. In addition, AOC established unit prices and  quantity allowances for some restoration tasks that are of indefinite  quantity, such as repairing cracks. According to AOC officials, this will  help to control project costs because, while their quantity estimates may  be imprecise, knowing the unit prices associated with the work should  mitigate some cost risk. While these actions are encouraging, they are  not documented within the context of a quantitative-risk and uncertainty  analysis. As a result, we do not know how these actions relate to other  risk mitigation efforts that may have been considered and why AOC  chose these actions over other efforts. Moreover, not having a  quantitative-risk and uncertainty analysis precludes establishing a level of  confidence associated with achieving the estimated cost and limits AOC  management’s ability to determine an appropriate level of contingency  reserves that may be necessary to address risks that AOC has identified.

AOC project’s cost-estimating guidance policy and two resulting project  estimates that we reviewed may not be fully reliable because they  incorporated some, but not all, leading practices in cost estimating.  Because AOC’s project cost estimates inform Congress’s funding  decisions and affect AOC’s ability to effectively allocate resources across  competing projects in its capital program, there is a risk that funding  decisions and resource allocations could be made based on information  that is not reliable. We recognize that incorporating GAO’s cost- estimating best practices into AOC’s cost-estimating policy and guidance  may involve additional costs—such as for conducting a risk and  uncertainty analysis for projects and conveying the confidence level of the  estimate to Congress and AOC managers. However, without investing in  these practices, Congress risks making funding decisions and AOC  management risks making resource allocation decisions without the  benefit that a robust analysis of levels of risk, uncertainty, and confidence  provide decision makers.

To improve the Architect of the Capitol’s project-cost-estimating process,  enhance the transparency of its related process, and allow for more  informed decision making related to projects’ costs, we recommend that  the Architect of the Capitol take the following two actions, to the extent  that the benefits exceed the costs:  incorporate leading practices we identified as lacking for cost  estimating in the AOC’s cost-estimating guidance and policies, and  for ongoing and future projects, submit the confidence level derived  from risk and uncertainty analyses along with budget documentation  to appropriate congressional decision makers, so that Congress is  aware of the range of likely costs and AOC’s associated confidence  levels.

We provided a draft of this report to the Architect of the Capitol for review  and comment. AOC agreed with our recommendations and provided us  with additional context and information on specific actions that the AOC  has taken or intends to take to more fully address our recommendations.  For example, AOC said that it planned to revise its policies and  procedures to require quantitative-risk and uncertainty analysis be done,  as specified in the Cost Guide, for high-dollar-value projects prior to  requesting construction funding. In addition, AOC said that it would  explore the most effective approach for communicating to congressional  decision makers the confidence level derived from risk and uncertainty  analyses along with budget documentation. We made no changes to our  draft based upon AOC’s comments. AOC’s comments are reprinted in  appendix IV, followed by our response to AOC’s detailed comments.

We are sending copies of this report to appropriate congressional  committees and the Architect of the Capitol. In addition, the report is  available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-2834 or stjamesl@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix V.

Lorelei St. James  Director, Physical Infrastructure Issues.

Appendix I: The Leading Practices for Developing High-Quality Cost Estimates and Associated Tasks

Determine estimate’s purpose, required level of detail, and overall scope;

Determine who will receive the estimate.

Determine the cost-estimating team and develop its master schedule;

Determine who will do the independent cost estimate;

In a technical baseline-description document, identify the program’s purpose and its  system and performance characteristics and all system configurations;  Any technology implications;  Its program acquisition schedule and acquisition strategy;  Its relationship to other existing systems, including predecessor or similar legacy  systems;  Support (manpower, training, etc.) and security needs and risk items;  System quantities for development, test, and production;

Define a work breakdown structure (WBS) and describe each element in a WBS  dictionary (a major automated-information system may have only a cost element  structure);

Choose the best estimating method for each WBS element;  Identify potential cross-checks for likely cost and schedule drivers;

Clearly define what the estimate includes and excludes;  Identify global and program-specific assumptions, such as the estimate’s base year,  including time phasing and life cycle;  Identify program schedule information by phase and program acquisition strategy;  Identify any schedule or budget constraints, inflation assumptions, and travel costs;  Specify equipment the government is to furnish as well as the use of existing facilities  or new modification or development;  Identify prime contractor and major subcontractors;

Determine technology refresh cycles, technology assumptions, and new technology to

Define commonality with legacy systems and assumed heritage savings;

Describe effects of new ways of doing business.

Create a data collection plan with emphasis on collecting current and relevant  technical, programmatic, cost, and risk data;  Investigate possible data sources;

Collect data and normalize them for cost accounting, inflation, learning, and quantity  adjustments;  Analyze the data for cost drivers, trends, and outliers and compare results against  rules of thumb and standard factors derived from historical data;  Interview data sources and document all pertinent information, including an  assessment of data reliability and accuracy;  Store data for future estimates.

Develop point estimate and  compare it to an independent cost  estimate

Develop the cost model, estimating each WBS element, using the best methodology  from the data collected, and including all estimating assumptions;  Express costs in constant year dollars;  Time-phase the results by spreading costs in the years they are expected to occur,  based on the program schedule;  Sum the WBS elements to develop the overall point estimate;  Validate the estimate by looking for errors like double counting and omitted costs;

Compare estimate against the independent cost estimate and examine where and why  there are differences;  Perform cross-checks on cost drivers to see if results are similar;

Update the model as more data become available or as changes occur and compare  results against previous estimates.  Test the sensitivity of cost elements to changes in estimating input values and key  assumptions;  Identify effects on the overall estimate of changing the program schedule or quantities;

Determine which assumptions are key cost drivers and which cost elements are  affected most by changes.

Determine and discuss with technical experts the level of cost, schedule, and technical  risk associated with each WBS element;  Analyze each risk for its severity and probability;

Develop minimum, most likely, and maximum ranges for each risk element;

Determine type of risk distributions and reason for their use;  Ensure that risks are correlated;

Use an acceptable statistical analysis method (e.g., Monte Carlo simulation) to  develop a confidence interval around the point estimate;  Identify the confidence level of the point estimate;  Identify the amount of contingency funding and add this to the point estimate to  determine the risk-adjusted cost estimate;

Recommend that the project or program office develop a risk management plan to  track and mitigate risks.

Document all steps used to develop the estimate so that a cost analyst unfamiliar with  the program can recreate it quickly and produce the same result;

Document the purpose of the estimate, the team that prepared it, and who approved  the estimate and on what date;

Describe the program, its schedule, and the technical baseline used to create the  estimate;  Present the program’s time-phased life-cycle cost;

Discuss all ground rules and assumptions;  Include auditable and traceable data sources for each cost element and document for  all data sources how the data were normalized;

Describe in detail the estimating methodology and rationale used to derive each WBS  element’s cost (prefer more detail over less);

Describe the results of the risk, uncertainty, and sensitivity analyses and whether any  contingency funds were identified;

Document how the estimate compares to the funding profile;  Track how this estimate compares to any previous estimates.

Present estimate to management  for approval

Develop a briefing that presents the documented life-cycle cost estimate;  Include an explanation of the technical and programmatic baseline and any  uncertainties;

Compare the estimate to an independent cost estimate (ICE) and explain any

Compare the estimate (life-cycle cost estimate (LCCE)) or independent cost estimate  to the budget with enough detail to easily defend it by showing how it is accurate,  complete, and high in quality;  Focus in a logical manner on the largest cost elements and cost drivers;

Make the content clear and complete so that those who are unfamiliar with it can  easily comprehend the competence that underlies the estimate results;

Make backup slides available for more probing questions;  Act on and document feedback from management;

Update the estimate to reflect changes in technical or program assumptions or keep it  current as the program passes through new phases or milestones;

Replace estimates with EVM, and independent estimate at completion (EAC) from the

Report progress on meeting cost and schedule estimates;  Perform a post mortem and document lessons learned for elements whose actual  costs or schedules differ from the estimate;

Document all changes to the program and how they affect the cost estimate.

Appendix II: Objectives, Scope, and Methodology

The House Appropriations Committee report accompanying the fiscal  year 2014 Legislative Branch Appropriations Bill (H.R. 2792) mandated  that we review the Architect of the Capitol’s (AOC) cost estimating  methodology to ensure that AOC is accounting for all of the variables that  should contribute to project cost estimates. This report addresses the  extent to which AOC’s policies and guidance for developing cost  estimates conform to leading practices identified in GAO’s Cost  Estimating and Assessment Guide and provide a reliable basis to  support funding and capital program decisions. This report also examines  whether the estimates for the Capitol Dome and the Cannon Building  projects reflect leading practices.

To determine the extent to which AOC’s policies and guidance comply  with leading practices for cost estimating and provide a reliable basis to  support funding and program decisions, we compared AOC’s documents  to leading practices set forth in GAO’s Cost Estimating and Assessment  Guide. The Cost Guide identifies 12 leading practices that represent  work across the federal government and are the basis for a high-quality,  reliable cost estimate. An estimate created using the leading practices  exhibits four broad characteristics: it is accurate, well documented,  credible, and comprehensive. That is, each characteristic is associated  with a specific set of leading practices. In turn, each leading practice is  made up of a number of specific tasks. (See app, I for a listing of the  tasks that make up each of the 12 leading practices.) When the tasks  associated with the leading practices that define a characteristic are  mostly or completely satisfied, we consider the characteristic to be  “substantially” or “fully” met. When all four characteristics are at least  “substantially met”—we consider a cost estimate to be reliable.shared our analysis with AOC officials to review, comment, and provide  additional information, and we adjusted our analysis where appropriate.

In reference to our examination of the estimates for the Capitol Dome and  Cannon Building projects, we selected these projects based on their  significance to the Capitol Complex, comparatively high costs, and public  visibility. In addition, our review of the Cannon Building estimate is  responsive to a mandate in the House Appropriations Committee report  accompanying the fiscal year 2010 Legislative Branch Appropriations Bill  (H.R. 2918) requiring us to monitor the progress of the project.selected versions of the estimates to review that had the most complete  information for our assessments. We assessed the reliability of data used  in developing the estimates and found them to be sufficiently reliable for  the purposes of this report. For example, while some source data were  unavailable to us, we were able to assess AOC’s process for building its  estimates with the data and check for errors. We also interviewed AOC’s  staff and its technical consultant about these projects and observed  existing conditions at the Cannon Building and Capitol Dome. While our  review of these projects’ cost estimates provides key insights and  illustrates products of AOC’s cost-estimating policies and guidance, the  results of our review should not be used to make generalizations about all  AOC project-cost estimates. We shared our analysis with AOC officials to  review, comment, and provide additional information, and we adjusted our  analysis where appropriate.

Appendix III: GAO’s Summary Assessments of AOC’s Project Cost Estimates for the Cannon Building’s Renewal and Capitol Dome’s Restoration

Appendix III: GAO’s Summary Assessments of  AOC’s Project Cost Estimates for the Cannon  Building’s Renewal and Capitol Dome’s  Restoration Estimating leading practice characteristic and rationale for assessment   Comprehensive  The cost estimate should include both government and contractor costs of the program over its full life cycle, from  inception of the program through design, development, deployment, operation and maintenance, to retirement of the  program. It should also completely define the program, reflect the current schedule, and be technically reasonable.  Comprehensive cost estimates should be structured in sufficient detail to ensure that cost elements are neither  omitted nor double counted. Specifically, the cost estimate should be based on a product-oriented work breakdown  structure (WBS) that allows a program to track cost and schedule by defined deliverables, such as hardware or  software components. Finally, where information is limited and judgments must be made, the cost estimate should  document all cost-influencing ground rules and assumptions.

The Cannon Building cost estimate did not include all life cycle costs as AOC guidance did not require these life  cycle costs to be identified. The estimate tracked well with the work required, as it had a WBS structure consistent  with this type of effort, and clearly laid out and appeared to update ground rules and assumptions based on evolving  understanding of the scope of work.

The Capitol Dome cost estimate covers the construction phase of the dome’s restoration. The cost estimate relies on  specifications and drawings that exist in separate files. The cost estimate uses a standard WBS structure but does  not include the underlying data. AOC submitted an extract of the data for our review. While the specifications include  ground rules and assumptions, the cost estimate does not specify them.

Well documented  A good cost estimate—while taking the form of a single number—is supported by detailed documentation that  describes how it was derived and how the expected funding will be spent in order to achieve a given objective.  Therefore, the documentation should capture in writing such things as the source data used, the calculations  performed and their results, and the estimating methodology used to derive each WBS element’s cost. Moreover, this  information should be captured in such a way that the data used to derive the estimate can be traced back to, and  verified against, their sources so that the estimate can be easily replicated and updated. The documentation should  also discuss the technical baseline description and how the data were normalized. Finally, the documentation should  include evidence that the cost estimate was reviewed and accepted by management.

The Cannon Building cost estimate documentation did not provide source data. It did provide a reasonable  explanation for how the cost estimates were prepared, what type of data were used, and the basic estimating  methodologies indicating some step by step processes. The documentation provided consisted of tables of unit  values, quantities, and extended values. While extended costs were identified, it was difficult to follow the roll-up of  these costs. Without a copy of or access to the cost model, we were unable to verify the accuracy of escalation  calculations and we could not trace all of the logic from the source data to the resulting estimate. The calculations  and changes to the baseline were not well documented but subsequent data and explanations highlighted the  process for the changes. Additionally, soft costs and contingencies, comprising a substantial portion of total program  costs, were only superficially addressed. There was evidence of management approval of the estimate.

The Capitol Dome cost estimate captures all source documents used. The cost estimate relies on a WBS and shows  how the calculations were performed. A cost analyst unfamiliar with the program could develop the cost estimate,  although in places it is not clear. The cost estimate has a brief reference to the source documents used. Costs are  presented in constant-year and then-year dollars.

Estimating leading practice characteristic and rationale for assessment   Accurate  The cost estimate should provide for results that are unbiased, and it should not be overly conservative or optimistic.  An estimate is accurate when it is based on an assessment of most likely costs, adjusted properly for inflation, and  contains few, if any, minor mistakes. In addition, a cost estimate should be updated regularly to reflect significant  changes in the program—such as when schedules or other assumptions change—and actual costs, so that it is  always reflecting current status. During the update process, variances between planned and actual costs should be  documented, explained, and reviewed. Among other things, the estimate should be grounded in a historical record of  cost estimating and actual experiences on other comparable programs.

The Cannon Building cost estimate included an uncertainty analysis indicating what confidence level the budget fell  within, which was greater than 90 percent. Additionally, in the absence of having the cost model used to prepare the  estimate, we were only able to validate the accuracy of a very small sample of WBS elements. Furthermore, while  the estimate documentation identified inflation adjustments, because of missing calculation and conversion factors,  we were unable to determine if the estimate has been adjusted properly for inflation. However, the documentation did  provide a discussion of what and how the inflation adjustments were applied.

The Capitol Dome cost estimate did not include a risk and uncertainty analysis indicating what confidence level for  the estimate the budget was set at. The estimate is shown in constant-year and then-year dollars using indices for  inflation. The indices used to adjust for inflation appear to be out-of-date. The estimate has few errors. AOC updated  its cost estimate prior to beginning the project, but once the project began, AOC did not update the estimate. AOC  did not document variances between planned and actual costs. AOC relied on historical records and on an industry  database to support its estimate. The estimate uses an engineering build-up throughout, which appears appropriate  for this project.  Credible  The cost estimate should discuss any limitations of the analysis because of uncertainty or biases surrounding data or  assumptions. Major assumptions should be varied, and other outcomes recomputed to determine how sensitive they  are to changes in the assumptions. Risk and uncertainty analysis should be performed to determine the level of risk  associated with the estimate. Further, the estimate’s cost drivers should be crosschecked, and an independent cost  estimate conducted by a group outside the acquiring organization should be developed to determine whether other  estimating methods produce similar results.

The Cannon Building cost estimate includes analyses of sensitivity, risk, and uncertainty. However, the sensitivity  analysis was performed after the fact and was not used to inform the budget or provided for management  consideration of risk prior to project start. AOC’s risk and uncertainty analysis indicates that the budget for the project  is set above the 90% confidence level, which may be unreasonably high. AOC has received multiple estimates from  different sources (architect, construction manager, pre-construction contractor) as a cross-check for reasonableness.  While these estimates are not necessarily outside the AOC program office’s influence, they do appear to represent  independent estimates based on each company’s interpretation of the project’s requirements and helped to validate  the reliability of the estimates.

Appendix IV: Comments from the Architect of the Capitol

1.  We continue to believe that a quantitative risk and uncertainty  analysis enables more effective oversight than can be obtained by  taking a qualitative approach. A quantitative risk and uncertainty  analysis conveys the confidence level in achieving the most likely cost  and provides actionable information about cost, schedule, and  technical risks that cannot be obtained qualitatively.  2.  While AOC’s contractors act independently, AOC’s Project  Management Division is responsible for the estimates prepared via  contract, and AOC’s Cost Estimating Group is responsible for  reviewing them. Because AOC’s Project Management Division and  Cost Estimating Group staff are part of the same Planning and Project  Management organization, the Cost Estimating Group staff could be  biased because of organizational influences and unable to provide a  fully objective review of costs to the Project Management Division.   3.  While our leading practices do not specify target confidence levels,  experts we consulted with in developing our leading practices agreed  that program cost estimates should be budgeted to at least the 55  percent confidence level and potentially as high as 80 percent. While  AOC has good reasons for its interest in maintaining the project’s  budget at a high confidence level, we believe that the confidence level  for the project’s budget—exceeding 90 percent—may be excessive.    4.  AOC identified 60 risks to the project and modeled them in aggregate,  as opposed to individually, in conducting its quantitative risk and  uncertainty analysis. Modeling the aggregated risks precludes AOC  from identifying relationships among risk elements and determining  which risks have the greatest influence on project costs. We continue  to believe that modeling the risks in aggregate likely contributes to the  overly narrow range of estimated costs over the confidence levels,  and can overstate the effect of the risks.  5.  Our leading practices indicate that an estimate’s documentation  should be detailed enough so that the derivation of each cost element  can be traced to all sources allowing the estimate to be easily  replicated and updated. Because some information was not readily  available in documents maintained by AOC, we were unable to  determine if the estimate has been adjusted properly for inflation.   6.  AOC has taken positive steps in conducting a life-cycle cost analysis  that considered fixed costs and ongoing maintenance costs to inform  the AOC’s selection of mechanical systems for heating and cooling  the Cannon Building. However, AOC did not conduct life-cycle  analyses for other components of the project, such as the building’s  roof and alarm system. This precludes AOC from capturing the total  cost of the project and evaluating design alternatives on a total-cost  basis.   7.  AOC takes positive steps to qualitatively identify risks and uncertainty  and assess sensitivity. While this analysis is useful in setting  contingencies and managing risks during construction, a quantitative  risk and uncertainty analysis enables more effective oversight  because it conveys the confidence level in achieving the most likely  cost and provides actionable information about cost, schedule, and  technical risks that cannot be obtained qualitatively.   8.  While AOC has used some information from completed phases of the  Capitol Dome’s restoration project to inform development of the  estimate for later phases, not having information on actual costs  makes it difficult for AOC to assess the difference between the price it  paid and its contractors’ costs to determine the estimate’s  reasonableness.  9.  Our leading practices indicate that an estimate’s documentation  should be detailed enough so that the derivation of each cost element  can be traced to all sources allowing the estimate to be easily  replicated and updated. Because some information was out-of-date at  the time of our review, we were unable to determine if the estimate  had the proper escalation adjustments.

Appendix V: GAO Contact and Staff Acknowledgments

In addition to the contact named above Michael Armes, Assistant  Director; Karen Richey, Assistant Director; George Depaoli, Analyst-in- Charge; Laura Erion; Emile Ettedgui; Colin Fallon; Geoffrey Hamilton;  James Manzo; Faye Morrison; and Vanessa Welker made key  contributions to this report.

Question: What does GAO's Cost Guide contain?

Answer: GAO's Cost Estimating and Assessment Guide ( Cost Guide ) defines 12 leading practices that are associated with four characteristics— c omprehensive, well documented, accurate , and credible —that are important to developing high-quality, reliable project-cost estimates.

Question: What does the GAO say about the AOC?

Answer: Using the Cost Guide, GAO determined that the Architect of the Capitol's (AOC) cost-estimating guidance conforms to leading practices for developing estimates that are, in general, comprehensive and well-documented.

Question: What flaws did GAO find in AOC's guidance?

Answer: However, AOC's guidance does not substantially conform to leading practices related to developing cost estimates that are accurate and credible .

Question: What are some examples of AOC's failure to conform to leading practices?

Answer: For example, pertaining to the credible characteristic, AOC's guidance does not require determining the confidence level of estimates or quantifying the extent to which a project's costs could vary due to changes in key assumptions. GAO found the strengths and weaknesses of AOC's guidance generally reflected in the cost estimates for AOC's Cannon House Office Building's (Cannon Building) renewal project ($753 million) and Capitol Dome's restoration project ($125 million).

AOC’s Policies and Guidance Conform to Leading Practices for Two Characteristics of Reliable Cost Estimates, but Fall Short in Two Others

AOC Partially and Minimally Conforms to the Leading Practices Associated with Accurate and Credible Cost Estimates, Respectively

Strengths and Weaknesses in AOC’s Policies and Guidance Are Reflected in the Cannon Building and Capitol Dome Projects

Appendix II: Objectives, Scope, and Methodology

Question: Why is the AOC's cost estimating guidance not fully reliable?

Answer: Overall, AOC's cost-estimating guidance may not enable fully reliable estimates because it incorporates some, but not all, leading practices.

Question: How do unreliable cost estimates affect AOC's projects?

Answer: Without reliable cost estimates that convey their confidence levels, AOC's projects risk experiencing cost overruns or budget surpluses, missed deadlines, and performance shortfalls.

Question: How does unreliable estimates affect Congress?

Answer: Potential limitations in the reliability of AOC's estimates may make it difficult for Congress to make well-informed funding decisions and affect how AOC allocates resources across competing projects in its capital portfolio.

Question: What does the report focus on?

Answer: This report addresses the extent to which AOC's policies and guidance for developing cost estimates adheres to leading practices.

Question: How did GAO examine AOC’s guidance?

Answer: GAO analyzed AOC's cost-estimating guidance, interviewed AOC officials, and compared AOC's cost-estimating guidance and documentation and two projects' cost estimates to leading practices in GAO's Cost Guide .

Question: How can a characteristic be fully or substantially met, according to GAO?

Answer: When most or all of the practices associated with each characteristic of a high-quality, reliable estimate are followed, GAO considers the characteristic to be “fully” or “substantially” met.

Question: How can a cost estimate be fully reliable?

Answer: When, in turn, all four characteristics are at least “fully” or “substantially” met, GAO considers a cost estimate to be reliable.

Background
Cost-Estimating Leading Practices