Regulatory Guidance Processes: Selected Departments Could Strengthen Internal Control and Dissemination Practices

Agencies rely on guidance to clarify regulatory text or statutes, to respond to the questions of affected parties in a timely way, and to inform the public about complex policy implementation topics. Unlike regulations, guidance is not legally binding.

GAO was asked to examine guidance processes at four departments. This report reviews how (1) agencies use guidance and decide to issue guidance rather than regulations; (2) follow applicable criteria and leading practices in their policies, procedures, and practices for producing guidance; and (3) agencies disseminate guidance to ensure public access and feedback. GAO reviewed guidance processes at all 25 components in the four departments that (1) were within the requesting committee's jurisdiction, and (2) engaged in regulatory or grant activities. GAO reviewed relevant requirements, written procedures, guidance and websites, and interviewed agency officials.

The four departments—Agriculture (USDA), Education (Education), Health and Human Services (HHS), and Labor (DOL)—and their selected components used guidance for multiple purposes, such as clarifying or interpreting regulations and providing grant administration information. The terminology used for agency guidance varied and agency components issued varying amounts of guidance, ranging from about 10 to over 100 guidance documents each year. The key criterion used when deciding whether to issue a regulation or guidance was whether it needed to be binding; in such cases agencies proceeded with regulation. Officials reported that they routinely consulted with legal counsel when making these choices. Departments typically identified few of their guidance documents as “significant,” generally defined by the Office of Management and Budget (OMB) as guidance with a broad and substantial impact on regulated entities.

All four departments identified standard practices to follow when developing guidance. They addressed OMB's requirements for significant guidance to varying degrees and could strengthen internal controls for issuing guidance. Education and USDA had written departmental procedures for approval of significant guidance as required by OMB. DOL's procedures were not available to staff and required updating. HHS had no written procedures. Ensuring these procedures are available could better ensure that components consistently follow OMB's requirements. In the absence of specific government standards for non-significant guidance—the majority of issued guidance—the application of internal control standards is particularly important. The 25 components GAO reviewed addressed some control standards more regularly than others. For example, few components had written procedures to ensure consistent application of guidance processes. All components could describe standard review practices and most used tools to document management approval of draft guidance. Of the 25 components, 15 cited examples in which they conferred with external nonfederal stakeholders while developing guidance and nearly half did not regularly evaluate whether issued guidance remained current and effective.

Components used different strategies to disseminate guidance and all relied primarily on posting the guidance on their websites. As such, components should follow applicable requirements for federal websites. One of these requirements—easy access to current and relevant guidance—could also facilitate opportunities for affected parties and stakeholders to provide feedback on those documents. USDA, DOL, and Education posted their significant guidance on a departmental website as directed by OMB; HHS did not. Components used several strategies—including organizing guidance by audience or topic and highlighting new or outdated guidance—to facilitate access. However, GAO identified factors that hindered online access, including long lists of guidance and documents dispersed among multiple web pages. All components GAO studied collected web metrics and many used them to evaluate online guidance dissemination. However, many of these components did not use metrics to improve how they disseminated guidance through their websites. Beyond their websites, components found other ways to disseminate and obtain feedback on issued guidance, including focus groups, surveys, and direct feedback from the public at conferences, webinars, and from monitoring visits.

GAO is recommending that HHS and DOL ensure consistent application of OMB requirements for significant guidance. GAO also recommends that USDA, Education, HHS, and DOL strengthen the use of internal controls in guidance production processes and improve online guidance dissemination. USDA, Education, HHS and DOL generally agreed with the recommendations.

Guidance documents, sometimes referred to as sub-regulatory guidance,  set forth policy on or interpret statutory, regulatory, or technical issues  and come in a variety of formats and names. Agencies rely on guidance  documents—which are not legally binding—to clarify statutes or  regulatory text and to inform the public about complex policy  implementation topics. The number of guidance documents related to  regulations issued by agencies has often been reported to outnumber the  agency regulations that these documents can help explain. However, little  is known about internal agency procedures for developing this guidance  and the strategies agencies use to solicit public comment and feedback  on these documents.

Congress and affected parties often request additional guidance to  elaborate on implementation of a statute or regulation. At the same time,  concerns have been raised about the level of oversight for agencies’  guidance, whether agencies seek feedback from affected parties on  guidance, and how to ensure that agencies do not issue guidance when  they should undertake rulemaking. guidance and the concerns about its use, in 2007 the Office of  Management and Budget (OMB) recognized the need for good guidance  practices. In particular, OMB established review processes for the  documents with the most broad and substantial impact.

Regulations are legally binding and typically require a desired action or prohibit certain  actions by regulated parties. Guidance documents are not legally binding and are often  used by agencies to explain how they plan to interpret a regulation or provide additional  clarifications.

You asked us to examine guidance processes at the four departments  under your jurisdiction. For this report, we reviewed how the Departments  of Agriculture (USDA), Education (Education), Health and Human  Services (HHS), Labor (DOL), and selected subagencies or components  at these four departments (1) use guidance and the processes and  criteria they use to determine whether to issue guidance or undertake  rulemaking; (2) follow applicable criteria and leading practices in their  policies, procedures, and practices for producing guidance; and (3)  ensure they follow dissemination requirements and facilitate end users’  access to and comment on documents.

The scope of this review is limited to guidance issued to external parties  and includes guidance explaining regulatory and grant-related  requirements and other policies. We reviewed guidance processes at the  4 departments and 25 of their selected subagencies, or components, that  (1) were within the requesting committee’s jurisdiction and (2) engaged in  regulatory or grantmaking activities. To identify these components, we  searched the Federal Register and the Catalogue of Federal Domestic  Assistance to identify regulatory and grantmaking activities, and  confirmed the resulting list of components with department officials. For  the purposes of this report, “agencies” refers to both departments and  department components, and will be used when describing both. See  table 1 below for the full list of audited departments and components.

To describe how selected departments and components used guidance  and the processes and criteria they used to determine whether to issue  guidance or undertake rulemaking, we reviewed agencies’ written  procedures, guidance documents, and websites. We also interviewed  department and component officials on guidance practices. To develop  themes and examples for all objectives, we analyzed information from  relevant agency documents and interviews to identify and confirm  common patterns as well as differences across selected agencies. We  determined a data reliability assessment was not necessary because  computer-processed data were not expected to materially affect our  findings. To assess the extent to which agencies adhered to requirements  for written procedures for approval of significant guidance under the  Office of Management and Budget’s (OMB) Final Bulletin for Agency  Good Guidance Practices (OMB Bulletin), we reviewed applicable agency  written procedures and agency websites and spoke to officials about their  practices for development and review of significant guidance.

We selected and applied four government-wide internal control standards  to guidance processes for non-significant guidance and identified both  opportunities for improved internal controls related to guidance and  highlighted agency practices that could be applied to guidance processes  in other agencies. To ensure we applied selected internal controls to  guidance processes appropriately, we reviewed applicable literature and  spoke to OMB staff and legal scholars. We also spoke with Food and  Drug Administration (FDA) officials to gain a better understanding of their  related statutory requirements for guidance. We spoke with FDA because  certain provisions of the OMB Bulletin were informed by written FDA  practices for the initiation, development, issuance, and use of their  guidance documents.

To evaluate guidance dissemination strategies, we assessed the extent to  which departments adhered to OMB’s public feedback and comment  requirements; reviewed agency websites and digital government strategy  reports; evaluated written statements from officials on components’ use of  web and customer satisfaction metrics; and interviewed relevant agency  officials. We used Guidelines for Improving Digital Services—guidelines  for improving digital services developed under the President’s digital  government strategy—to assess the usability of component websites for  accessing guidance documents.

We conducted this performance audit from March 2014 to April 2015 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

Agencies implement specific elements of laws through regulations. One  of the main purposes of guidance is to explain and help regulated parties  comply with agency regulations. As shown in figure 1, agencies use  guidance documents to explain how they plan to interpret regulations.  Agencies also use guidance for circumstances they could not have  anticipated when issuing a regulation and when additional clarifications  are needed. Similarly, our past work has recommended that agencies  issue guidance to clarify policies when we found confusion among  grantees or others about a component’s policy or practices.

How Does OMB Define Guidance?  OMB defines the term “guidance document”  as an agency statement of general  applicability and future effect, other than a  regulatory action, that sets forth a policy on a  statutory, regulatory, or technical issue or an  interpretation of a statutory or regulatory  issue.  Guidance documents often come in a variety  of formats and names, including interpretive  memoranda, policy statements, guidances,  manuals, circulars, memoranda, bulletins,  advisories, and the like. Guidance documents  include, but are not limited to, agency  interpretations or policies that relate to: the  design, production, manufacturing, control,  remediation, testing, analysis or assessment  of products and substances, and the  processing, content, and evaluation/approval  of submissions or applications, as well as  compliance guides. Guidance documents do  not include solely scientific research. publish the final regulation.creating binding legal obligations. Regulations are generally subject to  judicial review by the courts if, for example, a party believes that an  agency did not follow required rulemaking procedures or went beyond its  statutory authority.

Regulations affect regulated entities by  To balance the need for public input with competing societal interests  favoring the efficient and expeditious conduct of certain government  affairs, the APA exempts certain types of rules from the notice and  comment process. These include “interpretative rules” and “general  statements of policy” that can be made effective immediately upon  publication in the Federal Register. and general statements of policy, agencies use guidance documents that  may not be published in the Federal Register. In addition, agencies may  use the preambles of their regulations to further interpret the  regulations. An agency may use any of these documents to provide  more explanation on how the agency plans to interpret a regulation.  Defining guidance can also be difficult. To illustrate that difficulty, several  of the components in our scope told us that they do not consider many of  the communication documents they issue to the public to be guidance.

5 U.S.C. §§ 552(a), 553(b), 553(d). There is general agreement that the public interest is  served by prompt dissemination of the guidance contained in agency interpretations and  policy statements. Jeffrey S. Lubbers, A Guide to Federal Agency Rulemaking, 5th Edition  (Chicago, Illinois; American Bar Association, 2012) p. 64.

Agency guidance documents are not legally binding. Agencies sometimes  include disclaimers in their guidance to note that the documents have no  legally binding effect on regulated parties or the agencies. Even though  not legally binding, guidance documents can have a significant effect on  regulated entities and the public, both because of agencies’ reliance on  large volumes of guidance documents and the fact that the guidance can  prompt changes in the behavior of regulated parties and the general  public. Due to the potential for these effects, agencies’ use of guidance  has been the subject of scrutiny from policymakers and the public.  Despite the general distinctions between regulations and guidance  documents, legal scholars and federal courts have at times noted that it is  not always easy to determine whether an agency action should be issued  as a regulation subject to the APA’s notice and comment requirements or  is guidance or a policy statement, and therefore exempt from these  requirements.

Among the reasons agency guidance may be legally challenged are  procedural concerns that the agency inappropriately used guidance rather  than the rulemaking process or concerns that the agency has issued  guidance that goes beyond its authority. Other concerns raised about agency use of guidance include consistency of the information being  provided, currency of guidance, and whether the documents are  effectively communicated to those affected.

OMB Requirements for Significant Guidance

Although the APA does not generally prescribe processes for review of  agency guidance, the OMB Bulletin establishes policies and procedures  for the development, issuance, and use of “significant” guidance  documents. The Bulletin defines “significant guidance document” as a  guidance document disseminated to regulated entities or the general  public that may reasonably be anticipated to (1) lead to an annual effect  on the economy of $100 million or more or adversely affect in a material  way the economy, a sector of the economy, productivity, competition,  jobs, the environment, public health or safety, or state, local, or tribal  governments or communities; (2) create a serious inconsistency or  otherwise interfere with an action taken or planned by another agency; (3)  materially alter the budgetary impact of entitlements, grants, user fees, or  loan programs or the rights and obligations of recipients thereof; or (4)  raise novel legal or policy issues arising out of legal mandates, the  President’s priorities, or the principles set forth in Executive Order 12866,  as further amended.

Pursuant to a memo issued by the Director of OMB in March 2009,  OMB’s Office of Information and Regulatory Affairs (OIRA) reviews some  significant guidance documents prior to issuance. All significant guidance  documents, whether reviewed by OIRA or not, are subject to the OMB  Bulletin. “Economically significant guidance documents” (those  guidance documents under the first item in the definition above) are also  published in the Federal Register to invite public comment. The OMB  Bulletin directs each agency to develop written procedures for the  approval of significant guidance, establishes standard elements that must  be included in significant guidance documents, and requires agencies to  maintain a website to assist the public in locating significant guidance  documents. Non-significant guidance is not subject to the OMB Bulletin,  and guidance procedures are left to agency discretion.

Guidance documents are considered “significant” when they have a broad and  substantial impact on regulated entities, on the public or on other federal agencies.  Guidance can have coercive effects or can lead parties to alter their conduct. For  example, under a statute or regulation that would allow a range of actions to be  eligible for a permit or other desired agency action, a guidance document might  specify fast track treatment for a particular narrow form of behavior, but subject other  behavior to a burdensome application process with an uncertain likelihood of success.  Even if not legally binding, such guidance could affect behavior in a way that might  lead to an economically significant impact. Similarly, an agency might make a  pronouncement about the conditions under which it believes a particular substance or  product is unsafe. While not legally binding, such a statement could reasonably be  anticipated to lead to changes in behavior by the private sector or governmental  authorities such that it would lead to a significant economic effect.

For example, the following guidance documents issued by our audited agencies were  considered significant:  In response to questions from state officials, DOL’s Employment and Training  Administration issued guidance in 2005 clarifying that distance learning can be  considered approvable classroom training under the Trade Adjustment Assistance  program.

In response to requests for technical assistance, Education’s Office for Civil  Rights determined that elementary and secondary schools and postsecondary  institutions could benefit from additional guidance concerning Title IX obligations  to address sexual violence. In 2014, the office issued guidance that included  questions and answers to further clarify legal requirements, and discussed the  interplay between applicable laws and proactive efforts schools can take to  prevent sexual violence.

Good Guidance Practices at the Food and Drug Administration

Certain provisions of the OMB Bulletin were informed by written agency  practices established by the Food and Drug Administration (FDA) for the  initiation, development, issuance, and use of its guidance documents. In  1997, Congress established certain aspects of FDA’s guidance processes  as law and directed the agency to evaluate the effectiveness of its  practices and develop and issue regulations specifying its procedures.FDA’s Good Guidance regulations define the subset of guidance that  must be published in the Federal Register.internal policies and practices to ensure appropriate adherence to their  good guidance practices, including a written process to document  decisions about the appropriate level of review for each guidance  document. FDA officials told us that by default, guidance will receive a  higher level of review unless a justification is presented to warrant lesser  review. FDA officials told us that they use tools, such as “guidance  initiation forms” or “concept papers” to, among other things, ensure they  avoid duplicative or overlapping guidance and to prioritize proposed  guidance.

Departments and Components Used Guidance for Multiple Purposes and Weighed Various Factors when Deciding whether to Issue Regulations or Guidance

Components Issued a Wide Variety of Guidance Serving Multiple Purposes

Component officials used guidance for multiple purposes, including  interpreting regulations, disseminating suggested practices, and providing  grant administration information.

Interpret new regulations. Component officials told us they used  guidance to summarize regulations or explain ways to meet regulatory  requirements. For example, Education officials told us that they often  follow their regulations with guidance issued to restate the regulation  in plainer language, to summarize requirements, to suggest ways to  comply with the new regulation, or to offer best practices.  Occupational Safety and Health Administration (OSHA) officials told  us that they issue guidance to help employers and workers  understand their legal rights and responsibilities.

Distribute information on suggested practices. Officials told us  that they used guidance to distribute information on program  suggestions sometimes called best practices. In particular, component  officials who administered formula grants in which wide discretion is  given to grantees, such as states, told us that they often used  guidance to encourage certain leading practices. For example, the  Administration for Children and Families (ACF) Office of Child Care  issued Information Memorandums encouraging partnerships between  state child care and child welfare agencies.

Provide guidance on grant administration. Components that  administered grants also issued procedural guidance related to grant  administration. For example, the Bureau of Labor Statistics (BLS)  issued routine administrative memoranda to remind state partners of  federal grant reporting requirements.

The impetus for developing and issuing guidance varied, including  reasons such as: (1) explaining new regulations, (2) responding to  questions from external stakeholders, (3) clarifying policies in response to  compliance findings, and (4) disseminating information on leadership  priorities and initiatives.

Explaining new regulations. Components initiated guidance in  coordination with publication of a new regulation to help regulated  entities understand new requirements. For example, in 2014, DOL’s  Wage and Hour Division released both a fact sheet and Frequently  Asked Questions (FAQs) to coincide with the issuance of its final rule  establishing a minimum wage for federal contractors.

Clarifying policies in response to questions. Multiple component  officials told us that they used guidance to clarify policies in response  to questions received from the field or regional office input about  questions they are receiving from grantees or regulated entities.  DOL’s Office of Labor-Management Standards officials told us that  ideas for guidance often come from questions from the field or the  regulated community, particularly if multiple unions had similar  questions about a new regulation.

Clarifying policies in response to compliance findings. Officials at  Education’s Office for Civil Rights and OSHA told us that they often  initiated guidance in response to findings resulting from their  investigatory or monitoring efforts, among other things. USDA Food  and Nutrition Service officials told us they have issued guidance in  response to our or Office of Inspector General audits.

Disseminating information on leadership priorities and  initiatives. In some cases, component officials told us they may issue  guidance in response to directives from senior management or in  response to administration priorities. ACF’s Office of Child Care  officials told us that they use Information Memorandums to emphasize  leadership or other legislative priorities and changes. Officials at  Education’s Office of Postsecondary Education told us that  component leadership initiates guidance related to priorities the  administration wants to accomplish. When we asked for an example,  they noted that a Dear Colleague letter the Office issued explaining  that students confined or incarcerated in locations such as juvenile  justice facilities were eligible for federal Pell grants was issued in  response to inquiries from the field as well as administration  priorities.

We found that agencies did not use standard terminology for their  guidance. Components we reviewed within USDA, HHS, Education, and  DOL used many names for their external guidance documents. Some  departments or components generally used uniform names for guidance;  at other departments components used different names for guidance.  Education offices often issued Dear Colleague letters or FAQs, among  other types of guidance. Within HHS, ACF officials told us that they  consistently used Program Instructions and Information Memorandums to  communicate information to grantees and other recipients. While many  DOL components issued documents to assist with regulatory compliance,  they used varied terms for that guidance, including bulletins,  Administrator Interpretations, directives, fact sheets, and policy letters.

The amount of guidance components issued varied, ranging from about  ten guidance documents to over a hundred documents issued by a  component in a single year. BLS officials told us that they issued about  ten routine administrative memorandums each year related to the  operation of two cooperative agreement statistical programs. In contrast,  OSHA officials told us they could easily produce 100 new or updated  products each year to provide guidance to stakeholders. Component  officials cited the varying missions or types of programs as one  explanation for the different amounts of guidance. OSHA regularly issued  guidance to assist with regulatory compliance, while BLS officials told us  that, as a non-regulatory component, they rarely issued guidance.  Although the Office of Workers’ Compensation Programs has regulatory  authority, officials told us that they did not frequently issue guidance  because their authorizing statutes have not changed recently and their  programs focus on administering benefits. Officials at the Office of Child  Care and at DOL’s Employment and Training Administration (ETA) told us  they rarely issued guidance for formula or block grant programs or were  generally limited to issuing guidance with encouragements and  recommendations. These programs include the Office of Child Care’s  Child Care Development Fund Grants and ETA’s Workforce Investment  Act formula grants (both to states), which provide wide discretion to  grantees.

Component Officials Weighed Various Factors When Choosing Whether to Issue a Regulation or Guidance

Officials considered a number of factors before deciding whether to issue  guidance or undertake rulemaking. However, a key criterion in making  this decision was whether they intended for the document to be  binding. Officials from all components that issue regulations told us that  they understood when guidance would be inappropriate and when  regulation was necessary and consulted with legal counsel as they  decided whether to initiate rulemaking or issue guidance. Officials told us  that they often based the decision between guidance and regulation on  whether the direction was meant to be binding (in which case they issued  a regulation). In some cases, issued guidance clarified existing  regulations, educated the public, addressed particular circumstances, or  shared leading practices.

According to DOL officials, if components determined that current  regulations could not reasonably be interpreted to encompass the best  course of action, the solution was not case-specific, or the problem was  very wide-spread, then they may determine that issuing a new regulation  was necessary. An Education department official told us that they  considered multiple factors, including the objective to be achieved, when  choosing between guidance and regulations. For example, they used a  regulation to fill in gaps in statutory provisions. Following issuance of  regulations, they also provided guidance in the form of technical  assistance.

Similarly, HHS Administration for Community Living officials told us that  they considered a number of factors, including whether the instructions to  be disseminated were enforceable or merely good practice. For example,  they noticed that states were applying issued guidance related to  technical assistance and compliance for the state long-term care  ombudsman program differently. Administration for Community Living  officials decided it would be best to clarify program actions through a  regulation, as they could not compel the states to comply through  guidance. They believed that a regulation would ensure consistent  application of program requirements and allow them to enforce those  actions. They issued the proposed rule in June 2013.

USDA’s Food and Nutrition Service (FNS) officials told us that the  decision to issue guidance or undertake rulemaking depended on (1) the  extent to which the proposed document was anticipated to affect  stakeholders and the public, and (2) what the component was trying to  accomplish with the issued document. OIRA staff concurred that agencies  understood what types of direction to regulated entities must go through  the regulatory process.

In a few cases, components used guidance to alert affected entities about  immediate statutory requirements or to anticipate upcoming requirements  to be promulgated through the rulemaking process. While this may  provide timely information about new or upcoming requirements, it also  may cause confusion as details are revised during the rulemaking  process. FNS officials told us that when a new statute becomes effective  immediately and there is little ambiguity in how the statute can be  interpreted, they use a “staging process.” In this process, they issue  informational guidance so their partners are aware of and consistently  understand new requirements before the more time-consuming  rulemaking process can be completed. In 2014, we reported that five FNS  memorandums related to new statutory requirements for the content of  school lunches were distributed prior to the issuance of the final rule on  the changes to the content and nutrition standards for school lunches, in  part because of statutory timeframes. As FNS implemented the finalized  regulation, it also issued guidance containing new flexibilities or  substantive changes to previously issued guidance. While state and  school food authority officials said that some of these changes were likely  made by USDA to respond to problems they were having implementing  the new lunch requirements, the guidance changes were difficult to keep  up with and led to increased confusion about the requirements.

Education department officials told us they often used guidance to help  the field understand and apply new statutory requirements. Other officials  told us that in rare instances, they have issued guidance while a  proposed rule is out for comment. They noted that statutory deadlines for  implementation may necessitate the issuance of guidance prior to the  issuance of a final rule. For example, in 2014 DOL’s ETA issued an  Unemployment Insurance Program Letter interpreting statutory provisions  on permissible drug testing of Unemployment Insurance applicants. ETA noted that the program letter did not provide interim guidance on the  substance of the rule, but interpreted related statutory provisions. The  guidance stated that certain provisions related to the regulation could not  be implemented until publication of the final regulation. ETA officials told  us they will issue revised guidance after issuance of the final regulation.

Unemployment Insurance Program Letter 1-15, Permissible Drug Testing of Certain  Unemployment Compensation Applicants Provided for in Title II, Subtitle A of the Middle  Class Tax Relief and Job Creation Act of 2012. 79 Fed. Reg. 61,013 (Oct. 9. 2014)  (proposed rule). future guidance. However, the website was not updated in a timely  manner when the regulation was issued as final, which could have  caused confusion. ACF’s Office of Child Care issued recommendations to  its grantees that foreshadowed future binding requirements. For example,  in September 2011 the Office issued an Information Memorandum  recommending criminal background checks. It later published a proposed  rule in May 2013 to mandate the recommendations as requirements.

Departments Considered Few of Their Guidance Documents as Significant Under OMB’s Definition

Selected departments considered few of their guidance documents  significant as defined by OMB. Education considered a greater number of  its guidance documents significant, while USDA and DOL issued  relatively few significant guidance documents. We were unable to  determine the number of significant guidance documents issued by  HHS. As of February 2015, Education listed 139 significant guidance  documents on its website, while DOL listed links to 36 pieces of  significant guidance on its website. USDA listed links to 34 significant  guidance documents on its webpage for significant guidance (see figure  2). All selected components told us that they did not issue any  economically significant guidance.

OIRA staff told us they accepted departments’ determinations of which  types of guidance meet the definition of significant guidance. However,  given the unique circumstances of each program, the selected  departments differed in their interpretation of the significance of guidance  issued to explain eligibility changes resulting from the Windsor Decision   Education deemed its initial guidance  recognizing same-sex marriages.explaining eligibility changes for student aid in response to Windsor to be  non-significant, but considered later guidance providing clarifications  significant. HHS and USDA did not consider corresponding guidance  they released to be significant.

Officials at several components told us that, rather than invest resources  in OMB review of significant guidance, they would typically decide to  propose a regulation, which would allow them to assert binding  requirements. Employee Benefit and Security Administration officials told  us they would ordinarily use a regulation if they were considering a  guidance project that would meet the OMB bulletin definition of significant  or economically significant guidance. Officials at Labor’s Employment and  Training Administration told us they would typically opt to issue a rule  rather than guidance if the content was considered significant. OIRA staff  told us that OIRA examiners work closely with department officials and  may discuss what types of documents warrant OIRA review.

Agency Guidance Processes Differed and Could Be Strengthened by Enhanced Internal Controls

Two of Four Selected Departments Had Written Procedures for Significant Guidance as Directed by OMB’s Good Guidance Practices

What are the OMB Good Guidance  Practices for Approval of Significant  Guidance?  OMB’s Final Bulletin for Agency Good  Guidance Practices requires agencies to  develop the following procedures for approval  of significant guidance:

Agencies should develop or have written  procedures for the approval of significant  guidance documents. Those procedures  shall ensure that the issuance of  significant guidance documents is  approved by appropriate senior agency  officials.

Education and USDA had written procedures for the approval of  significant guidance as directed by OMB. However, HHS did not. While  DOL had written approval procedures, they were not available to  appropriate officials and DOL officials noted that they required updating.  Education and USDA’s written procedures explained the approval and  clearance procedures for significant guidance. DOL officials told us that,  although their written procedures were not readily available during our  audit, officials had been trained in 2007 on review and approval of  significant guidance documents. HHS officials told us that each  component tracked guidance development differently and a lack of written  procedures did not mean that guidance did not receive appropriate  departmental review. However, without written procedures or wide  knowledge of these procedures—a basic internal control standard—HHS  and DOL may be unable to ensure that their components consistently  follow other requirements of the OMB bulletin, such as required standard  elements for significant guidance, and cannot ensure consistency in their  processes over time.

In the Absence of Specific Government-Wide Standards for Non- significant Guidance, Components Could Strengthen Application of Internal Controls

As previously discussed, the Administrative Procedures Act does not  establish standards for the production of guidance. Therefore,  departments and components produce and issue the bulk of guidance— guidance that is considered non-significant—without government-wide  standards for those processes. In the absence of government-wide  standards for the production of non-significant guidance, officials must  rely upon internal controls—which are synonymous with management  controls—to ensure that guidance policies, processes, and practices  achieve desired results and prevent and detect errors. By incorporating  internal control standards, departments and components can promote  consistent application of management processes. We identified four  selected components of internal control and applied them to agencies’  guidance processes (see table 2 below).

Departments and components identified diverse and specific practices  that addressed these four components of internal control. While all  departments and components identified standard processes for internal  review of their guidance documents, these processes were typically not  documented. Further, agencies did not consistently apply other  components of internal control. The following sections identify practices  that select components have used to address these internal controls, as  well as opportunities for broader application of these practices. Internal  controls help to guide departments and components in the guidance  production process in a fluid environment where processes and  anticipated goals for non-significant guidance vary. Even within  components, types of guidance may need to be treated differently or may  warrant differing levels of review. A level of standardization of the process  that may be appropriate for one component or type of guidance may be  inappropriate for another. For information on specific components’  processes for initiation, development, review, dissemination, and tracking  and evaluation of guidance, see appendix II.

Most Components Managed Risk by Determining Appropriate Levels of Review

Agencies identified and discussed risk as they initiated guidance,  prioritized among different guidance documents to be developed, and  made decisions about the necessary level of review. Although no  component can insulate itself completely from risks, it can manage risk by  involving management in decisions to initiate guidance, prioritizing among  proposed guidance, and determining the appropriate level of review prior  to issuance. According to internal control standards, agencies should use  techniques and processes to identify and manage risk. Agencies face  multiple risks when going through the guidance production process. Risks  include legal challenges that issued guidance is asserting binding  requirements without having gone through the rulemaking process, or  that a guidance document goes beyond the agency’s statutory authority.  In addition, if leadership is not included in discussions related to initiation  of guidance, agencies risk expending resources developing guidance that  is unnecessary or inadvisable.

At a few components, officials told us that leadership (such as component  heads and department-level management) decided whether to initiate  certain guidance and guidance did not originate from program staff for  these components. Employee Benefits Security Administration office  directors presented guidance proposals considering legal, policy, and  programmatic factors to their Assistant Secretaries and Deputy Assistant  Secretaries for approval to start developing guidance. In most other  cases, ideas for additional guidance originated from program staff and  field offices or from leadership, depending on the nature of the guidance.  Education officials told us that component program staff and leadership  work together to identify issues to address in guidance. USDA Food and  Nutrition Service officials told us they may decide to initiate guidance  based on both input from regional offices and directives from senior  leadership. In a recent example, the Under Secretary for Food,  Consumer, and Nutrition Services initiated guidance to remind state  agencies of existing requirements prohibiting the online sale of benefits  for the Special Supplemental Nutrition Program for Women, Infants, and  Children.

Officials at several components indicated that they prioritized the  development of certain guidance documents over others to ensure that  guidance expediently responded, for example, to constituents with  pressing needs to comply with current regulations and that staff resources  were allocated in alignment with these priorities. Education officials told  us they prioritized among possible guidance documents based upon  conversations about staffing resources as well as the needs of their  constituents. For example, Education’s Office of Management officials  told us they used an ongoing list to prioritize certain guidance documents,  considering factors including (1) whether the guidance document  explained new statutory requirements, (2) whether it responded to  questions from their constituents, (3) the importance of the guidance to  other Education programs, or (4) whether the proposed guidance  addressed issues identified through technical assistance or calls to their  compliance hotline. HHS Administration for Children and Families’ (ACF)  Office of Child Care officials explained that they prioritized guidance in  areas that (1) had the largest positive impacts; (2) needed clarification to  address questions from grantees, stakeholders, and others; and (3) were  most relevant to the program’s rules or requirements.

At most components, officials told us that they determine the appropriate  level of review and final clearance of proposed guidance, and in many  cases guidance was reviewed at a higher level if the document was  anticipated to affect other offices or had a particular subject or scope.  Risk was one factor agency officials considered when determining the  anticipated appropriate level of review and final clearance of proposed  guidance. At the Employee Benefits Security Administration, for example,  the need for department-level clearance depended on various factors.  These included likely congressional interest, potential effects on areas  regulated by other DOL components, expected media coverage, and  whether the guidance was likely to be seen as controversial by  constituent groups. Two other factors that a few agencies reported  considering in determining whether the guidance warranted a higher level  of review were whether it was related to a major priority or would be  “impactful.”

Most Components Did Not Have Written Procedures for the Production of Non- significant Guidance

Most components did not have written procedures for guidance initiation,  development, and review. Control activities (such as written procedures)  help ensure that actions are taken to address risks and enforce  management’s directives. In the absence of written procedures,  components relied on officials’ understanding of the guidance process,  including when certain guidance documents should have been reviewed  by leadership and when it was unnecessary to have that review. In these  cases, officials told us that the guidance process was well understood by  program staff or followed typical management hierarchies. For example,  HHS’s Administration for Community Living, DOL’s Office of Labor- Management Standards and Education’s Office of Special Education and  Rehabilitative Services officials told us program staff have a good  understanding of the processes involved in developing and obtaining  approval of guidance. Control activities may help components assure that  management officials approve of the content of the guidance and concur  on the document’s relative priority. Control activities outlined in written  procedures can provide a central approach to guidance initiation,  development, and review.

A total of 6 of the 25 components had written procedures for the entire  guidance production process, and several of these components  highlighted benefits of these procedures for their guidance processes.  These components included ACF’s Office of Head Start and five  components at DOL: the Occupational Safety and Health Administration  (OSHA), the Mine Safety and Health Administration, the Employment and  Training Administration, the Office of Federal Contract Compliance  Programs, and the Bureau of Labor Statistics. Education’s Office of  Innovation and Improvement and Office of Elementary and Secondary  Education and Labor’s Veterans’ Employment and Training Service had  written procedures only for the review and clearance phase. The Mine  Safety and Health Administration’s written procedures contain information  that they describe as essential to the effective and consistent  administration of the component’s programs and activities. As shown in  figure 3, OSHA’s written procedures are designed to ensure that the  program director manages the process for a specific policy document by  considering feedback and obtaining appropriate concurrence to ensure  that guidance incorporates all comments and has been cleared by  appropriate officials. The Deputy Assistant Secretary resolves any  disagreements about substance, potential policy implications, or assigned  priority of the document. Documented procedures are not just an internal  controls issue; agencies benefit from them. OSHA’s procedures were  meant to ensure effective management of the issuance of guidance  documents.

Internal control standards do not prescribe either centralization or  decentralization for managing guidance processes, and the departments  we reviewed had varied approaches. One department, Education, had  centralized processes for guidance development, review, and  dissemination, while the other three departments were decentralized. At  Education, an Office of the General Counsel official told us she was  involved in decisions about whether guidance is considered significant  and Education’s Office of the Executive Secretariat managed the  document clearance and approval processes for all guidance. This office  also maintained frequently asked questions to explain the process to  components. In contrast, officials told us that DOL gave its components  the flexibility to develop individual procedures for developing and issuing  non-significant guidance. In addition, HHS departmental officials told us  they played a secondary review and approval role and that each HHS  component approached the development and issuance of guidance  documents differently. USDA officials told us that its guidance process  was also decentralized, as guidance was typically initiated, developed,  and approved at the program level, while significant guidance was shared  with the department for review.

Although a few components had written procedures for guidance  initiation, development, and review, officials from all components could  describe standard review practices to provide management the  opportunity to comment and ensure that its comments were addressed by  program staff. For example, the Administration for Community Living had  its officials circulate draft guidance for internal review and typically  required three to four officials to sign off on the draft, including center  directors and its Executive Secretariat. At Education’s Office of Innovation  and Improvement, program staff shared draft guidance with senior  leadership, who in turn provided feedback. Once senior leadership  officials and program staff were satisfied with and approved the  document, it was sent to the Office of the Executive Secretariat to be  placed into clearance.

Most Components Had Methods to Ensure Communication During the Guidance Development and Review Process

Most selected components had guidance practices to ensure either intra- agency and interagency review (or both) of guidance documents before  issuance. Internal controls require (1) that information is recorded and  communicated to management and others, and (2) the components have  an adequate means of communicating with and obtaining information  from external stakeholders that may significantly affect the component’s  ability to achieve its goals.

Intra-agency communications. To ensure that management  concurrence was recorded, most components we reviewed used  communication tools, such as electronic or hard-copy routing slips, to  document approval for guidance clearance or to communicate with  management and other offices about proposed or upcoming  guidance. In particular, officials at 20 components used a routing slip  to document management concurrence. For example, the Mine Safety  and Health Administration used two forms to track the clearance of  guidance documents. Education’s Office of Management used a  routing slip to document internal component approvals and convened  a working group to resolve comments and edits on the guidance  documents. The two components within ACF used a “policy calendar,”  a tool for communicating with management about the guidance  documents being drafted and their projected issuance dates. ACF’s  policy calendar listed the name and status of guidance, whether it was  a presidential or secretarial priority, whether the affected program was  mandatory or discretionary, and the proposed date of issuance to alert  appropriate ACF officials of upcoming guidance and to facilitate  appropriate review.

Interagency communications. Most component officials told us that  they conferred with other affected components or federal departments  to ensure consistency of their guidance during the development of  guidance. Officials at Education’s Office for Civil Rights told us they  sometimes reached out to other federal agencies and interested  stakeholders to have “listening sessions” on new guidance  documents, such as a 2013 pamphlet on academic success for  pregnant and parenting students. DOL’s Office of Disability and  Employment Policy officials told us they often worked with multiple  departments that addressed disability issues and contributed to the  fact sheets or other guidance documents issued by these  departments.

External stakeholders. Officials told us that feedback from external  nonfederal stakeholders often served as the impetus for the initiation  of guidance, and 15 of the 25 selected components cited examples in  which they conferred with external nonfederal stakeholders during the  guidance development process. At OSHA, external stakeholders were  not involved in developing directives or policy issuances, but assisted  with developing educational, non-policy guidance, such as hazard  alerts. Food and Nutrition Service officials told us that state and local  agency staff, industry representatives, advocacy organizations, and  the general public were involved in the development of their guidance,  generally through a comment period. For example, during the  development of a policy memorandum on vendor management, the  Special Supplemental Nutrition Program for Women, Infants, and  Children solicited comments from regional offices and all state  agencies through a 45-day comment period.

Almost Half of the Components Regularly Evaluated Whether Issued Guidance was Effective and Up-to-Date

Although some components did not have a formalized process to assess  the effectiveness of their guidance, many of these components told us  they have updated or revised certain guidance documents. According to  internal control standards, agencies benefit from procedures to  continually reassess and improve guidance processes and documents to  respond to the concerns of regulated entities. In the absence of  monitoring and evaluation strategies, components cannot assess whether  guidance meets intended goals or whether they need to provide  additional guidance to supplement and improve upon prior guidance.

Nearly half of the components we reviewed (11 of the 25) did not  regularly evaluate whether issued guidance was effective and up to date.  Without a regular review of issued guidance, components can miss the  opportunity to improve their guidance. DOL’s Office of Labor- Management Standards officials told us they had not evaluated the  relative success of old guidance and did not often revise guidance. ACF’s  Office of Child Care regularly tracked and updated guidance on grantee  reporting requirements. However, officials said there was little need to  track or update other guidance, as it had been 18 years since its  authorizing statute was changed. However, with the recent passage of  the Child Care and Development Block Grant reauthorization, officials  said they intended to assess their old guidance and update it to reflect the  new law.

GAO, Fair Labor Standards Act: The Department of Labor Should Adopt a More  Systematic Approach to Developing Its Guidance. GAO-14-69 (Washington, D.C.: Dec.  18. 2013). systematic effort to evaluate guidance. For example, DOL’s Wage and  Hour officials updated guidance to reflect new standards during their 2010  regulatory initiative on the temporary agricultural employment of H-2A  immigrant workers.

A few selected components had initiated or established a process for  tracking and evaluating guidance to identify necessary revisions. For  example, in November 2011, DOL’s Office of Federal Contract  Compliance Programs officials initiated a 2-year project to review their  directives system to ensure that they only posted up-to-date guidance. As  a result of the project, in 2012 and 2013 officials identified necessary  updates to guidance, clarified superseded guidance, and rescinded  guidance where appropriate. Officials told us that these actions reduced  the original number of directives by 85 percent. Officials also told us that  they did this to ensure that their guidance was more accurate and correct,  and the actions resulted in officials posting only relevant and current  guidance information on the component’s website. Officials told us they  now routinely monitor their directives about once a year and review other  guidance documents each time they issue new regulations or change a  policy to decide if they need to revise them. The Employment and  Training Administration used a checklist to review a list of active guidance  documents and identified whether to continue, cancel, or rescind the  guidance. In addition, officials indicated which documents were no longer  active on their website. The Mine Safety and Health Administration also  ensured that programs periodically reviewed and updated guidance  documents and canceled certain guidance.

Agencies Could Do More to Improve Online Guidance Dissemination

Components Disseminated the Bulk of Guidance Online

All components told us they relied primarily on their websites to  disseminate guidance but also used many other dissemination methods.  As shown in figure 4, the components in our review used various  strategies to distribute guidance to the public.

While all agencies posted guidance online, a few components also made  documents available to specific audiences on intranet websites. For  example, USDA’s Food and Nutrition Service officials told us that they  posted operational guidance on upcoming or proposed regulations to their  PartnerWeb, an intranet site that is only accessible by state agencies.  One component, DOL’s Bureau of Labor Statistics, e-mailed guidance to  state agencies and posted it on an intranet site for state agencies that is  not publicly accessible. Components also relied on other government  websites to distribute guidance. For example, Education used ADA.gov to  jointly issue guidance related to disability discrimination with the  Department of Justice and Stopbullying.gov to publicize guidance related  to antibullying laws and policies. Components also designed specialized  websites to disseminate guidance on specialized topics. For example,  DOL’s Office of Disability Employment Policy posted information about  disability programs on disability.gov and Education’s Federal Student Aid  used a separate websites to serve different audiences.

Almost all components used e-mail as another key dissemination method.  Components also used listservs (which manage e-mails to and from a list  of subscribers), e-mail delivery service (such as GovDelivery) or  newsletters. Officials told us they compiled listservs of individuals  interested in specific issues. They also explained that these lists were  developed in a number of ways, including by program offices that add  interested parties or directly from members of the public who sign up to  be on these lists through component websites. These listservs could be  very large. For example, DOL’s Employee Benefits Security  Administration list has 336,000 subscribers. Recognizing the importance  of listservs as a dissemination method, officials at several components  told us they periodically verify and update their e-mail lists.

Components also used other methods to disseminate guidance. Some  held press conferences or issued press releases, while others distributed  and discussed guidance during conferences, webinars, or conference  calls. Components also reported using social media, such as Facebook,  Twitter, or blogs. A few components told us that they posted guidance in  the Federal Register. Lastly, component officials said that external  partners—such as states, advocacy groups, and trade associations— sometimes distributed guidance for them at their request.

Officials used different strategies to reach certain groups and noted that it  was more resource intensive to distribute guidance to a wider audience.  For example, officials from HHS’s Administration for Community Living  explained that because their subgrantees are defined in statute, they  were able to effectively target their guidance to that group. Similarly,  Education’s Office for Civil Rights officials had readily available e-mail  lists for the purpose of sending guidance to all public school  superintendents or college presidents. DOL Employee Benefits Security  Administration officials noted that disseminating guidance to financial  institutions was fairly easy because that audience was receptive to  receiving information through their website and generally vocal when they  were unable to find the information they were seeking. On the other hand,  OSHA officials told us they use social media to communicate with hard- to-reach populations, such as non-English speakers and  temporary/contract workers who were more likely to be working in  dangerous jobs, and used hard-copy guidance during disaster recovery  efforts or to reach those who did not have access to the Internet. Officials  noted that states and stakeholder groups were helpful in reaching wide  audiences, especially when disseminating guidance to large groups  nationwide, such as parents or students and all employers or employees.  Components also reached wider audiences by engaging with the public  directly through conferences, webinars, media outreach, or public  awareness campaigns.

Ease of Accessing and Identifying Guidance Online Varied

Our ability to access and find significant and non-significant guidance  online varied. We found that Education, USDA, and DOL consistently  applied OMB Bulletin requirements for public access and feedback for  significant guidance while HHS did not. HHS’s website did not link to  significant guidance documents. In addition, we were unable to find these  documents by searching the department’s website.

HHS officials could not explain why these documents were not posted on  its website. Because components rely on their websites to disseminate  guidance, it is important that they generally follow requirements and  guidelines for online dissemination. For significant guidance, agencies are  required by the OMB Bulletin to maintain a current list of their significant  guidance on their websites. Agencies must also provide a means for the  public to submit comments on significant guidance through their websites.  Without providing the public an easy way to access significant guidance,  agencies cannot ensure that the public can know about or provide  feedback on these documents. an agency’s website to the electronic list  posted on a component website—a  current list of its significant guidance  documents in effect. The list shall include  the name of each significant guidance  document, any document identification  number, and issuance and revision dates.  The agency shall provide a link from the  current list to each significant guidance  document that is in effect. The list shall  identify significant guidance documents  that have been added, revised or  withdrawn in the past year.

While the OMB Bulletin does not have requirements for agencies related  to the online dissemination of non-significant guidance, there are several  resources agencies can use to improve how they post and update those  documents. One such resource is the Guidelines for Improving Digital  Services developed by the federal Digital Services Advisory Group.  These guidelines are aimed at helping federal agencies improve their  communications and interactions with customers through websites (see  table 3). website a means for the public to submit  comments electronically on significant  guidance documents, and to submit a  request electronically for issuance,  reconsideration, modification, or  rescission of significant guidance  documents. Public comments under  these procedures are for the benefit of  the agency. No formal response to  comments by the agency is required.

Components Took Steps to Make Online Guidance Easy to Find and Access

Made guidance easily accessible from component home pages.  All components linked key guidance documents on their websites so  that guidance could be easily found. We were able to navigate from  the homepage to the guidance itself in just a few clicks for all the  websites we reviewed. All components also used common terms for  guidance—including publications, resources, policy, grant guidance,  fact sheets, memorandums, and reports—to help users identify those  documents. Components used these terms to create links or menus to  facilitate users’ ability to find guidance. : HHS reported using its  department-wide governance structure for  developing and delivering digital services.  Governance, policies, and standards:  Education established guiding principles  to reinforce a governance structure for  developing and delivering digital services  and managing data.  Cross-agency collaboration and shared  services and tools website to showcase digital strategy best  practices and to test new technology and  tools.  Technical considerations: that it was modernizing its technical  infrastructure by adhering to business  requirements and technical trends, such  as increased use of and support for  mobile devices.  Usability and accessibility: target for all its websites and digital  content to become accessible and  compliant with Section 508—which  requires that federal electronic and  information technology are accessible to  people with disabilities—by May 31, 2013.  As of September 2013, the Administration  for Children and Families website was 92  percent compliant and the Administration  for Community Living’s website was 99  percent compliant.  Privacy and security:  that the information it collects is protected  by the privacy and confidentiality  provision of federal statutes, including the  Family Educational Rights and Privacy  Act, the Individuals with Disabilities  Education Act, the Education Sciences  Reform Act, and the Privacy Act of 1974.  It has also implemented a disclosure  review process and established a review  board to ensure that data are reviewed  and approved before they are publicly  released.

Improved search. We found that most components had search tools  on their websites that generally functioned well. Searches are a key  way that users access guidance. A number of components had  taken steps to improve their website searches. These included adding  meta tags to the code on their pages so that the most relevant content  appeared higher in the results of external searches and adding a  wider range of keywords to internal search engines to improve  searches. Components that had not made improvements explained  that this was because they used the department’s search tools and  did not have the ability to make changes on their own.

Highlighted new or important guidance. Components highlighted  new or important guidance on their homepages to draw users’  attention to that information. For example, HHS’s Office of Child Care  highlighted the passage of the Child Care and Development Block  Grant Act of 2014 on its homepage by providing reauthorization  resources. The website included key guidance related to the act,  including program instructions, technical assistance, and trainings.

Posted contact information to allow for questions and feedback  from the public. Components used websites to provide contact  information to the public. Specifically, components posted toll-free  numbers which could facilitate the public’s ability to ask questions or  provide feedback on published guidance. A few components provided  direct e-mails or phone numbers for specific offices and key program  staff. Opportunities for affected parties and other stakeholders to  submit questions and feedback on guidance documents are important  because, as discussed above, public interactions have served as the  impetus for new guidance. Further, because not all components we  reviewed provided examples of taking steps to solicit and respond to  public comments as guidance was developed, ensuring effective  mechanisms for affected parties and others to submit feedback is  crucial.

Categorized guidance. Components organized guidance by type,  topic, date, or audience to help users sort through the sometimes long  lists of guidance posted online, as shown in figure 5.

Several factors hindered the ease of access to component guidance  online. Components posted long lists of guidance, which could make it  difficult for users to find particular guidance documents. In addition, we  found that few components effectively distinguished whether their online  guidance was current or outdated to ensure the relevance of their online  information. As discussed earlier, we found that DOL’s Office of Labor- Management Standards did not update its website in a timely manner to  reflect guidance that would be affected by finalized regulation. Clearly  marking whether guidance is current is important. As previously  discussed, DOL’s Office of Federal Contract Compliance Programs  efforts to ensure the relevancy and currency of its directives system  resulted in the removal of 85 percent of their documents. Ensuring that  online content is accurate and relevant is one of the guidelines for federal  digital services (see table 3 above). Easy access to current and relevant  guidance could also facilitate opportunities for affected parties and  stakeholders to provide feedback on those documents.

Another factor that hindered public access was that it was not always  clear where to find guidance on a component website. We found  guidance was sometimes dispersed across multiple pages within a  website, which could make guidance hard to find and could contribute to  user confusion. The labeling of these links was not distinctive enough for  users to know where to go for the various guidance documents or topics  they may be seeking (see figure 6). These issues could be a result of  the requirement that Education components use a departmental template  for their websites. A few components created navigational links to  supplement departmental toolbars. Education officials told us that they  have learned from their grantees that the department’s guidance was not  easy to find and that online resources were hard to navigate. Federal  digital services guidelines direct agencies to publish digital information so  that it is easy to find and access (see table 3 above).

More Components Could Use Web Metrics to Improve Guidance Dissemination Strategies

While components used web metrics to evaluate their online guidance  dissemination strategy, many did not use that information to change their  existing approach. Further, many component officials told us that they did  not have a systematic way to evaluate whether the public could access  their guidance online. Web and customer satisfaction metrics—data  which allow agencies to measure performance, customer satisfaction,  and engagement to make continuous improvements to serve its  customers—could be a good source of this information.web metrics can inform officials about which guidance is being accessed   For example,  and searched. Similarly, customer satisfaction metrics could provide  qualitative information about how easily users were able to find the  guidance they were seeking. The thoughtful analysis and application of  these data would allow components to regularly evaluate the  effectiveness of disseminating guidance through their websites (see table  3 above). Further, internal controls call for the continual monitoring of  results and for management to take proper actions in response to  findings.

All components collected web metrics. This data could help agencies  evaluate its online products, which is a guideline for federal digital  services (see table 3 above). Every department in our review used  Google Analytics to collect website performance data. Many  components also used web metrics to evaluate the effectiveness of how  they disseminated guidance online. For example, USDA’s Food and  Nutrition Service used web metrics to track overall use of its guidance  online. In another example, HHS’s Administration for Community Living  learned about ways to drive traffic to its website when new guidance is  posted through its evaluation of web metrics. However, many  components did not use web metrics to improve how they disseminated  guidance through their websites (see table 4). Only 8 of the 24  components that made guidance publicly available online reported using  web metrics to improve how they used websites to disseminate these  documents. Because all components we studied primarily relied on their  websites to disseminate guidance, there is an opportunity for them to  build on their use of web metrics to improve how they disseminate  guidance online. Doing so will also facilitate components’ efforts to  evaluate its online guidance, which is also a guideline for federal digital  services (see table 3 above).

Components found ways to evaluate the effectiveness of guidance  dissemination outside their websites. This included seeing how many  applications had been viewed, downloaded, or submitted through  grants.gov, and how many e-mails and newsletters were opened through  GovDelivery. Components also conducted usability tests, focus groups,  and surveys of users, adhering to the federal guideline calling for  agencies to collect and address customer feedback (see table 3 above).  Other components convened internal task teams to identify and  implement changes while others hired contractors to redesign their  websites. Further, officials reported receiving useful feedback directly  from the public at conferences, webinars, stakeholder/grantee meetings,  or from monitoring visits. For example, Education’s Federal Student Aid  officials changed their search function to allow guidance to be searched  by relevance and date based on feedback received during training with  outreach groups. Additionally, OSHA officials are in the process of  surveying subscribers of its biweekly e-newsletter to seek feedback and  improvements so that they can provide useful, educational, and up-to- date information to the public. and Adult Education learned through the  use of metrics that many users sought  information about states and reorganized  its website to make that information more  prominent and easier to locate.

Administration updated its pages to  facilitate user access to guidance related  to lead because of web metrics indicating  that users were searching the topic. The  component also reconfigured its  publications page based on the number  of downloads. It also used metrics to  guide how often to reprint guidance and  which ones to translate.

Administration used metrics to determine  how long to keep guidance on its  homepages and provided navigational  links to reflect the different ways visitors  search its website.  metrics to populate its list of trending  topics and most requested pages. It also  used metrics on high-traffic pages to  inform decisions about where to post new  guidance. Further, decisions about what  materials should be translated into other  languages and whether more guidance is  needed on certain topics are based on  web metrics.

Guidance documents are an important tool that agencies use to  communicate timely information about the implementation of regulatory  and grant programs to regulated parties, grantees, and the general public.  Guidance documents also provide agencies valuable flexibility to clarify  their requirements and policies, and to address new issues and  circumstances more quickly than may be possible using rulemaking.  However, agencies must also exercise diligence when using guidance.  Although guidance documents are not legally binding, they can affect the  actions of agencies’ staffs, stakeholders and other interested parties,  because guidance articulates agencies’ interpretations and policy  choices. The potential effects of these documents—and the risks of legal  challenges to agencies—underscore the need for consistent and well- understood processes for the development, review, dissemination, and  evaluation of guidance.

We found mixed compliance with requirements established by the Office  of Management and Budget’s (OMB) Final Bulletin for Agency Good  Guidance Practices (OMB Bulletin) for the subset of guidance documents  considered “significant” under OMB’s definition. Education and USDA had  written departmental procedures for the approval of significant guidance,  as directed by the OMB bulletin. DOL officials had not made their  procedures available to component staff to ensure consistent application  of review processes for significant guidance, and those procedures  required updating. HHS had no procedures for significant guidance  approval. Though officials from both departments told us that they  believed their components understood the OMB requirements, HHS and  DOL could better ensure that their components consistently followed  OMB’s requirements for significant guidance if they made departmental  written procedures available. Education, USDA, and DOL consistently  applied other OMB Bulletin requirements on public access and feedback  for significant guidance, but HHS did not. HHS did not explain why the  department had not posted a website for significant guidance online. HHS  should ensure that the public can easily access and provide feedback on  its significant guidance, as required by OMB. Without providing an easy  way to access and comment on significant guidance, HHS cannot ensure  that the public is aware of or can provide feedback on these documents.

Government-wide guidance that specifically addresses processes for  non-significant guidance does not exist. Non-significant guidance  accounts for the bulk of components’ guidance documents. In the  absence of government-wide guidance specifically targeted at non- significant guidance, internal control principles and standards provide the  key criteria for components to apply to their policies and procedures.

Component officials identified many practices that they use to address  internal control standards regarding risk assessment, control activities,  communication, and monitoring. In particular, officials at most  components told us that they determine the appropriate level of review  and final clearance of proposed guidance documents. However, the  components less consistently identified practices to address other  elements of internal controls. For example, though all components could  describe standard practices for developing guidance, only 6 of the 25  components had written procedures for the entire process, and another 3  only had written procedures for the review and clearance phase. Written  procedures could help components define management roles in decisions  to initiate development of guidance documents, prioritize among them,  and determine their appropriate level of review to manage risk. Further,  not all components documented approval for guidance clearance and  nearly half of them did not regularly evaluate whether issued guidance  was effective and up to date. Opportunities exist for components to  strengthen their internal controls. For example, components could adapt  practices that others already use and have found to be an effective use of  resources. Wider adoption of these practices could better ensure that  components have internal controls in place to promote quality and  consistency of their guidance development processes.

To be effective, guidance documents must also be accessible by their  intended audiences. The departments and components primarily relied on  their websites to disseminate guidance. Consequently, components’  application of relevant federal guidance and best practices for web  dissemination is particularly important for ensuring that the intended  audiences can access and are aware of these documents. Certain  component websites for disseminating guidance were easy to use—for  example, because guidance was well organized or clearly marked—but  others were hard to navigate or did not effectively distinguish between  current and outdated guidance. Further, components did not always  leverage the web and customer satisfaction metrics they collected to  evaluate their guidance and its dissemination. By more consistently  analyzing the metrics they have already collected, components could  better ensure that their online guidance is easy to access, accurate, and  relevant. Ensuring effective mechanisms for affected parties and other  stakeholders to submit feedback on guidance documents is also crucial.  Opportunities for feedback on issued guidance are important, not only  because public comments and questions are often the impetus for  components initiating new or revised guidance, but also because  components we reviewed did not consistently take steps to confer with  external stakeholders while guidance was being drafted and reviewed.

To better ensure the adherence to requirements for approval and public  access to and feedback on significant guidance in accordance with  OMB’s Final Bulletin for Agency Good Guidance Practices (M-07-07), we  recommend that the Secretary of HHS take the following two actions:  1.  Develop written procedures for the approval of significant guidance  documents.  2.  Ensure that the department’s significant guidance is accessible online  and that the public can provide comments on significant guidance  documents.

To better ensure the adherence to requirements for approval of significant  guidance in accordance with OMB’s Final Bulletin for Agency Good  Guidance Practices (M-07-07), we recommend that the Secretary of  Labor take the following action:  1.  Review and update the department’s written procedures for approval  of significant guidance and make them available to appropriate  component staff.

To improve agencies’ guidance development, review, evaluation, and  dissemination processes for non-significant guidance, we recommend  that the Secretaries of USDA, HHS, DOL, and Education take the  following two actions:  1.  Strengthen their selected components’ application of internal controls  to guidance processes by adopting, as appropriate, practices  developed by other departments and components, such as  assessment of risk; written procedures and tools to promote the  consistent implementation and communication of management  directives; and ongoing monitoring efforts to ensure that guidance is  being issued appropriately and has the intended effect. Examples of  practices that could be adopted more widely include written procedures for guidance production to, among other  things, clearly define management roles;  improved communication tools, such as routing slips to document  management review; and  consistent and ongoing monitoring to determine if guidance is  being accessed and having the intended effect.  2.  Improve the usability of selected component websites to ensure that  the public can easily find, access, and comment on online guidance.  These improvements could be informed by the web and customer  satisfaction metrics that components have collected on their websites.  Some examples of changes that could facilitate public access to  online guidance include  improving website usability by clarifying which links contain  guidance; highlighting new or important guidance; and ensuring that posted guidance is current.

We provided a draft of this report to the Secretaries of Agriculture,  Education, Health and Human Services, and Labor. We received written  comments from Education, HHS, and DOL, which are reprinted in  appendixes III, IV, and V, respectively. USDA provided oral comments. In  addition, Education, DOL, and USDA provided technical comments,  which we incorporated as appropriate. We also shared a copy of the  report with the Office of Management and Budget, and incorporated its  technical comments as appropriate.

Education concurred with our recommendations. Education stated that,  although it believes that its internal controls for developing and producing  guidance are effective and that its online guidance can be easily  accessed by the public, it is committed to continuously looking for  opportunities to improve its processes. Education stated that it will review  components’ procedures for guidance development and production and  develop and provide to its components standard protocols they can use to  clarify management roles, document management review and approval of  guidance, and review posted guidance to ensure it is current and  accessible to the public. In addition, Education will review the  presentation of guidance on Education’s and its components’ websites  and identify best practices to improve the online presentation and  accessibility of guidance documents.

HHS concurred with our recommendations. While HHS pointed to its  established practices for developing and internally reviewing significant  guidance, it stated that it would explore the best mechanism for  distributing written procedures for approval of these documents. HHS  noted that it regularly engages with the public and regulatory stakeholders  to receive feedback and distributes its guidance in accordance with this  feedback, but will work with its agencies to update links to published  guidance and explore ways to make published guidance easier to find on  HHS webpages. In response to our recommendation on internal controls,  HHS stated that it will continue to work with its subagencies to share best  practices and ensure that agency practices are aligned with departmental  standards. HHS concurred with our recommendation on improving  website usability and stated that it will review current links to guidance  documents and explore ways to enhance their visibility and usability.

Labor concurred with our recommendations. Labor stated that it will  update the department’s written procedures for the approval of significant  guidance, disseminate them to component agencies, and ensure they are  easily accessible. In addition, Labor stated that it will work with  component agencies to share best practices and promote more  consistent application of internal control standards in the guidance  production process and encourage agencies to consider website  improvements and better use web metrics to ensure access and public  comments on guidance.

On March 19, 2015, USDA officials representing the Food and Nutrition  Service and the Department’s Office of Budget and Program Analysis  provided oral comments on the report. USDA generally concurred with  our recommendations.

As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the Secretaries of  Agriculture, Education, Health and Human Services, and Labor, and other  interested parties. We are also sending copies of this report to the  appropriate congressional committees. In addition, the report will be  available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-6806. Contact points for our Offices of Congressional  Relations and Public Affairs may be found on the last page of this report.  GAO staff who made key contributions to this report are listed in appendix  VI.

Appendix I: Objectives, Scope, and Methodology

You asked us to examine guidance processes at the four departments  under your jurisdiction. For this report, we reviewed how the Departments  of Agriculture, Education, Health and Human Services, and Labor and  selected subagencies or components at these four departments (1) use  guidance and the processes and criteria they use to determine whether to  issue guidance or undertake rulemaking; (2) follow applicable criteria and  leading practices in their policies, procedures, and practices for producing  guidance; and (3) ensure they follow dissemination requirements and  facilitate end users’ access to and comment on documents. We limited  our review to guidance agencies provide to external parties.

The scope of review included the 25 selected subagencies, or  components, in the four selected departments that (1) were within the  requesting committee’s jurisdiction, and (2) engaged in regulatory or  grantmaking activities, as the components engaged in these activities  were likely to issue guidance interpreting regulations or other  requirements to external parties (in contrast to agencies that issue only  informational guidance or other resources). To identify these components,  we searched in the Federal Register and the Catalogue of Federal  Domestic Assistance to confirm regulatory and grantmaking activities. We  confirmed the resulting list of identified components with department  officials. See table 5 below for the full list of audited department  components. We determined that computer-processed data were not  expected to materially affect our findings, conclusions, or  recommendations, thus rendering a data reliability assessment  unnecessary. We used computer-processed data solely to confirm that  the departments and components within our scope engaged in regulatory  or grant-making activities.

To describe how selected department and component processes used  guidance and the processes and criteria they used to determine whether  to issue guidance or undertake rulemaking, we reviewed agency written  procedures, guidance documents, and websites. We also interviewed  department and component officials on guidance practices.themes and examples from our documentary and testimonial evidence for  all objectives, we analyzed information from relevant documents and  interviews to identify and confirm common patterns as well as differences  across selected agencies.

To evaluate whether agency policies, procedures, and practices for  producing guidance followed applicable criteria and leading practices, we  assessed the extent to which agencies adhered to requirements for  written procedures for approval of significant guidance under the Office of  Management and Budget’s (OMB) Final Bulletin for Agency Good  Guidance Practices (OMB Bulletin).  To do so, we reviewed applicable  agency written procedures and agency websites and spoke to officials  about their practices for development and review of significant guidance.

Office of Management and Budget, Final Bulletin for Agency Good Guidance Practices,  72 Fed. Reg. 3432 (Jan. 25, 2007). authority, addressed in our discussion of the level of review of guidance,  are reflected in our application of the other internal controls. To ensure we  applied selected internal controls to guidance processes appropriately,  we reviewed applicable literature and spoke to OMB staff and legal  scholars identified due to their published work on the subject. In addition,  we spoke with officials at the Food and Drug Administration (FDA) to gain  a better understanding of their related statutory requirements for  guidance. We spoke with FDA because certain provisions of the OMB  Bulletin were informed by written FDA practices for the initiation,  development, issuance, and use of their guidance documents.

To evaluate guidance dissemination strategies, we assessed the extent to  which departments adhered to OMB’s public feedback and comment  requirements; reviewed agency websites and digital government strategy  reports; evaluated written statements from officials on components’ use of  web and customer satisfaction metrics; and interviewed relevant agency  officials. We used Guidelines for Improving Digital Services developed  under the President’s digital government strategy to assess the usability  of component websites for accessing guidance documents.

We conducted this performance audit from March 2014 to April 2015 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

Appendix II: Guidance Processes at Selected Components

This appendix describes guidance processes at the 25 components we  reviewed. Agency profiles include the following information:

Component Mission. In almost all cases we used information from  the 2013 United States Government Manual.

Target Audience and Dissemination Methods. Component officials  identified their intended audience and dissemination methods. We did  not corroborate this testimonial evidence.

Yes/No Questions. We asked the agency for evidence of these  practices and reported on our analysis based on this support.

Agency Use of Guidance. We relied on component officials to  explain the types of guidance they use most often and how they use  these types of guidance. In most cases, we corroborated this by  reviewing agency websites.

Guidance Processes (Initiation, Development, Review,  Dissemination, and Feedback on Guidance and Dissemination).  We provided information from component interviews about processes  for initiation, development, review, dissemination, and tracking and  evaluation. We did not corroborate this testimonial evidence. In cases  in which components referred to written procedures, tracking sheets,  or formalized review processes, we requested and reviewed relevant  documentation.

Highlighted Practices. In certain cases, we presented practices  identified by components that (1) applied or strengthened identified  internal controls and (2) could be helpful to other components if  similarly adopted. These practices are not exhaustive, but rather are  meant to illustrate useful practices.

Overview  What FNS Does  FNS administers the Department of  Agriculture’s (USDA) domestic food  assistance programs.

Target Audiences  Agency officials told us the  guidance issued by FNS was  targeted to state agencies, local  agencies, and other partner  organizations to assist them with  policy implementation and  compliance with regulations.

3.  General Guidance: Covered such topics as eligibility, program  management, breastfeeding, and nutrition.

Initiation, Development, Review, and Dissemination  Initiation: FNS officials told us that guidance was typically initiated and  drafted at headquarters by program staff. Guidance was often used to  interpret regulations. Officials at the FNS Special Supplemental Nutrition  Program for Women, Infants and Children (WIC) told us that they issued  guidance in response to the release of new regulatory and legislative  provisions or regulations issued by another agency that may have affected  WIC programs. They also issued guidance in response to input from FNS  regional offices as well as questions and other input from WIC state  agencies to clarify requirements and expectations for program operations  and directives from senior management, or in response to Inspector  General or GAO audits.

Dissemination methods  FNS website, hard copy (pre-2004  guidance), PartnerWeb internal  website, and e-mails to regional  offices.

Development and Review: FNS officials told us that after guidance was  drafted and depending on the nature of the guidance, agencies other than  the Department of Agriculture (USDA) FNS may be consulted or contacted  as appropriate. Guidance was cleared by the Deputy Administrator for  Supplemental Nutrition and Safety Programs and Office of General  Counsel. FNS obtained input from General Counsel to ensure that it did  not overstep its legal authority as provided under the program’s governing  legislation. Guidance was cleared by the Deputy Administrator, the  Administrator, or the Undersecretary if it contained sensitive issues of  interest to management or had broader implications. Guidance documents  that contained specific, extensive requirements for state agencies  administering the relevant program went through the same review process  as for rulemaking, going first to the division director and then to the Chief  of the Division of Planning and Regulatory Affairs for clearance. The  clearance process was tracked within the agency using an e-routing  system. FNS used a routing slip that accompanied all guidance  documents for clearance through the Administrator’s Office.

Initiation, Development, Review, and Dissemination, cont.  website. Afterwards, the officials notified state offices. FNS WIC officials  told us that sometimes guidance documents were e-mailed to offices that  did not have consistent access to the website, such as the Indian Health  Service offices. FNS also sent out notices if information published in the  Federal Register affected state program implementation.

USDA officials told us that because their programs were high profile, they  got a lot of public comments. They added that there was no routine  process for reviewing public comments on guidance received through  USDA’s website. FNS also received feedback from state and local  agencies through its regional offices, which had a program-specific point  of contact for the agencies. FNS officials told us that going through the  regional offices was the most effective way to receive feedback from state  agencies (i.e. the end users), as the regional offices worked with the  states doing day-to-day technical assistance and monitoring and  evaluation. State agencies contacted their regional offices when they had  difficulty accessing the web site or had questions about issued guidance  and policies. FNS officials told us this type of feedback served as a  reference point for future documents. FNS also occasionally issued  optional surveys or asked regional offices to poll state agencies.

Supplemental Nutrition Assistance Program: Enhanced Detection Tools  and Reporting Could Improve Efforts to Combat Recipient Fraud. GAO- 14-641. Washington, D.C.: August 21, 2014.

School-Meals Programs: USDA Has Enhanced Controls, but Additional  Verification Could Help Ensure Legitimate Program Access. GAO-14-262.  Washington, D.C.: May 15, 2014.

School Lunch: Implementing Nutrition Changes Was Challenging and  Clarification of Oversight Requirements Is Needed. GAO-14-104.  Washington, D.C.: January 28, 2014.

School Lunch: Modifications Needed to Some of the New Nutrition  Standards. GAO-13-708T. Washington, D.C.: June 27, 2013.

WIC Program: Improved Oversight of Income Eligibility Determination  Needed. GAO-13-290. Washington, D.C.: February 28, 2013.

Overview  What OCR Does  OCR’s mission is to ensure equal  access to education and to promote  educational excellence through  enforcement of civil rights. It serves  student populations facing  discrimination and the advocates  and institutions promoting systemic  solutions to civil rights problems.  OCR also provides technical  assistance to help institutions  achieve voluntary compliance with  the civil rights laws that OCR  enforces.

Agency Use of Guidance  1.  Dear Colleague Letters and Frequently Asked Questions: Covered  topics related to OCR-enforced regulations.  2.  Pamphlets: Offered general informational guidance directed toward a  wider audience, typically students and parents.

Initiation, Development, Review, and Dissemination  Initiation: OCR relied on field offices, other programs, and the public for  ideas for new guidance. OCR used its Program Legal Group (PLG) to  collect and assess the need for new guidance. PLG submitted guidance  ideas to OCR leadership for approval. Some factors considered to  determine whether to issue guidance included: (1) the number of people  who would be affected; (2) the need for guidance; and (3) how resource  intensive guidance development would be, such as the availability of  technical expertise or amount of collaboration needed.

School officials, parents, students,  and the general public.

Development and Review: OCR informally involved external  stakeholders, such as associations representing educational institutions  and officials, students, and civil rights advocates, and has used listening  sessions to get their views on draft guidance. However, it did not share  drafts externally until they were finalized. OCR followed the department’s  clearance processes for all of its guidance.

Review decisions documented?    Yes ✔      No   Dissemination methods  Website, e-mails, listservs,  conferences, webinars, press  conferences/releases, social media,  and external partners.

Dissemination: OCR disseminated guidance through its website, press  releases, e-mail listservs, and social media. OCR has scheduled calls with  press and stakeholders as necessary to announce the public release of  guidance. For guidance documents written for school officials, OCR has  created and distributed a less technical fact sheet to describe the  guidance for parents and school staff. OCR staff also disseminated and  answered questions about guidance during conferences, webinars, and  through the provision of technical assistance.

Feedback on Guidance and Dissemination  OCR officials monitored (1) statistics on the number of clicks on e-mailed  links, (2) the number of website hits, and (3) mentions of guidance in the  news and specialized publications. OCR also distributed feedback forms  on its guidance after technical assistance sessions and webinars.

Examples of Recent Guidance-Related GAO Reports  Child Welfare: Federal Agencies Can Better Support State Efforts to  Prevent and Respond to Sexual Abuse by School Personnel, GAO-14-42  (Washington, D.C.: January 27, 2014).

Charter Schools: Additional Federal Attention Needed to Help Protect  Access for Students with Disabilities, GAO-12-543 (Washington, D.C.:  June 7, 2012).

Students with Disabilities: More Information and Guidance Could Improve  Opportunities in Physical Education and Athletics, GAO-10-519  (Washington, D.C.: June 23, 2010).

Office of Career, Technical, and Adult Education  (OCTAE)

Overview  What OCTAE Does  OCTAE administers and  coordinates programs that are  related to adult education and  literacy, career and technical  education, and community colleges  that enable adults to acquire the  basic literacy skills necessary to  function in today’s society. OCTAE  also helps students acquire  academic and technical skills and  prepare for high-skill, high-wage,  and high-demand occupations in  the 21st-century global economy.  OCTAE provides national  leadership and works to strengthen  the role of community colleges in  expanding access to  postsecondary education for youth  and adults and in advancing  workforce development.

Agency Use of Guidance  1.  Frequently Asked Questions: Guidance relevant to OCTAE’s two  formula grant programs.   2.  State Plan Guidance: Information provided to states on the  requirements for plans that must be submitted before a state can  access federal funding under certain programs.  3.  Reports and Resources: Fact sheets that highlight efforts currently  underway that support OCTAE’s top priorities, and reports to  Congress on state performance under the Carl D. Perkins Vocational  and Technical Education Act and the Adult Education and Family  Literacy Act. OCTAE also published research and data.

Target Audiences  The primary audience for guidance  is states that are grantees of  OCTAE’s formula and discretionary  grant programs.

Written guidance review policy? same departmental process for significant and non-significant  guidance. Program officials consulted with policy staff, the Office of the  General Counsel (OGC), and the Office of Planning, Evaluation, and  Policy Development on guidance development. The draft guidance  then went to OCTAE’s executive officer, the Chief of Staff, and the  Deputy Assistant Secretary for review. The Office of the Assistant  Secretary cleared the guidance and consulted with the OGC to  determine if the guidance was significant.

Review decisions documented?   Yes ✔      No

Dissemination: OCTAE sent guidance directly to state directors via  OCTAE’s listserv. They also posted guidance to OCTAE’s website and  its blog. OCTAE also included links to new guidance in its newsletter.

OCTAE officials tracked and evaluated users’ awareness and  understanding of existing guidance through interactions with grantees  including monitoring visits, regular telephone communications, audits, and  meetings and conferences.

Used web metrics to change  online guidance dissemination?

Office of Elementary and Secondary Education  (OESE)

Overview  What OESE Does  OESE directs, coordinates, and  formulates policy relating to early  childhood, elementary, and  secondary education. OESE also  focuses on supporting states,  school districts, and schools in  improving K–12 education;  providing children with language  and cognitive development, early  reading, and other readiness skills;  and improving the quality of  teachers and other instructional  staff.

Agency Use of Guidance  1.  Frequently Asked Questions (FAQ): Issued to states and school  districts to answer questions spanning multiple programs.  2.  Dear Colleague Letters: Addressed issues concerning a particular  program or multiple programs outlining new policies.

Initiation, Development, Review, and Dissemination  Initiation: OESE officials said that they often produced guidance in  response to grantee inquiries or to questions from stakeholders. Further,  OESE may receive feedback from quarterly meetings with grantees and  technical advisors which result in the development of guidance.

Target Audiences  State and local educational  agencies, school administrators,  teachers and support staff, parents,  and the general public.

Development and Review: OESE officials told us that they follow the  departmental guidance on determining whether other guidance should be  categorized as significant or non-significant. Officials said that they have  formed a working group to draft and review significant guidance that  included the Office of the General Counsel and staff from OESE and other  components. OESE subcomponents were responsible for the drafting of  non-significant guidance. From there, the review and approval process  differed depending on the type and content of the guidance. OESE has  developed an internal guide to advise staff on who should be approving  different types of guidance and the time frames required for that review.  See the highlighted practices section on the next page for more  information.

Dissemination methods  OESE website, e-mails, webinars,  and newsletters.

Dissemination: Officials said that OESE typically posted guidance on its  website and issued a message from either the Assistant Secretary or the  Director of the issuing office to grantees or state contacts stating that the  new guidance was available. OESE also held webinars and frequently  communicated with national associations to provide information about the  guidance. In addition, OESE has highlighted new guidance in the  department’s “ED Review” newsletter. Finally, OESE typically held  quarterly calls with grantees and technical assistance advisors who  suggested possible improvements to guidance on grant program  implementation.

Feedback on Guidance and Dissemination  Officials noted that OESE did not formally track how it disseminates  guidance although some offices within the component may have their own  tracking systems. OESE officials said that they met with national  associations each month to discuss clarifications and technical issues.  OESE sought feedback from grantees about their experiences with  receiving guidance and have used those responses to improve  dissemination strategies.

Highlighted Practice  In addition to using a routing slip to  track the clearance of draft  guidance, OESE has developed an  internal guide to advise staff on  who should be approving different  types of guidance and the time  frames required for that review. For  example, this document calls for a  3-to-5 business day window for the  Office of the Secretary to approve  nonregulatory guidance and FAQs.  There is also an internal contact  identified who is responsible for  coordinating clearances within the  Office of the Assistant Secretary.

Feedback on Guidance and Dissemination, cont.

OESE officials also noted that they received frequent and helpful feedback  on guidance directly from the public through e-mails and phone calls. A  point of contact is identified on each piece of guidance to whom feedback  could be submitted. Additionally, OESE staff regularly informed  departmental leadership and policy officials of the need to revise guidance  when gaps and inconsistencies were identified.

Overview  What FSA Does  FSA partners with postsecondary  schools and financial institutions to  deliver programs and services that  help students finance their  education beyond high school. This  includes administering  postsecondary student financial  assistance programs authorized  under Title IV of the Higher  Education Act of 1965, as  amended.

Agency Use of Guidance  1.  Electronic Announcements: Announced administrative information.  2.  Operational and Policy Clarifications: Clarified issues such as  reporting requirements.

Initiation: FSA officials told us that questions from schools, program  partners, and the public often serve as the impetus for FSA guidance. FSA  officials met with OPE staff to discuss how to address questions that have  been raised and whether it was appropriate to issue guidance. FSA took  the lead on developing guidance that was operational in nature, while  OPE was the lead if the guidance was policy related. Guidance is typically  developed in a collaborative manner and often includes Education’s Office  of the General Counsel.

Target Audiences  The primary audience for FSA  guidance is students, parents,  borrowers, financial institutions,  counselors, mentors, schools, and  institutions of higher education that  disburse Direct Loans and other  federal aid authorized under Title  IV. Other audiences include  postsecondary education  associations and interest groups,  lenders and guarantors, federal and  state agencies, legal rights  advocates, and Title IV program  partners.

Development and Review: While operational guidance may be signed by  managers, officials told us that any guidance of importance was reviewed  at all levels. FSA officials told us they do not generally work with external  stakeholders during the development of electronic announcements.

Dissemination: According to FSA officials, FSA’s primary mechanism for  disseminating guidance was through the Information for Financial Aid  Professionals website at http://www.ifap.ed.gov/ifap/index.jsp. Officials  said all finalized guidance was posted on this website, which featured a  “What’s New” page on which guidance issued in the past 14 days was  posted. FSA also used e-mail blasts and newsletters to send out new  guidance and disseminated guidance during trainings and webinars.

According to FSA officials, FSA managers and staff have considerable  interaction with program participants, program partners, and  recipients. FSA receives a significant amount of feedback informally.  Feedback also came through FSA customer care centers.

Websites, e-mail blasts, webinars,  and newsletters.

Managing for Results: Selected Agencies Need to Take Additional Efforts to  Improve Customer Service. GAO-15-84. Washington, D.C.: October 24, 2014.

Used web metrics to change  online guidance dissemination?

Department of Education: Improved Oversight and Controls Could Help  Education Better Respond to Evolving Priorities. GAO-11-194.  Washington, D.C.: February 10, 2011.

Higher Education: Stronger Federal Oversight Needed to Enforce Ban on  Incentive Payments to School Recruiters. GAO-11-10. Washington, D.C.:  October 7, 2010.

Office of Innovation and Improvement (OII)

Overview  What OII Does  OII oversees competitive grant  programs that support innovations  in the educational system and  disseminates lessons learned. OII  administers, coordinates, and  recommends programs and policy  for improving the quality of activities  designed to support and test  innovations throughout the   K–12 system in areas such as  parental choice, teacher quality, use  of technology in education, and arts  in education.

Agency Use of Guidance  1.  Policy Guidance and Policy Letters: Intended to help state and local  educational agencies, educational service agencies, consortia of these  agencies, nonprofit organizations or institutions of higher education  that receive federal financial assistance to fulfill their obligations under  specific federal laws and regulations.   2.  Publications: Included guides, booklets, fact sheets, and brochures  on a variety of topics, such as innovative school models, promising  practices, school choice, private education, and supplemental  educational services.  3.  Frequently Asked Questions (FAQ): Related to grant competitions  and new statutory requirements.

Target Audiences  The audience for OII guidance  includes parents, administrators,  teachers, students, and  grantees/applicants.

Initiation, Development, Review, and Dissemination  Initiation: OII officials told us that grant competitions were the impetus for  most guidance it issues. In addition, questions from external stakeholders  have led OII officials to initiate new guidance or use FAQs to quickly  respond to questions. When deciding whether to issue a rule or guidance,  officials noted working with the Department of Education’s (Education)  Office of the General Counsel (GC). However, if OII made legally binding  programmatic changes, officials would use the rulemaking process.   Officials also stated that they worked with GC on all guidance to ensure  they were providing clear and accurate information to their grantees and to  the field.

Website, e-mail blasts, listserv  announcements, and press  releases.

Used web metrics to evaluate  online guidance dissemination?   Yes         No ✔  Used web metrics to change  online guidance dissemination?

Development and Review: Program staff were responsible for drafting  guidance in consultation with GC and budget officers. Drafts were shared  with the office director and submitted to the OII Executive Office where  they were reviewed for clarity and consistency with other initiatives. The  draft guidance was then reviewed by the Assistant Deputy Secretary,  Associate Assistant Deputy Secretary, and Executive Officer who then  provided feedback to program staff to make required revisions. OII officials  said that they did not use a routing slip to document concurrence on draft  guidance. Once approved, the guidance was sent to the Office of the  Executive Secretariat in the Office of the Secretary where it was finalized  (see figure 7). This process was also used for the development and  approval of significant guidance. For OII’s significant guidance, the Office  of the Executive Secretariat sought reviews from relevant Education  components and then GC forwarded it to OMB for review. Officials told us  external stakeholders were not involved in commenting on drafts before  issuance.

Dissemination: OII officials told us that they posted all guidance on their  website and notified the public about new guidance in a number of ways.  For example, OII sent out links to the guidance to its listserv subscribers  and used e-mail blasts to inform interested parties of the issuance of the  guidance.

Initiation, Development, Review, and Dissemination, cont.

Agency Use of Guidance  1.  Privacy Technical Assistance (TA): Provided TA through responses  to written inquiries, e-mails, and telephone calls on the Family  Educational Rights and Privacy Act (FERPA), the Protection of Pupil  Rights Amendment (PPRA), and the military recruiter provisions of the  Elementary and Secondary Education Act of 1965, as amended  (ESEA).  2.  Dear Colleague Letters: Addressed topics related to privacy matters.  3.  Frequently Asked Questions: Answered questions related to privacy  matters.

Target Audiences  School officials and parents.

Initiation, Development, Review, and Dissemination  Initiation: OM created new guidance based on feedback or questions  from the field, and addressed statutory and regulatory amendments made  to the laws it administers. OM held weekly meetings to discuss complex  inquiries received from the field and to address the need for any new  guidance.

Review decisions documented?    Yes ✔       No   Website, listservs, webinars, and  conferences.

Development and Review: OM officials told us that they follow  departmental guidance on determining whether other guidance should be  categorized as significant or non-significant. OM developed guidance  documents on FERPA, PPRA, and ESEA military recruiter provisions. A  working group provided input into the draft guidance documents and  recommended the appropriate level of departmental review. This group  included officials from OM and the Office of the General Counsel (OGC)  and other department program offices. OM has reached out to other  federal agencies when necessary. For example, OM has worked with the  Department of Justice on juvenile justice issues and with the Department  of Health and Human Services (HHS) on guidance on the amendments  made to FERPA by the Uninterrupted Scholars Act. Together, OM and  HHS hosted a joint webinar on this amendment. OM also circulated draft  guidance to OGC and appropriate components for review. OM followed  the department’s significant guidance clearance process for all documents  considered significant by OMB.  Generally, non-significant guidance  documents received a less formal clearance process.

Dissemination: OM officials told us that they posted guidance on their  website and e-mailed it out through a listserv. OM also introduced new  guidance through webinars and conferences.

Feedback on Guidance and Dissemination  OM officials told us that they have received feedback on guidance through  e-mails, interactions during conferences, and through OM’s technical  assistance network.

Overview  What OPE Does  OPE formulates federal  postsecondary education policy  and administers programs that  address national needs in support  of the mission to increase access  to quality postsecondary education.

Agency Use of Guidance  1.  Dear Colleague Letters: Clarified regulations.  2.  Electronic Announcements: Provided administrative information.  3.  Frequently Asked Questions: Answered questions on  postsecondary education policies and programs.

Target Audiences  The primary audience for OPE  guidance is entities and individuals  involved in postsecondary  education. This includes institutions  of higher education, postsecondary  education associations and interest  groups, lenders and guarantors,  students, federal and state  agencies, legal rights advocates,  program partners, and grantees.

Initiation, Development, Review, and Dissemination  Initiation: OPE officials told us that questions from schools, program  partners, and the public were typically the impetus for issuing guidance.  This was particularly true if OPE had received a number of similar  questions on a topic. Typically, it decided to issue guidance in close  consultation with the department’s Office of the General Counsel (OGC).  OPE officials also said that agency leadership initiated guidance  development to accomplish administration priorities.

Development and Review: OPE officials told us that they used a  standardized process to review all guidance and used a routing slip to  document the review. OPE officials told us they used different routing slips  for higher education program guidance and for guidance related to policy,  planning, or innovation. OPE officials reviewed the draft guidance and,  after final approval from the Assistant Secretary, submitted it for further  review in the department and at OGC. OPE officials did not generally work  with external stakeholders during the development of Dear Colleague  letters and electronic announcements. However, if the guidance  addressed an operational issue, OPE may consult with the National  Association of Student Financial Aid Administrators or other relevant  nongovernmental organizations.

Used web metrics to evaluate  online guidance dissemination?

Used web metrics to change  online guidance dissemination?

Dissemination: According to agency officials, OPE’s primary mechanism  for disseminating regulatory guidance was through the Information for  Financial Aid Professionals website. Guidance to grantees was mainly  distributed through the OPE website. For example, OPE distributed  guidance through national conferences for grantees, e-mail blasts to  inform grantees of program changes and upcoming grant competitions,  technical assistance webinars, and newsletters. OPE officials also said  that they may contact external stakeholders (including those that  represent students) as new guidance is being released to explain the  guidance, establish the objective of issuing the guidance, and answer any  questions that the stakeholders may have. OPE said guidance that has  been superseded by new guidance was clearly marked accordingly to  prevent confusion about which policies were in effect.

OPE officials described a variety of ways in which they obtained feedback  on guidance dissemination. For example, OPE was able to track how  many members were on its listserv and to track undeliverable e-mail.  According to OPE officials, program staff had considerable interaction with  people in the field and they received a significant amount of feedback  informally.

Postsecondary Education: Many States Collect Graduates' Employment  Information, but Clearer Guidance on Student Privacy Requirements Is  Needed. GAO-10-927. Washington, D.C.: September 27, 2010.

Grant Monitoring: Department of Education Could Improve Its Processes  with Greater Focus on Assessing Risks, Acquiring Financial Skills, and  Sharing Information. GAO-10-57. Washington, D.C.: November 19, 2009.

Office of Special Education and Rehabilitative  Services (OSERS)

Overview  What OSERS Does  OSERS helps ensure that people  with disabilities have equal  opportunities and access to  education, employment, and  community living. Its mission  includes administering the  Individuals with Disabilities  Education Act and the  Rehabilitation Act of 1973. These  statutes help states meet early  intervention and educational needs  of disabled children and youth,  support state and private programs  that provide people with disabilities  the resources they need to gain  meaningful employment and lead  independent lives, and support  research and development  programs.

Agency Use of Guidance  1.  Dear Colleague Letters: Highlighted departmental or administration  initiatives or provided program-related information.   2.  Grants and Funding: Provided information on grants, funding  opportunities, and other resources.   3.  Frequently Asked Questions: Contained issue-specific guidance  about a pending funding opportunity, provided background  information, or clarified certain topics.  4.  Model Individualized Education Program (IEP): Created in  response to a statutory mandate to be used by advocates, parents,  grantees, and school administrators.

Initiation, Development, Review, and Dissemination  Initiation: OSERS officials said that guidance was often produced in  response to grantee inquiries or questions from stakeholders. OSERS has  received feedback from quarterly meetings with grantees and technical  assistance providers which resulted in the development of guidance.  Further, OSERS has identified the need to clarify or issue guidance during  the course of monitoring its grant programs. It has also convened a focus  group with external stakeholders to identify needed guidance.

Target Audiences  Educational administrators,  vocational rehabilitation  administrators, grantees and  potential applicants, and the special  education and vocational  rehabilitation communities,  including advocates and parents of  students with disabilities.

Development and Review: OSERS officials told us that they followed  departmental guidance on determining whether documents should be  categorized as significant or non-significant. OSERS has an informal  process for developing and reviewing guidance. Officials explained that  there was no need to have a documented process because it had a  congenial and close group of experienced OSERS staff with a clear  understanding of policy. As needed, OSERS staff coordinated with staff  from other departmental components.

Written guidance review policy?    Yes         No ✔  Review decisions documented?    Yes ✔      No   Dissemination methods  Websites, newsletters, webinars,   e-mails, and mail.

Dissemination: Officials said that OSERS typically posted guidance on its  homepage as well as on idea.ed.gov. OSERS also uses its listserv to  announce new guidance to grantees or contact information. Additionally,  program officers held meetings with project directors during which new  guidance was announced. OSERS also held webinars and frequently  communicated with state associations to provide information about  guidance. Lastly, OSERS highlighted new guidance in the department’s  “Ed Review” newsletter or in the OSERS’ monthly newsletter.

Feedback on Guidance and Dissemination  OSERS officials noted that they received frequent and helpful feedback on  guidance directly from the public through e-mails and phone calls.  Specifically, a point of contact was identified on each piece of guidance  through which feedback could be submitted.

Examples of Recent Guidance-Related GAO Reports  Charter Schools: Additional Federal Attention Needed to Help Protect  Access for Students with Disabilities. GAO-12-543. Washington, D.C.:  June 7, 2012.

Students With Disabilities: More Information and Guidance Could Improve  Opportunities in Physical Education and Athletics. GAO-10-519.  Washington, D.C.: June 23, 2010.

Administration for Children and Families (ACF)  Office of Child Care (OCC)

OCC supports low-income working  families by providing access to  affordable, high-quality early care  and afterschool programs. OCC  administers the Child Care and  Development Fund (CCDF) and  works with state, territory, and tribal  governments to provide support for  children and their families juggling  work schedules and struggling to  find child care programs that will fit  their needs and that will prepare  children to succeed in school. transmit requirements to grantees:  Information Collections: Typically related to reporting requirements  for grantees.

Information on Related Legislation: Used to transmit information  about new legislation that affected the program. For example, OCC  used PIs to issue information on American Recovery and  Reinvestment Act (ARRA) funding. OCC has also issued PIs to  grantees about targeted funds appropriated for specific activities.  2.  Information Memorandums (IM): Used IMs to emphasize leadership  or other legislative priorities and changes, including recommendations  or encouragements, flexibilities in use of funds, and related information  on partner agencies.

Audiences for OCC guidance  include grantees, the 50 states, the  District of Columbia, five territories  and 260 federally-recognized  tribes. (Some of these tribes  represent consortia of tribes)

3.  Policy Interpretation Questions: Provided policy guidance in  response to questions from the field. OCC officials told us they rarely  use this type of guidance.

Initiation, Development, Review, and Dissemination  Initiation: Information guidance was typically developed in response to  feedback provided by regional office officials about questions received  from grantees. Officials at the central office held monthly calls with the  regional offices and a biweekly call with the regional program managers to  supplement day-to-day communication.

Website, regional office meetings  with state representatives, regional  and national conference calls, and  webinars.

Used web metrics to evaluate  online guidance dissemination?

Development and Review: The OCC Director was involved in the  development of OCC guidance and reported to the Deputy Assistant  Secretary. The Assistant Secretary reviewed any guidance that was  considered new, or novel. ACF’s General Counsel reviewed all OCC  guidance. PIs were flagged for leadership review at the beginning of the  clearance process. Officials typically created a routing slip for each  document. Officials did not typically provide draft guidance to external  stakeholders for comment prior to issuance. They often shared draft  guidance with regional offices. OCC officials cleared documents with other  ACF offices when the subject matter was directly relevant to other  programs.

Used web metrics to change  online guidance dissemination?

Dissemination: Because the audience for guidance was the OCC  grantees, officials typically posted new guidance on their website and then  e-mailed it to the grantees. If the guidance was of interest to grantees,  they often held a conference call with the grantees to explain the  guidance.

Administration of Children and Families (ACF)  Office of Child Care (OCC)

Highlighted Practices  The Child Care and Development  Block Grant (CCDBG) Act of 2014  reauthorized the child care  program for the first time since  1996. In response to new  requirements outlined in the law,  OCC created a “Reauthorization  Resource” webpage. The page  featured an overview of the law  and detailed the new health and  safety requirements for child care  providers and changes to eligibility  policies. It also provided answers  to frequently asked questions  (FAQs). In those FAQs, OCC  explained provisions of the law,  clarified who is affected by the law,  and stated that more detailed  guidance on effective dates for  certain program requirements in  the new law would be forthcoming.

OCC officials told us they had processes to ensure the currency of their  guidance, including labeling if the guidance was not current or applied only  to a certain fiscal year. OCC also maintained a Technical Assistance  Network that could identify implementation issues through its work with  grantees. This allowed OCC to receive comments from grantees online.  OCC regional offices communicated more with the grantees through  quarterly calls and often relayed any issues to headquarters officials.  Guidance documents listed regional office officials as the point of contact  for questions. Because the guidance was largely informational, officials  had not issued revisions to their guidance.

Administration for Children and Families (ACF)  Office of Head Start (OHS)

OHS administers grant funding and  oversight to the agencies that  provide OHS services. OHS also  provides federal policy direction  and a training and technical  assistance system to assist  grantees in providing  comprehensive services to eligible  young children and their families. requirements and submissions essential to program function. OHS  officials told us they used PIs when discussing requirements of the  OHS Act.  2.  Information Memorandums (IMs): Provided informational and  qualitative updates.  3.  Policy Clarifications: Responded to questions received through the  Early Childhood Knowledge and Learning Center (ECLKC) website.

Audiences for OHS guidance  included Head Start grantees.

Initiation, Development, Review, and Dissemination  Initiation: OHS officials told us they initiated guidance in response to  confusion identified in the field. In addition, officials might also issue a  piece of guidance if there was a change to statute or regulation of which  their grantees should be aware. Feedback leading to guidance came from  a variety of sources, including regional offices, conferences, directly from  programs, and from calls to congressional offices that were communicated  to the office.

Review decisions documented?  Yes ✔           No  Electronically though ECLKC and  e-mails to grantees.

Development and Review: OHS officials told us that guidance  development started at the program office level unless, for some types of  guidance, the decision was made to engage earlier with General Counsel  (GC). The unit’s division director and the Office of the Executive Secretary  approved guidance. Guidance then went to the Director of the Office of  OHS, GC, and finally to the Deputy Assistant Secretary or Assistant  Secretary of ACF. The guidance process was documented and a review  slip was used (see figure 8 below). Officials told us that they input  information on the potential guidance onto the ACF policy calendar so that  ACF officials could determine who should review and approve the  documents. If guidance was relevant to another federal agency, OHS  would work with the agency to write and approve the guidance.

Dissemination: OHS officials told us they disseminated guidance  electronically through ECLKC, which provided all grantees with access to  key documents. They maintained a directory of grantees and could send  information to the entire directory (anyone could sign up to receive  information). They may broadly disseminate items or share them only with  management staff, depending on the content. They also sent e-blasts with  the guidance based on the audience and subject matter.

Administration of Children and Families (ACF)  Office of Head Start (OHS)

Initiation, Development, Review, and Dissemination, cont.

Initiation, Development, Review, and Dissemination  Initiation: Before drafting, ACL officials told us they typically checked  whether similar guidance had been issued. This allowed them to take prior  guidance into consideration and to consult with lawyers to check on any  legal issues they needed to be aware of before drafting began.

Website, regional office meetings  with state representatives, regional  conferences, webinars, and  newsletters.

Used web metrics to evaluate  online guidance dissemination?

Development and Review: ACL officials told us that they circulated  guidance for internal review to center directors and ACL’s Executive  Secretary using an e-mailed routing slip. Officials wrote a memorandum to  accompany the guidance for review if background or additional context  was needed. ACL officials involved their lawyers during the internal review  process when legal interpretation was needed. During this phase, ACL  officials discussed whether external review was needed. External reviews  were not typical, but ACL worked with OMB to ensure that its guidance  accurately reflected new OMB instructions and federal requirements for  grant programs.

Used web metrics to change  online guidance dissemination?

Dissemination: Officials told us that all guidance was posted on ACL’s  website. ACL also distributed new and updated guidance through regional  conferences, webinars, newsletters, and the Federal Register. Regional  offices met with their states in group meetings to review new guidance  each quarter. From there, it was the responsibility of state representatives  to pass that information on to their respective partners, with whom they  have had more direct contact. Officials explained that sub-grantees were  identified in the law and thus easily identifiable.

Administration for Community Living (ACL)

Highlighted Practices  Guidance production at ACL often  included a discussion of how best  to publish and disseminate that  guidance. In one example, ACL  officials used a decision  memorandum to accompany draft  guidance that explained to  reviewers (1) the issue the policy  guidance document addressed, (2)  provided background on the policy  and impetus for the guidance, and  (3) requested the reviewer’s  signature. The background  discussion in the decision  memorandum included a plan for  posting the policy guidance on the  website and a discussion of other  dissemination methods to publicize  the guidance. The decision  memorandum suggested that a  blog post could be developed to  publicize the guidance, or that ACL  staff could reach out to key  stakeholders when the policy was  posted. ACL officials stated that for  any type of policy guidance they  would generally discuss how best  to disseminate it and would follow  up with staff to provide technical  assistance or answer questions.

Feedback on Guidance and Dissemination  ACL officials told us that they received feedback on their guidance in a  number of ways. ACL conducted quarterly and biennial meetings with  grantees during which guidance was discussed. The field office also  organized conference calls that covered guidance. ACL officials stated  that one-on-one interactions were the most effective manner to receive  feedback. There were program updates for grant programs that were re- released each year. Officials told us that the most common reason for  guidance updates was to address questions received from the states,  territories, and tribal organizations or from their regional offices; however,  they noted that guidance did not change frequently. ACL received  information from its information technology and communications offices on  website analytics. Meanwhile, it was also redesigning its website. ACL  officials told us that web metrics will be an important tool used to guide  how it shares its content, including guidance, once the site is redesigned.

In addition, officials told us that  when ACL issued state plan  guidance on the Older Americans  Act, ACL headquarters and  regional staff coordinated the dates  for release, the timeline, and the  process staff would use as follow- up.

Bureau of International Labor Affairs (ILAB)

Overview  What ILAB Does  ILAB improves working conditions,  raises living standards, protects  workers’ rights, and addresses the  workplace exploitation of children  and other vulnerable populations.

Target Audiences  Grantees that are subject to  cooperative agreements with ILAB,  including non-governmental  organizations and members of the  public who have knowledge of  labor conditions and practices in  countries with which the U.S. signs  free trade agreements. Guidance  related to cooperative agreements  was generally not intended for  widespread public consumption.

Covered the receipt and handling of public submissions on labor  provisions of U.S. free trade agreements.

Written guidance review policy?   Yes            No ✔   3.  Research-Related Resources  Initiation, Development, Review, and Dissemination  Initiation: ILAB officials told us that its program offices initiated the  development process when guidance was needed. The impetus for new  guidance could be grantee questions, or a program office could determine  that an adjustment in language was needed to clarify a requirement or that  a new requirement was needed. Most ILAB guidance clarified  government-wide grant regulations and was prompted by a change in  those grant regulations. ILAB officials told us little, if any, clarification was  required if there had not been a change in regulation.

Review decisions documented?   Yes ✔       No   Dissemination methods  Website, listserv, and press  releases.

Development and Review: ILAB officials told us that once the guidance  was drafted, the Office of the Solicitor reviewed it. If the guidance was  related to grants, the grant office also reviewed it. External stakeholders  were not typically involved in developing grant-related guidance. In one  instance, ILAB put forward guidance related to submissions on labor  provisions of U.S. free trade agreements through formal notice and  comment in the Federal Register. In another instance, for research-related  resources ILAB sought feedback from interagency partners and through a  peer review by external experts from business, academia, unions, and civil  society groups. Officials told us that they did not have written procedures  for their guidance processes.

Dissemination: ILAB officials told us that they disseminated guidance  directly to grantees as part of their cooperative agreement and via the  office’s webpage and various listservs. Grant solicitations were posted on  grants.gov. The Department of Labor’s Office of Public Affairs could also  issue a press release.

Officials told us that grantees could provide feedback to ILAB through its  project managers. Depending on the subject matter, ILAB could also  obtain public feedback through publishing a notice in the Federal Register  that solicits public comments. Officials considered this to be an effective  practice.

Examples of Recent Guidance-Related GAO Reports  International Labor Grants: DOL's Use of Financial and Performance  Monitoring Tools Needs to Be Strengthened. GAO-14-832. Washington,  D.C.: September 24, 2014.

International Labor Grants: Labor Should Improve Management of Key  Award Documentation. GAO-14-493. Washington, D.C.: May 15, 2014.

BLS collects, analyzes, and  disseminates economic information  to support public and private  decision making. BLS serves as a  statistical resource for the  Department of Labor.

Agency Use of Guidance  BLS officials told us that BLS awarded cooperative agreements to state  agencies to conduct two cooperative statistical programs- Labor Market  Information and Occupational Safety and Health Statistics. BLS issued  routine administrative memoranda that contained reporting requirements  and closeout procedures targeted to grantees.

Initiation, Development, Review, and Dissemination  Initiation: Because guidance was routine and issued annually, BLS  officials told us that their process did not involve a specific impetus for  initiation.

The audience for BLS  administrative memoranda was the  50 state agencies and territories  that receive cooperative agreement  funds and BLS Regional  Commissioners.

Development and Review: BLS officials told us the routine administrative  memoranda were sent out to the appropriate offices for review, including  the BLS Branch of Grants and Funds Management, the Office of Field  Operations, and BLS program offices. In addition, officials told us BLS had  written procedures for the development of guidance, but no external  stakeholders were involved during development.

Dissemination: BLS officials told us BLS administrative memoranda were  posted on the internal “Stateweb” website and e-mailed to the state  agencies. BLS provided an intranet link to its grantees to access the  stored documents. These documents were neither disseminated to the  general public nor posted to the BLS public website.

Internal “Stateweb” web site and e- mails to grantees with an intranet  link.

Feedback on Guidance and Dissemination  Officials told us that most administrative memoranda were issued  annually, so it was unnecessary to revise or update them or issue  correction memoranda. BLS officials told us that sometimes BLS got  feedback or questions on funding that it then answered.

We did not evaluate BLS’s use of  web metrics because it did not  use its public website to  disseminate guidance.

Employee Benefits Security Administration  (EBSA)

Overview  What EBSA Does  EBSA promotes and protects the  retirement, health, and other  benefits of the more than 141  million participants and  beneficiaries in more than 5 million  private sector employee benefit  plans. EBSA develops regulations,  assists and educates workers, plan  sponsors, fiduciaries, and service  providers, and enforces the law.  The Employee Retirement Income  Security Act is enforced through  regional and district offices  nationwide and a national office in  Washington, D.C.

Agency Use of Guidance  1.  Compliance Assistance Documents: Typically issued in response to  requests for advisory opinions and included advisory opinions,  information letters, interpretations, frequently asked questions, and  technical releases.   2.  Field Assistance Bulletins: Typically issued in response to issues  identified by EBSA personnel, including the regional and enforcement  staff who review them.  3.  Technical Guidance for Consumers: Typically provided information  to the public and included brochures, handouts, participant  information, and press releases.

Target Audiences  Guidance was targeted to  employee benefit plan participants  and beneficiaries, sponsors,  administrators, fiduciaries, service  providers including large financial  services firms, institutional record  keepers and asset custodians and  their representatives, EBSA  regional and enforcement staff, and  individual members of the public.

Initiation, Development, Review, and Dissemination  Initiation: Officials told us that any official in EBSA could initiate an idea  for a new piece of guidance. For example, each EBSA office had a chain  of command that employees could use to suggest ideas. However, the  Assistant Secretary decided whether to start developing guidance based  on discussion at regular executive staff meetings or with office directors  and their management staff. Officials told us EBSA sometimes issued  companion guidance to documents issued by other employee benefit  regulators, such as the Pension Benefit Guaranty Corporation or the  Internal Revenue Service.

Written guidance review policy?   Yes         No ✔  Review decisions documented?  Yes ✔      No  Dissemination methods  EBSA website, e-mails to website  listserv, press releases, e-mails to  stakeholders, calls, webcasts,  meetings with constituents, and  presentations at industry meetings.

Used web metrics to evaluate  online guidance dissemination?   Yes ✔        No  Used web metrics to change  online guidance dissemination?

Development and Review: EBSA officials told us office directors  considered legal, policy, and programmatic factors and then developed  guidance proposals to present to EBSA’s Assistant Secretary and Deputy  Assistant Secretaries for approval. Their procedures for guidance  clearance were dependent on the type of guidance. Guidance interpreting  regulations triggered a different level of review than informational  guidance and was cleared through the Director of the Office of  Regulations and Interpretations and/or the Director of the Office of Health  Plan Standards and Compliance Assistance, the Office of the Solicitor,  and the Deputy Assistant Secretary for Program Operations. EBSA’s  Assistant Secretary cleared all guidance except for very routine matters  and typically alerted department leadership upon release of the guidance.  EBSA officials regularly discussed the status of draft guidance using a  written agenda of pending regulations, exemptions, and active guidance  products in a weekly meeting with key EBSA and departmental officials.  Officials told us that the need for departmental review depended on  various factors, including likely congressional interest, potential impacts on  areas regulated by other Department of Labor (DOL) agencies, and  expected media coverage. EBSA did not use a formal or codified routing  slip. Instead, it used e-mail to contact the Assistant Secretary and Deputy  Assistant Secretaries after guidance was developed and vetted through  the appropriate national and/or regional office components and the Office  of the Solicitor.

Initiation, Development, Review, and Dissemination, cont. keepers and asset custodians, and the financial services industry.  Dissemination to these groups was relatively easy. Officials told us this  audience closely followed updates to the EBSA website and characterized  this audience as resourceful and very vocal. EBSA officials usually knew  whether the guidance had been received and was clear. However, officials  also told us that EBSA had challenges reaching their other audience,  which included small- to medium- sized employers and the participants in  the 5-to-6 million existing employee plans. To reach this audience, they  used a “multi-pronged strategy,” including posting guidance on their  website, which has a dedicated page for guidance, e-mailing and meeting  with stakeholders, and webcasts. Any special guidance could be posted  on the “New and Noteworthy” portion of their website and could be e- mailed to EBSA’s website listserv, which had about 336,000 subscribers.  Labor’s Public Affairs office assisted by drafting press releases and  handling press calls.

EBSA officials told us that EBSA sometimes revised and updated its  guidance and identified which documents had been superseded by the  new guidance. EBSA officials told us that both new and replaced guidance  documents were posted on EBSA’s website, which was actively monitored  by the regulated community and media outlets that focus on labor and  benefit issues. Various media reports and benefit-specific websites could  also provide information on new guidance. Officials also told us that the  most effective means of soliciting feedback on guidance had generally  been to post the guidance documents on EBSA’s website. To obtain input  on how to improve the quality of guidance and gauge whether guidance  has reached the intended audiences, EBSA met regularly with stakeholder  associations, individual companies, and consumer groups. EBSA also  participated in educational conferences sponsored by industry groups and  interacted with the Employee Retirement Income Security Act Advisory  Council.

401(K) Plans: Improvements Can Be Made to Better Protect Participants  in Managed Accounts. GAO-14-310. Washington, D.C.: June 25, 2014.

Private Pensions: Clarity of Required Reports and Disclosures Could Be  Improved. GAO-14-92. Washington, D.C.: November 21, 2013.

Private Pensions: Revised Electronic Disclosure Rules Could Clarify Use  and Better Protect Participant Choice. GAO-13-594. Washington, D.C.:  September 13, 2013.  401(K) Plans: Labor and IRS Could Improve the Rollover Process for  Participants. GAO-13-30. Washington, D.C.: March 7, 2013.

Employment and Training Administration (ETA)

Overview  What ETA Does  ETA provides job training,  employment, labor market  information, and income  maintenance services, primarily  through state and local workforce  development systems. ETA also  administers programs to enhance  employment opportunities and  business prosperity.

Agency Use of Guidance  1.  Training and Employment Guidance Letters (TEGL): Issued to a  broad audience of states and sub-grantees, they transmitted both  policy and operational guidance. Often used for programs to identify  funding allotments.  2.  Training and Employment Notices (TEN): Informational notices  directed to broad audiences. Officials told us that they used TENs to  communicate technical assistance resources, publications, and  updates on research and evaluation and the status of available agency  issuances.

Target Audiences  Audiences for guidance at ETA are  both narrow and broad, depending  on the program. ETA funds  American Jobs Centers, which has  guidance targeting a narrow  audience of those that administer  these centers, while other  programs are broader and have  guidance that could be directed to  a wide range of people. Other  audiences include Workforce  Investment Boards, governors,  both discretionary grantees and  formula subgrantees, and state  workforce agencies, among others.

3.  Unemployment Insurance Program Letters (UIPL): Interpreted  statute or policy in the form of guidance. For example, when  unemployment insurance was extended and then terminated, ETA  issued guidance more than a dozen times from June 2008 to  December 2013.  4.  Program Information Notices (PINs): Information for Job Corps  Centers, which were operated both by private contractors and the  Forest Service in the Department of Agriculture in partnership with the  Department of Labor (DOL).

Written guidance review policy?  Yes ✔      No  5.  Frequently Asked Questions  Initiation, Development, Review, and Dissemination  Initiation: ETA officials told us they initiated guidance in response to  issues that arose in the field, and worked with ETA program and policy  officials as well as discussed the issue with their Office of the Solicitor  (SOL) colleagues and the program office.

Review decisions documented?  Yes ✔      No  Dissemination methods  Website, e-mail blasts to the public,  e-mails to regional offices,  webinars, site visits, phone calls,  and conferences.

Development and Review: ETA used a standard operating procedure for  reviewing guidance documents and a routing slip for internal review.  Officials told us guidance went through multiple review processes. The  officials included in the review process varied depending on the nature of  the guidance. Usually, the program office, policy office, SOL, and  leadership office all signed off on the guidance. The Office of the Assistant  Secretary for Policy (OASP) signed off on guidance documents that were  “major priorities.” Officials also discussed what level of review the  guidance required with the Office of the Executive Secretary. Officials told  us that when making review decisions they considered (1) the amount of  money affected by the guidance, (2) how much of a priority the guidance  was, and (3) whether other sub-agencies within DOL would be interested  in the guidance. If the proposed guidance required internal clearance, they  included OASP, the Offices of Congressional Affairs, the Secretary, and  Public Affairs and alerted Cabinet Affairs staff. If regional offices would be  affected, they concurred on the draft guidance.

Initiation, Development, Review, and Dissemination, cont.  sign up to receive an e-mail when a new advisory was available. Separate  e-mails were sent to the regional offices, which then followed up with  grantees about new guidance. ETA officials told us they also informed  OASP and the Office of Congressional Affairs. ETA officials hosted a  webinar to provide technical assistance and answer any questions that  grantees had. The regional offices also had site visits and phone calls with  grantees in which they discussed advisories and guidance as needed and  occasionally attended conferences.

Feedback on Guidance and Dissemination  Officials told us they were asked to identify whether to continue, cancel, or  rescind the guidance on the ETA website annually. They told us they  routinely monitored grantees and had conversations with  intergovernmental organizations to gain insights into potential changes to  guidance. They received feedback from program offices, regional offices,  intergovernmental organizations, and agency leadership on the content of  their guidance.

Workforce Investment Act: DOL Should Do More to Improve the Quality of  Participant Data. GAO-14-4. Washington, D.C.: December 2, 2013.

H-2A Visa Program: Modernization and Improved Guidance Could  Reduce Employer Application Burden. GAO-12-706. Washington, D.C.:  September 12, 2012.

Mine Safety and Health Administration (MSHA)

Overview  What MSHA Does  MSHA promulgates and enforces  mandatory health and safety  standards by thoroughly inspecting  mines; targeting the most common  causes of fatal mine accidents and  disasters; reducing exposure to  health risks from mine dusts and  other contaminants; improving  training, particularly for  inexperienced miners and  contractors; strengthening MSHA  and the industry’s emergency  response preparedness; enforcing  miners’ rights to report hazardous  conditions without fear of  retaliation; and emphasizing  prevention. MSHA also assists  states in the development of  effective state mine safety and  health programs and contributes to  mine safety and health research  and development.

Agency Use of Guidance  1.  Program Information Bulletins: Provided information and best  practices to mine operators, miners, and MSHA enforcement officials.  2.  Program Policy Letters: Explained regulations to mine operators,  miners, and MSHA enforcement officials.  3.  Procedural Instruction Letters: Instructed MSHA’s staff on  procedures for enforcing applicable standards.  4.  Program Policy Manual: Consolidated MSHA policies.  5.  Handbook: Provided instructions to MSHA inspectors and specialists.  6.  Compliance Guidance and E-Laws: Posted on MSHA’s website.  7.  Best Practice Pocket Cards: Provided miners with health and safety  information, including an explanation of their rights  8.  Frequently Asked Questions: Explained MSHA standards and  regulations to operators and miners.

Target Audiences  Coal mine operators, metal and  non-metal mine operators, unions,  associations, and safety and health  professionals.

Initiation, Development, Review, and Dissemination  Initiation: MSHA officials told us that new or revised guidance was  typically initiated in response to questions from the field or issues  identified by miners, operators, or MSHA Field Managers. MSHA officials  discussed whether to issue guidance or undertake the rulemaking process  with the Office of the Solicitor, the Office of Standards, Regulations, and  Variances and the Office of the Assistant Secretary. MSHA also issued  guidance as part of the normal rollout of new standards or regulations.

Dissemination methods  Website, e-mails, compliance visits  and other meetings, and through  partners.

Development and Review: MSHA had written procedures for guidance  formulation, distribution, and maintenance. Administrators and Directors  initiated guidance that was reviewed by appropriate officials. MSHA  officials told us that guidance went through multiple reviews by affected  programs, and significant guidance was flagged during the review  process. For urgent guidance documents that need to be disseminated  quickly (for example, hazard alerts and information on respiratory  protective devices), the review process was shortened and senior  management were involved earlier in the process. The Directorate of  Program Evaluation and Information Resources (PEIR) coordinated and  monitored guidance development and clearance. PEIR’s Office of  Program Policy Evaluation (OPPE) officials managed the directives  process and used a form to manage and track the review and  dissemination of directives.

Initiation, Development, Review, and Dissemination, cont.  miners, labor organizations, industry associations, and other stakeholders  when guidance was developed or to introduce new standards or  regulations to explain and discuss the guidance. MSHA’s Field Managers  also discussed guidance with miners and operators at multiple compliance  visits each year.

Feedback on Guidance and Dissemination  MSHA officials told us they had developed procedures to ensure that  programs periodically reviewed and updated guidance documents. They  revised previously issued guidance if they determined that the guidance  was out of date due to advances in technology or if other new information  was available from stakeholders. MSHA officials told us that the most  effective way to gather feedback on guidance is by speaking with  stakeholders. MSHA officials told us that OPPE also had developed  policies to review existing guidance to ensure that it was valid and that  MSHA had made changes to its website to help the public easily find  guidance information.

Examples of Recent Guidance-Related GAO Reports  Mine Safety: Basis for Proposed Exposure Limit on Respirable Coal Mine  Dust and Possible Approaches for Lowering Dust Levels. GAO-14-345.  Washington, D.C.: April 9, 2014.

Mine Safety: Additional Guidance and Oversight of Mines' Emergency  Response Plans Would Improve the Safety of Underground Coal Miners.  GAO-08-424. Washington, D.C.: April 8, 2008.

Occupational Safety and Health Administration  (OSHA)

Agency Use of Guidance  Policy issuances—established internal policies or policy interpretations.

OSHA assures safe and healthful  working conditions for men and  women by promulgating  protective  health and safety standards;  enforcing workplace safety and  health rules; providing training,  outreach, education, and  assistance to workers and  employers in their efforts to control  workplace hazards; preventing  work-related injuries, illnesses, and  fatalities; and partnering with states  that run their own OSHA-approved  programs.

OSHA officials told us policy issuance documents have been used to  explain internal procedures for inspections and interpretations of  regulations for specific programs. An example of a policy issuance was  the inspection procedure directive, which contains procedures used to  investigate and cite violations of particular OSHA regulations.

Non-policy issuances—provided information consistent with regulations.  1.  Fact sheets, information sheets, hazard alerts, and small entity  compliance guidance: Provided hazard identification and prevention  information on critical safety and health hazards that often must be  disseminated quickly.  2.  Booklets: Provided information for constituents at all education levels.  3.  Fatal Facts: Contained information about how to identify and prevent  hazards that lead to fatalities at worksites. Written for employers,  safety and health professionals, and workers.

Employers and their  representatives, such as trade  associations; covered workers and  their representatives, such as  unions, community groups and  worker centers; and other safety  and health professionals.

4.  Quick cards: Small laminated cards that provided safety and health  information for employers, professionals, and workers with some  safety and health background.  5.  Low-literacy materials: For workers and employers with limited  English proficiency and young workers.

6.  Letters of interpretation: Clarified ambiguities in regulations.  Initiation, Development, Review, and Dissemination  Initiation: OSHA officials told us that they were often prompted to issue or  revise guidance for clarification in response to feedback from regional  offices and external stakeholders with questions on existing guidance.

OSHA website, hard copy delivery  to area offices, mass mailings to  employers, webinars, outreach in  the school system, social media,  newsletter (QuickTakes), and e- mail.

Used web metrics to evaluate  online guidance dissemination?

Development and Review: Officials told us policy issuances were  cleared by the Office of the Solicitor and OSHA leadership, and were  sometimes sent to the Office of the Executive Secretariat for coordination  of department-level review. OSHA program directors obtained input and  technical and policy clearance for both policy and non-policy issuances  from each of the other program directors and their directorate offices and  resolved any comments. The final draft was sent to the Director of  Administrative Programs for approval. The Deputy Assistant Secretary  addressed unresolved disagreements concerning the substance or policy  implications of proposed policy guidance. Officials told us that in some  circumstances, OSHA sought expert input or input of the target audience  for non-policy guidance materials to provide the most accurate and  applicable information—and to make sure it is accessible—for workers  and employers on a specific topic.

Highlighted Practices  OSHA had separate written  procedures in the form of  instructions for both policy and  non-policy issuances. Policy  issuances are internal directives  and supplementary guidance that  have implications for internal  statements of policy and  procedure, while non-policy  issuances are technical and  educational guidance documents  that provide information consistent  with regulations and include such  supplementary guidance materials  as letters of interpretation and  other non-policy statements issued  by OSHA. These procedures  outlined the roles and  responsibilities for the Assistant  Secretary of Labor for  Occupational Safety and Health,  OSHA program directors, and the  Director of Administrative  Programs and identified conditions  under which guidance should be  published.

Initiation, Development, Review, and Dissemination, cont.  produces an e-mail based newsletter called QuickTakes that publicized  new policy and non-policy guidance documents. Area offices distributed  new educational guidance materials to the stakeholders who were difficult  to contact electronically. They also conducted mass mailings and  webinars, posted on social media sites, and reached out at conferences  and schools. Dissemination also occurred with cooperative program  participants such as Alliance and Partnership members and state  partners. Participants and partners disseminate materials and guidance to  members and constituencies.

Feedback on Guidance and Dissemination  OSHA officials told us they tracked and evaluated guidance to determine  whether to revise guidance. If a guidance product was written for a  specific OSHA standard that had not changed, revisions were infrequent.  Guidance would be updated if based on a standard that has changed or a  hazard where new information was available to assure workers are  protected.

Workplace Safety and Health: OSHA Can Better Respond to State-Run  Programs Facing Challenges. GAO-13-320. Washington, D.C.: April 16,  2013.

Workplace Safety and Health: Further Steps by OSHA Would Enhance  Monitoring of Enforcement and Effectiveness. GAO-13-61. Washington,  D.C.: January 24, 2013.

Office of Disability Employment Policy (ODEP)

Overview  What ODEP Does  ODEP seeks to increase the  number and quality of employment  opportunities for people with  disabilities by promoting the  adoption and implementation of its  policy strategies and effective  practices and bringing focus to the  issue of disability employment.

Target Audiences  People with disabilities, employers  (in both the private and public  sectors), service providers, and  government entities. employers (in both the private and public sectors), service providers,  and government entities.

While ODEP does not have regulatory authority, it has assisted  enforcement agencies in reaching stakeholders in the disability community  regarding regulations that impact them. ODEP also provided input on  guidance issued by other Department of Labor (DOL) components, such  as the Employment and Training Administration, related to disabilities.

Review decisions documented?   Yes         No ✔  Dissemination methods  Website, e-mail, auxiliary websites  such as Disability.gov, listening  sessions, and webinars, social  media, public service  announcements, speaking  engagements, and press releases.

Guidance Initiation, Development, Review, and Dissemination  Initiation: ODEP officials told us that items addressing how to comply with  regulations, if developed, would emanate from ODEP’s policy or research  team supervisors. The item would then require input and approval from  other team leads at the agency, the Executive Officer, the Deputy  Assistant Secretary, the Chief of Staff, and the Assistant Secretary. ODEP  would work closely with DOL’s enforcement agencies and the Solicitor’s  Office in developing all such items.  All non-regulatory items produced for  the public would emanate from ODEP’s policy, outreach or research team  supervisors. The item would also require input and approval from other  team leads at the agency, the Executive Officer, the Deputy Assistant  Secretary, the Chief of Staff, and the Assistant Secretary.

Development and Review: According to ODEP officials, all guidance was  reviewed by either the Outreach Supervisor or the Executive Officer to  determine the level of both internal or external review needed (both  intraagency and interagency). Guidance was cleared internally by the  Outreach Supervisor or the Executive Officer, all relevant Policy,  Administrative and/or Research Supervisors, the Deputy Assistant  Secretary, the Chief of Staff, and the Assistant Secretary. In addition,  there is an expectation that items affecting stakeholders or touching legal  or policy issues were cleared by the relevant DOL agencies, such as the  Office of the Assistant Secretary of Policy, the Office of Public Affairs, the  Solicitor’s Office, the Office of Congressional and Intergovernmental  Affairs, the Office of the Secretary, or affected DOL program agencies.  ODEP worked closely with all these agencies to determine if further  external review was required, including the Office of Management and  Budget or other relevant federal agencies.

Initiation, Development, Review, and Dissemination, cont.  disseminated guidance through their website. Officials used the “Gov  Delivery” e-mail subscription service to disseminate guidance and  information to about 50,000 subscribers. ODEP officials also reached out  to their employer stakeholders and maintained an online community of  practice. ODEP held primary responsibility for managing the Disability.gov  website. In addition to these methods of dissemination, ODEP officials  said that each of their four technical centers maintained its own website  and offered webinars throughout the year.

Feedback on Guidance and Dissemination  ODEP officials told us that items shared with the public were reviewed  regularly and updated so that the information was current and relevant.   ODEP officials told us they had just started to widely use web metrics for  their website and intended to use the information gathered to improve how  they communicated with the public. Survey tools were consistently used to  evaluate how Disability.gov served the public and that information was  used to improve its service model.

Office of Federal Contract Compliance  Programs (OFCCP)

OFCCP administers and enforces  equal opportunity mandates  prohibiting federal contractors and  subcontractors from discriminating  on the basis of race, color, religion,  sex, national origin, disability, or  protected veteran status and  require federal contractors and  subcontractors to take affirmative  steps to ensure equal employment  opportunities. anticipated to have operational impact or to result in enforcement  action.  2.  Federal Contract Compliance Manuals: Explained broad OFCCP  policy.  3.  Frequently Asked Questions (FAQ)  4.  Fact Sheets and Brochures: Targeted to a more general audience or  the public.  5.  Technical Assistance Guides: Assisted federal contractors and  subcontractors in complying with laws and regulations on employment  discrimination and equal employment.

The audience for most OFCCP  guidance and technical assistance  is contractors, particularly new  contractors. Guidance was also  targeted to communities from  whom OFCCP most often received  violation complaints. These  communities represent all  protected classes.

6.  Equal Employment Opportunity Posters: Employers covered by  non-discrimination and equal employment opportunity laws were  required to display posters on their premises.

Initiation, Development, Review, and Dissemination  Initiation: OFCCP officials told us that after they identified an issue that  might require guidance, they discussed it with the Office of the Solicitor  (SOL).

Written guidance review policy?  Yes ✔   No  Review decisions documented?  Yes ✔   No  Website, e-mails, webinars, public  meetings, and media outreach.

Development and Review: OFCCP officials told us that the national  office typically developed guidance with feedback from SOL and  comments from the regions. OFCCP guidance received varying levels of  review based on the type of guidance, but SOL reviewed almost all  guidance. Directives were more formal, used a template, and were always  reviewed and signed by the OFCCP director. FAQs were generated  internally and might not rise to the level of director review. Departmental  officials and officials from other Department of Labor (DOL) or  components reviewed guidance (1) on a sensitive subject, (2) for which  heightened scrutiny was anticipated, (3) that might affect other DOL  programs or agencies, (4) that might be considered newsworthy, or (5)  that is part of an initiative of the Administration. OFCCP officials used a  routing slip to document clearance and its processes for review were  written in administrative procedures. See figure 12 below.

Dissemination: OFCCP officials told us they had recently standardized  the process to centrally monitor and control the guidance dissemination  process. To supplement sending new guidance to the regional office,  officials also sent an e-blast to more than 54,000 e-mail subscribers  explaining (1) why guidance was issued, (2) what the guidance was, and  (3) where the guidance could be found. If appropriate, the e-blast directed  the subscriber to a hyperlink to guidance on OFCCP’s website. OFCCP  also used media packets, toolkits, webinars, and public appearances to  further publicize new guidance. OFCCP posted all guidance intended for  the public online.

Highlighted Practices  In 2011, OFCCP officials started a  2-year project to review their  directives system. They told us that  this effort was intended to make  their guidance more accurate and  correct. As part of these efforts,  they identified necessary updates  to guidance, clarified superseded  guidance, and rescinded guidance  when appropriate, reducing the  original number of directives by  85%.

Initiation, Development, Review, and Dissemination, cont.

OLMS conducts criminal and civil  investigations to safeguard the  financial integrity of unions and to  ensure union democracy. The  Office conducts investigative audits  of labor unions to uncover and  remedy criminal and civil violations  of the Labor-Management  Reporting and Disclosure Act and  related statutes and explains the  reporting, election, bonding, and  trusteeship provisions of the act.  The OLMS promotes labor union  and labor-management  transparency through reporting and  disclosure requirements. either stand alone or be issued to accompany a regulation. Standalone  facts sheets often explained what statute required.  2.  Guides for Union Officers: Provided general information on  requirements that applied to unions and union officers, and offered  suggestions on how to comply with those requirements.  3.  Guidance: Information on how a complaint with OLMS could be filed.

Initiation, Development, Review, and Dissemination  Initiation: OLMS officials told us that the impetus for guidance varied.  Guidance was initiated if (1) numerous unions had similar questions after  a new regulation had been finalized about compliance, (2) officials had  issued a regulation or were about to issue one, or (3) field personnel  encountered a consistent issue through OLMS’s compliance assistance  programs. OLMS received feedback when its program staff reached out to  union officials. However, OLMS did not typically initiate new guidance and  rather answered questions individually. If officials saw questions come into  their OLMS-Public@dol.gov e-mail box on a related issue, OLMS program  officials flagged the questions as a potential impetus for new guidance.

1) Union officials and union  members, 2) employers with a  union or with a workforce  attempting to unionize, and 3) labor  relations management  organizations.

OLMS website, field staff,  webinars, and listserv e-mails.

Used web metrics to evaluate  online guidance dissemination?

Development and Review: Officials stated that guidance development  depended on the impetus. OLMS officials stated that most often the  director or other senior manager decided to initiate guidance and then  tasked the drafting of the guidance to program staff. OLMS had no written  procedures for the guidance production process. After the program staff  drafted the guidance, it went to the Division Chief, then the Deputy  Director, then the Director. OLMS divisions and the Office of the Solicitor  (in particular the Division of Civil Rights and Labor Management) were  also involved at this stage of the guidance process. Documentation of  concurrence on draft guidance depended on the type of guidance. Some  documents were typically approved through e-mail, while for other  documents a physical folder with the sign-off chart was used for  stakeholder initials of concurrence. OLMS officials stated that they drafted  decision memorandums (typically used for regulation) to accompany draft  guidance when departmental clearance was required. OLMS officials told  us they rarely coordinated with other federal agencies when developing  and reviewing guidance, although they recently coordinated with the  Department of Labor’s Office of Federal Contract Compliance Programs  (OFCCP) on development and review of guidance.

Used web metrics to change  online guidance dissemination?

Office of Labor-Management Standards (OLMS)

Initiation, Development, Review, and Dissemination, cont. hosted webinars. OLMS had a listserv to notify interested parties when  there was a new regulation or guidance being issued.

Feedback on Guidance and Dissemination  OLMS officials told us they found webinars to be the most effective way to  solicit feedback. Officials solicited questions from listeners at webinars  and contacted those with questions directly. Officials also received  questions and comments from the public through the OLMS- Public@dol.gov e-mail box. Questions typically related to how forms  should be filled out, and officials contacted the person commenting  directly. If the question was substantive, they would raise it to a higher  level to address the comment. As of October 2014, OLMS officials told us  they had not received substantive comments related to guidance  dissemination. However, they had received feedback on technical issues  related to locating materials, navigating the website, and ease of  accessibility. OLMS officials said that they had not assessed whether their  guidance has been effective.

Office of Workers’ Compensation Programs  (OWCP)

Overview  What OWCP Does  OWCP protects the interests of  workers who are injured or become  ill on the job. OWCP serves  specific employee groups that are  covered under four major disability  compensation statutes by  mitigating the financial burden  resulting from workplace injury or  illness and promoting return to  work when appropriate.

Agency Use of Guidance  1.  Training materials: OWCP posted training materials directed to  internal claims examiners to the web.   2.  Industry notices: Procedural information sent to employers, carriers,  medical providers, and injured workers.   3.  Compliance Guidance: Issued for Black Lung and Longshore  programs to explain certain new regulations and how small entities  could comply with them.  4.  Educational Guidance: Included medical benefit guides, frequently  asked questions, webinars, and other educational materials.

Target Audiences  The audience for OWCP guidance  included beneficiaries and  employers.

Initiation, Development, Review, and Dissemination  Initiation: OWCP officials told us that program directors decided to initiate  guidance in response to (1) questions from users or confusion in the field,  (2) new procedures, (3) new initiatives, (4) litigation, or (5) results of  accountability reviews.

Written guidance review policy?   Yes         No ✔  Review decisions documented?  Yes ✔      No  Dissemination methods  Website, emails, specific task  forces, and meetings with unions.

Development and Review: OWCP officials told us they developed and  issued guidance with the assistance of the Office of the Solicitor. The  development of guidance was typically informal and was conducted  through email, revisions, and comments in documents. OWCP officials  told us they had an informal review process for guidance within each  program. OWCP officials used a routing slip and hierarchical process for  document review. A guidance document could be reviewed by the  Department based on the significance and sensitivity of the issue.

Used web metrics to evaluate  online guidance dissemination?     Yes         No ✔  Used web metrics to change  online guidance dissemination?

Dissemination: OWCP officials told us that guidance documents were  posted to the OWCP website and disseminated through email blasts to  stakeholders. For example, there were 12,000 subscribers to their  Longshore Electronic Filing email list. OWCP officials told us that they  discussed who may want or need the information before placing the  information on their website. OWCP had a joint task force which  disseminated information regarding the Energy program primarily to the  concentrated population affected in jurisdictions where there have been  nuclear activities. OWCP also held routine meetings with unions,  advocacy groups, and other stakeholder groups and maintained lists of  these groups for dissemination purposes.

Feedback on Guidance and Dissemination  OWCP officials told us that program directors reviewed guidance on an  ongoing basis and updated guidance as necessary.

Veterans’ Employment and Training Service  (VETS)

Overview  What VETS Does  VETS is responsible for  administering veterans’  employment and training programs  and compliance activities that help  veterans and service members  succeed in their civilian careers.

Target Audiences  Program service providers,  including agency grantees, sub- grantees, and other entities that  provide services to veterans,  transitioning service members, and  other eligible persons. grant administration, including formula funding updates.  3.  Solicitations for Competitive Grant Applications  4.  Frequently Asked Questions: Clarified application and program  details.

Guidance Initiation, Development, Review, and Dissemination  Initiation: According to VETS officials, the primary impetus for guidance  was to maximize the effectiveness of their programs by issuing updated  and clarifying guidance that is easily understood and can be carried out  effectively.

Dissemination methods  Website, e-mail, social media, and  through regional administrators and  state workforce grantees.

Development and Review: VETS officials told us that because the  component is small, it was easy to complete internal review of their  documents. Officials used a routing slip to circulate draft guidance to the  Offices of the Solicitor, Deputy Secretary, Assistant Secretary, and  Secretary. Departmental officials reviewed guidance if it represented a  major policy change or affected other Department of Labor agencies and  other departments.

Used web metrics to evaluate  online guidance dissemination?

Dissemination: VETS posted all publicly available documents on its  websites. Officials told us they also relied on regional administrators and  state workforce grantees to disseminate guidance related to grants. When  VETS issued technical assistance on competitive grants, it sent out an e- mail blast to notify grantees.

Used web metrics to change  online guidance dissemination?

Feedback on Guidance and Dissemination  VETS officials told us they conducted periodic meetings and listening  sessions with Veterans Service Organizations and stakeholders. VETS  coordinated closely with the Departments of Defense and Veterans  Affairs. The coordination requirements were included in a memorandum of  understanding. VETS officials also participated in forums such as the  Interagency Council on Homelessness and the Transition Assistance  Program Executive Committee. VETS’ practice had been to issue new  guidance annually. As of January 2015, it was now working to give  guidance a longer “shelf life” by, for example, only revising due dates for  routine reports rather than reissuing the entire guidance.

Examples of Recent Guidance-Related GAO Reports  Veterans’ Employment and Training: Better Targeting, Coordinating, and  Reporting Needed to Enhance Program Effectiveness. GAO-13-29.  Washington, D.C.: December 13, 2012.

WHD enforces federal minimum  wage, overtime pay,  recordkeeping, and child labor law  requirements of the Fair Labor  Standards Act. WHD enforces a  number of other worker protection  laws including those concerning  family and medical leave, migrant  and seasonal farm workers, and  several temporary foreign worker  visa programs. WHD also  administers and enforces the  prevailing wage requirements  applicable to federal contracts for  construction and for the provision  of goods and services.

Written guidance review policy?  Yes       No ✔  Review decisions documented?  Yes ✔   No  WHD website, webinars, trainings,  and outreach.

Initiation, Development, Review, and Dissemination  Initiation: WHD officials told us they initiated new guidance in response to  (1) issues identified by WHD’s investigators in their field offices, (2)  questions from outreach and education sessions with employers and  employee groups, (3) recurring questions received in correspondence, (4)  feedback from stakeholders on specific programs, (5) information  collected on the types and frequency of questions at WHD’s national call  center, and (6) interaction with other federal and state agency partners.  Many officials and offices within the Department of Labor (DOL) were  involved in WHD’s decisions to initiate guidance, including the Office of  the Solicitor, Office of the Assistant Secretary for Policy, national and  regional WHD offices, and other enforcement personnel.

Development and Review: WHD officials told us that they had a  multilayered review process that differed based on the substance of the  proposed guidance. Officials documented review with a routing slip. To  clear guidance materials, WHD worked with the Office of the Solicitor and,  depending on the substance or public interest, may have worked with  other offices within the Department as appropriate. Some of the factors  that may be considered in guidance being reviewed at the departmental  level include: whether the guidance involved a new interpretation or policy  or whether it could impact other DOL programs.

Initiation, Development, Review, and Dissemination, cont. depended on the type of guidance. WHD staff routinely conducted  webinars, training, and outreach during which guidance was distributed.  For example, they have hosted webinars, conference calls, meetings and  presentations, including engaging relevant state associations, consumers,  disability and senior citizens’ advocates, worker representatives, and  industry groups.

Feedback on Guidance and Dissemination  WHD officials told us that they revised guidance “as appropriate.” As of  January 2015, WHD officials told us they did not have a systematic way to  determine whether end users were accessing their guidance. WHD  officials told us that they meet with stakeholders, including employers,  human resources organizations, attorneys, employees, worker  organizations, and unions to hear their views on areas WHD can provide  additional guidance to achieve better compliance with the worker  protection laws it administers.

Examples of Recent Guidance-Related GAO Reports  Fair Labor Standards Act: Department of Labor Needs a More Systematic  Approach to Developing Its Guidance, GAO-14-629T. Washington, D.C.:  July 23, 2014.

Fair Labor Standards Act: Department of Labor Needs a More Systematic  Approach to Developing Its Guidance, GAO-14-69. Washington, D.C.:  December 18, 2013.

Overview  What Women’s Bureau Does  The Women’s Bureau is  responsible for promoting the  status of wage-earning women,  improving their working conditions,  increasing their efficiency, and  advancing their opportunities for  profitable employment. The  Women’s Bureau also focuses on  the needs of vulnerable women in  the workforce.

Agency Use of Guidance  1.  Informational Fact Sheets: Used to support the Women’s Bureau’s  role in disseminating its own research.  2.  Technical Assistance: Women’s Bureau officials told us 10 regional  offices as well as staff in the national office provided technical  assistance.

Target Audiences  Either targeted groups or broadly to  more than 50,000 people. The  target audiences for its  informational fact sheets are  typically working women, their  employers, and organizations that  represent them.

3.  Frequently Asked Questions  Initiation, Development, Review, and Dissemination  Initiation: When deciding to initiate guidance, officials told us that the  relevant program office typically decided whether a fact sheet was  needed. It then either researched the subject itself or contracted out for  the research. Leadership sometimes initiated guidance if it was on a  “burning issue.” The ideas for potential Women’s Bureau initiatives and/or  academic research came from all staff—both in the national and regional  offices. These ideas stemmed from internal research, including current  news and events.

Written guidance review policy?   Yes     No ✔  Review decisions documented?   Yes ✔   No  Dissemination methods  Website, e-mail, social media,  workshops, and listening sessions.

Development and Review: Women’s Bureau officials told us that after  initiating guidance, the relevant program office drafted the guidance  document. Managers and staff in the Women’s Bureau’s Office of Policy  and Programs reviewed the draft guidance document before it was  reviewed by the Deputy Directors and Director. Officials used a routing slip  or memo to guide the draft through the review process and to document  the review. Women’s Bureau officials told us they worked closely with the  Office of the Solicitor, the Office of the Assistant Secretary for Policy, the  grant officer, and other impacted Department of Labor components as  they drafted and reviewed guidance. They also used the Departmental  “Secretary’s Information Management System” program for internal  tracking. Fact sheets were shared with external stakeholders when the  guidance was issued.

Dissemination: Women’s Bureau officials told us that most documents  and other resources were available on their website and they used the  “Gov Delivery” system to disseminate new information and documents.  Officials also disseminated information through their Director’s blog and  through workshops conducted by regional office staff, such as those  that  led up to the national White House Summit on Working Families. Listening  sessions were another example of dissemination conducted by the  regional offices.

Officials told us they received feedback from end users though e-mails,  phone calls, and comments made by participants at conferences.

Appendix III: Comments from the Department of Education

Appendix IV: Comments from the Department of Health and Human Services

Appendix V: Comments from the Department of Labor

Appendix VI: GAO Contact and Staff Acknowledgments

Michelle Sager at (202) 512-6806 or sagerm@gao.gov.

In addition to the contact named above, key contributors to this report  were Tim Bober, Assistant Director, Alexandra Edwards, Sandra Baxter,  Shirley Hwang, Shelby Kain, Andrea Levine, Sarah Sheehan, and Wesley  Sholtes. In addition, Jacques Arsenault, James Bennett, Marcia Crosse,  Rachel Frisk, Ricky Harrison, Anne K. Johnson, Sarah Kaczmarek,  Jacqueline M. Nowicki, Cynthia Saunders, Stewart W. Small, and Betty  Ward-Zukerman provided key assistance.

Question: How did the four departments use guidance?

Answer: The four departments—Agriculture (USDA), Education (Education), Health and Human Services (HHS), and Labor (DOL)—and their selected components used guidance for multiple purposes, such as clarifying or interpreting regulations and providing grant administration information.

Question: How did guidance vary between agencies?

Answer: The terminology used for agency guidance varied and agency components issued varying amounts of guidance, ranging from about 10 to over 100 guidance documents each year.

Question: How was it determined whether to issue a regulation or guidance?

Answer: The key criterion used when deciding whether to issue a regulation or guidance was whether it needed to be binding; in such cases agencies proceeded with regulation.

Question: How were legal counsel involved in these decisions?

Answer: Officials reported that they routinely consulted with legal counsel when making these choices.

Question: How did departments identify guidance documents as "significant"?

Answer: Departments typically identified few of their guidance documents as “significant,” generally defined by the Office of Management and Budget (OMB) as guidance with a broad and substantial impact on regulated entities.

OMB Requirements for Significant Guidance

Departments and Components Used Guidance for Multiple Purposes and Weighed Various Factors when Deciding whether to Issue Regulations or Guidance

Components Issued a Wide Variety of Guidance Serving Multiple Purposes

Component Officials Weighed Various Factors When Choosing Whether to Issue a Regulation or Guidance

Question: How did the departments respond to OMB's requirements?

Answer: All four departments identified standard practices to follow when developing guidance. They addressed OMB's requirements for significant guidance to varying degrees and could strengthen internal controls for issuing guidance. Education and USDA had written departmental procedures for approval of significant guidance as required by OMB. DOL's procedures were not available to staff and required updating. HHS had no written procedures.

Question: How would ensuring the availability of these procedures be beneficial?

Answer: Ensuring these procedures are available could better ensure that components consistently follow OMB's requirements.

Question: Why is the application of internal control standards so important?

Answer: In the absence of specific government standards for non-significant guidance—the majority of issued guidance—the application of internal control standards is particularly important.

Question: How were control standards addressed by the 25 reviewed components?

Answer: The 25 components GAO reviewed addressed some control standards more regularly than others. For example, few components had written procedures to ensure consistent application of guidance processes. All components could describe standard review practices and most used tools to document management approval of draft guidance. Of the 25 components, 15 cited examples in which they conferred with external nonfederal stakeholders while developing guidance and nearly half did not regularly evaluate whether issued guidance remained current and effective.

Agency Guidance Processes Differed and Could Be Strengthened by Enhanced Internal Controls

Two of Four Selected Departments Had Written Procedures for Significant Guidance as Directed by OMB’s Good Guidance Practices

In the Absence of Specific Government-Wide Standards for Non- significant Guidance, Components Could Strengthen Application of Internal Controls

Most Components Did Not Have Written Procedures for the Production of Non- significant Guidance

Almost Half of the Components Regularly Evaluated Whether Issued Guidance was Effective and Up-to-Date

Agencies Could Do More to Improve Online Guidance Dissemination

More Components Could Use Web Metrics to Improve Guidance Dissemination Strategies

Appendix I: Objectives, Scope, and Methodology

Question: How did the components disseminate guidance?

Answer: Components used different strategies to disseminate guidance and all relied primarily on posting the guidance on their websites.

Question: Why must all components follow the requirements for federal websites?

Answer: Components used different strategies to disseminate guidance and all relied primarily on posting the guidance on their websites. As such, components should follow applicable requirements for federal websites.

Question: How are the website requirements beneficial?

Answer: One of these requirements—easy access to current and relevant guidance—could also facilitate opportunities for affected parties and stakeholders to provide feedback on those documents.

Question: How did the agencies respond to the website requirements?

Answer: USDA, DOL, and Education posted their significant guidance on a departmental website as directed by OMB; HHS did not.

Question: How did the components facilitate access?

Answer: Components used several strategies—including organizing guidance by audience or topic and highlighting new or outdated guidance—to facilitate access.

Question: What factors hindered online access?

Answer: However, GAO identified factors that hindered online access, including long lists of guidance and documents dispersed among multiple web pages.

Question: How were web metrics used by components?

Answer: All components GAO studied collected web metrics and many used them to evaluate online guidance dissemination. However, many of these components did not use metrics to improve how they disseminated guidance through their websites.

Question: How did the components disseminate information and obtain feedback apart from their websites?

Answer: Beyond their websites, components found other ways to disseminate and obtain feedback on issued guidance, including focus groups, surveys, and direct feedback from the public at conferences, webinars, and from monitoring visits.

Components Disseminated the Bulk of Guidance Online

Ease of Accessing and Identifying Guidance Online Varied

Components Took Steps to Make Online Guidance Easy to Find and Access

Question: What is the purpose of this report?

Answer: GAO was asked to examine guidance processes at four departments.

Question: What issues are examined in this report?

Answer: This report reviews how (1) agencies use guidance and decide to issue guidance rather than regulations; (2) follow applicable criteria and leading practices in their policies, procedures, and practices for producing guidance; and (3) agencies disseminate guidance to ensure public access and feedback.

Question: How did GAO conduct their review?

Answer: GAO reviewed guidance processes at all 25 components in the four departments that (1) were within the requesting committee's jurisdiction, and (2) engaged in regulatory or grant activities. GAO reviewed relevant requirements, written procedures, guidance and websites, and interviewed agency officials.