International Cash-Based Food Assistance: USAID Has Established Processes to Monitor Cash and Voucher Projects, but Data Limitations Impede Evaluation

For more than 60 years, the United States provided assistance to food-insecure countries primarily in the form of food commodities procured in the United States and transported overseas. In recent years, the U.S. government has increasingly provided food assistance in the form of cash transfers or food vouchers. In fiscal years 2010 through 2015, USAID funding for Emergency Food Security Program (EFSP) for cash transfers and food voucher projects grew from about $76 million to nearly $432 million.

GAO was asked to review USAID's monitoring and evaluation of cash-based food assistance. This report examines, among other things, (1) USAID's and implementing partners' processes for monitoring cash transfer and food voucher projects and (2) the extent to which monitoring data reported to USAID can be used to evaluate the performance of such projects. GAO analyzed program data, interviewed relevant officials; and conducted fieldwork in Kenya and Liberia, selected on the basis of criteria such as funding and types of projects. GAO also reviewed the final reports for a nonprobability sample of closed cash transfer and food voucher projects.

The United States Agency for International Development (USAID) and its implementing partners have established processes to monitor cash transfer and food voucher projects. To monitor the implementation of these projects, USAID has assigned monitoring roles and responsibilities to staff, is developing country monitoring plans and monitoring tools, and is working to verify information that partners have provided through actions such as conducting site visits, and speaking with beneficiaries. To ensure that assistance is delivered according to their procedures and to the targeted beneficiaries, implementing partners monitor distributions, and interview beneficiaries regarding the distribution of the assistance. In addition, implementing partners conduct postdistribution surveys to gather information about the relevance, efficiency, and effectiveness of the assistance (see figure).

Incomplete reporting and weaknesses in certain performance indicators limit USAID's ability to use monitoring data to evaluate cash transfer and food voucher projects' performance. GAO's review of 14 final reports, which USAID requires for each project, found that a majority of the reports lacked required data elements, such as prices for key staple foods. Only 1 report included all 12 required data elements, and the other reports were missing up to 8 elements. As a result, USAID has limited ability to assess the overall performance of these projects. Further, GAO found weaknesses in USAID's indicators for measuring cash and voucher projects' timeliness, cost-effectiveness, and appropriateness. USAID's indicator for timeliness does not track delays in implementation. In addition, the indicator for cost-effectiveness does not include a standardized unit for measuring project costs. Further, the indicator for project appropriateness does not have associated benchmarks for measuring cash transfer and food voucher projects' impact on local markets. As a result, USAID lacks information that would be useful for evaluating the projects' effectiveness relative to that of in-kind food aid. According to standards for internal control in the federal government, management should use quality information, including relevant data from reliable sources, to achieve an agency's objectives.

USAID should (1) take steps to ensure compliance with its requirements for data in final reports and (2) strengthen the indicators it uses to measure the timeliness, cost-effectiveness, and appropriateness of cash transfer and food voucher projects. USAID concurred with GAO's recommendations.

The United States is the largest donor of international food assistance to  food-insecure countries around the world. In recent years, this assistance  has been used to address food emergencies resulting from, among  others, the Ebola crisis in West Africa, the Syria humanitarian crisis, and  drought in countries stricken by the effects of El Niño. For more than 60  years, the United States provided this assistance primarily in the form of  in-kind food aid (i.e., commodities purchased in the United States and  transported overseas). More recently, the U.S. government has also  provided several modalities of cash-based assistance: cash transfers and  food vouchers, which recipients can use to purchase food on their own,  as well as locally and regionally procured food aid (LRP).

According to the U.S. Agency for International Development’s (USAID)  Office of Food for Peace (FFP), which manages the majority of U.S.  international food assistance, it uses cash-based assistance when, for  example, U.S. in-kind assistance cannot arrive in time to respond to an  emergency and when cash transfers or food vouchers may be more  appropriate because of local market conditions and beneficiaries’  preferences. In fiscal year 2015, FFP provided about $432 million for  cash transfer and food voucher projects through its emergency cash- based food assistance program, the Emergency Food Security Program  (EFSP), with funding from the International Disaster Assistance  account. In the same year, USAID provided about $1.07 billion for  emergency food aid, primarily in-kind assistance, authorized by Title II of  the Food for Peace Act. For fiscal year 2017, the administration has  requested additional flexibility to respond to international food  emergencies by expanding the use of cash-based assistance.

In light of proposals for U.S. food aid reform that call for expanded use of  cash-based assistance, monitoring and evaluation are fundamental to  assessing and demonstrating—with timely and credible evidence—the  effectiveness of the various approaches that the United States employs to  deliver such assistance. This is the second of two reports that you  requested on USAID’s use of cash transfers and food vouchers. In this  report, we (1) examine USAID’s and implementing partners’ processes for  monitoring cash transfer and food voucher projects; (2) analyze the extent  to which monitoring data that partners reported to USAID can be used to  evaluate the performance of such projects; and (3) review studies that  examined the relative impacts of cash transfers, food vouchers, and food  transfers on food security outcomes and the relative costs of these  modalities.

To address our objectives, we analyzed data and reviewed documents  provided by USAID and its implementing partners, including the United  Nations World Food Programme (WFP) and selected nongovernmental  organizations (NGO). We met in Washington, D.C., with USAID officials  and with implementing partner officials representing NGOs that received  USAID EFSP grants. We also met in Rome, Italy, with officials from WFP  and the U.S. Mission to the United Nations. In addition, we visited Kenya  and Liberia, where we conducted reviews of cash and voucher projects  implemented by USAID partners in three countries—Kenya, Liberia, and  Somalia. We selected these countries using criteria that included the  amount of EFSP funding received, the type of project (cash transfer or  food voucher), and the type of implementing partner. While in Kenya and  Liberia, we met with USAID officials from the U.S. missions; with  representatives of implementing partners, vendors, and financial  institutions; and with beneficiaries, among others. In addition, we took  the following steps to address each of our three objectives (see app. I for  further details):

To examine USAID’s and implementing partners’ monitoring of cash  transfer and food voucher projects, we reviewed USAID’s and  implementing partners’ activities and relevant documentation. We also  interviewed seven FFP field officers from eight countries where EFSP  cash transfer and food voucher projects were ongoing in fiscal year  2016.

To analyze the extent to which reported data can be used to evaluate  the performance of cash transfer and food voucher projects, we  examined the final reports that implementing partners submitted to  USAID for a sample of 14 projects. In each report, we reviewed data  responding to minimum programmatic reporting requirements in  USAID’s 2013 Annual Program Statement (APS) and grant  agreements. Because the 14 projects represent a nonprobability  sample, our findings may not be generalized to all USAID EFSP  projects. In addition, we reviewed the indicators and data USAID uses  to measure cash transfer and food voucher projects’ timeliness, cost  effectiveness, and appropriateness—namely, time from award to  distribution, cost per beneficiary, and food price data.

To review studies examining the relative impacts and costs of food  assistance modalities, we searched the relevant literature and  identified 14 rigorous studies that (1) compared cash transfers, food  vouchers, food transfers with at least one other modality; (2)  evaluated at least one food security outcome; and (3) considered food  security outcomes by means of randomized control trials or groups,  carefully selected comparison groups, or a quasi-experimental design  that used statistical techniques to make precise comparisons. These  studies—examining projects in Bangladesh, the Democratic Republic  of the Congo (DRC), Ecuador, Ethiopia, Malawi, Mexico, Niger, Sri  Lanka, Yemen, and Uganda—evaluated whether cash transfers, food  vouchers, or food transfers were more successful in achieving  intended impacts on food security outcomes in situations where at  least two modalities were implemented and could be compared. Our  review of the studies was not intended to assess any individual donor  such as USAID, and thus the conclusions we analyzed cannot be  inferred to assess the performance of USAID’s projects. In addition,  several of the projects examined were pilots conducted to assess the  comparative performance of multiple modalities on household food  security. We analyzed the 14 studies using a data collection  instrument that examined the studies’ design, quality, and major  findings. Further, after determining that 11 of the 14 studies  considered the modalities’ costs, we reviewed those studies’  conclusions regarding the least expensive modality.

We conducted this performance audit from June 2015 to September 2016  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

In 2015, FFP provided EFSP funding for cash transfer and food voucher  projects in 30 countries (see fig. 1). To deliver assistance through these  projects, USAID’s implementing partners employ a variety of  mechanisms. These mechanisms include (1) distributing cash manually  or electronically through accounts at banks or other financial institutions  and (2) distributing paper or electronic food vouchers that entitle the  holder to buy goods—typically, approved items from participating  vendors—or services up to the voucher’s designated cash value. The  value of the cash transfers and food vouchers is generally based on a  formula that attempts to bridge the gap between the beneficiaries’ food  needs and their capacity to meet those needs.

Additionally, in fiscal years 2010 through 2015, USAID funded the  following EFSP regional awards for cash transfer and food voucher  projects: (1) the Syria Regional Award, in countries hosting Syrian  refugees—Egypt, Iraq, Jordan, Lebanon, and Turkey—awarded each  year since 2012; (2) the Central America Drought Award—for Guatemala,  El Salvador, and Honduras—awarded in 2014; and (3) the Ebola  Regional Response—for Guinea, Sierra Leone, and Liberia—awarded in  2015.

In fiscal years 2010 through 2015, USAID awarded EFSP grants totaling  about $3.27 billion, including $1.42 billion for cash transfer and food  voucher projects. Awards for such projects grew from about $76 million in  fiscal year 2010 to nearly $432 million in fiscal year 2015 (see fig. 2). In  fiscal years 2010 through 2015, awards for cash transfers increased from  about $42 million to approximately $119 million, while awards for food  vouchers increased from about $34 million to approximately $312  million. During the same period, USAID awards of Title II funding for  emergency food aid—primarily in-kind assistance—decreased from about  $1.52 billion to approximately $1.07 billion.

USAID Requirements for Proposing Emergency Cash-Based Food Assistance Projects

Since 2010, USAID’s APS for International Emergency Food Assistance  has outlined agency requirements for EFSP cash-based food assistance  proposals. The APS, which serves as a primary source of information  for prospective applicants for EFSP grants, requires applicants to explain  the rationale for the assistance modality they propose—cash transfer,  food vouchers, LRP, in-kind food aid, or some combination of these  modalities. Since 2011, the APS has required grant applicants to provide  justification for a proposed project that includes cash-based assistance or  in-kind food aid, in terms of the project’s timeliness, cost-effectiveness, or  appropriateness.

Timeliness. Since 2011, the APS has required grant applicants to  explain whether in-kind food aid or prepositioned stocks can arrive in  a sufficiently timely manner through the regular ordering process to  address urgent or emergency needs.

Cost-effectiveness. Since 2011, the APS has required grant  applicants to provide cost-effectiveness information that affected their  choice of modality. Further, according to the APS, in certain cases the  cost of cash-based food assistance may be lower than that of in-kind  assistance, while in other cases the difference may be negligible.

Appropriateness. Since 2011, the APS has required grant applicants  to explain why cash transfers or food vouchers, or both, may be more  appropriate than in-kind food distributions. For example, potential  beneficiaries may have physical access to functioning markets but  lack sufficient purchasing power. (Fig. 3 shows other examples of  reasons that EFSP grant applicants have cited for deeming cash  transfer or food voucher projects to be most appropriate.) The 2015  APS states that, depending on market conditions, cash-based food  assistance may be deemed more or less appropriate than in-kind food  aid to address specific emergency food security needs. Since 2015,  the APS has required applicants to justify their selection of either cash  transfers or food vouchers as the preferred modality.

USAID released a draft revision of the APS in May 2016 for public  comment; according to USAID officials, the revised version will be  finalized in the autumn of 2016. The draft APS requires grant applicants  proposing EFSP projects to provide a justification that addresses how  market appropriateness, feasibility, project objectives, and cost-efficiency  influenced the modality selection. According to the draft APS, when  addressing how feasibility influenced the selection of the assistance  modality, applicants are to consider the time-sensitive nature of the  emergency, including availability of rapid-response options such as  prepositioned commodities or prenegotiated cash transfer or voucher  response mechanisms.

USAID Monitoring Requirements for Cash Transfer and Food Voucher Projects

Monitoring and evaluation perform two separate but interrelated functions.  Monitoring is the collection of data to determine whether projects are  being implemented as intended and the tracking of progress with  preselected indicators throughout the life of a project. Data collected  through monitoring can be used by project managers to make incremental  changes and adjustments to project implementation. Evaluations consist  of ad hoc or periodic studies to assess whether and how projects  achieved their expected goals. An evaluation may also identify outcomes  that can be attributed to the project and may assess the project’s cost- effectiveness. Evaluations may rely on a range of quantitative and  qualitative measures in addition to preselected indicators, comprehensive  research designs, and appropriate statistical analysis of data, including  data collected through monitoring activities. Results of evaluations can  provide program managers with critical evidence for future program  design.

USAID’s APS contains monitoring and evaluation–related requirements  for cash transfer and food voucher grant applications as well as minimum  reporting requirements for data in projects’ final reports.

Requirements for grant applications. Since 2010, the APS has  required grant applications to include a monitoring and evaluation  plan. In addition, the 2015 APS required that the monitoring and  evaluation plan include a logical framework showing the causal  linkages between activities, outputs, outcomes, and goals; identify  assumptions and potential risks that are critical to the success of a  project; and specify key indicators with proposed targets to track the  project’s performance. In 2015, the APS also began requiring grant  applicants to plan and budget for a baseline survey and a final  evaluation survey for projects that propose an implementation period  of greater than 12 months. Under the 2015 APS, such projects that  include activities to influence beneficiary behaviors must also include  at least one high-level food security or nutrition outcome indicator for  each project purpose.

Minimum reporting requirements for project final reports. Since 2010,  the APS has required implementing partners to submit final reports  containing monitoring data that meet minimum programmatic  reporting requirements. The reports are to include data on the  number of beneficiaries targeted and reached; the cost per  beneficiary; retail price information on key staples (before, during, and  after project implementation); and the actual number and value of  cash transfers or food vouchers distributed and redeemed by  beneficiaries. The reports are also to include information about  beneficiaries’ use of resources provided through cash transfer  projects. In addition, since 2011, the APS has required that final  reports include the time from donor-signed agreement to first  distribution; a description of how the program addressed gender  needs; the planned number and value of cash transfers or food  vouchers distributed to, and redeemed by, beneficiaries; and  information on the types and quantities of commodities they procured  with food vouchers. Further, in 2015, USAID began requiring final  reports to include the average cost per beneficiary per month for each  modality as well as learning on the appropriateness of selected  modalities.

USAID and Implementing Partners Have Established Processes for Monitoring Cash and Voucher Projects

USAID has established a process for monitoring implementation of EFSP  cash transfer and food voucher projects by assigning monitoring roles  and responsibilities to headquarters and in-country mission staff,  developing country monitoring plans, and developing tools to assist its  field staff. USAID’s process includes actions such as visiting distribution  and project sites, speaking with beneficiaries and retailers, and meeting  regularly with partners’ in-country staff. Implementing partners have  established processes to monitor cash and voucher projects during and  after distributions of assistance. To ensure that assistance is delivered  according to their procedures and to the targeted beneficiaries,  implementing partners monitor distributions and interview beneficiaries  about the distribution process. In addition, implementing partners conduct  postdistribution surveys to gather information about the relevance,  efficiency, and effectiveness of the assistance.

USAID Has Established a Process to Monitor Implementation of Cash and Voucher Projects

USAID has assigned monitoring roles and responsibilities to headquarters  and in-country mission staff. According to USAID officials, FFP grant  officers, in headquarters, are primarily responsible for reviewing  implementing partners’ quarterly and final reports. The grant officers are  additionally responsible for meeting with implementing partner  headquarter officials to discuss project progress and performance. Also,  FFP’s Monitoring and Evaluation Team is responsible for developing  monitoring tools, standards, guidance, implementing partner reporting  requirements, and providing training for FFP officers and implementing  partners who are responsible for monitoring cash and voucher projects.  The FFP field officers are primarily responsible for verifying information  provided by implementing partners, communicating regularly with  implementing partners in country, and providing grant officers with  information on project progress and performance.

The seven FFP field officers we interviewed told us that they conduct  various monitoring activities to verify information that partners provide,  such as visiting distribution and project sites, completing site visit reports,  speaking with beneficiaries and retailers, and meeting regularly with  partners’ in-country staff. In addition, the FFP field officers said they  discuss the results of their monitoring efforts and site visit reports with the  grant officers on a regular basis. To aid FFP field officers in conducting  site visits, the FFP Monitoring and Evaluation Team, in 2015, developed a  monitoring tool that includes sample questions for FFP officers to  consider when making site visit observations and to ask respondents  during monitoring visits. The tool is organized by type of respondent (i.e.,  implementing partner staff, beneficiaries, service providers, and retailers  or market vendors), and the information collected is used to complete trip  reports. According to FFP officials, as of December 2015, the Monitoring  and Evaluation Team had shared the tool with every USAID field office.

To ensure regular monitoring of all FFP programs in a country, the FFP  Monitoring and Evaluation Team began training FFP field officers in  developing country monitoring plans in 2014, according to USAID  officials. Agency officials said the country monitoring plans use a risk- based approach to prioritize the monitoring of country projects across the  FFP portfolio (including EFSP and Title II projects), to establish the  number of site visits per month and year, to determine which monitoring  activities to conduct, and to allocate staff resources. In addition, USAID  officials said they plan to institute a requirement that FFP officers in  missions with FFP programs must complete country monitoring plans and  report periodically on progress in implementing objectives identified in the  country monitoring plan. As of June 2016, USAID had developed  monitoring plans for 26 of 28 countries. According to the FFP officers  we spoke with, country monitoring plans have helped them prioritize  monitoring site visits for both Title II and EFSP projects despite  challenges such as limited embassy resources (e.g., motor-pool  availability and staffing shortfalls), adverse operating conditions (e.g.,  road closures or weather), and security concerns. According to USAID  officials, to mitigate security-related constraints, FFP has awarded  contracts to third-party monitors in countries where USAID has limited  access. For example, for Somalia, USAID contracted with a third-party  monitor to verify project activities and conduct post-distribution surveys on  a monthly basis for 20 percent of the project sites. According to a  representative of the monitor, it uses a risk-based approach to monitoring,  prioritizing site visits and rotating the sites visited on a monthly basis.

Implementing Partners Have Established Processes to Monitor Cash and Voucher Projects during and after Distribution of Assistance

To monitor projects and collect required information, implementing  partners have established processes to monitor activities during and after  distributions of EFSP cash transfer and food voucher projects.

Distribution monitoring. According to implementing partner  representatives for projects in Kenya, Liberia, and Somalia, their  monitoring processes include observing distribution processes to  ensure that the cash transfers and food vouchers are delivered  according to standard procedures, for the stated purposes, and to  targeted beneficiaries. The representatives said that during  distribution, monitors collect information on the actual number of  beneficiaries served, the actual transfer value provided, the timing of  the distribution, and beneficiary participation by gender and age. They  also collect information about the distribution site and other aspects of  the distribution process, such as queue management, waiting time,  beneficiary verifications, access, security, and safety. In addition,  implementing partner monitors may speak with beneficiaries about  their satisfaction with the distribution process. Implementing partners  also monitor financial service providers to ensure that their  implementation of the cash transfer is according to plan. In addition,  implementing partner representatives monitor the voucher process at  the vendor or retailers’ level.

Figure 4 shows implementing partner representatives conducting  distribution monitoring for a cash transfer project in Liberia.

Postdistribution monitoring. According to implementing partner  representatives for programs in Kenya, Liberia, and Somalia, their  processes include surveying beneficiaries and non-beneficiaries at  their residences after the assistance is distributed. According to the  implementing partners, postdistribution surveys are one of the main  information sources for assessing the relevance, efficiency, and  effectiveness of the assistance provided. During the surveys,  implementing partner representatives gather information about  beneficiary households, including, among others, income,  expenditures, food security outcomes, coping strategies, perceived  problems with the assistance, and modality preferences. In addition,  implementing partner representatives said their processes also  include monitoring markets after distributions by sampling traders,  shops, and markets for changes in commodity prices. Further, for food  voucher projects, implementing partner representatives for the project  in Somalia said that they also monitor vendors or retailers to collect  information on food voucher use and availability of commodities,  accuracy of commodity price displays, and to determine if the food  vouchers are used for intended purposes.

We observed implementing partner representatives using technology and  tools that may enhance the collection and analysis of monitoring data. For  example, implementing partner representatives for the project in Somalia,  based in Nairobi, Kenya, were using beneficiary management systems to  register beneficiaries and were remotely monitoring beneficiary purchases  and redemptions, and vendor sales and activities, for an EFSP food  voucher project in Somalia. In Kenya and Liberia, we observed  representatives of implementing partners for projects in those countries  using mobile devices with standardized forms to conduct on-site  distribution monitoring, postdistribution monitoring, and market monitoring  surveys (see fig. 5). Implementing partner representatives showed us the  tablets and standardized forms they use for collecting monitoring  information for their projects in Somalia. According to the implementing  partners, using this technology enables rapid aggregation and analysis of  the data collected. We also observed implementing partner  representatives conducting remote phone-based surveys with  beneficiaries for projects in Kenya and Somalia. The survey included a  short series of questions on household food consumption and coping  strategies. Further, the implementing partners established hotlines in  Kenya and Somalia for beneficiaries to provide feedback, including  complaints, about project implementation and the assistance provided.

Data Gaps and Indicator Weaknesses Limit USAID’s Ability to Evaluate the Performance of Cash and Voucher Projects

Incomplete reporting of data that USAID requires, as well as weaknesses  in USAID’s performance indicators, limits the agency’s ability to evaluate  the projects’ performance. Our review of the final reports for 14 cash  transfer and food voucher projects found that most of the reports lacked  some of the required data that are necessary for performance evaluation.  In addition, the indicators that USAID uses to assess projects’ timeliness,  cost-effectiveness, and appropriateness—three criteria that it considers in  approving grant applications—have weaknesses. For example, USAID’s  indicator for timeliness does not capture delays in actual implementation  against what was planned. Moreover, USAID’s indicator for cost- effectiveness—cost per beneficiary—does not produce data that can be  compared across projects or modalities, because it does not include a  standardized unit for measuring costs. Additionally, USAID has not set a  benchmark for assessing market appropriateness based on market  prices. These limitations affect USAID’s ability not only to evaluate the  overall performance of cash transfer and food voucher programs but also  to learn from experience and make informed decisions on future projects.

Most Final Reports We Reviewed Lacked Required Data about Project Performance

Our review of the final reports submitted for the 14 cash transfer and food  voucher projects found that most of the reports lacked some data about  the projects’ performance that USAID’s APS required. According to  USAID officials, implementing partners may have communicated the  missing information through other means, such as during meetings or in  e-mails or quarterly reports; however, the 2013 APS, to which the projects  were subject, required partners to submit the information in their projects’  final report. The APS required the final report for each project to include  the number of beneficiaries targeted and reached, disaggregated by sex  and age, and to verify that the program assessed and addressed gender  needs and issues. In addition, the APS required the final report to list the  planned and actual number and value of food vouchers or cash transfers  that implementing partners distributed and beneficiaries redeemed.  Further, the APS required the final report to include information on how  the beneficiaries used cash transfers as well as information on the types  and quantities of commodities that beneficiaries procured with food  vouchers. Finally, the APS required the final report to include data that  the agency will use to measure projects’ overall performance in terms of  timeliness, cost-effectiveness, and appropriateness: the time from donor- signed agreement to first distribution to beneficiaries; the cost per  beneficiary; and retail price information on key staples in the area of the  program before, during, and after the distribution.

While most of the 14 final reports that we reviewed included most of the  required data on project beneficiaries, only 1 report, for a food voucher  project in Sudan, included all 12 data elements required by USAID; the  other reports lacked up to 8 of the required elements. Figure 6 shows the  required data elements that were included in the 14 final reports that we  reviewed.

Data on beneficiaries. All 14 final reports that we reviewed included data  on the number of beneficiaries reached, although several reports were  missing other required data about the beneficiaries who received the cash  transfers or food vouchers. The reports indicated that cash transfer or  food voucher projects reached almost 926,000 beneficiaries in Burundi,  Chad, the DRC, Haiti, the Philippines, Somalia, Sudan, Yemen, and  Zimbabwe. Two reports did not list the number of beneficiaries targeted  by the project. In the 12 reports that listed these data, our analysis found  that 9 of the 12 projects reached or exceeded their target numbers.  However, 4 reports did not enumerate beneficiaries by age range and 4  reports did not describe how gender needs were assessed and  addressed, as USAID requires.

Data on assistance distributed and redeemed. Half of the 14 reports  we reviewed did not list the planned value of cash transfers or the number  of food vouchers distributed as the APS requires. Only 5 reports—for  projects in Chad, DRC, Somalia, Sudan, and Zimbabwe—included all  required information about the actual number and value of cash transfers  or food vouchers distributed and redeemed. Further, of the 4 final reports  for cash transfer projects, only 1 report included the required information  about beneficiaries’ use of the transferred cash. Of the 8 final reports for  food voucher projects, only 4 listed, as required, the types and quantities  of commodities that beneficiaries procured with their vouchers. Only 1 of  2 projects that comprised both cash transfer and food vouchers included  the required information on the beneficiaries’ use of the cash and types  and on quantities of commodities procured with the vouchers.

Data on timeliness, cost-effectiveness, and appropriateness. Most of  the 14 reports we reviewed were missing required data that USAID uses  to analyze the timeliness, cost-effectiveness, and appropriateness of cash  transfer or food voucher projects.

Timeliness. Twelve of the 14 reports did not list the number of days  from the award agreement date to the cash transfer or food voucher  distribution date. In addition, 5 of the 14 reports did not list the first  date of distribution, making it difficult to determine the time from the  partner’s signing of the agreement with USAID to the first distribution  of the cash transfers or food vouchers. We requested any data that  implementing partners had submitted through other documents, such  as e-mails or quarterly reports; however, USAID did not provide such  data.

Cost-effectiveness. Six of the 14 final reports did not list a cost per  beneficiary as required by USAID. According to a country officer, the  final report for one project excluded the cost per beneficiary because  the project, which was jointly funded by FFP and the Office of U.S.  Foreign Disaster Assistance (OFDA), used an OFDA award  mechanism that does not require this calculation.

Appropriateness. Twelve of the 14 final reports did not list retail price  information on key staples in the area of the program before, during,  and after the cash transfer or food voucher distribution as required by  USAID. Some of the 12 reports listed commodity prices during  distributions but did not list the prices 2 weeks before the program  began or 2 weeks after it ended. According to USAID, country officers  do not request additional price data from partners if there is no  evidence of projects’ having a negative impact on the local market.

USAID’s Current Indicators for Assessing Cash and Voucher Projects’ Timeliness, Cost- Effectiveness, and Appropriateness Have Weaknesses

USAID’s indicators for measuring cash transfer and food voucher  projects’ timeliness, cost-effectiveness, and appropriateness have  weaknesses that limit the agency’s ability to evaluate these aspects of  projects’ performance. USAID’s indicator for timeliness produces data  that show how quickly the distribution occurs after the award agreement  is signed; however, the indicator does not compare actual and planned  distribution schedules, which would identify delays in project  implementation. Moreover, USAID’s indicator for cost-effectiveness does  not produce data that can be compared across projects or modalities,  because it does not include a standardized unit for measuring costs.  Further, USAID does not have a benchmark for the market  appropriateness indicators that measure the impact of cash or voucher  assistance on local markets. According to federal standards for internal  control, management should use quality information, including relevant  data from reliable sources, to achieve an agency’s objectives. As a  result of the weaknesses in these indicators, USAID and its implementing  partners have limited ability to fully evaluate cash transfer and food  voucher projects’ timeliness, cost-effectiveness, and appropriateness and  may miss opportunities to learn from past experiences to improve the  program.

USAID’s Indicator for Timeliness Does Not Track Implementation Delays

The indicator that USAID and its implementing partners use to assess  cash transfer and food voucher projects’ timeliness provides a measure of  how quickly assistance is delivered after an award is signed, but it does  not systematically capture any delays in project implementation. Since  2011, USAID’s APS has required that the final report for each cash  transfer and food voucher project include, as an indicator of the project’s  timeliness, the number of days between grant approval and distribution of  assistance. According to USAID officials, the agency uses this measure  to determine how quickly assistance can be provided under different  modalities. USAID also told us that it considers the duration of time  between award and first distribution, based on the proposed project  timeline, as a general indicator of implementing partners’ initial progress  in starting a project. USAID officials noted that anecdotal evidence  suggests that the number of days between grant approval and first  distribution of assistance has generally decreased for cash transfer and  food voucher projects.

The number of days between grant approval and first distribution of  assistance does not register delays in project implementation—an  important aspect of timeliness, since delays in implementing cash and  voucher projects can have a severe impact on beneficiaries who rely on  the assistance for their livelihood. USAID does not systematically collect  data that would show cash transfer and food voucher project delays, such  as the planned dates of distribution for comparison with the actual dates.

Since 2011, the APS has not required partners to list in their final reports  the planned date of first distribution, which appears in the implementing  partner’s award proposal. Delays have been noted for many EFSP  projects. For example, a recent USAID Inspector General report  compared the planned and actual distribution dates for four cash transfer  and food voucher projects in West Africa during the Ebola response and  concluded that delays in distribution—resulting from delays in award  approvals and challenges in staff recruitment and coordination—averaged  3 months. We requested and obtained the planned first-distribution  dates for the 14 cash transfer and food voucher projects we reviewed and  then compared the planned dates with the actual dates listed in the  projects’ final reports. We found that for 7 of the 14 projects, first  distribution of assistance was delayed by an average of 2 months.

USAID’s Indicator for Cost- Effectiveness Does Not Produce Comparable Data

The indicator that USAID uses to measure the cost-effectiveness of cash  transfer and food voucher projects does not include a standardized unit  for measuring cost. As a result, USAID is unable to use the cost data it  collects to assess the relative cost-effectiveness of its EFSP cash transfer  and food voucher projects or to compare the cost-effectiveness of such  projects with that of projects that used other modalities. Since 2011,  USAID’s APS has required that the final report for each cash or voucher  project include, as an indicator of cost-effectiveness, the cost per  beneficiary. This cost is calculated by dividing the aggregate cost of  providing assistance by the actual number of beneficiaries of a cash  transfer or food voucher project.

We found that in the 8 final reports that listed the cash transfer or food  voucher project’s cost per beneficiary—USAID’s indicator for cost- effectiveness—the reported costs could not be used to compare the  projects’ cost-effectiveness because the cost units were not standardized.  In these 8 reports, the reported costs ranged widely, from $10 to $219 per  beneficiary. According to USAID, per-beneficiary costs may range widely  for a number of reasons, including variance in the size of the transfers,  the frequency of the transfers, the presence of complementary services,  the overall scale of the projects, and overhead costs for each project. For  example, given economies of scale, a project that prints vouchers for  50,000 beneficiaries may have lower overhead costs than a project that  prints vouchers for 10,000. Also, indirect costs, such as for security or  transportation, may be higher for projects that require additional security  or are located in remote, less accessible districts. In addition, according to  an implementing partner responsible for a cash transfer project and a  food voucher project in Liberia, its cost-per-beneficiary calculation for the  voucher project did not include indirect costs because they were included  in its calculation for the cash transfer project. USAID officials  acknowledged that without a standardized cost unit, partners have  applied different methods to calculate the cost per beneficiary for cash  transfer and food voucher projects.

To weigh the cost-effectiveness of various food aid modalities during  project design, one of USAID’s implementing partners, the United Nations  World Food Programme (WFP), uses a method that compares different  modalities’ costs for delivering the same numbers of calories and for  delivering the same nutritional values. Using this method—known as the  Omega tool analysis—to compare potential costs, WFP determined that  in Burkina Faso and Niger, food transfers could deliver the same  nutritional value at a lower cost than cash transfers or food vouchers.  However, WFP’s analysis show that in both cases, combining modalities  could deliver the same nutritional value at a lower cost than using a single  modality. In Senegal, WFP determined that food vouchers or a  combination of food transfers and cash transfers or food vouchers could  deliver the same nutritional value at a lower cost than food transfers.  According to WFP officials, as of July 2016, WFP had used this tool while  designing projects in 24 countries and plans to rely on this tool to assess  cost-efficiency and cost-effectiveness. WFP officials also noted that  WFP’s country offices planned to use this tool to validate cost-efficiency  assumptions at the close of programs.

According to a USAID official, the agency is considering revising the APS  to require grant applicants to use the cost of a food ration for cash-based  and in-kind food aid projects when estimating potential cost-effectiveness.  The official indicated that the change is intended to standardize methods  for justifying proposed projects on the basis of cost-effectiveness. The  draft version of the APS that USAID released for public comment in May  2016 provides more explicit parameters for estimating cost-effectiveness  that could enable the agency to compare proposed modalities during the  application process.

USAID’s Indicator for Cash and Voucher Projects’ Appropriateness Lacks Associated Benchmarks for Market Impact

USAID has not required implementing partners to establish benchmarks  for measuring cash transfer and food voucher projects’ impact on prices  in local markets—an indicator it uses to measure the projects’  appropriateness. Since 2010, the APS has required that implementing  partners’ final reports include retail price information on key staples in the  area around the project before the project begins, monthly during the  project, and after the project ends. However, USAID has not required the  partners to establish any associated benchmarks for this indicator and as  a result, USAID and implementing partners may not be able to assess  whether market price fluctuations are within acceptable ranges.

Implementing partners are using different thresholds to assess market  impact. For example, WFP’s policy is to reassess the value of the  assistance provided if prices for staple foods fluctuate by 10 percent or  more. According to WFP officials, WFP may also change the assistance  modality as a result of price fluctuations. Representatives of other  implementing partners told us that they reassess project implementation  when prices fluctuate by 20 percent or more. Implementing partner  representatives also stated that, because of a lack of guidance on this  topic, they were unsure of how USAID assesses the market information  they submit. As of the end of June 2016, USAID’s draft revisions to the  APS included a reference to market monitoring guidance for  implementing partners. According to this guidance, implementing partners  are to set price thresholds as a basis for determining when price changes  must be investigated and explained. The guidance also states that when  prices increase or decrease beyond this set parameter, the change  should be flagged and the cause should be investigated. The draft  revisions to the APS also require partners to include market analysis  information and any significant changes in their quarterly and final project  reports. The draft revisions to the APS do not explicitly require partners to  report the price thresholds in their final reports.

Studies Showed Cash, Vouchers, and Food Transfers Can Improve Food Security, and Contextual Factors May Explain Impact Variations

Our review of 14 rigorous studies of the relative impacts of cash transfers,  food vouchers, and food transfers found the studies demonstrate that all  three modalities can improve food security outcomes for people facing  food emergencies. The modalities’ impacts on food security outcomes  varied by study, with no modality consistently outperforming the others.  Contextual factors, such as the severity of the food crisis and the capacity  of local markets, may have contributed to this variation. Studies that  compared the relative costs of the three modalities generally reported that  cash transfers were least expensive, although most of the studies did not  account for the full costs associated with the modalities.

Studies Demonstrated That All Three Modalities Can Improve Food Security Outcomes

Among the studies we reviewed, those comparing recipients of cash  transfers, food vouchers, and food transfers with control groups of  individuals who received no assistance demonstrated that all three  modalities can lead to significant improvements in dietary quantity and  quality. For example, a study in Ecuador found that, compared with the  control group, all recipients of cash transfers, food vouchers, and food  transfers experienced significantly improved outcomes in terms of the  value and volume of food consumed, caloric intake, and dietary diversity.  Similarly, a study in Bangladesh found that all recipients of cash transfers,  food transfers, or combinations of the two modalities experienced  significantly improved outcomes for value and volume of food and for  caloric intake.

Eight of the 14 studies we reviewed included control groups, which  allowed us to compare food security outcomes for recipients of cash  transfers, food vouchers, or food transfers with outcomes for groups of  individuals that did not receive any type of assistance. Among 6 studies  that used control groups to examine the value and volume of food  provided, 5 studies showed statistically significant improvements as a  result of receiving all three types of assistance. Similarly, 3 of 6 studies  that used control groups to examine caloric intake also found statistically  significant improvements for all three modalities.

The studies showed that all three assistance modalities generally led to  significantly improved outcomes for the most frequently assessed  outcomes in the studies—value and volume of food consumed, caloric  intake, and dietary quality. However, the results for some other outcomes  were mixed. In particular, 3 of the 4 studies that considered nutritional  status showed some modalities leading to significant improvements, while  the fourth study showed no modality leading to significant improvements.

Studies Showed the Modalities Had Varying Impacts on Food Security Outcomes

Our analysis of the 14 studies comparing the impacts of cash transfers,  food vouchers, and food transfers on food security outcomes found that  the modalities’ performance varied by study and project. Moreover, none  of the three modalities consistently outperformed the others. Contextual  differences among the projects may have contributed to the variations in  the modalities’ impacts. (For detailed results of our literature review, see  app. II.)

Modality Performance Varied by Study, and No Modality Consistently Outperformed the Others

Our review of the 14 studies found that the impacts of the modalities on  food security outcomes varied by study. For example, the study of the  project in Yemen reported that recipients of the cash transfers bought a  wider range and higher value of food items, showing that cash transfers  provided significantly greater improvements in dietary quality than did  food transfers. The Yemen study also found that food transfers provided  higher levels of caloric intake than did cash transfers, which the study’s  authors attributed to the relatively inexpensive staples, such as wheat and  oil, included in the food transfers.

In contrast, a study of a project in Niger found that food transfers resulted  in significantly greater improvements in dietary quality than did cash  transfers, indicating that, in this instance, food transfers provided a more  varied and higher-quality diet. This finding was attributed to the fact that  cash recipients in Niger bought significantly cheaper bulk grains with their  transfers than food recipients. The study authors determined that the  cash beneficiaries purchased these cheap bulk grains in anticipation of  seasonal price increases—essentially stocking up on supplies for the  “hungry” season. As a result, the food recipients, who relied on the food  transfers provided by the project, had a more varied and higher-quality  diet.

In some instances, a modality significantly outperformed one or both of  the other modalities for a particular outcome in some countries but not in  others. For example, studies of projects in Niger, Sri Lanka, Uganda, and  Yemen showed that cash transfers had a greater impact on the value and  volume of food than food transfers had, but studies of projects in  Bangladesh, the DRC, Ecuador, and Mexico showed that the three  modalities had comparable impacts on this outcome. Moreover, a study of  a project in Niger comparing the impacts of the three modalities found  that food transfers provided the greatest dietary diversity, while studies of  projects in Malawi, Uganda, and Yemen found that cash transfers  provided greater dietary diversity than food transfers. The DRC and  Ecuador studies did not find that any modality had a consistently and  significantly greater impact on dietary diversity.

Differences in Modality Impacts Were Often Not Statistically Significant

In 11 of the 14 studies, some of the differences in modalities’ impacts on  food security outcomes were not statistically significant and therefore may  have resulted by chance. For example, the study of the Ecuador project  found that cash transfers, food vouchers, and food transfers all led to  improvements in beneficiaries’ dietary diversity. While statistically  significant compared with the control group, the improvements that  resulted from each of the three modalities were not statistically different  from each other across the metrics studied. Similarly, a study of a project  in Mexico noted that both cash transfers and food transfers increased the  value and volume of food consumed, and levels of nutrients, with  statistically insignificant differences in the two modalities’ impacts.

Contextual Factors May Help Explain Variations in Modality Impacts

While we could not determine precisely why the three modalities’ impacts  varied in the studies we reviewed, contextual differences in the projects  the studies examined may help explain such variation. According to  researchers we spoke with, the effectiveness of cash transfers, food  vouchers, and food transfers are heavily influenced by contextual factors  such as the severity of the food crisis, beneficiaries’ specific needs when  projects began, changes in market prices, and the projects’ designs.

While each of the 14 studies we reviewed controlled for multiple factors  when comparing the modalities’ impacts on food security outcomes, we  noted numerous contextual differences among the evaluated projects that  may have contributed to variation in these impacts. The projects’  purposes and goals ranged from providing disaster relief to responding to  seasonal food emergencies, aiding displaced persons, addressing long- standing emergencies, or helping achieve development objectives. In  addition, the projects varied in factors such as the value and frequency of  the transfers and whether the assistance was conditional or  unconditional. For example, a study in Sri Lanka examined a 3-month  project that took place in tsunami-affected regions of the country. The  study found that cash transfers had a greater impact than food transfers  on dietary quantity. However, its findings are difficult to apply to many  other situations because of the post-tsunami conditions that prevailed, the  relatively short duration of the emergency relief project, and some  unevenness in the frequency of food transfers relative to cash transfers.

Changes in markets and prices can affect the modalities’ relative impacts  on food security outcomes. For example, a study of a project in Ethiopia  found that both food transfers and a mixture of food and cash transfers  significantly outperformed cash-only transfers in reducing the periods of  time when beneficiaries experienced food shortages. However, the  study’s authors noted that the project took place during a period of high  food-price inflation that considerably reduced the value of the cash  transfers compared with that of the food transfers.

Moreover, differences in projects’ design—specifically, whether the  project provided any assistance in addition to the food or cash transfers— may have contributed to variation in the modalities’ impacts. For example,  a study of a project in Bangladesh concluded that aspects of project  design, and in particular the use of complementary programs designed to  achieve project outcomes, were associated with the greatest impacts.  This study found that modalities combined with complementary programs  that provided guidance and training on nutrition significantly outperformed  cash transfers, food vouchers, and food transfers that were not combined  with complementary programs. According to the study’s authors, these  results help demonstrate that food security outcomes can be improved by  including complementary programs designed to achieve those objectives  and that these improvements occur when such programs are combined  with any of the three modalities.

Studies Generally Reported Cash Transfers Were Least Expensive, but Fewer Than Half Considered Full Costs

Among the 14 studies we reviewed, 11 considered the relative costs of  cash transfers, food vouchers, and food transfers. Seven of these 11  studies reported that cash transfers were least expensive, while 1 study  reported that food transfers were least expensive. Of the 3 remaining  studies that considered the modalities’ relative costs, 1 study did not  identify the least expensive modality and the other 2 studies had mixed  results, depending on the method used to estimate costs. The studies  considered costs for a range of activities related to delivering the  assistance, such as administration, staffing, banking for cash transfers,  production for food vouchers, and storage and transportation for food  transfers. However, only 3 of the 11 studies considered not only the  activity costs related to delivering the assistance but also the costs of  purchasing food.

The three studies that considered the costs of purchasing food reached  varying conclusions about the modalities’ overall relative costs. One  study, of a project in Niger, found that food transfers were least expensive  overall. Another study, of a project in Malawi, reported mixed results,  finding that cash was more cost-effective but food transfers were more  cost-efficient. The third study, of a project in Bangladesh, did not clearly  identify the least expensive modality.

According to researchers, food transfers may be the least expensive  modality overall if market conditions allow implementing partners to  purchase food at prices low enough to offset the costs of delivering it,  since these costs are usually higher than the costs of delivering cash  transfers or food vouchers. For example, in some situations,  implementing partners’ savings from purchasing large amounts of food  wholesale—that is, for considerably lower prices than beneficiaries would  pay in their local markets—could offset the higher costs of delivering the  food.

With USAID’s additional flexibility to choose among modalities of food  assistance, the ability to demonstrate the performance of each modality is  of increasing importance. Monitoring and evaluation are essential to  assessing and demonstrating, with timely and credible evidence, the  effectiveness of the various modalities employed to deliver assistance,  including cash transfers and food vouchers. USAID and its implementing  partners have established processes to monitor EFSP cash transfer and  food voucher projects. Specifically, USAID has assigned roles and  responsibilities and developed monitoring plans and tools to aid FFP field  officers who conduct site-visits and meet with implementing partners.  Further, implementing partners have developed processes to monitor the  effectiveness of their projects through distribution and postdistribution  monitoring. However, the incompleteness of required information about  project performance in partners’ final reports limits USAID’s ability to  assess whether EFSP cash and voucher projects met their performance  goals. In addition, weaknesses in USAID’s indicators for measuring EFSP  cash transfer and food voucher projects’ timeliness, cost, and  appropriateness—criteria that USAID considers in approving the  projects—limit the extent to which reported data can demonstrate the  effectiveness of such projects and be used to evaluate the performance  of cash and vouchers relative to in-kind food aid.

The 14 studies we reviewed showed that cash transfers, food vouchers,  and food transfers can all improve food security. At the same time, the  studies’ findings of variation in the modalities’ impact on food security  suggest that when selecting modalities for emergency food assistance,  USAID and other donors should carefully consider contextual factors that  could influence project outcomes. By taking steps to strengthen  monitoring and evaluation of its cash transfer and food voucher projects,  USAID will ensure access to information about each implemented project  that it can use in planning future projects—including selecting the  appropriate modality—and will help USAID optimize its efforts to respond  to continuing food emergencies around the world.

To strengthen USAID’s monitoring and evaluation of cash transfer and  food voucher projects and help ensure improved program oversight of  these projects, we recommend that the USAID Administrator take the  following two actions:

Take steps to ensure that final reports submitted for cash transfer and  food voucher projects comply with USAID’s minimum data  requirements.

Strengthen the indicators USAID uses to measure the timeliness,  cost-effectiveness, and appropriateness of EFSP cash transfer and  food voucher projects.

We provided a draft of this report to USAID, which provided both written  and technical comments. In its written comments, reproduced in appendix  IV, USAID agreed with our findings and recommendations. Regarding our  first recommendation, USAID agreed to standardize data collected in final  reporting by soliciting and hiring program support officers for each of its  geographic teams. Regarding our second recommendation, USAID  agreed to improve indicators to ensure evaluation and comparison across  its emergency food assistance portfolio by updating these indictors in its  forthcoming Annual Program Statement for International Emergency Food  Assistance. We incorporated USAID’s technical comments as  appropriate.

We are sending copies of this report to the appropriate congressional  committees; the Administrator of USAID; and the Secretary of State. The  report is also available at no charge on GAO’s website at  http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-9601 or melitot@gao.gov. Contact points for our Offices  of Congressional Relations and of Public Affairs may be found on the last  page of this report. GAO staff who made major contributions to this report  are listed in appendix V.

Appendix I: Objectives, Scope and Methodology

In this report, we (1) examine the U.S. Agency for International  Development’s (USAID) and implementing partners’ processes for  monitoring cash transfer and food voucher projects; (2) analyze the extent  to which monitoring data that partners reported to USAID can be used to  evaluate the performance of such projects; and (3) review studies that  examined the relative impacts of cash transfers, food vouchers, and food  transfers on food security outcomes and the relative cost of these  modalities.

To address these objectives, we analyzed data and reviewed program  documents provided by USAID and its implementing partners for  Emergency Food Security Program (EFSP) cash transfer and food  voucher projects, including the United Nations World Food Programme  (WFP) and nongovernmental organizations (NGOs). We met in  Washington, D.C., with USAID officials and with implementing partner  officials representing NGOs that received USAID EFSP grants, and we  met in Rome, Italy, with officials from WFP and the U.S. Mission to the  UN. In addition, we visited Kenya and Liberia, where we reviewed  projects that were under way in Kenya, Liberia, and Somalia. We selected  these projects using criteria that included the amount of EFSP funding,  the type of project (cash transfer or food voucher), and the implementing  partner. While in Kenya and Liberia, we met with USAID officials from the  U.S. missions; representatives of implementing partners, vendors, and  financial institutions; and project beneficiaries, among others.

To examine USAID’s and implementing partners’ processes for  monitoring EFSP cash and voucher projects, we reviewed activities that  USAID and implementing partners, including WFP and NGOs, undertook.  We also reviewed relevant program documents that they provided, such  as award agreements, project quarterly and final reports, market  monitoring tools, site visit reports, and results of distribution surveys. We  interviewed USAID officials in Washington, D.C., including award  agreement officers, country backstop officers, and members of FFP’s  monitoring and evaluation team, among others. We also interviewed FFP  regional monitoring officers located in Dakar, Senegal, and in Nairobi,  Kenya, and we interviewed FFP field officers responsible for monitoring  cash and voucher projects in eight countries. In addition, we met in  Washington, D.C., with implementing partner officials representing NGOs  that were awarded USAID EFSP grants and in Rome, Italy, with officials  from WFP and the U.S. Mission to the UN. Further, we visited Kenya and  Liberia, where we reviewed EFSP-funded cash and voucher projects  under way in Kenya, Liberia, and Somalia. In Kenya and Liberia, we also  visited project sites, observed implementing partners conducting  monitoring activities, and met with beneficiaries of projects in those  countries.

To examine the extent to which reported data can be used to evaluate the  performance of cash transfer and food voucher projects, we reviewed  USAID’s 2013 Annual Program Statement (APS) for International  Emergency Food Assistance and grant agreements for final reporting  requirements. In addition, we reviewed 14 final reports that implementing  partners submitted to USAID for, respectively, four cash transfer projects,  eight food voucher projects, and two projects with both cash transfer and  food voucher components. We selected these projects using the following  criteria, intended to ensure a diverse sample of implementing partners:  (1) the funding was awarded for NGO projects in fiscal years 2013 and  2014 or for WFP projects in fiscal year 2014, all of which were subject to  USAID’s 2013 APS; (2) the award was at least $2 million for NGO  projects or at least $3 million for WFP projects; and (3) the projects  closed by October 31, 2015. Of the 36 NGO projects funded in fiscal  years 2013 and 2014, 22 received awards of at least $2 million; of these,  only 11 were closed by October 31, 2015, and therefore were included in  our review. Of the 18 WFP projects funded in fiscal year 2014, 7 received  awards of at least $3 million; of these 7 projects, only 3 were closed by  October 31, 2015, and therefore were included in our review. Because  the 14 projects represent a nonprobability sample, our findings may not  be generalizable to all USAID EFSP programs. We reviewed the APS and  grant agreements to identify minimum programmatic reporting  requirements, and we reviewed the final reports for the 14 selected  projects for data responding to these requirements. In addition, we  reviewed the 14 final reports for data and the indicators that USAID uses  to measure cash transfer and food voucher projects’ timeliness, cost- effectiveness, and appropriateness—namely, time from award to first  distribution, cost per beneficiary, and food price data. We met with USAID  officials in Washington, D.C., to discuss APS requirements, implementing  partner reports, and the quality of indicators.

To review studies’ conclusions about the relative impacts of cash  transfers, food vouchers, and food transfers on food security and the  modalities’ relative costs, we took the following steps:

To identify relevant studies, we (1) considered prior reviews of  rigorous evaluations related to cash-based humanitarian assistance  and food security; (2) conducted our own search of literature using  appropriate terms; (3) and asked six researchers who had conducted  prior reviews of rigorous evaluations whether they knew of any  additional evaluations. We screened the studies identified by these  sources to determine whether any of the studies met the following  criteria: (1) compared cash transfers, food vouchers, or food  transfers with at least one other modality; (2) evaluated at least one  food security outcome; and (3) considered food security outcomes by  means of randomized control trials or groups, carefully selected  comparison groups, or a quasi-experimental design that used  statistical techniques to make precise comparisons. This process  resulted in our selecting 14 rigorous studies, made public since 2006,  that evaluated whether cash transfers, food vouchers, or food  transfers were more successful in achieving intended impacts on food  security outcomes in situations where at least two modalities were  implemented and could be compared. These 14 studies examined  projects in 10 countries: Bangladesh, the Democratic Republic of the  Congo, Ecuador, Ethiopia, Malawi, Mexico, Niger, Sri Lanka, Yemen,  and Uganda. (See app. II for a list of these studies.)

To review the 14 studies we selected, we used a data collection  instrument (DCI) designed to examine the studies’ design, quality, and  major findings. (App. II lists the studies we reviewed and provides a  detailed summary of the results of our review.)

We developed the following eight key food security outcomes based  on our analysis of the metrics in the 14 selected studies and on our  discussions with researchers with relevant expertise: value and  volume of food, caloric intake, dietary diversity, food consumption  score, nutrient number and levels, caloric value, experiential  measures, and nutritional status. We reviewed each study’s results to  determine whether any modality demonstrated a statistically  significant improvement in one or more of these outcomes compared  with the other modality, or modalities, that the study examined. Two  independent analysts used the DCI to review the studies, discussing  and reconciling any differences in their initial assessments. In some  instances, studies presented the results for several metrics for the  same outcome, such as a household dietary diversity score and a  household dietary diversity index. In those instances, our decision rule  was that one modality had to demonstrate a statistically significant  improvement over all other modalities examined for all metrics that the  outcome comprised. If this criterion was not met, we determined that  the modalities’ impacts were comparable. We did not consider results  for subpopulations, such as by region, income group, gender, or age  range, except when results for a particular outcome were reported  only for a subpopulation. We also did not consider impacts on factors  that did not qualify as costs or food security outcomes, such as  beneficiaries’ income, assets, and overall consumption. When  examining findings about dietary diversity, we relied on indices and  metrics that had been created for this purpose; we did not assess  results for individual food groupings, as these varied greatly between  studies and did not lend themselves to the overall methodology that  we employed. (App. II summarizes the results of our review.) The  researchers who helped us identify relevant studies that met our  criteria generally stated that they believed the 14 studies we selected  constituted a reasonable body of evidence that no one modality  significantly outperforms the other and that all modalities can lead to  improvements in food security. However, one researcher reported that  the evidence base does not allow for generalizable conclusions about  the situations in which specific modalities are most appropriate.

We also considered whether the 14 studies examined the relative  costs of the modalities, and in the studies that did, we considered  which modality was found to be least expensive at delivering the  same level of assistance. The studies employed a variety of data and  methods to estimate relative costs. For example, some studies  examined cost-efficiency, others examined cost-effectiveness, and  still others examined both. In addition, some studies used a costing  method adapted from the health economics field that involves  identifying the costs of specific “activities” required to implement a  modality, such as staffing, banking, and production. Meanwhile, three  studies comparing cash and food transfers examined the costs of both  delivering the modalities and purchasing the food beneficiaries  consumed. If more than one method was used to estimate costs in a  study, we considered the results for all methods used and reported  that one modality was the least expensive only if all methods used  found it so. Otherwise, we reported that the results were mixed. In one  instance, a study examined four projects in one country: one project  provided food transfers, another provided cash transfers, and two  others provided a mix of cash and food transfers. This study  examined both the costs and the cost-efficiency of the transfers. In  that instance, because the results differed by project and method  used, we could not determine which modality was least expensive.

Our review of the 14 studies was not intended to assess any individual  donor such as USAID and thus the conclusions we analyzed cannot be  inferred to assess the performance of USAID’s projects. In addition,  several of the projects examined were pilots conducted to assess the  comparative performance of food and cash on household food security.

The projects that the studies examined varied in terms of contextual  factors such as the projects’ purposes and goals, projects’ design and  outcomes examined. While the studies demonstrate that no modality  consistently outperformed the others across a range of settings and  situations, this variance does not allow us to determine the precise  conditions in which one modality might outperform the others. To obtain  context and background, we also considered additional studies on cash- based assistance that did not meet our criteria for inclusion in our  systematic review but provided insights into projects that used cash  transfers, food vouchers, food transfers, or a combination of these  modalities. (See app. III for a list of these additional studies.)

We conducted this performance audit from June 2015 to September 2016  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

Appendix II: Summary Results of GAO Literature Review

Appendix III: Additional Studies Examining Food Aid Modalities

In addition to reviewing 14 rigorous studies that compared the impacts of  cash transfers, food vouchers, and food transfers on food security  outcomes, we consulted the following 23 studies for insights into projects  that included one or more of these modalities. In some cases, these  studies examined large-scale cash transfer or food voucher projects or  the relative costs of the modalities and were recommended by food  security researchers we interviewed.  1.  Attanasio, David, Erich Battistin, and Alice Mesanard. “Food and  Cash Transfer: Evidence from Colombia.” The Economic Journal,  vol. 122 (March 2011): 92-124.  2.  Bailey, Sarah, and Paul Harvey. “State of Evidence on  Humanitarian Cash Transfer.” Background note for the High Level  Panel on Humanitarian Cash Transfers, Overseas Development  Institute. March 2015.  3.  Bailey, Sarah, and Sophie Pongracz. “Humanitarian Cash  Transfers: Cost, Value for Money and Economic Impact.”  Background note for the High Level Panel on Humanitarian Cash  Transfers, Overseas Development Institute. March 2015.  4.  Creti, Pantaleo. The Impact of Cash Transfers on Local Markets:  A Case Study of Unstructured Markets in Northern Uganda. Cash  Learning Partnership, April 2010.  5.  Creti, Pantaleo. The Voucher Programme in the Gaza Strip: Mid- Term Review (Final Report). Commissioned by the World Food  Programme and Oxfam, United Kingdom, 2011.  6.  De Sardan, Jean Pierre, Hannatou Adamou, Oumarou Hamani,  Younoussi Issa, Nana Issaley, and Issaka Oumarou. “Cash  Transfers in Niger: The Manna, the Norms and the Suspicions.”  Translation of a paper undertaken by LASDEL and originally  published in French. 2013.  7.  Doocy, Shannon, and Hannah Tappis. Cash-Based Approaches in  Humanitarian Emergencies: A Systematic Review. Funded by the  U.K.’s Department of International Development (DFID). Johns  Hopkins. 2015.  8.  Doocy, Shannon, Emily Lyle, and Hanna Teppis. Emergency  Transfers in Northern Syria. An Economic Evaluation of GOAL  Food Assistance Programs in Idleb Governorate. Johns Hopkins,  Bloomberg School of Public Health, September 2015.  9.  Dunn, Sophia, Mike Brewin, and Aues Scek. Cash and Voucher  Monitoring Group: Final Monitoring Report of the Somalia Cash  and Voucher Transfer Programme. London: Humanitarian Policy  Group, Overseas Development Institute, 2014.  10. Gentilini, Ugo. Our Daily Bread: What is the Evidence on  Comparing Cash versus Food Transfers? Social Protection &  Labor Discussion Paper No. 1420. Washington, D.C.: World Bank  Group, July 2014.  11. Gentilini, Ugo. “Revisiting the ‘Cash vs. Food’ Debate: New  Evidence for an Old Puzzle.” World Bank Research Observer, vol.  31 (2016): 135-167.  12. Gentilini, Ugo. The Other Side of the Coin: The Comparative  Evidence of Cash and In-Kind Transfers in Humanitarian  Situations. Washington, D.C.: World Bank Group, July 2016.  13. Gilligan, Daniel, Melissa Hidrobo, John Hoddinott, Shalini Roy,  and Benjamin Schwab. “Much Ado about Modalities: Multicountry  Experiments on the Effects of Cash and Food Transfers on  Consumption Patterns.” International Food Policy and Research  Institute paper prepared for the Agriculture & Applied Economics  Association Annual Meeting, Minneapolis, July 2014.  14. Hedlund, Kerren, Ben Allen, Maria Bernandez, Muriel Cala, Saul  Guerrero, Chloe Milloz Baudy, Juilien Morel, Panos Navrozidis,  Silkie Pietzsch, and Michael Yemene. Meta-Evaluation of ACF  Fresh Food Voucher Programmes. ACF International, with funding  from the Cash Learning Partnership and the European  Commission on Humanitarian Aid and Civil Protection. January  2012.  15. Hoddinott, John, Daniel Gilligan, Melissa Hidrobo, Amy Margolies,  Shalini Roy, Susanna Sandström, Benjamin Schwab, and Joanna  Upton. Enhancing WFP’s Capacity and Experience to Design,  Implement, Monitor and Evaluate Vouchers and Cash Transfer  Programmes. Study Summary. International Food Policy  Research Institute, 2013.  16. Husain, Arif, Jean-Martin Bauer, and Susanna Sandstrӧm.

Economic Impact Study: Direct and Indirect Impact of the WFP  Food Voucher Programme in Jordan. World Food Programme,  2014.  17. Kardan, Andrew, Ian MacAuslan, and Ngoni Marimo. Evaluation of  Zimbabwe’s Emergency Cash Transfer (ZECT) Program—Final  Report. Oxford Policy Management, supported by WFP and  Concern Worldwide, 2010.  18. Majewski, Brian, Lois Austin, Katherine George, Carol Ward, and  Kurt Wilson. WFP’s 2008 Cash and Voucher Policy (2008-2014):  A Policy Evaluation. Evaluation Report—Volume 1. Konterra  Group paper for World Food Programme. 2014.  19. Margolies, Amy, and John Hoddinott, “Costing Alternative Transfer  Modalities,” IFPRI Discussion Paper 01375. International Food  Policy Research Institute, September 2014.  20. Maunder, Nick, Victoria De Bauw, Neil Dillon, Gabrielle Smith, and  Sharon Truelove. “Evaluation of the Use of Different Transfer  Modalities in ECHO Humanitarian Aid Actions, 2011-2014.”  Analysis for Economic Decisions. Evaluation commissioned by the  European Commission. 2016.  21. Michelson, Hope, Christopher Barrett, Laura Cramer, Eric Lentz,  Megan McGlinchy, Mitchell Morey, and Richard Mulwa. “Cash,  Food, or Vouchers? An Application of the Market Information and  Food Security Response Analysis Framework in Urban and Rural  Kenya.” Food Security, vol. 4 (2012): 455-469.  22. Mountfield, Ben. Unconditional Cash Transfers in Gaza: An  External Review. Commissioned by Oxfam, Great Britain. 2012.  23. Poulsen, Lene, Sophia Dunn, Sado Hashi, Mohamed Adnan  Ismail, Colleen McMillon, Caroline Tanner and Njoroge Thuo.

Somalia Protracted Relief and Recovery Operation: Strengthening  Food and Nutrition Security and Enhancing Resilience, June  2012–December 2015. Mid-Term Evaluation Report. World Food  Programme, 2015.

Appendix IV: Comments from the U.S. Agency for International Development

Appendix V: GAO Contact and Staff Acknowledgments

In addition to the individual named above, Joy Labez (Assistant Director),  Sushmita Srikanth (Analyst-in-Charge), David Blanding, Carol Bray, Ming  Chen, Martin De Alteriis, Neil Doherty, Mark Dowling, Reid Lowe, Julia  Ann Roberts, and Shannon Roe made key contributions to this report.

International Food Assistance: Cargo Preference Increases Food Aid  Shipping Costs, and Benefits Are Unclear. GAO-15-666. Washington,  D.C.: September 25, 2015.

International Food Assistance: USAID Should Systematically Assess the  Effectiveness of Key Conditional Food Aid Activities. GAO-15-732.  Washington, D.C.: September 10, 2015.

International Cash-Based Food Assistance: USAID Has Processes for  Initial Project Approval but Needs to Strengthen Award Modification and  Financial Oversight. GAO-15-760T. Washington, D.C.: March 26, 2015.

USAID Farmer-to-Farmer Program: Volunteers Provide Technical  Assistance, but Actions Needed to Improve Screening and Monitoring.  GAO-15-478. Washington, D.C.: April 30, 2015.

International Cash-Based Food Assistance: USAID Has Developed  Processes for Initial Project Approval but Should Strengthen Financial  Oversight. GAO-15-328. Washington, D.C.: March 26, 2015.

International Food Aid: Better Agency Collaboration Needed to Assess  and Improve Emergency Food Aid Procurement System. GAO-14-22.  Washington, D.C.: March 26, 2014.

International Food Aid: Prepositioning Speeds Delivery of Emergency Aid,  but Additional Monitoring of Time Frames and Costs Is Needed.  GAO-14-277. Washington, D.C.: March 5, 2014.

Global Food Security: USAID Is Improving Coordination but Needs to  Require Systematic Assessments of Country-Level Risks. GAO-13-809.  Washington, D.C.: September 17, 2013. E-supplement GAO-13-815SP.

International Food Assistance: Improved Targeting Would Help Enable  USAID to Reach Vulnerable Group. GAO-12-862. Washington, D.C.:  September 24, 2012.

World Food Program: Stronger Controls Needed in High-Risk Areas.  GAO-12-790. Washington, D.C.: September 13, 2012.

International Food Assistance: Funding Development Projects through  the Purchase, Shipment, and Sale of U.S. Commodities Is Inefficient and  Can Cause Adverse Market Impacts. GAO-11-636. Washington, D.C.:  June 23, 2011.

International School Feeding: USDA’s Oversight of the McGovern-Dole  Food for Education Program Needs Improvement. GAO-11-544.  Washington, D.C.: May 19, 2011.

International Food Assistance: Better Nutrition and Quality Control Can  Further Improve U.S. Food Aid. GAO-11-491. Washington, D.C.: May 12,  2011.

International Food Assistance: A U.S. Governmentwide Strategy Could  Accelerate Progress toward Global Food Security. GAO-10-212T.  Washington, D.C.: October 29, 2009.

International Food Assistance: USAID Is Taking Actions to Improve  Monitoring and Evaluation of Nonemergency Food Aid, but Weaknesses  in Planning Could Impede Efforts. GAO-09-980. Washington, D.C.:  September 28, 2009.

International Food Assistance: Local and Regional Procurement Can  Enhance the Efficiency of U.S. Food Aid, but Challenges May Constrain  Its Implementation. GAO-09-570. Washington, D.C.: May 29, 2009.

Sending cash and vouchers instead of food

Historically, the United States shipped most of its emergency food aid overseas. In recent years, the U.S. Agency for International Development has also provided cash or vouchers that recipients—such as Syrian refugees—can use to buy food. USAID funding for this assistance grew from $76 million to $432 million in fiscal years 2010-2015 and could continue to rise.

USAID has processes to monitor cash and voucher projects. Yet it hasn't collected all the data it needs to evaluate their effectiveness, and some monitoring measures are flawed. We made recommendations to improve USAID's data and measures.

Question: What has USAID and its partners established?

Answer: The United States Agency for International Development (USAID) and its implementing partners have established processes to monitor cash transfer and food voucher projects.

Question: How do they monitor these projects’ implementation?

Answer: To monitor the implementation of these projects, USAID has assigned monitoring roles and responsibilities to staff, is developing country monitoring plans and monitoring tools, and is working to verify information that partners have provided through actions such as conducting site visits, and speaking with beneficiaries.

Question: How are interviews and surveys part of this monitoring process?

Answer: To ensure that assistance is delivered according to their procedures and to the targeted beneficiaries, implementing partners monitor distributions, and interview beneficiaries regarding the distribution of the assistance. In addition, implementing partners conduct postdistribution surveys to gather information about the relevance, efficiency, and effectiveness of the assistance (see figure).

USAID and Implementing Partners Have Established Processes for Monitoring Cash and Voucher Projects

USAID Has Established a Process to Monitor Implementation of Cash and Voucher Projects

Implementing Partners Have Established Processes to Monitor Cash and Voucher Projects during and after Distribution of Assistance

Question: What happened as a result of the USAID’s incomplete reporting?

Answer: Incomplete reporting and weaknesses in certain performance indicators limit USAID's ability to use monitoring data to evaluate cash transfer and food voucher projects' performance. As a result, USAID has limited ability to assess the overall performance of these projects. As a result, USAID lacks information that would be useful for evaluating the projects' effectiveness relative to that of in-kind food aid.

Question: What did GOA find in the reports for USAID's cash and voucher projects?

Answer: GAO's review of 14 final reports, which USAID requires for each project, found that a majority of the reports lacked required data elements, such as prices for key staple foods.

Question: How did the reports differ in the amount of data they included?

Answer: Only 1 report included all 12 required data elements, and the other reports were missing up to 8 elements.

Question: What other weaknesses did GAO find with USAID’s current projects?

Answer: As a result, USAID has limited ability to assess the overall performance of these projects. Further, GAO found weaknesses in USAID's indicators for measuring cash and voucher projects' timeliness, cost-effectiveness, and appropriateness. USAID's indicator for timeliness does not track delays in implementation. In addition, the indicator for cost-effectiveness does not include a standardized unit for measuring project costs. Further, the indicator for project appropriateness does not have associated benchmarks for measuring cash transfer and food voucher projects' impact on local markets.

Question: What is quality information important to help do?

Answer: According to standards for internal control in the federal government, management should use quality information, including relevant data from reliable sources, to achieve an agency's objectives.

Data Gaps and Indicator Weaknesses Limit USAID’s Ability to Evaluate the Performance of Cash and Voucher Projects

Most Final Reports We Reviewed Lacked Required Data about Project Performance

USAID’s Current Indicators for Assessing Cash and Voucher Projects’ Timeliness, Cost- Effectiveness, and Appropriateness Have Weaknesses

USAID’s Indicator for Timeliness Does Not Track Implementation Delays

USAID’s Indicator for Cash and Voucher Projects’ Appropriateness Lacks Associated Benchmarks for Market Impact

Question: How has the United States provided support to food-insecure countries?

Answer: For more than 60 years, the United States provided assistance to food-insecure countries primarily in the form of food commodities procured in the United States and transported overseas.

Question: How has this support to food-insecure countries changed in recent years?

Answer: In recent years, the U.S. government has increasingly provided food assistance in the form of cash transfers or food vouchers. In fiscal years 2010 through 2015, USAID funding for Emergency Food Security Program (EFSP) for cash transfers and food voucher projects grew from about $76 million to nearly $432 million.

Background
USAID EFSP Funding since 2010

Question: What was GAO asked to do regarding USAID’s monitoring of cash-based food assistance?

Answer: GAO was asked to review USAID's monitoring and evaluation of cash-based food assistance.

Question: What does GAO’s review examine?

Answer: This report examines, among other things, (1) USAID's and implementing partners' processes for monitoring cash transfer and food voucher projects and (2) the extent to which monitoring data reported to USAID can be used to evaluate the performance of such projects.

Question: What research and analysis did GAO do in the making of this report?

Answer: GAO analyzed program data, interviewed relevant officials; and conducted fieldwork in Kenya and Liberia, selected on the basis of criteria such as funding and types of projects. GAO also reviewed the final reports for a nonprobability sample of closed cash transfer and food voucher projects.

Appendix I: Objectives, Scope and Methodology