Evidence Map compass image

About the map's creation

Kid washing his hands

This Map reflects the broad range of evidence identified as part of a systematic literature review. It was created to:

Organize 500+ impact evaluations on integrated development programs in ways that succinctly depict where the most evidence lies (and where it is thin); and

Allow for easy searching and identification of evidence relevant to various areas of specific interest (including types of interventions, sector-based outcomes, study design, or geography)

In 2015, FHI 360 conducted a systematic review of the evidence on global development programs that were defined as ‘integrated development’: Integrated development approaches intentionally link the design and delivery of programs across more than one core sector.  Given the extreme heterogeneity of the types of programs that can be characterized as integrated in nature, the review did not seek to collectively synthesize their substantive findings with regard to effectiveness or impact (like comparing apples to oranges!). Rather, the primary objective was to assess whether the assumed synergistic, amplified impacts are being routinely and adequately measured for these types of multi-sector programs. A publication reporting the results of the review is forthcoming in 2016. This work was completed with support from the Bill & Melinda Gates Foundation and the FHI Foundation.

Above and beyond the conclusions drawn about the research methodologies, however, we feel strongly that the high volume of impact evaluations collected through the systematic review exercise are directly applicable for people currently making decisions about funding, advocacy, policy, program design, and evaluation of integrated development approaches. Typically, this type of information is only available as an alphabetical list of citations from a review manuscript, or in a table or database used as supplementary material. Given the unusually high number and extreme diversity of articles included in our review, these formats are insufficient for promoting practical use and uptake of the evidence available.

Therefore this Map is a secondary product of the systematic evidence review. In their simplest sense, ‘evidence maps’ are graphical displays of the evidence base available on a particular issue. We owe a double debt of gratitude to the International Initiative for Impact Evaluation (3ie), as a) we utilized their Impact Evaluation Repository to conduct our review (see more information below); and b) the 3ie Evidence Gap Maps helped to inspire the idea of how to translate this type of knowledge in a user-friendly way.

Methodology for the systematic literature review

We utilized the 3ie Impact Evaluation Repository as the sampling frame for this review. The Repository was first launched in 2014 and is an index of all published impact evaluations of development interventions. The systematic search reflected all publications up to March 2014, but the 3ie database continues to be updated on a regular basis. To be included in the Repository, an impact evaluation must: be published (as a journal article, book chapter, report, or working paper), take place in a developing country, examine the effectiveness of a specific development intervention, and use a specifically defined experimental or quasi-experimental estimation strategy. The 3ie review process had no restrictions on publication date. This Repository was chosen as the source for this review because it represents a high-quality database that is not solely dedicated to any one particular sector of global development.

We manually screened all 2,648 abstracts which comprised the entire Repository as of August 7, 2015 to identify which impact evaluations were associated with integrated development interventions. Using a two-phased process we ultimately identified 500+ studies that met this criteria. Note that our definition of integrated development encompasses studies that would be classified as “multi-sector” or “multi-disciplinary” by others. More precisely, our definition focuses on the integrated nature of the intervention itself and excludes programs that:

  • Only integrate different subsectors of a core sector (e.g., health programs that link family planning and HIV/AIDS)
  • Measure outcomes across multiple sectors but do not include multi-sector intervention components (e.g., education programs that measure both education and nutrition outcomes but only deliver education services).

For this review we used the following core sector categories and illustrative subsectors. These were used as general guiding parameters rather than exclusionary definitions:

  • Agriculture and food security (e.g., farming, markets, value & food supply chains, famine prevention)
  • Economic development (e.g., economic strengthening, income generation, livelihoods, cash transfers, microfinance)
  • Education (e.g., early education, primary/secondary/tertiary school, adult literacy)
  • Environment (e.g., environmental/land/marine management, conservation, climate change)
  • Governance (e.g., peacebuilding, conflict management, election monitoring, democracy)
  • Health (e.g., HIV, tuberculosis, maternal and child health, sexual and reproductive health, non-communicable disease, malaria, immunization vaccine, mental health, dental health)
  • Humanitarian (e.g., short-term emergency, conflict, and disaster relief)
  • Nutrition (e.g., micronutrients, food fortification, malnutrition, feeding programs, household diet diversification)
  • Water, sanitation, and hygiene (e.g., water quality, management, supply, hygiene practices)

A noted limitation with our overall categorization is that the application of the definition of integrated development was a subjective process, as was assignment of core sectors to study interventions and outcomes. To address this, we utilized two independent coders and employed inter-coder agreement procedures to enhance reliability in our screening process. Regardless, in some cases assigning sectors to an intervention was difficult and would be considered open to reasonable interpretation (e.g., depending on its particular aim, aid to small-scale farmers could conceivably be an economic development/livelihoods, agriculture, or nutrition intervention).

An additional limitation is that the 3ie Impact Evaluation Repository is primarily aimed at research on conventional development programs, and not necessarily on interventions in the humanitarian sector. We included a category for that field because some of the projects included humanitarian objectives, but we fully recognize that the majority of this literature is absent from this database and therefore not adequately represented within the Evidence Map.