3 July 2018

Contribution of international development: Randomized controlled trials and outcome mapping

Summary: International development actors have a great interest in knowing the impact of their interventions. The field offers a wide variety of approaches. Some development economists propose to measure the impact of an intervention using experimental methods.

The impact evaluation practices developed in Western countries, and in the United States in particular, notably for large infrastructure projects, quickly spread to the field of international development, a field that was likewise undergoing expansion, namely through the activities of the World Bank. Economic models such as CBA and impact studies such as the social impact assessment (SIA), as formalized by the International Association for Impact Assessment (IAIA) (see next section of the timeline), were thus deployed throughout the world.

The focus progressively switched from merely mitigating potentially negative impacts to also maximizing positive impacts. To determine whether an intervention generates the expected results, some development economists recommend the use of experimental approaches inspired by the health and natural science communities (Banerjee and Duflo, 2009). The notion of impact, much more “quantitative,” was then approached retrospectively with the objective of establishing causal links. A publication of the World Bank defines impact evaluation as follows:

Impact evaluation. An evaluation that makes a causal link between a program or intervention and a set of outcomes. An impact evaluation answers the question: What is the impact (or causal effect) of a program on an outcome of interest. (Gertler et al., 2016, p. 328)

It is probably in the field of international development that the methods in use have come closest to the ideal of a social impact measurement defined as gathering evidence of a causal link between an intervention and an observed outcome. Indeed, large philanthropy-funded interventions (private or public) usually pay much more attention to the use of a convincing counterfactual than smaller initiatives led locally by social economy organizations. They do so by deploying randomized controlled trials (RCT) or other quasi-experimental designs that come close to it.

Questions about the desirability and feasibility of this vision of social impact measurement are regularly raised. In the early 2000s, for example, authors associated with Canada’s International Development Research Centre (IDRC) (Mayne, 2001; Smutylo, 2001) moved away from the approach of establishing evidence of a causal link between an action and an effect, instead advocating that a plausible association be identified to show that “in light of the multiple factors influencing a result, […] the intervention made a noticeable contribution to an observed result” (Mayne, 2012, p. 273). This led to the development of contribution analysis and the outcome mapping methodology, the latter of which is used by a sizeable community of evaluators in developing countries.

Yet, twenty years later, as the founders of the Abdul Latif Jameel Poverty Action Lab (J-PAL) receive the Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel for “their experimental approach to alleviating global poverty” (J-PAL, 2019), it seems the issue is far from being settled. This debate regarding what is the right level of evidence to expect when talking about impact is explored in the section “Proving impact: Causality, attribution and contribution”. Regardless of one’s opinion on the issue, these debates are relevant to the extent that they define how social impact measurement is understood today.