In 2014, the Millennium Challenge Corporation invested about US$275 million to upgrade and extend piped water and sewer networks in Zarqa, Jordan, as well as increase the capacity of the country’s largest wastewater treatment plant. The goal was to reduce poverty and stimulate economic growth by improving water supply to urban areas without cutting flows to agricultural users. While these goals are simple and intuitive, complex infrastructure projects can entail a slew of varying short and long-term effects that demand more careful examination. Bottomline: While it is relatively easy to determine how much a project costs, it is not so easy to measure its impact.
Unfortunately, the methods to learn about impacts and mechanisms in the water sector remain imperfect. The fundamental challenge is to isolate the causal effects of infrastructure from the many other contemporaneous changes that affect water supply and sanitation services, water resources systems, and human welfare and well-being. The answer is to use multiple, carefully chosen methods.
Using this type of multi-pronged evaluation strategy, researchers found that the Zarqa project appears net beneficial on balance. On the plus side, it improved service quality, reduced water loss, and increased the number of households connected to the wastewater system while augmenting the supply of treated wastewater for agriculture. However, there was no evidence that the project’s improvements reduced expenses for high-cost non-network water (i.e., money spent on water purchased from vendors such as shops and tankers), which was a key expected channel of benefits to households. There was also no evidence that small and medium enterprises benefited from the investments, likely owing to their continued low rate of connection to the piped water and sewer system.
Methods and Results
Given concerns over the validity and plausibility of previous assessments, the particular challenges of these long-lived projects, and the complex causal chains that determine their impacts, we propose a set of three key criteria—validity, relevance, and practicality—that researchers working in this domain should consider when designing and implementing evaluation research. As there are tradeoffs across these criteria, scholars must thoughtfully weigh the particular pros and cons, and opportunities and vulnerabilities of different methods, to advance knowledge and achieve greater policy impact.
Researchers seeking to carry out policy-relevant evaluations of large infrastructure investments must work harder to engage with project planners to understand these interventions’ complete theories of change and to track the most important set of anticipated impacts. They must also be cognizant of potential spillovers, evaluation contamination risks, statistical power limitations, and overall impacts on welfare. The eventual evaluation design in this case, with data analysis focused on a range of stakeholder groups and both pecuniary and non-pecuniary quality of life outcomes, allowed researchers to make a critical appraisal of planners’ most critical assumptions, rather than focusing on a narrow set of questions. Second, and relatedly, this multi-dimensional approach allowed for nuanced understanding of the distributional effects of the investment. Specific and documented failures, for example, related to water consumers’ lack of confidence in utility water in this case, can inform development of future remedies to address them. Thus, Zarqa policy-makers might consider investing to convince users of the safety of network water, given our results showing that this water may even be safer than more expensive water purchased from vendors.
Marc Jeuland (Duke University), Jennifer Orgill-Meyer (Franklin & Marshall College), Seth Morgan (Duke University), Daniel Hudner (Social Impact), Mateusz Pucilowski (Social Impact), Alan Wyatt (RTI International), Mohammed Shafei (Independent Consultant), James Cajka (RTI International), Jeff Albert (Aquaya Institute)
Millennium Challenge Corporation (MCC)
Climate & Sustainability, Governance, Economic Development, Impact Evaluation, MENA