The first two articles in this series (here and here) have covered the Student Movement for Real Change (SMRC) and its keystone project, the Kenya Water Project. In a nutshell, the Water Project, which is entirely designed and implemented by young people, will end with the construction of a 28 km ( 17.4 mi) pipeline that will bring clean water to over 35,000 residents of Kayafungo, Kenya.
As part of the research for this series, I interviewed Ryan Knight, a leading member of the Kenya Water Project. Ryan, a former economics major currently working at NERA Economic Consulting, is leading an analysis of the effects of the project that will extend far beyond Kayafungo. He is receiving assistance and advice from a team of Harvard graduate student researchers and his former Pomona College professors.
Ryan Knight: Our goal with the evaluation is to identify the causal outcomes of the project. This goes beyond a typical "case study" or even a standard quantitative evaluation. It is easy to show that there is a change in outcomes post-intervention. It is much harder to show that the intervention caused the change. There are any number of reasons why the outcome could have changed that are unrelated to the intervention. We're taking advantage of the fact that the pipeline will be constructed in phases to address the impact of water provision.
The reasoning is as follows: If we compare our indicators [measurements of changes in sanitation, hygiene, health, and other social factors] to the baseline data, we can't really say that it's the pipeline that's driving the changes because we don't know how the indicators would have changed had we not constructed the pipeline. It may be that it was something other than providing water that was driving the changes, such as a seasonal variation. If we compare the changes in our community to the changes in a control community, it may be something that is different [between] the communities that is driving the change. For example, it could be that our community is so excited about the intervention, their school attendance and sanitation are going to improve next year whether or not we construct the pipeline. This may not sound like a very big deal, but it has proven to be huge in empirical studies.
Alternatively, if we compare the changes in our indicators to different groups within Kayafungo, we can compare those groups that received access to the pipeline -- for no reason other than being lucky enough to live closer to the start of the pipeline--to those who have not yet gotten access to the pipeline but will as soon as it is financially and technically feasible. This allows us to avoid sample bias and get results that would translate to other communities, without having to withhold treatment from anyone or otherwise impede the intervention.
SK: So what is the importance of knowing the causal effects of building a water pipeline?
RK: Given the vast amount of resources currently being thrown at water projects, identifying causality is extremely important. If we can show in a convincing manner that pipelines cause significant improvements in outcomes, not just in our community but in any East African community that self-identifies as lacking access to water, it would suggest that this aid should be directed to pipeline projects in such communities. On the other hand, if our evidence suggests that pipelines might not be the best way to provide water, it would warrant a re-examination of policies that promote interventions similar to ours. Only by identifying and promoting those interventions that work will we be able to achieve the lofty goals of governments and donors.
SK: How did this all come about? It doesn't seem like a clear connection between "water pipeline" and "academic analysis."
RK: When Lily [Muldoon, Project Director] first came to me with the idea to build a water pipeline in Kayafungo, Kenya, we looked for a theoretical justification and empirical support. We incorporated what we found in academic literature—primarily, the importance of sanitation—but we also identified several areas where the literature doesn't provide clear guidance, such as effective schemes for maintaining water infrastructure. Recognizing that our project provides an excellent opportunity to address some of these issues, we brought on a team of graduate students to help us with a causal evaluation.
SK: Ultimately, how can the KWP be used as an example for other projects, be they water-based or not?
RK: Our goal is for the KWP to be an example of the capabilities of young people. As young professionals administering our first project, we face a lot of disadvantages. We don't have any practical experience, which can be a real setback at times (for example, one of our engineers from Engineers Without Borders at Washington State University forgot to bring closed-toed shoes to the initial site assessment and couldn't participate in the surveying). On the other hand, our inexperience also means that we don't have any bad habits. Anytime we don't know what to do about something, we don't say, "well, how did we do it last time?" we say, "what's the best way to do this?"
In most fields, inexperience would be more of a disadvantage. But, the simple fact is that so many development projects are not successful. For most of the problems that we're looking at, there are no "best practices" because no one has ever done it right. We want to show everyone that young people can not only get an intervention done, but that we can do the best interventions.
SK: How does one go about finding "the best practices" for a project like the Kenya Water Project?
RK: The unfortunate reality is that best practices haven't been established for the vast majority of development interventions. Lots of people think they have the answer—be it microcredit, women's involvement, community involvement, sanitation, or whatever the current fad may be—but most don't have any evidence that what they believe in works. In a world of limited resources, rigorous evaluations often don't make the cut. The result is that, while many NGOs may be doing great work, there's no way to separate the good work from the bad.
The good news is that there are people working hard to identify what works in development policy, like Michael Kremer and the other researchers at MIT's Poverty Action Lab, who are identifying effective interventions by [using randomized groups in their studies]. However, until meticulous evaluations become a routine part of development interventions, there will be a gap between what we know and what we need to know.
That said, there are some great resources out there for development practitioners. Refereed journals are a great place to start. Generally, a refereed journal will only publish the high quality research. After reviewing the academic research, there is the plethora of "case studies" that NGOs publish about their work. These are generally of lower quality, and should not be relied upon to formulate an intervention. However, reading case studies can be a great way to identify practices that should be avoided. Generally, it's much harder to publish a study about something that doesn't work than a study about something that does. We've learned a lot about what not to do by reading case studies. That said, case studies are also a good way to get ideas or to choose interesting practices that you may want to test rigorously. Finally, our project would not have been possible without the personal advice we've received from people who have spent their lives working in development.
SK: What is the potential practical value of the analysis you will undertake?
RK: The questions that we're addressing are some of the most relevant in development policymaking today. One of the United Nations Millennium Development Goals is to cut in half "by 2015, the proportion of people without sustainable access to safe drinking water and basic sanitation." Massive amounts of resources are being thrown at this problem, without much guidance in terms of effective ways to address water and sanitation. For example, there has been no comparison between the relative benefits of piped water provision versus spring source cleaning versus water delivery. With all of these forms of water provision, there are problems with the maintenance of the infrastructure, which is why we're taking an innovative approach in maintenance that includes privatizing the water rights. With respect to sanitation, we know that good hygiene practices are associated with better health outcomes, but we don't know practical ways to encourage the adoption of better hygiene practices. These are some of the issues we're addressing with our research that will have practical implications for policymaking.
One of the things that I'm committed to is improving the exchange of knowledge between researchers and the rest of the world. After our research is completed, we're going to work hard to make it widely available so that smaller NGOs like us can learn about what we did, and can conduct their own evaluations to build on ours.
To learn more about, join, or contribute to the Kenya Water Project, please visit http://www.studentmovementusa.org/
Further Reading:
