http://www.wws.princeton.edu/~dkarlan/downloads/impactperils.pdf
Microfinance Impact Assessments: The Perils of Using New Members as a Control Group
Microfinance institutions aim to reduce poverty. Some assess their impact through a cross-sectional impact methodology which compares veteran to new participants, and then calls any difference between these two groups the "impact" of the program. Such studies have risen recently in popularity because they are cheap, easy to implement, and often encouraged by donors. USAID, through its AIMS project, encourages this methodology with its SEEP/AIMS practitioner-oriented tools1. This paper intends to inform practitioners about the perils of using such a strategy, and suggests a couple solutions to some of the larger problems with this approach.
This approach makes many assumptions that are untested, and others that are tested and false. For example, this approach assumes that dropouts have, on average, identical income and consumption levels to those who remain. Furthermore, this approach assumes that dropouts are not made worse off by participating in the program. This approach also assumes that when lending groups form they do not sort themselves by economic background. These assumptions not only are brave theoretically, but are contradicted by existing empirical research. This paper suggests a method to address the attrition biases, and suggests further research be conducted on the other implicit assumptions before expending resources on a plausibly unreliable assessment
methodology.