Field Experiments (Econ 2041/HBS 4430)

Course Description:
This course is for doctoral students who want to learn how to design and run field experiments as a research methodology. The objective is for students to refine their own experimental designs and be able to run them by the end of the course, leading to an academic paper.

The course will be hands-on and oriented towards providing technical skills for the design and implementation of field experiments, including overcoming the many possible associated pitfalls. We will examine in-depth examples of how field experiments are designed, implemented and analyzed, including the “back story” of several published field experiments.

We will also discuss at length throughout the course how to use field experiments to test academic theory as opposed to only for policy/impact evaluation. The last third of the course is dedicated to introducing and studying particularly fruitful areas for research using field experiments and to students’ presentations of their own research ideas.

Advanced MBAs, MPPs, and MPA-IDS, who want to learn the technical skills of running and managing a field experiment, for the purpose of conducting randomized impact evaluations of innovative programs in the companies and NGOs that they will be part of and/or advising, will be allowed to take the course upon permission of the instructor.

All students should be intending to run a field experiment in the near future, and ideally in the summer of 2009.  The course assignment will be a completed proposal (of approximately 15 pages) outlining the theory and design for a field experiment, and a completed IRB application for human subjects approval.
 

Course Outline:

January 29, 2009

Session 1: Why do field experiments?

Causal Inference:
LaLonde, Robert J. 1986. “Evaluating the Econometric Evaluations of Training Programs with Experimental Data.”American Economic Review, 76: 604-20.

Angrist, Josha D.  & Alan B. Krueger. 2001. “Instrumental Variables and Search for Identification: From Supply and Demand to Natural Experiments." Journal of Economic Perspectives, 15 (4):  69-85.

Arceneaux, Kevin, Ian Gerber, and Donald Green. 2006. “Comparing Experimental and Matching Methods using a Large-Scale Field Experiment on Voter Mobilization.” Political Analysis, 14 (1): 37-62.

Lab experiments vs field experiments:
List, John A., and Steven Levitt. 2006. "What Do Laboratory Experiments Tell Us About the Real World?" University of Chicago and NBER.

Roth, Al. 1988. “Laboratory Experimentation in Economics: A Methodological Overview.” The Economic Journal, 98(393): 974-1031.

Harrison, Glenn and John A. List. 2004. "Field Experiments." Journal of Economic Literature, XLII: 1013-1059.

In this class, we will discuss the two main reasons for field experiments: to improve causal inference, and to provide more natural settings and external validity for laboratory experiments.  We will discuss the evolution of field experiments as policy evaluation tools and as a methodology for testing economic theory.

Helpful technical guide going  forward:
Duflo, Esther,  Abhijit Banerjee, Rachel Glennerster, and Michael Kremer. 2006. “Using Randomization in Development Economics: A Toolkit.” Forthcoming in Handbook of Development Economics.

February 5, 2009

Session 2: Timeline of a Field Experiment, including Finding Field Partners and Convincing Practitioners

Ashraf, Nava, Dean Karlan, and Wesley Yin. 2006. "Tying Odysseus to the Mast: Evidence from a Commitment Savings Product in the Philippines." Quarterly Journal of Economics, 121, 2 : 635-672.

Ashraf, Nava, Dean Karlan, and Wesley Yin. 2003. "Testing Savings Product Innovations Using an Experimental Methodology." Asian Development Bank Technical Note Series Series, No. 8.

Ashraf, Nava and Laura Winig, HBS Case Study N9-907-040 : “Evaluating Micro-savings Programs: Green Bank in the Philippines (A)”

Ashraf, Nava and Laura Winig, HBS Case Study B-case “Implementing Evaluation: Green Bank’s Decision”
 

This cluster of readings illustrates one field experiment, on a commitment savings product in the Philippines, from beginning to end: the QJE article is the final academic published result; the ADB article uses the field experiment and its timeline (including sample surveys) to illustrate a more general guide for practitioners on using randomization to test savings innovations; and the HBS Cases are about a particular moment in time when the field partner had to be convinced to do individual-level randomization and the concerns they had.

In this class, we will discuss the time horizon of a field experiment, from conception to implementation to results (not including publication!), and will include:
-Using qualitative work to find and develop testable hypotheses
-Finding field partners
-Convincing and training practitioners within the partner institution
-Keeping the field partner from bailing/not following the randomization and ruining 3 years of your graduate career
-Pilot testing
-what happens if your pilot fails? When should you cut your losses?

Midterm Assignment: Write a brief for an organization about why run field experiments and how to run them (this can be as simple as the correspondence you have with the field partner with which you will be running your field experiment)

February 12, 2009

Session 3: Measurement of Outcomes, and Power and Sample Size

Designing Behavioral Indicators and Survey Questions
Olken, Ben. 2007. “Monitoring Corruption: Evidence from a Field Experiment in Indonesia.” Journal of Political Economy, 115 (2): 200-249.

Deaton, Angus. 1997. The Analysis of Household Surveys. Baltimore: John Hopkins University Press for the World Bank.

Banerjee, Abhijit, Shawn Cole, Esther Duflo, and Leigh Linden. 2003. “Remedying Education: Evidence from Two Randomized Experiments in India.” NBER Working Paper 11904.

Chattopadhyay, Raghabendra and Esther Duflo. 2004. “Women as Policy Makers: Evidence from a Randomized Policy Experiment in India.”Econometrica, 72(5): 1409-1443. (also see NBER Working Paper No. 8615; BREAD Working Paper No. 001, 2001).

In this class, we will discuss how to design appropriate and  innovative behavioral indicators to measure the outcomes of field experiments, and how to design household surveys, when they are necessary.We will also be discussing how to do power calculations, in order to determine the sample size you will need for the results you expect to have statistical significance, so that you don’t end up spending a large amount of money and time only to discover you didn’t have a large enough sample size to see an effect. We will be using the software Optimal Design.

February 19, 2009

Session 4: Designing Field Experiments to test theory and have external validity (aka, what can field experiments tell us about the real world?)

Rodrik, Dani. 2008. “The New Development Economics: We shall experiment, but how shall we learn?”Working Paper.

Ashraf, Nava, James Berry, and Jesse M. Shapiro. "Can Higher Prices Stimulate Product Use? Evidence from a Field Experiment in Zambia." Harvard Business School Working Paper, No. 07-034, December 2006.

Cohen, Jessica and Pascaline Dupas, “Free Distribution or Cost-Sharing? Evidence from a Randomized Malaria Prevention Experiment."

(optional: Levitt, Steve and John List. 2008. “Field Experiments in Economics: the Past, the Present and the Future”, NBER Working Paper No. W14356)
 

**February 27, 2009

Session 5: Ethics of Randomization and Human Subjects Review

Beecher, Henry K. "Ethics and Clinical Research." New England Journal of Medicine (1966).

*Taking the Harvard Ethics Training will be very helpful, both for what we will talk about in this class session and also for your future work:
https://hethr.harvard.edu/HethrLogin.jsp

Interim Assignment: IRB Application

March 5, 2009

Session 6: How to Actually Randomize

Bruhn, Miriam and David McKenzie. 2008. "In Pursuit of Balance." The World Bank Policy Research Working Paper 4752.
 

Mobius, Marcus, Paul Niehas, and Tanya Rosenblat. 2005. “Social Learning and Consumer Demand.”

[Guest speakers describing challenges in randomization]
 

March 12, 2009

Session 7: Intent-To-Treat (ITT) and Treatment-On-theTreated (TOT): Evaluating Results; More Statistical Inference (Spillovers and Attrition)

Kling, Jeffrey R., Jens Ludwig, and Lawrence F. Katz. 2005. “Neighborhood Effects on Crime for Female and Male Youth: Evidence from a Randomized Housing Mobility Experiment.” Quarterly Journal of Economics, 120: 87-130.

Miguel, Edward and Michael Kremer. 2004. “Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities.” Econometrica, 72 (1): 159-217. (NBER Working Paper #8481, August 2001).

**March 19: no class, individual meetings

March 26: Spring Break

April 2, 2009

Session 8: Ripe Areas for research with Field Experiments (I): Incentives and Company Experiments

Fehr, Ernst and Lorenz Goette. 2008. "Do Workers Work More if Wages Are High? Evidence from a Randomized Field Experiment." American Economic Review, 97(1): 298-317.

Bandiera, Oriana, Iwan Barankay and Imran Rasul. 2007. “Incentives for Managers and Inequality Among Workers: Evidence From a Firm Level Experiment.”Quarterly Journal of Economics, 122: 729-75.

(Shearer, Bruce. 2004. "Piece Rates, Fixed Wages and Incentives: Evidence from a Field Experiment." Review of Economic Studies, 71 (2): 513-34.)

April 9, 2009

Session 9:
Guest Speaker

April 16, 2009

Session 10: Ripe Areas for research with Field Experiments II:
Health and Persuasion; Finance; Intra-Household Dynamics and Consumer Demand

Ashraf, Nava, Diego Aycinena, Claudia Martinez A. and Dean Yang “Remittances and the Problem of Control: A Field Experiment Among Migrants from El Salvador”, HBS Working Paper

Ashraf, Nava “Spousal Control and Intra-Household Decision Making: An Experimental Study in the Philippines”, American Economic Review (forthcoming)

 [Guest speakers on Small Decisions, Health Behavior Change, and Mental Accounting in Finance]
 

April 23, 30

Sessions 12-15: Presentation by students of design proposals
 

** date change