View
213
Download
0
Category
Tags:
Preview:
Citation preview
Dana BurdeVisiting Assistant Professor
Department of Humanities and Social Sciences
New York Universitydana.burde@nyu.edu
Panacea or Straitjacket? A Qualitative Researcher Reflects
on a Randomized Trial in Afghanistan
Randomized Trial Assessing Community-Based Schools in Afghanistan
Co-Investigator (for the large-scale randomized trial): Leigh Linden, Assistant Professor, Economics Department and School of International and Public Affairs, Columbia University
Agency studied: Catholic Relief ServicesResearch assistants: Matt Hoover, Dr. Saeed Mahmoodi, Nathan Falkner, Nafi Olomi, Nicole Rigg, Amy Kapit.
Funders (from the inception of the study): Columbia University Institute for Social and Economic Research & Policy ($15,000); Weikart Family Foundation ($75,000); United States Institute of Peace ($40,000); Spencer Foundation ($40,000—pilot; $483,000—large scale study); National Science Foundation ($100,000).
Outline of today’s talk
1. Brief overview of arguments: Heated pro/con debate between qualitative researchers and quantitative researchers regarding the uses and effectiveness of randomized trials
2. Study design: Description of the randomized trial of community-based schools in Afghanistan and the ways in which qualitative study components complemented the quantitative design
3. Discussion: Design, execution, ethics, and research culture—how these issues affected our randomized trial
The Pro/Con Debate
Those in favor of randomized trials say they provide: Evidence-based findings that show more robust
results (than other kinds of studies) Answers to key questions in education research
This position is epitomized by this quotation:“…[randomized controlled] experiments provide a
better warrant [for causal conclusions]…than any other method. So if experiments can be conducted in schools they should be. Not to use them requires a very strong justification” (Cook, 2003, p. 117).
The Pro/Con Debate
Those who oppose randomized trials accuse them of: Weakness in the research design and execution of the
study; Ethical shortcomings; and Promoting a culture of privileging quantitative methods
over qualitative.
This position is epitomized by this quotation:“…there are formidable limits to the randomized control approach—limits
which the proponents of this methodology have not sufficiently recognized. To achieve ‘rigor’ in education research, we need to adopt a more realistic grasp of the problems that plague attempts to implement research results as well as a greater openness to other approaches that offer crucial insights into social processes in organizations. For these purposes, qualitative methods, including participant observation and in-depth interviews, are likely to be the most promising” (Lareau, forthcoming, p. 146).
Argument
Based on our study in Afghanistan, I argue that: Randomized trials are excellent tools for assessing program
impact and are underused to study education programs conducted by international NGOs
Strong design and execution of place-based randomized trials can overcome some weaknesses identified by critics
Questions raised regarding ethics can be addressed and should not preclude using randomized trials
In agreement with the critics, I argue randomized trials should not eclipse other forms of research; they are not appropriate for every impact study; qualitative methods can address their weaknesses and complement randomized trials to great effect
Overview of StudyThe study: investigates whether community-based education programs
improve child welfare by comparing villages that receive schools to villages that have not yet received them and by comparing children in these villages.
examines attendance/enrollment, academic achievement, child labor, and social benefits (protection, social networks).
Phases: Qualitative pilot study: Tested field procedures 2005-2006 Randomized trial 2007-2008: Worked with a large international
US-based NGO, Catholic Relief Services (CRS), to randomly assign their community-based schools to eligible villages (i.e., villages that had none previously)
Complementary qualitative case study of program
Program: Partnership for Advancing Community Education in Afghanistan (PACE-A)
PACE-A is a five-year, $24 million USAID-funded program: meant to “expand quality learning and life opportunities for
marginalized communities and their children in Afghanistan” (PACE Summary, 2006, p. 1)
Consortium of 4 NGOs tasked with providing hundreds of community-based schools in 20 provinces across Afghanistan
The CRS portion of the program was intended to reach approximately 100 villages by the end of 2008 Schools were to be phased in over several years
CRS Community-Based Schools
CRS selects districts and communities according to: Security; Availability of teachers; Level of community
interest; Community willingness to mobilize resources (teachers’ salaries); Support from the Ministry of Education (agreement); Community willingness to provide space for the school (room in a home or mosque).
CRS provides training to teachers, materials for the classroom (government curriculum), and regular monitoring to track progress over time.
Study Hypotheses Hypothesis 1 Attendance/Enrollment: Children living in a
village that receives a CRS school will be more likely to enroll in school and will attend more regularly than children in other villages.
Hypothesis 2 Learning: Children who live in villages that receive a CRS school will perform better on academic achievement tests on math and language skills than children in the villages that do not receive a CRS school.
Hypothesis 3 Indirect Social Outcomes: Children who live in villages that receive a CRS school will report higher levels of social integration, higher levels of security, and higher levels of safety than children in other villages.
Original Study Design Yes - CBS in 2007 No - CBS in 2007
(but yes for 2008)
Group 1:
Normal CRS Class,
10 Villages, 500 HHs Group 5
No CRS Schools,
40 Villages,
2,000 HHs
Group 2:
Community education,
10 Villages, 500 HHs
Group 3:
Disciplinary Instruction,
10 Villages, 500 HHs
Group 4:
Community Education and Disciplinary Instruction,
10 Villages, 500 HHs
Actual Study Design
CRS community-based schools
N = 31 villages; N = 1,253 households;
N = 2,036 children aged 6-11
Yes in 2007 No in 2007
Group 1: Treatment
CRS Schools,
13 Villages,
637 HHs
Group 2: Control
No CRS Schools,
18 Villages,
616 HHs
Qualitative component
Case study of the CRS program to complement the RT and learn more about:
program implementation; place, i.e., country, province, districts, and villages in
which we are collecting data; institutions that are part of our study or influence it: i.e.,
households, families, schools (government and CBS), mosques, SMCs, village shuras, ministry of education.
individuals that are part of our study or influence it, i.e., children, parents/heads of households, teachers, arbobs, CRS education staff members, CRS managers, PACE-A directors
Discussion: Design Criticism: Addressing only narrow questions misses significant
variables in education
Our study: “Schools” constitute a significant variable
Criticism: Extent to which variables can be manipulated accurately, e.g., networks
Our study: Intent was not to manipulate networks, but rather to measure the impact of schools on networks, and other secondary effects
Limitation: We can say that the program did not have an impact on friendship networks, but need qualitative methods to say why
Discussion: Execution Criticism: Adherence to treatment; “contamination” of
control by treatment
Difficulties our study faced in execution:• Persuading CRS to carry out a randomized trial
• Avoiding contamination • Preserving the integrity of the randomization:
• Communication breakdowns (sample size decreased)• Inclement weather (delayed start to survey)• Security (attacks on staff, tribal warfare, etc. sample size
decreased to 31 villages)• Hiring and training survey staff
Discussion: Ethics
Criticism: Unethical to deny treatment
Ethical issues our study faced: Sample selection: “Needs assessment” versus
randomized selection Temporary denial of treatment: Program was
intended to be phased in over time, regardless Program benefits not clear given the basic program
intervention
Discussion: Ethics
Additional ethical issues our study faced: Protection for the researchers was a far greater
problem than the protection of subjects IRB was silent on this point In fact, IRB protections for subjects put researchers
at risk: Confidentiality Signed documents
Discussion: Research CultureA culture of preference for randomized trials pervades among some
research circles.
Qualitative methods are invaluable in addressing weaknesses of randomized trials in several ways:
A strong qualitative pilot study enhances the randomized study design and increases the probability of successful program intervention as well as successful execution of the study.
Qualitative methods used in conjunction with a randomized trial counteract the tendency of randomized trials to focus on narrow cause and effect relationships.
Complementary qualitative case study and ethnographic methods provide more confidence in quantitative data analysis, interpretation, and provide additional detail to explain findings that randomized trials cannot.
Conclusion
Randomized trials -- crucial method to test multiple empirical questions across the social sciences.
Critics have valid concerns -- randomized trials should not be allowed to eclipse qualitative methods.
As in all research, ethical questions raised by randomized trials must be taken seriously.
Properly designed and executed, randomized trials can produce robust and significant data even in the most difficult circumstances.
Recommended