Flipping the Research Coin
Randomized field trials are not the best or only way to address all important research questions, but they are often described as the "gold standard." This month's column explores why researchers and NCLB express enthusiasm for experimental research designs, why schools may hesitate to participate, and possible win-win solutions.
The researcher's view A randomized field trial is an experimental research design used to estimate the effect of an intervention or "treatment" (e.g., a math program) on a particular outcome (e.g., math achievement). Such trials are randomized and controlled.
Random selection refers to the way participants are selected from a defined population (e.g., high-poverty rural schools). Random assignment occurs when investigators use a random process like the toss of a coin to decide which participants are assigned to an "experimental" group (which gets the intervention) and which are assigned to a "control" group (which does not).
An experiment is controlled when investigators regulate all factors and conditions that could possibly influence the outcome or confound the results.
Researchers are especially fond of randomized field trials when evaluating educational programs because they can increase the validity of their findings. For example, if the experimental group improves while the control group does not, the difference isn't so easily attributed to factors such as prior achievement or classroom conditions.
The educator's view School and district leaders want rigorous research information to help them select programs. But here's the rub: randomized field trials can't be done without the participation of schools and teachers--and testing educational theories in the real world is challenging.
For example, in some cases, the ideal unit of study is individual students in the same classroom, but it's not always possible to simultaneously implement treatment and control programs (e.g., using textbook vs. computer-based exercises) without disrupting classroom flow. If the unit of study is the classroom or the school, teachers and principals may perceive a control-group designation as a "sentence to reform purgatory," even though researchers may find the "old way" works best. Also, researchers must get parental consent before involving children in a study, and parents may perceive a control-group designation as unfair--or an experimental-group designation as "using their child as a guinea pig."
Possible solutions How can random assignment be made feasible and palatable? What could be done to make educators more likely to consider serving as the control group for a year or more, based on a coin flip?
Johns Hopkins University researcher Robert Slavin has proposed a solution called "paired awards." Districts wishing to try an intervention would apply for a grant and, at the same time, identify pairs of schools (or teachers within schools) willing to participate in research. Those randomly selected as controls would continue using their regular programs for one year but would then be eligible to adopt the treatment under reduced costs or other concessions.
University of Memphis researcher Steve Ross recommends refining the paired-awards approach so that the treatment-control comparison lasts at least two years, citing the implementation dip: "Given new strategies to understand and put into practice, teachers may struggle for a while, with short-term negative effects on teaching and learning. It would seem ironic if increasing randomized field trial usage caused the effectiveness of many potentially valuable programs to be systematically underestimated."