Conforming to Standards: New Designs for Evaluation
Incorporating experimental research into an external evaluation of school-based technology initiatives presents challenges, most notably in logistics, recruitment, and method according to conference panelists at the American Education Research Association (AERA). The tasks are made even more difficult when an intervention is already underway and must reconform to the US Department of Education’s Gold Standards for research and evaluation.
First articulated in 2003, the standards arrived two or more years into the project cycle of many technology-related projects, which has had profound implications on the scope of work required for evaluation teams. Charlotte (Lani) Gunawardena, external evaluator of the Star Schools project HbL4U and speaker at AERA 2004, pointed out that a central dilemma in her evaluation focused on the incongruity of using the Gold Standard’s variable-oriented research approach for a process-oriented project. Since it is impossible to reorganize participating schools into treatment and comparison groups at this stage of the project, the evaluators have opted to rely on a repeated measures design in a quasi-experimental framework. Frank Zittle, an external evaluator for the Star School program’s TECH Share initiative, agrees that the original intent of a project that was awarded four years ago may frustrate attempts to work an experimental approach into the evaluation. For the CEER evaluation team, the variety in modules’ goals and content that has been the hallmark of TECH Share has challenged attempts to create a standard assessment that reflects teacher and student outcomes.
Paula Szulc Dominguez, Hezel Associates’ director of research and evaluation, noted in her presentation that the format and delivery of PBS TeacherLine’s online courses have lent themselves to an experimental approach. Still, the recent emphasis on experimental findings and student outcomes has meant that other kinds of evaluation data and activities – for example, those that center on TeacherLine’s sustainability and dissemination – cannot be investigated as thoroughly.
Paula Szulc Dominguez presented “Beyond the Glitter of Online Professional Development” as a member of an AERA panel exploring the topic of online teacher professional development. A Reality Check on the Gold Standard: Issues and Challenges of Evaluating Department of Education Technology Projects during the AERA national conference held in San Diego, California in April, 2004.