The Cart vs. the Horse: NCLB, School Reporting, and Research Directions
Statewide and locally, schools and school districts face calls for increased accountability in order to meet the requirements of the federal No Child Left Behind Act (ESEA 2001). Under NCLB, states must include in their reporting particular indicators for academic performance at the school, district, and state levels, with special concern for documenting the academic progress of students who are racial minorities, who attend high poverty schools, and who are members of other identified subgroups. These performance indicators (usually tied to standards-based assessments) are “high stakes” — failure to meet annual performance improvement objectives carries with it specific consequences.
Undeniably, the added reporting places considerable burdens on educators. But for research organizations like Hezel Associates, there has never been a richer array of information available for analysis. The greatest challenge typically associated with conducting research is not a lack of good models of inquiry, issues of concern to address, or even lack of suitable funding. Instead, the largest obstacle to conducting research has been the actual collection of data. Schools are understandably protective of student privacy and instructional time — any activities that take away from classroom instruction can potentially detract from student learning. For this reason, even credible education researchers who have studies that closely map onto schools’ improvement needs — studies of course content, professional development, and college going — have encountered difficulties convincing schools to provide access to student- and school-level data.
The expansion of school- and student-level performance data means that education researchers can design studies that take advantage of the existing information. This has to take place carefully, however. Most performance data are linked to scores on state standards-based tests. To make use of such scores as outcome variables, they have to be persuasively connected to the research question, the intervention, and the predictor variables used. For example, it makes no sense to use student scores on a standardized reading test as an outcome variable if an intervention (e.g., introduction of vending machines to school hallways) has little or nothing to do with the reading curriculum or the reading instruction that takes place.
One example of how standardized test scores can be used by researchers to examine compelling questions is a recently completed quasi-experimental study by Hezel Associates that was conducted as part of our independent evaluation of PBS TeacherLine. In this study, the evaluation team looked at teachers’ participation in TeacherLine’s online professional development courses in math. The content of these courses aligned well with the math content emphasized on the Florida Comprehensive Assessment Test (FCAT), which prompted us to use school-level FCAT scores as outcome variables. Using course participation data provided by the Miami-Dade County Public Schools, the evaluation team was able to identify those schools that had high proportions of teachers participating in TeacherLine. We were able to combine this information with publicly available school demographic data to create a matched sample of comparison schools. In examining school-level FCAT scores, as well as scores in subareas of the FCAT, we were able to detect differences between the TeacherLine and non-TeacherLine schools that closely approached statistical significance, which suggested that the online courses appeared to be a professional development strategy that, perhaps in tandem with other school improvement efforts, was associated with student success. During the next year, the evaluation team will follow up on this initial research to investigate school-level factors that might have supported TeacherLine in the treatment schools and contributed to the differences in FCAT scores that emerged in the analysis.
NCLB’s impact, then, goes far beyond school walls. By defining the information for which schools are held accountable and by mandating the reporting of such data, NCLB has also influenced the design and implementation of research. Certainly, not all important research centers on the results of standards-based tests. But the emphasis on these data, coupled with their widespread availability, means that researchers will continue to integrate such information into studies whose bottom line — students’ academic performance, as measured by standardized testing — are meaningful to school improvement under NCLB.