If you told us about all of your work activities during the past hour, could you tell us accurately?  Would you tell us accurately?  We suspect you might put a particular sheen on your activities, either because you wanted to make yourself look good or because you just couldn't remember.  (We all have those challenges.)   

But in research, our job at Hezel Associates is to discover as precisely as possible the reality of perceptions and behaviors.  So we have been analyzing the veracity of self-reports, particularly of teacher behaviors.    

Traditionally, researchers choose from self-reported methods-collected through surveys, self-assessments, interviews, logs, focus groups and group interviews-and objective sources of data-gathered from observations and secondary research.  Self-reporting from study participants is the sole means of collecting data on certain behaviors, perceptions, and attitudes, yet researchers are challenged by many difficulties of collecting self-reported data.  Hezel Associates has gained ground in validating study participant behaviors with the help of technology through electronic tracking.
 
Researchers bridge the divide between self-reporting and objective data collection with electronic tracking-a method which uses technology to provide an objective log of study participant actions.  We're not saying that self-reports are not objective data, just that they are individually subjective data points that together form an aggregate picture of what people say about themselves.
 
Hezel Associates has seized the opportunity provided by our clients' focus on education technology to mine resources for previously unavailable data.  This new data is more rigorous, provides a means for triangulation of traditional data, and assists in providing a more accurate and nuanced view of research results.  In turn, electronic tracking helps alleviate some of the challenges to collecting accurate self-reported data.  Electronic tracking also opens an entire new pool of information that proves useful for creating richer sets of data.
 
For an external evaluation of Video in Teaching and Learning (VITAL) online instructional resources conducted by Hezel Associates, VITAL's facilitators provided the evaluators with access to the electronic tracking of pilot teachers' use of the resources, with the study participants' consent.  They used this data to determine the frequency of each teacher's VITAL login, which and how often they downloaded resources (i.e. video clips, worksheets), and their activity as they navigated the website.
 
This data allowed Hezel Associates researchers to create three profile groups of VITAL study participants[1]:

• those who typically used VITAL less than once a month 
• those who used VITAL once a month 
• those who used VITAL greater than once a month.

These profiles, based on electronic tracking data, proved tremendously helpful in elucidating some interesting findings.  Those participants who used VITAL greater than once a month typically had more teaching experience and/or had received in-person training.  Those who typically used VITAL less than once a month tended to teach in small schools, urban districts, and/or were ELA teachers.  All activity by users decreased over time--a troublesome finding--indicating a novelty effect associated with VITAL.  This could be due to users accessing all resources of interest early on in the study. 
 
An even more troublesome finding, with implications for the ethics of study design and instrument choice, was the surprising discrepancy between self-reported data and electronic tracking data.  If researchers based their findings solely on self-reported data, they would have found that nearly ten percent of study participants used VITAL less than once per month - yet electronic tracking showed that 22 percent of study participants accessed VITAL less than once per month (a 12 percent difference).  Additionally, nearly 90 percent of participating teachers indicated they downloaded at least one VITAL resource each month, whereas online tracking data clearly showed that about 32 percent of teachers averaged fewer than one resource download per month.
 
There are a number of potential rationales for these findings. For one, users of VITAL may have commitments and time constraints beyond their control that influence their ability to access VITAL as frequently as was required by the study, and possibly even as much as they might wish.  Also, the Hawthorne Effect could have implications for these discrepancies, where performance increases when participants are aware of the researcher's presence - even if it is only knowledge that their actions will be tracked for future reference.  With the necessary involvement of researchers in troubleshooting technology issues, communicating to keep participants engaged in a longitudinal study, and encouraging them to complete other study and data collection activities, researcher influence was unavoidable.  Therefore, participants' inclination to please the researcher and fulfill their study requirements may have conflicted with their previously mentioned constraints, possibly motivating participants to skew their reported level of performance.
 
Electronic data can be extremely helpful in creating a more accurate and nuanced representation of study participant activity.  Hezel Associates' researchers found the use of technology important to their research on, and evaluation of, VITAL's online instructional resources.  However, electronic tracking does come with some constraints: Fidelity measures are costly and time-consuming.  Also, the client must work with the research team to make these data available.  The data may not be in a useable format, which may require many man-hours to clean before it can be analyzed.  The actual analysis may also take considerable time and resources due to the sheer quantity of data available. 
 
Still, with the discrepancies highlighted between self-reported and electronic tracking data, the richer and more nuanced analysis and findings, and the increasing use of technology in research and evaluation, use of electronic tracking will certainly continue to increase---at Hezel Associates and elsewhere in the field.

[1] *The minimum expectation of VITAL use during the experiment was once each month