Josh was asked recently to review the state of adolescent literacy interventions during his appearance at the Ohio Literacy Academy. His presentation highlighted one of the challenges school and district leaders face in developing multi-tiered systems of support in our middle and high schools: the lack of interventions designed for this age group that have been researched with experimental or quasi-experimental methods.
The Institute for Education Sciences (IES) is one of the leading agencies in educational research, evaluation, and statistics. They host a website called the What Works Clearinghouse (WWC), which provides detailed research and reviews on educational strategies, materials, and programs.
For a literacy intervention to be published on the What Works Clearinghouse website, it must meet the qualifications set out in the WWC Procedures and Standards Handbooks (WWCH, 2013).
This process includes five steps.
- Developing the review protocol. The WWC defines the purpose and parameters for conducting a review of a particular educational practice.
- Identifying the relevant literature. In the second phase, they gather all the relevant research on the subject that has already been published.
- Screening studies. In the third step, the WWC then screens the research for certain qualifications set out in step one. The WWC only includes research that has been conducted through primary analysis, has an experimental or quasi-experimental design, and has a legitimate sample population. The WWC also looks at publication time frame and outcome measurements.
- Reviewing studies. After collecting and screening research, What Works Clearinghouse then performs an in-depth review of the research. Review requirements are set out within the WWC Standards Handbook and aim to synthesize the research to see if the intervention has enough evidence to be valid.
- Reporting on findings. The fifth and final step is to publish their findings for other educators. The final results of this process are on the What Works Clearinghouse website.
Because of this thorough review process, the WWC is usually a high quality resource for ideas for effective practice. However, when one starts digging around, it becomes apparent that even on such a credible site, there is a gap in the research. Of the 233 literacy interventions listed on the What Works Clearinghouse, only 29 are labeled for use with middle and secondary. Of those 29, only 17 are listed on the WWC as having positive or potentially positive effectiveness based on the WWC criteria. And of those 17, many of these interventions are programs that require extensive time and professional development to be implemented.
Here, it should be noted that the IES What Works Clearinghouse focuses primarily on the internal validity of educational interventions. While this approach provides educators with a good understanding of an intervention’s theoretical effectiveness, it excludes other important factors like the time it takes to implement, the professional development needed for staff to integrate interventions into their instruction, and additional obstacles in the implementation process for new interventions.
Other organizations have begun to develop more comprehensive reviews to help literacy leaders understand both the efficacy and the practicalities of available Tier 2 and Tier 3 literacy interventions.
- Evidence for the Every Student Succeeds Act (ESSA; https://www.evidenceforessa.org/) provides reviewed resources that meet evidence-based standards set out by the ESSA. In addition to providing a rating on effectiveness, Evidence for ESSA also provides information regarding how the practice is implemented, including professional development, staffing, and technology requirements.
- The National Council on Intensive Interventions (NCII; https://charts.intensiveintervention.org/aintervention) connects educators with resources in a similar way as well. On their website, they host charts of examined interventions with an analysis of their efficacies. The charts provide detailed descriptions of the interventions as well as the resources needed to implement them. NCII focuses on strategies that are built on data-based individualization. As such, the majority of their listed interventions require data collection, instruction, assessment, and adaptation to be part of the intervention.
- Reading Ways resource catalog includes a section dedicated to program and intervention evaluation. We review the research on each intervention and evaluate the challenges for implementation, but we also do in-depth interviews with our partner schools that have direct experience with implementation. These reviews are invaluable for our consultants when conducting readiness evaluations, supporting our literacy leadership course, and providing MTSS technical assistance.
It is important to remember that according to section 8101(21) of the Elementary and Secondary Education Act (ESEA), federal funds must be spent on interventions that have evidence of effectiveness. Middle and high school leaders have few options that are at the highest tiers of evidence. Therefore, we need to be prepared to “demonstrate a rationale based on high-quality research findings or positive evaluation that such activity, strategy, or intervention is likely to improve student outcomes” and “includes ongoing efforts to examine the effects of such activity, strategy, or intervention.” Learn more about how we support this work through our MTSS technical assistance.
References
Clearinghouse, W. W. (2013). What Works Clearinghouse procedures and standards handbook. Washington, DC: Institute of Education Sciences. https://ies.ed.gov/ncee/wwc/Docs/referenceresources/wwc_standards_handbook_v4.pdf