“Weathering the Storm”: Federal Efforts Helped Bolster U.S. Education Standing Among Peer Nations
The OECD released international testing data from the Program for International Student Assessment (PISA). The PISA measures 15-year-olds’ performance in math, science, and reading worldwide every three years.[1] The previous assessment was done in 2018 and the recent release was conducted in 2022–allowing for an examination of COVID’s impact on students and of global recovery efforts.
The 2022 PISA results reveal that the COVID-19 pandemic had substantial effects on education systems around the globe. As shown in Figure 1, the OECD as a whole saw test scores decline in math and reading by 15 and 10 percent of a standard deviation between 2018 and 2022 in math and reading, respectively.[2] While there were small measured declines in science, these declines were not statistically significant, and were within the margin of error. Though it also experienced real declines in math, the U.S. seems to have fared better than many of its peers, indicating considerable resilience at least in some communities: the U.S. saw a score declines of 11 percent of a standard deviation in math, and no statistically significant change in reading or science.[3] These declines represent U.S. score changes that are approximately 86 percent of the size of OECD averages in math and less than 10 percent of the size of OECD averages in reading.
To examine this relative U.S. resilience, this issue brief discusses the Federal government’s policy response to combating test score declines due to COVID-19, sheds lights on successful models, and outlines what remains to be accomplished.
U.S. Policy Approach/Overview of Funding
In response to COVID-19’s likely impact on student learning, the Biden-Harris administration made an unprecedented Federal investment in K-12 schools – representing $122 billion of the $190 billion allocated across three COVID relief packages (Figure 2). This funding, part of the Elementary and Secondary Emergency Relief Fund (ESSER), allows states to spend on academic recovery, health and safety, and mental health and wellbeing across their schools and districts. The ESSER funds were to be spent across five years, representing an increase of roughly $40 billion in additional federal funding per year. For comparison, during the Great Recession, the American Recovery and Reinvestment Act (ARRA) of 2009 distributed $16.6 billion annually to schools over three years to offset K-12 expenditure declines. Annual federal appropriations for public elementary and secondary schools, have been around $67 billion per year since 2012. As such, these ESSER funds represent a historic boost in Federal support for public schools over the time period. Even so, it is important to point out that relative to total public K-12 spending (which predominantly comes from state and local sources), this represents a roughly 5 percent increase.
Correctly anticipating that low-income schools would be hardest hit by the pandemic, ESSER funds were allocated based on Title I rules. Figure 3 illustrates that districts with a higher share of low-income students received more per-student funding. On average, ESSER distributed around $1,000 per student annually. However, this varied from less than $400 in more affluent districts to almost $1,600 in districts with the most students qualifying for free or reduced-price lunch.
While recovery funds have gone towards strategies for which there is evidence of real causal impact, it is too early still for research that establishes a causal link between funding and academic recovery from COVID-19 per se. The U.S. has taken an aggressive approach compared to many peer nations. Available data suggest that many peer OECD countries haven’t sustained similar levels of education spending in recovery efforts as the U.S. This suggests that federal recovery efforts may have supported U.S. resilience.
What COVID Relief Dollars Can Do
Schools spent ESSER funds on a wide variety of authorized uses such as staff salaries, digital learning resources, COVID safety measures like HVAC upgrades, and maintaining meal services for students during remote learning. As shown in Figure 4, about 60 percent of the expenditure was “traditional instructional spending”, with “academics” and “staff” accounting for the bulk of instructional spending (44 percent and 42 percent respectively). Models of optimal behavior suggest districts will allocate funds optimally based on their specific needs (Jackson and Mackevicius 2023). That is, a district with many computers but very few social workers will benefit more from spending on social workers than technology, while the converse would be true for a school with little technology infrastructure and many social workers. In both cases, spending on the needed input is more productive than it would have been in the other district. This makes a comparison of the effect of specific inputs very difficult to interpret, as it combines the effect of the input with differences in the extent to which that input was needed across settings. A more meaningful question then, is what happens when schools have more money to spend as they see fit. For this reason, we do not differentiate between spending categories and instead consider the probable impact of a $1,000 per-pupil spending increase.
Existing studies find that increasing spending by 7-8 percent is associated with test score increases of 3.2 percent of a standard deviation when sustained over four years—with effects as large as 8.2 percent of a standard deviation for “high-impact” spending. As such, based on existing work, one can infer that test scores would have been 3-8 percent of a standard deviation lower than what we see reflected in the PISA scores had there been no additional COVID relief funding.[4] This level of decline would put U.S. test scores at or below the OECD average in all three subjects. More plainly, if you asked us to predict (based on the best available evidence) the effect of ESSER’s $1,000 per student annual investment in education in the United States, our prediction about would be about what we see: a modest but meaningful improvement in U.S. standing.
While the U.S. has made commendable relative progress—compared to other OECD countries, there remains room for improvement. Test score declines are less pronounced in lower grades, but as of Spring 2022, National Assessment of Educational Progress (NAEP) data show an approximately 8 percent and 20 percent of a standard deviation learning shortfall in 8th grade English and Math scores, respectively. However, it’s not all bad news. As we detail below, there are proven strategies to enhance student outcomes, and some states have leveraged, and continue to leverage, the Biden-Harris Administration’s policy leadership to achieve significant gains.
What Kinds of Evidence-Backed Interventions Can Aid Recovery?
There are many approaches that schools and districts can take to remediate test score declines. As one solution, many researchers and administrators have highlighted one-on-one or small group tutoring, which can be an important instructional accelerator for students of all ages. The sizable and relatively rapid impacts of high-intensity tutoring on student test scores and grades are well–documented in small-scale settings and particular contexts. However, CEA analysis provides promising signs that high-dosage tutoring can work at scale.
We consider the effects of state-wide tutoring programs implemented during the 2021-2022 school year using NAEP score changes from 2019-2022. To avoid simply comparing states with tutoring to those without tutoring, which may be subject to selection bias, we compare the change in test scores (in standard deviations) between 2019 and 2022 between states with and without state-wide tutoring implemented during that time. While all states may have had similar ill-effects of the pandemic, those states with tutoring in place appear to have “weathered the storm” better.[5] While we do not claim causality using this approach, these patterns are encouraging.
Because there was significant variability on how tutoring was offered, the patterns we identify do not speak to the benefits of a full statewide tutoring intervention with high fidelity. One example of a state that has undertaken high-fidelity efforts after the pandemic is Illinois. Using ESSER funds, the Illinois Tutoring Initiative provided high-impact tutoring for students at 125 schools in districts disproportionately impacted by the pandemic beginning in 2020. During the pandemic, Chicago Public Schools also adopted a new tutoring program: Tutor Corps, which represents one of the largest tutoring efforts in the country. The program started in 2022 with at least 5,500 students receiving at least one tutoring session and as of March 2023, nearly 10,000 students had received at least one tutoring session and about 3,700 students receive regular sessions. To shed light on the potential successes of a state-wide high quality tutoring intervention, Figure 5 shows the evolution of NAEP scores (math and ELA combined) for Illinois relative to the average for other states in each year of the NAEP since 2009.[6] Whereas test scores in Illinois track those of other states before 2019, Illinois scores jump considerably relative to other states in 2022. This is an increase of about 8 percent of a standard deviation (a large effect relative to the pandemic-related test score declines). While Illinois’ relative success in 2022 is noteworthy, it may reflect all unique aspects of the state’s approach, not just tutoring. However, the significant role of high-quality, high-dosage tutoring in their recovery efforts suggests it is a successful strategy worth replicating elsewhere.
While high-dosage tutoring is among the most discussed interventions to accelerate academic recovery, there are other proven and promising interventions that have been shown to improve student achievement. These include expansions of afterschool programs and summer programming; instructional supports to increase the capacity of educators, including through literacy and math coaches; and strategies to curb the increase in student absences.
Promoting Best Practices
In addition to providing necessary resources, providing leadership on best-practices, the Administration, in collaboration with major organizations, launched the National Partnership for Student Success (NPSS) to provide additional tutors, mentors, student success coaches, wraparound service coordinators, and post-secondary transition coaches. Many states, under the Administration’s guidance, have allocated funds to proven recovery solutions. For instance, North Carolina used ESSER I funding for early adoption of high-dosage tutoring. Louisiana Accelerate supports local school systems implementing tutoring with roughly $1 million in seed funding from ESSER. Colorado established a High-Impact Tutoring Program using $6.4 million in ESSER funds to train and deploy tutors, while Oklahoma is investing in math tutoring for up to 1,500 secondary students per year with their ARP ESSER funds. Virginia’s “All in VA” ESSER funds to evidence-backed high-intensity tutoring for at-risk students. Of course, the structure of training and compensation for tutors can vary substantially based on the unique needs and constraints of schools. Stanford’s National Student Support Accelerator suggests various tutor trainings along with online pre-service trainings such as Saga Coach. The training level of tutors also impacts tutor payment with some models proposing payments of $20 per hour for non-professionals and payments of up to $50 per hour for teachers and professional tutors.
In addition to tutoring initiatives, other states have invested in summer and after school programs: states like Michigan, Minnesota, Texas and Alabama have either established or expanded funding for after school programs.
Conclusion
The Biden-Harris administration’s unprecedented investment in U.S. public education and leadership has sparked creative initiatives nationwide. These efforts helped the U.S. navigate the COVID-19 pandemic and emerge relatively resilient compared to peer nations. Schools, teachers, and districts have contributed significantly to this outcome. But the work is not done and significant opportunities to address these challenges remain. The latest federal data indicates that districts are still in the process of spending tens of billions in ESSER resources as we move towards the September 2024 obligation deadline—investments that can promote success in the future. As we continue to work to accelerate the recovery from learning loss, the administration is dedicated to expediting this process through continued investment and support.
[1] Given that spending on personal protective equipment and infrastructure to facilitate hybrid learning and promote greater student attendance was foundational to other instructional inputs, it is reasonable to assume that the spending could have been high impact.
[2] This year’s PISA testing was delayed one year due to the COVID-19 pandemic.
[3] To aid interpretation, 10 percent of a standard deviation is equal to about 4 percentile points.
[4] Note that the OECD change in scores for science, as well as the US change in scores for both reading and science are not statistically distinguishable from zero and thus cannot be characterized as a “decline.”
[5] See data appendix for further analysis details.
[6] In general, NCES does not recommend combining scale scores. In this analysis, however, we combine math and reading standardized scores, which is shown to be an effective way of averaging across scales. See Kling, Liebman, and Katz (2007) for further discussion.
