Learning Loss in Reading and Math in U.S. Schools Due to

Learning Loss in Reading and

Math in U.S. Schools Due to the

COVID-19 Pandemic

March 2021

Victoria N. Locke, PhD

Istation

Chalie Patarapichayatham, PhD

Southern Methodist University

Sean Lewis, MA

Istation

1

Executive Summary

This study investigates students’ achievement loss in reading and math due to

COVID-19 pandemic school closures. Using a nationally stratified sample from the

extensive Istation database across two academic years, we found that

students showed learning losses in both reading and math due to the school closures.

Learning loss in reading

Students lost approximately 2 months’ learning in reading due to the COVID-19

school closures in addition to the regular 1 month of summer learning loss. Most of this

was due to loss of learning gains typically experienced in April and May. Schools

implemented remote learning during this time, and it helped students to maintain

learning gains they had made from September March of the 2019-2020 school year.

Learning loss in math

Students typically have greater summer learning losses in math than they do in

reading, and these results show that students lost additional months of learning due to

the COVID-19 pandemic school closures. Students entering first grade in fall 2020 had

higher scores in September than they had in March. When compared with typical

summer learning loss in math, students entering second grade lost an extra month,

students entering third and fourth grades lost an extra two months, students

entering fifth and sixth grades lost an extra three months.

2

Learning loss varied by type of school

We evaluated the learning losses by the poverty level of each school, categorized

as low , mid-low, mid-high, or high. For students entering first grade in 2020-2021,

students enrolled in mid-low and low-poverty schools had larger learning losses as

compared to similar students in first grade in 2019-2020, and students in high-poverty

schools had less learning loss. In second grade, the biggest differences were in mid-high

and mid-low poverty schools, and in third grade the low-poverty schools had the biggest

mean differences.

Students in upper elementary school had scores in September 2020 that are

lower than in September 2019, with a few exceptions of students in sixth grade in high-

poverty schools and seventh grade students in mid-high poverty schools. Sixth and

seventh grade students in low-poverty schools had scores that were comparable to

students in the previous year’s cohort.

Conclusion

• Learning losses were greater in math than in reading.

• Learning losses varied by grade.

• Learning losses varied by poverty status at the school, and typically students

enrolled in high or mid-high poverty schools had learning losses that were lower

than, or similar to, students enrolled in other types of schools.

3

Introduction

In March and April of 2020, schools across the country closed their doors and

sent their students home in response to concerns about spreading the novel coronavirus,

which causes the illness commonly known as the coronavirus disease 2019 (COVID-19).

Schools implemented at-home learning using educational technology, conference calls,

and increased parental involvement (Malkus, et al., 2020).

From the beginning, there were concerns about student equity and access to

technology. Many districts provided students with electronic devices and hot spots for

internet access if they did not have access to technology at home. Student engagement in

education was mixed during this time. Some students had access to technology and

parents who monitored their participation in their learning, while others simply

disappeared from schools (Swaby, 2020). Students also lost instructional days as

districts and schools extended spring breaks, shortened academic calendars, and spent

time implementing remote learning (Malkus, et al., 2020).

Istation, a leading provider of educational software, including progress

monitoring and adaptive curriculum, expanded its offerings to include progress

monitoring at home for its math and reading assessments, known as Istation’s

Indicators of Progress (ISIP™). Istation also offered a parent portal, where parents cand

track student participation. Istation’s school-based curriculum was offered to all

participating school districts, as was Ipractice, Istation’s at-home curriculum. Istation

also made its teacher-directed lessons available to parents to help them with at-home

instruction. Istation also offered a series of videos for parents and teachers on how to

conduct in-home, remote learning.

4

When schools opened in fall of 2020, school districts across the country instituted

a mixture of in-person learning, remote learning, and a hybrid where students

participated some at home and some in the classroom. These learning models were

adapted to local conditions and were subject to change if there were COVID-19

outbreaks in the area.

Istation continued to offer progress monitoring at home, and districts had the

option to allow progress monitoring at home or only in school, where testing conditions

are controlled by the teacher. Schools were advised that if the progress monitoring was

conducted at home, the scores could be used to inform instruction, and Istation also

advised schools to remind parents not to assist with the assessment, as it was important

for the students to work independently.

There is also concern amongst educators and researchers regarding learning loss

and whether the pandemic’s school closures have created a learning slide that puts

students further behind (Dorn, 2020; Kuhfeld, Soland, et al., 2020). Since remote

learning at home on a large scale was new, it is unknown how much this might impact

student achievement for students of different socioeconomic backgrounds, and if the fall

2020 cohort of students had lower achievement, indicating a greater summer slide.

Researchers from the Northwest Educational Association (NWEA) (Kuhfeld,

Soland, et al., 2020; Kuhfeld, Tarasawa, et al., 2020), projected the potential impact of

COVID-19 school closures. Their projections were first disseminated in the spring, and

they projected students would begin the 2020-2021 school year with 63% to 68% of the

learning gains in reading and 37% to 50% of the learning gains in mathematics. Other

research using fall 2020 data showed that students had lost 13% of their gains in

5

reading and 33% of their gains in math. Learning loss was greater in schools that served

students of color (Dorn et al., 2020). Similar results were available from Stanford

University, which showed that there were differences by state. South Carolina had the

largest estimated learning loss, and losses were greater in math than they were in

reading (Stanford, 2020).

Summer Learning Loss

To better understand the impact due to the pandemic, it is important to

distinguish between losses due to early school closures and remote learning, versus what

is typical learning loss over the summer. Research on summer learning loss shows that

there are differences between reading and math and differences in grade. Some research

indicates that students’ achievement scores decline over the summer by 1 month’s worth

of school-year learning. Students lose their math ability more rapidly than reading, and

these losses are greater for students in higher grades (Quinn & Polikoff, 2017). Other

research demonstrates that students from all socioeconomic backgrounds forget more of

what they learned in math over the summer than the amount they lose in reading skills

(Shafer, 2016).

However, students who are lower achieving may experience less loss, and may

even experience some gains, as compared to students who are higher achieving, perhaps

due to attendance at summer learning programs (Campbell et al., 2019). Many school

districts across the country offer summer learning or enrichment programs designed to

keep students learning over the summer. These programs are especially common in

areas that serve students living in higher-poverty households. Other research shows that

students who are high achieving continue to grow over the summer, and average

6

students will not, and thus high-achieving students’ growth, while slower in the

summer, still helps them to gain more compared to average or lower-achieving students

(Rambo-Hernandez & McCoach, 2015)

This study investigates students’ learning loss from COVID-19 school closures

using Istation’s Indicators of Progress (ISIP™) assessments in reading and math.

Istation is an integrated learning system that provides a formative assessment that can

be used for either progress monitoring or benchmarking. Over 4 million students have

been assessed using ISIP Early Reading (ER), ISIP Advanced Reading (AR), and ISIP

Math.

This study has several goals. The first goal is to evaluate whether the students in

fall of 2020 had lower achievement than similar students in the fall of 2019, and if those

results differed by socioeconomic status at the school level.

This study will specifically address the following research questions:

1. What were the learning losses from March to September of 2020 due to school

closures during the pandemic and the institution of remote learning?

2. Did these losses vary by socioeconomic status?

Methodology

Measures

ISIP assessments are web-delivered, computer-adaptive testing (CAT)

assessments. ISIP gathers and reports frequent information about student progress in

the critical domains throughout and across academic years (Mathes, 2011;

Patarapichayatham et al., 2013).

7

The purpose of ISIP Reading is to measure students’ reading ability and identify

deficits in critical areas to provide continuous differentiated instruction. ISIP ER is

available for prekindergarten through third grade, and ISIP AR is available for fourth

through eighth grade. ISIP Math is designed for students in prekindergarten through

eighth grade (Istation, 2018).

Methods and Data

We used two separate methods for this analysis, and they are separated by the

two research questions. A piecewise analysis was used to determine the learning

trajectory of students from September 2019 to September 2020. For the second research

question, we used a cohort analysis that compares mean scores from a subset of the

sample used in the piecewise analysis.

Piecewise Analysis

For the first research question, we used a piecewise growth model to estimate

students’ COVID-19 learning loss. It is a type of time series analysis for nonlinear

growth with longitudinal data. Growth models examine the development of individuals

on one or more outcome variables over time. The outcome variables can be observed or

continuous latent variables. A model was fit by grade level using Mplus software. Mplus

handles the relationship between the outcome and time by allowing time scores to be

parameters in the model so that the growth function can be estimated. This is the

approach used in structural equation modeling (Muthén & Muthén, 1998-2017).

In a piecewise growth model, different phases of development are captured by

more than one slope growth factor, and it is used when growth is not linear (Muthén &

Muthén, 1998-2017). In this study, it allows for multiple slope factors in the model.

8

COVID-19 school closures occurred in mid-March, and we expected that growth would

vary from March to the following September. During April and May growth is harder to

predict given that many students did not have scores, and those that did were from at-

home progress monitoring. Growth over the summer would also vary depending on

student achievement level, grade, and access to resources within the home. Therefore,

we expect that students will have greater summer learning loss than expected in typical

years, and it might vary by achievement level.

Kamata et al. (2013) demonstrated ways to model nonlinear growth using three

testing events. They demonstrated the growth models in the context of curriculum-

based measurement with the fall, winter, and spring reading fluency benchmark

assessments using various models including a linear growth model, a piecewise growth

mixture model, a growth mixture model, and a growth model with an estimated time

score model. Their research showed that a piecewise growth mixture model performed

well with three test events. Therefore, we applied a piecewise growth model with our

longitudinal data with eight test events to estimate an effect of learning loss due to the

school closures during the COVID-19 pandemic on students’ achievement in reading

and math.

Piecewise models were less stable when we attempted to run them on models

from each SES separately. Therefore, we used a cohort approach and compared mean

differences in September 2020 to assess the differences by grade and socioeconomic

status at the school (SES).

The piecewise analyses were completed by grade level for ISIP ER, ISIP AR, and

ISIP Math. Figure 1 shows a model in this study. There are eight test events. The model

9

shows the intercept of the growth factors and the slope for students’ learning. The “i” is

growth, which includes the first seven test events, which cover their growth before the

school closures (SEP 2019, OCT 2019, NOV 2019, DEC 2019, JAN 2020, FEB 2020, and

MAR 2020). The “s2” is the slope for the second phase of growth, which has only the

September 2020 test event (SEP 2020), when students headed back to school to start

their new academic year.

Figure 1: Piecewise growth model

September is considered the beginning-of-the-year assessment month, and May

is the end-of-the-year assessment month. April, May, June, July, and August were not

included in the analysis. The students’ grade level in the 2020-2021 school year is used.

In reading, we do not have results for students in fourth grade because of the differences

in scale scores, and eighth grade is eliminated due to a convergence issue.

Cohort Analysis

For the second research question, we used a cohort analysis approach and

compared the mean scores for September 2019 to September 2020 by grade. For this

analysis, we also compared mean scores by type of school to evaluate whether some

10

schools had greater loss due to the COVID-19 pandemic school closures. We calculated

statistical significance and effect sizes, as well as differences in mean percentile ranks.

Data

The data for this study came from the extensive Istation database. Istation serves

millions of students with their reading, math, and Spanish-language assessments. We

selected students that had data in both the 2019-2020 and 2020-2021 school years. This

helped to control for missing data in key assessment months and helped ensure that the

sample was equivalent for both academic years.

Since Istation offered home progress monitoring during the pandemic and there

may be mean differences in scores between home and school assessments due to a

difference in the testing environment, we only selected students who had progress

monitoring at school in September 2020. This helped to control for the possible effects

of the testing environment.

While the Istation database is extensive, we did not use all available students as

this may introduce some selection bias into the results. Typically, students from higher

socioeconomic status households have higher achievement than students from lower

socioeconomic status households. Since the Istation database is slightly skewed toward

schools that have higher percentages of students receiving free or reduced priced lunch

(FRPL, which is a measure used to indicate SES), and this may impact the results, we

selected a stratified sample according to the school SES.

To do this, we created four categories for SES, using the categories from the

National Center for Education Statistics. SES 1 consists of schools that have 75% or

11

more of their students enrolled in the FRPL program. SES 2 schools have 50% to 74.9%

enrolled in FRPL, and SES 3 schools have 25% to 49.9% of students enrolled in FRPL.

SES 4 schools have less than 25% of students enrolled in FRPL. Next, we calculated the

percentage of students that were enrolled in each of the four SES school types according

to enrollment data available from NCES for public and public charter schools and used

this information to create sample targets.

For the piecewise models, we created code in R statistical software to conduct

sampling with replacement at all grade levels. In grades 1-5, we used a sample size of

10,000–10,100 students per grade. In grades 6 and 7, we had lower numbers of

students at some SES levels and used a smaller sample size to limit the sampling with

replacement. This was accomplished by using code in R statistical software to select a

sample that matched enrollment percentages. For the cohort analysis, we selected cases

that had both September 2019 and September 2020 scores; they were a subset of the

piecewise sample. A description of the sample is available in Table 1.

Table 1. Description of the ISIP Reading and ISIP Math Analytical Samples

Piecewise Models

Assessment Grade as of September

2020

Sample Size

SES 1 SES 2 SES 3 SES 4

Reading 1 10,090 36% 16% 20% 28%

2 10,090 36% 16% 20% 28%

3 10,090 36% 16% 20% 28%

4 10,065 36% 16% 20% 28%

5 10,087 36% 16% 20% 28%

6 4,037 29% 18% 24% 30%

7 1,515 29% 18% 24% 30%

Piecewise Analytic Sample Grades 1-5 36.3% 16.1% 19.4% 28.2%

NCES Enrollment Grades 1-5 36.2% 16.1% 19.5% 28.2%

12

Piecewise Analytic Sample Grades 6 and 7 28.9% 17.8% 23.7% 29.7%

NCES Enrollment Grades 6 and 7 28.9% 17.7% 23.7% 29.7%

Math 1 10,090 36% 16% 20% 28%

2 10,090 36% 16% 20% 28%

3 10,090 36% 16% 20% 28%

4 10,090 36% 16% 20% 28%

5 10,090 36% 16% 20% 28%

6 4,039 29% 18% 24% 30%

7 1,515 29% 18% 24% 30%

Piecewise Analytic Sample Grades 1-5 36.3% 16.1% 19.4% 28.2%


Piecewise Analytic Sample Grades 6 and 7 28.9% 17.8% 23.7% 29.7%


Cohort Analysis


2020

Sample Size

SES 1 SES 2 SES 3 SES 4

Reading 1 7,427 33% 16% 21% 30%

2 8,233 34% 16% 20% 30%

3 8,302 33% 17% 20% 30%

4 7,658 35% 17% 21% 27%

5 7,790 34% 17% 20% 29%

6 3,258 28.6% 18.1% 24.7% 28.5%

7 1,153 29.2% 18.6% 22.5% 29.7%

Cohort Analytic Sample Grades 1-5 33.9% 16.7% 20.4% 29.0%


Cohort Analytic Sample Grades 6 and 7 28.1% 18.6% 24.6% 28.7%


Math 1 7,097 37.6% 15.4% 19.7% 27.3%

2 7,201 37.7% 16.8% 19.4% 26.1%

3 7,401 36.8% 16.6% 19.4% 26.1%

4 7,561 35.6% 16.6% 19.1% 27.5%

5 7,668 34.7% 15.9% 20.1% 29.3%

6 3,398 26.9% 17.5% 23.1% 32.5%

7 1,344 26.8% 17.9% 23.3% 32.0%

Cohort Analytic Sample Grades 1-5 36.5% 16.4% 19.7% 27.4%


13

Cohort Analytic Sample Grades 6 and 7 27.0% 17.5% 23.1% 32.4%


In the 2019-2020 school year data, students had ISIP scores from the September

assessment month through March. Students with some missing data were included in

the analysis. Some students had home progress monitoring scores in April and May

during the pandemic. Most students had ISIP scores up to the March assessment

month, some students had scores up to April, and some students had scores up to May

2020. Because there were large amounts of missing data in April and May, and the

potential for sample bias due to access to technology or a conducive home environment,

the April and May scores are dropped from this study. Using information from the

students’ growth trajectories, we projected the expected growth for April and May,

which will be described in the methods section.

We used the students’ grade level in the 2020-2021 school year to compute the

longitudinal growth. For example, a fifth grade student means that the student was in

the fourth grade in the 2019-2020 school year and in fifth grade in the 2020-2021

school year. In reading, fourth grade students were excluded from this study because of

a scaling issue between ISIP ER and ISIP AR. ISIP ER has a different scale than ISIP

AR, and fourth grade students who took ISIP ER in the 2019-2020 school year when

they were in third grade and also took ISIP AR in the 2020-2021 school year had scores

from two different scales.

ISIP Math was renormed for the 2020-2021 school year with a new vertical scale.

For this study, we converted the old scale scores to the new scale scores for this analysis.

Due to a smaller sample in eighth grade that would require heavier sampling with

14

replacement in some SES categories, this grade was dropped from analysis for both

reading and math.

Results

Piecewise Model

Table 2 shows the observed mean score by month. Table 3 shows the estimated

intercepts, estimated slope 1, and estimated slope 2. The estimated intercept is a cut

point on the y-axis, with the ISIP scores on the x-axis. The intercept shows the estimated

scale scores in September 2019. Estimated slopes are the students’ growth trajectories

in each segment in the growth model. The estimated intercepts and estimated slopes in

Table 3 are used to derive the estimated mean scores for each assessment month using

the formula 1 2ˆ i *( 1) *( 2)i i iy s time s time= + + and the results are shown in Table 4.

Observed mean scores and estimated mean scores were comparable across grades in

both reading and math, indicating reasonable results from a piecewise growth model.

15

Table 2: Observed Scale Scores of Each Assessment Month


2020

SEP 2019

OCT 2019

NOV 2019

DEC 2019

JAN 2020

FEB 2020

MAR 2020

SEP 2020

Reading

1 179 185 189 192 194 196 198 199

2 202 205 208 211 213 215 217 221

3 224 226 228 230 231 233 234 236

5 1851 1866 1889 1902 1917 1894 1923 1930

6 1956 1966 1995 2002 2021 1980 2004 1956

7 2030 2041 2063 2066 2083 2030 2044 2072

Math

1 304 318 335 349 357 366 374 391

2 394 410 424 435 443 451 459 450

3 454 462 470 476 479 481 482 476

4 478 483 491 496 499 506 510 497

5 505 510 515 519 523 526 530 504

6 510 511 517 522 531 531 536 526

7 542 539 544 548 561 548 549 542

Table 3: Estimated Intercept, Estimated Slope 1, and Estimated Slope 2


2020

Estimated Intercept

Estimated Slope 1

Estimated Slope 2

Reading

1 180.626 3.153 -0.680

2 202.132 2.535 3.048

3 223.101 1.986 1.270

5 1849.769 12.775 0.366

6 1962.396 9.737 -2.516

7 2035.408 7.166 -8.867

Math

1 306.455 11.610 14.582

2 397.712 10.400 -10.406

3 458.181 4.329 -8.542

4 477.857 5.222 -12.130

5 505.009 4.208 -26.270

6 509.759 4.522 -11.218

7 539.940 3.514 -19.309

16

Table 4. Piecewise Scaled Scores for Each Assessment Month

Reading In reading, students grew consistently from September 2019 through March

2020, as expected. To determine the growth for April and May, we looked at the mean

score and its percentile rank for each month from September through March. Most

months had the same mean percentile rank, which might vary by a point or two. We

took the average of the percentiles across the months and used that percentile to project

what growth would have been in April and May. We also projected the typical learning

loss in the summer of 2020 by using the incoming scores in September of the 2019-2020

school year. This may not completely capture the full learning loss, as some students

may have taken their May assessment at the beginning of the month and school ended

four to six weeks later. Students also began school in August, and therefore there may be

Assessment

Grade as of

September 2020

SEP 2019

OCT 2019

NOV 2019

DEC 2019

JAN 2020

FEB 2020

MAR 2020

SEP 2020

Reading

1 181 184 187 190 193 196 200 199

2 202 205 207 210 212 215 217 220

3 223 225 227 229 231 233 235 236

5 1850 1863 1875 1888 1901 1914 1926 1927

6 1962 1972 1982 1992 2001 2011 2021 2018

7 2035 2043 2050 2057 2064 2071 2078 2070

Math

1 306 318 330 341 353 365 376 391

2 398 408 419 429 439 450 460 450

3 458 463 467 471 475 480 484 476

4 478 483 488 494 499 504 509 497

5 505 509 513 518 522 526 530 504

6 510 514 519 523 528 532 537 526

7 540 543 547 550 554 558 561 542

17

a month or more of instruction that the learning loss method described here does not

fully capture.

The growth and the learning loss that the students experienced varied by grade.

Kindergarten students entering first grade had scores (199) that were between the

average of the previous year’s scores in February (196) and March (200), indicating 3

and a half months of learning loss. Students entering second grade had a mean of 220,

which showed a slight learning gain from the previous March (217), indicating they

grew slightly some over the spring, but their scores were lower than the projected scores

(221), indicating a loss of one month. Students entering third grade had scores (236)

that were higher than March (235), and lower than the projected scores of 238.

Students in the upper grades experienced the pandemic differently than students

in early elementary. Students entering fifth grade had scores in September 2020 (1927)

similar to the average in March (1926), and students entering sixth grade had

September 2020 scores (2018) that were between the averages for February (2011) and

March (2021). Students entering seventh grade had September 2020 scores (2070) that

were similar to those in February (2071).

When comparing September 2019 and September 2020 scores, incoming

students performed lower when compared to similar students in the previous year.

Students entering first or second grade in September 2020 had scores that were 3 points

less than students from the previous cohort, indicating that they came in approximately

2 months behind the previous cohort. Students entering third grade had a mean ISIP

score that was 6 points less than the previous cohort. Students in upper elementary also

had lower scores than students from the previous cohort.

18

Across the grades, the means in September 2020 were approximately equivalent

to their mean scores around March or April in their prior year, with the exceptions

noted above. This indicates that once schools closed in March, students did not have

increased gains, with the possible exception of students entering second grade.

Although some students may have been able to maintain their reading ability

until the end of the school year, the majority of students did not continue to grow in

April and May. The remote learning may have helped them maintain the growth they

experienced through March.

Typical summer learning loss in reading is one month when using the ISIP

reading scores. This study demonstrates that in reading, the COVID-19 school closures

contribute approximately 1 to 3 months’ loss in reading in addition to the typical

summer loss. These results are consistent with many studies on COVID-19 learning loss

as well as summer learning loss in reading (Kuhfeld, Soland, et al., 2020; Kuhfeld,

Tarasawa, et al., 2020; Lewis et al., 2020; Patarapichayatham et al., 2021; Quinn and

Polikoff, 2017). Figures 2 and 3 show the difference between the expected summer

learning loss and the actual COVID-19 learning loss by grade for ISIP ER, and Figure 4

shows the learning loss for students assessed with ISIP AR. These graphs show that the

biggest loss was that student learning did not progress in April and May and that for the

most part, students picked up in September where they left off in March of 2020.

19

Figure 2. Typical summer loss and COVID-19 learning loss in reading: grades 1-3.

170

180

190

200

210

220

230

240

250

SEP OCT NOV DEC JAN FEB MAR APR MAY SEP

ISIP Early Reading

K - 1 K - Spring projection K - Summer loss

1 - 2 1 - Spring projection 1 - Summer loss


1800

1850

1900

1950

2000

2050

2100

2150

SEP OCT NOV DEC JAN FEB MAR APR MAY SEP

ISIP Advanced Reading

4 -5 4 - Spring projection 4 - Summer loss



20

Figure 3. Typical summer loss and COVID-19 learning loss in reading: grades 5-7.

Math

The results for math demonstrate that while students in the early grades had less

learning loss, the learning loss was greater in math than in reading. Students entering

first grade had September 2020 scores (391) that were higher than March (376) but

lower than the projected scores (395). After that, students showed greater learning loss

in math. We note that summer learning losses are greater in math than they are in

reading, so for the remaining grades, we will report the losses in comparison not with

the March scores, but rather with the typical summer learning losses.

Students entering second grade would typically have 2 months of summer

learning loss, and in this research, they had 3 months of learning loss in math. Students

entering third or fourth grade would typically have 3 months of summer loss, and the

learning loss from the pandemic was 5 months. Students entering fifth grade would

have 5 months of loss, and in this research, they have 8 months of learning loss —

almost an entire year. Students entering sixth grade would have 3 months of loss, and in

this research, they had 5 and a half months of learning loss. Figures 5-7 show the

projected gains, typical summer learning loss, and the actual loss in math.

Similar to ISIP ER and ISIP AR students, ISIP Math students showed increased

COVID-19 learning loss when comparing their September 2020 performance scores to

their previous assessment performance scores. They also had mean math scores in

September that were lower than the previous cohort. Students entering first grade had

ISIP Math scores that were 7 points lower than students in the prior year. Students

entering second grade had ISIP Math scores that were 8 points lower, third graders had

21

scores that were 2 points lower, and fourth and fifth graders had scores that were 8 and

6 points lower, respectively. Students entering sixth grade had scores that were 14

points lower than the previous cohort.

Understanding the ISIP Math norms is important to put these results in context.

The norms assume that the learning gains are greater in the fall and winter months, and

then they taper in the spring. They use a parabola method that mirrors the learning

gains expected during the year, and they also assume the greater learning loss over the

summer that was present in the norming data. While the learning losses were greater

from the pandemic closures and greater than expected summer learning losses, it is

typical that students will lose more in math than they will in reading, and that the losses

will be greater in the upper elementary years when the curriculum is more difficult to

learn and easier to forget if the skills are not practiced.

Figure 5. Typical summer loss and COVID-19 learning loss in math: grades 1 and 2.

250

300

350

400

450

500

SEP OCT NOV DEC JAN FEB MAR APR MAY SEP_2021

K-2 ISIP Math

K - 1 K - Spring projection K - Summer loss


22

Figure 6. Typical summer loss and COVID-19 learning loss in math: grades 2 and 3.

Figure 7. Typical summer loss and COVID-19 learning loss in math: grades 4-6.

450

475

500

525


Grades 2-4 ISIP Math



500

510

520

530

540

550


Grades 5-6 ISIP Math



23

Our results are consistent with many studies on COVID-19 learning loss and

summer learning loss in reading and math (Kuhfeld, Soland, et al., 2020; Kuhfeld,

Tarasawa, et al., 2020; Lewis et al., 2020; Patarapichayatham et al., 2021; Quinn &

Polikoff, 2017; Shafer, 2016; Dorn, 2020). In summary, our findings show that in

reading, students’ growth trajectories kept going up until they finished their 2019-2020

school year. Once they headed back to school in September 2020, their scores dropped,

especially in the upper grades. Learning losses were greater in math than they were in

reading, and the students entering fifth grade had steep losses in their math ability as

compared to their previous year’s achievement.

Cohort Analysis

To answer the second research question, we compared the differences among

mean scores in reading and math for students in September 2020 versus the previous

cohort, by grade and SES level at the school, using the same categories as described in

the sample selection process. All differences were statistically significant, and therefore

we calculated effect sizes using Hedges’s g to better see the magnitude of the difference.

Most effect sizes were moderate or small. The results are available in Table 5.

24

Table 5. Cohort comparisons for students ISIP Reading

SES Sept. 2019

Sept. 2020

Difference in Scale Score

Points Effect Size

Difference in Percentile

Points

Grade 1

SES1 198 195 -3 -0.21 -7

SES2 200 197 -3 -0.22 -7

SES3 204 200 -4 -0.28 -9

SES4 208 205 -3 -0.21 -8

All 202 199 -3 -0.19 -7

Grade 2

SES1 218 216 -2 -0.11 -4

SES2 222 218 -4 -0.22 -7

SES3 226 222 -4 -0.25 -7

SES4 230 227 -3 -0.18 -6

All 224 221 -3 -0.16 -6

Grade 3

SES1 232 231 -1 -0.09 -2

SES2 237 235 -2 -0.14 -5

SES3 241 239 -2 -0.13 -5

SES4 246 243 -3 -0.17 -7

All 239 236 -3 -0.12 -7

Grade 4

SES1 1786 1754 -32 -0.18 -7

SES2 1831 1793 -38 -0.21 -6

SES3 1872 1835 -37 -0.22 -9

SES4 1925 1883 -42 -0.25 -9

All 1850 1813 -37 -0.20 -9

25

SES Sept. 2019

Sept. 2020

Difference in Scale Score

Points

Effect Size

Difference in Percentile

Points

Grade 5

SES1 1893 1861 -32 -0.16 -7

SES2 1917 1912 -5 -0.03 -1

SES3 1979 1954 -25 -0.13 -6

SES4 2024 2009 -15 -0.09 -3

All 1955 1930 -25 -0.20 -6

Grade 6

SES1 1950 1965 +15 0.08 +3

SES2 2022 1978 -44 -0.23 -10

SES3 2057 2038 -19 -0.10 -4

SES4 2097 2086 -11 -0.06 -2

All 2030 2021 -9 -0.05 -2

Grade 7

SES1 2005 1980 -25 -0.12 -4

SES2 2034 2043 9 0.04 +2

SES3 2144 2097 -47 -0.24 -10

SES4 2150 2156 6 0.03 +1

All 2088 2072 -16 -0.08 -3

For ISIP Early Reading, the cohort of students in the 2020-2021 school year is

behind the 2019-2020 school year cohort of students in all grades and SES levels with a

few exceptions. In first through third grades, the students have lower scores in

September 2020 in all SES levels. In grade 1, the schools that are SES 3 and SES 4 have

the greatest difference in mean percentile points, and in grade 2, the biggest differences

in mean percentile points are with SES 2 and SES 3 schools. In grade 3, the SES 4

26

schools have the biggest mean differences in percentile points. These results are

depicted in Figure 8.

Students in upper elementary school who took the ISIP Advanced Reading

assessment have a more variable pattern of results, and the differences between SES

levels are more pronounced. These results can be seen in Figure 9. The scores have more

of a sawtooth pattern, where students in fifth grade in SES 1 schools have scores similar

to students in fourth grade in SES 3 schools, and fifth grade students in SES 2 schools

have scores similar to fourth grade students in SES 4 schools. A similar pattern is

evident in sixth and seventh grades. The scores in September 2019 are higher than in

September 2020, with the exception of students in sixth grade in SES 1 schools, seventh

grade in SES 2 schools, and SES 4 schools in sixth and seventh grade, where scores were

comparable to the prior year.

Figure 8. Differences in cohort performance by school SES in reading, grades 1-3.

195

205

215

225

235

245

SES 1 SES 2 SES 3 SES 4 SES 1 SES 2 SES 3 SES 4 SES 1 SES 2 SES 3 SES 4

Grade 1 Grade 1 Grade 1 Grade 1 Grade 2 Grade 2 Grade 2 Grade 2 Grade 3 Grade 3 Grade 3 Grade 3

ISIP Early Reading

SEP 2019 SEP 2020

27


For ISIP Math, the cohort of students in the 2020-2021 school year scored lower

than students in the 2019-2020 school year cohort of students in all grades and SES

levels, and the differences in the mean percentile points has a wide range. These results

are available in Table 6. Students in first grade in SES 1 schools have scores that are

similar to each other in September of 2019 and September 2020. SES 2 schools have the

biggest gap of 9 scale points, which is equivalent to 7 percentile points. Students in SES

3 and SES 4 schools also had lower scores in September 2020 than in September 2019.

In second grade, students in SES 4 schools had the biggest drop in ISIP Math scores,

particularly in the number of percentile points.

1750

1800

1850

1900

1950

2000

2050

2100

2150

SES 1 SES 2 SES 3 SES 4 SES 1 SES 2 SES 3 SES 4 SES 1 SES 2 SES 3 SES 4 SES 1 SES 2 SES 3 SES 4

Grade4

Grade4

Grade4

Grade4

Grade5

Grade5

Grade5

Grade5

Grade6

Grade6

Grade6

Grade6

Grade7

Grade7

Grade7

Grade7

ISIP Advanced Reading

SEP 2019 SEP 2020

28

Table 6. Cohort comparisons for students, ISIP Math SES Sept. 2019 Sept. 2020 Difference

Scale Score Points

Effect Size Difference Percentile

Points

Grade 1

SES1 384 383 -1 -0.02 -1

SES2 394 385 -9 -0.20 -7

SES3 398 394 -4 -0.09 -3

SES4 406 401 -5 -0.16 -4

All 394 391 -3 -0.06 -2

Grade 2

SES1 446 444 -2 -0.06 -1

SES2 453 448 -5 -0.16 -5

SES3 455 452 -3 -0.09 -4

SES4 464 457 -7 -0.20 -8

All 454 450 -4 -0.12 -4

Grade 3

SES1 471 468 -3 -0.10 -3

SES2 475 473 -2 -0.07 -2

SES3 481 479 -2 -0.07 -3

SES4 486 484 -2 -0.07 -2

All 478 476 -2 -0.07 -2

Grade 4

SES1 497 488 -9 -0.29 -10

SES2 502 494 -8 -0.26 -9

SES3 507 501 -6 -0.18 -7

SES4 515 508 -7 -0.24 -8

All 505 497 -8 -0.25 -9

29

SES Sept. 2019 Sept. 2020 Difference

Scale Score Points

Effect Size Difference Percentile

Points

Grade 5

SES1 499 494 -5 -0.16 -4

SES2 509 502 -7 -0.21 -7

SES3 506 508 2 0.06 -2

SES4 522 515 -7 -0.20 -6

All 510 504 -6 -0.17 -6

Grade 6

SES1 527 513 -14 -0.36 -11

SES2 538 523 -15 -0.37 -16

SES3 542 526 -16 -0.39 -17

SES4 556 539 -17 -0.49 -15

All 542 526 -16 -0.40 -17

Grade 7

SES1 518 527 9 0.26 -8

SES2 535 538 3 0.07 -3

SES3 550 545 -5 a-0.12 -4

SES4 568 555 -13 -0.30 -10

All 545 542 -3 -0.07 -3

In third grade the drops in mean scores and percentile points were comparable

across all types of schools. In fourth grade, the biggest drop in mean scores and

percentile points was in SES 1 and SES 2 schools. In contrast, students in fifth grade

had bigger drops in SES 2 and SES 4 schools, and slight gains in SES 3 schools.

Students entering sixth grade showed the largest drop in scores across all grades

and SES, and the drops in mean scores and difference in percentile ranks were large,

with effect sizes between −.49 and −.39. These results show that students in all types of

schools came to school in September 2020 with scores that were substantially lower

30

than their September 2019 counterparts, indicating that teachers would need to spend

time catching students up on what they forgot since school closures were instituted in

March 2020. Students entering seventh grade had a mixed pattern of results, with

students in SES 1 and SES 2 schools having higher scores, and students in SES 3 and

SES 4 schools having lower scores.

These results are not surprising given the results from the piecewise models that

showed these students had the greatest learning loss. Figures 10, 11, and 12 depict these

results. With the exception of students entering seventh grade, the gaps between

September 2019 and September 2020 scores were greater in the upper grades than in

the lower grades.

Figure 10. Differences in cohort performance by school SES in reading, grade 1.

370

375

380

385

390

395

400

405

410

SES1 SES2 SES3 SES4

Grade 1 Grade 1 Grade 1 Grade 1

ISIP Early Math Grade 1

September 2019 September 2020

31

Figure 11. Differences in cohort performance by school SES in reading, grades 2 and 3.


440

450

460

470

480

490

SES4 SES1 SES2 SES3 SES4 SES1 SES2

Grade 1 Grade 2 Grade 2 Grade 2 Grade 2 Grade 3 Grade 3

ISIP Math Grades 2 - 3


480

490

500

510

520

530

540

550

560

570

580

SES1 SES2 SES3 SES4 SES1 SES2 SES3 SES4 SES1 SES2 SES3 SES4 SES1 SES2 SES3 SES4

Grade4

Grade4

Grade4

Grade4

Grade5

Grade5

Grade5

Grade5

Grade6

Grade6

Grade6

Grade6

Grade7

Grade7

Grade7

Grade7

ISIP Math Grades 4-7


32

Discussion

Our findings in both reading and math are consistent with previous research that

found differences in learning gains and losses in reading and math during the COVID-

19 pandemic (Kuhfeld, Soland, et al., 2020; Dorn, 2020; Huff, 2020; Patarapichayatham

et al., 2021). Students made learning gains through March in the 2019-2020 school

year, but they did not make the typical gains in April and May. In reading, students held

on to their learning gains through March, but for the most part missed out on the

learning gains they would have achieved in April and May had schools been open. The

results varied by the SES level of the school, and students in higher-poverty SES 1

schools did not have learning losses that were substantially greater than other students.

Our research is also similar to Quinn & Polikoff (2017) which found that students’

achievement scores declined over the summer by one month’s worth of school-year

learning, and students in higher grades had more loss than the lower grades.

The situation in math is somewhat different. Students entering first and second

grade appear to have held on to their gains through March from the previous year, and

first grade students gained a bit during the pandemic. However, as the students’ grade

increased — and thus the complexity of the math concepts increased — they had a

harder time maintaining their gains. While students in the elementary grades typically

lose more months in math than they do in reading when looking at typical summer loss,

the losses were more substantial due to the school closures from the pandemic. Losses

were the greatest for students in sixth grade across all SES levels, indicating that

students entering sixth grade came in further behind with more substantial learning

losses than other grades.

33

Research from Lewis et al. (2020) showed that students in lower SES schools who

engaged in online learning for reading narrowed the achievement gap compared to

students in higher SES schools. It is likely that online learning helped to stave off some

of the anticipated learning losses from the pandemic closures for those who were able to

obtain the needed resources to do so.

The patterns of learning loss follow what we know about summer loss in a typical

school year. Shafer (2016) found that students had greater summer learning loss in

reading than they did in math. The loss from the pandemic was greater in math than it

was in reading. This may be because there are greater opportunities for a student to use

their reading skills when they are not in the classroom, especially if reading is part of a

family’s daily life. Many parents will read to their children and ask questions about what

they read, and it is likely that fewer parents help their children practice math skills when

school is out. This may be especially true with older students, since parents may have

forgotten parts of the math curriculum. As a result, when the school year ends, students

may have very few opportunities to engage in any type of mathematical thinking.

Recommendations

Our results indicate that students fell behind in their learning in both reading and

math, with greater learning loss in math. This will put a burden on teachers and parents

to help students get caught up so that there are not lifelong repercussions for the

students. First, while remote learning may have helped stave off some of the losses when

schools closed in the spring (Lewis et al., 2020), this research indicates that it was not as

effective as in-person learning. Students did not make learning gains in April and May

like they would have in a normal year. In reading, teachers will need to review basic

34

concepts and teach some of the more advanced concepts from the previous year to help

students catch up. Parents may also want to emphasize literacy in the home, such as by

reading to their children and checking out books from the library if possible. Many

libraries have apps that patrons can download and check out books to read on a device,

and schools across the country have provided devices to students during the pandemic

so that students can participate in remote learning. Educational technology can also

help students with their learning growth, including lower-achieving students (Sutter, et

al., 2019).

The situation will be more urgent in math, especially in the upper grades, when

the concepts are more difficult and students may have a greater tendency to forget the

concepts of decimals, fractions, number lines, graphing, and other more complex and

abstract concepts. Teachers across grades may need to strategize on which concepts to

review more extensively and how to keep students engaged in math concepts. In

addition, teachers should reach out to parents to help them incorporate math skills at

home so that students have time to practice their math skills as well as their reading

skills.

Long-Term Implications of COVID-19 Learning Loss

Unknown is the longer-term impact of COVID-19 learning loss since the

pandemic has continued through the 2020-2021 school year. Some states and districts

have returned to in-person learning, and others have remained closed, with students

only participating remotely. Still others have maintained a hybrid model where some

students are remote and others are attending in person on different days, so as to

minimize class sizes and exposure. Likely there will be long-term impacts for students,

35

especially if states and districts do not implement recovery learning strategies for their

students such as expanding summer school.

Learning losses from five to nine months could occur by June 2021 (Dorn et al.,

2020). In research conducted by McKinsey & Company, they recommended that

districts work with teachers to make remote learning engaging and structured and

eliminate the digital divide. Acceleration plans using evidence-based programs and

formative assessments to identify students who are falling behind may help students

catch up academically (Dorn et al., 2020). Another approach may be to expand learning

time during the day and consider in-person summer programs after the pandemic has

waned.

The question of whether to implement state-level summative assessment in

spring of 2021 has been controversial. Having a year-end summative assessment may be

helpful for states and districts to determine the extent, if any, of the learning loss and

whether there are differences by type of instruction received in the 2020-20201 school

year. These results could be helpful for instructional planning for the following school

year in the event the districts may need to implement plans to help students catch up.

Some high-accountability states such as Texas have opted to go forward with the

summative assessment but not use it to grade teachers, schools, and districts on student

performance. Others, such as California have opted to replace the summative

assessment with a shorter version, and others have sought waivers to cancel all

standardized testing. Georgia and Colorado will reduce the number of summative

assessments, and Massachusetts required districts to develop academic recovery plans

36

(Olneck-Brown, 2021). The results from these tests, if administered, should be

interpreted with caution, especially if the assessments are administered at home.

Limitations and Future Directions

This study shows the impact of the COVID-19 pandemic’s school closures on

students’ achievement in reading and math with a nationally representative sample of

students in the US that use Istation and the ISIP assessments. This study focused on

students who took the assessment in school in September 2020.

Our results indicate that students lost their math ability more rapidly, and while

remote learning may have helped students retain their progress in reading, it was not

the same for math. More research needs to investigate how remote learning is different

in math than in reading and what can be done to help students maintain their math

ability as well as their reading ability.

37

References

Campbell, L. O., Sutter, C. C., Lambie, G. W. (2019). An investigation of the summer learning effect

on fourth grade students’ reading scores. Reading Psychology, 40, 5, 465-490. Retrieved

February 1, 2021 from

https://www.tandfonline.com/doi/full/10.1080/02702711.2019.1629516

Dorn, E., Hancock, B, Sarakatsannis, J. and Viruleg, E. (2020). COVID-19 and learning loss-

disparities grow and students need help. Retrieved February 20, 2020 from

https://www.mckinsey.com/industries/public-and-social-sector/our-insights/covid-19-and-

learning-loss-disparities-grow-and-students-need-help.

Huff, K. (2020). National data quantifies impact of COVID learning loss; raises questions about at-

home testing. Curriculum Associates. Retrieved from

https://www.curriculumassociates.com/-/media/mainsite/files/i-ready/caimpact-of-covid-

learning-loss-fall-2020.pdf

Kamata, A., Nese, J. F. T., Patarapichayatham, C., & Lai, C. F. (2013). Modeling nonlinear growth

with three data points: Illustration with benchmarking data. Assessment for Effective

Intervention, 38, 2 (p 105–116).

Kuhfeld, M., Soland, J., Tarasawa, B., Johnson, A, Ruzek, E. & Liu, J. (2020). Projecting the potential

impact of COVID-19 school closure on academic achievement. Educational Researcher, 48, 8

(p 549–565).

Kuhfeld, M., Tarasawa, B., Johnson, A, Ruzek, E. & Lewis, K. (2020). Learning during COVID-19:

initial findings on students’ reading and math achievement and growth. Portland, OR:

NWEA.

Lewis, S., Locke, V. N., & Patarapichayatham, C. (2020). Student engagement in online learning

during COVID school closures predicts lower learning loss in Fall 2020. Dallas, TX: Istation.

https://www.tandfonline.com/doi/full/10.1080/02702711.2019.1629516

https://www.mckinsey.com/industries/public-and-social-sector/our-insights/covid-19-and-learning-loss-disparities-grow-and-students-need-help

https://www.mckinsey.com/industries/public-and-social-sector/our-insights/covid-19-and-learning-loss-disparities-grow-and-students-need-help

https://www.curriculumassociates.com/-/media/mainsite/files/i-ready/caimpact-of-covid-learning-loss-fall-2020.pdf

https://www.curriculumassociates.com/-/media/mainsite/files/i-ready/caimpact-of-covid-learning-loss-fall-2020.pdf

38

Malkus, N., Christensen, C., & Schurz, J. (2020). (Rep.). American Enterprise Institute.

doi:10.2307/resrep25360

Mathes, P., Torgesen, J. & Herron, J. (2011). Technical Manual: Istation’s Indicators of Progress,

Early Reading, Version 4. Istation: Dallas, TX.

Muthén, L.K. and Muthén, B.O. (1998-2017). Mplus User’s Guide. Eighth Edition. Los Angeles, CA:

Muthén & Muthén.

Olneck-Brown, B. (2021). Public Education’s Response to the Coronavirus(COVID-19) Pandemic.

Retrieved February 20, 2021 from https://www.ncsl.org/research/education/public-

education-response-to-coronavirus-covid-19.aspx.

Patarapichayatham, C., Fahle, W., & Roden, T.R. (2013). ISIP Reading versus STAAR reading: The

predictability study. Dallas, TX: Istation.

Patarapichayatham, C., & Locke, V. N. (2019). An evaluation of Istation curriculum on student

reading growth: quasi-experimental study using propensity score analysis. Dallas, TX:

Istation.

Patarapichayatham, C., Locke, V. N., & Lewis, S. (2021). Covid Learning Loss in Texas. Dallas, TX:

Istation.

Quinn, D., & Polikoff, M. (2017). Summer learning loss: What is it, and what can we do about it.

Brookings Institution. Retrieved from https://www.brookings.edu/research/summer-

learning-loss-what-is-it-and-what-can-we-do-about-it.

Rambo-Hernandez, K. E., & McCoach, D. B. (2015). High-Achieving and Average Students’ Reading

Growth: Contrasting School and Summer Trajectories. Journal of Educational

Research, 108(2), 112–129. https://doi.org/10.1080/00220671.2013.850398

Shafer, L. (2016). Summer math loss: why kids lose math knowledge, and how families can work to

counteract it. Retrieved from https://www.gse.harvard.edu/news/uk/16/06/summer-math-

loss.

https://www.ncsl.org/research/education/public-education-response-to-coronavirus-covid-19.aspx

https://www.ncsl.org/research/education/public-education-response-to-coronavirus-covid-19.aspx

https://www.brookings.edu/research/summer-learning-loss-what-is-it-and-what-can-we-do-about-it

https://www.brookings.edu/research/summer-learning-loss-what-is-it-and-what-can-we-do-about-it

https://www.gse.harvard.edu/news/uk/16/06/summer-math-loss

https://www.gse.harvard.edu/news/uk/16/06/summer-math-loss

39

Sutter, C., Campbell, L., & Lambie, G. (2019). Computer-adaptive reading to improve reading

achievement among third-grade students at risk for reading failure. Journal of At-Risk Issues,

22, 2, 31-38.

Stanford University, Center for Research on Education Outcomes (2020). Estimates of Learning Loss

in the 2019-2020 School Year. Retrieved October 30, 2020 from

https://credo.stanford.edu/sites/g/files/sbiybj6481/f/short_brief_on_learning_loss_final_

v.3.pdf.

Swaby, A. (2020). As the school year begins online, thousands of Texas students are being left out of

virtual learning. Texas Tribune, September 1, 2020. Retrieved January 30, 2020 from

https://www.texastribune.org/2020/09/01/texas-schools-reopening-virtual-learning/

https://credo.stanford.edu/sites/g/files/sbiybj6481/f/short_brief_on_learning_loss_final_v.3.pdf

https://credo.stanford.edu/sites/g/files/sbiybj6481/f/short_brief_on_learning_loss_final_v.3.pdf

https://www.texastribune.org/2020/09/01/texas-schools-reopening-virtual-learning/

Documents

Learning Loss in Reading and Math in U.S. Schools Due to