Idaho Literacy Intervention Program Evaluation 2020-21

Report Authors:

Matthew May, PhD, Senior Research Associate

Mcallister Hall, Research Associate

This report was prepared by Idaho Policy Institute at Boise State University and commissioned by the Idaho State Board of Education.

This evaluation, of the Idaho Literacy Intervention Program (Program) annually mandated by the Idaho Legislature, considers (a) program design, (b) use of funds, including the funding amounts, (c) local education agencies (LEAs) that have utilized all-day kindergarten (all-day K), (d) program effectiveness, and (e) any other relevant matters. For the fourth year, Idaho Policy Institute (IPI) conducted the evaluation.

To complete this report, IPI received relevant financial, performance, and enrollment data from Idaho’s Office of the State Board of Education and Idaho State Department of Education. This includes student-level Idaho Reading Indicator (IRI) scores, demographic data, all-day K enrollment data, and LEA literacy intervention expenditures.

Key Findings

Program Design

The Program is designed to help students who need support as identified by the fall administration of the IRI test. Teachers agree the IRI test accurately identifies students who need support in achieving literacy proficiency. Spending in approved funding categories directly impacts students.

Use of Funds

LEAs continue to use a majority of funds each year to hire more personnel or increase pay of current personnel. Most administrators indicate if their LEA received more literacy funds, they would increase personnel spending. Current reporting standards do not require schools to report how much is spent on all-day K. Program spending toward all-day K is likely reflected in the personnel category.

All-day Kindergarten

About 58% of schools offer some form of all-day K. In a 2020 IPI survey of school administrators, 37% of LEAs indicated using at least some of their state literacy funds toward a version of a free all-day K program. Some schools funding all-day K programs may not be able to serve all kindergarten students with a full day of instruction.

Program Effectiveness

The impact of the COVID-19 pandemic is seen in spring 2021 IRI scores. All grades and demographic subgroups saw a decrease in proficiency from spring 2019 scores. Grade 1 students saw the biggest drop, likely due to learning loss during the final months of their kindergarten school year. Students who are economically disadvantaged, students with disabilities, and students learning English continue to perform lower than their counterparts. There is an opportunity to increase student proficiency by ensuring that schools with higher percentages of these groups have adequate resources to meet the needs of these students. Students in city schools and students with disabilities saw the smallest gap between 2019 and 2021 proficiency rates. Further study is needed to understand any strategies used to mitigate this gap.

In 1999, the National Reading Panel was convened by the United States Congress. The 14-member panel reviewed more than 100,000 studies to determine evidence-based best practices for teaching reading. The findings prompted broad scale incorporation of policies across the states.

That same year, Idaho passed the Idaho Comprehensive Literacy Act. The associated legislation sought to mandate regular assessments of kindergarten to third grade (K-3) students, make school-level assessment data available to stakeholders, provide intervention for students not meeting grade-level reading proficiency, and implement associated professional development for teachers and administrators. The legislation experienced substantive updates in response to the outcomes of the 2015 Comprehensive Literacy Plan. One of the updates, implemented in 2016 by legislative statute, established the new Literacy Intervention Program (Program), the focus of this report.

In 2018, Idaho implemented a new computer-adaptive Idaho Reading Indicator (IRI) assessment statewide that measures five foundational skills of literacy, including alphabetic knowledge, phonemic awareness, vocabulary, spelling and comprehension. The following year (2019) the Idaho Legislature doubled Program funding, making it approximately $26 million annually.

In 2020, the state completed another update to the Comprehensive Literacy Plan, which is mandated by Idaho Code every five years. The following legislative session (2021), the Idaho Legislature passed the Idaho Literacy Achievement and Accountability Act, which addressed issues raised by the plan review and consolidated existing sections of Idaho Code dealing with literacy intervention into a new section (Title 33, Chapter 18).

METHODS

In 2021, the Idaho Legislature again authorized an independent, external evaluation of the state’s literacy intervention program. As mandated, this evaluation considers (a) program design, (b) use of funds, including the funding amounts, (c) local education agencies (LEAs) that have utilized all-day kindergarten (all-day K), (d) program effectiveness, and (e) any other relevant matters.

For this report, Idaho Policy Institute (IPI) received relevant financial, performance, and enrollment data from Idaho’s Office of the State Board of Education (OSBE) and Idaho State Department of Education (SDE). IPI also used school locale data from the National Center for Education Statistics (NCES). This led to three main sets of data:

• Student-level IRI scores and demographic data
• All-day K enrollment data
• LEA Literacy Intervention expenditure data

Information is reported at the state level and by subcategories within each dataset. IPI also drew from its analysis in prior evaluations, including surveys administered in 2020 to K-3 teachers and LEA administrators (Appendix C). Further details about the methodology can be reviewed in Appendix B.

SCOPE AND LIMITATIONS

This report is IPI’s fourth evaluation of the Program. The initial evaluation, completed in January 2019, relied on 2016-17 and 2017-18 data from what is known as the legacy IRI. In the 2018-19 school year, all Idaho schools started using the IRI by Istation testing instrument. The second report (completed in January 2020) includes data from up through the 2018-19 school year. The report did not compare scores across years because the IRI by Istation and the legacy IRI testing instrument measure different aspects of literacy. IPI’s third report (completed January 2021) focused on qualitative data collected from teachers and administrators across the state, as spring 2020 IRI performance data was unavailable because of the COVID-19 pandemic.

Similar to the first two evaluations, this evaluation compares IRI student performance data across demographic categories. This report only uses scores from the IRI by Istation (2018-19 and 2020-21 school years). A comprehensive comparison of student performance will require at least four consecutive years of data.

The IRI was not required in spring 2020 because of the COVID-19 pandemic. As such, spring 2020 data is omitted from data analysis throughout the report as few K-3 students in Idaho took the assessment. The COVID-19 pandemic caused students to experience precarious learning situations in spring 2020 which continued into the 2020-21 and 2021-22 school years.

In 2020-21, schools used varying instructional delivery methods, including in-person, online, and hybrid approaches. In both the 2020-21 and 2021-22 school years, some school districts closed for days or weeks due to outbreaks or staffing shortages. It is unclear how long the impacts of these precarious learning situations may be reflected in student performance and growth (See Appendix D).

The legislation mandating this evaluation requires a summary of LEAs using Program funds on all-day K. Currently, budget and expense reports do not require schools to indicate whether any funds are dedicated to all-day K. Under the current categories on expense reports, all-day K spending is likely reflected in the personnel section, but not all personnel spending is likely related to all-day K. If LEAs are dedicating Program funds for all-day K programs, it is likely the funds are not covering the whole cost of all-day K.

PROGRAM DESIGN

The Program design evaluation echoes previous reports, as the legislative design of the Program has not changed. The Program is well designed. One of the most notable Program strengths is the focus on students. Students qualify for support based on their fall IRI score, and if identified, have an individual reading plan created for them to receive 30 or 60 hours of intervention services during the school year. Districts receive funding based on a three-year rolling average of qualifying students and use those funds as outlined in their district-created Literacy Intervention Program Plan. Each LEA chooses how to distribute these funds and as a result, students receive targeted instruction and resources to maximize their potential for growth.

Based on information from the 2020 surveys, the impact of spending in approved funding categories is evident in student learning. Funding is being used for items that can generally be traced directly to the students. Students receive intervention supports using curriculum materials and purchased technology. Increased personnel allow for small group and one-on-one instruction to individualize student learning. Professional development provides teachers with literacy-focused teaching strategies they use regularly for interventions and core instruction.

The Program is designed to foster teacher autonomy. The literacy standards are clearly defined while allowing teachers flexibility in how they choose to approach literacy instruction. This flexibility improves results and benefits students, as teachers can research and apply best practices for specific students and literacy standards. Teachers in the 2020 survey report that autonomy also allows for instructional changes when current strategies are not improving literacy.

The Program’s funding formula focuses on the number of students scoring basic and below basic on the IRI by Istation. The 2020 teacher and administrator surveys asked respondents their confidence in the Istation assessment’s ability to identify student performance status. Most teachers (80%) are moderately or very confident in the Istation performance indicators.

USE OF FUNDS

EXPENSE CATEGORIES

LEAs are required to submit an expense report of Program expenditures at the end of each academic year. Expenses are broken down into six major categories: personnel, curriculum, (student) transportation, professional development, technology, and other. There is not a state definition of each category and where expenses are accounted for is at the discretion of each LEA. Not all LEAs submit expenditure reports, as such, the number of LEAs completing expenditure reports vary by year. IPI analyzed the proportion of annual LEA expenditures in each funding category. The proportion of expenditures per category spent by all LEAs in years 3-5 of the Program is summarized in Table 1.

TABLE 1: PROPORTION OF LEAS PROGRAM EXPENSE REPORT BUDGET CATEGORIES

Personnel expenses increased by nearly 10 points after FY 2019, accounting for over three-quarters of all literacy program expenditures each following year. If LEAs use Program funding for all-day K purposes, it would most likely be reflected in personnel spending. Expanding kindergarten may require schools to hire additional kindergarten staff or extend full-time positions to previously budgeted part-time staff. In the 2020 administrator survey, 89% of respondents indicated they would dedicate literacy funds toward paying personnel if their funding amount increased.

Curriculum costs were relatively stable in years 3-5 of the Program, averaging 12.8%. Curriculum spending is likely greater in years 1-2 as LEAs may have purchased new curriculum to implement the Program but have not made large updates since. Ongoing costs may reflect annual licenses or updates for virtual learning programs.

Transportation expenses continues to account for the smallest proportion of expenditures. Per state law, transportation funding is capped at $100 per student, which could be a factor keeping this expense low.

The proportion of professional development (PD) expenses has nearly halved since FY 2019. That said, PD opportunities might have been less expensive during the FY 2021 school year due to the increased availability of online opportunities. Additionally, dedicated PD funding opportunities are available in both statewide and LEA budgets, which LEAs may opt to use instead of literacy funds in order to direct Program funding to other areas. Regardless of funding stream, teachers in the 2020 survey indicated value in being offered literacy-focused PD, with 88% using strategies learned through such opportunities regularly in the classroom.

ALL-DAY KINDERGARTEN

Kindergarten is not required by state law in Idaho. The current state funding formula provides districts with funds for students to attend kindergarten part-time only. As a result, schools have found creative ways to implement part-time kindergarten, particularly for students who need additional support. This may include students receiving a half-day of instruction with a teacher and then additional time with a paraeducator. Though these students may spend a full day at the school, they are not receiving all-day K instruction. The same applies to students who attend school all day with a teacher but only on alternating days of the week. These students are classified as part-time kindergarten students. All-day K students are students who attend a full day of instruction with a teacher four to five days per week, ultimately completing hours equivalent to other elementary grade students.

Some schools find the benefit of all-day K valuable enough to dedicate additional funds to provide full-day instruction to their students. This is typically done using a combination of funds, which may include Program funds. In the 2020 survey, 37% of LEAs indicated using at least some of their state literacy funds toward a version of a free all-day K program. Table 2 enumerates schools who offer all-day K to all students, only part-time K to all students, or a combination.

TABLE 2: KINDERGARTEN OFFERINGS BY SCHOOLS

* Past data does not account for schools who offer a unique combination of kindergarten opportunities

The combination category represents schools funding all-day K programs who may not be able to serve all kindergarten students with a full day of instruction. In these schools, all-day K is usually prioritized to students based on need, sometimes using early IRI testing to determine which students qualify for the program. However, some LEAs offer all-day or full-time K based on parent preference and fee payment which is against state law.

PROGRAM EFFECTIVENESS

The IRI from Istation, first implemented statewide in the 2018-19 school year, serves as the earliest data that is directly comparable to present day. Both changing testing procedures and the COVID-19 pandemic have impacted the number of directly comparable years available. In the 2019-20 school year, the spring IRI assessment was not required and was not universally administered because of the pandemic.

Teachers in the 2020-21 school year attempted to mitigate the impact of ending the prior school year early. In the 2020 survey completed for last year’s evaluation, some reported needing to cover much of the previous grade’s content with the new grade standards. Additionally, the 2020-21 school year experienced its own instructional disruption, as some schools implemented online or hybrid learning.

Students entering kindergarten in the first year of the Istation instrument implementation (2018-19) will finish third grade in spring 2022. The pandemic’s outsized impact on their learning in first and second grades makes them a unique cohort and not ideal to evaluate overall Program effectiveness.

That said, it remains useful to examine literacy scores over time to see if larger trends in the data observed in prior years’ reports are still present in the current year. Additionally, identifying where the COVID-19 pandemic impacted literacy is a critical step in mitigating those impacts.

IPI’s analysis is limited to students who took both fall and spring IRI assessments in a particular academic year. Table 3 provides a descriptive overview of the students who took the IRI assessment over the last three years of the Program, which serves as the basis of this evaluation.

TABLE 3: DESCRIPTIVE STATISTICS OF STUDENTS WHO TOOK IRI ASSESSMENT

Approximately 3,000 fewer students took both fall and spring IRI assessments in 2020-21 compared to pre-pandemic levels in 2018-19. The pandemic may have caused more parents to shift their children to homeschooling. But for missing students among vulnerable student populations, decreased enrollment could signify a year of lost learning. As a result, proficiency levels among vulnerable subgroups may not represent the full impact of the pandemic on learning loss. However, without information on which students exited the system and why, it is only possible to speculate.

IRI PROFICIENCY

Each fall and spring, students in grades K-3 take the IRI. The intention of the IRI is to help instructors identify students who need additional support achieving grade-level reading proficiency, not to evaluate students or their instructors. Generally, fall scores are lower than spring scores. This may be due to younger students adjusting to the format of standardized testing and older students experiencing some regression during a summer without instruction. Those who do not score proficient on the fall exam are required to receive additional instructional hours within the school year (30 hours if scoring basic, 60 hours if below basic).

TABLE 4: LITERACY PROFICIENCY BY GRADE (SPRING IRI)

Table 4 summarizes spring IRI scores statewide by grade level for 2018-19 and 2020-21. While proficiency levels in 2018-19 show a gradual increase that continues into second grade before plateauing in third grade, results from 2020-21 indicate proficiency is down across all grade levels, likely because of the impacts of the COVID-19 pandemic. The most substantial decline from 2018-19 levels is in first grade (6.9pp), where not only are proficiency levels lower than pre-pandemic levels, but also lower than kindergarten levels. Second grade proficiency also saw a substantial decline (5.9pp), while kindergarten (1.5pp) and third grade (3.2pp) declines were less pronounced.

There are a few reasons that may explain this. The kindergarten experience of 2020-21 first grade students was incomplete, as schools closed to comply with COVID-19 protocols. The end of kindergarten is valuable to reinforce and retain learning from the year. As a result, first grade teachers may have allocated extraordinary instructional time reviewing kindergarten concepts. Time spent reteaching these concepts, combined with teaching an already robust first grade curriculum and shifts in instructional delivery, may help explain the outsized decrease in first grade proficiency (see Appendix D).

The same pandemic disruptions experienced by kindergartners advancing to first grade were likely felt by first graders advancing to second grade. With an abbreviated first grade experience in 2019-20 due to the pandemic, first grade literacy concepts were likely not covered, requiring second grade teachers to dedicate significant time to a review of first grade concepts.

LOCALE

Urban and rural LEAs may have different access to resources and different student populations that impact proficiency levels. The NCES indicator of school locale (see Appendix A) was used to create categories for comparison at the school level. In addition to the NCES locale categories, IPI added a virtual category to classify statewide virtual charter schools. These virtual schools use virtual-specific rather than location-specific resources to serve students across the state. In this context, virtual represents a school designed to be attended virtually and serves students statewide—not LEA-specific virtual schools or in-person schools that shifted to online or hybrid instruction in response to the COVID-19 pandemic. Table 5 provides a breakdown of the number of students and number of schools in each locale category.

TABLE 5: COUNT OF SCHOOLS & STUDENTS BY SCHOOL LOCALE (SPRING)

Note: Locale analysis is only possible for schools for which NCES locale data is available. NCES data is not available for more recent schools. As a result, 501 students in 2018-19 and 1,801 students in 2020-21 are not included.

The distribution of students is balanced across physical locales. While the number of students attending city schools declined in 2020-21, students in virtual schools more than doubled. Table 6 summarizes proficiency levels by school locale for 2018-19 and 2020-21.

TABLE 6: LITERACY PROFICIENCY BY SCHOOL LOCALE (SPRING IRI)

In 2020-21, students in city schools had the highest levels of proficiency. By contrast, in 2018-19 students in suburban schools performed best across all grade levels.
Students in suburban, town, and rural schools saw the biggest post-pandemic gaps. First grade was the only grade level to see declining proficiency across all locales, with substantial declines among suburban (-9.6pp), rural (-7.1pp), and town (-6.0pp) students.
While students in virtual schools show the lowest levels of proficiency across all grade levels in each year (except for third grade in 2020-21), it is notable that kindergarten, second grade, and third grade all saw increases over pre-pandemic proficiency levels—only first grade exhibited a decline. This improvement may be related to increased enrollment of populations not usually represented in virtual schools.

RACIAL AND ETHNIC DIVERSITY

Race and ethnicity can be associated with academic performance, especially in students learning a second language. According to the 2020 U.S. Census, Idaho’s population is 82% white, which suggests most Idaho schools will have a high percentage of white students.
As a result, IPI created a relative measure for Idaho schools using the racial/ethnic makeup of its K-3 students to calculate diversity.
Schools with a student body that is over 90% white are classified as low diversity, 85-90% white are medium diversity, 75-84% are high diversity, and less than 75% white are very high diversity. Table 7 provides a breakdown of total number of students and total number of schools included in each diversity category.

Table 7: Count of Schools & Students by School Diversity (Spring IRI)

Table 8: Literacy Proficiency by School Diversity (Spring IRI)

Similar to 2018-19, 2020-21 scores (Table 8) indicate that schools with very high racial/ethnic diversity performed substantially below all other diversity classifications across all grade levels, including kindergarten. More diverse schools may have a higher concentration of non-white students learning English as a second language, which can substantially affect proficiency rates. However, high diversity schools consistently rank among the two highest in proficiency in 2020-21, suggesting that greater diversity does not automatically mean lower test scores.

While schools of all diversity levels saw declines in proficiency relative to their scores in 2018-19, schools with very high diversity saw the smallest decline in three of four grade levels: kindergarten (-0.5pp), first grade (-5.3pp), and third grade (-2.1pp). By comparison, schools with medium levels of diversity saw the largest declines across all grade levels: kindergarten (-2.6pp), first grade (-10.1pp), second grade (-8.8pp), and third grade (-5.6pp).\

ENGLISH LEARNERS (EL)

The IRI assesses students’ literacy proficiency in English. Consequently, students for whom English is not their first language have lower levels of proficiency. Idaho schools identify these students through a system with eleven designations for English Learners (EL). For ease of analysis, these classifications are collapsed into two designations: EL students (those in the program or have exited within the past two years) and non-EL students (those now fluent, who have exited three or more years ago, screened out, or not applicable). Table 9 summarizes the results from 2018-19 and 2020-21.

Table 9: Literacy Proficiency by EL Status (Spring IRI)

The gap between EL and non-EL students in both years is more pronounced in kindergarten and shrinks in subsequent grades, but there remains a 22-24pp gap in third grade rates. 2020-21 results indicate the gap between these groups grew by 1-3pp each grade level from pre-pandemic results. While proficiency among first graders in 2020-21 fell relative to kindergarten among non-EL students, EL students’ proficiency increased over the prior grade level (5.3pp).

The gap between EL and non-EL students indicates the difficulties of learning literacy in a new language. EL students may also need different levels of intervention based on their previous language skills. Improving programs and interventions that support underperforming EL students or allocating more resources to schools with large EL populations could have an overall positive impact on IRI proficiency.

STUDENTS WITH PRIOR LEARNING ACCOMMODATIONS

Some students have disabilities that necessitate an Individualized Education Plan (IEP) to support their learning. Table 10 summarizes IRI proficiency levels according to disability status.

Table 10: Literacy Proficiency by Disability Status (Spring IRI)

Results for 2018-19 and 2020-21 show students with disabilities perform below students without disabilities, with the margin between the two growing wider with each successive grade level, starting at an approximate 30pp gap in kindergarten and growing to an approximate 45pp gap by third grade. In 2020-21, the gap for individual grade levels grew only 2-5pp relative to pre-pandemic levels.

The overall performance among students with disabilities is not much different from pre-pandemic levels. Kindergarten proficiency improved by 0.9pp compared to 2018-19, while other grade levels only fell 1-2pp. These results suggest that students without disabilities fell further behind due to the pandemic than their counterparts, with their proficiency levels declining by 7.2pp in first grade and 6.3pp in second grade.

The reasons for these significantly lower learning gaps among students with disabilities are unknown. It is possible hybrid learning is more beneficial for students with disabilities or staff in special education programs adjusted instruction differently than their counterparts and experienced better scores as a result. Further study would improve understanding this mitigation of pandemic impact and potentially reveal successful strategies that could be implemented statewide.

Improving support programs for students with disabilities or allocating additional resources remains one way to help increase literacy proficiency among this group, which would in turn increase overall reading proficiency.

ECONOMICALLY DISADVANTAGED STUDENTS

Economic disadvantage can negatively affect student performance. Idaho measures economic status by categorizing students as economically disadvantaged or not economically disadvantaged (see Appendix A). Table 11 breaks down proficiency by economic status and grade level for 2018-19 and 2020-21.

Table 11: Literacy Proficiency by Economic Status (Spring IRI)

In both 2018-19 and 2020-21, economically disadvantaged students substantially lagged behind their non-disadvantaged counterparts by 17-19pp and 16-18pp respectively. The most substantial decline is found in first grade following the pandemic. Proficiency increased in later grades, although all proficiency rates lagged compared to pre-pandemic levels.

Though the gaps are similar, the significantly fewer enrolled economically disadvantaged students in 2020-21 suggests learning loss among this population could be greater as many students not enrolled lost a year of education. Data from 2021-22 may better identify the loss among these students.

Economically disadvantaged students account for 40% of all K-3 students. As this is almost half of all students, addressing challenges associated with economically disadvantaged students would likely improve overall state performance.

Students experiencing housing insecurity are counted as economically disadvantaged; however, understanding the impacts of housing security alone is valuable. Insecurely housed students have no permanent home of their own and may be moving from place to place, have multiple families living in a single home, or experiencing literal homelessness. This uncertainty may lead to inconsistent attendance and more distractions among affected students that impacts their learning. The results, summarized in Table 12, illustrate this.

Table 12: Literacy Proficiency by Housing Status (Spring IRI)

Results suggest students experiencing homelessness consistently lag securely housed students. Once again, the pandemic impact was most pronounced among first graders. Students experiencing homelessness increased proficiency from kindergarten to first grade in 2018-19 (+6.2pp) but declined in 2020-21 (-1.8pp).

The overall trend in the gap between both types of students across later grade levels is consistent in that after first grade the gap is either flat or decreases with each successive grade. Even so, the gaps increase from 17-20pp in 2018-19 to 21-24pp in 2020-21, which suggests that the impact of the pandemic exacerbated the learning gap between these student groups.

In terms of the Program, there is an opportunity to increase student proficiency by ensuring that this affected population is better served, so they are able to focus on learning.

This report is IPI’s fourth evaluation of Idaho’s Literacy Intervention Program. As in previous years, there are limitations to properly evaluating this Program. Changes in both the IRI testing instrument and testing procedure in 2018-19 makes is difficult to make meaningful comparison across years.

Across-year evaluation is not possible until four consecutive years of data are available. This is exacerbated by the COVID-19 pandemic, which interrupted instruction and data collection as schools closed and the spring 2020 IRI assessment was not required nor was it universally administered. Consequently, there is a gap in data.

IPI’s analysis of program design is positive, finding it accurately identifies and targets resources towards the students in most need. Prior evaluations of the Program confirm this view is generally supported by K-3 teachers in the state.

In terms of all-day kindergarten, analysis finds the number of schools offering at least some form of all-day K substantially increased over the past two years. Program funds dedicated to all-day K are most likely used to hire additional teachers.

Analysis of use of funds finds personnel is consistently the largest expense category throughout the state. Over the past two years, more than three-quarters of all Program funds were spent on personnel. While curriculum and technology are the next largest expense categories, they constitute a much smaller proportion of Program expenses overall. Last year’s survey of administrators confirmed most LEAs would spend increased funding on more personnel.

Finally, IPI’s analysis of program effectiveness indicates difficulty in separating discrete Program effects from the impact of the COVID-19 pandemic. While proficiency scores were down across the board, this is not unexpected given the precarious learning environments starting in March 2020.

Some student groups did not experience as substantial a decline in proficiency levels following the COVID-19 pandemic’s impact as others. This suggests a potential opportunity to further study those groups to identify successful strategies for mitigating pandemic impact on literacy across the state.

Economically disadvantaged students, students with disabilities, and English learning students continue to perform lower than their counterparts. This suggests a continuing opportunity to increase student proficiency by better serving schools with higher percentages of these groups.

NATIONAL CENTER FOR EDUCATION STATISTICS LOCALES

NCES defines school locales as follows:

• City is defined as “territory inside an urbanized area and inside a principal city”
• Suburb is defined as “territory outside a principal city and inside an urbanized area”
• Town is defined as “territory inside an urban cluster”
• Rural is defined as “Census-defined rural territory”

NCES further subdivides these categories—City and Suburb are subdivided by Large, Midsize and Small, while Town and Rural are subdivided by Fringe, Distant and Remote.

To simplify analysis, only the four overriding categories were used.

ECONOMICALLY DISADVANTAGED STUDENTS

Per Idaho Code § 33-1001(8), an “economically disadvantaged student” means a student who:

• Is eligible for a free or reduced-price lunch under the Richard B. Russell national school lunch act, 42 U.S.C. 1751 et seq., excluding students who are eligible only through a school’s community eligibility program;
• Resides with a family receiving assistance under the program of block grants to states for temporary assistance for needy families (TANF) established under part A of title IV of the social security act, 42 U.S.C. 601 et seq.;
• Is eligible to receive medical assistance under the medicaid program under title XIX of the social security act, 42 U.S.C. 1396 et seq.; or
  Is considered homeless for purposes of the federal McKinney-Vento homeless assistance act, 42 U.S.C. 11301 et seq.

Student-level data from the 2016-17, 2017-18, 2018-19, 2019-20, and 2020-21 academic years was provided to IPI. The dataset included spring and fall IRI scores (if available), grade level, gender, race/ethnicity, free and reduced lunch status, individualized educational plan (IEP) status, limited English proficient (LEP) status, 504 Plan status, homeless status, economic disadvantage status, school, and LEA. The dataset includes over 790,000 unique test scores for 198,335 students over five academic years.

As with prior evaluations, LEA-level literacy program expenditure reports for academic years 2016-17, 2017-18, 2018-19, 2019-20, and 2020-21 were also collected.

TEACHER SURVEY

IPI developed and administered an online survey of K-3 teachers using the Qualtrics platform. The survey was in the field from November 4th, 2020 through November 20th, 2020. In order to reach as many K-3 teachers in Idaho as possible, IPI worked with staff at the Idaho State Department of Education (SDE) to facilitate distribution of an anonymous survey link to teachers with instructions on how to participate.

There were 494 teacher responses with usable data from 71 different LEAs from every region in the state. A summary of respondent characteristics follows:

• By Grade Level
  • 21% Kindergarten teachers (105)
  • 29% 1st grade teachers (140)
  • 22% 2nd grade teachers (106)
  • 21% 3rd grade teachers (105)
  • 3% Multi-grade teachers (15)
• By Region
  • Region 1: 5% (23)
  • Region 2: 6% (28)
  • Region 3: 42% (208)
  • Region 4: 21% (101)
  • Region 5: 13% (62)
  • Region 6: 12% (59)
  • Virtual Schools: 1% (5)
  • N/A: <1% (3)
• By School Type
  • Traditional Public: 85% (341)
  • Brick and Mortar Charter: 14% (56)
  • Virtual Charter: 1% (6)
• By 2020-21 Instruction Type
  • In-person 51% (208)
  • Online 7% (28)
  • Hybrid 42% (174)
• On average, teachers have been teaching in Idaho for 12 years (sd of 8.5)
• On average, teachers have been teaching in their current grade for 7 years (sd 6.6)
  • Only 10% are new to their current grade
• No patterns were found between any demographic and literacy-focused data.

ADMINISTRATOR SURVEY

IPI developed and administered an online survey of LEA literacy program administrators using the Qualtrics platform. The survey was in the field concurrently with the teacher survey from November 4th, 2020 through November 20th, 2020. In order to reach as many administrators in Idaho as possible, IPI worked with staff at the Idaho Office of the State Board of Education (OSBE) to facilitate distribution of an anonymous survey link to all literacy plan contacts with instructions on how to participate.

There were 101 administrator responses with usable data from 72 different LEAs from every region in the state.

Summary of respondents by region:

• Region 1: 8% (8)
• Region 2: 9% (9)
• Region 3: 27% (27)
• Region 4: 14% (14)
• Region 5: 27% (28)
• Region 6: 14% (14)
• Virtual Schools: 1% (1)

A summary of positions respondents held include:

• 18% Administrators (general)
• 5% Curriculum Coordinators
• 3% Directors of Accountability
• 9% Federal Programs Specialists
• 5% Instructional Specialist
• 4% Literacy Coordinators
• 33% Principal or Assistant Principals
• 3% Reading Specialists
• 21% Superintendents or Assistant Superintendents

The 2020 Literacy Intervention Program Evaluation included a section explaining the impacts of COVID-19 based on information collected through teacher and administrator surveys. That section is included here as an appendix as the observations are still relevant in this report. Please keep in mind the narrative is written from the perspective of the situation in December 2020.

COVID-19 IMPACTS

The impacts of COVID-19 related school closures are likely to be seen in future evaluations. The survey included questions about teacher perceptions of student performance, changes made to regular instruction, and the process of providing virtual interventions to high-need students. This information is provided here and could be considered for inclusion in future evaluations to help contextualize scores.

PERFORMANCE

Teachers were asked how their students performed on the fall 2020 IRI and how they feel students will perform in the coming spring IRI compared to previous years (Table 13).

Table 13: Teacher Perception of Future Student IRI Performance

Teachers are noticing differences in students this year in addition to obvious knowledge gaps caused by school closures and precarious instruction at the end of the last school year. Some teachers have reported that students are lacking in educational stamina and have social and behavioral learning gaps that are impacting their ability to learn and progress.

Although schools are not closed this year, the continuing pandemic is impacting student learning. Many LEAs are providing instruction in a hybrid format, with students alternating between in-person and virtual learning. Some schools are allowing students to attend completely virtually while their peers are physically in school. In some cases, teachers are expected to teach both sets of students concurrently. Teachers are aware that their in-person instruction may be moved to complete virtual learning if enough students or teachers have been exposed to the disease.

INSTRUCTION

In response to these conditions, 80% of respondents reported needing to adjust their usual instruction patterns. To address learning gaps, many teachers began the school year teaching content students would have learned in the previous grade and reteaching foundational skills. Teachers are also teaching current required curriculum, pacing their curriculum slower to account for student stamina, and simplifying student expectations. Another strategy is to increase the amount of small group work and differentiated learning to their instruction to specifically account for the large range of student abilities in the classroom. This allows full group instruction to remain similar, while still ensuring individual students are closing learning gaps at their own pace.

Teachers have had to adjust their curriculum to account for time lost. The most common response is prioritizing curriculum and eliminating content depth. The second most common response is to focus on instruction and dedicating less time to practice and engaging projects. Some teachers have increased homework assignments to provide students with more practice and review opportunities.

Respondents teaching students in-person on alternating days described attempting to complete all necessary instruction in-person and having online learning days dedicated to practice. This makes in-person instruction content heavy and requires student concentration. Many teachers, both those teaching hybrid and all in-person classes, reported increasing digital learning within the classroom. These teachers feel the need to prepare students for virtual learning in the event virtual learning becomes necessary again.

ENDNOTES

1 “The literacy intervention program(s) shall continue its independent, external evaluation that includes an analysis of key performance indicators of student achievement. The results of the updated evaluation shall be reported … on the program design; use of funds, including the funding amounts and local education agencies that have utilized all-day kindergarten; program effectiveness; and any other relevant matters.” Senate Bill 1202 (2021) accessed November 15, 2021 from https://legislature.idaho.gov/sessioninfo/2021/legislation/S1202/.

2 Idaho Code § 33-1801 to § 33-1810

3 In prior evaluations, expense report analysis was limited to LEAs that submitted both a budget before the school year and an actual expenditure report after the school year. Due to time constraints, analysis of pre-year budgets was not possible this year and no LEAs were excluded.

4 Idaho Code § 33-208