Literacy Intervention Program Evaluation 2020

Report Authors

• Matthew May, PhD, Senior Research Associate
• McAllister Hall, Graduate Research Associate
• Vanessa Crossgrove Fry, PhD, Research Director

This report was prepared by Idaho Policy Institute at Boise State University and commissioned by the Idaho Office of the State Board of Education.

Recommended citation: May, M., Hall, M.,& Crossgrove Fry, V. (202o). Idaho literacy intervention program evaluation. Idaho Policy Institute, Boise State University.

Download a printable pdf of this report

Executive Summary

For over 20 years, Idaho leaders have recognized the critical importance of early childhood literacy. In this time, the Idaho State Legislature, State Board of Education and State Department of Education have developed policies, rules and plans as well as implemented programs to support reading proficiency in Idaho’s kindergarten through third grade students. In 2015, the Board of Education published an updated Comprehensive Literacy Plan for the state. The Legislature responded in 2016 by amending statutes related to early literacy development and establishing the current Literacy Intervention Program. In 2018, the Idaho Policy Institute completed an external evaluation of the Literacy Intervention Program requested by the Legislature. In 2019, the Legislature requested an additional, updated external evaluation. This report serves as that evaluation. The report evaluates the Literacy Intervention Program and discusses its design, use of funds and effectiveness during its first three years.

The Literacy Intervention Program enables tailored literacy intervention plans at the Local Education Agency-level, allowing for flexibility to account for local needs. The mandated collection of data such as Idaho Reading Indicator scores, program budgets and annual expense reports is necessary for ongoing evaluation.

Three years of expense data indicate a trend in which Local Education Agencies are better anticipating costs associated with the Program. This allows them to allocate their resources more efficiently.

The testing instrument and procedures were changed (moving from the legacy IRI to the new IRI by Istation) in the Program’s third year in order to more effectively evaluate student literacy achievement. This change in testing limits the ability to compare overall literacy achievement across all Program years. However, early indications of patterns within the data can inform the Program’s implementation and evaluation in the future.

Return to the beginning of the report

In 1999, the National Reading Panel was convened by the United States Congress. The 14 member panel reviewed over 100,000 studies on how children learn to read, attempting to determine the most effective evidence-based methods for teaching reading. A major finding was that early reading acquisition depends on the understanding of the connection between sounds and letters. These findings prompted broad scale incorporation of policies across the states.

That same year, indicating continuing recognition of the critical importance of reading skills, Idaho passed the Idaho Comprehensive Literacy Act. The legislation associated with this act sought to mandate regular assessments of kindergarten to third grade (K-3) students (and make school-level assessment data available to stakeholders), provide intervention for students not meeting grade-level reading proficiency and implement associated professional development for teachers and administrators. The original legislation has morphed over time, with the most substantive updates in response to the outcomes of the 2015 Comprehensive Literacy Plan. One of the updates, implemented in 2016 by legislative statute, established the new Literacy Intervention Program (Program), the focus of this report. The Program is now in its fourth year.

Return to the beginning of the report

For this report and its predecessor, Idaho Policy Institute (IPI) reviewed recent peer-reviewed academic literature and studies surrounding literacy intervention to identify best practices, contextualize Idaho’s program and inform IPI’s data collection and analysis of the Program.

IPI collaborated with Idaho State Board of Education (OSBE) and State Department of Education (SDE) staffs to obtain data on performance metrics, specifically the Idaho Reading Indicator (IRI) assessment. IPI requested additional data elements deemed appropriate for the evaluation and supplemented this data with additional data elements on school locale from the National Center for Education Statistics (NCES). This led to three main sets of data:

• Student-level IRI scores and demographic data
• Individual Local Educational Agency (LEA) Literacy Intervention Plans (Plans)
• LEA Literacy Intervention Expenditures

The Plans’ data was combined with the IRI data and NCES data (described in Appendix A) to create a dataset indicating each LEAs’ impacted population, budget and expenditures. This information is reported at the state-level and used to identify patterns by different categories. Further details about the methodology can be reviewed in Appendix B.

Return to the beginning of the report

While the 2018-19 school year represents the third year of the Literacy Intervention Program’s existence, in many ways it must be treated as the first year of an entirely new intervention. This is primarily due to the complete overhaul of the IRI testing instrument and testing process.

The change in assessment instrument in the 2018-19 school year, from the legacy IRI to the new IRI by Istation, makes direct comparisons of future years with the first two years of the Program difficult and problematic.

In 2017-18, Idaho piloted a new IRI testing instrument, developed by the company Istation, in a handful of schools. The purpose of the pilot was to identify best practices for test administration and identify challenges that could impact the statewide rollout. As such, results from these tests could not be used for rigorous analysis. The 2017-18 academic year was the last year the legacy IRI test was still administered to all Idaho students. While legacy IRI scores allow for direct comparison of results over time, they cannot be directly compared with scores from the new Istation IRI.

The legacy IRI testing procedure was a one-on-one assessment between the proctor and student. It was approximately one minute in length and measured only a single aspect of literacy, reading fluency.

By contrast, the new IRI from Istation is a computer-adaptive assessment taken on a tablet or computer that can last approximately 30-45 minutes. It measures five foundational skills of literacy, including alphabetic knowledge, phonemic awareness, vocabulary, spelling and comprehension.

Since all Idaho schools are administering the new IRI from Istation as of the 2018-19 school year and these results cannot be compared with historical results, this essentially makes year three of the Literacy Intervention Program a new de facto year one. As such, it will take several years of Program data under the new IRI by Istation before a comprehensive evaluation will be possible.

In the 2019 Legislative Session, the Legislature doubled Program funding. While not as drastic a limitation to comparison as changing the testing mechanism, it still serves as a potential confounding explanation for any observed changes between results in the current year and results in the next year.

Return to the beginning of the report

Program Design

As noted last year, generally speaking, the Program is well-designed. The ability to tailor literacy intervention plans at the LEA-level allows for flexibility to account for local context and shape interventions to suit local needs. Additionally, the mandated collection of data such as IRI scores, program budgets and annual expense reports is extremely beneficial for ongoing evaluation.

Another strength of the Program is that it distributes resources and intervention directly to the students that need it. IRI scores are a relatively consistent assessment of students’ literacy proficiency because they are administered over time and tracked by both the LEAs and the SDE. The funding formula is directly tied to a three-year rolling average of LEAs’ aggregate student proficiency, requiring the State allocate more funds to high-need LEAs. As noted, once distributed, LEAs can direct these resources to students who need it.

Last year IPI noted a limitation in the Program’s design, namely divided reporting of financial data and restrictions surrounding that information’s usage. LEAs submit budgets to OSBE within the LEA’s Literacy Intervention Plan at the beginning of the year based on estimated literacy funding distribution. Actual funding distributions may vary. Therefore, expense reports submitted at the end of the year may be different from the original budgeted amount. While distribution amounts do not perfectly match the budgeted amounts from the LEA Literacy Intervention Plans used in IPI’s analysis, tracking Program budgeting practices over time will better aid LEAs in anticipating actual Program costs and ensure more efficient allocation of LEA financial resources among expense categories in the future.

Comparison of budgets with expense reports can help in identifying LEAs that may need more assistance in Program implementation if actual expenses consistently exceed budget expectations. The better LEAs are at anticipating how much funding they will have in a given year, the better they can direct those resources to where they will be most effective. This makes the Program itself more efficient over time.

Use of Funds

LEAs are required to submit an expense report of Program expenditures at the end of each academic year. In 2016-17 and 2017-18, expenditures were broken down into four major categories: Personnel, Curriculum, [Student] Transportation and Other. In 2018-19, two additional expense categories were reported: Professional Development and Technology. IPI’s analysis was limited to LEAs for which both Literacy Intervention Plan budgets and end of year expense reports were available. As such, results are reported for 142 LEAs in FY 2017, 147 LEAs in FY 2018 and 135 LEAs in FY 2019. The exclusion of LEAs for which both data points were not available is also why percentages reported for FY 2019 differ from those reported in SDE’s expense report summary. IPI analyzed the proportion of annual LEA expenditures in each funding category. The averaged results across LEAs is summarized in Table 1.

^{Note: Professional Development and Technology expense categories were not present on FY 2017 or FY 2018 expense reports. As such, percentages cannot be reported in those categories for those years.}

Distribution of expenses across categories is generally stable under the first two years of the Program, followed by substantial changes in both the Curriculum and Other categories in FY 2019. The change in Curriculum is due to some curriculum costs being upfront and decreasing over time, while the change in Other is most likely to accommodate the addition of the professional development and technology expense categories. On average, Personnel expenses accounted for 68 percent of the overall cost in the first year of the program, 71 percent in year two and 69 percent in year three, making it relatively stable over time in terms of proportion of total program expenses. Curriculum costs accounted for 23 percent of overall costs in year one, 20 percent in year two, but only 15 percent in year three, a sharp decline attributable to some Curriculum costs being one-time expenses in early years of the Program that would not need to be repeated annually, as well as the addition of two new expense categories.

Transportation expenses remain the smallest funding category, reflecting that few districts expend funds on travel relative to the literacy intervention program—less than one percent in each year. A related factor may be that transportation funding is capped at $100 per student. Professional Development was not reported as a discrete category in previous years’ data and accounted for an average 2.4 percent of literacy program expenses in FY 2019. Likewise, Technology is only reported as a discrete category in FY 2019 and on average accounted for 10 percent of literacy program expenses. Finally, the Other funding category accounted for eight percent and seven percent of expenditures for years one and two, respectively. By FY 2019, that share declined to three percent as more specific categories were added.

Comparing budgeted dollars to actual expenses is a relative indicator of Program efficiency year-to-year. As in last year’s study, IPI compared the start-of-year Literacy Plan budgets to the end-of-year expense reports to produce a measure indicating how accurately the budgets anticipated costs. We classified LEAs “near budget” if expenses were within +/- 25 percent of anticipated costs. If expenses were greater than +/- 25 percent of budgeted costs, then they were either classified “over budget” (if actual expenses were greater) or “under budget” (if actual expenses were less). This allows us to track an element of financial efficiency over time. Figure 1 and Table 2 summarize LEA performance over the last three years of the Program.

In FY 2017, a little over half of LEAs—53 percent—were near their projected budget (the outer bounds of which are represented by blue lines on the graph—the orange line represents the point where budgets and expenses perfectly match). About one-third of LEAs were over budget, while 14 percent were under budget. In FY 2018, the proportion of LEAs near budget increased to 82 percent (a 28 percentage point increase), while those over budget decreased to 12 percent and those under budget decreased to six percent.

In FY 2019, the proportion of LEAs near budget declined to 79 percent, a change of three percentage points. Those over budget increased to 16 percent, while those under budget decreased to five percent.

Three years of expense data is not enough to draw definitive conclusions, but it nevertheless indicates a trend in which LEAs are better anticipating costs associated with the Program. This allows them to allocate their resources more efficiently. As Figure 2 shows, each subsequent year of the program indicates fewer and fewer LEAs at either extreme, suggesting more LEAs are anticipating actual costs. While the categorization percentages may decrease, LEAs are contracting around the “near budget” zone between the two lines, which suggests that even those who fall outside the zone are closer to it than they were in previous years. To be clear, this measure compares expenses to the budgeted amounts submitted with each LEA’s Literacy Intervention Plan at the beginning of the year, not the State’s actual distribution, which LEAs are required to either spend or return to the state.

As noted last year, the remaining outliers are likely due to the budgets of small schools being far more susceptible to small changes in expenses, which would constitute a greater percentage of their initial budget. Additional data points from subsequent years will help improve Program implementation, as it will allow the State to identify LEAs that could benefit from additional financial planning resources.

^{Note: New data allowed for inclusion of an additional 22 LEAs in 2017 and 8 LEAs in 2018 calculations, which accounts for the difference in percentages observed between this report and last year’s study.}

Program Effectiveness

As noted, only three complete years of data from the Program and the added complication of changing the IRI test instrument and testing procedure prevents comparison of 2018-19 results with any of the previous years.

Students who entered kindergarten during year one of the Program (fall 2016) will finish third grade in spring 2020. Even then, with the change of IRI test instrument and testing procedure, the third year of the Program has become a new de facto year one, meaning that students who entered kindergarten during that year will not finish third grade until 2022. Evaluation of the overall effect of the Program will not be possible until then, and even so, those students would constitute only a single cohort.

For this reason, we cannot compare any changes of magnitude between years 1-2 and the current year. To help illustrate this limitation, we have made tables depicting data from the first two years of the program visually distinct from those depicting data from the current year. Tables with blue headers contain data from the first two years of the Program, under the legacy IRI, while tables with orange headers contain data from the most recent year, under the new IRI from Istation. While blue tables may be compared to other blue tables, and orange tables to other orange tables, IPI stresses that data between blue and orange tables cannot be directly compared.

That said, there is still some utility in looking at both sets of data. For example, it is useful to see if larger trends in the data that were observed in last year’s report—such as the gap between English-language learners and non-English language learners—is still present in the current year. While the magnitude of those scoring “Proficient” is not as important, confirmation of the existence (or absence) of gaps such as these can be useful indicators of patterns within the data and can inform the manner in which the Program is implemented by identifying where resources may best be allocated in order to effect change at the LEAs overall reading proficiency level.

IPI’s analysis was limited to students who took both Fall and Spring IRI assessments in a particular academic year. Table 3 provides a descriptive overview of the Program’s first two years in the dataset used for this evaluation. Table 4 provides the same descriptive information for 2018-19.

IRI Proficiency

One key component of the Program is assessing student literacy proficiency to best direct resources to students requiring more learning support. As such, each fall and spring, students in grades K-3 take the IRI. It is important to note, the intention of the IRI is to assist instructors in identifying students who may need additional support to achieve grade-level reading, rather than to evaluate students or their instructors. Generally, fall scores are lower than spring scores. For kindergarten, which is not compulsory in Idaho, this can be due to many students being exposed to a formalized education for the first time. For higher grade levels, another explanation is the so-called “summer slide,” the months of summer vacation when students are without daily classroom instruction. Those who do not score proficient on the fall exam are required to receive additional instructional hours (30 hours if scoring Basic, 60 hours if Below Basic), within the school year, to bring them to grade level.

Table 5 breaks down Spring IRI scores by grade level for 2016-17, while Table 6 does for 2017-18. In the first two years of the program, reading proficiency levels were generally stable at each grade level between years. Kindergarten students achieved the highest reading proficiency level, which was followed by a substantial drop-off in proficiency rate from kindergarten to first grade. Scores improved only slightly for second grade and more so in third grade.

Collectively, the results from these years suggest that students do not have a particularly difficult time grasping kindergarten-level reading concepts but begin to struggle as they are introduced to more advanced concepts in first and second grades. More specifically, under the legacy IRI in kindergarten, children are tested on their ability to identify letters and their sounds. Starting in first grade, they begin to learn to read and the assessment includes additional skills that are necessary in the typical learning progression of literacy, which can be challenging for some and thus result in lower test scores.

Second grade appeared to be especially challenging to students during these years, as it is the only grade in both years where Below Basic is the second-most frequent outcome.

Table 7 breaks down Spring IRI scores for 2018-19 using the new IRI from Istation testing instrument and new testing procedures. Data for this year no longer indicates a substantial gap in proficiency from kindergarten to first grade, but instead a gradual increase that continues into second grade before plateauing in third grade. Additionally, Below Basic is the lowest proportion for all four grade levels.

The results for kindergarten in 2018-19 could be attributable to multiple factors. It could be that the new IRI from Istation is a better reflection of actual literacy proficiency across all grades and the expected progression of learning; it could be difficulties relating to the use of computers or tablets in the administering of the test account for the difference, particularly when prior access to technology is differentiated among students; or it could be an aberration. Without additional years of data to compare these results against, IPI cannot conclusively say which factor is at work.

Locale

An indication of proficiency differences between urban and rural students may be useful in directing support to underperforming LEAs. To determine if this was the case in Idaho, NCES’s indicator of school locale was used to create categories for comparison at the school-level. NCES currently defines school locale along four overriding categories: City, Suburb, Town and Rural (for how each category is defined, see Appendix A). For this study, IPI added a fifth category, Virtual, to better account for the geographic distribution of those using online school options in Idaho. Table 8 provides a breakdown of the number of students and number of schools in each locale category. Tables 9 and 10 summarize proficiency levels by school locale for the first two years of the program, while Table 11 does so for 2018-19.

With the exception of virtual schools, the distribution of students was fairly consistent across locales for all three years, with slightly more students attending suburban schools than other classifications. Virtual schools had the fewest number of students. Similarly, city, suburb and town locales had roughly the same number of schools, while rural schools had nearly twice as many. With only five virtual schools included in the dataset, they accounted for the fewest.

As depicted in Table 9 and Table 10, during the Program’s first two years, suburban schools performed best across most grade levels. Virtual schools had the lowest proficiency, followed by town schools.

Results from 2018-19, shown in Table 11, were generally consistent with previous observations. Suburban schools performed best among locales across all grade levels, while virtual schools had the lowest proficiency levels in all grade levels.

Racial and Ethnic Diversity

As race and ethnicity have been associated with academic performance, IPI once again created an indicator of school diversity. According to the U.S. Census Bureau’s American Community Survey (2018), Idaho’s population is 91 percent white, which suggests most Idaho schools will have predominantly white students. Therefore, we created a relative diversity measure for Idaho schools by coding all schools in the dataset according to the racial/ethnic makeup of its K-3 students and dividing the schools into subgroups. Schools with a student body that is over 90 percent white are classified low diversity, those that are 85-90 percent white are classified medium diversity, 75-84 percent as high diversity and those with less than 75 percent white students as very high diversity. Table 12 provides a breakdown of total number of students and total number of schools included in each diversity category. Tables 13, 14 and 15 breakdown IRI proficiency by grade level and school diversity for each year.

As noted in IPI’s previous report, during the first two years of the Program, there is not much difference in kindergarten proficiency levels across diversity classifications. There is a much more pronounced effect in subsequent grades, where schools with very high racial diversity generally have lower proficiency levels than schools with other diversity classifications. This is likely a result of more diverse schools having a higher concentration of non-white students for whom English is a second language, which can substantially affect proficiency rates.

Results from 2018-19 in Table 15 indicate that schools with very high racial diversity substantially perform worse than other diversity classifications across all grade levels, including kindergarten. Once again, kindergarten proficiency levels in 2018-19 could be due to a variety of factors, including the change in testing instrument/procedures, or possibly be an anomaly. Alternatively, it could be that the effect of higher concentrations of non-white students and students who are not fluent in English are amplified under the new testing procedures, making the variation more dramatic across all grade levels. More years of data are necessary before IPI can reach a conclusion.

English Learners (EL)

As the IRI assesses students’ proficiency in reading English, students for whom English is not their first language have lower levels of proficiency. Idaho schools identify such students through a system with eleven designations for English Learners (EL). For ease of analysis, we have collapsed these classifications into two designations: EL students (those in the program or those who have exited within the past two years) and non-English learners (those now fluent, who have exited three or more years ago, screened out or not applicable). Tables 16 and 17 summarize the results for the first two years of the Program, while Table 18 depicts data from 2018-19.

Results for the first two years (Tables 16 & 17) follow the same patterns observed up to this point, with a large drop from kindergarten to first grade followed by gradual recovery. These results are far more pronounced for EL students, however.

In 2018-19, similar to earlier observations, the kindergarten to first grade drop-off is no longer evident in results. The gap between EL and non-EL students is far more pronounced in kindergarten and shrinks in subsequent years, but not enough where EL students find parity with their non-EL counterparts.

The gap between EL and non-EL students exists across testing instruments and is indicative of the difficulties of learning a second (or additional) language compounding the inherent challenges of learning how to read. This means EL students learn at a different pace from their non-EL peers, which begins to explain why schools with very high diversity tend to have lower proficiency rates — they likely have a higher concentration of EL students facing unique challenges. In addition, these challenges vary depending on a student’s previously learned languages. In other words, EL students may need different levels of intervention based on their previous language skills.

Knowing that the added challenges EL students face may contribute to lower reading proficiency scores suggests a possible area for improvement in allocating state resources. Improving programs and interventions that aid and support high-need EL students, or that help mitigate those challenges, could produce an overall positive impact on reading proficiency. The earlier these challenges can be mitigated, the better it allows these students to reach higher proficiency rates by the third grade.

Students with Prior Learning Accommodations

Some students have disabilities that necessitate an Individualized Education Plan (IEP) to accommodate their learning. As with some students for whom English is not their first language, some students with an IEP can face challenges when learning to read. Tables 19, 20 and 21 summarize IRI proficiency levels according to disability status.

Across 2016-17 and 2017-18, reading proficiency among students with disabilities consistently lagged students without disabilities. The margin between the two grows wider with each successive grade level.

Results for 2018-19 in Table 21 also show that students with disabilities perform below students without disabilities. These results once again indicate that the new IRI from Istation realigned kindergarten results to the point where a large drop off is no longer evident. As noted in last year’s report, improving support programs for students with disabilities or allocating additional resources can help increase reading proficiency among this group, which would, in turn, increase overall reading proficiency.

Economically Disadvantaged Students

Economic disadvantage is known to affect student performance. While there is no direct measure of a student’s level of economic security available, a common proxy is whether they are eligible for free or reduced-price lunches. For 2016-17 and 2017-18, state data sorts students into five possible categories—free lunch eligible, reduced price eligible, district eligible, community eligible school and not eligible. It is important to note that while the state records this data as a single variable, they are actually determined at two separate levels of analysis.

Free lunch eligible, reduced-price eligible and not eligible are all student-level classifications determined by the student’s own personal status. Conversely, a student is classified as district eligible or community eligible school if a high enough proportion of the LEAs’/schools’ students qualify for free or reduced lunch. In that case, eligibility is granted to the entire LEA or school population, regardless of their personal eligibility status. As such, it is important to consider these classification groupings separately, since they are not directly comparable with one another. Tables 22 and 23 breakdown IRI proficiency by these classification levels in the first two years of the Program.

For the first two years of the Program, the data indicates that both free or reduced lunch eligible students generally lag behind those who do not qualify, especially after kindergarten. Students who qualify for free lunches—generally an indicator of greater economic disadvantage than reduced-price lunch eligibility—have the lowest level of proficiency among student-level lunch metrics.

Of the school or district level metrics, proficiency rates differ based on Program year. In year one, district eligible students generally performed better than students in community eligible schools. In year two, however, results were mixed.

In 2018-19, district-level classifications were no longer used in order to make data more comparable. Two additional student-level categories were added: Direct Certification, which identifies children eligible for free meals under the National School Lunch Program, and students whose eligibility is determined through a Home Income Survey. These results are broken down by grade level in Table 24.

Data from 2018-19 reinforce the student-level observations we noted in prior years. Free or reduced lunch eligible students continue to lag behind those who do not qualify, an observation that extends to include both new lunch status classifications.

An additional challenge faced by some students is housing insecurity. For some, this means having no permanent home of their own, in which case they may be moving from place to place, have multiple families living in a single home or literally experiencing homelessness. This uncertainty may lead to inconsistent attendance and more distracted behavior among affected students that impacts their learning. The results, summarized in Tables 25, 26 and 27 help demonstrate this.

Data from the Program’s first two years suggest that students experiencing homelessness consistently lag behind securely housed students. Unlike Students with disabilities, the wider gaps in first and second grades start to contract by third grade, especially in year two.

Data from 2018-19 demonstrate the same gaps, with the exception (once again) of there no longer being a drop off from kindergarten to first grade. Students facing uncertainty due to homelessness trail students that do not by a substantial margin across all grades. What’s more, the gap appears to be relatively consistent across grade levels.

As IPI noted last year, in terms of the Program, this means that there is an opportunity to increase student proficiency by ensuring that this affected population is better served, so that they are able to focus on learning. Overall, it is important to recognize how these different programs affect each other and improvement in one may require attention elsewhere.

Return to the beginning of the report

The current Literacy Intervention Program, which IPI assessed in both 2019 and this report, is just one example of improvements made by the State of Idaho since implementing a strategic approach to early childhood literacy. By making a commitment to utilizing assessment data to make evidence-based decisions, the State will likely continue to improve on its ability to identify students most in need of additional literacy interventions and, thus, support all students’ efforts to achieve grade level reading by third grade. In this regard, the data put forth in this report once again demonstrates that some factors associated with students, outside their educational experience, may also influence student performance.

That said, there are limitations in properly evaluating this Program at this time. Changes in both the IRI testing instrument (moving from the legacy IRI to the new IRI by Istation) in 2018-19 and the IRI testing procedure (from a one-minute test of fluency to a 30-45 minute computer adaptive test) makes meaningful comparison of results across years not possible. As such, we are only able to examine whether or not service gaps or larger trends are still present. We are not able to assess the magnitude of improvement across years, although it is possible to do so within an individual academic year.

Furthermore, across-year evaluation will not be possible until several more years of directly comparable data are available. Changes in the Program, including additional funding and revisions in evaluative criteria for schools, may affect how data across academic years can be compared.

The most significant finding from the current year’s data is the absence of the previously observed drop-off in proficiency scores from kindergarten to first grade. As noted, this finding could be attributable to any number of factors. It is possible the new IRI from Istation is a better reflection of actual literacy proficiency across all grades. Alternatively, lower scores in kindergarten could be the result of difficulties relating to the use of computers or tablets in the administering of the test. It could also simply be an anomaly. With only a single year of data from the new IRI from Istation, IPI cannot conclusively explain the discrepancy.

Outside of this change, the addition of 2018-19 data has largely confirmed patterns that IPI previously identified within the data. These trends can inform the manner in which the Program is implemented, especially in identifying where resources may be allocated in order to effect change at the LEAs’ overall reading proficiency level. With further evidence-based interventions directed at students’ specific needs, there is a potential for further improvement in their proficiency levels and, thus, the overall proficiency level of the State.

Return to the beginning of the report

National Center for Education Statistics Locales

• City is defined as “territory inside an urbanized area and inside a principal city”
• Suburb is defined as “territory outside a principal city and inside an urbanized area”
• Town is “territory inside an urban cluster”
• Rural is defined as “Census-defined rural territory”

NCES further subdivides these categories—City and Suburb are subdivided by Large, Midsize and Small, while Town and Rural are subdivided by Fringe, Distant and Remote. To simplify analysis, only the four overriding categories were used.

Return to the beginning of the report

Student-level data from the 2015-16, 2016-17, 2017-18 and 2018-19 academic years was provided to IPI. The dataset included spring and fall IRI scores, grade level, gender, race/ethnicity, free and reduced lunch status, individualized educational plan (IEP) status, limited English proficient (LEP) status, 504 Plan status, homeless status, school and LEA. The dataset includes over 704,000 unique test scores for 171,755 students over four academic years.

Since last year’s study, OSBE and SDE identified individuals who had inadvertently been assigned multiple student identification numbers, collapsing their results into a single unique identification. Inspection of IPI’s dataset indicated seven instances of repeated records, although in each case the data in the record matched exactly. The duplicates were dropped from analysis and data from the 2016-17 and 2017-18 school years were re-calculated, which is why some percentages throughout this report may be different than those previously reported.

As with last year’s study, LEA-level data from the Literacy Intervention Plans for academic years 2016-17, 2017-18 and 2018-19 was also collected along with expenditure reports.