Research sheds light on which factors actually predict SOL pass rates
The percentage of students eligible for free or reduced lunches is one predictor of SOL pass rates, according to a Longwood research study.
Many of the most commonly cited remedies for improving educational test scores—shrinking class sizes, boosting teacher salaries or hiring teachers with advanced degrees—actually have little power to predict Standards of Learning (SOL) pass rates, says a Longwood economist.
What does seem to matter are economic and demographic variables related to students at the school—including ethnicity and family income, according to a study by Dr. David Lehr. The study analyzed 2008-09 SOL pass rates in reading and math for third-graders in nearly every Virginia elementary school at that time.
Lehr said the percentage of students eligible for free or reduced lunches and the percentage of African-American and Hispanic students in a given school are important predictors of SOL pass rates. This is the case even after controlling for other factors such as class sizes, levels of teacher education and teacher salaries, he said, adding that these and other variables often thought to impact test scores—including a school’s attendance rate, the number of days taught in the academic year and the percentage of teachers with doctorates— appear to have little predictive ability.
“This study should convince policymakers that there is no magic bullet for SOL pass rates,” said Lehr, associate professor of economics. “If you think that raising salaries or lowering class size or hiring teachers who are more educated will be the key to substantially higher pass rates, this paper should give you pause.”
The study by Lehr and two others was an effort to develop an accurate forecasting model to enable policymakers to predict SOL pass rates. This was the first time Lehr knows of that so-called machine learning techniques—now often applied to “big data”—have been used with data from K-12 education.
“We wanted to demonstrate to policymakers the utility of a data-driven approach to predicting SOL scores given different educational environments. If you redistrict or alter class sizes, for instance, our work can help policymakers understand the likely impact on school performance,” said Lehr. “We weren’t interested in cause and effect; the methodology we use is not designed for that purpose. Our goal was to come up with a way to slice the data so that it would provide the best possible prediction of pass rates.”
The study examined the importance of 22 school variables in predicting pass rates. For both math and reading, the most important predictors were the percentage of students eligible for free or reduced lunches and the percentage of African-American students (the higher those percentages, the lower the school’s pass rate, holding constant all other variables). The percentage of Hispanic students also was an important variable for predicting pass rates in both subject areas.
Lehr thinks the approach used to analyze the data can also be useful in high schools to improve graduation rates by identifying at-risk students.
The study analyzed data from 1,101 schools, or 99 percent of Virginia’s elementary schools at the time. The SOL scores from 2008-09 were the most recent data available when the study began.
Lehr’s co-authors are Dr. Melanie Marks, professor of economics at Longwood, who initiated the study and collected the data, and his son, David Lehr, an undergraduate at the University of Pennsylvania who provided expertise in statistical learning models.—Kent Booty