Harvard researchers find evidence that Skew The Script boosts AP Stats scores
Big Update: A research group at Harvard found evidence that Skew The Script lessons had a causal impact on AP Stats scores across Texas. Specifically, Skew The Script’s curriculum boosted pass rates about 5-6 percentage points after one year of implementation and 10-11 percentage points in subsequent years. As far as we know, this is the first-ever research of curricula showing causal improvement for scores in AP Stats - a course that's been in U.S. high schools since 1996.
Let’s break it down
The researchers, Eric Taylor and Sara Ji, analyzed the AP Stats scores of 125,000 Texas students from 2014 - 2022. Because an experiment wasn’t possible (high schools don’t randomly assign curriculum to students), they used the next-best method for investigating causality: a quasi-experiment. Specifically, they used a difference-in-differences study design. If you’d like to dive deeper into the methods, check out Figure I from the research brief.
The key takeaway is shown in Figure II from their analysis.
Figure II from the research brief, showing the effect of Skew The Script on AP Stats exam pass rates in Texas
Year -1 in the graph represents the year before schools in the treatment sample started regularly using Skew The Script. The y-axis is set to 0 at that year, to account for any baseline differences in pass rates between treatment schools (which adopted Skew The Script) and comparison schools. This figure shows two remarkable trends:
No difference before Skew The Script: In the 5 years before the treatment sample started using Skew The Script, there was no observable change in the pass rate trends between treatment schools and comparison schools. The values are all tightly hugging that zero line.
A marked difference after Skew The Script: At Year 0 (first year using Skew The Script), the treatment schools already show a 5-6 percentage point improvement relative to trends for comparison schools. At Year 1+ (second year and beyond using Skew The Script), that difference jumps to 10-11 percentage points, which is statistically significant.
When taken together, according to the assumptions of the difference-in-differences approach, these trends provide evidence that Skew The Script caused an increase in AP Stats scores. The case for causal impact becomes even stronger when the analysis is broadened to other AP Exams at these same schools. When the researchers looked at AP subjects where Skew The Script does not offer curriculum - AP English Language (Figure III) and AP Calculus (Figure IV) - we see no evidence of a treatment effect.
Analyses of AP English Language (Figure III) and AP Calculus (Figure IV) from the research brief, showing no change in other Texas AP subjects
Imagine that the treatment schools made other reforms (e.g. school leadership changes or adding teacher professional development) at the exact same time that they adopted Skew The Script. We’d expect the effect of those reforms to show up in other AP subjects. However, we do not see improvements in other subjects. We only see improvements in the subject (AP Stats) that adopted Skew The Script.
Importantly, researchers also wanted to make sure that higher pass rates weren’t coming at the cost of including fewer students in AP Stats. As shown in Figure 5 from their brief, even as AP Stats pass rates went up, AP Stats enrollment stayed consistent between treatment and comparison schools.
More to learn
In the fuller writeup of their results, the researchers discuss applying the same analysis from our home state (Texas) to their home state (Massachusetts). Unlike in Texas, they found no change in AP Stats exam pass rates between treatment schools and comparison schools. However, they did find suggestive evidence that the number of students taking AP Stats increased in schools that began regularly using Skew The Script:
Figure IV from the fuller write-up, showing AP Stats enrollment increases in Massachusetts schools that adopted Skew The Script
We also find these Massachusetts results to be quite promising. All things being equal, if you increase the breadth of students taking AP Stats, many would expect the percent passing the exam to go down. In Massachusetts schools that adopted Skew The Script, enrollment increased in AP Stats. However, instead of pass rates going down, they stayed the same. This could essentially be the same treatment effect we saw in Texas (increasing pass rates while keeping enrollment the same), just expressed a different way (increasing enrollment while keeping pass rates the same).
We also have other hypotheses for the differences in results between states. For example, it could be that the baseline pass rate was already so much higher in Massachusetts treatment schools than in Texas treatment schools, that there wasn't as much room to grow. In addition, it could be that the treatment sample in Texas had a higher share of Title I schools than in Massachusetts. Our longstanding theory is that genuinely relevant math is most impactful for increasing achievement among low-income students. If that's the case, the difference in effect could be due to difference in the sample composition.
Clearly, there’s more to learn. We’re excited to continue working with Harvard researchers to dig deeper into these results and to expand the analysis to later life outcomes (e.g. college attendance, performance, and career outcomes).
Five years
In the meantime, it’s worth noting that this week marks the 5th anniversary of the launch of skewthescript.org. Five years after an AP Stats teacher in San Antonio decided to share genuinely relevant math lessons online, we now have quantitative evidence of something that our 20,000+ teacher-users already knew: Skew The Script works.
To the students who’ve inspired our work, to the fellow teachers who’ve trusted our materials in their classrooms, and to the supporters who’ve made it all possible: thank you. We couldn’t do any of it without you, and we can’t wait to see what the future brings.
Let’s skew it!