• As students proceed through the lesson, they discover a surprising fact: small schools aren’t just more likely to show big growth. They’re also more likely to show big declines. To avoid giving away this “punchline,” it can be helpful to avoid having students look closely at the percent of schools in each size category that show big growth, moderate change, and big declines until they reach the Discussion Question. At that point, students will experience the full surprise and be tasked with explaining why it occurs.
• Although the chi-square test is new, the central idea is familiar: compare what actually happened (observed counts) with what we would expect to happen if the null hypothesis were true (expected counts). Returning to this observed-versus-expected framing throughout the lesson can help students see the chi-square test as an extension of earlier inference, rather than an entirely new concept.
• Consider lingering on Question 4 in the Handout before introducing the formula for the chi-square statistic. Students can already identify where observed counts differ from expected counts. However, they may not be able to determine if a difference is substantial enough to be meaningful or significant. This motivates the purpose of the chi-square statistic before students encounter its calculation.

First, download this lesson's Handout Key and read through its Discussion Question section. Then, check out our model discussion norms and the additional background notes below.

• Students may initially attribute the higher rate of improvement among small schools to characteristics of small schools themselves. Once they notice that small schools also have the highest rate of large declines, encourage them to reconsider that explanation and ask what features of smaller schools could produce more extreme outcomes in either direction.
• If students need a bridge, connect the discussion to earlier work with sampling distributions: smaller samples tend to produce greater sample-to-sample variability. Because fewer students contribute to a small school’s overall rating, differences in student performance from one year to the next can produce larger swings in the school-level measure.

• Advocates for small schools have long argued that smaller schools are easier to reform, as they provide easier settings for collaboration and consensus-building. For example, a 1989 New York Times op-ed by Deborah Meier, an early advocate of the modern small schools movement, argued that “Small schools offer opportunities to solve every one of these critical issues…staff can meet to discuss issues and differences without complex governance structures; understanding the budget does not require an advanced degree in accounting. Looking in on colleagues and, sharing ideas, becomes possible.”
• The pattern surfaced by this lesson – that smaller schools have more variability in their performance due to their smaller sample size of students – is also discussed in several popular books, including Thinking, Fast and Slow and How Not to Be Wrong.
• The context provides a useful opportunity to revisit the principle that correlation does not imply causation. The data show an association between school size and changes in school performance, but they do not establish a causal link. In fact, the higher rates of both big improvements and big declines among small schools suggest that sampling variability may be the primary reason for the pattern, rather than any other inherent traits of small schools.

• Although chi-square looks very new, the underlying reasoning is familiar: describe what we would expect under the null hypothesis (expected counts), measure how far the observed data (observed counts) depart from that expectation, and determine whether such a discrepancy could reasonably occur by chance. Emphasizing this common structure can help students understand chi-square as another application of statistical inference, rather than a totally new procedure to memorize.
• Expected counts connect directly to students’ earlier work with independent events in lesson 2.A.5. Under the null hypothesis, knowing a school’s size does not affect the probability of its performance falling into a certain category. In probability notation: P(Big improvement in ratings | school is small) = P(Big improvement in ratings). This mirrors the definition of independence: P(A | B) = P(A). Making this connection can help students see expected counts as a familiar concept, rather than as another calculation to memorize.
• One advantage of the chi-square test is that it allows us to ask a single overall question about whether the distributions differ across the groups, rather than conducting many separate pairwise tests (e.g. conducting many two-sample z-tests for the difference between population proportions). A significant result tells us that there is evidence of a difference somewhere in the table, but it does not tell us exactly which groups or categories account for that difference. In more advanced statistical work, researchers may conduct follow-up comparisons while adjusting for the increased risk of Type I error that comes from conducting multiple tests.
• For a more detailed look at how the p-value for a chi-square test is calculated from a chi-square distribution, check out this supplemental video.