• There has been no well-designed study that has established a causal link between vaccines and autism. However, rather than beginning the lesson with that final conclusion, it’s more powerful to allow students to openly analyze the strengths and weaknesses of Wakefield’s study. This provides them the opportunity to apply principles of experimental design and reach the conclusion themselves that Wakefield’s study falls short of showing a causal relationship.
• This lesson provides an opportunity to distinguish between statistical error and design flaws. Type I and Type II errors arise from random variation in well-designed studies, whereas poor study design introduces bias or confounding. Emphasize that statistical inference addresses uncertainty due to chance – not problems caused by flawed study design. Adjusting a significance level cannot correct for the bias or confounding introduced by poor study design.

First, download this lesson's slide deck and handout key to see the prompt and sample responses for the Lesson Starter. Then, check out the additional background notes below.

Instructional routine: This Lesson Starter is a modification of the Notice & Wonder instructional routine. Specifically, this Lesson Starter is a “slow reveal” Notice & Wonder that plays out over the first several slides in the lesson slide deck. Students engage in two iterations of the noticing and wondering – one round for each slide – with more information revealed on the “Round 2” slide. Each round should have silent independent time for students to add noticings and wonderings to their lists. Optionally, partner sharing could be added during the first round as well. However, the whole group share-out and conversation should be saved for the final round. For more precise guidance:

• Begin the routine as usual, but after 1 minute, advance to the next slide (slide 3) and ask, “How about now?”
• Still on slide 3, engage with the rest of the routine, surfacing noticings and wonderings from students in a whole class discussion.
• Finally, reveal the graph title (slide 4), discussing which questions are now answered or changed and what new wonderings students now have.

Purpose & Background: The goal of this lesson starter is to introduce students to a graph that they’ll analyze later in the lesson. The graph is provided without much context, allowing students to observe trends and wonder about their causes without strong preconceived notions. In the “reveal” slide, students discover that the graph describes measles case counts in the U.S. during the past several decades. Students will explore this context – and how it may relate to claims linking the measles vaccine to autism – throughout the lesson. The data was compiled by the CDC.

First, download this lesson's handout key and read through its Discussion Question section. Then, check out our model discussion norms and the additional background notes below.

• Some students may point to the relatively small sample size of the University of Tokyo study as evidence that it was more likely to produce a Type I error. It can be helpful to clarify that the probability of a Type I error is determined by significance level, α, not by sample size. The stronger concern in this study is the large number of outcomes that were tested (the problem of multiple tests).
• Consider asking students to contrast the University of Tokyo study with the later Duke study. The Duke researchers focused on a single primary outcome. Testing one outcome is less likely to result in a Type I error than testing many outcomes at once.
• The higher likelihood of committing a Type I error when testing multiple outcomes is known as the problem of multiple tests. Although the problem of multiple tests is not generally listed in most state learning standards for high school statistics, it’s an important extension of the concept of Type I error. For a light-hearted visual representation of the problem of multiple tests, consider sharing this xkcd cartoon with students.

• Wakefield’s study serves as an effective example of how flaws in experimental design can limit the conclusions that can be drawn from data. The lack of comparison groups, random assignment, replication, and controls provides a useful review of the principles of experimental design.
• The contrast between the Wakefield study and the later oxytocin studies helps illustrate the difference between poor study design and statistical error. Wakefield’s study suffers from design flaws, whereas the Tokyo study provides an example of how a well-designed study can still produce a statistical error due to chance.
• The progression from the Tokyo study to the larger Duke replication study highlights the role of replication in scientific research. Repeated studies provide additional evidence and help researchers distinguish genuine effects from unusual results that arise due to sampling variability.

• The problem of multiple tests can be derived from the Type I error rate and the rules of probability. Imagine researchers are conducting 10 independent hypothesis tests within the same study, and each test has a Type I error rate of α = 0.05. In this scenario, the probability of making at least one Type I error is: P(at least one Type I error) = 1 − P(no Type I errors) = 1 − (0.95)10 ≈ 0.401. This means there is approximately a 40% chance of observing at least one statistically significant result by chance. This is the problem with multiple tests.
• Type I and Type II errors are not equally controllable. Researchers directly control the probability of a Type I error by choosing the significance level, α, before conducting the study. In contrast, the probability of a Type II error depends on several factors, including sample size, effect size, sampling variability, and the significance level α. Because researchers cannot simply choose a desired Type II error rate, they often focus on increasing a study’s sample size. Higher sample sizes produce more precision, which make a treatment effect easier to detect – thereby reducing undetected effects (reducing Type II errors).