• Natural experiments are not covered on the AP Exam. So, naturally, students might wonder why we’re discussing natural experiments in this lesson. It can help to name three reasons:
• 1) Natural experiments involve fascinating methodology, and they help us answer interesting questions.
• 2) Studying them now provides an early window into more advanced statistics courses and methods.
• 3) We can use methods that are covered on the AP Exam to analyze them, including the method that we’re focusing on in this lesson: the two-sample t-interval for the difference between population means.
• When discussing this lesson’s key question – does college have a causal impact on wages? – students may bring up examples of famous individuals who didn’t attend college and who are very wealthy (e.g. athletes, entertainers, actors, etc.). It’s helpful to emphasize that this lesson is about averages. We’re investigating whether college has an impact on wages, on average. There will always be examples of individuals who don’t go to college, become a star, and become rich. However, there are also examples of individuals who don’t go to college, attempt to become a star, and never get their big break. Rather than discussing individual cases, we can use data to analyze general trends and the labor market at a higher level.
• When students interpret their confidence intervals throughout the lesson, encourage them to identify the point estimate, using questions like: “What is the center of your interval? What does that center represent?” This will help students internalize that their intervals represent plausible values for the difference between two means. Then, ask students whether the value of 0 is included in their interval. If the interval includes 0, what does that mean? If it doesn’t include 0, what does that mean?

First, download this lesson's Handout Key and read through its Discussion Question section. Then, check out our model discussion norms and the additional background notes below.

• Students may veer into unproductive discussion of applicants’ “innate ability.” Some extra framing:
• Imagine that two applicants – applicant A and B – have the same innate ability. Applicant A attended a high-performing high school. Applicant B attended a lower-performing school. Due to this inequity, applicant B is applying with a lower GPA and lower test scores.
• In theory, Applicant B should perform just as well as Applicant A if they are at the same college. However, if the academic preparedness gap created by attending a lower-performing high school is too severe, it may prevent Applicant B from being successful.
• What do the study’s results suggest about the above scenario if Applicant B is just above the admission margin? Just below the admission margin? Far below the margin?
• Encourage students to think about this problem in terms of what was studied: the outcomes of admitted students. Students may have ideas about how lowering a GPA cutoff might affect a university’s ranking, alumni relations, and other factors. However, we don’t have any evidence that speaks to these considerations.
• This discussion question can be connected back to the Unit 1 idea of generalizability. Because the researcher only studied students who were close to the GPA cutoff, the results may not generalize to students who were far above or far below the GPA cutoff. Because natural experiments often study particular subgroups of samples, they tend to have a narrower scope of generalizability than typical observational studies.

• The paper and data studied in this lesson can be found here: Zimmerman, Seth. “The Returns to College Admission for Academically Marginal Students.” Journal of Labor Economics, 2014, vol. 32 (4). https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1962185
• In the introduction, we identify high motivation and skill as possible confounding factors that may imply correlation rather than causation. However, as the New York Federal Reserve notes: “a number of studies that attempt to correct for this possibility find a similar return. In part, this is likely because in today’s labor market a college degree continues to serve as a gateway to professional occupations that offer better opportunities for wage growth over the life cycle.” This is further evidence, beyond the natural experiment explored in the lesson, that college may have a causal impact on wages.
• In addition to estimating the direct impact of college attendance on wages, the study at the center of this lesson also estimated the return on investment from college – that is, the percent boost in net earnings after subtracting the cost of college and the lost wages from not working for four years (while a full-time student). The study author notes: “I find that the internal rate of return on investment is r = 0.074 … this indicates that the direct cost of school attendance and the opportunity cost of forgoing earnings while in school are relatively small compared to the earnings gains that accrue to students who are able to attend school.”

• In addition to regression discontinuity (covered in this lesson), there are other types of natural experiments and quasi-experiments, including instrumental variable analysis and difference-in-differences. Each method is designed to detect situations in which “as good as randomly assigned” groups are found in observational data. When these scenarios meet all assumptions, they can be analyzed for causal inference. Note: natural experiments will not be covered on the AP Exam. They are just an extension topic that we find particularly cool :).
• Natural experiments have broad applications in fields where random assignment to treatment isn’t commonplace, such as in education. In fact, researchers used a difference-in-differences natural experiment to determine that Skew The Script had a positive causal impact on AP Statistics test scores in Texas.
• For two-sample t procedures, most software and calculator platforms use the Welch–Satterthwaite equation to determine the degrees of freedom. The equation and its derivation are far beyond the scope of our course. But students may benefit from knowing that the degrees of freedom from this equation usually are higher than the more conservative estimate we use when determining the degrees of freedom for two-sample procedures by hand (n – 1, where n is the sample size from the smaller group). So, the underlying t-distribution used by software and calculator platforms will tend to be more precise than the manually-calculated t-distribution, yielding slightly narrower intervals. The degrees of freedom and associated interval widths from both methods (from software or by hand) will both be accepted on the AP Exam.