• Because this study is a well-designed experiment with random assignment to treatment, the observed difference in callback rates can be interpreted as a causal effect of the assigned names within the context of the experiment. That said, the underlying mechanism behind the name advantage is less clear. Intentional racial discrimination is one possible explanation. However, it’s also possible that the differences in callback rates could arise from unintentional biases. For example, commonly-white names may tend to be more familiar for the people making hiring decisions, so hiring managers may unconsciously gravitate to such names. Framing these considerations with students is important for interpreting the study and provides a natural opportunity to revisit the concept of study generalizability. While the experiment supports a causal conclusion, teachers can also discuss the extent to which the findings apply to other employers, industries, locations, or stages of the hiring process.
• The study covered in this lesson is similar to the audit study from Lesson 1.B.4, but this lesson has a different statistical focus. In Lesson 1.B.4, students focused on the principles of experimental design and how experiments support causal conclusions. Here, students focus on the size of the treatment effect and determine whether the observed difference is statistically significant. Making this connection can help students see the lesson as building on prior knowledge, rather than repeating it.
• Throughout the lesson, reinforce that confidence intervals do more than answer whether a difference exists. They also estimate the range of plausible values for the true effect size, laying the foundation for the Discussion Question about practical importance.
• Before releasing students to do the Practice problems, it’s helpful to point out that each question stem defines the order of subtraction for the proportions in the problem. Reinforcing this will help students’ responses have the same signs (positive or negative), which allows for easier collaboration and interpretation.

First, download this lesson's Handout Key and read through its Discussion Question section. Then, check out our model discussion norms and the additional background notes below.

• A 3.4 percentage point difference may initially appear modest. Encouraging students to consider the overall callback rate provides additional context. Because relatively few applications received callbacks overall, even a difference of a few percentage points represents a substantial relative advantage. As the original study noted: “a white name yields as many more callbacks as an additional eight years of experience” (pg. 3). Framing numerical differences in terms of equivalent differences among other variables – such as years experience – can be a helpful way to gauge practical importance.
• Multiple viewpoints about the broader causes of racial wage gaps may emerge during discussion. The statistical evidence from this experiment supports a causal conclusion about the effect of the assigned names in this particular hiring context. Broader questions about labor market outcomes may extend beyond the scope of this single study. These limits to the study’s generalizability will be explored in the Discussion Question of the next lesson (Lesson 3.B.3).

• This lesson is based on a landmark audit study (free working paper version here) by economists Marianne Bertrand and Sendhil Mullainathan. Researchers sent nearly 5,000 fictitious resumés to job advertisements in Chicago and Boston, randomly assigning each resumé either a commonly-white or commonly-black first name. The jobs represented a variety of occupations that required different levels of experience and education, strengthening the study's relevance across multiple hiring settings. The researchers used birth certificate records to identify the names that were most exclusively associated with each racial group. Multiple male and female names were used within each group, helping ensure that the results were not driven by any single name.
• The Bertrand and Mullainathan study became one of the best-known examples of an audit study. Since its publication, researchers have conducted numerous studies using the same basic audit study design in a variety of hiring contexts. Together, these studies illustrate how statistical evidence accumulates through repeated investigation.

• This lesson introduces an important conceptual shift. Although students are working with two populations, the parameter of interest is still a single quantity: the difference between the two population proportions. Reinforce that a difference of 0 represents no difference between the populations, while positive and negative values indicate which population has the higher proportion. This also provides a natural opportunity to emphasize the importance of clearly defining the order of subtraction before beginning a two-sample analysis.
• Technically, because this lesson describes an experiment with random assignment, the distribution used for inference is a randomization distribution rather than a sampling distribution. Randomization distributions describe the results of repeatedly reassigning treatments, whereas sampling distributions describe the results of repeatedly drawing random samples. For these procedures, however, the mathematical results are equivalent, so AP Statistics does not formally distinguish between the two.
• This lesson strengthens the connection between confidence intervals and hypothesis tests. As a useful rule of thumb, if a confidence interval for a difference contains 0, a corresponding hypothesis test will typically fail to reject the null hypothesis. If the interval does not contain 0, the corresponding test will typically reject the null. For two-sample procedures, this relationship is not exact, as intervals and tests use slightly different standard errors. However, it remains a valuable conceptual rule of thumb.