• Although this lesson analyzes the same study as a prior lesson (Lesson 4.7), the materials are designed without presuming that students have seen the prior lesson. Therefore, instructors can use this lesson even if they haven’t used the prior lesson. However, instructors who have used the prior lesson with their students can reduce the discussion of design of the study to a brief resurfacing.
• Because this study is a well-designed experiment with random assignment to treatment, the observed difference in callback rates can be interpreted as a causal effect from the assigned names. That said, the underlying mechanism behind the name advantage is less clear. Intentional racial discrimination is one possible explanation. However, it’s also possible that the differences in callback rates could arise from unintentional biases. For example, commonly-white names may tend to be more familiar for the people making hiring decisions, so hiring managers may unconsciously gravitate to such names. Framing these considerations with students is important for interpreting the study and provides a natural opportunity to revisit the concept of study generalizability. While the experiment supports a causal conclusion, instructors and students can also discuss the extent to which the findings apply to other employers, industries, locations, or stages of the hiring process.
• Before releasing students to do the Practice problems, it’s helpful to point out that each question stem defines the order of subtraction for the proportions in the problem. Reinforcing this will help students’ responses have the same signs (positive or negative), which allows for easier collaboration, interpretation, and checking of solutions.

First, download this lesson's slide deck and handout key to see the prompt and sample responses for the Lesson Starter. Then, check out the additional background notes below.

Instructional routine: Ten-Minute Talk. The lesson provides space for students to jot down their thoughts on the prompt, before discussing with a partner and engaging in whole group discussion. You can find more background on implementing a Ten-Minute talk here.

Purpose & Background: Now that we have come to the final lesson of the course’s statistical inference topics (final lesson for hypothesis tests and confidence intervals), this 10-minute talk provides students with a chance to reflect on their learning. Encouraging students to look through their notes for Unit 4 (Hypothesis Tests) and Unit 5 (Confidence Intervals) may help elicit more precise responses. Instructors can use the students’ responses to design review / synthesis days before the end of unit assessment.

First, download this lesson's handout key and read through its Discussion Question section. Then, check out our model discussion norms and the additional background notes below.

• In general, when the null hypothesis is outside a confidence interval (i.e. the null is not a plausible value), the corresponding hypothesis test would reject it. When the null hypothesis is within a confidence interval (i.e. the null is a plausible value), the corresponding hypothesis test would fail to reject it. This relationship is true in most cases where the significance level (e.g. α = 0.05) of a two-sided hypothesis test is the complement of the confidence level (e.g. 95% confidence).
• The above relationship is not always true, especially when utilizing simulation-based intervals and hypothesis tests. Because results vary across simulations, intervals or tests that are borderline significant (p-values very close to 0.05 or confidence intervals very close to 0) should have more simulations run than usual (e.g. run 1,000 simulations instead of 100) in order to establish a more robustly defined p-value or confidence interval.
• Multiple viewpoints about the broader causes of racial wage gaps may emerge during discussion. The statistical evidence from this experiment supports a causal conclusion about the effect of the assigned names in this particular hiring context. However, broader questions about labor market outcomes may extend beyond the scope of this single study and beyond the scope of this course.

• This lesson is based on a landmark audit study (free working paper version here) by economists Marianne Bertrand and Sendhil Mullainathan. Researchers sent nearly 5,000 fictitious resumés to job advertisements in Chicago and Boston, randomly assigning each resumé either a commonly-white or commonly-black first name. The jobs represented a variety of occupations that required different levels of experience and education, strengthening the study's relevance across multiple hiring settings. The researchers used birth certificate records to identify the names that were most exclusively associated with each racial group. Multiple male and female names were used within each group, helping ensure that the results were not driven by any single name.
• The Bertrand and Mullainathan study became one of the best-known examples of an audit study. Since its publication, researchers have conducted numerous studies using the same basic audit study design in a variety of hiring contexts. Together, these studies illustrate how statistical evidence accumulates through repeated investigation.

• Although students are working with two samples in each problem of this lesson, the parameter of interest is still a single quantity: the difference between the two population proportions. Reinforce that a difference of 0 represents no difference between the populations, while positive and negative values indicate which population has the higher proportion. This also provides a natural opportunity to emphasize the importance of clearly defining the order of subtraction before beginning a two-sample analysis.
• To generate the types of computer simulations we use in the lesson, our preferred tool is stapplet. For simulations comparing two proportions, utilize the One Categorical Variable, Multiple Groups applet. Input the observed data. Then, scroll down to the inference section and choose simulation.
• Technically, because this lesson describes an experiment with random assignment, the distribution of simulations is a randomization distribution rather than a sampling distribution. Randomization distributions describe the results of repeatedly reassigning treatments, whereas sampling distributions describe the results of repeatedly drawing random samples. For these procedures, however, the mathematical results are equivalent, so it is not necessary to formally distinguish between the two.