• It’s important to start the lesson by explicitly naming several potential confounding factors at play (e.g. number of applicants, applicant qualifications, types of labs). That way, students can directly connect the experimental design choices in the Yale study to controlling/balancing these confounders.
• It’s worth noting that, when analyzing gender gaps in STEM, most of the data collected in the field measures gender as binary (male or female). To represent the full STEM workforce, there needs to be more data collected on STEM professionals who don’t identify by this binary. However, even if the data we analyze in this lesson is incomplete, it’s still important to analyze trends among the data we already have.
• Just as it was important in the sampling lessons to draw a distinction between bias and variability, it’s also important in this lesson to draw a distinction between confounding and variability. In particular, emphasize that random assignment reduces confounding, not variation. In addition, replication reduces variation, not confounding.

First, download this lesson's Handout Key and read through its Discussion Question section. Then, check out our model discussion norms and the additional background notes below.

• This question provides an excellent opportunity to loop back in the topic of generalizability. Because the Yale study measures gender gaps in a specific context – science faculty evaluating written lab manager job applications – its results may not fully generalize to other parts of the hiring process (e.g. interviews) or other STEM positions (e.g. non-research STEM positions).
• Hence, the information we learn from the Yale study does not, on its own, provide conclusive evidence about whether there are other factors that influence gender gaps in STEM. So, a wide variety of opinions on this question are consistent with the study results and can be expected in class conversation.
• Although the differences measured in the study are statistically significant (so large that they’re unlikely to happen by chance alone), students may also wonder if they are practically important (so large that they have a felt impact in the context of hiring). Generally, the 0.8-0.9 point differences on the 7 point scales can be considered practically important. Especially in a competitive job market, gaining almost an additional full point out of 7 could be a substantial advantage.

• Full citation for the Yale study: Moss-Racusin, C., Dovidio, J., et al. “Science faculty’s subtle gender biases favor male students.” PNAS October 9, 2012 109 (41) 16474-16479; https://doi.org/10.1073/pnas.1211286109
• The Yale study is an example of an audit study. Audit studies are field experiments in which identical artifacts (e.g. application materials) are sent to evaluators, with only one attribute changed (e.g. the applicant name). Differences in results can then be attributed to the changed attribute. Audit studies are often used in the context of testing for discrimination. The most famous audit study is the 2004 study by Marianne Bertrand and Sendhil Mullainathan, in which the researchers sent identical resumes to employers, but changed the name to be either a White-sounding name or a Black-sounding name (based on birth certificate records). The researchers found a statistically significant difference in the rate of callbacks received by each name.
• The use of the Yale study in our curriculum was inspired by conversation with authors of Advanced High School Statistics, which also uses the experiment as an example in their textbook.

• The Yale study is an example of a completely randomized design experiment. In the next lesson (1.B.5), we’ll discuss other types of experimental designs and formally define the completely randomized design. However, the focus of this lesson is on the principles of experimental design that apply to all types of experiments.
• Although experiments unlock the ability to make causal inferences, they can also be expensive and difficult to implement. Because of the cost, researchers often have to consider tradeoffs between causal inference and generalizability. For instance, it’s relatively easy and inexpensive to gather observational data about gender gaps across many STEM fields and positions. However, this observational data doesn’t allow for causal conclusions. By contrast, the Yale experiment allows for causal conclusions. However, its results can’t be generalized to unstudied parts of the hiring process (e.g. interviews), other types of STEM positions (e.g. non-research STEM positions), and other STEM fields (the researchers only recruited biology, chemistry, and physics professors to participate in the study).
• Quasi-experiments or natural experiments describe specific scenarios in which “as good as random” variation is naturally found in observational data. In these specific scenarios, researchers can use advanced methods to draw causal conclusions from observational data. In fact, Harvard researchers used a quasi-experiment to determine that Skew The Script had a positive causal impact on AP Statistics test scores in Texas. That said, these methods are beyond the scope of AP Statistics. In AP Statistics, causal conclusions can only be made from experiments. They cannot be made from observational data.