• When setting up the World War II problem, invest time into telling Abraham Wald’s story and why solving the bomber problem — like every problem he solved during the war — was personal for him. Telling his story is key to investing students in this historic problem. For a model, see how Mr. Young-Saver shares the story from 2:59 - 4:16 in the lesson video. In addition, the segment from 5:04 - 6:23 in the video also shows a model of the “big reveal” of Wald’s answer and the reasoning behind it.
• In the lesson video, Mr. Young-Saver also shares how Abraham Wald’s story relates to his own family’s personal history. This isn’t necessary to share when you’re presenting the lesson in your own class. But if you assign students to watch the lesson video, this can provide another powerful moment.
• Abraham Wald’s key insight was that the sample of planes that returned from fighting was not representative of the full population of planes that embarked on the bombing mission. In statistics, we called this kind of unrepresentative sample a biased sample. We’ll cover the idea of bias thoroughly later in the course. For now, rather than add more vocabulary (e.g. “bias”), it’s best to simply emphasize that samples can sometimes provide distorted pictures of populations. This idea will return again in the lesson’s Discussion Question.
• Make sure to spend time on the most subtle variables to classify in the student data set: shoe size and zip code. The shoe size example helps students see that values with decimals can still be discrete. The zip code example helps students see that values with numbers can still be categorical.

First, download this lesson's Handout Key and read through its Discussion Question section. Then, check out our model discussion norms and the additional background notes below.

• It’s important to emphasize that Palo Alto College is a community college. One key function of community colleges is to be a stepping stone towards transferring to a four-year college. So, both outcomes described here - obtaining full-time work and proceeding to the next level of schooling - are generally considered positive outcomes.
• When brainstorming ways that the statement may be misleading, students sometimes discuss whether the “full-time work” consists of graduates’ first-choice careers or merely “fallback” jobs outside their fields of study. Students also sometimes bring up whether the “next level of schooling” consists of graduates’ first-choice schools or “fallback” schools. These are valid questions and could point out additional issues beyond the practice of sampling just the graduates.
• Graduation rate data can be found for almost every U.S. college through the IPEDS website. If there’s a college near your community that shares job placement rates only among graduates, you can use the IPEDS website to look up its graduation rate. If the graduation rate is low, consider replacing the Palo Alto College example with the more local college, to add extra relevance for students.

• Abraham Wald’s original analysis utilized far more than 10 planes. In the lesson, we use the hypothetical example of 10 planes that went on a specific bombing mission (6 of which returned) in order to provide a simple example of sample size and population size for students.
• The nose is one of several possible places where Abraham Wald may have actually told the Allied Forces to put extra armor. The only public reprint of Wald’s original analysis can be found here. In the paper, Wald shares a “hypothetical example” in which the engines are the most vulnerable part of the planes. However, this example is hypothetical - the real data may have been classified at the time. That real data is either still classified or lost to history. So, you’ll find variations of the airplane problem in which different areas are identified as the vulnerable regions on the planes. In our lesson, we chose the nose to honor Mr. Young-Saver’s former professor Joseph Blitzstein, who uses the nose in his retelling of the problem. Ultimately, every retelling has the same conclusion: we should put armor where we observe the least shots - not the most.

• The Abraham Wald example highlights an idea that we’ll formally introduce later in the course: sampling bias. The planes that returned are not representative of all the planes that embarked on the mission. This bias informs Wald’s conclusion to reinforce the areas with the least shots – not the most. The specific type of sampling bias on display here is called undercoverage bias – bias that arises when “the sampling method fails to include part of the population or a part of the population is less likely to be selected based on the sampling method” (AP Statistics CED, Topic 1.12). This is the same type of bias on display in the Discussion Question, as the sampling method fails to include the students who don’t graduate. We’ll discuss bias later in the course. It does not need to be explicitly defined in this lesson.
• The student data set in the lesson is provided in tidy format. Students in AP Statistics do not need to know what tidy format is; however, it’s helpful for instructors interested in preparing their own data sets to be familiar with tidy format. Data provided in tidy format follow three basic rules (shown below). When these rules are not followed, data sets can be difficult to manipulate, visualize, and analyze. Here are the rules of tidy data:
• Every column is a variable.
• Every row is an observation.
• Every cell is a single value.
• Note that the far left column in the student data set, which shows the student names, is not a variable. Rather, it’s a unique identifier. Variables are descriptive values that vary between observation units (e.g. height, dominant hand, shoe size). It’s possible for two observational units to share the same variable values (e.g. two students share the same height or shoe size). Unique identifiers are non-descriptive names, codes, or numbers whose sole function is to identify a unique observation in a data set. In this case, the names provide a way to uniquely identify each student in the data set.