CHAPTER 16 Measure Your Mission Flashcards
Why is it important to measure outcomes and treatments that correspond to your mission?
To ensure that improvements are not misleading and accurately reflect the achievement of your goals.
What can happen if you measure an outcome in an incomplete way?
Apparent improvements may be misleading.
What should you consider when applying lessons drawn from data to a new context?
Whether the contexts are sufficiently similar for the lessons to hold.
What is a potential issue when trying to use a relationship to achieve a goal?
The relationship may disappear once you try to use it.
What is a common issue when measuring outcomes related to a mission?
Measuring an outcome or treatment that doesn’t correspond to what you are really interested in.
What are partial measures in the context of measuring missions?
Measures that capture only part of the overall mission, like standardized test scores for educational achievement.
What is a potential problem with partial measures?
Improvements on one dimension might coincide with losses on other dimensions.
What is a simple reason for negative correlations across dimensions of a problem?
Resource constraints.
What is strategic adaptation?
Efforts to improve outcomes on some dimensions lead people to adjust their behavior to circumvent those efforts.
What significant security measure was implemented in U.S. airports in the early 1970s?
Metal detectors.
What might be a misleading conclusion drawn from the reduction of hijackings after installing metal detectors?
It may not reflect overall security improvements, as other types of terrorist attacks could increase.
What are intermediate outcomes?
Steps along the path of the mission that are easier to measure than the ultimate outcome.
Why might measuring intermediate outcomes be problematic?
They may not accurately reflect the ultimate mission or final outcome.
What is an example of an intermediate outcome in medical research?
Blood pressure as a predictor of heart attacks.
What is one issue with using tumor size as an intermediate outcome in cancer research?
It may not accurately predict mortality since many tumors are not harmful.
What should researchers consider when interpreting the relationship between intermediate outcomes and their missions?
The clarity of the causal relationship between the intermediate outcome and the actual mission.
What does it mean for a mission to be ill-defined?
There may be multiple reasonable ways to measure it, impacting the choice of outcomes and treatments.
What can be misleading when following the strategies of the richest people to maximize earnings?
Assuming that dropping out of college and starting a tech company guarantees success.
What concept is illustrated by examining the correlation between dropping out of college and wealth?
Correlation does not imply causation.
What is p-hacking?
Studying a small population and looking for commonalities that may not represent a true relationship.
What is p-hacking?
Engaging in questionable research practices to find statistically significant results
P-hacking involves manipulating data or analyses to achieve desired outcomes.
Why might dropping out of college to start a tech company be a bad idea?
It may lead to serious debt and doesn’t guarantee higher expected earnings
Correlation observed may not imply causation.
What is the primary concern when measuring outcomes for decision making?
Correctly measuring your mission and defining what outcome you care about
This includes considering expected earnings versus probability of extreme wealth.
What can be a consequence of dropping out of college?
Increased likelihood of being in serious debt
It may also reduce expected earnings.
What is the difference between internal and external validity?
Internal validity pertains to unbiased estimations; external validity concerns the applicability of results across contexts
External validity is crucial for applying findings from one context to another.
What was the conclusion of the study by Burke, Hsiang, and Miguel regarding climate change?
Unmitigated warming is expected to reduce average global incomes by 23% by 2100
This conclusion is based on the effects of temperature fluctuations on GDP growth.
What is a selected sample?
A sample of observations not drawn randomly but selected based on specific characteristics
Selected samples may not represent the broader population.
What issue did the Bangladesh Integrated Nutrition Project face compared to the Tamil Nadu Integrated Nutrition Project?
Cultural differences in decision-making authority regarding food
The BINP failed to target key decision-makers as effectively as the TINP.
Why might standardized test scores not correlate with college performance among enrolled students?
Because the enrolled students are a selected sample with other strong characteristics
This can lead to a weak correlation in the selected sample despite test scores being good predictors among all applicants.
What is the batting average of major league pitchers compared to non-pitchers?
Pitchers: .125; Non-pitchers: .259
This reflects the significant difference in hitting ability between the two groups.
What does external validity assess?
Whether a relationship estimated in one context will hold in another context
It is essential for applying research findings appropriately.
What is a common mistake in decision-making regarding educational outcomes?
Studying the wrong outcome, like correlates of extreme wealth instead of expected earnings
This can lead to misguided conclusions about educational choices.
What is the hitter rule in baseball?
Specifically so that pitchers don’t have to bat.
Why are major league pitchers considered poor hitters?
They spend so much time practicing pitching that they don’t practice hitting.
What was the batting average for pitchers compared to non-pitchers in high school baseball?
.322 for pitchers and .317 for non-pitchers.
What trend is observed in the correlation between pitching and hitting ability as players age?
It changes from slightly positive in high school to negative in professionals.
How did the batting averages of pitchers and non-pitchers compare from 1871 to 2017?
In the nineteenth century, they had comparable batting averages; in the twentieth century, pitchers’ hitting declined relative to non-pitchers.
What is a suspected reason for the changing correlation between pitching and batting ability over time?
It has to do with selected samples and increasing selectivity in player recruitment.
What is meant by ‘selected samples’ in the context of baseball players?
Players are chosen based on their above-average skills in hitting and pitching.
What happens to the correlation between pitching and batting ability as selectivity increases?
It can turn from positive to negative.
What does the term ‘strategic adaptation’ refer to in the context of the window tax?
The behavior change of homeowners to minimize tax burden by altering their properties.
What was the purpose of the window tax established in England in 1696?
To raise revenue in a way that taxed wealthier individuals without invading privacy.
What impact did the window tax have on architecture over time?
Larger homes began to include fewer windows to reduce tax liability.
What is the ‘shift’ strategy in baseball?
A defensive adjustment based on data showing where batters are likely to hit the ball.
How did the introduction of big data affect defensive strategies in baseball?
Teams began using spray charts to optimize defensive setups for each batter.
What correlation was observed when teams started using the shift strategy?
A negative correlation between using the shift and runs allowed.
What was the percentage of shifts used by the Tampa Bay Rays in 2010?
10 percent of all shifts.
By what year had the number of shifts in Major League Baseball grown significantly from 2011?
2014.
Fill in the blank: The correlation between pitching and hitting ability in the general population is likely to be _______.
positive.
True or False: The correlation between hitting and pitching ability in high school players is negative.
False.
List the three types of players needed to make a Major League Baseball team.
- Amazing hitters
- Amazing pitchers
- Great pitchers who can also hit.
What is the negative correlation observed in baseball between using the shift and runs allowed?
Teams that shifted allowed fewer runs.
This correlation inspired teams to implement the shift more frequently.
How many shifts were used in Major League Baseball in 2011?
Approximately 2,000 shifts.
This number increased significantly in the following years.
What was the number of shifts used in Major League Baseball by 2016?
Over 28,000 shifts.
This indicates a dramatic increase in the use of shifting strategies.
What adaptation did batters make in response to the shift?
Batters began hitting more balls to the opposite side and in the air.
This adaptation reduced the effectiveness of the shift.
What is strategic adaptation?
Changes in behavior that result from an attempt to avoid the effects of a change in someone else’s behavior.
What was the effect of the U.S. government’s offensive against Colombian drug cartels in the 1980s?
Drugs stopped flowing through the Caribbean and Florida and began entering the U.S. through Mexico instead.
What percentage of cocaine in the U.S. entered through Mexico by 1989?
One-third of all cocaine.
This figure increased to one-half by 1992 and 90 percent in subsequent years.
What was the estimated worth of the Mexican drug trade by the mid-1990s?
Roughly $20 billion.
This was significantly larger than Mexico’s largest legal commodity, oil.
What does internal validity refer to?
An estimate is internally valid if it is a credible estimate of the estimand.
What does external validity refer to?
An estimate is externally valid if there is good reason to think the relationship will hold in a context other than the one from which the data is drawn.
What is a selected sample?
A sample of data that wasn’t drawn at random from the population of interest but rather was selected to be studied because it possessed some particular set of characteristics.
True or False: The correlation that inspired teams to shift in baseball remains strong today.
False.
Hitters have adapted, reducing the correlation between shifting and runs allowed.
Fill in the blank: Strategic adaptation can create both _______ and changes in behavior.
partial measures.
How did the U.S. government’s actions impact the drug trafficking routes in the 1980s?
They led to a shift from the Caribbean to Mexico for drug trafficking.
What was a significant consequence of the adaptation by drug organizations in Mexico?
Increased violence and fragmentation among Mexican drug trafficking organizations.
What is the relationship between high-stakes testing and educational outcomes?
High-stakes testing is a partial measure of educational achievement.
Improvements in test scores may not reflect overall educational improvement.
What might explain why students assigned to popular instructors perform worse in subsequent math classes?
The focus on immediate performance may lead to gaps in foundational knowledge.
This relates to failing to measure the overall mission of educational success.
What is the primary critique of using passing rates as a measure of educational success?
Passing rates can be misleading and do not capture the full picture of educational outcomes.
What does the term ‘partial measures’ refer to?
Measures that do not fully capture the entirety of a mission or objective.
