Exam 2 Lecture 2

Question 1

True story of being a scientist

Answer 1

Asked to critique a study that looked at a group of people who drink too much.

PROTOCOL: Heart rate measured
- During a relaxation exercise
- When they were showed photos of booze
- As they told a favorite story about drinking
- While they smell their favorite drink, and taste it
- Measured craving after all of this

RESULTS: Heart rate changed during the storytelling

CONCLUSION: Craving causes changes in heart rate measures

I don’t believe you!
- The heart and lungs are linked, when you inhale, your heart accelerates. What does talking do to your breathing?
When you concentrate, your heart rate (&BP) changes
Maybe concentrating on all this changes HR, not craving

Question 2

We need a way to vet whether a study was done correctly and whether the data are meaningful… whether we should believe what we read/see/hear

Answer 2

Validity testing

Question 3

When you are testing something that is vague, murky, complicated, multifaceted, squishy, subjective… most things being studied are… that something there needs to be some way to determine if you are actually testing

Answer 3

VALIDITY TESTING

Question 4

Using a measure that measures what you think you’re measuring

Answer 4

Validity

Does an online IQ test really measure intelligence?
Does the Myers-Briggs test really reflect who you are?
Before you believe a conclusion, you need to believe the process

Question 5

Designing a study that studies what you think you’re studying

Answer 5

Validity

Studying how well antidepressants work by studying differences in SAT scores between AD users and non-users. NOT VALID

Studying how people respond to social situations using a lab-based experiment where people watch movie clips about awkward social situations while hooked up to physiological sensors
MAYBE

Question 6

Before you believe a conclusion, you need to believe __________

Answer 6

The process

Question 7

Validity helps measure what?

Answer 7

The APPROPRIATENESS of your data and data collection protocol.
The validity of a measurement tool (questionnaire, equipment) must be tested. The validity of a study must also be tested.

There are many different types of validity.

Validity is NOT about precision. It’s about RELEVANCE

Question 8

Is validity about precision or relevance?

Answer 8

Relevance

Question 9

Validity of a measurement examples

Answer 9

You want to assess stress levels of college students. What data should you collect?

• Blood glucose
• How much deodorant they wear
• Answer to “you stressed rn?”
• Antibiotic use in past week
• A 106-item survey to measure stress
• Salivary cortisol at wake time

Implications to all of these, some don’t even assess stress levels, others are not helpful indicators/may generate too much noise/error compared to others

Question 10

Validity of a Study protocol

Answer 10

Am I actually answering the question I’m asking?

Ex: Is heart rate (HR) different after a stroke?
Protocol: Collect HR before, during, and after a stroke using an electrocardiogram
- a plethsymogram
- counting carotid pulse
- asking spouse about patient palpitations

Which are valid? They are all valid, but some of these measurement strategies vary in how precise/accurate they are.

Question 11

Validity (measuring what you think you’re measuring). When is it easier to demonstrate and when is it harder to demonstrate?

Answer 11

• Easier to demonstrate when the study is about tangible/concrete construct (body composition, enzyme activity, heart rate)
• Harder to demonstrate when measuring complex ideas (fatigue and stress), human behavior, and self-reported data (attitudes and feelings)- these days, we measure A LOT of complex things

Question 12

TESTING and PROVING validity is critical when measuring things that are

Answer 12

Hard to define, hard to measure, and hard to quantify

Question 13

What are the types of validity?

Answer 13

MEASUREMENT VALIDITY aka CONSTRUCT VALIDITY

and

STUDY VALIDITY

Question 14

What are the subcategories of MEASUREMENT AKA CONSTRUCT VALIDITY

Answer 14

• Content validity
• Face validity
• Criterion validity (Convergent and discriminant)

Question 15

What are the subcategories of STUDY VALIDITY

Answer 15

• Internal validity
• External validity

Question 16

What does MEASUREMENT VALIDITY AKA CONSTRUCT VALIDITY aim to ask

Answer 16

Are YOUR DATA relevant to your measure?

Question 17

What does STUDY VALIDITY aim to ask

Answer 17

Is your study design relevant to answer the question you are asking?

Question 18

Measurement Validity aka Construct Validity
What is the difference between constructs and indicators?

Answer 18

Construct= A concept or idea you want to measure
Sometimes it isn’t directly measurable, so you get a variety of relevant (but not exact) data called indicators/items

There are a variety of strategies to determine whether the indicators/items (aka- the data you actually collected) really capture the construct (aka the big idea that you were trying to measure/measurement goals)

Question 19

Construct vs. Indicators (Alcohol Example)

Answer 19

Construct= Alcohol misuse

Indicator= Frequency of use (days per week)
Indicator= Quantity used (drinks per occasion)
Indicator= Hangover severity (symptoms checklist)
Indicator= Consequences (negative outcomes checklist)
All of the indicators are used to determine the construct.

Question 20

Construct vs. Indicators (Exercise Difficulty Example)

Answer 20

Indicator= Knowledge (breath control)
Indicator= Experience (fitness history)
Indicator= Preparedness (good sneakers)
Indicator= Motivation (enthusiasm to try)

Question 21

Most psychological states require a bunch of questions to get at. This 21-item survey measures 3 psychological states. Explain.

Answer 21

21-item survey= 21 indicators
3 psychological states= 3 constructs

Question 22

To have construct validity, what three things must you test?

Answer 22

1. Face validity
2. Content validity
3. Criterion validity

Question 23

What is construct validity?

Answer 23

The way you are trying to answer your question makes sense based on what is currently known

Do the tools you are using to answer your question capture all relevant, current knowledge of the construct?

Constructs change definitions over time as knowledge expands
- Optimal nutrition (“good fat” etc)
- Autism is a spectrum (very complicated history)

Constructs require differentiation from similar constructs
- Anxiety versus depression
- Fatigue from overexertion, depression, sleep disorders, thyroid conditions

Question 24

Face Validity- a component of Construct Validity

Answer 24

Face validity= yep. makes sense
- On the surface, your measure seems appropriate for what you are trying to measure
- On the surface, it seems relevant and related

This is a more subjective measure of validity.
- This type of validity might vary depending on who is participating in the study and what the overall study design is
- This is one place where bias can sneak in
- Generational differences
- Racial/ethnic/cultural differences

Question 25

Face Validity Example

Answer 25

Asking “you stressed rn” on a stress survey has more face validity for younger people than older people.
Without face validity, there is no construct validity.
With face validity… ok, but that’s not enough

Question 26

Content Validity- a component of Construct Validity

Answer 26

Content validity= yep. that’s the whole story. Does your measure represent all aspects of your construct?
Does your measure include all relevant indicators?
To be a complete measurement tool, it must include all the different parts, components, perspectives.
It also must not include irrelevant things.

Question 27

Content validity Example

Answer 27

If my next exam does not include items about validity, it will not have content validity
If my next exam includes items about best practices for orthopedic surgeons, it will not have content validity.

Without content validity, you don’t have construct validity.

Question 28

Criterion Validity- a component of construct validity

Answer 28

Criterion validity= yep. true not matter how I measure it

• Study A measures stress by asking “you stressed rn”
• Study B gives a 106-item stress symptoms survey
• Study C collects salivary cortisol at wake time

If the same person does all 3 studies, would they say the same thing? This is criterion validity

To test criterion validity, you can use a statistical test. You want to measure the relationship between the studies and make sure a high stress person from Study A is also labeled high stress in Study B and Study C.

Question 29

Am I answering only the question I’m asking? What two types of CRITERION validity describe this (C+D)

Answer 29

1. Convergent validity (same construct, different measures)- Yep, the whole story is consistent.
2. Discriminant validity (different constructs, different measures)- Nope, we are not testing another similar thing. We are testing the right thing.

Question 30

Convergent Validity

Answer 30

Category under Criterion Validity

Convergent validity (same construct, different measures)
- Making sure that what you are measuring is related to other things it is known to be related to (Heart rate measured by ECG and counting from radial artery)
Yep, the whole story is consistent.

Question 31

Discriminant Validity

Answer 31

Category under Criterion Validity

Discriminant validity (different constructs, different measures)
- Making sure you aren’t accidentally picking up something else with your measure (Does the SAT measure intelligence or test-taking ability?)
- Making sure that two different constructs are not highly related (How can you be sure a person is anxious and not just stressed out?)
- Making sure things that shouldn’t be related aren’t related (A Division 1 soccer player shouldn’t have low fitness scores)
Nope, we are not testing another similar thing. We are testing the right thing.

Question 32

Concurrent vs. Predictive are 2 types of criterion validity. Describe concurrent validity.

Answer 32

Concurrent: A new measure/indicator is similar to the ‘gold standard’.
- You have created a new technology that measures step count.
- Does it get the same answer as FitBit? If so, has concurrent validity.

Question 33

Concurrent vs. Predictive are 2 types of criterion validity. Describe predictive validity.

Answer 33

Predictive: Can a measure/indicator (meaningfully) predict a future outcome?
- A moving company measures how much a person can deadlift because they need workers who can lift heavy items. If people who can deadlift more get jobs done quicker, then deadlifting has PREDICTIVE VALIDITY (of job success)

Question 34

Criterion Validity- This is a strong form of construct validity but often hard to come by because of what?

Answer 34

1. You need to measure more things which complicates the study design
2. You might find out the different measures aren’t all related, or only partially related. Then what? Which one is ‘correct’?
   - You really need a ‘gold standard’ but that doesn’t always exist.

Question 35

The Borg Scale

Answer 35

The Borg Scale= rating of perceived exertion
How hard is this exercise?
- On the surface, it makes sense that the more you feel you are exerting, the more difficult it is.
- If you agree with this, you’d say that the Borg scale has face validity.

A little weird though.
How hard is this exercise? “6”, not hard at all
Weird because the ratings go from 6-20

The 6-20 scale is intentional! To line it up with HR (gold standard)
If not, criterion validity is poor.

Question 36

Complex constructs require __________

Answer 36

Multiple indicators!

Does RPE/BORG score completely capture the construct of difficulty?
If not, content validity is poor.