Module 7, Evaluating Quantitative Research

Question 1

Reliability - What is a Reliable Instrument?

Answer 1

an instrument that gives the same measurements when you repeatedly measure the same unchanged objects or events
- a test must be reliable for it to be valid!

my notes:
- if you were to step on a scale 5 times within 30 seconds of each other and you get the same weight from the scale, we would say those scores are reliable and consistent (getting same measurement)
◦ it is important we would not
expect the weight to change
within 30 seconds but if we
do it 5 times throughout the
day it is possible the weight
could change
- must produce consistent scores when we do not expect anything to change in order for it to be accurate (if the weight was different each time, the scale would be unreliable and not accurate

Question 2

Internal Validity

Answer 2

the extent to which the change in the dependent variable can be attributed to the manipulation of the independent variable
- what is really the manipulation of the independent variable that caused the change in the DV or was there other factors?
- can we establish cause and effect
- extraneous variables are alternative explanations (do not want to see them show up)

Question 3

External Validity

Answer 3

extent to which researchers can generalize their findings to other people, situations and times
- people is population validity (is the extent to which we can generalize our findings to other people)
◦ example: do the findings
generalize from adolescents
to young adults?
- situations is ecological validity (whether we can generalize the findings to other situations or contexts)
◦ example: does a lab based
study or intervention, what
that study generalize to be
able to say more real world
situations where exercises
are actually working out in a
gym
- times (different time periods) - do not consider much but we need to
◦ different time periods bring
about different attitudes and
beliefs and so research done
earlier on may not actually
be applicable now
◦ example: study interested in
woman participation in
sports done in 1980 (may not
be applicable in today’s
social context)

Question 4

Internal Vs. External Validity

Answer 4

• want to maximize both but very challenging to do both in one study
  trade-off between internal and external validity
• the more you maximize one, you have to give up the other just as much
• when researchers try to maximize internal validity, they try to control for extraneous variables or alternative explanations for findings - in doing so what happens is the study starts to look less like the real world or they might start to restrict the inclusion criteria
• if a researcher is trying to maximize external validity and the study to look more like real world where it is generalizable then the researcher has to give up tight control; where they move from lab based to real world context like a gym
  series of experiments (studies)
• each study has a specific goals and limitations
• real world intervention vs. lab based intervention (more of a continuum
• you would apply your lab based results to the real world

Question 5

Threats to Internal Validity

Answer 5

what is going to threaten our ability to establish cause and effect
- experimental procedures, treatments, or experiences of the participants that threaten the researcher’s ability to draw correct inferences from the data about the population in an experiment

Question 6

10 Threats to Internal Validity

Answer 6

history, maturation, testing (reactivity, warm-up effect and loss of naivety), instrumentation, selection bias, experimental mortality, selection-maturation interaction and expectancy

Question 7

Threats to Internal Validity - History

Answer 7

events occurring in the course of experiment that cause changes in the DV and are not the intervention (IV)
- an unplanned event that coincides with the IV could be responsible for the observed changes in the DV (true cause)
- not history of participant rather coinciding

Question 8

Threats to Internal Validity - Maturation

Answer 8

processes within the participants that operate as a result of time passing
- eg. DV is a physical fitness test (beep test)
my notes:
- may have nothing to do with PA rather they are just physically maturing
- looking at expected or anticipated physiological changes or maturation over time but it does not always mean improvement could also mean decline
- natural development is key

Question 9

Threats to Internal Validity - Testing (Reactivity)

Answer 9

the effects of one test on subsequent administrations of another test
- the pre-test is affecting post-test not the intervention
reactivity: exposure to pretest changes behaviour
- eg. self-monitoring - athletes increase their effort simply because they were asked to record it (apple watch is given and all of a sudden you start moving around more) - impact of pre-test

Question 10

Threats to Internal Validity - Testing (Warm-Up Effect)

Answer 10

do better on the subsequent measure/test because they are more familiar with it
- they are better at the test not that their aerobic capacity has increased for yo-yo test
- based on a performance measure, it is not filling out a questionnaire on emotion regulation, or wellbeing (this is an assessment) - only an issue for performance measure like yo-yo

Question 11

Threats to Internal Validity -Testing (Loss of Naivety)

Answer 11

people start catching on to what you are testing/measuring
- example: we do not tell them we are measuring self compassion because that will impact how they are answering the questions themselves

Question 12

Threats to Internal Validity - Instrumentation

Answer 12

quantitative research relies on measurement:
- eg. leadership style, MRI, motivation etc.
instrumentation:
- changes in instrument calibration, including lack of agreement within and between observers
- it is important that the data acquired from instruments be accurate or else the interpretation of results will be inaccurate

Question 13

Threats to Internal Validity - Selection Bias

Answer 13

choosing comparison groups in a non-random manner (not randomly assigned)
pre-existing differences between groups - functions as confounding or extraneous
- control groups and experimental group - identified extraneous variables - you allow people to pick which group (people who are more motivated about PA will sign up for experimental group)

Question 14

Threats to Internal Validity - Experimental Mortality

Answer 14

loss of participants from comparison groups for non-random reasons (participants decide the study is not for them and they drop out)
- participants drop out and those who remain tend to be more motivated (results in higher scores on post-test)
example:
- control and experimental group both have 50 participants and at the pre test they are scoring same amount of moderate to vigorous activity but at the end of the study you find that the experimental group has more moderate to vigorous activity than control group
- since these are based on averages it could be that in the experimental group who were doing less activity in pre-test are less motivated in first place so they drop out of the study (did not like PA intervention) and you are left with people who were already scoring high, thus it will inflate your post test scores

Question 15

Threats to Internal Validity - Selection-Maturation Interaction

Answer 15

the passage of time affects one group but not the other in nonequivalent group designs (groups have not been formed randomly)
- could be the case if we are studying teams or children (can be hard to put people randomly into groups)
- may randomly assign teams, classes, groups as compared to people individually
- randomly assign coaches - does not have to do with the intervention (leadership) - maybe the field they plan on or maturation

Question 16

Threats to Internal Validity - Expectancy

Answer 16

researchers anticipating that certain participants will perform better
- people expectations will impact peoples motivation, effort, behaviour
- if people act people to perform better they may interact with them differently (encouraging) ~ could be sub-conconscious

Question 17

3 Ways to Control Threats to Internal Validity

Answer 17

randomization, placebos, blind & double-blind studies and standardize experiments and instruments as much as possible

Question 18

Controlling Threats to Internal Validity - Randomization

Answer 18

• random assignment – controls for history
• matched pairs – e.g, age, gender
  ◦ if we cannot do random
  assignment - good
  secondary option
• randomizing treatments or counterbalancing
  ◦ ABC v. ACB v. BAC v. BCA v.
  CAB v. CBA
• interaction of timing - example: randomized other of clips because they wanted to figure out if features for penalties were valid not the order the clips came in (just getting at information)
  my notes:
• random assignment is equal probability of an individual being placed into any level of the IV
• on average, the groups are the same on a IV (everyone in the groups is not same but on average-
• random assignment is equal probability of an individual being placed into any level of the IV
• on average, the groups are the same on a IV (everyone in the groups is not same but on average the groups are the same) (trying to even out any pre-existing differences) which controls for a lot of extraneous variables - internal validity is impacted
  ◦ any changes in the dependent variable or differences between group and dependent variable can be attributed back to the IV - random assignment is best tool to control for threats to internal validity

Question 19

Controlling Threats to Internal Validity - Placebos, Blind and Double-Blind Studies

Answer 19

helps reduce experimenter bias/participant expectancies

Question 20

Placebo

Answer 20

often times people think they are receiving the treatment and will report improvements even though they received placebo (its psychological)
- do not want the participant to know what group they are in - whether they are getting treatment or not
- intervention group is getting aerobic capacity and the other group is not getting it but in order to blind it they could do PA that is less aerobic, like stretching, yoga etc. - this is called an attention control group
- attention control group: self-compassion study - she had another group, an attention control group (not impact IV but are doing something) - they did writing activities (around sport) whether the writing was impacting the outcomes as imposed to the self-compassion writing activities (intervention)

Question 21

Single Blind

Answer 21

single blind study either the participant does not know what group they are in (typical) or rare scenario whether people who are collecting data do not know what group they are in
- single blind is typically when research participant is unaware of an aspect of the study, so for example a participant would not be aware when they are taking the placebo medication and when they are taking the actual medication

Question 22

Double Blind

Answer 22

• in double blind study both the participants and the researcher who is collecting data or administrating test do not know what groups they are in (both)

in a double blind study neither the participant or investigator know that information and double blind can sometimes be more difficult to do but would be more robust from a research point of view
- this way participants cannot use any preconceptions or biases while in the study as they are unaware and investigators need to be blinded so they do not ask questions or treat the participants differently based on whether they are getting air pollution or filtered air

Question 23

Standardize experiments and instruments as much as possible

Answer 23

eg. script every researcher uses, everything in the same order, make sure instruments are calibrated after every test

Question 24

Threats to External Validity

Answer 24

arise when experimenters draw incorrect generalization from the sample data to other persons, other settings, and past or future situations

Question 26

Threats to External Validity -Reactive/Interactive Effects of Testing

Answer 26

the pretest may make the participant more aware of or sensitive to the upcoming treatment
- as a result, the treatment is not as effective without the pretest
- in the study the pretest may be driving folks to reflect on their behaviours and how they approach situations for example and make them more likely to transform and work on certain things through the intervention (more engaged with intervention)
◦ intervention has bigger
impact on them
◦ when you take it out of
research context without
pretest, they may not take it
as seriously
- the findings do not generalize to context without pretest

Question 27

Threats to External Validity - Interaction of Selection Bias & the Experimental Treatment

Answer 27

when a group is selected on some characteristics, the treatment may work only on groups possessing that characteristic
- eg. gender, age, sport, nationality
- population validity
- so context specific to what it is you are studying
- Game Plan is more so for Canadian athletes, might not be as effective in Sweden as there are different education, pensions, etc.

Question 28

Threats to External Validity - Multiple-Treatment Interference

Answer 28

when participants receive more than one treatment, the effects of previous treatments may influence subsequent ones
- we cannot isolate specific movements for internal validity, becomes challenging because we get multiple treatment interference
- not only one treatment is causing change rather the combination of treatments combined
- example: aerobic capacity intervention is not as effective as if you are building muscular strength at the same time

Question 29

Controlling Threats to External Validity (2 mechanisms)

Answer 29

selecting participants, treatments, experimental situation, and tests to represent a broader population
- random selection: based on our inclusion criteria (we can open it up: to generalize but we give up our ability for control for internal validity)
researcher might restrict claims about groups to which the results cannot be generalized
- conduct additional experiments with groups with different characteristics

my notes:
each research study is limited by its scope and the number of people you can reach
- we need to have series of studies, select people who are representative of the community we are trying to reach