Week 3 - Research Design

Question 1

What is a cross-sectional Correlational Survey?

Answer 1

• Examines relationships between variables (personality, demographics, behavior)
• a type of cohort study where only one comparison is made between exposed and unexposed subjects.

Question 2

What is a Longitudinal Survey (no manipulation)?

Answer 2

• Examines changes over time (different pathways)
• a correlational research study that involves repeated observations of the same variables over long periods of time, often many decades.

Question 3

What is a quasi-experimental design?

Answer 3

• Compares different existing groups with different
  treatments/experiences
• over time: add longitudinal component
• Researchers control administration of treatment (when and to whom) AND collection of dependent measures
• Random Assignment to Condition is not possible

Question 4

What is an experimental design?

Answer 4

• testing for casual influence of X on Y

Question 5

Elements of the experimental design?

Answer 5

• manipulation of IV (the cause)
• measurement of DV (the effect)
• random assignment to condition (removes self-selection problem)
• control over extraneous variables (removes other differences between groups)

Question 6

What is internal validity?

Answer 6

the extent to which one can assume that differences in dependent variable measures are due to the manipulation of the independent variable, and the independent variable alone

Question 7

What is external validity?

Answer 7

refers to the generality of an effect, to the populations and settings to which it can be applied

Question 8

What is the goal of non-experimental research designs?

Answer 8

to overcome their inherent weaknesses…
• Provide some points of comparison
• Try to eliminate alternate hypotheses

Question 9

Advantages and disadvantages of correlational designs?

Answer 9

Advantages:

• no manipulation
• can be quick
• provides basis for theory building

Disadvantages:

• does not test cause and effect
• 3rd variable problem

Question 10

What is the 3rd Variable Problem?

Answer 10

occurs when an observed correlation between two variables can actually be explained by a third variable that hasn’t been accounted for

Question 11

What is a cross-lagged panel design?

Answer 11

• a study of the relationships between two or more variables across time in which one variable measured at an earlier point in time is examined with regard to a second variable measured at a later point in time

Question 12

Advantages and Disadvantages of cross-lagged panel design?

Answer 12

Advantages:
- Solves “directionality” problem

Disadvantages:
- Still has “3rd Variable” problem

Question 13

What is a pseudo experimental method?

Answer 13

unlike a true experiment, a quasi-experiment does not rely on random assignment. Instead, subjects are assigned to groups based on non-random criteria.

pseudoexperimental methods are unable to control for extraneous variables, such as: specific interfering events, passage of time, presence of the tester, changes in the tester over time, or participant attrition

• do not involve ‘treatment’ or manipulation of the IV
• do not involve appropriate standards of comparison
• no random assignment to condition

Question 14

What is a One Shot Case Study (a.k.a post-test only)?

Answer 14

a single group is observed on a single occasion after experiencing some event, treatment, or intervention. Because there is no control group against which to make comparisons, it is a weak design

(tute example: experiment seeing if at a party, drink consumption increases touching)

Question 15

Advantages and Disadvantages of a one-shot case study (post-test only):

Answer 15

Advantages:
- At least allows us to study events

Disadvantages:
- No basis for comparison

Question 16

What is a one group pre test post test design?

Answer 16

type of quasi-experiment in which the outcome of interest is measured 2 times: once before and once after exposing a non-random group of participants to a certain intervention/treatment.

Question 17

Advantages and Disadvantages of one group pre-test post-test design:

Answer 17

Advantages:
- Allows some comparison

Disadvantages:
- Many Threats to Internal Validity

Question 18

What are the threats to internal validity?

Answer 18

• history
• maturation
• testing
• instrumentation
• mortality

Question 19

Threats to internal validity: History

Answer 19

An unrelated event influences the outcomes.

e.g., Most participants are new to the job at the time of the pre-test. A month later, their productivity has improved as a result of time spent working in the position.

Question 20

Threats to internal validity: Instrumentation

Answer 20

Different measures are used in pre-test and post-test phases.

e.g., In the pre-test, productivity was measured for 15 minutes, while the post-test was over 30 minutes long.

Question 21

Threats to internal validity: Testing

Answer 21

The pre-test influences the outcomes of the post-test.

e.g., Participants showed higher productivity at the end of the study because the same test was administered. Due to familiarity, or awareness of the study’s purpose, many participants achieved high results.

Question 22

Threats to internal validity: Mortality

Answer 22

a problem that arises when employing a longitudinal design and participants who start the research process are unable to complete the process

Question 23

What is a static-group comparison design?

Answer 23

quasi-experimental design in which the outcome of interest is measured only once, after exposing a non-random group of participants to a treatment, and compared to a control group

Question 24

Advantages and disadvantages of static-group comparison design:

Answer 24

Advantages:
- Allows some comparison

Disadvantages:
- Many Threats to Internal Validity—but especially “Selection”

Question 25

What is a simple time-series design?

Answer 25

an experimental design that involves the observation of units (e.g., people, countries) over a defined time period.

Question 26

Advantages and disadvantages of a simple time-series design:

Answer 26

Advantages:
- Allows us to see the distinctiveness of the
treatment
- “Within-subjects” Design

Disadvantages:
- History could still be a threat to internal
validity (if something odd happened on Thursday!)

Question 27

What is an equivalent time-samples design?

Answer 27

quasi-experimental method where the intervention is administered several times and the intervention is taken away several times

Question 28

Advantages and disadvantages of an equivalent time-samples design:

Answer 28

Advantages:

• Reduces likelihood of history confounds
• “Within-subjects” Design

Disadvantages:
- Can only be used with “treatments” that are
temporary—or with enough delay between treatments

Question 29

What is a non-equivalent control group design?

Answer 29

where the assignment of participants to groups is not controlled by the investigator. When group assignment is not controlled there is a significant threat to internal validity. Since group assignment is not random, there is a chance that the groups are not similar.

Question 30

Advantages and disadvantages to non-equivalent control group design:

Answer 30

Advantages:
- Reduces problems with selection (can
compare baseline rates of 2 groups)
- “Between-subjects” Design

Disadvantages:
- Other Confounds are still a possibility
(things that differed for the control group and treatment group other than the IV).

Question 31

Why are control groups important?

Answer 31

• important in ruling out alternative explanations for results.

Question 32

What is a confound?

Answer 32

• factor other than the IV that varies between the conditions of an experiment, calling into question what caused any effects on the DV

Question 33

What is an experimenter expectancy effect?

Answer 33

• effects produced when an experimenters expectations about results of an experiment affect behaviour toward a participant, therefore influence participants responses
• KEEP EXPERIMENTERS BLIND TO CONDITION

Question 34

What is external validity?

Answer 34

• degree to which there can be reasonable confidence that the results of the study would be obtained for other people and in other situations
• considerations:
  – Representative vs. convenience samples
  – Increased use of Internet-based data collection allows for more diverse sets of participants
  – What is the setting in which the research is conducted?

Question 35

Laboratory vs Field Settings:

Answer 35

Laboratory settings permit a high degree of control but also impose limits:

• Relatively short time
• Measures may seem “artificial”
• Demand characteristics: Cues in a research setting that lead participants to infer what researchers expect or desire and that therefore bias their act

Field settings potentially offer high realism, but…

• less control of various factors
• May affect people’s motivations, thoughts, feelings, behaviours in some specific ways

Question 36

What is mundane realism?

Answer 36

• the extent to which the research setting resembles the real-world setting of interest

Question 37

What is experimental realism?

Answer 37

The degree to which the experimental setting and procedures are real and involving to the participant

Question 38

Using mundane or experimental realism?

Answer 38

Using both methods to focus on the same question gives you all the benefits and makes up for all of the weaknesses

Question 39

What is a between subject design?

Answer 39

participants are randomly assigned to one level of each independent variable (one condition of the study). Some random error expected.

every participant experiences only one condition, and you compare group differences between participants in various conditions

Question 40

What is a within subjects design?

Answer 40

each participant experiences and responds to every level of the independent variable. Each participant is essentially their own “control”.

also called a dependent groups or repeated measures design because researchers compare related measures from the same participants between different conditions. All longitudinal studies use within-subjects designs to assess changes within the same individuals over time.

Question 41

What is a mixed design?

Answer 41

both between subjects and
within subjects independent variables may be
included.

Question 42

Problems with Within-subject designs?

Answer 42

• context effects (practice, sensitisation, carry-over)
• counter balancing (an attempt to reduce or avoid carryover effects and order effects.) may help but contrast (unconscious bias that happens when two things are judged in comparison to one another, instead of being assessed individually.) is unavoidable

Question 43

Problems with between-subjects designs:

Answer 43

• Random Assignment, especially with small samples, still leaves some individual differences between groups
• Any differences between groups end up as “noise” (random error) or “confounds” (systematic error).
• Confounds may still exist if participants are run in groups or if experimenters are not careful to keep all procedures consistent across levels of the independent variables.
• But random assignment is better than no random assignment…
• Effects may be smaller if participants don’t know the other levels of the independent variable – ecological validity may
  be an appropriate guide.