Evaluative research exam 1

Question 1

what are the features of Random Assignment

Answer 1

People assigned to conditions based on chance.
Each person has a nonzero probability of being assigned to any given condition.
Random sampling is a different thing

Question 2

What does Random assignment do for your study

Answer 2

Evenly distributes participant characteristics across conditions.

Rules out selection threat.

Makes it unlikely that other threats to internal validity are confounded with condition.

Question 3

Basic random assignment design

Answer 3

subjects randomly assigned to treatment (Could be more than one) or control condition(or other treatment condition).
Post-test measure of outcome

Question 4

Potential problem with random assignment

Answer 4

Attrition-people dropping out
Feasibility in terms of ethics and time or availibility of people that meet the criteria
it is difficult to randomize correctly
participants receive assigned treatment and no other

Question 5

Why is attrition a problem in a random assignment design?

Answer 5

The people who drop out could be different in important ways (for particular conditions or the study as a whole) and their leaving could threaten validity.
Pretest helps assess the extent of the problem

Question 6

Pretest-posttest design

Answer 6

Same as basic random assignment with a pretest condition, which could be used as a covariate or a repeated measures ANOVA can be performed.

Question 7

Factorial design

Answer 7

Comnine two or more independent variables (factors) that each have at least two levels.
Major advantage: Test interactions along with main effects

Question 8

Longitudinal design

Answer 8

Multiple pretest and posttest measurements of coutcome variable.
Show changes over time

Question 9

Problems with longitudinal design

Answer 9

attriction, and can be unethical to withhold treatment for long periods of time

Question 10

Crossover design

Answer 10

Two groups (randomly assigned), two treatment levels, put each group through both treatment conditions at different times: R O Xa O Xb O and R O Xb O Xa O

Question 11

list the different Random assignment designs

Answer 11

Basic, Pretest-posttest, Factorial, Longitudinal, crossover

Question 12

When would you use random assignment in feild research

Answer 12

when demand is greater than supply

when a treatment can’t be delivered to everyone at once
when temporal isolation is possible
when people are spatially seperated or don’t communicate much
when change is needed but it’s unclear which solution will work best
When there is ambiguous need
when some people have no preference among alternatives
when you can create your own organization
when you have control over experimental units
when loteries are expected

Question 13

When should you not use random design

Answer 13

Short on time
research question is not about causation
impossible or unethical to manipulate the IV
More conceptual or empirical work must be done to determine whether an experiment is a good use of resources

Question 14

What are some techniques of randomization

Answer 14

Simple random assignment
Restricted random assignment to force equal cell sizes
restricted random assignment to force unequal cell sizes
Haphazard assignment

Question 15

Regression discontinuity design

Answer 15

subjects assigned to condition on the basis of a cutoff score on assignment variable
assignment variable mus be continuous and taken before the treatment
post-test measure after treatment

Question 16

Interrupted time series

Answer 16

Building on the pre-test post test design by increasing number of both
the interuption is the treatment
time series because pretest and posttest measurements are taken at intervals
100 is a generally acceptable number of data points

Question 17

interupted time series designs are vulnerable to what threats of validity?

Answer 17

History (biggest)-to alleviate make measurements intervals smaller.
Instrumentation
attrition

Question 18

ITS additions to increase validity

Answer 18

Nonequivalent control group
Nonequivalent DV
Introducing and then removing treatment
Multiple replications
Switching replications

Question 19

ITS potential difficulties

Answer 19

Gradual interventions
Delayed causation
Short time series
limitations of archival data

Question 20

Quasi-experiments

Answer 20

attempt to test causal hypothesis

called “quasi” because they lack random assignment

Question 21

How do you determine causation in quasi-experiments?

Answer 21

Cause precedes effect
cause co-varies with effect
Alternative explanations are implausible

Question 22

Validity

Answer 22

The approximate truth of an inference
A matter of degree
A property of inferences, not designs or methods

Question 23

Four types of validity

Answer 23

Statistical conclusion
Internal
Construct
External

Question 24

Statistical Conclusion Validity

Answer 24

Validity of inferences about the covariance between treatment and outcome

I.E. How large and how reliable is the co-variation between the presumed cause and effect?

Question 25

Methods of assessing statistical conclusion validity

Answer 25

NHST: P values, Effect size and confidence intervals

Question 26

P values

Answer 26

tell us the probability that the results observed in an experiment could have been obtained by chance from a population in which the null hypothesis is true

Question 27

Threats to statistical conclusion validity

Answer 27

``` Low statistical power Violated assumptions of statistical tests Fishing and error rates Unreliability of measures restriction of range Unreliable treatment implementation extraneous variance in setting Heterogeneity of respondents inaccurate effect size estimation ```

Question 28

Power

Answer 28

The probability that a statistical test will reject the null hypothesis when it is actually false and should be rejected Power is usually set at .80

Question 29

Fishing and error rate

Answer 29

Increasing the number of statisitical tests increases probability of a type 1 error

Question 30

Restriction of range

Answer 30

Independent variable-Levels too few or too similar | Dependent variable-categorical instead of continuous:floor or ceiling effects

Question 31

Internal validity

Answer 31

Validity of the inference that the treatment as implemented had a causal effect on the outcome as measured Is the co variation causal or would the same co variation have been obtained without the treatment?

Question 32

Local molar causal validity

Answer 32

Local- causal conclusions are limited to the particular treatments, outcomes, settings, and persons studied. Molar-treatments are a complex package with many components (INUS conditions) Causal-did the treatment as implemented cause changes in the outcome as measured?

Question 33

Threats to internal validity

Answer 33

``` Ambiguous temporal precedence Selection history maturation regression to the mean attrition Testing instrumentation ```

Question 34

Selection threat

Answer 34

Participant characteristics are confounded with treatment condition

Question 35

History

Answer 35

Occurs when an event outside of the study could have produced the outcome

Question 36

Maturation

Answer 36

when natural changes in participants could have produced the outcome

Question 37

Regression to the mean

Answer 37

Extreme scores at one point (or measure) tend to be followed by scores closer to the mean when people are selected for a study because of a high school, a lower score after treatment could be regression instead of treatment

Question 38

Testing

Answer 38

Taking a test once can lead to participant changes that are responsible for an observed effect

Question 39

Instrumentation

Answer 39

Changes over time in measuring instruments might be responsible for an observed effect

Question 40

Construct Validity

Answer 40

Validity of inferences about higher order abstract constructs based on specific manipulations and operationalizations Which general constructs are involved in the persons, settings, treatments, and observations used in the experiment?

Question 41

Ways to increase construct validity

Answer 41

Pilot testing manipulation checks for IV explication of constructs using existing, workable manipulations and measures

Question 42

Inadequate explication of constructs

Answer 42

Failure to adequately explicate a construct may lead to incorrect inferences about the relationship between operation and construct

Question 43

Construct confounding

Answer 43

Operations usually involve more than one construct and failure to describe all the constructs may result in incomplete construct inferences

Question 44

mono-operation bias

Answer 44

Any one operationalization of a construct both under-represents the construct of interest and measures irrelevant constructs, complication inference

Question 45

Mono-method bias

Answer 45

When all operationalizations use the same method (e.g. self report) that method is part of the construct actually studied

Question 46

Confounding constructs with levels of constructs

Answer 46

Inferences about the construct that best represent study operations may fail to describe the limited levels of the construct that were actually studied

Question 47

Treatment sensitive factorial structure

Answer 47

The structure of a measure may change as a result of treatment change that may be hidden if the same scoring is always used

Question 48

Reactive self report changes

Answer 48

Self reports can be affected by participant motivation to be in a treatment condition motivation that can change after assignment is made

Question 49

Reactivity to the experimental situation

Answer 49

Participant responses reflect not just treatments and measures but also participants' perceptions of the experimental situation and those perceptions are part of the treatment construct actually tested

Question 50

Experimenter expectancies

Answer 50

The experimenter can influence participant responses by conveying expectations about desirable responses and those expectations are part of the treatment construct as actually tested

Question 51

Novelty and disruption effects

Answer 51

Participants may respond unusually well to a novel innovation or unusually poorly to one that disrupts their routine a response that must then be included as part of the treatment construct description

Question 52

Compensatory equalization

Answer 52

When treatment provides desirable goods or services administrators, staff , or constituents may provide compensatory goods or services to those not receiving treatment and this action must them be included as part of the treatment construct

Question 53

Compensatory rivalry

Answer 53

Participants not receiving treatment may be motivated to do as well as those receiving treatment and this must be included in the treatment construct description

Question 54

Resentful demoralization

Answer 54

Participants not receiving the treatment may become so resentful or demoralized that they respond more negatively that otherwise

Question 55

Treatment diffusion

Answer 55

Participants may receive services from a condition to which they were not assigned

Question 56

External validity

Answer 56

Validity of inferences about generalizing the cause-effect relationship to other person. settings, treatments, and measurements How generalizable is the locally embedded causal relationship over variation in persons treatment, observations, and settings?

Question 57

Four types of replication

Answer 57

Exact replication conceptual replication constructive replication participant replication

Question 58

Conceptual replication

Answer 58

Investigates the relationship between the same conceptual variables studied in previous research using different operational definitions

Question 59

Constructive replication

Answer 59

Tests the same hypothesis of original experiment with added condition(s) to asses specific variables that might change the previously observed relationship

Question 60

Participant replication

Answer 60

Conduct the original experiment using a new type of participants