Chapter 10 Flashcards
When is an experiment considered a true experiment?
if all aspects of a basic experimental design are present, and absent any threats to internal validity
Are all topics amenable to the use of a true experiment?
no
When are single case experimental designs used?
when only one appropriate participant is available.
When are quasi experimental designs used?
used when random assignment for a variable is not possible
Can anything that departs from a true experiment e sued to support claims about causality?
no.
Can applied research and evaluation research present difficulties that prevent researchers from using true experiments?
yes.
What are some reasons that may prevent applied researchers and evaluation researchers from conducting true experiments?
Sometimes researchers are invited to participate to late in the process and are not able to provide input on the best measurement techinques.
Budget limitations can rule out some form of data collection
Program evaluation research raises many of the issues found in what contexts?
applied contexts
What is program evaluation? What are they usually looking to achieve? Where are the programs often implemented?
Research designed to evaluate programs (e.g., social reforms, innovations) designed to produce changes or certain outcomes in a target population.
Usually looking to achieve some positive effect on a group of people.
Such programs can be implemented in schools, work settings, or entire communities.
Who urged a culture of evaluation in which all programs are thoroughly evaluated to determine whether they are effective?
Donald Campbell
How has the field of program evaluation progressed?
the initial focus of evaluation research was outcome evaluation (did the program result in the positive outcome for which it was designed?)
Evaluation research then broaden the scope to many other questions which can be categorized into 5 types.
What are the 5 types of questions that can guide program evaluations? Is this a sequential process? What type of questions are emphasized depend on?
This is a sequential process and the type of questions that are emphasized depend on the purpose of evaluation.
The categories/5 types of program evaluation research
- needs assessment
- are there problems that need to be addressed in a target population? - Program Theory Assessment
- How will the problems be addressed? Will the proposed program actually address the needs appropriately? - Process Evaluation
- Is the program addressing the needs appropriately? Is it being implemented properly? - Outcome Evaluation
- Are the intended outcomes of the program being realized? - Efficiency Assessment
- Is the cost of the program worth the outcomes?
What are examples of questions researchers could ask relating to needs assessments (evaluation research?)
- what services do homeless people need most?
- Do repeat juvenile offenders have particular personal and family problems that could be addressed by an intervention program?
- What are some of the most overall important needs among Portuguese Canadians?
How do researchers usually collect data for a needs assessment (evaluation research)?
often come form surveys, interviews, and existing archival data maintained by public health, criminal justice, and other agencies.
Explain we need to think about when assessing program theory?
Who does assessing theory usually involve?
What does assessing program theory include?
After identifying needs, a program can be designed to address them. The program must be based on valid assumptions about the causes of problems and a cogent rationale for the best way to address these problems.
usually involves researchers, service providers, a prospective client program clients collaborating to ensure that the program assesses the target populations’ needs.
includes articulating a rationale for how members will benefit from the program, including how they will access the program services.
What is another name for process evaluation?
When the program is underway, what do the researchers monitor?
What is this research very important for?
How is data for this research often collected?
program monitoring.
monitor whether it reaches the target pop., whether its attracting enough clients, and whether staff is providing he planned services.
This research is really important to avoid concluding that a program is ineffective, when really it is not being implemented properly.
Often collected through questionnaires, interviews, observational studies, and analyses of records kept by program staff.
What is another name for outcome evaluation?
How do researchers determine this?
What do we need to know when addressing this aspect of program evaluation?
What methods do we use to conduct outcome evaluation?
Impact assessment.
the researchers must devise a way of measuring the outcome and then study and then study the impact of the program on the outcome measure.
We need to know what participants of the program are like and what they would be like if they had not completed the program.
Ideally, a true experiment with random assignment to conditions would be carried out to answer questions about outcomes. However, other approaches like the quasi-experimental and single case designs, can be useful ways of assessing the impact of the intervention when random assignment is not possible or is not ethical.
What are researchers trying ot figure out hen doing efficeny assessment?
once it is shown that the progrma does have the intended affect, researchers can determien whtehr the benefits are worth the cost, or if the resources used to implement this program coudl be better used.
What is a shoestring evaluation? Who came up with it?
full program evaluations can be extensive, longterm undertakings, particularly if every question is being addressed. Researchers liek Bamberger and colleagues, are developing systematic approaches to respond to the specific challenges that arise while doing evaluation research. Shoestring evaluation is when there are constraints related to time, budget, and data collection options.
What are quasi experimental designs?
A study design that has many features of an experiment, but due to necessity lacks some aspects of a true experimental design (and so cannot support causal inferences).
What is quasi a latin term for?
As if
What do Quasi-experiments lack from a true experimental design out of necessity?
What is the result of this?
What do quasi experimental designs provide a good example of?
control conditions, random assignment
As a result, they cannot be used to make causal inferences, but rather indicate how variables are related. (i.e associations, similar to correlational studies with measured variables.)
Provide a good example of how studies that claim to b using true experimental designs might fall short of that goal, and inadvertently include elements that prevent them from being true experimental designs, and therefore do not permit causal inferences.
When using quasi experimental designs, what is it important to distinguish between?
(1) quasi-experimental designs used intentionally and out of necessity
(2) flawed experiments that purport to be true experiments but are actually quasi experimental in nature.
Are Quasi-experiments bad or poor experiments?
no. in many circumstances they are used out of necessity.
What is the simplest quasi experiment design?
the one-group post-test only design.
What are the 3 categories of research designs for special circumstances. What are these compared to?
Compared to true experiments.
- single case
- Quasi-experiment
- Developmental
What are all of the characteristics of a true experiment? (design, # of conditions, # of participants, assignment, pretest?, post-test?)
- between subjects design
- 2 or more conditions
- random assignment
- lots of participants (often 30+ per group)
- Sometimes includes a pre-test
- includes a post-test (dependent variable)
What are all of the characteristics of a single case experiment? (design type (s), # of conditions, # of participants, assignment, pretest?, post-test?)
Reversal ABA (B) :
- 1 condition (more if re-institute treatment, as in ABAB)
- random assignment is not applicable
- 1 participant
- a pretest (A which is a baseline)
- a post-test (reversal to the baseline)
Multiple baseline:
- 1 condition(more if multiple settings or behaviours)
- no random assignment
- 1 participant (more if multiple participants)
- a pre-test (A, baseline)
- a post-test (B, treatment)
What are all of the characteristics of a Quasi-experiment? (design type (s), # of conditions, # of participants, assignment, pretest?, post-test?)
One Group posttest- only:
- 1 condition
- no random assignment
- many participants (often 30+)
- no pre-test
- a posttest
One Group pretest -posttest:
- 1 condition
- no random assignment
- many participants (often 30+)
- a pre-test
- a posttest
Non-equivalent control group:
- 2+ conditions
- no random assignment
- many participants (often 30+ per group)
- no pre-test
- a posttest
Non-equivalent control group pretest-posttest:
- 2+ conditions
- no random assignment
- many participants (often 30+ per group)
- a pre-test
- a posttest
Interrupted time series:
- 1 condition
- no random assignment
- many participants (often archival data)
- multiple pre-tests
- a posttest (often multiple)
Control series:
- 2+ conditions
- no random assignment
- many participants (often archival data)
- multiple pre-tests
- a posttest (often multiple)
What are all of the characteristics of a developmental experiment? (design type (s), # of conditions, # of participants, assignment, pretest?, post-test?)
Longitudinal:
- 1 condition
- no random assignment
- many participants (often 30+)
- sometimes a pretest
- multiple posttests
Cross Sectional:
- 2+ conditions
- many participants (often 30+ per group)
- sometimes a pre-test
- a posttest
Sequential:
- 2+ conditions
- many participants (often 30+ per group)
- sometimes a pre-test
- multiple post tests
What is the sequence for one group posttest only design?
participants, then independent variable, then dependent variable
What is the one group posttest only design? Can you interpret these results?
A quasi-experimental design that has no control group and no pretest comparison; a very poor design in terms of internal validity.
- cannot interpret these results
What is the one group posttest only design used as (weak and insufficent) evidence for? Example?
- effectiveness of programs or advertised products
Ex: employees may participate in a four hour information session on emergency procedures , after which they score an average of 90% on a knowledge test. Without any comparison, you cannot conclude that the company is successfully educating the employees.
Ex: a sports drink company may advertise that people reported high athletic performance after drinking a sport drink. BUT this doesnt allows us to make conclusions because we don;t know how they would have performed otherwise.
Do you need comparison data to be able to interpret any result?
yes.
Why would a researcher chose to use a one group posttest only design?
there may be a unique circumstance in which a one group posttest-only design is the only possible design, making it necessary.
Is one group pretest or one group pretest-posttest design higher on internal validity?
one group pretest-posttest design
What is one group pretest-posttest design?
A quasi-experimental design in which the effect of an independent variable is inferred from the pretest-posttest difference in a single group.
What would be the sequence of a one group pretest-posttest design?
participants, dependent variable pretest, independent variable, dependent variable posttest.
Can you make casaul inferences with the one group pretest-posttest design? Why or why not?
no you cannot. The design fails to take into account several possible alternative explanations. These alternative explanations are called threats to internal validity.
What do threats to internal validity include?
history, maturation, testing, instrumental decay, and regression towards the mean.
What does internal validity refer to? When do we consider something a threat to internal validity?
our ability to claim, based on a true experiment, that the independent variable causes changes in the dependent variable
something is a threat to internal validity when it allows for some other reasonable alternative explanations for changes to the dependent variable.
When a true experiment is well designed, are there any threats to internal validity?
no.
Is a confounding variable a threat to internal validity?
yes.
Provide a summary of the history threat to internal validity and explain how it undermines internal validity.
Summary: Historical event that affects all or most participants (e.g. natural disaster, media event) and that is not of interest to the researcher.
Explanation: Provides an alternative explanation for change between pretest and posttest. if there are multiple groups and the event affects the groups differently, this becomes a confound and an alternative explanation for group differences.
Provide a summary of the Maturation threat to internal validity and explain how it undermines internal validity.
Summary:
Natural changes to participants short term states (e.g. fatigue), or long-term development (e.g. education), not of interest to the researcher.
Explanation:
Provides an alternative explanation for change between pretest and posttest. if there are multiple groups and the groups mature differently, this becomes a confound, and therefore an alternative explanation for group differences.
Provide a summary of the Testing threat to internal validity and explain how it undermines internal validity.
Summary: Simply taking the pretest influences people’s responses to the posttest
Explanation: Provides an alternative explanation for changes between pretest and posttest
Provide a summary of the Instrument decay threat to internal validity and explain how it undermines internal validity.
Summary: Characteristics of the measurement instrument changes with repeated use.
Explanation: Provides an alternative explanation for changes between pretest and posttest
Provide a summary of the Regression Towards the mean threat to internal validity and explain how it undermines internal validity.
Summary: May occur when participants are chosen or groups are divided based on extreme scores on the pretest. because extreme scores tend to become less extreme on repeated measurement .
Explanation: Provides an alternative explanation for changes between pretest and posttest
Provide a summary of the Attrition threat to internal validity and explain how it undermines internal validity.
Summary: Participants leave the study.
Explanation: Differences in attrition between groups may create group differences, even if groups were initially randomly assigned, offering an alternative explanation for differences between groups.
Provide a summary of the Selection Effects threat to internal validity and explain how it undermines internal validity.
Summary:
Groups are divided based on any reason other than random assignment.
Explanation:
Pre-existing group differences offer alternative explanations for differences between groups.
Provide a summary of the Cohort Effects threat to internal validity and explain how it undermines internal validity.
Summary:
Groups are divided by age. A special type of selection effect.
Explanation:
Instead of differences being due to age, the unique characteristics of a particular cohort offer an alternative explanation for differences between groups.
What is a history effect? When does it occur? how does this effect one group study designs?
Threats to internal validity, in which outside events not part of the manipulation influence the dependent variable, providing an alternative explanation for the results.
occurs during or after the experimental manipulation, after the pretest (if there is one), but before the posttest.
In one group study designs any such event is confounded with the manipulation.
What are maturation effects? How does this effect one group study designs?
effects that occur systematically over time. Threats to internal validity, in which any naturally occurring change within individuals that occurs over time could provide an alternative explanation for the results.
in the one group pretest posttest design, maturation is confounded with the manipulation.
Can a human observer (researcher) be subject to instrument decay?
yes. a person rating behaviour can become more skilled, fatigued, or change standards on which observations are based.
What is regression towards the mean? What is another name for this?
also called statistical regression..
A statistical phenomenon in which extreme scores on a variable tend to be closer to the mean when a measurement is repeated; this change can be mistakenly attributed to some manipulation or intervention. Regression to the mean is an alternative explanation for an observed change.
What is the problem of regression towards the mean rooted in?
the reliability of our measures.
What is any measurement composed of? how is this related to repeated measures?
a true score plus measurement error.
because of this measurement error, scores will vary around the true score with most scores close to the true score, but some will be higher and some will be lower.
In practice, is there such thing as a completely reliable measure?
no. All measures have measurement error.
If a one group pretest posttest design is the only design that is possible, what can we do to help rule out some threats to internal validity?
if it is not possible for us to create a control group, replicating the study at other times with other participants can help rule out some of the threats to internal validity.
Can threats to internal validity affect true experiment designs?
yes.
How can many of the threats to internal validity be addressed?
by the use of an appropriate control group we can help rule out the effects of history, regression towards the mean etc.
Provide scenarios for how threats to internal validity can impact true experiments.
- perhaps an outside event occurs that only affects on group of participants, but not the other. Evn if the event effects both groups equally it coudl pose a threat to making causal claims.
What is a non-equivalent control group pretest-posttest design? Can it make causal claims? Is it a good design? Is assignment to groups random?
A quasi-experimental design in which non-equivalent groups are used, but a pretest allows assessment of equivalency and pretest-posttest changes.
this is still not a true experiment and cannot support causal inferences but it is one of the most useful quasi experimental designs.
assignment to groups is not random.
What is the sequence for a non-equivalent pretest posttest design?
2 groups:
- participants, dependent variable pretest, treatment, dependent variable posttest
- participants, dependent variable pretest, non-treatment (control), dependent variable posttest
What are interrupted time series designs similar to? What is the difference? What are they often used to measure? what type of data is often used?
similar to one group pretest-posttest, and non-equivalent pretest-posttest designs respectively.
The key addition is that there are multiple pre and post tests.
commonly used to examine the effects of naturally occuring manipulations in society. Like passing of laws.
archival data
What is an interrupted time series design?
A quasi-experimental design in which a treatment is investigated by examining a series of measurements made over an extended time period, both before and after the treatment is introduced.
Which is the next best quasi experimental design after the interrupted time series design?
the control series design
what is a control series design?
An extension of the interrupted time series quasi-experimental design in which there is a comparison or control group.
Why were single case experiments created?
so that experiments could be conducted within the context of a case study, with just a single research participant.
What is a single case experimental design?
A research design in which the effect of the independent variable is assessed using data from a single participant.
in a single case experimental design what is the participants behaviour first measured by? When is the manipulation introduced? What offers evidence for effectiveness of the manipulation?
a baseline control time period
manipulation is introduced after baseline during a treatment period, then the participants behaviour is continued to be observed.
A change in participant behavioru from baseline to treatment periods offers evidence of effectiveness
What is a baseline control time period?
A form of control condition in which participant behaviour is measured during a control period, before introduction of the manipulation.
Where did much of the early interest in single case studies come from?
classic research in reinforcement, pioneered by B.F. Skinner
Whats the biggest challenge in single case study experiments? What method can be used to demonstrate this? What is this called?
how to detemrien that the treatment had an effect on the dependent variable
one method is to determine that treatment can be undone or reversed by removing treatment.
This is called a reversal design
What is the sequence of a reversal design?
A (baseline period) -> B (treatment period) -> A (baseline period)
What does the ABA (reversal design) require? What is this also sometimes called?
requires that behaviour be observed and measured during A, the baseline period, B, the treatment period, and also during a second baseline control beriod (a) after removing the experimental treatment.
Sometimes called a withdrawal design.
How can the ABA design be greatly improved? how can this address issues with interpreting results?
by extended it to an ABAB design or ABABAB design. Adding more reversals can address 2 issues with interpreting results.
1. a single reversal can be caused by a random fluctuation in the person’s behaviour.
- it allows us to consider an ethical issue. It doesn’t seem right to end the design on a withdrawal of a treatment that might be beneficial to the participant. ABAB design means that the sequence ends with a treatment rather than a withdrawal.
What is a multiple baseline design? Are there multiple variations? What is a version?
Observing behaviour before and after a manipulation under multiple circumstances (across different individuals, different behaviours, or different settings).
yes.
a version is where hte baseline is measured across participants with behaviour of several participants measured overtime. The key element is that for eahc participant the manipulation is introduced at a different point in time.
Do reports of research using single case experiments, typically present results separately for each particpant?
yes.
Are multiple behaviours of a single participant measured overtime in a multiple baseline across behaviours design?
yes.
what is the third variation of multiple baseline designs?
the same behavioru is measured across multiple different situations, such as at home and at school .
Why might some multiple baseline designs include a reversal while others do not? In cases where there isn’t a reversal, what should researchers expect?
sometimes a reversal fo a behaviour is not possible or is unethical.
in this case multiple measures overtime can be made before and after the manipulation. if it is effective, a change in behaviour may be observed right away and the change should continue to be reflected in further measurements of the behaviour.
What do single case designs have the same limitations as? how can this be enhanced?
descriptive case studies (questionable generalizability)
When the results observed with a single participant can be replicated with other participants, the generalizability is enhanced.
When can single case designs be especially valuable?
for applying treatment to help someone in particular improve their behaviour
Can the reuslts from many single case study designs be combined using metanalysis to reveal overall patterns?
yes.
In all developmental researhc designs, what is the major variable?
age
What are the 2 general methods developmental researchers use ot study peopel of different ages? What do these 2 methods resemble? what is a third approach?
the corssectional method and hte longitudinal method.
crosssectional share similarities with between subjects designs
longitudinal shares similarities with within subjects designs.
there is a hybrid approach called a sequential method.
What is the longitudinal method?
A developmental research method in which the same people are observed repeatedly as they grow older; conceptually similar to a within-subjects design.
overtime
is longitudinal or cross sectional more common? Why? What is this one more susceptible to?
corss sectional
it is less expensive to conduct and yields immediate results
very susceptible to cohort effects.
What are cohort effects?
A cohort is a group of people born at about the same time and exposed to the same societal events; cohort effects are confounded with age in a cross-sectional study.
What is the cross sectional method?
A developmental research method in which people of different ages are studied at a single point in time; conceptually similar to a between-subjects design
single point in time.
What 2 things are confounded in cross sectional design?
the variable of interest (age) and cohort effects.
When are cohort effects most likely to be a problem in cross sectional designs?
when the researcher is examining age effects across a wide range of ages.
What is a retrospective approach? What problems does this introduce?
alternative to longitudinal where you study people of a certain age and ask parents about their earlier environments etc.
problem = difficulty remembering past events accurately.
What are the major limitations of a longitudinal approach?
its difficulty and its expense. It’s hard to keep track of people over a number of years. People may move, die, or lose interest in the study. AKA there is a great risk of high participant attrition.
What must researchers do to account for high attrition when analyzing data?
researchers must compare the scores of people who drop out with those who stay. If those who stay become quite different than those who leave, then we might be concerned about whethr there is something unique and unidentified about the final sample that has not been analyzed.
What is the sequential method? What is a benefit?
A combination of cross-sectional and longitudinal designs to study developmental research questions. First thing is to compare age groups through cross sectional methods, then use longitudinal methods to retest each group at least one more time at a point in the future.
using this method may take fewer years to complete than a traditional longitudinal study, and cross sectional data are ready to analyze while longitudinal data are being collected.
