Final Flashcards
Control Groups and Treatments of Questionable Efficacy
Intervention Research Issues
In research on the effectiveness of psychotherapies, there is often a group that is not expected to improve. This situation raises ethical issues:
- Over the course of the study, people in a control group may stay the same or get worse.
- Participants in the group that does not receive an effective treatment may be discouraged from seeking psychotherapy in the future.
Unfortunately, research using control conditions that raise ethical issues is “fundamental to progress in understanding treatment.” To find out whether or not a treatment works, a no-treatment or waiting-list control group is necessary. To find out why a treatment works, a nonspecific treatment control group is necessary.
Some of the ethical issues can be addressed by providing treatment to participants in the control group after the study is over.
Obtrusive measures
When subjects are aware their performance is being assessed
Longitudinal studies
make comparisons over an extended period of time often involving several years
a study with pre, post and follow up is longitudinal
Quasi-experiment
when researcher cannot control who is in each group
often occurs when doing research in schools or hospitals (because classes or wards already exist)
Sometimes researcher is able to randomly assign P in some groups but not all groups
Major qualitative research methods
Interviews Focus Groups Direct observations Statements of personal experience Documents Photographs Audio or video recordings Films
Test Bias
A test is biased when it does not predict as accurately for one group as it does for another. Bias occurs when the data for two groups have different slopes or different intercepts. When two groups get different mean scores on a test, that alone does not mean that the test is biased.
4 conditions for mediator
- “The intervention (e.g. exercise) leads to change on [an] outcome measure (e.g. depression).
- The intervention alters the proposed mediator (e.g. …stress level).
- The mediator is related to [the] outcome (stress level is related to symptoms).
- Outcome effects (changes in depression) are not evident or substantially less evident if the proposed mediator (stress in this example) did not change.”
Reactive measures
If awareness of assessment leads persons to respond differently from ow they would usually respond
Concurrent validity
correlation of a measure with performance on another measure or criterion at the same point in time
Sample Characteristics
External Validity Threat
The extent to which the results can be extended to subjects or clients whose characteristics may differ from those included in the investigation
Internal Consistency
Degree of consistency or homogeneity of the items within a scale. Different reliability measures are used toward this end such as split-half, kuder-richardson 20 formula, and coefficient alpha
Steps in Scale Development
- Determine what you want to measure, including level of specificity.
- Generate an item pool.
Redundancy is fine.
Begin with more items than you want in the final scale. Internal consistency
reliability is related to the number of items. - Determine the format of the questions. Decide on the number of response categories,
whether you want an odd or even number of responses, and type of response format.
Possible formats:
Likert scale
Semantic differential
Visual analog
Binary options - Have initial item pool reviewed by experts.
- Consider inclusion of validation items.
- Administer items to a development sample. It should be large and representative of
the population for which the scale will be used. - Evaluate the items—including computing coefficient alpha (or Cronbach’s alpha).
Usually, .70 is considered acceptable. Alpha depends on the number of items and on
the average inter-item correlation. - Optimize scale length—longer scales are more reliable.
Discriminant validity
correlation between measures that are expected not to relate to each other or to assess dissimilar and unrelated constructs
Narrow Stimulus Sampling
External Validity Threat
The extent to which the results might be restricted to a restricted range of sampling materials (stimuli) or other features the experimenters used in the experiment
Stimulus characteristics include the experimenters, setting, interviewers or other factors
Most commonly occurs when there is one experimenter, one therapist, one setting, one taped story, etc.
Sensitivity of the measure
“The measure ought to be sensitive enough to reflect the type and magnitude of change or group differences that the investigator is expecting.”
Desirable characteristics of a measure of the DV:
a. It should allow a large range of scores so that it can pick up differences between
groups or conditions.
b. It should for bi-directional changes—increases or decreases. Should not allow
ceiling or floor effects.
c. Looking at the measure’s items should show that the measure could pick up changes
or group differences. Sometimes existing literature shows that the measure can pick
up changes.
When the measure of the DV used is not sensitive, that can lead to finding no change or no difference between groups.
Translational research
applies finding from basic research (e.g. laboratory research) to people in real life (applied research)
Moderated mediation
occurs when strength (or direction) of the relation of the mediator to outcome depends on the level of some other variable
mediator that doesn’t work for everyone
Predictive validity
correlation of a measure at one point in time with performance on another measure or criterion at some point in the future
Nonspecific Treatment or Attention-Placebo Control Group
Get everything except actual treatment (including attention)
Difficult to have expectations for improvement without using an intervention
Issues with APCG can be avoided by using Treatment as Usual (TAU)
Stratified Sampling
members of groups in the pop are selection in proportion to their representation in the pop
Fairness in Treatment During the Testing Process
“Regardless of the purpose of testing, the goal of fairness is to maximize, to the
extent possible, the opportunity for test takers to demonstrate their standing on
the construct(s) the test is intended to measure” (p. 51).
Accordingly, procedures “for the standardized administration of a test should be
carefully documented by the test developer and followed carefully by the test
administrator” (p. 51).
Multiple Measures
Multiple Measures should be used to assess each construct being measured in a study. Usually, one measure alone cannot capture all facets of a construct. Also, when only one measure is used, the results are restricted “to the construct as assessed by a particular” measure. There are exceptions, e.g. when the DV is survival.
“Evidence for a particular hypothesis obtained on more than one measure increases the confidence that the construct of interest has been assessed.”
Confidentiality
Source of Protection
“means that the information will not be disclosed to a third party without the awareness and consent of the participant.” Are exceptions—e.g. if child abuse is found.
Cultural Influences on Data Analysis
The major issue is cultural response sets—a tendency to respond in a certain way on tests or scales.
When two cultures’ means differ, do not know whether there is a difference between the groups in levels of a construct or whether the two groups use a scale differently.
People from collectivistic cultures may not use the ends of bipolar scales.
People from one culture may check off more items on a list than people from another culture.
Guidelines for Research with Human Participants
- Institutional Approval—provide accurate information and conduct the research as described
- Informed Consent to Research
- Informed Consent for Recording Voices and Images in Research—usually required prior to the recording
- Client/Patient, Student, and Subordinate Research Participants—potential participants are not penalized for deciding not to participate or withdrawing from a study. “When research participation is a course requirement or an opportunity for extra credit, the prospective participant is given the choice of equitable alternative activities.”
- Dispensing with Informed Consent for Research—possible when participation is viewed as harmless and certain other conditions exist (e.g. use of anonymous questionnaires that would not put participants at risk).
- Offering Inducements for Research Participation—payment should not be so large as to constitute coercion.
- Deception in Research
- Debriefing
“Subjects must be guaranteed that all information they provide will be anonymous and confidential and told how these conditions will be achieved.”
Interrelations among Validities
Trade-off between controlling a situation in a study and being able to generalize it
Informing Clients about Treatment
Intervention Research Issues
whether the treatment has been shown to be effective.
Interestingly, telling participants that a treatment is experimental and has not been shown to be effective may actually reduce the effectiveness of the treatment because hope and expectancy of change are not aroused in the client.
Participants should be told that a number of treatments are being given and that assignment to a particular treatment is random (when that is actually true). Only those who agree to these conditions will participate in the study, which may affect external validity. That is, those who agree may not be a representative group.
Standardized Path Coefficients
Numbers in arrows on path diagrams that reflect the strength of the causal relationship between the latent variable and each item
In opportunity to learn
Threats to Fair and Valid Interpretations of Test Scores
Especially for achievement tests, this term refers to “the extent to which
individuals have had exposure to instruction or knowledge that affords
them the opportunity to learn the content and skills targeted by the test”
(p. 56). People who have attended schools that do not have adequate
resources may not have learned the material necessary for the test. In
these cases, “the validity of inferences about student ability drawn from
achievement test scores may be compromised” (pp. 56-57).
Debriefing
“If there is any deception in the experiment or if crucial information is withheld, the experimenter should describe the true nature of the experiment after the subject is run.” The purpose of the research should be explained.
Purposes of debriefing are:
- To eliminate or minimize any negative of effects of participation in the study.
- To show the participant why the research is valuable.
The elements of debriefing are provided in writing.
There are rare situations where debriefing can be omitted.
Systematic Sampling
take every Kth person from your list
problem can occur if there is a cyclical pattern in the list that coincides with the sampling interval
Retrospective Case-Control Design
compares groups that differ on the particular characteristic being studied on other variables that occurred in the past
Qualitative research - mixed methods
combine quantitative and qualitative research.
True Experiment
allow the most control over the IV
randomly assign each person to a condition
Variations of Information
Type of Manipulation
Here the question is: did the participants receive, attend to, and believe the information? How to check this: Give a questionnaire. The groups should differ on the information that they received.
Treatment integrity or treatment fidelity
Checking on whether a treatment was delivered as intended
Pretest-Posttest Control Group Design
Type of true experiment design
R = random assignment A = Assessment or Observation X = Intervention
Group 1: R A1 X A2
Group 2: R A1 A2
History
Internal Validity Threat
Any event inside (except the IV/intervention) or outside of the experiment that may account for the results but it has to be a plausible explanation of the results
Control group can help mitigate
Threats to External Validity
Multiple-Treatment Interference Sample Characteristics Narrow Stimulus Sampling Reactivity of Experimental Arrangements Reactivity of Assessment Test Sensitization Novetly Effects Generality across Measues, Setting and Time
Mneumonic: M. SNRRTNG
Testing
Internal Validity Threat
Effects of repeated assessment
“practice effect”
group that receives pre and post without intervention can help rule this out
Direct Observations of Behavior
Modality of Assessment
—“look at what the client actually does.” Samples of behavior.
Not totally simple because researchers need to develop codes and define the behavior to be observed, e.g. child getting out of seat in class.
Need to demonstrate reliability of the observations.
Can be done in natural or laboratory settings.
Issues:
- “decisions regarding what to observe could restrict interpretation and generality of the measure.” For example, what is emotional abuse?
- Observations are obtained at a particular time, and may not represent what the client does at other times. People may behave differently when they are aware that they are being observed. This concern can be reduced by the use of mobile devices to collect data.
- How a person behaves in a laboratory may be different from how they behave normally.
- Validity must be shown.
Multiple Comparisons and Error Rates
Data-Evaluation Validity
When multiple statistical tests are completed the likelihood of chance finding is increased
i.e. .05 alpha is only for one test. alpha goes above .05 when you do multiple tests a.k.a “experiment-wise error rate”
Cross-validation
is when you split the sample in half, develop the scale on one half, and then assess the validity of the scale on the other half as well.
Data-Evaluation Validity
Facets of the evaluation that influence the conclusions we reach about the experimental condition and its effects
Criteria for Inferring a Causal Relationship between Variables
- A strong relationship between the independent variable and the dependent variable.
- Consistency or replication, although some inconsistency can occur when there is a moderator variable.
- The cause comes before the effect.
- A “does-response relation”: more of the IV is associated with greater change in the DV.
- A reasonable process that explains how the IV leads to the DV.
- Experiment: when the IV is altered, a change in the DV occurs.
- Existence of similar findings in other areas.
Quality of Life
Measures to Evaluate the Clinical Significance of Change in Intervention Studies
refers to the client’s evaluation of how she is doing in multiple spheres (e.g. work, friendships).
Alternative-Form Reliability
Correlation between forms of the same measure when the items of the two forms are considered to represent the same population of items
Different measures
Using multiple measures may lead to finding different results for different measures. This is OK. Three reasons why different measures may yield different results:
- Using multiple informants, e.g. parent, child, and teacher may respond differently.
Getting different responses makes sense because different informants may see
different behavior (e.g. behavior at home vs. at school). - Many constructs have several facets. For example, when assessing depression, one
might look at affect, behavior, and biological symptoms. So, “the lack of
correspondence between measures is to be expected.” - The different measures may or may not correspond, depending on the client’s level of
the construct, e.g. the “magnitude of the client’s anxiety.”
Null hypothesis
specifies there is no differences between groups
Instrumentation
Internal Validity Threat
Changes in how the DV is measured over time
Can occur when any of the following is not constant:
measuring instruments
observers, raters, or interviewers
remarks or directions form the experimenter
test conditions
Most common occurrence is where raters change the criteria they are using over time
Qualitative Research
Qualitative research studies people’s experience in depth.
Qualitative research is inductive while quantitative research is deductive.
Qualitative research tends to involve open-ended responses to questions, while
quantitative research usually involves selecting from possible answers given.
While quantitative research tests theory, qualitative research generates theory.
Qualitative research can assess reliability and validity.
While quantitative research employs statistical analysis to analyze data, data analysis in
qualitative research often involves identifying themes and categories.
Qualitative research - grounded theory
theory that is developed from observation and analysis. The theory comes out of the data.
Brief Measures and Short Forms
Brief measures can be used, but their reliability and validity must be demonstrated.
Short forms of longer scales can be used if their reliability and validity are OK, e.g.
Symptom Checklist has 90 questions, but shorter forms can be used. Sometimes use of a short form leads to having a possible range of scores that is restricted; thus it is harder to find group differences. Also, short forms “may not be appropriate if there are multiple subscales or characteristics in a measure.”
Randomized controlled trails (RCTs)
When the IV involves an intervention, a true experiment becomes a RCT
Cause
One variable influences either diretly or through other variables the apprearance of the outcome
Deception
ranges from intentionally misleading the participant to not giving the participant certain information. Some deceptions are not harmful to participants, but others can be (e.g. Milgram experiment).
The decision of whether deception is justified in a particular experiment weighs the possible risks to participants against the potential knowledge the study may produce. Generally, the risks to participants are weighed against the benefits to society. “The safest way to proceed is to minimize or eliminate risk to the subject by not using active forms of deception.”
Deception in clinical psychology research generally does not involve actually misleading participants, but rather, withholding information from them. Often giving the participant in a psychology experiment all information (e.g. hypotheses) could change the results, raising validity issues.
If a researcher wants to deliberately mislead participants, he must show that this deception is necessary to achieve the goals of the research, specifically:
- that the deception is justified because the experiment will yield important information.
- that less deceptive methods would not produce the information.
- the aversiveness of the deception is justified.
Since individual rights are to be protected in research, a potential research participant must be given enough information about the study to be able to make an informed decision about whether to participate. If possible, it is better not to use deception.
Anonymity
Source of Protection
ensuring that the identity of the subjects and their individual performance are not revealed.” Accomplished by the researcher not getting identifying information (e.g. name) or keeping data coded and separate from names. [But in some cases, the researcher has a list of codes and corresponding names.]
Subjective Evaluation
Measures to Evaluate the Clinical Significance of Change in Intervention Studies
subjective evidence of clinical significance is provided is there is a large improvement. Can be assessed with the Reliable Change Index.Evaluation—by the client or those in the client’s life (e.g. family).
Effect size (ES)
magnitude of the difference between two (or more) conditions or groups
M1 - M2/SD
The smaller the variablity (the more we minimize error) the larger the effect size because SD is the denominator
Nonrandomly Assigned or Nonequivalent Control Group
—“help rule out specific rival hypotheses and decrease the plausibility of specific threats to internal validity,” like history, maturation, or testing. “Such a group may be used when a no-treatment control group cannot be formed through random assignment.”
Cross-Sectional Case-Control Design
compares groups that differ on the particular characteristic being studied on other variables that exist currently.
Competence (Informed Consent)
does the individual understand the information given and is he able to make a decision? Some groups are not considered competent to give consent—e.g. young children, people with cognitive deficits.
Posttest-Only Control Group Design
Type of True Experimental Design
R = random assignment A = Assessment or Observation X = Intervention
Group 1: R X A1 Group 2: R A1
Convergent validity
extent to which two measures that assess similar or related constructs correlate with each other
Reactivity of Experimental Arrangements
External Validity Threat
Issue of how partcipants’ knowledge that they are being studied (or in a special program or that a relationship is being examined between specific variables) changes their behavior
Clinical Significance or Practical Importance of Changes
did the treatment make a real difference in the client’s everyday life?
Falling within Normative Levels of Functioning
Measure to Evaluate the Clinical Significance of Change in Intervention Studies
evidence of clinical significance is provided if clients move from outside the normative range before treatment to within the normative range after treatment.
Choices in Using a Measure
- Using a Standardized Measure—does it really measure what the researcher wants to
measure? - Varying the Use or Contents of an Existing Measure—reliability and validity need to be demonstrated for the modified measure.
- Developing a New Measure—reliability and validity must be demonstrated.
Construct Validity
Extent to which the measure reflects the construct (concept, domain) of interest
Checking manipulation of IV
The researcher needs to assess the impact of the experimental manipulation. This is done to make sure that the IV was as intended. This is done differently depending on the type of manipulation (variation of information, variation in subject tasks and experience, or variation of intervention conditions).
Checking on the manipulation can be helpful in interpreting the results.
A manipulation check may, with certain types of manipulations, increase the reactivity of the assessment. This problem can be avoided by doing a manipulation check in a pilot study.
One has to be careful when removing subjects who were not affected by the manipulation as intended.
Qualitative Assessment
Measures to Evaluate the Clinical Significance of Change in Intervention Studies
open-ended questions to provide an in-depth evaluation. Can include ways therapy did and did not help the client.
Efficacy research
conducting treatment in highly controlled conditions
Cohort Designs
Type of observational design
here the researcher follows and studies a group or groups of people over time. So, this is a prospective, longitudinal design. A cohort “is a group of people who share a particular characteristic such as being born” in a particular year.
Theory
conceptualization of the phenomenon of interest
provides a tentative explanation of how variables are related
To be a scientific theory it must generate testable hypotheses
organizes existing research in a way that guides further studies
Can explain the basis for change and give us an idea of which moderators to investigate
<p>criterion validity</p>
<p>correlation of a measure with some other criterion (can encompass concurrent or predictive validity)</p>
