Research & Program Evaluation Flashcards
Research definition
the systematic process of collecting and analyzing data for some purpose such as investigating a problem or answering a question
Evidence-based inquiry
is the search for knowledge using empirical data that has been gathered systematically
Inductive research
- begins at the real world level
- tends to be descriptive, correlational, historical and leads to the building of a theory
I make observations of xyz and create a theory based on trends, observations, correlations that form a theory (bottom up approach)
Deductive research
-springs from theory that’s already established
- tries to determine what the reln’ps are bw elements of the theory
- may be experimental in nature
I apply a theory and then make deductions about the data I collect that highlight the reln’ps therein (top-down approach)
Types of non-experimental designs
(quant) SEDCC
Survey
Ex Post Facto
Descriptive
Comparative
Correlational
types of experimental designs
(quant)
true experiment
quasi-experiment
types of quantitative research designs
non-experimental
experimental
survey research
non-experimental quantitative research
- questionnaires, surveys
- used to measure attitudes, perceptions
- response rate is often low (<50%)
ex: public opinion poll
descriptive research
non-experimental quantitative research
- describes an existing state of events
- numbers used to characterize individuals or groups
comparative research
non-experimental quantitative research
- investigates differences bw >=2 groups
- no manipulation of conditions experienced by group
correlational research
non-experimental quantitative research
- uses the correlation coefficient to determine the degree of the relnp bw >=2 variables
ex: income level and attitude towrd counseling
Ex Post Facto
non-experimental quantitative research
- aka ‘causal-comparative’
- studies possible causal reln’ps among variables ex post facto (after the fact)
- no variables manipulated, focus is on what already happened (after the fact)
- may generate several causes/reasons for the reln’ps they discover
- typical statistics used are t-test and analysis of variance
true experiment
experimental design (quantitative)
- uses experimental and control groups w/random assignment to each
experimental designs used to determine cause and effect reln’ps
quasi-experimental
experimental design (quantitative)
- similar to experimental research but w/o randomization of subjects to treatment and control group
- results not as unequivocal as true experime
What are the typical statistics used for ex post facto research ?
t-test and analysis of variance
What is another name for ex post facto?
causal-comparative
what are the 2 types of qualitative research designs?
1) Interactive
- case studies, which = a prgm, activity, or set of ppl in same place
- ethnography
–> a description and interpretation of a cultural or social group, or system.
–>data collected via observation and interview
–> observer bias is important
2) NonInteractive
- analytical research conducted primarily through document analysis
ex: historical analysis (using docs to describe former events); biographical analysis (oral or written); legal analysis (court and legal decisions)
what is ethnography?
a type of qualitative research design that is interactive in nature
–> a description and interpretation of a cultural or social group, or system.
–>data collected via observation and interview
–> observer bias is important
mixed-method design
both qual and quant
typically the designs are sequential (ie, using quant first and then qual to further explain the findings)
other specialized research designs and types
SLAPC
single-subject
longitudinal
action research
pilot study
cross-sectional
single study design
studies the effects of a program or treatment on an individual or group as an individual, usually after a baseline has been established
action research
conducted in an attempt to improve services or a prgm.
- often viewed as having an evaluative function
pilot study
a small scale research effort often used to determine the feasibility of a large scale effort w/emphasis on refining procedures and instrumentation
Longitudinal study
consists of collecting data from the same group of individuals over a period of time.
- aka a ‘panel study’
cross-sectional research
consists of collecting data from different groups at the same time and examining the differences
how may research outcomes be measured?
in 2 ways:
1) within-subjects: examining what changes occur with the members of a group
2) between subjects: examining what changes occur bw >=2 groups
meta-analysis
refers to research comparing findings across studies
(ie results of many studies examined and one or more questions answered)
internal validity
the extent to which extraneous values are controlled (ie, the exten that the treatment variable is the only one producing the observed changes)
threats to internal validity
SIMMESH
Selection of subjects (differences in ppl) - differences in the results bw groups due to composition
Instrumentation (differences in results due to instruments or recorders became bored and recorded differently)
Maturation - results may be due to maturational or other changes rather than the treatment
Mortality or attrition
Experimenter bias - responses of the subjects may be influenced by the researcher
Statistical regression
History - may be extraneous incidents (national, local, tragedies, etc…)
External validity
experiment is valid to the extent that the results may be generalized to ppl and situations beyond the study.
threats to external validity
Selection of subjects - if subjects aren’t randomly selected then results can only apply to one group in the study, and generalized only to ppl with similar characteristics
Ecological validity - exists if results can be generalized from one setting/circumstance to another (sometimes they’re so unique that they can’t)
Subject reactions (Reactivity) - results may not be generalized if any of these are present: Hawthorne effect, Demand Characteristics, Experimenter Bias, Placebo
Novelty and disruption effect - the measured effect of the treatment may be due to its novelty or the disruption it causes. when novelty or disruption wear off, there may be long-term effects of the treatment
hawthorne effect
refers to the influence in performance when subjects know they’re part of research or receive attn
Demand characteristics
are all the cues, info, knowledge, rumors the subject has heard about the experiment
Experimenter bias
aka Rosenthal Effect
refers to the changes in the subject’s behavior brought on by the researcher’s expectations, behaviors, attitudes
How did the Rosenthal Effect come about?
He did research on experimenter bias and called it the Pygmalion Effect, which refers to the self-fulfilling expectation of doing well because it is expected
Ecological validity
a type of threat to external validity
exists if results can be generalized from one setting/circumstance to another (sometimes they’re so unique that they can’t)
Novelty and disruption effect
a type of threat to external validity
the measured effect of the treatment may be due to its novelty or the disruption it causes. when novelty or disruption wear off, there may be long-term effects of the treatment
Subject reactions (Reactivity)
a type of threat to external validity
results may not be generalized if any of these are present: Hawthorne effect, Demand Characteristics, Experimenter Bias, Placebo
what does the level of measurement determine, how many are there, and what are they?
- determines the type of statistic you can use
4 levels (NOIR):
Nominal - the #s represent qualities or categogies (ie, male or female)
Ordinal - [think ORDER] #s represent differences in some magnitude of the variable; can order the data from high to low, top to bottom (ie, scores arranged on a test from highest to lowest)
Interval - the intervals bw #s are the same (ie, distance bw 4 & 5 is same as bw 23 & 24)
Ratio - the #s are on a scale which has a true value of zero. #s can also be compared by ratios (ie, smn who weighs 200lbs is twice as much as smn who weighs 100lbs, but in counseling we can’t say smn is twice as introverted as smn else)
types of sampling
CROPPS
Cluster sampling
Random sampling
Other samples
Proportional sampling
Purposeful sampling
Stratified sampling
Cluster sampling
the sampled unit is a group of naturally occurring individuals rather than the individuals themselves
ie, classes, city blocks
random sampling
there’s an equal and independent chance of being selected
Other sampling types
samples of convenience or volunteer samples
Proportional stratified sampling
randomly selecting same proportion of individuals as they represent proportionally in the major subgroups in the population
(ie, if half the population [in a neighborhood or wherever] is white and other half is hispanic, then set the sample set to half white and half hispanic)
Purposeful sampling
- used when there’s zero interest in generalizing the findings, as it’s a comprehensive sampling when every case or event is selected
Stratified sampling
major subgroups in population are sampled
what does the sample size do
influences statistical hypothesis testing
what are suggested minimum sample sizes for correlational research?
30
what are suggested minimum sample sizes for ex post facto and experimental research?
15
what are suggested minimum sample sizes for survey research?
100
types of statistical analysis
DIPN
descriptive or inferential
parametric or nonparametric
descriptive: used to describe the data
inferential: used to make inferences about the sample to the population; goal is to determine probability of an event
parametric: represents normally distributed data; normal curve because ‘para’=2 equal sides; used when variance of the sample is homogenous/similar to variance of the population
nonparametric: used when you cannot make assumptions about the shape of the curve or variance of population (scores not normally distributed, variances not homogenous). For example, you want to talk about the school’s academic performance but you only have data from the gifted classes, so you won’t have a normal distribution of scores
independent vs dependent variables
the IV is the variable you manipulate/control to see what changes. May be grouped into a category by gender, class, sex etc..
- the IV precedes the DV
-sometimes called the stimulus variable, experimental variable, or predictor variable (SEP because it’s separate from the DV being tested)
the DV is the variable you are measuring or trying to change. The value of the DV depends on the value of the IV you selected.
ie, the effect of three different therapeutic techniques (IVs) on anxiety (DV).
- sometimes called the response variable, outcome variable, criterion variable
other names for the independent and dependent variables
IV
- sometimes called the stimulus variable, predictor variable, or experimental variable
DV
- sometimes called the response variable, outcome variable, criterion variable
What are the 3 types of hypotheses?
Null hypothesis: there is no difference between the variables or groups measured.
- Ie, there are no significant differences in GPA between boys and girls finishing 10th grade
Directional hypothesis: states that one group score will be significantly different from others’
Ie., the GPA of girls will be higher than the GPA of boys finishing 10th grade
Non-directional hypothesis: there will be differences between groups, but what the difference is is not indicated
Ie., the GPA of girls and boys finishing 10th grade will be different
Significance level
Refers to the likelihood of making the mistake of either rejecting or accepting the null hypothesis when you shouldn’t.
Common significance levels are .05, .01, and .001.
- For example, at .05 you were willing to accept the possibility of rejecting the null hypothesis in error five times out of 100 (if you perform the experiment 100 times)
Type I vs Type II errors
Type I error is rejecting the null hypothesis when you shouldn’t
Ie., a type one error would mean you say there is a significant difference in XYZ when in fact, the difference is not significant.
Type II error is retaining the hypothesis when you shouldn’t should
Ie., a type two error would mean that you would say there is not a significant difference in XYZ when in fact there is.
As the significance level goes down (ie .05 to .01), type I error decreases, but type II error increases.
As one type of error goes up, the other goes down and vice versa.
T-test
Used to determine whether the main scores of two groups are significantly different from each other.
Can only be used when there are 2 groups (2 mean scores)
How many types of analysis of variance and what are they?
One-way
Factorial
Multivariate
When to use a One way analysis of variance
When you have only one variable at 3 or more levels
p259-260
What is ANOVA and when would you use it?
factorial analysis of variance
Used to simultaneously determine whether mean scores on 2 or more variables (factors) differ.
Ie, the effects of two or more factors (independent variables) are being measured
What is MANOVA
Multivariate analysis of variance
Used when the analysis involves more than one DEPENDENT variable
What is ANCOVA
Analysis of covariance
Used similarly to analysis of variance except that the influence of one or more IVs on the DV is controlled.
- analyzes the differences between three or more group means while controlling for the effects of at least one continuous covariate
Post hoc or multiple comparisons test
Use post hoc (after the fact) to determine whether different group means or combinations thereof are significantly different.
Examples:
Scheffe’s
Tukey’s Honestly significant difference
Newman-Keuls
Duncan’s new multiple range test
What is homogeneity?
When the variance of your sample is similar to the variance of the population
When would you use a non-parametric test? Examples?
When you cannot assume that your distribution of scores is normally distributed, or there is a lack of homogeneity (when the variance of your sample is similar to the variance of the population)
Examples:
Mann-Whitney U Test:
- When you collect data from two samples that are independent from each other, and the scores are not normally distributed
Wilcoxen Signed-rank test:
- when you have scores for two samples, and these scores are correlated, but they do not approximate a normal distribution
Kruskal- Wallis test:
- when you have more than two mean scores on a single variable. This is a non-parametric one way analysis of variance.
Chi-square:
- used when you have nominal data such as groups or categories. Used to determine whether two distributions differ significantly compared to expectations
Solomon four-group design
Examines the effect of any pretest used on the experimental treatment. Ie, maybe a pretest sensitized subjects to the “need” for the treatment being tested
Multiple regression
Uses the correlation coefficient to determine the strength of the relationship of predictor (independent) variables on a criterion (dependent) variable.
Multiple regression ads together with the predictive power of several independent variables (predictors)
Ex: predict your variables, like high school GPA, class rank, and ACT scores may be used to predict the criterion (outcome) variable, such as end of freshman year GPA
Chi-square
Non parametric test used when you have nominal data such as groups or categories.
Used to determine whether two distributions differ significantly compared to expectations
It’s also known as a “goodness of fit” statistic because it measures how well observed data aligns with expected distributions. The test’s primary purpose is to determine if two variables are independent, meaning they are unrelated.
Scatterplot
Graphic representation of the relationship between two variables for a group of individuals
Factor analysis
A statistical method using the correlation coefficient to determine whether a set of variables can be reduced to a small smaller number of factors.
For example, a factor analysis of 15 items on an inventory may uncover that only four of them are independent of each other. Thus many of the constructs of the 15 items overlap.
Biserial Correlation
A correlation coefficient to use when one variable yields continuous data and the other yields data that is dichotomous
Degrees of freedom
Refers to the number of observations that are free to vary
Double blind technique
Occurs when neither the researcher nor the subject knows who is getting the active substance or placebo
Halo effect
The tendency for the observer to form an early impression of the person being observed, and then letting this impression in influence observations for ratings of that individual
“They seem nice so they must have good intent”
Heteroscedasticity
When one end of a distribution of scores has more variability than the other end, resulting in a fan like appearance
Homoscedasticity
When there’s an equal distribution of scores throughout the range of scores, i.e. around a line of best fit
Inter-rater reliability
In qualitative research, the reliability calculated by correlating the responses of several raters
Placebo also means what?
Control treatment
Rank order correlation
Used when the values of the variables are reported in rank form rather than continuous
What does parsimonious mean and what is another term for it?
It refers to the Tendency to be miserly and not overspend
Occam’s Razor
When is an experiment confounded?
When an undesirable variable which is not controlled by the researcher is introduced in the experiment
Hypothesis testing is most closely related to the work of
RA Fisher
What is used by police departments to screen candidates to become police officers?
The law enforcement aptitude battery assessment (LEAB)
Barnum effect
“There’s a sucker born every minute”
Aka The Forer Effect
A cognitive bias that causes people to believe that general descriptions applies to them specifically (psychic readings)
A set personality trait or characteristics that’s so vague can be used for anyone
Is the Rosenthal effect an aspect of informal or formal observation, both, or neither?
Both
The Rosenthal effect suggests that an experimenter’s beliefs about an individual can cause the person observed to behave in a certain way and therefore skew experimental data. It applies to both informal and formal ways of observation.
What is the GAF?
The Global Assessment of Functioning (GAF), a number from 1 to 100 that roughly assessed a client’s overall level of functioning, is no longer used as part of a formal diagnosis. Some counselors may still find it useful.
Can training raise IQ scores?
Yes, in some circumstances
In some cases, training can raise an IQ score; keep in mind that the assertion is that the person can learn skills which can raise their score, not that their brain has changed per se.
What is the ‘matching hypothesis’?
We tend to choose partners who match our level of attractiveness
The matching hypothesis asserts that when choosing long-term mates, we tend to choose partners who match our overall level of attractiveness.
The matching hypothesis does not assert that attitude, personality type, or mental illness is a factor in matching.
When is it appropriate for supervisors to counsel their supervisees?
Under no circumstances
Counselors providing supervision to supervisees should not provide actual counseling services under any circumstances.
What does “terminal drop” refer to?
The loss of intelligence in the last five years of life; aka “terminal decline”
Terminal drop refers to a theory about intelligence in older people that states there is a marked decline in intellectual functioning in the last five years of life.
The term does not refer to general physical decline or breathing.
what is the most common eating disorder?
According to research, binge eating disorder (in which a person engages in a period of concentrated overeating) is the most common eating disorder.
What do psychotherapy notes include?
Psychotherapy notes form a special class of clinical documentation. They are to be kept separate from the rest of the clinical record, and contain items such as (a CAT in the notes):
- clinical impressions
- analysis
- therapeutic conversations
What is the most well-known criticism of behavior therapy?
It treats symptoms but not causes
Behavior therapy, in its many forms as descended from the work of B. F. Skinner, does seem to have a good track record of proven results in treating various issues. However, in directly addressing specific behaviors and treating them, it tends to ignore the inner life of the person and therefore can be said to treat symptoms of problems rather than causes of those problems (as in more insight-oriented psychotherapy).
What is the point of informed consent?
To help clients make decisions with the best information possible
Clients should be in a position to make the best decisions possible with the best information possible, whether in counseling or as participants in research. This process is referred to as informed consent.
What is the paradox of religious and spiritual involvement for clients?
Spirituality offers benefits but can create strain
There is a paradox involved in clients embracing religion or spirituality, or in therapists relying on religious or spiritual content. Though spirituality and religion can offer a number of benefits, the same content can create issues. For example, religion is capable of increasing a client’s well being through mood, identification with an other, and marital satisfaction; however, religion can also create anxiety and depression.
To what does psychotherapy integration refer?
Using strategies from various schools of counseling
There are many schools of counseling, each of which has unique strengths and potential liabilities. Psychotherapy integration refers to using strategies from various schools of counseling.
According to research, how many people does it take to produce conformity in a group?
3
The phenomenon of conformity in groups has been studied to indicate exactly when a process of groupthink begins to take hold. The available research seems to suggest that it takes as little as three persons in a group to begin producing conformity.
Which traits have been proven by research to be the most useful in group leaders?
None
Despite the fact that such items as flexibility, enthusiasm, and common sense are considered helpful to some degree, research has yet to reliably identify any single trait as more valuable than the rest in group leaders.
What is the model in which a recovering addict helps a practicing addict?
Chemical dependency (CD)
The chemical dependency, or CD, model of substance use treatment, is notable due to its use of a recovering addict to help a practicing addict. It is believed, according to this model, that the experience of being an addict is so unique and unknown to those who have not experienced it that a recovering addict is much more helpful in substance use treatment of addicted persons.
When does marital satisfaction tend to be highest?
at the time of the wedding
Research indicates that marital satisfaction tends to be highest at the time of the wedding, though there is idiosyncratic ebb and flow in most marriages.
Does EMDR require eye movement to be effective in treating conditions caused by trauma?
No, eye movement is not required
Suprisingly, it has been found that, in treating some kinds of conditions caused by trauma, eye movement desensitization and reprocessing therapy (EMDR), does not always need the eye movement component to be effective.
What is the relationship between birth order and the need for affiliation?
The need to affiliate, or to have associational bonds with others, appears to decrease with birth order, according to research. Thus, a third child would have the least need for affiliation according to this research.
What do defense mechanisms protect against, according to Sigmund Freud?
Protection against anxiety
The defense mechanisms, such as denial, rationalization, projection, and so on, were, according to Sigmund Freud, protection against the anxiety produced by one’s life. Thus, if one practices denial, one does not have to face whatever is being denied.
Freud did not suggest that defense mechanisms were protection against depression, psychosis, or violence.
What is androcentric bias?
Bias oriented toward men
“Androcentric bias” is a term that emerged from feminist therapy, which suggests that most therapeutic interventions and theories have at their heart a bias oriented toward men in general.
What are the phases of ‘stress inoculation’?
A three-step process to reducing stress
Stress inoculation refers to the three-step process innovated by Donald Meichenbaum which seeks to deal with stress.
1) educational stage where the client monitors themselves
2) rehearsal stage in which the learning is applied to practice
3) application stage where the learning is put to use in the real world situation causing stress.
In terms of family therapy, what is an “irrelevant distractor”?
communicative role
Virginia Satir, as a part of experimental conjoint family therapy, offered terminology for various roles that do not serve the purpose of addressing family issues when they are present. One of these is the “irrelevant distractor,” who, when the family system is under stress, will resort to the strategy of using irrelevant information to block progress. The term applies to the person and not the material of their distracting behavior.
What does codependency refer to?
The phenomenon of being addicted to an addicted person
Though the term can refer to various aspects of relational dynamics in substance abuse, the concept of codependency essentially refers to one person being “addicted” to caregiving, supporting, or otherwise engaging in an unhealthy relationship of support for a person who is experiencing substance use issues.
The term does not refer to two people being addicted to the same substance, though this does sometimes occur in codependent scenarios.
What is “therapeutic surrender”?
the moment when a client lets down their guard with someone from a different culture
In multicultural counseling, therapeutic surrender refers to the moment when a client from one culture lets down their guard with someone from a different culture. This moment is key to the establishment of trust and productive therapeutic rapport.
what is miliieu therapy?
Managing a client’s whole physical and social environment
Milieu therapy seeks to help clients with their issues by managing the whole social and physical environment in which therapy is taking place; in effect, the management of the milieu amounts to a kind of therapy in its own right.
what is “flight from reality” in counseling?
Resorting to psychosis to escape current reality
In providing empathic clinical counseling, it is important to remember that psychosis is sometimes part of a mind’s inability to deal with current difficulties or other issues. This is known as “flight from reality.”
What is the purpose of open-ended questioning?
To give the client greater control in response
Open-ended questions, those in which there is not a concrete, specific response anticipated by the counselor, are of use in treatment chiefly because they give greater control to the client in terms of how they would like to respond to a given query.
The point of open-ended questions is not to make the client speak more often, encourage honesty, or to activate key insights.
What does social comparison theory suggest?
Social comparison theory suggests that we compare ourselves with others to evaluate ourselves and orient our aspirations and current status to an overall social standard.
Social comparison theory does not suggest that the objective of such comparison is feeling better or dismissing others.
what is an organismic variable?
one the research cannot control yet exists, such as height, weight, or gender
what happens if a research increases the sample size in an experiment?
raising the sample size helps lower the chance of risk of chance/error factors. Basically, differences revealed in large sample sizes are likely more genuine than differences revealed in a smaller sample
if a researcher uses two IVs then the statistic of choice would be ___?
a two-way ANOVA.
3 IVs would be a three-way ANOVA, etc…
Hawthorne vs Placebo effect
The Hawthorne effect and the placebo effect are both psychological phenomena that can affect experimental results.
The Hawthorne effect is when participants change their behavior because they are being observed, while the placebo effect is when participants change their behavior because they believe in a treatment
which way would a positive distribution lean?
the tail would be toward the right
hint: the direction of the tail indicates whether the distribution is positively or negatively skewed
