Research and Program Evaluation Flashcards
1
Q
Research
A
the systematic process of collecting and analyzing data for some purpose such as investigating a problem or answering a question
2
Q
Evidence-based inquiry
A
the search for knowledge using empirical data which has been gathered systematically
3
Q
Quantitative research
A
- assumes social facts have a single objective reality
- tends to study samples or populations
- researchers try not to influence collection of data (instruments)
- statistical methods comparing and contrasting groups occurs
- researchers examine for causes and relationships
4
Q
Qualitative research
A
- assumes multiple realities socially constructed by individuals/groups
- tends to study individual units - person, family, community - in naturalistic setting
- researchers may be primary instrument for collecting data (through observation)
- researchers’ impressions, judgments, feelings may be used
- goal is to describe the nature of things
5
Q
When to choose qual vs. quant
A
both kinds of research are valued
- one is chosen over the other because it better fits the assumptions of the researcher and the nature of the problem under investigation
- some professional journals prefer to publish one kind of research over the other
6
Q
Inductive research
A
begins at the real world, practical level (small and builds to large theory)
- tends to be descriptive, correlational, historical
- leads to building of theory
- closer to qualitative research
7
Q
Deductive research
A
springs from theory which is already established (starts broad and goes smaller to specifics)
- tries to determine what the relationships are between elements of the theory and may be experimental in nature
- closer to quantitative research
8
Q
Quantitative; Non-experimental designs
Survey
A
may occur through questionnaires, interviews, etc. and is used to measure attitudes, perceptions, etc.
- ex. Public Opinion Poll
- often has low response rate, below 50%
- unless you know that characteristics of non-respondents are similar to respondents, must be cautious in generalizing
9
Q
Quantitative; Non-experimental designs
Descriptive
A
describes an existing state of events
- numbers may be used to characterize groups/individuals
10
Q
Quantitative; Non-experimental designs
Comparative
A
investigates whether there are differences between two+ groups
- no manipulation of conditions experienced by each group
11
Q
Quantitative; Non-experimental designs
Correlational
A
this research method uses the correlation coefficient to determine the degree of relationship between two+ variables or phenomena
ex. income level and attitude toward counseling
bivariate: correlational data describing the nature of two variables
multivariate: more than two variables
12
Q
Quantitative; Non-experimental designs
Ex post facto (Causal-comparative)
A
studies possible causal relationships among relationships ex post facto (after the fact) - no random assignment
- do not manipulate any variables; focus is on what has already happened
- may generate several reasons (causes) for the relationships you discover
- uses t-tests and ANOVAs
13
Q
Quantitative; Experimental designs
True experiment
A
characterized by the use of experimental and control groups with random assignment to each
- used to determine cause-and-effect relationships
- ex. 60 college freshmen are enrolled in English class. 30 are randomly assigned to one-hour per week writing lab, and the others comprise a control group. End-of-semester essay exam results are analyzed to see if the lab was associated with better writing skills
14
Q
Quantitative; Experimental designs
For experiments, there are design variations such as:
A
- treatment and control group with posttest only
- treatment and control group with pretest and posttest
- two different treatment groups with control groups and posttest
- etc.
15
Q
Quantitative; Experimental designs
Quasi-experiment
A
similar to experimental research except that randomization of subjects to treatment and control groups is not possible
- may be that no control or comparison group is available
- result from such research will not be as unequivocal as results from a true experimental study
- ex. a school has two classrooms of 4th graders. Each classroom is taught arithmetic by a different method for the school year. In May, arithmetic achievement is compared for the two classrooms using scores on a national exam
16
Q
Types of Research - Qualitative
Qualitative
A
emphasizes gathering data about naturally occurring phenomena (individual’s and groups’ living experiences) and events
- data collection may be in terms of words rather than numbers
17
Q
Types of Research - Qualitative
Two principal qualitative research designs:
Interactive - Case Study
A
the case may be a program, activity, set of individuals who are bounded in time and place
18
Q
Types of Research - Qualitative
Two principal qualitative research designs:
Interactive - Ethnography
A
a description and interpretation of a cultural or social group/system. Data is typically collected through observation and interviewing
- Issue of observer bias is important
19
Q
Types of Research - Qualitative
Two principal qualitative research designs:
Noninteractive - analytical research
A
conducted primarily through document analysis
- ex. historical analysis (collecting and analyzing docs describing former events)
- ex. biographical analysis (written or oral)
- ex. legal analysis (focuses on law and court decisions)
20
Q
Mixed-Method Research Designs
Mixed-Methods
A
combine quantitative and qualitative methods in the same research effort
- researcher retains the flexibility to use both types of design
- typically designs are used sequentially (quant. may be gathered first and then qual. used to further explain or elaborate on findings using surveys, interviews, focus groups)
21
Q
Other Specialized Research Designs and Types
Single-subject design
A
studies the effects of a program or treatment on an individual or group treated as an individual, usually after a baseline has been established
22
Q
Other Specialized Research Designs and Types
Action research
A
conducted in an attempt to improve services or a program
- may be viewed as having an evaluative function
23
Q
Other Specialized Research Designs and Types
Pilot study
A
small-scale research effort often used to determine the feasibility of a large scale effort with emphasis on refining procedures and instrumentation
24
Q
Other Specialized Research Designs and Types
Longitudinal research
A
collecting data from the same group of individuals over a period of time (panel study)
- ex. studying career development of school children by reinterviewing them every two years until they were high school seniors
25
# Other Specialized Research Designs and Types
Cross-sectional research
consists of collecting data from different groups at the same time and examining these differences
- ex. studying career development by interviewing each grade of students at the same time
also called Synchronic method
26
# Research outcomes may be measured two ways:
Within-subjects
examining what changes occur within the members of a group
| IPSATIVE
27
# Research outcomes may be measured two ways:
Between subjects
examining what changes occur between two or more groups
28
Meta-analysis
research comparing findings across studies
- i.e., the results of many studies are examined simultaneously and one or more research questions answered
29
# Internal Validity
Internal Validity
experiments are internally valid to the extent that extraneous variables have been controlled
- to the extent that the treatment variable is the only one producing the observed changes, the experiment is internally valid
30
# Threats to Internal Validity
Selection of subjects
differences in the results between two groups may not be due to the treatment variable experiement by one group because the composition of the two groups are different to begin with (probably not randomly selected)
31
# Threats to Internal Validity
Instrumentation
differences in results between two or more groups may be due to instruments which are unreliable or because the instruments are changed during the study
- or, perhaps the observers recording data become fatigued or bored and record behaviors differentially over time
32
# Threats to Internal Validity
Maturation
results may be due to maturational or other changes in the subjects and not due to the treatment being applied. This is especially important if research data is gathered over a long period of time
33
# Threats to Internal Validity
Mortality or attrition
losing subjects during the study could lead to different results than if everyone had stayed. Subjects with the most or least amount of important characteristics to the study may be the ones dropping out
Think of normal curve extremes
34
# Threats to Internal Validity
Experimenter bias
the responses of the subjects may be influenced by the researcher. This may occur by treating some subjects differently, reinforcing different behaviors, as well as the presence of many other variables which deliberately or unintentionally influence subjects
35
# Threats to Internal Validity
Statistical regression
sometimes subjects in a study are recruited because of extreme high or low scores (e.g., self-esteem, social skills) on the dependent variable being measured. Due to statistical regression, future measures would expect these individuals to score closer to the mean score even without any intervention
36
# External Validity
External Validity
an experiment is externally valid to the extent that the results may be generalized to people and situations beyond the study
- there are several threats to external validity of experiments and some of these are also threats to internal validity
37
# Threats to external validity
Selection of subjects
if subjects are not randomly selected, the results may only apply to the subjects in the study
- the results can only be generalized to people with similar characteristics
38
# Threats to external validity
Ecological validity
the research has ecological validity if the results can be generalized from one setting or circumstance to another. Sometimes the circumstances, conditions, physical surroundings of the research are so unique that the results cannot be generalized beyond that study
39
# Threats to external validity
Subject reactions (Reactivity): Hawthorne effect
| 1
the influence in performance which occurs when subjects receive attention or know they are participating in research
40
# Threats to external validity
Subject reactions (Reactivity):
Demand characteristics
| 2
all the cues, info, knowledge, even rumors the subject has heard about the experiment which are likely to influence their performance
41
# Threats to external validity
Subject reactions (Reactivity):
Experimenter bias (Rosenthal effect)
| 3
the changes in the subject's behavior brought about by the researcher's expectations, behaviors, attitudes
- Rosenthal conducted research into this phenomenon and called it Pygmalion effect, referring to the self-fulfilling expectation of doing well because it is expected
42
# Threats to external validity
Subject reactions (Reactivity):
Placebo
| 4
any control treatment
should be identical to the experimental treatment except for the critical item being studied
even so, control subjects may be influenced by the placebo adn react in unintended ways
43
# Threats to external validity
Novelty and disruption effects
the measured effect of the treatment on the subjects may be due to its novelty or the disruption it causes
- being selected for research may be exciting and energizing; as it continues it may begin to disrupt routine and one's typical schedule
- when novelty and disruption wear off or stabilize, there may be no long-term effects of the treatment
44
# Levels of measurement
Levels of measurement
determine the statistic you can use
45
# Levels of measurement
Nominal
variable's qualities or categories. use non-parametric test (ex. Chi-square)
ex. male/female
## Footnote
Categories YES
Rank order NO
Equal Spacing NO
True zero NO
46
# Levels of measurement
Ordinal
differences in some magnitude of the variable
ex. scores on exam can be ranked from highest to lowest
## Footnote
Categories YES
Rank order YES
Equal Spacing NO
True zero NO
47
# Levels of measurement
Interval
intervals between the numbers on a scale contain the same amount of the variable throughout the scale. provide a constant unit of measurement
**NO REAL ZERO**
Ex. Fahrenheit temperature (the distance/interval between 11 and 12 degrees is the same as the distance between 100 and 101 degrees)
## Footnote
Categories YES
Rank order YES
Equal Spacing YES
True zero NO
48
# Levels of measurement
Ratio
numbers are on a scale which has a true zero
Numbers can be compared by ratios
Ex. weight, distance, time
Ex. someone who weights 200 lbs is twice as heavy as someone who weighs 100 lbs. BUT we cannot say someone is twice as introverted as someone else (so this ex is not ratio)
## Footnote
Categories YES
Rank order YES
Equal Spacing YES
True zero YES
49
# Sampling
Sampling
how well samping is conducted will determine how validly we can generalize from a sample to a population
Involves the selection of a part of the population
50
# Probability Sampling
Random sampling
all the individuals in the population have an equal and independent chance of being selected
51
# Probability Sampling
Stratified sampling
sampling in such a way that major subgroups in the population will be sampled (ex. gender, age, ethnicity, etc.)
- can be proportional or disproportionate
52
# Probability Sampling
Proportional stratified sampling
randomly selecting the same proportion of individuals for the sample as they represent proportionally in the major subgroups in the population
Ex. if 1/2 of a population is Hispanic and 1/2 is white, you would randomly select your sample to be 1/2 Hispanic and 1/2 white
53
# Probability Sampling
Systematic sampling
researchers select members of the population at a fixed interval determined in advance
ex. selecting every 3rd study on a roster
54
# Probability Sampling
Cluster sampling
the unit is not an individual but a naturally occurring group of individuals (ex. classrooms, city blocks)
Clusters are randomly selected for the study
55
# Nonprobability Sampling
Non-random or nonprobability samples
samples of convenience or volunteer samples
- cannot be counted on to yield a normal distribution of scores but can yield useful/important data
56
# Nonprobability Sampling
Purposeful sampling
in some studies, there may be no interest in generalizing findings so this may be used.
- selecting participants based on specific criteria such as characteristics or lived experiences
- can be helpful to gather in-depth info on a topic
Possibilities:
- comprehensive sampling where every case/event is selected
- there is extreme-case or typical-case selection
Ex. researchers studying individual's lived experience with depression post flyers in counseling center
57
# Nonprobability Sampling
Convenience sampling
participants are recruited who are easily accessible and who are usually close in proximity
ex. you select your classmates for your research on graduate counseling experiences
58
# Nonprobability Sampling
Snowball sampling
participants are asked to assist researchers in identifying other potential subjects
Can be helpful when conducting research about people with specific traits
ex. you recruit 5 counselors in supervision and have each give you names of other counselors in supervision who you then recruit
Study tip: Michael Scott stuck in a pyramid scheme coded
59
# Nonprobability Sampling
Quota sampling
the researcher identifies participants meeting different criteria that are needed for the study and then sets a target number for each category in the sample. Then, participants are nonrandomly recruited to fill the quotas for each criteria
ex. you are studying EMDR for trauma recovery, so you recruit 4 veterans, 4 first responders, etc.
60
Difference between Quota sampling and Stratified sampling
- both methods specify subcategories and attempt to fill targeted numbers for each subcategory
**Quota sampling**: participants selected nonrandomly so have more control over who you select (e.g., convenience or purposive sampling)
**Stratified random sampling**: participants selected using random selection technique (once in a subcategory). Since this is a probability method, you can generalize
61
Sample Size
influences statistical hypothesis testing
- tables for determining appropriate sample sizes are available
- 5-10% of the population is generally used
Suggested minimum sample sizes for different research:
- correlational (30)
- ex post facto and experimental (15)
- survey (100)
62
# Statistical analysis may be
Descriptive
- sometimes called summary
- used to describe the data collected
- ex. means, SDs, frequency counts, percentages
63
# Statistical analysis may be
Inferential
used to make inferences from the sample to the population
- goal is to determine probability of some event occurring
- ex. t-test and ANOVA
64
# Statistical analysis may be
Parametric
used when a sample is randomly drawn from a population and the data is normally distributed
- para (two-sided) data that yields a bell curve
- assume the variance of the sample is homogeneous (similar) to the variance of the population from which your sample is drawn
- scores would give you normal bell curve
- ex. t-tests and ANOVA
65
# Statistical analysis may be
Nonparametric
used when you cannot make any assumption about the shape of the curve or variance of the population scores (they may not be normally distributed and variances may not be homogeneous)
ex. Chi-square, Mann-Whitney U Test, Wilcoxon Signed-Ranks Test
66
# Variables
Independent variable
the variable you manipulate or vary to see what change occurs in the DV
- precedes or is antecedent to DV
- sometimes you group/categorize IV (ex. you categorize a group of individuals by gender F/M; high school students into grades)
- sometimes called stimulus variable, predictor variable, experimental variable
67
# Variables
Dependent variable
variable you are measuring or trying to change. Value of this variable depends upon the value of the independent variable you selected
ex. the effects of three kinds of therapy (IV) on anxiety (DV)
- also called respone variable, outcome variable, criterion variable
68
# Research questions and hypotheses
Some ask research questions to be answered
ex. Is there a relationship between disciplinary practices and leadership styles for men in the military?
ex. is there is a significant difference in the mean number of client contacts between public and private counseling agencies?
69
# Research questions and hypotheses
Null hypothesis
states there is no difference between the variables or groups measured
ex. There are no significant differences in final academic grades (GPA) between boys and girls finishing 10th grade)
70
# Research questions and hypotheses
Alternative hypothesis (directional)
states that one group's scores will be significantly different from another group's score (one-tailed test)
Ex. the GPA of girls will be higher than the GPA of boys finishing 10th grade.
71
# Research questions and hypotheses
Alternative hypothesis (non-directional)
there will be differences between the groups but which group has higher/lower scores is not indicated
Ex. the GPA of girls and boys finishing the tenth grade will be different.
72
# Significance level
Significance level
will determine likelihood of making Type I or Type II errors
73
# Significance level
How to select level for significance
generally set at 0.05 or 0.01 (or up to 0.001)
level selected is willingness ot make an error (rejecting null hypothesis when there is not a significant difference between the groups)
- at 0.05, you are willing to accept the possibility of rejecting the null hypothesis in error 5 times out of 100
74
Type I error (Alpha)
rejecting the null hypothesis (which states that there is no difference) when it is correct
! You can change the probability of Type I error by changing significance level
- Saying there is a difference when there isn't
- Saying there is a wolf when there is not a wolf
75
Type II error (Beta)
failure to reject the null hypothesis when there is a difference
! Small sample sizes can result in Type II errors
- Saying there is no difference when there is a difference
- Villagers believing there is no wolf when there actually was one
76
Negative relationship between Type I and Type II errors
as significance level goes down (e.g., from 0.05 to 0.01), Type I error decreases but Type II error increases
As one goes up, other goes down
77
T-test
used to determine whether mean scores of two groups are significantly different from each other
- can only be used when there are two groups (two mean scores)
- You would compare your obtained value of t with the value of t presented in a Table of T values to make the determination
78
Difference between T-test and ANOVA
t-test looks at differences between two groups
ANOVA is 3+ groups
79
One-Way Analysis of Variance (ANOVA)
examines the differences between two or more (usually 3+) groups based on one IV (that has at least 2 levels)
- this yields an F-ratio which can be compared to other values listed in F Distribution Table to determine whether significant differences are present
- ex. types of therapy (levels: CBT, DBT, ACT)
**Use One-Way ANOVA when there is only 1 IV with multiple levels)**
80
Factorial ANOVA
used to determine if mean scores on 2+ IVs (factors) differ significantly from each other and whether the factors interact significantly with each other
- Types of counseling (3 levels: CBT, DBT, ACT)
- Duration of counseling (2 levels: 4 weeks, 8 weeks)
- One DV: symptoms of anxiety
- Factorial design: 3 x 2
81
Multivariate ANOVA (MANOVA)
** USE when have 2+ DVs**
- integrates all DVs into a single composite variable
- ! DVs must be related!!
- used to gain insight into how different factors/treatment influence multiple outcomes simultaneously
ex. effect of MSC-T program on internalizing disorders (depression, anxiety, somatic symptoms)?
- IV: two groups: treatment/no treatment
- DV: single composite variable of internalizing disorders (depression, anxiety, somatic symptoms)
**Study tip**: Man, what a lot of DVs
82
Analysis of Covariance (ANCOVA)
used when influence of one or more IVs on the DV is controlled (i.e., initial group differences are adjusted statistically on one or more variables that are related to DV)
**Covariate**: variable that influence the DV but are not of primary interest
- ex. Therapy (2 levels: DBT and Gestalt)
- DV: Depression level
- Covariate: peer support
83
Identifiers for each type of ANOVA
**One-Way ANOVA**: only one IV with multiple levels (ex. multiple types of therapy)
**Factorial ANOVA**: 2+ IVs, 1 DV (ex. medication and types of counseling)
**MANOVA**: 2+ IVs; 2+ DVs integrated into single composite variable
**ANCOVA**: used when influence of one or more IVs on the DV is controlled (covariate)
84
# Post hoc or multiple comparisons tests
Post hoc tests
if your ANOVA yields a significant F value, you still will not know which particular pair of mean scores is significantly different from each other
- must apply post hoc tests to determine between which groups the significance lies
Ex. Tukey's HSD, Scheffe
85
Nonparametric tests
use when you cannot assume that your distribution of scores is normally distributed (resembles a normal curve) or that the variance of your sample is similar to the variance of the population (homogeneity)
86
# Nonparametric tests
Mann-Whitney U Test
when you collect data from two samples that are independent (uncorrelated/unmatched) from each other and the scores are not normally distributed
**Study tip**: Mann-Whitney **U** = **u**ncorrelated or **u**nmatched
87
# Nonparametric tests
Wilcoxen signed-rank test
when you have scores for two samples and these scores are correlated (you matched them or got two scores for each individual - repeated measures)
However, the scores do not approximate a normal distribution
**study tip**: Wil**c**oxen = **c**orrelated
88
# Nonparametric tests
Kruskal-Wallis test
when you have more than two mean scores on a single variable. A nonparametric one-way ANOVA
89
# Parametric with Non-parametric Equivalent
Independent t-test
Mann-Whitney U test
90
# Parametric with Non-parametric Equivalent
Dependent t-test
Wilcoxon Signed-Rank Test
91
# Parametric with Non-parametric Equivalent
One-way between-groups ANOVA
Kruskal-Wallis test
92
# Parametric with Non-parametric Equivalent
One-way within-groups ANOVA
Friedman test
93
# Dumb Mnemonic Device for Parametric with Non-Parametric Equivalent
Man
What
Krispy
Fries
## Footnote
Just remember the order of Parametrics (T-test indep. dep. ; ANOVA between within)
Independent t-test | Mann-Whitney
Dependent t-test | Wilcoxon
One-Way Between Groups ANOVA | Kruskal-Wallis
One-Way Within Groups ANOVA | Friedman
94
Chi-square test
examines the relationship between two or more categorical (nominal) variables
- utilizes a contingency table: displays frequencies for categorical variables
Study tip: **c**hi = **c**ontingency
95
Chi-square test example
Is there a significant association between gender (F/M) and preference for learning mode (online/in-person) among students?
IV: Gender (F/M); Learning mode preference (online/in-person)
Then do a contingency table displaying the data of males who prefer online/in-person; females who prefer online/in-person; totals
96
# Between Groups Design Experiments
Posttest-Only Control Group Design
participants are randomly assigned to either a treatment or control group and are tested on DV once
- compares how groups differ from each other after treatment to determine the effect
97
# Between Groups Design Experiments
Pretest-Posttest Control Group Design
participants are randomly assigned to either treatment or control group and are tested on DV twice
- measures the effect of a treatment (IV) on an outcome variable (DV) by comparing the group scores before/after exposure
- should use same scales for pretest and posttest
98
# Between Groups Design Experiments
Solomon Four-Group Design
takes into account the influence of pretesting on susequent posttest results
- allows you to determine whether the pretest by itself made a difference, whether the treatment by itself made a difference, whether a combination of pretest and treatment made a difference, or whether nothing mades a difference
99
Multiple regression
the use of the correlation coefficient to determine the strength of the relationship of predictor (independent) variables on a criterion (dependent) variable
- adds together the predictive power of several IVs (predictors)
ex. predictor variables like high school GPA, class rank, ACT scores may be used to predict the criterion (outcome) variable, which could be freshman year GPA
100
Scatterplot (scattergram)
a graphic representation of the relationship between two variables for a group of individuals
each point on graph is an individual score (stress score on horizontal X axis and depression score on vertical Y axis)
- can draw a line of best fit to display the direction, form, and strength of the relationship
101
Correlation coefficient
measures and describes the relationship between two variables
** Pearson correlation coefficient (r)** measures a linear correlation
- sign (+/-) indicates direction of the relationship
- numerical value (0.0-1.0) indicates strength of relationship
**Study tip**: pearson r is used for interval or ratio data
102
Factor analysis
a statistical method using the correlation coefficient to determine whether a set of variables can be reduced to a smaller number of factors
ex. breaking SES into income, education, occupation factors etc.
103
Factor analysis example
ex. a factor analysis of all the items on a long inventory with 15 scales may uncover only 4 or 5 factors independent of each other underlying the scales. Thus many of the constructs of the 15 scales overlap
104
Likert scale
measures attitudes/opinions
allows for several response choices
0 - not at all
1 - occassionally
2 - often
3 - always
105
Biserial correlation
an appropriate correlation coefficient to use when one variable yields continuous data and the other yields data that is dichotomous
106
Cross-sectional
studying or measuring characteristics of several groups at the same time
107
Longitudinal
studying or measuring characteristics of a group over a period of time
also called diachronic method
108
Degrees of freedom
the number of observations that are free to vary
109
Single-blind technique
the subject does not know whether they are in control or experimental group
this helps eliminate **demand characteristics** which are cues/features of a study which suggest a desired outcome (the subject can manipulate/confound an experiment by purposely trying to confirm or disprove hypothesis
110
Double-blind technique
occurs when neither the researcher nor the subject knows who is getting the active substance or the placebo
reduces **experimenter** effects - which flaw an experiment because the experimenter might unconsciously communicate their intent or expectations to the client
111
Halo effect
the tendency for the observer (researcher or data collector) to form an early impression of the person being observed and then letting this impression influence observations or ratings of that individual
- can be positive or negative
Can happen with things that are not being evaluated having an impact on the ratings (like finding someone attractive makes you rate them higher on their counseling skills)
112
Heteroscedasticity
one end of a distribution of scores has more variability than the other end resulting in a fan-like appearance
113
Homoscedasticity
there is an equal distribution of scores throughout the range of scores (i.e., around the line of best fit)
114
Inter-rater reliability
In qualitative research, the reliability calculated by correlating the responses of several raters
115
Observer bias
the tendency of researchers to see, hear, remember what they want to
116
Pilot study
a preliminary trial/test of research techniques and measures
117
Placebo
control treatment that gives subjects the same amount and kind of attention as experimental group subjects get (reduces hawthorne and Rosenthal effects)
- could be sugar pill etc.
118
Rank-order correlation (Spearman rho)
used when the values of the variables are reported in rank form rather than continuous
used for ordinal data (rho ends in O!)
119
# Counseling program evaluation
Counseling program evaluation began as
emphasis on accountability in human services field accelerated in 1970s and continues today coming primarily from funders like Health Maintenance Organizations, insurance companies, gov. funding sources, etc.
- determines a 'bottom line'
120
# Counseling program evaluation
Need for counseling program evaluation
acute need to demonstrate the efficacy of counseling in general and effectiveness of specific theories, techniques, approaches in particular
- emphasis on short-term therapy (6-12 sessions) argues for research and evaluation to determine what works well for what kinds of problems with what clients under what circumstances
121
# Counseling program evaluation
Goals and measurable objectives must be
specified in advance
- without those, evaluation data has little relevance
122
# Counseling program evaluation
On what level (community, individual, etc.) does evaluation occur
the effectiveness of counseling techniques/processes often occurs on individual client basis
123
# Counseling program evaluation
Evaluation
the systematic collection of evidence of the worth of a program, process, technique
124
# Counseling program evaluation
Types of evaluation:
Formative evaluation
ongoing, process evaluation to measure the effectiveness of a technique or part of a program
- tries to determine how well a new technique, process, treatment works
- process evaluation
125
# Counseling program evaluation
Types of evaluation:
Summative evaluation
summary or product evaluation designed to measure the effectiveness of a program, usually conducted at the end of a cycle such as a school year, fiscal year, etc.
- conducted to see how well agency or program goals have been met
- usually a product (document) is generated so this is 'product' evaluation vs 'process' evaluation (formative)
126
# Ethical Issues in Research
Confidentiality
no one should have access to the data except for the researcher and research assistants
- release of research data to others is only ethical with consent of the subject
127
# Ethical Issues in Research
Deception
deception may be justifiable if no risk to subjects is involved. Such research should be followed by debriefing of the subjects
128
# Ethical Issues in Research
Informed consent
subjects should be informed of the research they will be participating in and give their consent
129
# Ethical Issues in Research
Ethical issues revolve around benefits of research
significance of research results should outweigh the potential benefits denied to the control group
130
# Ethical Issues in Research
Institutional Review Board (IRB)
approves human subject research projects when conducted within institutions or agencies when federal funding is involved
131
Writing research
- most writing should be done in APA format
- sexist language is to be avoided
- never submit a manuscript to more than one journal for publication consideration at a time
132
APA's Journal of Counseling Psychology
publishes more counseling research articles than any other periodical in our field
133
Ethics in research
- subject is informed of risks
- negative after-effects are removed
- subjects can withdraw at any time
- confidentiality of subjects is protected
- results will be presented in an accurate format that is not misleading
- use only techniques that you are trained in
134
Hypothesis testing pioneered by
R.A. Fisher
135
Biserial correlation
one variable is continuous (i.e., measured using an interval scale) while other is dichotomous
ex. trying to correlate state licensing exam scores to NCC status
136
Phi-coefficient correlation
both variables are dichotomous (two valued)
ex. correlating gender with certification status (does or does not have a certification)
137
Withdrawal designs
ABA or ABAB
ABA: baseline is taken (A), intervention is implemented (B), new baseline taken (A)
ABAB: if the pattern for the second AB administration mimics that of the first, then the chances increase that B (intervention) caused the changes rather than extraneous variable
- ABAB is good to rule out extraneous variables
138
Empirical rule
68-95-99.7
68% of scores fall within +/-1 SD, 95% within +/-2SD, 99.7% within +/-3 SD
139
Mode
the highest point on the curve!
the point of maximum concentration
140
The benefit of standard scores like percentiles, t-scores, z-scores, stanines, SDs over raw scores is that
a standard score allows you to analyze the data in relation to the properties of the normal bell curve
141
horizontal x axis is also known as
abscissa
142
vertical y axis is also known as
ordinate
143
Stanine
divide the distribution into 9 equal intervals with stanine 1 as lowest 9th and 9 as the highest 9th.
5 would be the mean
144
Survey problems
- need at least 50-75% completion rate to be accurate
- poor construction of the instrument
- low return rate
- subjects are often not picked at random
145
Nocebo effect
a placebo effect with a negative effect
ex. when a doctor says that individuals with this condition live for 6 weeks
146
Standard error of measurement (SEM)
tells the counselor what would most likely occur if the same individual took the same test again
