Research and Program Evaluation II Flashcards
Positivism
An objective truth exists and can only be understood if directly observable. Truth must be measurable
Post-positivism
Truth can only be approximated bc of inherent errors present when measuring reality
Constructivisim
There are multiple realities or perspectives for any given phenomenon
Critical/ideological paradigm
Researchers taking a proactive role and confronting the social structure and conditions facing oppressed or underprivileged groups
Nuremberg Code
Stemmed from Nazi Medical War Crimbes
Milgram Obedience study
Shocks
Belmont Report
Informed consent, right to withdraw, guidelines for use of deception. Stemmed from Tuskegee syphilis study
Institutional Review Board (IRB)
any institution receiving federal funding must go through IRB to conduct research with humans
Willowbrook Study
School for children with mental disabilities, parents who wanted to enroll signed a form to allow their children to be injected with hepatitis. Parents never told they could decline injections.
Jewish Chronic Disease Hospital Study
Healthy and unhealthy patients injected with cancer cells . participants never gave informed consent and weren’t told they were being injected w/ cancer
HIPAA
privacy rights of participants
Research hypothesis (2 types)
testable, concise statement involving the expected relationship between 2 or more variables.
- Nondirectional: Eg: There is a significant relationship btwn amount of sleep and career satisfaction
- Directional: There is a significant positive relationship between . . .
Null hypothesis
Statement that there is no relationship
Alternative hypothesis
Developed in order to be eliminated and addresses the question “what else could be causing the results?”
Significance level
Threshold for rejecting the null hypothesis, with values associated with alpha (typically .001, .01, .05)
Statistical significance
Cutoff point (critical value)
P value
likelihood of obtaining a result at least as extreme as the one observed assuming the null hypothesis is true
Type I Error
(alpha) Occurs when a decision is made to reject a null hypothesis when in fact that null hypothesis is true
Type II Error
(Beta) Occurs when a decision is made to retain the null hypothesis that should have been rejected bc it’s actually false
Power
Likelihood of detecting a significant relationship between variables when one is really there
Probability sampling
Sampling a known population
Non probability sampling
More common in counseling research. Involves accessing samples of convenience
Simple random sampling (probability sampling)
Every member of population has equal chance of being selected
Systematic sampling (probability sampling)
Every nth element is chosen
Stratified random sampling (probability sampling)
Population divided into subgroups based on important characteristics (gender, race, etc) and counselor draws randomly from subgroups
Cluster sampling (probability sampling)
Counselor identifies existing subgroups and not individual participants.
Multi-stage sampling (probability sampling)
2 stage (randomly select 60 schools then 10 classes from each school) 3 stage (select 200 school districts, 20 schools, 10 classes) , etc.
Convenience sampling (nonprob sampling)
Most common method. Counselor selects an easily accessible population that most likely doesn’t represent the pop of interest
Purposeful sampling (nonprob sampling)
Counselor selects sample from a population based on who will be most informative about a topic of interest. Participants selected bc they represent needed characteristics
Quota sampling (nonprob sampling)
Counselor draws needed number of participants with the needed characteristic (gender, race) from convenience sample
Random selection
Select participants from a population so that every member of the population has an equal chance of being selected
Random assignment
Randomly assigning participants to different groups, such as treatment or control group
Internal validity
Changes in the DV are due to the effects of the IVs
History (internal validity)
Extraneous incidents occur during research. Within or outside the study
Selection (internal validity)
Group differences exist before the intervention due to lack of random assignment
Statistical regression (internal validity)
Scores of participants who were selected because of their extreme score on a dependent variable (eg: extremely depressed, low achievement) are affected Those with extremely high or low scores will regress toward the mean
Testing (internal validity)
Test itself has an impact on individuals, especially when pretests are invovled. Practice effects (memory effects)
Instrumentation (internal validity)
Changes in the instrument affect results (computerized, paper and pencil, etc)
Attrition (internal validity)
Participants drop out of a research study
Maturation (internal validity)
Changes in a participant over time affect the DV
Diffusion of treatment (internal validity)
Effects of an intervention are felt by those in another group
Experimenter effects (internal validity)
Bias of the investigator influences participant responses. Halo effect (counselors subjective, usually positive perceptions of participant are genearlized to other traits) , Hawthorne effect (presence of investigator affects participant responses)
Subject effects (internal validity)
Participants change their behaviors or attitudes based on their understanding of their role as participants. Participants will pick up cues (demand characteristics) from the researcher, or research setting, that motivate them
External validity (2 types)
Ability to generalize the results of a study to a larger group
- Population external validity: involves the population to which one can generalize
- Ecological external validity: involves the conditions or settings to which one can generalize
Novelty effect (external validity)
New treatment produces positive results just because it is new to participants
Experimenter effect (external validity)
halo and hawthorne
History by treatment effect (external validity)
experiment is conducted in a particular time period replete with contextual factors that can’t be duplicated easily in another setting
Measurement of the dependent variable (external validity)
Effectiveness of a program may depend on the type of measurement used in the study
Time of measurement by treatment effect (external validity)
Timing of the administration of a posttest may influence results
Quantitative Research
Captures relationship between 2 things that can be measured numerically
Qualitative Research
Attempts to answer questions about HOW a behavior or phenomenon occurs
Mixed Method Research (2 types)
Blends or mixes qualities from quant and qual research
- Concurrent: quant and qual are collected at same time (aka triangulation)
- Sequential design: either quant or qual data is collected first. Exploratory: qual first. Explanatory: quan first
Single-Subject research designs (SSRD)
Measure how either receiving treatment or not receiving treatment affects a single subject or group of subjects
Descriptive Research
Used to describe a phenomenon and does not involve intervention (treatment). What is and how often something occurs. Usually done as precursor or in conjunction with other research methods
Longitudinal Research
Data collection for a particular group over time
Cross-sectional research
Examines different groups or cohorts at a particular point in time, with differences in experience being compared
Survey Research
Method of collecting quant and qual data
Action research
Research that is typically carried out by professional counselors in an effort to improve their own practice/efficiency
Pilot study
Smaller than full scale study and is designed to assess feasibility of expanding a study to a much larger scale
Nonexperimental Research designs
Quant. Exploratory and descriptive. No intervention involved, no variables or conditions manipulated. Goal is to observe and outline properties of variable.
Experimental research designs
Involves intervention (counselor manipulates conditions and variables). Assess cause and effect relationships
Descriptive design (nonexperimental research design)
Thoroughly describing a variable at one time (simple descriptive) or over time (longitudinal design).
Longitudinal design (trend, cohort, panel)
Trend=assess general population over time, with new people each time data collected. Cohort=assess the same population over time, panel = same individuals over time
Comparative design
Investigates group differences for a particular variable.
Correlational research design
Describes relationship between two variables
Ex post facto research design
(AKA causal-comparative design) involves examining how an IV affects a DV by assessing whether one or more pre-existing conditions possibly caused differences in groups
Within-subject design
assess changes that occur within the participants in a group as they experience some intervention
Between-group design
Exploring the effects of a treatment or intervention between 2 groups or among more than 2 groups
Split-pot design
ASsessing a general intervention on the whole plot and assessing other treatments to subplots within the whole plot
Pre-experimental designs
Do not use random assignment. Thus not considered true experimental designs
One-group posttest-only design (Pre-experimental designs)
Group receives intervention and change is measured
One-group pretest-posttest design (Pre-experimental designs)
Group evaluated before and after intervention
Nonequivalent groups posttest-only design (Pre-experimental designs)
No attempt made to begin the study with equivalent groups of participants.
True experimental designs (5 types)
(AKA randomized experimental designs) are gold standard for experimental designs: at least 2 groups for comparison and random assignment
Randomized pretest-posttest control group design (true experimental)
participants assigned to 2 groups (1 is control) and both are measured before and after intervention
Randomized pretest-posttest comparison group design (true experimental)
participants assigned to at least 2 groups, and each group receives a distinct intervention.
Randomized posttest-only control group designs (true experimental)
random assignment of participants to treatment or control, and then administer an intervention to one group, then measure outcome
Randomized posttest-only comparison group design (true experimental)
with at least 2 groups for comparison and no control group
Solomon four-group design (true experimental)
comprehensive true experimental design. using 4 randomly assigned groups, presence of pretest and intervention can be assessed more rigorously
Quasi-experimental designs
Useful when it’s impossible/inappropriate to randomly assign participants to groups
Time series design (quasi-exp)
Repeatedly measuring before/after intervention for one group or control group for comparison
Single-Subject Research Designs (SSRDs)
Allow for repeated measures of a target behavior over time for an individual or select group of individuals
Descriptive statistics
Organize and summarize data - describe data
Frequency distribution
Tabulation of the number of observations (or number of participants) per distinct response for a particular variable
Frequency polygon
Line graph of the frequency distribution
Histogram
Graph of connecting bars that shows the frequency of scores for a variable
Bar graph
Looks similar to histogram but displays nominal data
Central tendency
What is the typical score
Mean
average
Median
Middlemost score. Better to use if there are outliers
Mode
Most frequently occuring score (bimodal, multimodal)
Variability
How dispersed are scores from a measure of central tendency?
Range
Range of data. can be affected by outliers.
Interquartile range
eliminates top and bottom quartiles and provides range around median score
Standard deviation
most frequently reported indicator of variability for interval or ratio data
Variance
Standard deviation squared
Skewness
Asymmetrical distribution with data points that don’t cluster symmetrically around a mean
Kurtosis (3 types)
Meso (normal curve), letpo (tall and thin), platy (flat and wide)
Inferential statistics
Used to try and describe results beyond what is garnered from the data alone
Degrees of freedom
Number of scores, or categories of a variable that are “free to vary”. It’s a restrictor
Correlation coefficient
Information about relationship between two variables. Is there a relationship? What direction? Is it strong?
Spurious correlation
when a correlation overrepresents or underrepresents the actual relationship
Regression studies (3 types)
Prediction studies are extensions of correlational studies and known as regression studies
- Bivariate regression: how well scores from independent variable predict scores on dependent variable
- Multiple regression: Involves more than one predictor variable
- Logistic regression: dependent variable is dichotomous
T-test (2 types)
Compares 2 means for one variable
- Independent t test: involve comparing 2 independent groups on one dependent variable
- Dependent t test: involve similar groups paired or matched in some meaningful way, or the same group tested twice
Analysis of variance (ANOVA)
Involves having at least one independent variable in a study with 3+ groups or levels
Post hoc analysis
Allows one to examine every possible pairing of groups for a particular independent variable after one has concluded there are main effects (significant difference among 2 or more groups making up a single independent variable)
Analysis of covariance (ANCOVA)
Test includes independent variable as a covariate, or a variable that needs to be statistically adjusted and controlled in order to look at the relationship of the other independent variables and the DV
Multiple analysis of variance (MANOVA)
Similar to anova but involves multiple DVs
Multiple analysis of covariance (MANCOVA)
Similar to ancova but involves multiple DVs
Nonparametric Stats
Used when professional counselors are only able to make a few assumptions about the distribution of scores in the underlying population
Chi-square test
Used with 2 or more categorical or nominal variables, where each variable contains at least two categoires.
Factor analysis
The purpose is to reduce a large number of variables to a smaller number of factors
Meta analysis
allows researcher to combine and synthesize the results of numerous similar studies for a particular outcome or DVs
Phenomenology
Used to discover meaning of participants’ lived experiences with goal of understanding person
Grounded theory
Generate theory that is grounded in data from participants’ perspectives for a particular phenomenon
Consensual qualitative research (CQR)
approach that combines elements of phenomenology and grounded theory
Ethnography
Researcher describes and provides interpretations about the culture of a group or system. Observation
Purposive sampling
Obtain information-rich cases that allow for maximum depth and detail regarding a certain phenomenon
Saturation
Seek sample sizes that reach point of redundancy of information. Sample until no new data comes in.
Convenience sampling
Sampling based on availability or accessibility. Least desirable method. Least trustworthy
Maximum variation sampling
Sampling a diverse group and searching for core patterns and individual perspectives based on unique participant characteristics
Homogenous sampling
Selecting participants for a specific subgroup with theoretically similar experiences
Stratified purposeful sampling
Identify important variables pertaining to a research question and sampling subgroups that best isolate each variable
Purposeful random sampling
identify a sample and randomly select participants from that sample
Comprehensive sampling
Sampling all individuals within a system
Typical case sampling
selecting the “average” participants or those who represent the typical experience for a phenomenon
Intensity sampling
identify those with intense but not extreme experiences of a phenomenon
Critical case sampling
Sampling those with intense and irregular experiences
Extreme or deviant sampling
Looking for the bounds of difference, or those with the most positive and negative experiences
Snowball, chain, or network sampling
A pool of participants is derived through obtaining recommendations from earlier participants
Criterion sampling
develop criteria and selecting all cases that meet criteria
Opportunistic or emergent sampling
Taking advantage of an unexpected opportunity and changing one’s research design to include a particular individual in one’s pool
Theoretical sampling
as theory evolves, sampling those who best contribute information for one’s theory
Confirming/disconfirming case sampling
Including cases that confirm and add depth to one’s evolving theory can also provide exceptions or potentially disconfirm elements of one’s theory
Qualitative interviewing (3 types)
- Unstructured: no predetermined question
- Semi-Structured: preset interview protocol with flexibility to change/add/remove questions
- Structured: standardized interview protocol and ensures same level of info is collected from each individual
Participant observation
Counselors engaging in observations
Unobstrusive methods
Collecting photos, videos, documents, archival data, artifacts
Contact summary sheet
Single page snapshot of specific contact
Document summary form
Attached primarily to unobstrusive data sources (like newspapers or artifacts)
Data display
presents organized data in a table format or a figure containing interconnected nodes
Inductive analysis
Involves searching for keywords and potential traditions that involves searching for keywords and potential themes from the data without significant preconceived notions of what theory or theories fit the data
Trustworthiness
The validity or truthfulness of findings - why others should trust your data collection
Credibility (trustworthiness)
Believability of your findings
Transferability (trustworthiness)
degree to which data transfers to other contexts and participants
Dependability (trustworthiness)
degree of consistency of results over time and across researchers
Confirmability (trustworthiness)
reflects that interpretation of data is a genuine reflection of participants’ views
Needs assessment (program evaluation)
Initial step to explore whether a program is needed for a particular group of individuals
Process evaluation (program evaluation)
assessment of an ongoing program to ensure that program activities match plans
Outcome evaluation(program evaluation)
Determination of program success by investigating how participants are performing with respect to themselves as well as others who are not involved in the program
Efficiency analysis (program evaluation)
comparison of the costs of the program in relation to need, processes, and outcomes.
Accountability
Process of providing feedback about a program to its stakeholders
Stakeholders
Any individuals involved in or affected by the program
Formative evaluations
Ongoing evaluation of a program throughout its implementation to ensure that it is being conducted as planned and acted upon as needed
Summative evaluation
involves assessment of the entire program to determine the degree to which program goals and objectives have been met
Steps in program evaluation
- Identify the program to be evaluated
- plan the evaluation
- conduct a needs assessment and provide recommendations
- determine what success is
- Select data sources
- Monitor and evaluate the program progress
- Determine the degree to which a program is successful
- analyze the program’s efficiency
- continue, revise, or stop program based on findings
ABCD model
Audience, Behavior, Conditions, Description
