Research & Program Evaluation Flashcards
Compares two means for one variable; provide a ?-ratio
t-tests; t-ratio
Involve comparing two independent groups (usually assigned randomly) on one dependent variable.
EX: Gender and achievement
Independent t-tests
Involve similar groups paired or matched in some meaningful way or the same group tested twice.
EX: Same group of high school students; pre-test and post-test
Dependent t-tests
A statistical test that involves having at least one independent variable in a study with three or more groups or levels.
EX: Household income (Below $20,000; $20K-$40K; $40K-$60K; and $60,000+) divided into 4 levels/groups
Analysis of Variance (ANOVA)
ANOVA
Indicates 2+ more of the group means are statistically different
F ratio
Used with 1+ independent variable; yields both main effects and interaction effects (i.e., significant differences among groups across or more independent variables).
EX: 2 treatments (Cognitive therapy and Interpersonal therapy) compared for effectiveness on males and females
Factorial ANOVA
Statistical test that includes an IV as a covariate, or a variable that needs to be statistically adjusted & controlled in order to look at the relationship of other IVs & the DV. EX: Examining the relationship bw household income & work satisfaction w/gender as a covariate (statistical effects of gender are removed from the analysis to control for any effects that gender may have on work satisfaction)
ANCOVA
Allows examination of every possible pairing of group means for a particular IV after one has concluded that there are main effects (i.e., significant difference among two or more groups comprising a single IV) in an ANOVA.
Post hoc analysis
Chi-Square test, Mann-Whitney U test, Friedman’s rank test, Wilcoxon’s signed-rank test, Kolmogorov-Smirnov Z procedure, Kruskal-Wallis test
Types of nonparametric statistics tests
Used when researchers are only able to make a few assumptions about distribution of scores in the underlying pop. Specifically, their use is suggested when nominal or ordinal data are involved or when interval or ratio data are not distributed normally (i.e., are skewed).
Non-parametric statistics test
A nonparametric statistical test used to determine whether two or more categorical or nominal variables are statistically independent.
EX: investigating rxp of whether to terminate counseling (yes/no) and the gender of the counselor (male/female)
Chi-square test
Nonparametric statistical test. Analogous to a parametric independent t-test, except uses ordinal data instead of interval or ratio data. Compares the ranks from two groups.
EX: Compare students Grades 9-12 w/education aspiration (diploma, 2/4-yr degree, grad) as DV
Mann-Whitney U-test
Nonparametric statistical test. Similar to Mann-Whitney U test, but used with samples smaller than 25 participants
Kolmogorov-Smirnov Z Procedure
Nonparametric statistical test analogous to an ANOVA. Extension of Mann-Whitney test. Used with 3 or more groups per independent variable & an ordinal-scaled dependent variable.
Kruskal-Wallis test
Nonparametric statistical test. Equivalent to a dependent t-test. Involves ranking the amount & direction of change for ea pair of scores.
EX: Comparing perceived level of competence before and after a training program
Wilcoxon’s signed-ranks test
A nonparametric statistical test similar to Wilcoxon’s signed-ranks test in that it is designed for repeated measures. It may be used with more than two comparison groups.
Friedman’s rank test
Graphical Representations of Experimental Designs:
- ___ = A, B, C, D
- ___ = (R)
- ___ = O
- ___ = X, Y, Z
- ___ = n/a
- Groups
- Random Assignment
- Observation
- Intervention
- control group (no intervention or observation)
A type of experimental design used when it is impossible or inappropriate to randomly assign participants to groups. Often used with nested data (e.g., classrooms, counseling groups) or naturally occurring groups (e.g., males, African Americans, adolescents). Two common types:
(a) nonequivalent groups pretest-posttest control (sometimes called comparison group designs), and
(b) time series design.
Quasi-experimental designs
Common type of quasi-experimental design.
Observations made at equal time intervals with same testing procedures. Characterized by repeatedly measuring before and after an intervention for one group, & uses a control group for comparison.
(Graphical Representation)
control group interrupted time series design
A: 0 -> 0 -> 0 -> X -> O -> O -> O
B: 0 -> 0 -> 0 -> n/a -> O -> O -> O
Common type of quasi-experimental design.
Counselor keeps groups intact; administers a pretest; administers treatment to only one group
(Graphical Representation)
nonequivalent groups pretest-posttest control group design
A: O -> X -> O
B: O -> n/a -> O
Common type of quasi-experimental design. Observations made at equal time intervals with same testing procedures. Characterized by REPEATEDLY measuring before & after intervention; can measure one group, or include control group for comparison.
time series design
Common type of quasi-experimental design.
Counselor keeps groups intact; administers pretest, administers treatment to 2+ groups, and then gives the groups a posttest.
(Graphical Representation)
nonequivalent groups pretest-posttest comparison group design
A: O -> X -> O
B: O -> Y -> O
C: O -> Z -> O
Common type of quasi-experimental design.
Characterized by repeatedly measuring before and after an intervention for one group.
(Graphical Representation)
one-group interrupted time series design
A: 0 -> 0 -> 0 -> X -> O -> O -> O
Aka randomized experimental designs. The gold standard; involve 2+ groups for comparison & random assignment. Common types
(a) randomized pretest-posttest control group design
(b) randomized pretest-posttest comparison group design,
(c) randomized posttest-only control group design,
(d) randomized posttest-only comparison group design, and
(f) Solomon four-group design
true experimental designs
Common type of true experimental design.
Participants are assigned to one of 2+ groups, each group receives a distinct intervention, and the effectiveness of interventions is compared through the use of pre- and posttests
(Graphical Representation)
randomized pretest-posttest comparison group design
(R) A: O -> X -> O
(R) B: O -> Y -> O
(R) C: O -> Z -> O
Common type of true experimental design.
Uses 4 randomly assigned groups so the presence of a pretest and the presence of an intervention can be assessed more rigorously.
(Graphical Representation)
Solomon four-group design (R) A: O -> X -> O (R) B: O -> n/a-> O (R) C: n/a-> X -> O (R) D: n/a->n/a->O
Common type of true experimental design.
Involves the random assignment of participants to a tx or control group, administering an intervention to one group, and then measuring the outcome.
(Graphical Representation)
randomized posttest-only control group design
(R) A: X -> O
(R) B: n/a -> O
Common type of true experimental design.
Participants are assigned to 2 groups (one group serves as the control); both groups are measured before & after an intervention.
(Graphical Representation)
randomized pretest-posttest control group design
(R) A: O -> X -> O
(R) B: O -> n/a -> O
Common type of true experimental design.
Similar to a randomized posttest-only control group design, but with 2+ groups for comparison and no control group
(Graphical Representation)
randomized posttest-only comparison group design
(R) A: X -> O
(R) B: Y -> O
A type of experimental design that does not use random assignment, thus failing to control for internal validity threats. Three common types:
(a) one-group posttest only design,
(b) one-group pretest-posttest design, and
(c) nonequivalent groups posttest-only design.
pre-experimental designs
Common type of pre-experimental design.
No attempt to begin study w/equivalent groups; one group receives intervention; change is measured; other group serves as control w/o intervention & assessed at same time as the other group.
(Graphical Representation)
nonequivalent groups posttest-only design
A: X -> O
B: n/a -> O
Common type of pre-experimental design.
A group receives an intervention and change is measured.
(Graphical Representation)
one-group posttest only design
A: X -> O
Common type of pre-experimental design.
A group is evaluated before and after an intervention
(Graphical Representation)
one-group pretest-posttest design
A: O -> X -> O
A type of quantitative sampling that involves sampling a known population using randomization. Methods include (a) simple random sampling; (b) systematic sampling; (c) stratified random sampling; and (d) cluster sampling.
Probability sampling
A method of probability sampling.
Every nth element is chosen
systematic sampling
A method of probability sampling.
Every member of population has equal chance of being selected
simple random sampling
A method of probability sampling.
Population is divided into subgroups and the professional counselor draws randomly from the subgroups
stratified random sampling
A quantitative sampling method that typically involves accessible, convenient samples and does not use randomization. Methods include convenience, purposeful, and quota.
Nonprobability sampling
A method of Non-probability sampling.
The selection of a sample from a population based on who will be most informative about a topic of interest; participants are selected because they represent needed characteristics
purposeful sampling
A method of probability sampling.
The professional counselor identifies existing subgroups and not individual participants. Sometimes involve multiple stages, such as a two-stage random sample (e.g., randomly select 60 schools and then 20 classes from those schools), a three-stage random sample (e.g., randomly selecting 200 school districts, then 20 schools from each district, and then 10 classes per school), and so forth.
cluster sampling
A method of Non-probability sampling.
Drawing the needed number of participants with the needed characteristic (e.g., gender, race) from the convenience sample.
quota sampling
A method of Non-probability sampling.
The selection of an easily accessible population that most likely does not fully represent the population of interest
convenience sampling
The degree to which changes in the dependent variable are due to the effects of the independent variable. Threats include history, selection, statistical regression, testing, instrumentation, attrition, maturation, diffusion of treatment, experimenter effects, and subject effects.
Internal validity
Scores of participants who were selected because of their extreme score on a dependent variable are affected; regress towards the mean
Statistical regression
The phenomenon of research participants knowing what to expect and learning something from a pretest that helps to improve their performance on future tests.
Practice Effects (memory effects)
The degree of peakedness of a distribution. Distributions can be mesokurtic, leptokurtic, and platykurtic.
Kurtosis
Normal curve distribution
Mesokurtic
Distribution curve that is “flat and wide”
platykurtic
Distribution curve that is “tall and thin”
leptokurtic
Outlines policies that guide researchers who use human subjects; outlines participants’ rights and researchers’ responsibilities in conducting research. It requires these studies to be approved by an institutional review board (IRB). Protects the rights of those who may be more vulnerable to abuse, such as children, fetuses, and those with mental disabilities
“Title 45: Public Health, Part 46: Protection of Human Subjects” of the Code of Federal Regulations (45 CRF 46)
A report prompted by the ethical issues arising from the Tuskegee syphilis study. Created by the former U.S. Department of Health, Education, and Welfare to outline ethical principles and guidelines for research involving human participants.
Belmont report
______ is a set of ethical principles for research using humans; resulted from _______ following _______ Guarantees research participants’ choice to be involved in a research study (i.e., voluntary consent) and the right to terminate their participation at any time.
Nuremberg Code;
Nuremberg trials; World War II
Outlines the privacy rights of participants pertaining to health information
Health Insurance Portability and Accountability Act (HIPAA) of 1996
A part of “Title 45: Public Health, Part 46: Protection of Human Subjects” of the Code of Federal Regulations. It outlines policies that guide researchers who use human subjects. It requires these studies to be approved by an institutional review board (IRB).
Common Rule, or Subpart A, of 45 CFR 46
Infamous, unethical research study. Sought to investigate blind obedience through a series of shocks (15–450 volts). “Learners” (part of research team) pretended to be shocked by “teachers” (participants) when responding with incorrect answers. Although most participants showed signs of internal struggle, 65% “shocked” learners @ max level. No debriefing after the study.
Milgram obedience study
Any institution receiving federal funding must sponsor an ___ in order to approve proposals to conduct research w/human subjects. Consists of 5 members & typically operates under the same general guidelines.
Institutional Review Board (IRB)
Infamous, unethical research study in which both healthy & unhealthy patients were injected w/live cancer cells so that researchers could better understand the impact of cancer on the basis of health status. Participants never provided informed consent & were not told they were being injected with cancer cells.
Jewish Chronic Disease Hospital study
A philosophical paradigm that proposes an objective truth exists and can only be understood if directly observable (i.e., “truth” must be directly measurable). has been closely tied to quantitative research.
Positivism Paradigm
philosophical paradigm that proposes truth can only be approximated because of inherent errors present when measuring reality. The concept of measurement error in terms of validity and reliability is emphasized.
Post-positivism Paradigm
Paradigm that contends that there are Multiple realities or perspectives for any given phenomenon. The truth differs for individuals and is an internal manifestation. Foundation of most qualitative research today
Constructivism (interpretivism) Paradigm
A philosophical paradigm that centers on researchers taking a proactive role and confronting the social structure and conditions facing oppressed or underprivileged groups. Heavily tied to qualitative research design.
Critical/ideological Paradigm
A measure of the strength of the relationship between two variable in a population
Effect Size
The likelihood of obtaining a result at least as extreme as the one observed assuming the null hypothesis is true.
P value
Used to predict outcomes (dependent variable) from a predictor variable(s) (independent variable).
Regression studies
Type of regression study.
How well scores from an independent variable (predictor variable) predict scores on the dependent variable (criterion variable)
bivariate regression
Type of regression study.
Involves more than one predictor variable when each predictor variable is weighted (beta weights) in a regression equation to determine the contribution of each variable to the criterion variable
multiple regression
A qualitative approach used to discover or describe the meaning or essence of participants’ lived experiences with the goal of understanding individual and collective human experiences for various occurrences
Phenomenology
A qualitative approach used for the purpose of generating theory that is grounded in data from participants’ perspectives for a particular phenomenon.
Grounded theory
Qualitative research approach. Describes a case, a distinct system of an event, process, setting, or individuals or small group of individuals
Case study
Common types of Qualitative research (7)
Case Study Grounded theory Phenomenology Participatory Action Research (PAR) Consensual Qualitative Research (CQR) Ethnography Biography
A qualitative research tradition that focuses on change of the participants and researcher as a result of qualitative inquiry. Involves a collaborative approach to problem solving between the researcher and other key stakeholders.
Participatory Action Research (PAR)
A qualitative approach that combines elements of phenomenology and grounded theory and involves researchers selecting participants who are very knowledgeable about a topic and remaining close to data without major interpretation with some hope of generalizing to a larger population.
Consensual Qualitative Research (CQR)
A qualitative research tradition in which the researcher describes and provides interpretations about the culture of a group or system.
Ethnography
A qualitative research tradition that seeks to identify personal meanings individuals give to their social experiences. The researcher gathers stories and explores meanings for an individual as well as how the stories fit into a broader social or historical context.
Biography
Type of regression study.
Used when the dependent variable is dichotomous; may be similar to a bivariate or multiple regression
logistic regression
A data management tool used in qualitative research that provides a single-page snapshot of a specific contact, such as an interview or observation.
Contact summary sheet
In hypothesis testing, the likelihood of detecting a significant relationship between variables when one is really there.
Power
The amount of shared variance between the two variables; computed by squaring the correlational coefficient.
Coefficient of determination
4 categories of research
Qualitative
Quantitative
Mixed-method
Single-subject
Key Category of Research Design (4 total).
Blends or mixes designs from quantitative and qualitative research. Most important characteristic: can strengthen what either research designs provides individually.
Mixed-Methods Research
Mixed-Methods Research Designs.
Two types that are generally considered:
(a) concurrent design, (b) sequential design
Mixed-Methods Research design.
Quantitative & qualitative data collected AT THE SAME TIME
concurrent design
Mixed-Methods Research design.
Either quantitative or qualitative data are collected first, but NOT AT THE SAME TIME; Exploratory/Explanatory design.
sequential design
Mixed-Methods Research Sequential Design.
When researchers introduce a study with quantitative research strategies (before qualitative)
explanatory design
Mixed-Methods Research Sequential Design.
When researchers employ qualitative research strategies first (before quantitative)
exploratory design
Key Category of Research Design (4 total).
Measures how either receiving treatment or not receiving treatment affects a single subject (client) or a group of subjects (clients) who can be treated as a single unit.
Single-Subject Research Design (SSRD)
Three commonly used types of SSRDs:
(a) within-series designs;
(b) between-series designs; and
(c) multiple-baseline designs
Commonly used type of SSRD.
Examines the effectiveness of one intervention or program.
Examples: (a) Single phase changes; (b) A-B-C design; (c) changing criterion design; and (d) parametric design
Within-series design
Example of within-series design.
A-B designs; similar to a post-test only design
Single phase changes
Example of within-series design.
Examining the interaction among treatment components
A-B-C designs
Example of within-series design.
Becomes more restrictive to assess how much incentive is needed to achieve maximum level performance, or the desired target behavior
Changing-Criterion designs
Example of within-series design.
Treatments are assessed across phases
Parametric designs
Commonly used type of SSRD.
Compare the effectiveness of two or more interventions for a single variable
between-series designs
Commonly used type of SSRD.
Assess data for a particular target behavior across various situations or individuals.
multiple-baseline designs
More specific forms of research that may be classified as one or more of the broad categories of research
(Qualitative, Quantitative, Mixed-method, Single-subject)
- Descriptive Research: Longitudinal Design
- Descriptive Research: Cross-sectional Design
- Survey Research
- Action Research
Nonexperimental design (most prevalent category). Used to DESCRIBE a phenomenon. NO intervention/treatment. ONLY present what is & how often something occurs; CANNOT capture reason situation occurs. Often conducted as either a precursor to or in conjunction w/other research methods. Types: simple descriptive designs, cross-sectional designs, & longitudinal designs.
Descriptive Research
Nonexperimental descriptive research design.
Thoroughly describes a variable at ONE time
(i.e., “1-shot” surveys of a variable)
simple descriptive design
Key Category of Research Design (4 total).
Conducted in an effort to improve practice or organizational efficiency; tests new approaches, theories, or ideas and reflect on one’s own teachings to enhance effectiveness.
Action Research
Nonexperimental descriptive research design.
Thoroughly describes a variable over time.
longitudinal design
Nonexperimental descriptive research: longitudinal design.
Involves assessing the SAME POPULATION over time.
Cohort study
Nonexperimental descriptive research: longitudinal design.
A study that looks at the SAME INDIVIDUALS over time.
Panel study
Descriptive research design.
Method of collecting quantitative and qualitative data. Researchers select a sample of participants and administer a series of questions to them.
Survey research
Nonexperimental descriptive research: longitudinal design.
Involves assessing the GENERAL POPULATION over time, with NEW INDIVIDUALS sampled each time data are collected.
Trend study
Nonexperimental research design.
Allows the researcher to investigate group differences for a particular variable in order to determine if there is a diff bw the groups.
Comparative design
Nonexperimental research design. Involves looking at potential causes of a DV after the data have been collected. Specifically, examines how an IV affects a DV by assessing whether 1 or more pre-existing conditions possibly caused diff’s in groups. Unfortunately, IV or conditions can’t be manipulated; data have already been collected
Ex Post Facto Research Designs aka Causal-Comparative
Descriptive research design. Examines different groups or cohorts at a particular point in time, with differences in experiences being compared.
EX: Studying diff age groups of kids & how watching tv affects their performance scores on tests at a certain time of day. Limitations: Comparisons can only be inferred, since same kids are not studied
Cross-sectional research
SSRD’s series designs
A =
B =
C =
A = means baseline data (w/o tx) B = treatment C = additional treatment
The validity of qualitative findings- that is, why others should believe your results and findings.
Trustworthiness
The four components of trustworthiness
- Credibility
- Transferability
- Dependability
- Confirmability
Component of trustworthiness; “believability” of your findings
credibility
Component of trustworthiness; degree to which data transfer to other contexts and participants
transferability
Component of trustworthiness; degree of consistency of results over time and across researchers
dependability
Component of trustworthiness; reflects the fact that interpretation of the data is a genuine reflection of participants’ views
confirmability
Key Terms of Program Evaluation. From a program evaluation perspective, a process of providing feedback about a program to its stakeholders.
Accountability
Key Terms of Program Evaluation. Any individuals involved in or affected by the program. These are individuals to whom the program evaluator is accountable.
Stakeholders
Key Terms of Program Evaluation. Used for developing program objectives and includes A 5 audience (individuals influenced by the program objective), B 5 behavior (expected action or attitude), C 5 conditions (context or mode in which behavior will occur), and D 5 description (concrete performance criterion).
A-B-C-D model
A-B-C-D model. Used for developing. Includes:
A- individuals influenced by the program objective
B- expected action or attitude
C- context or mode in which behavior will occur
D- concrete performance criterion
A: audience
B: behavior
C: conditions
D: description
Key Terms of Program Evaluation. Report developed by program evaluators for advisory committee after needs assessment complete. Contains background info on needs asses., info on data sources & analyses used, & recommendations on findings for program implementation and future evaluation.
Executive summary
Used in program evaluation to weigh the benefits of a particular course of action (i.e., maintaining a program; terminating a program) against the costs.
Efficiency Analysis (Cost-Benefit Analysis)
Used in program development and evaluation. Typically, composed of representatives from various stakeholder groups and varies widely in form and function.
Advisory Committee
Smaller than a full-scale study, designed to assess the feasibility of expanding a small study to a much larger scale.
Pilot study
Types of non-experimental designs
Ex-post facto
Descriptive
Comparative
Correlational
Internal threats to validity.
When the person taking a test gets bored
Maturation
Internal threat to validity.
extraneous variable within or outside of the test that effects the outcome.
History
