Research & Program Evaluation Flashcards
Compares two means for one variable; provide a ?-ratio
t-tests; t-ratio
Involve comparing two independent groups (usually assigned randomly) on one dependent variable.
EX: Gender and achievement
Independent t-tests
Involve similar groups paired or matched in some meaningful way or the same group tested twice.
EX: Same group of high school students; pre-test and post-test
Dependent t-tests
A statistical test that involves having at least one independent variable in a study with three or more groups or levels.
EX: Household income (Below $20,000; $20K-$40K; $40K-$60K; and $60,000+) divided into 4 levels/groups
Analysis of Variance (ANOVA)
ANOVA
Indicates 2+ more of the group means are statistically different
F ratio
Used with 1+ independent variable; yields both main effects and interaction effects (i.e., significant differences among groups across or more independent variables).
EX: 2 treatments (Cognitive therapy and Interpersonal therapy) compared for effectiveness on males and females
Factorial ANOVA
Statistical test that includes an IV as a covariate, or a variable that needs to be statistically adjusted & controlled in order to look at the relationship of other IVs & the DV. EX: Examining the relationship bw household income & work satisfaction w/gender as a covariate (statistical effects of gender are removed from the analysis to control for any effects that gender may have on work satisfaction)
ANCOVA
Allows examination of every possible pairing of group means for a particular IV after one has concluded that there are main effects (i.e., significant difference among two or more groups comprising a single IV) in an ANOVA.
Post hoc analysis
Chi-Square test, Mann-Whitney U test, Friedman’s rank test, Wilcoxon’s signed-rank test, Kolmogorov-Smirnov Z procedure, Kruskal-Wallis test
Types of nonparametric statistics tests
Used when researchers are only able to make a few assumptions about distribution of scores in the underlying pop. Specifically, their use is suggested when nominal or ordinal data are involved or when interval or ratio data are not distributed normally (i.e., are skewed).
Non-parametric statistics test
A nonparametric statistical test used to determine whether two or more categorical or nominal variables are statistically independent.
EX: investigating rxp of whether to terminate counseling (yes/no) and the gender of the counselor (male/female)
Chi-square test
Nonparametric statistical test. Analogous to a parametric independent t-test, except uses ordinal data instead of interval or ratio data. Compares the ranks from two groups.
EX: Compare students Grades 9-12 w/education aspiration (diploma, 2/4-yr degree, grad) as DV
Mann-Whitney U-test
Nonparametric statistical test. Similar to Mann-Whitney U test, but used with samples smaller than 25 participants
Kolmogorov-Smirnov Z Procedure
Nonparametric statistical test analogous to an ANOVA. Extension of Mann-Whitney test. Used with 3 or more groups per independent variable & an ordinal-scaled dependent variable.
Kruskal-Wallis test
Nonparametric statistical test. Equivalent to a dependent t-test. Involves ranking the amount & direction of change for ea pair of scores.
EX: Comparing perceived level of competence before and after a training program
Wilcoxon’s signed-ranks test
A nonparametric statistical test similar to Wilcoxon’s signed-ranks test in that it is designed for repeated measures. It may be used with more than two comparison groups.
Friedman’s rank test
Graphical Representations of Experimental Designs:
- ___ = A, B, C, D
- ___ = (R)
- ___ = O
- ___ = X, Y, Z
- ___ = n/a
- Groups
- Random Assignment
- Observation
- Intervention
- control group (no intervention or observation)
A type of experimental design used when it is impossible or inappropriate to randomly assign participants to groups. Often used with nested data (e.g., classrooms, counseling groups) or naturally occurring groups (e.g., males, African Americans, adolescents). Two common types:
(a) nonequivalent groups pretest-posttest control (sometimes called comparison group designs), and
(b) time series design.
Quasi-experimental designs
Common type of quasi-experimental design.
Observations made at equal time intervals with same testing procedures. Characterized by repeatedly measuring before and after an intervention for one group, & uses a control group for comparison.
(Graphical Representation)
control group interrupted time series design
A: 0 -> 0 -> 0 -> X -> O -> O -> O
B: 0 -> 0 -> 0 -> n/a -> O -> O -> O
Common type of quasi-experimental design.
Counselor keeps groups intact; administers a pretest; administers treatment to only one group
(Graphical Representation)
nonequivalent groups pretest-posttest control group design
A: O -> X -> O
B: O -> n/a -> O
Common type of quasi-experimental design. Observations made at equal time intervals with same testing procedures. Characterized by REPEATEDLY measuring before & after intervention; can measure one group, or include control group for comparison.
time series design
Common type of quasi-experimental design.
Counselor keeps groups intact; administers pretest, administers treatment to 2+ groups, and then gives the groups a posttest.
(Graphical Representation)
nonequivalent groups pretest-posttest comparison group design
A: O -> X -> O
B: O -> Y -> O
C: O -> Z -> O
Common type of quasi-experimental design.
Characterized by repeatedly measuring before and after an intervention for one group.
(Graphical Representation)
one-group interrupted time series design
A: 0 -> 0 -> 0 -> X -> O -> O -> O
Aka randomized experimental designs. The gold standard; involve 2+ groups for comparison & random assignment. Common types
(a) randomized pretest-posttest control group design
(b) randomized pretest-posttest comparison group design,
(c) randomized posttest-only control group design,
(d) randomized posttest-only comparison group design, and
(f) Solomon four-group design
true experimental designs
Common type of true experimental design.
Participants are assigned to one of 2+ groups, each group receives a distinct intervention, and the effectiveness of interventions is compared through the use of pre- and posttests
(Graphical Representation)
randomized pretest-posttest comparison group design
(R) A: O -> X -> O
(R) B: O -> Y -> O
(R) C: O -> Z -> O
Common type of true experimental design.
Uses 4 randomly assigned groups so the presence of a pretest and the presence of an intervention can be assessed more rigorously.
(Graphical Representation)
Solomon four-group design (R) A: O -> X -> O (R) B: O -> n/a-> O (R) C: n/a-> X -> O (R) D: n/a->n/a->O
Common type of true experimental design.
Involves the random assignment of participants to a tx or control group, administering an intervention to one group, and then measuring the outcome.
(Graphical Representation)
randomized posttest-only control group design
(R) A: X -> O
(R) B: n/a -> O
Common type of true experimental design.
Participants are assigned to 2 groups (one group serves as the control); both groups are measured before & after an intervention.
(Graphical Representation)
randomized pretest-posttest control group design
(R) A: O -> X -> O
(R) B: O -> n/a -> O
Common type of true experimental design.
Similar to a randomized posttest-only control group design, but with 2+ groups for comparison and no control group
(Graphical Representation)
randomized posttest-only comparison group design
(R) A: X -> O
(R) B: Y -> O
A type of experimental design that does not use random assignment, thus failing to control for internal validity threats. Three common types:
(a) one-group posttest only design,
(b) one-group pretest-posttest design, and
(c) nonequivalent groups posttest-only design.
pre-experimental designs
Common type of pre-experimental design.
No attempt to begin study w/equivalent groups; one group receives intervention; change is measured; other group serves as control w/o intervention & assessed at same time as the other group.
(Graphical Representation)
nonequivalent groups posttest-only design
A: X -> O
B: n/a -> O
Common type of pre-experimental design.
A group receives an intervention and change is measured.
(Graphical Representation)
one-group posttest only design
A: X -> O
Common type of pre-experimental design.
A group is evaluated before and after an intervention
(Graphical Representation)
one-group pretest-posttest design
A: O -> X -> O
A type of quantitative sampling that involves sampling a known population using randomization. Methods include (a) simple random sampling; (b) systematic sampling; (c) stratified random sampling; and (d) cluster sampling.
Probability sampling
A method of probability sampling.
Every nth element is chosen
systematic sampling
A method of probability sampling.
Every member of population has equal chance of being selected
simple random sampling
A method of probability sampling.
Population is divided into subgroups and the professional counselor draws randomly from the subgroups
stratified random sampling
A quantitative sampling method that typically involves accessible, convenient samples and does not use randomization. Methods include convenience, purposeful, and quota.
Nonprobability sampling
A method of Non-probability sampling.
The selection of a sample from a population based on who will be most informative about a topic of interest; participants are selected because they represent needed characteristics
purposeful sampling
A method of probability sampling.
The professional counselor identifies existing subgroups and not individual participants. Sometimes involve multiple stages, such as a two-stage random sample (e.g., randomly select 60 schools and then 20 classes from those schools), a three-stage random sample (e.g., randomly selecting 200 school districts, then 20 schools from each district, and then 10 classes per school), and so forth.
cluster sampling
A method of Non-probability sampling.
Drawing the needed number of participants with the needed characteristic (e.g., gender, race) from the convenience sample.
quota sampling
A method of Non-probability sampling.
The selection of an easily accessible population that most likely does not fully represent the population of interest
convenience sampling
The degree to which changes in the dependent variable are due to the effects of the independent variable. Threats include history, selection, statistical regression, testing, instrumentation, attrition, maturation, diffusion of treatment, experimenter effects, and subject effects.
Internal validity
Scores of participants who were selected because of their extreme score on a dependent variable are affected; regress towards the mean
Statistical regression
The phenomenon of research participants knowing what to expect and learning something from a pretest that helps to improve their performance on future tests.
Practice Effects (memory effects)
The degree of peakedness of a distribution. Distributions can be mesokurtic, leptokurtic, and platykurtic.
Kurtosis
Normal curve distribution
Mesokurtic
Distribution curve that is “flat and wide”
platykurtic
Distribution curve that is “tall and thin”
leptokurtic