Test3 Flashcards
Quasi-experimental designs
One-group posttest-only design One-group pretest-posttest design Pre-existing groups design Non-equivalent control groups posttest-only design Non-equivalent
PURPOSE OF PROGRAM EVALUATION
Demonstrate program effectiveness to funders
Improve the implementation and effectiveness of programs
Better manage limited resources
Document program accomplishments
Justify current program funding
Support the need for increased levels of funding
Satisfy ethical responsibility to clients to demonstrate positive and negative effects of program participation
Document program development and activities to help ensure successful replication
BARRIERS TO PROGRAM EVALUATION
Funding
• Time
• Technical skills
• Interferes with other, primary, activities
• Concern about the program getting a “bad grade”
Overcoming Barriers to program evaluation
- (Collaboration)is key to successful program evaluation through involvement of stakeholders (i.e., individuals who are affected by the program and its evaluation)
- (Stakeholders)include but are not limited to program staff, program clients, decision makers, and evaluators
- (Respect for one another’s roles) and equal partnership in the evaluation process overcomes
PROGRAM EVALUATION TYPES
Context Evaluation: Assessing how the program operates in a particular social, political, physical and economic environment, could include a community needs or organizational assessment
Formative Evaluation: Assessing needs that a program should fulfill, examining the early stages of a program’s development, or testing a program on a small scale before broad dissemination
Process Evaluation: Examining the implementation and operation of program components
Impact Evaluation: Investigating the magnitude of both positive and negative changes produced by a program
Outcome Evaluation: Assessing the short and long-term results of a program
Assessing how the program operates in a particular social, political, physical and economic environment, could include a community needs or organizational assessment
Context Evaluation
Assessing needs that a program should fulfill, examining the early stages of a program’s development, or testing a program on a small scale before broad dissemination
Formative Evaluation:
Examining the implementation and operation of program components
Process Evaluation:
Investigating the magnitude of both positive and negative changes produced by a program
Impact Evaluation:
Assessing the short and long-term results of a program
Outcome Evaluation:
- What is the purpose of the evaluation? What do you want to learn?
- Who are the audiences for the information from the evaluation, e.g., customers, bankers, funders, board, management, staff, customers, clients?
- What kinds of information are needed to make the decision you need to make and/or enlighten your intended audiences?
- From what sources should the information be collected?
Designing the program evaluation
Qualities of good program evaluation
(Smart) Specifics Measurable Attainable Relevant Time-bound
Well written program goals use
Use action verbs
Are focus on client or customer
CONDUCTING THE EVALUATION
Timing: Evaluation should be incorporated during the initial stages of program development
Goals: Having clear program goals (i.e., a program description) ensures that program activities and objectives are clearly defined and that the objectives can be measured – these goals and objectives are, essentially, the hypothesis that the researcher is testing
Pragmatics: Evaluation should be feasible, useful, culturally competent, ethical and accurate
Data:
• Data should be collected over time using multiple instruments that are valid and reliable
• Using both qualitative and quantitative data (i.e., a mixed method evaluation) can provide a more comprehensive picture of the program
Communicating research
LANGUAGE USE
Use first person (sparingly) and third person
Avoid “you”
Use past tense to describe completed research (others’ and your own) Use present tense for conclusions
Use future tense for future work
Use edited American English
Use unbiased language
POSTER PRESENTATIONS
Contain the same content headings as a paper
Use bullet points (not blocks of text)
Should be 48 inches wide X 36 inches high
Use white-space to make the poster visually appealing and easier to read No content within an inch of any of the sides of the poster
Use color for emphasis and visual appeal
Fonts sizes should be at least 24 point
Use san serif font for a clean look to the poster
Use images as appropriate, but only if they are directly related to the poster Avoid having the entire poster in color – that is just a waste of ink
Include references if there is space
Cite your sources
(Type fact) 2 BETWEEN SUBJECTS IVS, EACH WITH 2 LEVELS A 2(ALCOHOL) X 2(CAFFEINE)
BETWEEN SUBJECTS FACTORIAL DESIGN
(Type fact) Pretest-posttest Design
• One group of participants is tested, given the treatment, then tested a second time; the scores on the first test are compared to the scores on the second test
Repeated Measures Design
• One group of participants is tested repeatedly across several different conditions; there may or may not be a true “control” condition
Benefits of Within Subjects Designs compared to Between Subjects Designs
• Fewer participants
• Reduced error variance (because each person serves as his or her one
“control”)
• More powerful test of the effect of the IV
Within subject factorial design
(Type of fact) 1 WITHIN SUBJECTS IV AND 1 BETWEEN SUBJECTS IV A 2(PRE/POST) X 2(TREAT/CONTROL)
Mixed factorial design
(Type of factor)random placing into groups
Solomon four group design
Factorial designs have more than one IV (and each IV has two or more levels)
IVs are called “factors”
Null Outcome: The factors (IVs) have no effect on the DV
Main Effect: A factor has an effect on the DV (with two IVs there are two possible main effects, with three IVs there are three possible main effects, etc.)
Interaction: When differences on one factor depend one the levels of the second factor. There are three ways you can determine whether an interaction exists:
• When running the statistical analysis, the statistical table will report on all main effects and interactions
• Knowing that there is an interaction when you can’t talk about an effect on one factor without mentioning the other factor
• Spot an interaction in the graphs, whenever lines are not parallel, an interaction is present (but the statistics are needed to determine whether the interaction is significant)
Factorial designs
Changing the time between treatments can help to control for order effects
• If history or maturation are a concern, shorted the time between treatments
• If carryover or fatigue are a concern, lengthen the time between treatments Use the order of the treatments as an IV via counterbalancing
Controlling order of effects
Complete Counterbalancing: Every possible order of treatment is used equally often; must have enough participants for each of the groups to meet the assumptions of the statistical tests
Latin Square Counterbalancing: An incomplete counterbalancing in which each treatment occurs once in each ordinal position
Controlling order effects
Counter balancing
COSTS WITHIN SUBJECTS DESIGNS
History: Something in the environment changes, which also changes the participants
Maturation: Participants “grow up”
Practice: Participants get better after taking repeated tests
Fatigue: Participants get tired after taking repeated tests
Carryover: Earlier treatments influence how participants respond to later treatments
Attrition (Mortality): People leave the study
Regression to the mean: Because extreme scores are not likely for a normally distributed DV, if a participant provides an extreme score on an early test, later scores from that participant are more likely to be closer to the mean, just based on probability, and not due to the IV
One group of participants is tested, given the treatment, then tested a second time; the scores on the first test are compared to the scores on the second test
Pretest-posttest Design
• One group of participants is tested repeatedly across several different conditions; there may or may not be a true “control” condition
Repeated Measures Design
• Fewer participants
• Reduced error variance (because each person serves as his or her one
“control”)
• More powerful test of the effect of the IV
Benefits of Within Subjects Designs compared to Between Subjects Designs
- Hold extraneous variables constant: Control any variables that potentially affect the DV
- Use blind and double blind procedures
- Blind Procedures: Either the researcher or the participants do not know who received the treatment and who was in the control group
- Double Blind Procedures: Neither the researcher nor the participants know who received the treatment and who was in the control group, often a third party who has no contact with the participants keeps track of who is in which group
- Change a weak manipulation to a strong manipulation (e.g., instead of 60 mg of caffeine for the treatment group, given them 200 mg of caffeine)
REDUCING ERROR VARIANCE
Different groups of people get different levels of the IV (e.g., one group gets caffeine and another group gets no caffeine)
The researcher needs to ensure that the groups are equivalent at the start of the study
The researchers needs to control for confounding variables
• Confounding variables are variables that are directly associated with the different levels of the IV and that provide an alternative explanation for differences or changes observed in the DV
• Confounding variables can mask effect of the IV (Type II Error)
• Greg Martin provides a good explanation of confounding variables and
experimental designs:
• Alternatively, confounding variables can provide a false effect, making it look like there is a difference between the groups due, when there really isn’t a difference (Type I Error)
BETWEEN SUBJECTS DESIGNS
INDEPENDENT VARIABLE (IV) TYPES OF MANIPULATION
• Stimuli participants experience (e.g., perception studies)
• Context that the participants are in (e.g., competitive vs. cooperative
situations)
• Information participants receive (e.g., dating or hiring studies)
• Social setting participants are in (e.g., alone or in groups)
• Physiological conditions participants experience (e.g., stress, arousal, drugs, or deprivation of food, water, sleep, sex)
At least 1 independent variable and the IV has at least 2 levels (conditions)
Experimental designs
- Because the data are observational, care is needed to ensure that they are also objective and reliable
- Validity of the research is increased when multiple measurements are taken in each phase
- Direct interventions (rather than naturally occurring changes in the environment) are desirable
- Changes to behavior that was occurring over a long period of time are more impressive than changes to behavior that recently developed
- Generalizability increases if the same treatment is applied to several people with similar results
- The treatment procedures should be standardized
- Multiple outcome measures should be used
- When the treatment includes several components, these should be applied separately, if possible
- The treatment should have social validation (e.g., demonstrated social importance)
QUALITY ISSUES (MERTENS, 2005)
- AB (baseline, treatment)
- ABA (baseline, treatment, baseline; also called return to baseline)
- ABC (baseline, treatment 1, treatment 2, etc.)
- Multiple Baselines across Behaviors: Several behaviors need to be changed. Each behavior is subjected, individually, to the treatment to see whether the behavior changes. Typically used in educational settings.
- Multiple Baselines across Subjects: One behavior across several people needs to be changed. Each person receives the treatment, one at a time, to determine whether the treatment was effective. Typically used in educational and coaching situations.
SINGLE SUBJECT (SMALL N) DESIGNS types
