quasi experimental designs Flashcards
empiricism and determinism
- Empiricism is the process of learning things through direct observation or experience and reflection on those experiences. this grounds what it means to ask an empirical question
- Determinism is the assumption that all events have causes.
Identifying causality involves covariation, temporal order, and control of other factors.
lots of complex steps most of which we will Never be able to address all at the same time but we need to be aware of them and try to get at them in different ways across different studies. - different issues that influence our ability to identify causality include covariation between variables.
Two disciplines?
In 1957 APA President Lee Cronbach described psychology as consisting of two disciplines.
Experimental research (manipulated variables)
Correlational research (subject variables)
Manipulated variables
- Experimental research always involves a manipulated variable
- Determined by the research question and design choices
Also called experimental factor or independent variable
example of manipulated variables
does everything get reduced to exactly the same thing or are there different types of mental representation?
koger shepherd came up with a measure he used to evaluate whether when you use some kind of visual spatial info in your environment it has visual spatial properties in the way that your brain uses it
mental rotation test - ptps asked are these two figures the same except for their orientation?
important variation was on the yes trials sometimes the degraded of rotation in terms of the difference between the figures was very small and sometimes it was very large.
rogers proposla was that if its true that the brain is using some kind of visual spatial coding them we could get some analogue result to the degrees of rotation that we could measure in terms of peoples reaction times.
the more degrees of rotations difference between figures the longer it took people to say yes. if orientations close it was a short time. allowed rogers to draw conclusions on what’s happening in the brain.
used to evaluate hypothesis about gender differences in cog abilities and researchers were interested in identifying potential differences across genders in spatial cognition.
Subject variables
when we are not interested in them we use random assignment
- Correlational research focuses on subject variables that vary across individuals and situations.
- Attributes that pre-exist the study or attributes that occur naturally during the study.
Subject variables can be studied with a range of methods.
Subject variables and sampling
- Because subject variables are not manipulated, in non-experimental research participants are selected or grouped on the basis of individual characteristics.
- In other words, individual differences are especially important in non-experimental research.
Whenever individual differences are important, we must pay special attention to sampling.
Quasi-experimental designs
- Like experimental designs, quasi-experimental designs contain a manipulated variable (IV) and a DV.
- Like correlational research, quasi-experimental designs also contain a subject variable or quasi-independent variable.
- Participants cannot be randomly assigned to a quasi-independent variable.
Studies of quasi-independent variables test differences in distributions between groups (x) on some other variable (y).
title et al 2008
was interested in if men and women had different reaction times in the rotation test.
conclusion = theres a difference in spatial cognition between men and women
In some cases, a third variable can help to clarify the relation between the quasi-experimental and manipulated variable.
Computer game practice improves mental rotation performance, and the effect is stronger for women.
Both groups improved as a result of training
the difference observed at pre-test disappeared
= third varaibe can be critical to interpreting differences that we observe in quasi-experimental designs
quasi experimental design
- The lack of random assignment in quasi-experimental designs means we need to be more cautious about causal inferences.
- In true experimental designs, assuming no confounds, we can infer that IV causes DV.
- In quasi-experimental designs, groups may differ in several ways, so IV cannot be said to cause DV.
- Quasi-experiments require the same processes of critical thinking required by randomized experiments
- Choosing independent & dependent variables wisely
- Identifying useful populations & settings to study
- Ensuring assumptions of statistical tests are met
- Thinking about validity & generalisation
Quasi-experiments require an extra task – critical thinking about confounds & other problems that might result from the lack of random assignment
Correlational designs
- Correlational designs involve two or more variables that you cannot manipulate experimentally.
- A correlation is also a statistical technique used to determine the degree to which two variables are related.
Not all correlational research designs reports correlations in their statistical tests. So the test is not the identifier of the design.
Correlation and causation
To accurately interpret the results of correlational research, we need to consider two problems.
Direction of causation problem: a correlation does not indicate which variable is the cause and which is the effect.
Third variable problem: the correlation between two variables may be the result of some third, unspecified variable.
why are correlational designs of interest?
have higehr external validity as often measure something that is quite consistent overtime
higher reliability - easier to observe same result over and over across different samples in a correlational design than an experimental design
Scatterplots
- Scatterplots graph data from two variables
- The predictor variable is usually plotted on the X-axis, and the outcome or criterion variable on Y-axis
Scatterplots help us recognise relations between variables
correlation tests are good at detecting linear relations but not good at detectimg other kinds of relations
Regression and prediction
Regression is a statistical process for predicting individual scores AND estimating the accuracy of those predictions
Regression allows you to use a predictor variable (X) to predict a criterion variable (Y)
Regression line – straight line on a scatterplot that best summarizes a correlation
on the basis of a regression line we can make some sort of prediction about what would happen if we would observe beyond the observed data.
personality development
Relationships between age and three personality trait scores of the dogs.
a Relationship on the full sample, the four outlier aged dogs are marked with red dots. b Relationship after excluding the four outlier aged dogs.
turcsan et al 2020
gender and degree are not true IV or you can’t assign someone to gender or degree so they are quasi independent or quasi experimental variables. if they bought it into the room its a subject variable.
Two disciplines?
Psychology can draw on the two disciplines of experimental & observational research to address a broader range of questions & at the same time maximise the validity & reliability of our research.
summary
- Goals of research in psychology
The goals of psychological research are broad.
To address those goals fully, we must consider both experimental and non-experimental research designs. - Quasi-experimental designs
Quasi-experimental designs involve groups based on a pre-existing variable.
The lack of random assignment & potential confounds in quasi-experimental designs challenge causal inferences. - Correlational research
Correlational research allows us to examine hypotheses about relations between variables.
Correlational research also challenges causal inferences.
analysising data from non experimental methods - correlation - describing relationships
A correlation exists whenever two variables are associated or related. This idea is implied by the term itself: co for two and relation for, well, relation. Correlations can occur for data of all different types of scales of measurement, but we will focus here on interval and ratio data. In a positive correlation, the relationship is such that a high score on one variable is associated with a high score on the second variable; similarly, a low score on one relates to a low score on the other. A negative correlation, on the other hand, is an inverse relationship. High scores on one variable are associated with low scores on the second variable, and vice versa.
scatterplots
An indication of the strength of a relationship between two variables can be discerned by examining a scatterplot, which is a visual representation of the relationship between the two measured variables. Generally speaking, the stronger the relationship between the two variables, the closer the points on the scatterplot will be a straight line. If there is more variability in the scores for the two variables, then the points on the scatterplot will be more spread out, that is, more scattered. In general, as any correlation weakens, the points on a scatterplot move farther away from the diagonal lines that would connect the points in a perfect correlation.
Some relationships are not linear, however, and applying statistical procedures that assume linearity will fail to identify the true nature of the relationship.
correlation coefficients
The strength and direction of a correlation is indicated by the size of a statistic called the coefficient of correlation. The most common coefficient is the Pearson’s r, named for Karl Pearson, the British statistician who rivals Sir Ronald Fisher (the ANOVA guy) in stature.6 Pearson’s r is calculated for data measured on either an interval or a ratio scale of measurement. Other kinds of correlations can be calculated for data measured on other scales. For instance, a correlation coefficient called Spearman’s rho (reads “row”) is calculated for ordinal (i.e., rankings) data and a chi‐square test of independence (also invented by Pearson) or the phi coefficient works for nominal data.
The correlation coefficient itself ranges from −1.00 for a perfect negative correlation, through 0.00 for no relationship, to +1.00 for a perfect positive correlation. The digit represents the strength of the relationship between two variables: the closer the coefficient is to 1 or −1, the stronger the relationship. The sign of the coefficient represents the direction of the relationship, either positive or negative.
Another way to interpret the correlation coefficient is in a form of effect size of the strength of the relationship between two variables. Psychologists often use Cohen’s (1988) conventions of .10 for a small effect size, .30 for a medium effect size, and .50 for a large effect size. So, if one obtains a Pearson’s r of .23, one may interpret the correlation as having a small‐to‐medium‐sized relationship between the two variables.
coefficient of determination
A better interpretation of a correlation is to use what is called the coefficient of determination (r2 ). It is found by squaring the Pearson’s r—hence, the coefficient will always be a positive number, regardless of whether the correlation is positive or negative. Technically, r2 is defined as the percent of variance in one variable that is explained by the other variable. Another way to think of this is how much variability is shared across both variables, a concept called shared variance.
Notice, for example, that for a correlation of +.70, the coefficient of determination is .49, while a correlation of +.50 has an r2 of .25. We might be tempted to think the relationships are both “strong” correlations, according to Cohen’s (1988) conventions. However, the reality is the amount of shared variance is almost twice as much in the first case as in the second. That is, a correlation of +.70 is much stronger than a correlation of +.50.
outliers
outlier is a score that is dramatically different from the remaining scores in a data set. with correlational research an outlier can seriously distort the calculated value of Pearsons r and the coefficient of determination (r^2). the best way to spot an outlier in an correlational study is to look at the scatterplot.
regression - making predictions
in non- experimental designs researchers can use regression techniques to predict behaviours based on the correlations between variables. making predictions on the basis of correlations is referred to as doing a regression analysis. If you know a statistically significant correlation exist between two variables, the knowing the score on one of the variables and they was due to predict a score on the other. A regression line is used for making the predictions and is also called the line of best fit. It provides the best possible way of summarising the points on the S scatterplot. Precisely if you look at the absolute values of the shortest distances between each part in the line low distances would be at a minimum. In regression analysis a regression equation is used to predict a value for y based on a given value of X. why is sometimes referred to as the criterion variable and x as a predictor variable. in Order to predict with confidence however the correlation must be seen significantly greater than zero. The higher the correlation the closer the points on the scatter plot will be to the regression line and the more confident you can be in your prediction. and that confidence can be expressed mathematically in the form of a confidence interval as a way of determining a range of scores within which the true mean of a population is likely to be found. When making a prediction in aggression analysis it is possible to establish a range of scores for the prediction within which the true prediction is likely to occur a higher percentage of times. In general as the correlation gets stronger can be more confident of the prediction. This will be reflected in a narrower range of scores when the confidence interval is calculated.
The actual regression analysis will yield standardized estimates of the strength of the predictor variable’s ability to predict changes in the outcome or criterion variable; this is estimate is usually a beta coefficient, represented as β. Technically, beta is the slope of the regression line, as represented in the formula for creating a straight line on a graph with X and Y coordinates, where X would be the predictor variable and Y would be the criterion variable:
Y = a + bX
Beta can be interpreted in a similar fashion as a correlation coefficient, but a regression analysis also yields information about how strong those predictors are. Statistical tests of whether the predictor variable is a statistically significant predictor are calculated in a regression analysis and may be reported as F‐ or t‐tests
we have described what is known as a bivariate approach to data analysis, which investigates the relationships between any two variables. A multivariate approach, on the other hand, examines the relationships among more than two variables (often many more than two). In the case of simple, linear regression, two variables are involved: the predictor variable and the outcome variable.
Multiple regression solves the problem of having more than one predictor of some outcome. A multiple regression analysis has one criterion variable and a minimum of two predictor variables. The analysis enables you to determine not just that these two or more variables combine to predict some criterion but also how they uniquely predict some criterion variable. Multiple regression allows the researcher to estimate the relative strengths of the predictors. These strengths are reflected in the multiple regression formula for raw scores, which is an extension of the formula for simple regression:
Y = a + b1X1 + b2X2 + …. + bnXn
where each X is a different predictor score; Y is the criterion, or the score being predicted; and the size of the b’s are the beta coefficients that reflect the relative importance of each predictor— they are also known as beta weights in multiple regression (Licht, 1995). A multiple regression analysis also yields a multiple correlation coefficient (R) and a multiple coefficient of determination (R2 R is a correlation between the combined predictors and the criterion, and R2 ). provides an index of the variation in the criterion variable that can be accounted for by the combined predictors. Note the use of upper case letters to differentiate the multivariate R and R2 r and r2 correlation, and both R2 tell you about the amount of shared, explained variation. from the bivariate Pearson’s . Their interpretations are similar, however. Both R and r tell you about the strength of a and r2 The advantage of a multiple regression analysis is that when the influences of several predictor variables are combined (especially if the predictors are not highly correlated with each other), prediction improves compared to the single regression case.
interpreting correlational results - directionality
If there is a correlation between two variables, A and B, it is possible that A is causing B to occur (A → B), but it also could be that B is causing A to occur (B → A). That the causal relation could occur in either direction is known as the directionality problem. The existence of the correlation by itself does not allow one to decide about the direction of causality.
Choosing the correct causal direction is not possible based on an existing correlation. However, the directionality problem can be addressed to some extent. The approach derives from the criteria for determining causality. research psychologists are generally satisfied with attributing causality between A and B when they occur together with some regularity, when A precedes B in time, when A causing B makes sense in relation to some theory, and when other explanations for their co‐occurrence can be ruled out. using a procedure called a cross‐lagged panel correlation, it is possible to increase one’s confidence about directionality. In essence, this procedure investigates correlations between variables at several points in time. Hence, it is a type of longitudinal design, adding the causal element of A preceding B.
third variables
correlational research may not attempt to control extraneous variables directly, these variables often provide an explanation for the correlation found—that is, rather than A causing B or B causing A, an unknown third variable, C, might be causing both A and B to happen. C is an uncontrolled third variable (or variables—it is often the case that more than one uncontrolled variable lies behind a correlation).
there is a possibility that both A and B result from a third variable C (C -> A and B)
sometimes trying to identify third variables is a purely speculative affair. on other occasions however one might have reason to suspect a particular third variable is operating. if so and if it is possible to measure this third variable its effects can be evaluated using a procedure called partial correlation which attempts to control for the third variable statistically.
there are two types of third variable that may help explain a correlation - a mediator and a moderator. Each is a company by advanced statistical analysis to test to see whether the variable is a factor in a correlation between two variables. a mediating variable is one that explains how or why a relationship exists between two variables. A moderating variable is one that explains under what conditions does the relationship between two variables exist. This can include for what types of people or when does the correlation exist. These factors are not experimentally manipulated.
combining non-experimental and experimental methods
Another common strategy for increasing confidence in causality is to do a correlation study use it to create causal hypotheses and then follow the correlation study with experimental studies.
beyond the lab
there is a close connection between basic and applied research, as illustrated by growing field of translational research. In Chapter 1, we defined translational research as research that is done for both better understanding of a particular phenomenon as well as for its application to promote physical and psychological well‐being. While basic research may serve as the “engine of discovery,” driving innovation and deeper understanding of human functioning, it is also important that basic research results apply to situations that enable users of research to inform their practice. Further, to best inform therapeutic interventions, basic research findings need to be translated and tested in clinical situations. The National Institutes of Health (NIH) has recognized this need and has made translational research a priority (Woolf, 2008). Broadly speaking, translational research has been called “bench‐to‐bedside” approaches for translating basic research into interventions and treatments for individuals. In psychology, it has been considered a type of research that can help bridge the science‐practice gap (Tashiro & Mortensen, 2006). Virtually, all applied research has the dual function of addressing applied problems directly and providing evidence of basic psychological phenomena that influence theory development. Furthermore, applied research often is rooted in theories and research findings derived from basic research.
applied psychology in historical context
from the time psychology emerged as a new discipline in the late 19th century, psychologists in the US have been interested in applied research and in applying the results of their bias research. institutional pressures in the early 20th century forced psychologists to show how their work could improve society. in order to get funding psychologists had to show the ideas deriving from their research could be put to good use. Psychologists trained as researchers focused on extending knowledge, but they often found themselves trying to apply basic research methods to solve problems in areas such as education, mental health, child rearing, and, in the case of Walter Miles, sports.
Psychologists at the beginning of the 21st century are as interested in application as were their predecessors at the beginning of the 20th century. That is, they design and carry out studies to help create solutions to real‐world problems while at the same time contributing to the basic core knowledge of psychology. However, applied research projects encounter several difficulties not usually found in the laboratory.
design problems in applied research
- Ethical dilemmas (Chapter 2). A study conducted outside of the laboratory may create problems relating to informed consent and privacy. Also, proper debriefing is not always possible. Research done in an industrial or corporate setting may include an element of perceived coercion if employees believe their job status depends on whether they volunteer to participate in a study (see Box 11.3 at the end of this chapter for more on ethics and applied research). * A trade‐off between internal and external validity (Chapter 5). Because research in applied psychology often takes place in the field, the researcher can lose methodological control over the variables operating in the study. Hence, the danger of possible confounding can reduce the study’s internal validity. On the other hand, external (and specifically, ecological) validity is usually high in applied research because the setting more closely resembles real‐life situations, and the problems addressed by applied research are everyday problems. * Problems unique to between‐subjects designs (Chapter 6). In applied research, it is often impossible to use random assignment to form equivalent groups. Therefore, the studies often use ex post facto designs and must therefore compare nonequivalent groups. This, of course, introduces the possibility of reducing internal validity by subject selection problems or interactions between selection and other threats such as maturation or history. When matching is used to achieve a degree of equivalence among groups of subjects, regression problems can occur, as will be elaborated in a few pages. * Problems unique to within‐subjects designs (Chapter 6). It is not always possible to counterbalance properly in applied studies using within‐subjects factors. Hence, the studies may have uncontrolled order effects. Also, attrition can be a problem for studies that extend over a long period of time.
quasi - experimental designs
Strictly speaking, and with Woodworth’s (1938) definitions in mind, so‐called true experimental studies include manipulated independent variables and equivalent groups formed by either straight random assignment or matching followed by random assignment. If subjects cannot be assigned randomly, however, the design is called a quasi‐experimental design. Although it might seem that quasi‐experiments are therefore lower in status than “true” experiments, it is important to stress that quasi‐experiments have great value in applied research. They do allow for a degree of control, they serve a researcher’s goals when ethical or practical problems make random assignment impossible, and they often produce results with clear benefits for people’s lives. Thus far, we have seen several examples of designs that could be considered quasi‐experimental: * Single‐factor ex post facto designs, with two or more levels * Ex post facto factorial designs * P x E factorial designs (the P variable, anyway) * All of the correlational research
nonequivalent control group designs
In this type of study, the purpose is to evaluate the effectiveness of some treatment program. Those in the program are compared with those in a control group who aren’t treated. This design is used when random assignment is not possible, so in addition to the levels of the independent variable, the members of the control group differ in some other way(s) from those in the treatment group—that is, the groups are not equivalent at the outset of the study. You will recognize this as a specific example of ex post facto design in Chapters 7 and 8, a type of design comparing nonequivalent groups, often selected with reference to a subject variable such as age, gender, or some personality characteristic. In the case of the quasi‐experimental nonequivalent control group design, the groups are not equal at the start of the study; in addition, they experience different events in the study itself. Hence, there is a built‐in confound that can complicate the interpretation of these studies. Nonetheless, these designs effectively evaluate treatment programs when random assignment is impossible. Following the scheme first outlined by Campbell and Stanley (1963), the nonequivalent control group design can be symbolized like this: Experimental group: O1 Nonequivalent control group: O1 where O1 and O2 T O2 O2 refer to pretest and posttest observations or measures, respectively, and T refers to the treatment program being evaluated. Because the groups might differ on the pretest, the important comparison between the groups is not simply a test for differences on the posttest, but a comparison of the amounts of change from pre‐ to posttest in the two groups. Hence, the statistical comparison is typically between the change scores (the difference between O1 and O2 ) for each group. Alternatively, techniques are available that adjust posttest scores based on the pretests.
regression to the mean and matching
a special threat to the internal validity of nonequivalent control group designs control when there is an attempt to reduce the nonequivalencey of the groups through a form of matching. matching is an alternative to random assignment under certain circumstances, and it works rather well to create equivalent groups if the independent variable is a manipulated variable and participants can be randomly assigned to groups after being paired on some matching variable (see Chapter 6 to review the matching procedure). However, it can be a problem in nonequivalent control group designs when the two groups are sampled from populations that differ on the factor being used as the matching variable. If this occurs, then using a matching procedure can enhance the influence of the regression to the mean problem and even make it appear that a successful program has failed.
most nonequivalnet control group designs use pretest-posttest designs but not all use pretests.
interrupted time series designs
if you take measures for an extended period before and after an event expected to influence behaviour.
Using the system in Campbell and Stanley (1963) again, the basic time series study can be symbolized like this:
O1O2O3O4O5TO6O7O8O9O10
where all of the O’s represent measured observations of behavior taken before and after T, which is the point at which some treatment program is introduced or some event (e.g., an earthquake) occurs. T is the interruption in the interrupted time series. Of course, the number of measures taken before and after T will vary from study to study and are not limited to five each. It is also not necessary that the number of pre‐interruption and post‐interruption points be the same. As a general rule, the more data points, the better, and some experts (e.g., Orwin, 1997) recommend at least 50 pre‐interruption data points.
Outcomes
The main advantage of a time series design is that it allows the researcher to evaluate trends, which are relatively consistent patterns of events that occur with the passing of time.
variations on the basic time series design
sometimes, the conclusions from an interrupted time series design can be strengthened if some type of control comparison is made. One approach amounts to combining the best features of the nonequivalent control group design (a control group) and the interrupted time series design (long‐term trend analysis). The design looks like this
O1O2O3O4O5. O6O7O8O9O10
O1O2O3O4O5. T O6O7O8O9O10
A second strategy for strengthening conclusions from a time series study is when a program can be introduced in different locations at different times, a design labeled an interrupted time series with switching replications by Cook and Campbell (1979), and operating like this:
O1O2O3TO4O5O6O7O8O9O10
O1O2O3O4O5O6O7TO8O9O10
With this procedure, the same treatment or program is put into place in two locations at two points in time. There is no control group, but the design provides the benefit of a built‐in replication. If the outcome pattern in Location 2 matches that of Location 1, the researchers can be more confident about the generality of the phenomenon being studied.
A third elaboration on an interrupted time series design, again in the absence of a control group, is to measure several dependent variables, some expected to be influenced by the interruption, others not expected to change.
Program evaluation
Applied research that attempts to assess the effectiveness and value of public policy (e.g., California’s three strikes law) or specially designed programs (e.g., Meals on Wheels) is sometimes given the name program evaluation. This research concept developed in the 1960s in response to the need to evaluate social programs like Head Start, but it is concerned with much more than answering the question “Did program X work?” More generally, program evaluation includes (a) procedures for determining if a need exists for a particular program and who would benefit if the program is implemented; (b) assessments of whether a program is being run according to plan and, if not, what changes can be made to facilitate its operation; (c) methods for evaluating program outcomes; and (d) cost analyses to determine if program benefits justify the funds expended.
planning for programs - needs analysis
a needs analysis is a set of procedures for predicting whether a population of sufficient size exists that would benefit from the proposed program whether the program could solve a clearly defined problem and whether members of the population would actually use the program. several methods exist for estimating need, and it is important to rely on at least some of these techniques because it is easy to overestimate need. One reason for caution follows from the availability heuristic, first introduced in Chapter 1’s discussion about ways of knowing. Events that grab headlines catch our attention and become more “available” to our memory. Because they come so readily to mind, we tend to overestimate how often they occur. All it takes is one or two highly publicized cases of children being abandoned by vacationing parents for a call to be made for new programs to fix this seemingly widespread problem. Also, a need for a new program can be overestimated by those in a position to benefit (i.e., keep their jobs) from the program’s existence. As outlined by Posavac and Carey (2010), there are several ways to identify the potential need for a program. These include: * Census data. If your proposed program is aimed at the elderly, it’s fairly obvious that its success will be minimal if few seniors live in the community. Census data (www.census.gov) can provide basic demographic information about the number of people fitting into various categories. Furthermore, the information is fine‐grained enough for you to determine the number of single mothers under the age of 21, the number of people with various disabilities, the number of older adults below the poverty line, and so on. * Surveys of available resources. There’s no reason to begin a Meals on Wheels program if one already exists in the community and is functioning successfully. Thus, one obvious step in a needs analysis is to create an inventory of existing services that includes a description of who is providing the services, exactly which services are being provided, and an estimate of how many people are receiving the services. * Surveys of potential users. A third needs analysis strategy is to administer a survey within the community, either to a broadly representative sample or to a target group identified by census data. Those participating could be asked whether they believe a particular program is needed.
* Key informants, focus groups, and community forums. A key informant is someone in the community who has a great deal of experience and specialized knowledge about the problem at hand that is otherwise unavailable to the researcher (Gilchrist & Williams, 1999). Such persons include community activists, clergy, people who serve on several social service agency boards, and so on. A focus group is a small group (typically 7‐9 people) whose members respond to a set of open‐ended questions about some topic, such as the need for a particular program (they might also be used to assess a program’s progress or its outcome). Focus groups are often used as a follow‐up to a community survey, but they also can be used to shape the questions that will appear in a survey. Finally, useful information can sometimes emerge from a community forum, an open meeting at which all members of a community affected by a potential program are invited to come and participate. Key informants, focus groups, and forums can all be helpful tools, but the researcher must be careful of weighing too heavily the arguments of an especially articulate (but perhaps nonrepresentative) informant, focus group member, or speaker at a forum.
monitoring programs - formative evaluation
rather than waiting until the programs completion the process of the program is carefully monitored while its in progress - the most common form of evaluation activity according to sechrest and figured 1993
a formative evaluation can include several components - it determines if the program is being implemented as planned. it also provides data on how the program is being used this can be called program audit - the program auditor examines whether the program as described in the agency’s literature is the same as the program that is actually being implemented. a final part of formative evaluation can be a pilot study =. program implementation and some preliminary outcomes can be assessed on small scale before extending the program.
evaluating outcomes - summative evaluation
formative evaluations are less threatening than summative evaluations politically which are overall assessments of program effectiveness. formative evaluation is aimed at program improvement and is less likely to call into question the programs very existence. summative evaluation on the other hand can do just that. if the program isn’t effective why keep it and by extension why continue to pay the programs director and staff/
as sechrest and figueredo 1993 stated
summative evalaution and even the rationale for doing it call into question the very reasons for existence of the organisation involved. formative evaluation, by contrast, simply responds to the question “how can we be better?” without strongly implying the question “How do (we) know (we) are any good at all?’
despite the political difficulty summative evaluations are the core of the evaluation process and are an essential feature of any program funded by the federal government. any agency wishing to spend tax dollars to develop a program is obligated to show those dollars are being used effectively.
the actual process of performing summative evaluations involves applying some of the techniques you already know about especially quasi-experimental designs with random assignment are possible sometimes especially when evaluating a program that has more people desiring it than space available. In such a case, random assignment in the form of a lottery (random winners get the program; others wind up in a wait list control group) is not only methodologically sound, it is also the only fair procedure to use. One problem that sometimes confronts the program evaluator is how to interpret a failure to find significant differences between experimental and control groups—that is, the statistical decision is “fail to reject the null hypothesis” Such an outcome is difficult to interpret, as you recall from the discussion in Chapter 4. It could be there just isn’t any difference, yet there’s always the possibility of a Type II error being committed (an effect is real, but your study failed to find it), especially if the measuring tools are not sensitive or reliable. The program might indeed have produced some small but important effect, but the analysis failed to discover it. Although a finding of no difference can be difficult to interpret, most researchers believe that such a finding (especially if replicated) contributes important information for decision making, especially in applied research. For instance, someone advocating the continuation of a new program is obligated to show how the program is better than something already in existence. Yet, if differences between this new program and one already well established cannot be shown, then it might be wise to discontinue the new program, especially if it is more expensive to implement than the older one. A “fail to reject the null” decision also can help evaluate exaggerated claims made by advocates of a new program. A finding of no difference has important implications for decision making for reasons having to do with cost, and this brings us to the final type of program evaluation activity.
weighing costs - cost effectiveness analysis
one type of cost‐effectiveness analysis: monitoring the actual costs of a program and relating those costs to the effectiveness of the program’s outcomes. If two programs with the same goal are equally effective but the first costs half as much as the second, then it is fairly obvious that the first program should be used. A second type of cost analysis takes place during the planning stages for a program. Estimating costs at the outset helps determine whether a program is feasible and provides a basis for the later comparison of projected costs and actual costs. Estimating costs with reference to outcomes can be a complicated process, often requiring the expertise of a specialist in cost accounting. Thus, a detailed discussion of the procedures for relating costs to outcomes is beyond the scope of this chapter. In addition, it is often difficult if not impossible to put a monetary value on the benefits that might result from the implementation and continuance of a program, especially one involving wellness.
a note on qualitative data analysis
during a need analysis qunatitative data from a community survey and census data can be combined with in-depth interview information from key informants and focus groups. in formative and summative assessments quantitative data can be supplemented with a qualitative analysis of interviews with agency workers and clients and with direct observations of the program in action. in program evaluation resrach it is seldom a question of whether quantitive or qualitative research is better. although there has been and continues to be debate about the relative merits of quantitive and qualitative evaluation thoughtful program evaluators rely on both.
