quasi experimental designs Flashcards

Question

third variables

Answer 1

correlational research may not attempt to control extraneous variables directly, these variables often provide an explanation for the correlation found—that is, rather than A causing B or B causing A, an unknown third variable, C, might be causing both A and B to happen. C is an uncontrolled third variable (or variables—it is often the case that more than one uncontrolled variable lies behind a correlation). there is a possibility that both A and B result from a third variable C (C -> A and B) sometimes trying to identify third variables is a purely speculative affair. on other occasions however one might have reason to suspect a particular third variable is operating. if so and if it is possible to measure this third variable its effects can be evaluated using a procedure called partial correlation which attempts to control for the third variable statistically. there are two types of third variable that may help explain a correlation - a mediator and a moderator. Each is a company by advanced statistical analysis to test to see whether the variable is a factor in a correlation between two variables. a mediating variable is one that explains how or why a relationship exists between two variables. A moderating variable is one that explains under what conditions does the relationship between two variables exist. This can include for what types of people or when does the correlation exist. These factors are not experimentally manipulated.

Answer 2

Another common strategy for increasing confidence in causality is to do a correlation study use it to create causal hypotheses and then follow the correlation study with experimental studies.

Answer 3

there is a close connection between basic and applied research, as illustrated by growing field of translational research. In Chapter 1, we defined translational research as research that is done for both better understanding of a particular phenomenon as well as for its application to promote physical and psychological well‐being. While basic research may serve as the “engine of discovery,” driving innovation and deeper understanding of human functioning, it is also important that basic research results apply to situations that enable users of research to inform their practice. Further, to best inform therapeutic interventions, basic research findings need to be translated and tested in clinical situations. The National Institutes of Health (NIH) has recognized this need and has made translational research a priority (Woolf, 2008). Broadly speaking, translational research has been called “bench‐to‐bedside” approaches for translating basic research into interventions and treatments for individuals. In psychology, it has been considered a type of research that can help bridge the science‐practice gap (Tashiro & Mortensen, 2006). Virtually, all applied research has the dual function of addressing applied problems directly and providing evidence of basic psychological phenomena that influence theory development. Furthermore, applied research often is rooted in theories and research findings derived from basic research.

Answer 4

from the time psychology emerged as a new discipline in the late 19th century, psychologists in the US have been interested in applied research and in applying the results of their bias research. institutional pressures in the early 20th century forced psychologists to show how their work could improve society. in order to get funding psychologists had to show the ideas deriving from their research could be put to good use. Psychologists trained as researchers focused on extending knowledge, but they often found themselves trying to apply basic research methods to solve problems in areas such as education, mental health, child rearing, and, in the case of Walter Miles, sports. Psychologists at the beginning of the 21st century are as interested in application as were their predecessors at the beginning of the 20th century. That is, they design and carry out studies to help create solutions to real‐world problems while at the same time contributing to the basic core knowledge of psychology. However, applied research projects encounter several difficulties not usually found in the laboratory.

Answer 5

* Ethical dilemmas (Chapter 2). A study conducted outside of the laboratory may create problems relating to informed consent and privacy. Also, proper debriefing is not always possible. Research done in an industrial or corporate setting may include an element of perceived coercion if employees believe their job status depends on whether they volunteer to participate in a study (see Box 11.3 at the end of this chapter for more on ethics and applied research). * A trade‐off between internal and external validity (Chapter 5). Because research in applied psychology often takes place in the field, the researcher can lose methodological control over the variables operating in the study. Hence, the danger of possible confounding can reduce the study’s internal validity. On the other hand, external (and specifically, ecological) validity is usually high in applied research because the setting more closely resembles real‐life situations, and the problems addressed by applied research are everyday problems. * Problems unique to between‐subjects designs (Chapter 6). In applied research, it is often impossible to use random assignment to form equivalent groups. Therefore, the studies often use ex post facto designs and must therefore compare nonequivalent groups. This, of course, introduces the possibility of reducing internal validity by subject selection problems or interactions between selection and other threats such as maturation or history. When matching is used to achieve a degree of equivalence among groups of subjects, regression problems can occur, as will be elaborated in a few pages. * Problems unique to within‐subjects designs (Chapter 6). It is not always possible to counterbalance properly in applied studies using within‐subjects factors. Hence, the studies may have uncontrolled order effects. Also, attrition can be a problem for studies that extend over a long period of time.

Answer 6

Strictly speaking, and with Woodworth’s (1938) definitions in mind, so‐called true experimental studies include manipulated independent variables and equivalent groups formed by either straight random assignment or matching followed by random assignment. If subjects cannot be assigned randomly, however, the design is called a quasi‐experimental design. Although it might seem that quasi‐experiments are therefore lower in status than “true” experiments, it is important to stress that quasi‐experiments have great value in applied research. They do allow for a degree of control, they serve a researcher’s goals when ethical or practical problems make random assignment impossible, and they often produce results with clear benefits for people’s lives. Thus far, we have seen several examples of designs that could be considered quasi‐experimental: * Single‐factor ex post facto designs, with two or more levels * Ex post facto factorial designs * P x E factorial designs (the P variable, anyway) * All of the correlational research

Answer 7

In this type of study, the purpose is to evaluate the effectiveness of some treatment program. Those in the program are compared with those in a control group who aren’t treated. This design is used when random assignment is not possible, so in addition to the levels of the independent variable, the members of the control group differ in some other way(s) from those in the treatment group—that is, the groups are not equivalent at the outset of the study. You will recognize this as a specific example of ex post facto design in Chapters 7 and 8, a type of design comparing nonequivalent groups, often selected with reference to a subject variable such as age, gender, or some personality characteristic. In the case of the quasi‐experimental nonequivalent control group design, the groups are not equal at the start of the study; in addition, they experience different events in the study itself. Hence, there is a built‐in confound that can complicate the interpretation of these studies. Nonetheless, these designs effectively evaluate treatment programs when random assignment is impossible. Following the scheme first outlined by Campbell and Stanley (1963), the nonequivalent control group design can be symbolized like this: Experimental group: O1 Nonequivalent control group: O1 where O1 and O2 T O2 O2 refer to pretest and posttest observations or measures, respectively, and T refers to the treatment program being evaluated. Because the groups might differ on the pretest, the important comparison between the groups is not simply a test for differences on the posttest, but a comparison of the amounts of change from pre‐ to posttest in the two groups. Hence, the statistical comparison is typically between the change scores (the difference between O1 and O2 ) for each group. Alternatively, techniques are available that adjust posttest scores based on the pretests.

Answer 8

a special threat to the internal validity of nonequivalent control group designs control when there is an attempt to reduce the nonequivalencey of the groups through a form of matching. matching is an alternative to random assignment under certain circumstances, and it works rather well to create equivalent groups if the independent variable is a manipulated variable and participants can be randomly assigned to groups after being paired on some matching variable (see Chapter 6 to review the matching procedure). However, it can be a problem in nonequivalent control group designs when the two groups are sampled from populations that differ on the factor being used as the matching variable. If this occurs, then using a matching procedure can enhance the influence of the regression to the mean problem and even make it appear that a successful program has failed. most nonequivalnet control group designs use pretest-posttest designs but not all use pretests.

Answer 9

if you take measures for an extended period before and after an event expected to influence behaviour. Using the system in Campbell and Stanley (1963) again, the basic time series study can be symbolized like this: O1O2O3O4O5TO6O7O8O9O10 where all of the O’s represent measured observations of behavior taken before and after T, which is the point at which some treatment program is introduced or some event (e.g., an earthquake) occurs. T is the interruption in the interrupted time series. Of course, the number of measures taken before and after T will vary from study to study and are not limited to five each. It is also not necessary that the number of pre‐interruption and post‐interruption points be the same. As a general rule, the more data points, the better, and some experts (e.g., Orwin, 1997) recommend at least 50 pre‐interruption data points.

Answer 10

The main advantage of a time series design is that it allows the researcher to evaluate trends, which are relatively consistent patterns of events that occur with the passing of time.

Answer 11

sometimes, the conclusions from an interrupted time series design can be strengthened if some type of control comparison is made. One approach amounts to combining the best features of the nonequivalent control group design (a control group) and the interrupted time series design (long‐term trend analysis). The design looks like this O1O2O3O4O5. O6O7O8O9O10 O1O2O3O4O5. T O6O7O8O9O10 A second strategy for strengthening conclusions from a time series study is when a program can be introduced in different locations at different times, a design labeled an interrupted time series with switching replications by Cook and Campbell (1979), and operating like this: O1O2O3TO4O5O6O7O8O9O10 O1O2O3O4O5O6O7TO8O9O10 With this procedure, the same treatment or program is put into place in two locations at two points in time. There is no control group, but the design provides the benefit of a built‐in replication. If the outcome pattern in Location 2 matches that of Location 1, the researchers can be more confident about the generality of the phenomenon being studied. A third elaboration on an interrupted time series design, again in the absence of a control group, is to measure several dependent variables, some expected to be influenced by the interruption, others not expected to change.

Answer 12

Applied research that attempts to assess the effectiveness and value of public policy (e.g., California’s three strikes law) or specially designed programs (e.g., Meals on Wheels) is sometimes given the name program evaluation. This research concept developed in the 1960s in response to the need to evaluate social programs like Head Start, but it is concerned with much more than answering the question “Did program X work?” More generally, program evaluation includes (a) procedures for determining if a need exists for a particular program and who would benefit if the program is implemented; (b) assessments of whether a program is being run according to plan and, if not, what changes can be made to facilitate its operation; (c) methods for evaluating program outcomes; and (d) cost analyses to determine if program benefits justify the funds expended.

Answer 13

a needs analysis is a set of procedures for predicting whether a population of sufficient size exists that would benefit from the proposed program whether the program could solve a clearly defined problem and whether members of the population would actually use the program. several methods exist for estimating need, and it is important to rely on at least some of these techniques because it is easy to overestimate need. One reason for caution follows from the availability heuristic, first introduced in Chapter 1’s discussion about ways of knowing. Events that grab headlines catch our attention and become more “available” to our memory. Because they come so readily to mind, we tend to overestimate how often they occur. All it takes is one or two highly publicized cases of children being abandoned by vacationing parents for a call to be made for new programs to fix this seemingly widespread problem. Also, a need for a new program can be overestimated by those in a position to benefit (i.e., keep their jobs) from the program’s existence. As outlined by Posavac and Carey (2010), there are several ways to identify the potential need for a program. These include: * Census data. If your proposed program is aimed at the elderly, it’s fairly obvious that its success will be minimal if few seniors live in the community. Census data (www.census.gov) can provide basic demographic information about the number of people fitting into various categories. Furthermore, the information is fine‐grained enough for you to determine the number of single mothers under the age of 21, the number of people with various disabilities, the number of older adults below the poverty line, and so on. * Surveys of available resources. There’s no reason to begin a Meals on Wheels program if one already exists in the community and is functioning successfully. Thus, one obvious step in a needs analysis is to create an inventory of existing services that includes a description of who is providing the services, exactly which services are being provided, and an estimate of how many people are receiving the services. * Surveys of potential users. A third needs analysis strategy is to administer a survey within the community, either to a broadly representative sample or to a target group identified by census data. Those participating could be asked whether they believe a particular program is needed. * Key informants, focus groups, and community forums. A key informant is someone in the community who has a great deal of experience and specialized knowledge about the problem at hand that is otherwise unavailable to the researcher (Gilchrist & Williams, 1999). Such persons include community activists, clergy, people who serve on several social service agency boards, and so on. A focus group is a small group (typically 7‐9 people) whose members respond to a set of open‐ended questions about some topic, such as the need for a particular program (they might also be used to assess a program’s progress or its outcome). Focus groups are often used as a follow‐up to a community survey, but they also can be used to shape the questions that will appear in a survey. Finally, useful information can sometimes emerge from a community forum, an open meeting at which all members of a community affected by a potential program are invited to come and participate. Key informants, focus groups, and forums can all be helpful tools, but the researcher must be careful of weighing too heavily the arguments of an especially articulate (but perhaps nonrepresentative) informant, focus group member, or speaker at a forum.

Answer 14

rather than waiting until the programs completion the process of the program is carefully monitored while its in progress - the most common form of evaluation activity according to sechrest and figured 1993 a formative evaluation can include several components - it determines if the program is being implemented as planned. it also provides data on how the program is being used this can be called program audit - the program auditor examines whether the program as described in the agency's literature is the same as the program that is actually being implemented. a final part of formative evaluation can be a pilot study =. program implementation and some preliminary outcomes can be assessed on small scale before extending the program.

Answer 15

formative evaluations are less threatening than summative evaluations politically which are overall assessments of program effectiveness. formative evaluation is aimed at program improvement and is less likely to call into question the programs very existence. summative evaluation on the other hand can do just that. if the program isn't effective why keep it and by extension why continue to pay the programs director and staff/ as sechrest and figueredo 1993 stated summative evalaution and even the rationale for doing it call into question the very reasons for existence of the organisation involved. formative evaluation, by contrast, simply responds to the question "how can we be better?" without strongly implying the question "How do (we) know (we) are any good at all?' despite the political difficulty summative evaluations are the core of the evaluation process and are an essential feature of any program funded by the federal government. any agency wishing to spend tax dollars to develop a program is obligated to show those dollars are being used effectively. the actual process of performing summative evaluations involves applying some of the techniques you already know about especially quasi-experimental designs with random assignment are possible sometimes especially when evaluating a program that has more people desiring it than space available. In such a case, random assignment in the form of a lottery (random winners get the program; others wind up in a wait list control group) is not only methodologically sound, it is also the only fair procedure to use. One problem that sometimes confronts the program evaluator is how to interpret a failure to find significant differences between experimental and control groups—that is, the statistical decision is “fail to reject the null hypothesis” Such an outcome is difficult to interpret, as you recall from the discussion in Chapter 4. It could be there just isn’t any difference, yet there’s always the possibility of a Type II error being committed (an effect is real, but your study failed to find it), especially if the measuring tools are not sensitive or reliable. The program might indeed have produced some small but important effect, but the analysis failed to discover it. Although a finding of no difference can be difficult to interpret, most researchers believe that such a finding (especially if replicated) contributes important information for decision making, especially in applied research. For instance, someone advocating the continuation of a new program is obligated to show how the program is better than something already in existence. Yet, if differences between this new program and one already well established cannot be shown, then it might be wise to discontinue the new program, especially if it is more expensive to implement than the older one. A “fail to reject the null” decision also can help evaluate exaggerated claims made by advocates of a new program. A finding of no difference has important implications for decision making for reasons having to do with cost, and this brings us to the final type of program evaluation activity.

Answer 16

one type of cost‐effectiveness analysis: monitoring the actual costs of a program and relating those costs to the effectiveness of the program’s outcomes. If two programs with the same goal are equally effective but the first costs half as much as the second, then it is fairly obvious that the first program should be used. A second type of cost analysis takes place during the planning stages for a program. Estimating costs at the outset helps determine whether a program is feasible and provides a basis for the later comparison of projected costs and actual costs. Estimating costs with reference to outcomes can be a complicated process, often requiring the expertise of a specialist in cost accounting. Thus, a detailed discussion of the procedures for relating costs to outcomes is beyond the scope of this chapter. In addition, it is often difficult if not impossible to put a monetary value on the benefits that might result from the implementation and continuance of a program, especially one involving wellness.

Answer 17

during a need analysis qunatitative data from a community survey and census data can be combined with in-depth interview information from key informants and focus groups. in formative and summative assessments quantitative data can be supplemented with a qualitative analysis of interviews with agency workers and clients and with direct observations of the program in action. in program evaluation resrach it is seldom a question of whether quantitive or qualitative research is better. although there has been and continues to be debate about the relative merits of quantitive and qualitative evaluation thoughtful program evaluators rely on both.