Research and Program Evaluation Flashcards
Research
the systematic process of collecting and analyzing data for some purpose such as investigating a problem or answering a question
Evidence-based inquiry
the search for knowledge using empirical data which has been gathered systematically
Quantitative research
- assumes social facts have a single objective reality
- tends to study samples or populations
- researchers try not to influence collection of data (instruments)
- statistical methods comparing and contrasting groups occurs
- researchers examine for causes and relationships
Qualitative research
- assumes multiple realities socially constructed by individuals/groups
- tends to study individual units - person, family, community - in naturalistic setting
- researchers may be primary instrument for collecting data (through observation)
- researchers’ impressions, judgments, feelings may be used
- goal is to describe the nature of things
When to choose qual vs. quant
both kinds of research are valued
- one is chosen over the other because it better fits the assumptions of the researcher and the nature of the problem under investigation
- some professional journals prefer to publish one kind of research over the other
Inductive research
begins at the real world, practical level (small and builds to large theory)
- tends to be descriptive, correlational, historical
- leads to building of theory
- closer to qualitative research
Deductive research
springs from theory which is already established (starts broad and goes smaller to specifics)
- tries to determine what the relationships are between elements of the theory and may be experimental in nature
- closer to quantitative research
Quantitative; Non-experimental designs
Survey
may occur through questionnaires, interviews, etc. and is used to measure attitudes, perceptions, etc.
- ex. Public Opinion Poll
- often has low response rate, below 50%
- unless you know that characteristics of non-respondents are similar to respondents, must be cautious in generalizing
Quantitative; Non-experimental designs
Descriptive
describes an existing state of events
- numbers may be used to characterize groups/individuals
Quantitative; Non-experimental designs
Comparative
investigates whether there are differences between two+ groups
- no manipulation of conditions experienced by each group
Quantitative; Non-experimental designs
Correlational
this research method uses the correlation coefficient to determine the degree of relationship between two+ variables or phenomena
ex. income level and attitude toward counseling
bivariate: correlational data describing the nature of two variables
multivariate: more than two variables
Quantitative; Non-experimental designs
Ex post facto (Causal-comparative)
studies possible causal relationships among relationships ex post facto (after the fact) - no random assignment
- do not manipulate any variables; focus is on what has already happened
- may generate several reasons (causes) for the relationships you discover
- uses t-tests and ANOVAs
Quantitative; Experimental designs
True experiment
characterized by the use of experimental and control groups with random assignment to each
- used to determine cause-and-effect relationships
- ex. 60 college freshmen are enrolled in English class. 30 are randomly assigned to one-hour per week writing lab, and the others comprise a control group. End-of-semester essay exam results are analyzed to see if the lab was associated with better writing skills
Quantitative; Experimental designs
For experiments, there are design variations such as:
- treatment and control group with posttest only
- treatment and control group with pretest and posttest
- two different treatment groups with control groups and posttest
- etc.
Quantitative; Experimental designs
Quasi-experiment
similar to experimental research except that randomization of subjects to treatment and control groups is not possible
- may be that no control or comparison group is available
- result from such research will not be as unequivocal as results from a true experimental study
- ex. a school has two classrooms of 4th graders. Each classroom is taught arithmetic by a different method for the school year. In May, arithmetic achievement is compared for the two classrooms using scores on a national exam
Types of Research - Qualitative
Qualitative
emphasizes gathering data about naturally occurring phenomena (individual’s and groups’ living experiences) and events
- data collection may be in terms of words rather than numbers
Types of Research - Qualitative
Two principal qualitative research designs:
Interactive - Case Study
the case may be a program, activity, set of individuals who are bounded in time and place
Types of Research - Qualitative
Two principal qualitative research designs:
Interactive - Ethnography
a description and interpretation of a cultural or social group/system. Data is typically collected through observation and interviewing
- Issue of observer bias is important
Types of Research - Qualitative
Two principal qualitative research designs:
Noninteractive - analytical research
conducted primarily through document analysis
- ex. historical analysis (collecting and analyzing docs describing former events)
- ex. biographical analysis (written or oral)
- ex. legal analysis (focuses on law and court decisions)
Mixed-Method Research Designs
Mixed-Methods
combine quantitative and qualitative methods in the same research effort
- researcher retains the flexibility to use both types of design
- typically designs are used sequentially (quant. may be gathered first and then qual. used to further explain or elaborate on findings using surveys, interviews, focus groups)
Other Specialized Research Designs and Types
Single-subject design
studies the effects of a program or treatment on an individual or group treated as an individual, usually after a baseline has been established
Other Specialized Research Designs and Types
Action research
conducted in an attempt to improve services or a program
- may be viewed as having an evaluative function
Other Specialized Research Designs and Types
Pilot study
small-scale research effort often used to determine the feasibility of a large scale effort with emphasis on refining procedures and instrumentation
Other Specialized Research Designs and Types
Longitudinal research
collecting data from the same group of individuals over a period of time (panel study)
- ex. studying career development of school children by reinterviewing them every two years until they were high school seniors
Other Specialized Research Designs and Types
Cross-sectional research
consists of collecting data from different groups at the same time and examining these differences
- ex. studying career development by interviewing each grade of students at the same time
also called Synchronic method
Research outcomes may be measured two ways:
Within-subjects
examining what changes occur within the members of a group
IPSATIVE
Research outcomes may be measured two ways:
Between subjects
examining what changes occur between two or more groups
Meta-analysis
research comparing findings across studies
- i.e., the results of many studies are examined simultaneously and one or more research questions answered
Internal Validity
Internal Validity
experiments are internally valid to the extent that extraneous variables have been controlled
- to the extent that the treatment variable is the only one producing the observed changes, the experiment is internally valid
Threats to Internal Validity
Selection of subjects
differences in the results between two groups may not be due to the treatment variable experiement by one group because the composition of the two groups are different to begin with (probably not randomly selected)
Threats to Internal Validity
Instrumentation
differences in results between two or more groups may be due to instruments which are unreliable or because the instruments are changed during the study
- or, perhaps the observers recording data become fatigued or bored and record behaviors differentially over time
Threats to Internal Validity
Maturation
results may be due to maturational or other changes in the subjects and not due to the treatment being applied. This is especially important if research data is gathered over a long period of time
Threats to Internal Validity
Mortality or attrition
losing subjects during the study could lead to different results than if everyone had stayed. Subjects with the most or least amount of important characteristics to the study may be the ones dropping out
Think of normal curve extremes
Threats to Internal Validity
Experimenter bias
the responses of the subjects may be influenced by the researcher. This may occur by treating some subjects differently, reinforcing different behaviors, as well as the presence of many other variables which deliberately or unintentionally influence subjects
Threats to Internal Validity
Statistical regression
sometimes subjects in a study are recruited because of extreme high or low scores (e.g., self-esteem, social skills) on the dependent variable being measured. Due to statistical regression, future measures would expect these individuals to score closer to the mean score even without any intervention
External Validity
External Validity
an experiment is externally valid to the extent that the results may be generalized to people and situations beyond the study
- there are several threats to external validity of experiments and some of these are also threats to internal validity
Threats to external validity
Selection of subjects
if subjects are not randomly selected, the results may only apply to the subjects in the study
- the results can only be generalized to people with similar characteristics
Threats to external validity
Ecological validity
the research has ecological validity if the results can be generalized from one setting or circumstance to another. Sometimes the circumstances, conditions, physical surroundings of the research are so unique that the results cannot be generalized beyond that study
Threats to external validity
Subject reactions (Reactivity): Hawthorne effect
1
the influence in performance which occurs when subjects receive attention or know they are participating in research
Threats to external validity
Subject reactions (Reactivity):
Demand characteristics
2
all the cues, info, knowledge, even rumors the subject has heard about the experiment which are likely to influence their performance
Threats to external validity
Subject reactions (Reactivity):
Experimenter bias (Rosenthal effect)
3
the changes in the subject’s behavior brought about by the researcher’s expectations, behaviors, attitudes
- Rosenthal conducted research into this phenomenon and called it Pygmalion effect, referring to the self-fulfilling expectation of doing well because it is expected
Threats to external validity
Subject reactions (Reactivity):
Placebo
4
any control treatment
should be identical to the experimental treatment except for the critical item being studied
even so, control subjects may be influenced by the placebo adn react in unintended ways
Threats to external validity
Novelty and disruption effects
the measured effect of the treatment on the subjects may be due to its novelty or the disruption it causes
- being selected for research may be exciting and energizing; as it continues it may begin to disrupt routine and one’s typical schedule
- when novelty and disruption wear off or stabilize, there may be no long-term effects of the treatment
Levels of measurement
Levels of measurement
determine the statistic you can use
Levels of measurement
Nominal
variable’s qualities or categories. use non-parametric test (ex. Chi-square)
ex. male/female
Categories YES
Rank order NO
Equal Spacing NO
True zero NO
Levels of measurement
Ordinal
differences in some magnitude of the variable
ex. scores on exam can be ranked from highest to lowest
Categories YES
Rank order YES
Equal Spacing NO
True zero NO
Levels of measurement
Interval
intervals between the numbers on a scale contain the same amount of the variable throughout the scale. provide a constant unit of measurement
NO REAL ZERO
Ex. Fahrenheit temperature (the distance/interval between 11 and 12 degrees is the same as the distance between 100 and 101 degrees)
Categories YES
Rank order YES
Equal Spacing YES
True zero NO
Levels of measurement
Ratio
numbers are on a scale which has a true zero
Numbers can be compared by ratios
Ex. weight, distance, time
Ex. someone who weights 200 lbs is twice as heavy as someone who weighs 100 lbs. BUT we cannot say someone is twice as introverted as someone else (so this ex is not ratio)
Categories YES
Rank order YES
Equal Spacing YES
True zero YES
Sampling
Sampling
how well samping is conducted will determine how validly we can generalize from a sample to a population
Involves the selection of a part of the population
Probability Sampling
Random sampling
all the individuals in the population have an equal and independent chance of being selected
Probability Sampling
Stratified sampling
sampling in such a way that major subgroups in the population will be sampled (ex. gender, age, ethnicity, etc.)
- can be proportional or disproportionate
Probability Sampling
Proportional stratified sampling
randomly selecting the same proportion of individuals for the sample as they represent proportionally in the major subgroups in the population
Ex. if 1/2 of a population is Hispanic and 1/2 is white, you would randomly select your sample to be 1/2 Hispanic and 1/2 white
Probability Sampling
Systematic sampling
researchers select members of the population at a fixed interval determined in advance
ex. selecting every 3rd study on a roster
Probability Sampling
Cluster sampling
the unit is not an individual but a naturally occurring group of individuals (ex. classrooms, city blocks)
Clusters are randomly selected for the study
Nonprobability Sampling
Non-random or nonprobability samples
samples of convenience or volunteer samples
- cannot be counted on to yield a normal distribution of scores but can yield useful/important data
Nonprobability Sampling
Purposeful sampling
in some studies, there may be no interest in generalizing findings so this may be used.
- selecting participants based on specific criteria such as characteristics or lived experiences
- can be helpful to gather in-depth info on a topic
Possibilities:
- comprehensive sampling where every case/event is selected
- there is extreme-case or typical-case selection
Ex. researchers studying individual’s lived experience with depression post flyers in counseling center
Nonprobability Sampling
Convenience sampling
participants are recruited who are easily accessible and who are usually close in proximity
ex. you select your classmates for your research on graduate counseling experiences
Nonprobability Sampling
Snowball sampling
participants are asked to assist researchers in identifying other potential subjects
Can be helpful when conducting research about people with specific traits
ex. you recruit 5 counselors in supervision and have each give you names of other counselors in supervision who you then recruit
Study tip: Michael Scott stuck in a pyramid scheme coded