Exam #2 (2 of 2) Flashcards
SAMPLING AND GENERALIZATION
SAMPLING AND GENERALIZATION
Population and Census
Population = the entire set of people the reseacher desires to learn about
Census = measures each person in the population (not feasible for the research we do)
Sampling and Sample
Sampling = the selection of certain individuals to participate in the research
* The researcher uses information about these individuals to make inferences about the population from which they were drawn
* relates to inclusion/exclusion criteria, etc.
Sample = the people who actually participate in the research
* Researchers will not be able to know exactly the true characteristics of a population, but a representative sample helps facilitates these inferences
* representative samples are approximately the same as the population in every important respect
Probability sampling
Probability sampling is used to draw a representative sample from the population of interest
* –> each person in the population has a known and non-zero (though not necessarily equal) chance of being selected
* probability sampling often used in survey research/political polling
Types of probability sampling
Types of probability sampling:
- Simple random sampling
- Systematic random sampling
- Stratified sampling
- Disproportionate stratified sampling
- Cluster sampling
Simple random sampling
Simple random sampling:
* Participants are randomly selected from a list of all members of the population of interest
* Every person on the list has the same chance of being selected
* Sampling frame = complete list of all the people in the population
Systematic random sampling
Systematic random sampling
- With systematic random sampling, the names on the sampling frame are known to be listed in random sequence
- A random starting point can be chosen
- Ever nth name can be selected
- Example: 5 is the random starting point chosen, then every 5th person chosen until we get 20 people
Stratified sampling
Stratified sampling
- Stratified sampling is used when we want to make sure that some characteristic is represented in the sample in the same proportion as it is represented in the population
- Involves drawing separate samples from a set of known subgroups called strata rather than sampling from the population as a whole
- Example: Two strata of IUP graduate and undergraduate students
Disproportionate stratified sampling
Disproportionate stratified sampling
* Disproportionate stratified sampling is used when the strata differ in size and we want to include enough people from each one to allow us to compare the characteristics of the strata
* This approach involves a technique called oversampling = drawing a sample that includes a larger proportion of some strata than they are represented in the population
* Example: Inviting all students from a small department and a small # of students from a large department to attempt getting equal groups in each stratum
Cluster sampling
Cluster sampling
* Cluster sampling can be used when a complete sampling frame does not exist
* We break the population into a set of smaller groups, called clusters
* Then we randomly choose some of the clusters until we reach a level where there are sampling frames
Example: study about alcohol use and the way students spend their time on college campuses. Not possible to get students from every university. Instead, draw from 5 random states. Then draw from 5 universities from list of state’s universities. Then conduct research on 25% of students
* –> go through a randomized process to get sampling frames
Representative sampling & sampling bias
Representative sampling requires:
* The existence of one or more sampling frames listing the entire population of interest, AND
* All selected individuals must be sampled
Sampling bias occurs when either of these conditions is not met
* There is the potential that the sample is not representative of the population
Nonprobability sampling
Nonprobability sampling
- When probability sampling is not possible, nonprobability samples must be used
- Examples of nonprobability sampling include:
- –Snowball sampling
- –Convenience sampling
Snowball sampling
Snowball sampling is used when members of the population of interest are rare or difficult to reach
* One or more individuals from the population are contacted
* These individuals lead the researcher to other members of the population
* Example: New parents sharing study with other new parents they know
Convenience sampling
Convenience sampling
* When convenience sampling is used, the researcher samples individuals who are readily available without any attempt to draw a sample that is representative of a certain population
* Even though psych 101 students are all psych students, the researchers aren’t studying them for that specific characteristic. Hence, convenience sampling
Random sampling vs. Random assignment
Random sampling selects participants for the research on the basis of chance
* Random sampling strengthens EXTERNAL validity by drawing a representative sample
Random assignment places people into experimental conditions on the basis of chance
* Random assignment strengthens INTERNAL validity by evenly distributing participant characteristics across conditions and thus ruling out selection threat
GENERALIZED CAUSAL INFERENCE
GENERALIZED CAUSAL INFERENCE
Relationships between generalizability and construct validity/external validity
Construct validity: validity of inferences about generalizing from specific operationalizations to higher order abstract constructs
External validity: validity of inferences about generalizing the cause-effect relationship to other persons, settings, treatments, and measurements
Mnemonic for remembering the 5 principles of generalized causal inference
5 principles of generalized causal inference mnemonic:
Snakes rarely meet in caves
5 Principles of Generalized Causal Inference
5 Principles of Generalized Causal Inference:
- Surface similarity
- Ruling out irrelevancies
- Making discriminations
- Interpolation and extrapolation
- Causal explanation
- Surface similarity
- Surface similarity
- How similar do the particulars of the study seem to be to the prototypical features of the target of generalization?
- Surface similarity:
Applications to construct validity
- Face validity: do the study particulars appear to capture the intended constructs?
- Content validity: do the study particulars capture all the prototypical features of the intended constructs?
- Inadequate explication of constructs impedes these judgments
- Surface similarity:
* Applications to external validity
- We often make judgments about likely generalizability of research findings based on how similar the study particulars are to the target of generalization (will the intervention work well in both Illinois and Rhode Island? Consider similarities of settings)
- Campbell’s principle of proximal similarity = greater similarity leads to greater confidence about generalization
2. Ruling out irrelevancies
2. Ruling out irrelevancies
- What characteristics of the people, settings, treatments, and outcomes are irrelevant (because they do not limit generalization)?
- They do not qualify the effect that we are finding –> don’t limit generalizability
2. Ruling out irrelevancies
Applications to construct validity
Multiple operationalism: Across the operationalizations of a construct, there should be heterogeneity of irrelevant features. This rules out the mono-method and mono-operation threats to construct validity
- Example: Social psych studies. Some involve computers, some ink blots, etc., These features are irrelevant as long as they are measuring the same construct
Campbell: heterogeneity of irrelevancies help make sure the construct is not confounded even with apparently irrelevant features - Think about within context of mono-method and mono-operation: are we really getting at this bigger construct we’re interested in?
2. Ruling out irrelevancies
Applications to external validity
- Identifying variables that are NOT moderators (what other variables don’t matter?)
- If a strong effect is found despite heterogeneity, those variables are unlikely to be moderators (e.g., we find a strong effect even though the individuals differ)
3. Making discriminations
3. Making discriminations
- What characteristics of the people, settings, treatments, and outcomes DO limit generalization?
3. Making discriminations
Applications to construct validity
- (Discriminant validity: distinguishing a measure of a construct from measures designed to assess other related constructs) (e.g., making sure an anxiety measure isn’t also measuring similar constructs like depression)
- Similar process for persons, settings, and treatments. Defining by showing how they are different from other persons, settings, and treatments
- –In addition to describing the setting, also want to describe what is different between settings
- Example: Testing effectiveness of shorter versions of a treatment–explain what the differences are, what are we shortening, etc.
3. Making discriminations
Applications to external validity
- Identifying moderators of the causal effect
4. Interpolation and Extrapolation
Interpolation: generalizing to unsampled values within the range of the sampled persons, settings, treatments, and outcomes
* Example: Medication tested at 50mg, 100mg, 150mg. Generalizing to 75mg (which wasn’t tested) would be interpolation–still within range
Extrapolation: generalizing beyond the range of the sampled persons, settings, treatments, and outcomes
* Same example: Generalize to 250mg? That’s extrapolation–outside of range
4. Interpolation and Extrapolation
Applications to construct validity
- Careful to avoid confounding constructs with levels of constructs, as this can lead us to extrapolate too far beyond the study particulars
- Example: Sure, 10 sessions of therapy could reasonbly be expected to be superior to 5 sessions of therapy. But 1,000,000 sessions of therapy? –> can’t extrapolate too far. Keep it close to range of sampled values
4. Interpolation and Extrapolation
Applications to external validity
(What are interpolation and extrapolation aided by?)
Interpolation and extrapolation are aided by:
1. sampling more values,
2. well-specified functional form of the relationship over the sampled range (e.g., curvi-linear relationship? linear relationship?)
3. making judgments about values close to those included in the study
5. Causal explanation
- What processes underlie/explain the relationship to be generalized?
- Causal explanation
Applications to construct validity
- Deep/structural similarity: similarities in the underlying structural relations in different operational instances that give rise to a common concept that characterizes all instances
- IOW: making sure all operationalizations are tapping into the bigger/underlying construct
- Example: we want the experience of I-sharing to be ‘felt,’ however it’s operationalized
5. Causal explanation
Applications to external validity
5. Causal explanation: Applications to external validity
- Knowing the full causal system (including mediators) allows us to reproduce a causal effect with other people, in other settings, and using other treatments and outcomes
Feasibility of the 5 principles of generalized causal inference
Feasibility of the 5 Principles of Generalized Causal Inference
Surface similarity and causal explanation are feasible in single studies
- Example: Social program is effective. Can then examine the surface similarity between that program and where it was implemented and the target of generalization. How similar the populations, how similar the settings?
Ruling out irrelevancies, making discriminations, and interpolation/extrapolation are more feasible in research programs
- Heterogeneity of irrelevant features–having multiple studies will help us find multiple irrelevancies, make more discriminations, etc.
Random sampling (for generalizability)
Random sampling is sometimes a good way to increase the generalizability of research findings
* But mostly for generalization to persons (and even then, not always feasible)
Much more difficult to randomly sample treatments or outcomes
* We usually want to choose these based on theoretical reasons
Purposive Sampling
(purposeful sampling)
PSI-Het
Purposive sampling of heterogeneous instances (PSI-Het)
- Drawing a sample that captures the range of values present in the population of interest
- Develop a sample that is heterogeneous and broad
- –> helps identify irrelevancies, make discriminations, identify what makes a difference, interpolation and extrapolation
- SCC: PSI-Het can reduce power due to the heterogeneity of units threat to stat. conclusion validity
Purposive Sampling
(purposeful sampling)
PSI-Typ
Purposive sampling of typical instances (PSI-Typ)
- Drawing a sample close to the central tendency of the population of interest
- Develop a sample that is average or most typical
- Results generalize to typical cases
Not just applicable to people, but also setting, treatment type, and outcomes we measure
Purposive sampling and surface similarity
Purposive sampling and surface similarity
Generalization is facilitated by similarity between the study particulars and the target of generalization
- PSI-Typ and PSI-Het both useful, depending on the target of generalizations
- Example: More interested in widely applying findings to other peoples/treatments? Use PSI-Het
- Researchers should clearly define study particulars so that readers can assess surface similarity for themselves (e.g., who participated, what settings, what measures, what outcomes?)
Purposive sampling and ruling out irrelevancies
Purposive sampling and ruling out irrelevancies
Generalization is facilitated by identifying attributes of persons, settings, treatments, and outcomes that are irrelevant
* PSI-Het allows testing those characteristics that are presumed to be irrelevant
Might want to make heterogenous those study particulars presumed irrelevant but that critics might argue with
* Example: If we don’t think it matters whether program targets 18–25 yo or 40-45yo, it’s helpful to confirm this
If PSI-Het is not used, there will still be variability in the study, which we can measure to determine whether those characteristics are truly irrelevant
* IOW: Even if we don’t purposefully try to recruit a heterogenous sample, we can still conduct analyses because there will still be some kinds of differences
Purposive sampling and making discriminations
Purposive sampling and making discriminations:
Generalization is facilitated by distinguishing the people, settings, treatments, and outcomes for which generalization does or does not hold
- Can include potential moderating variables
- Can use stratification
- Can add differentiating measures (nonequivalent DVs)
- Can measure features of persons, settings, and treatments that may affect generalization
PSI-Het and PSI-Typ could both be useful here
- Ex: PSI-Het and Big 5 –> continuum of extroversion, whereas PSI-Typ has those closer to central tendency (e.g., recruiting some people with high extroversion and some people with low extroversion –> high contrast)
- PSI-Het and psychotherapy –> 10 different types of therapy
- PSI-Typ and psychotherapy –> two most typical therapies/time spent in therapy
Purposive sampling and interpolation/extrapolation
Purposive sampling and interpolation/extrapolation
Generalization is faciliated by having many levels of the study feature on which generalization is desired
- Can often diversify people and outcomes (easy to find different people, add additional outcome measures, and add different measures that operationalize the same construct)
May not always be possible to diversify settings and treatments
- But if unplanned variability in treatment occurs, the researcher should measure it (e.g., we expect participants to attend 10 sessions but they only come to 5)
- And the researcher should measure/describe features of the setting(s)
Purposive sampling and causal explanation
Purposive sampling and causal explanation
- Relies less on sampling techniques
Three techniques:
Qualitative methods (e.g., observation, interviews)
- If we are trying to determine what the treatment does for people that leads to outcomes –> could observe what’s happening, interview subjects, etc.
Statistical methods that help establish mediation (e.g., SEM, bootstrapping)
Experiments that manipulate explanatory variables
- E.g., directly manipulate the mediator and see if it has direct impact on the outcome variable
GENERALIZED CAUSAL INFERENCE ACROSS MULTIPLE STUDIES
GENERALIZED CAUSAL INFERENCE ACROSS MULTIPLE STUDIES
Directed programs of experiments
Directed programs of experiments = series of replications designed to systematically investigate treatments, outcomes, moderators, and mediators
How to synthesize this information?
Narrative Reviews of Existing Literature
Narrative Reviews of Existing Literature:
Author synthesizes the results from the existing research about topic
Vote-counting = tally research with significant positive effect, negative effect, and no effect
- This approach is limited. Many studies underpowered, not informative about effect size, may not consider sample size
A lot of information to keep track of, especially with many studies on a given topic
- Can be subjective–influenced by author’s perspective
Meta-Analysis
Meta-Analysis
- Quantitative way of synthesizing the results of multiple studies on a given topic
- Basic procedure: Compute common effect size for each study and average them
Mnemonic for remembering 5 meta-analytic steps
5 meta-analytic steps mnemonic:
Incredible chefs cook amazing Italian
What are the 5 meta-analytic steps?
5 meta-analytic steps:
- Identify problem and perform literature review
- Coding of studies
- Computing effect sizes
- Analysis
- Interpreting and presenting results
Meta-Analysis steps
1. Identify problem and perform literature review
1. Identify problem and perform literature review:
Need clearly defined research question to determine criteria for inclusion
- How broad/narrow is the scope of the project?
- Include moderators/mediators or just main effect?
- What participant populations to include?
- Cultural/Linguistic range? (e.g., articles in other languages?)
- Time frame?
The more uncertainty about the research question, about the articles to include, etc., the more room for subjectivity
Identify problem and perform literature review
* Find relevant literature (all studies that meet inclusion criteria)
* Try to avoid file-drawer problem (look for fugitive literature)
* (electronic databases, articles’ references, asking people who work in the relevant areas, email relevant listservs)
Meta-Analysis steps
#2. Coding of studies
Meta-Analysis steps
#2. Coding of studies:
Must capture relevant characteristics of each study, such as:
- Participant characteristics, sample size
- Setting characteristics, year of study
- Treatment methodology/delivery
- Outcome measure
Missing or ambiguous information complicates the coding process
Can be very time consuming, especially as the number of studies increases
* Whether you need the same IV operationalization, methodology, etc., depends on the researcher, the topic, etc.
Meta-Analysis steps
*#3. Computing effect sizes
Meta-Analysis steps
#3. Computing effect sizes
- Many studies have different outcome measures, so it’s important to calculate a common effect size (like d or r) for each
Meta-Analysis steps
*#4. Analysis
Meta-Analysis steps
#4. Analysis:
- Calculate the average effect size (may weight studies according to sample size)
- Homogeneity testing indicates whether variance among studies reflects only sampling error or systematic error that could be accounted for by other predictors
- Sensitivity analysis can indicate how many nonsignificant studies would need to be included to alter the conclusions of the meta-analysis
Meta-Analysis steps
#5. Interpreting and presenting results
Meta-Analysis steps
#5. Interpreting and presenting results:
- Present the findings
Explain the significance of the findings and how they add to the literature
Meta-analysis assists SCC’s 5 principles of generalized causal inference.
What are those principles again?
5 principles of generalized causal inference:
- Surface similarity
- Ruling out irrelevancies
- Making discriminations
- Interpolation and extrapolation
- Causal explanation
Meta-analysis and surface similarity
Meta analysis and surface similarity
- Collection of studies increases the number of construct inferences that can be made (e.g., similar populations –> more generalizable)
- Collection of studies increases the chance that one will present similar characteristics to a targeted generalization
- If the literature is limited in some way, the meta-analysis makes this known
Meta-analyses help highlight whether and how the literature is limited
Meta-analysis and ruling out irrelevancies
Meta analysis and ruling out irrelevancies:
- Probably heterogeneous irrelevancies in the many studies collected for the meta-analysis
- Can collapse across irrelevances (by calculating single effect size across studies)
- If strong finding still emerges, indicates robust effect, and suggests those variables are irrelevant
If there are enough studies (and therefore enough power), we can analyze those variables as potential moderators to determine whether they are really irrelevant
* (e.g., gender of experimenter)
Meta-analysis and making discriminations
Meta analysis and making discriminations
Many more study features varied across studies than within any given study
- Helps to clarify the relevant part(s) of a treatment
- Helps to clarify the boundaries of a treatment effect (e.g,. the treatment is really effective for people of a certain age range, or for a certain diagnosis)
Meta-analysis and interpolation/extrapolation
Meta analysis and interpolation/extrapolation
Likely to be more values, and a wider range of values in the studies collected for a meta-analysis than in any single study
Meta-analysis and causal explanation
Meta analysis and causal explanation
- More information available from diverse studies to help elucidate relevant parts of causal effect and mediators
- We’re looking for the process, mediators, etc. –> what explains how this treatment’s outcomes?
