Brainscape's Knowledge GenomeTM

Browse over 1 million classes created by top students, professors, publishers, and experts.


See full index

Applied Research Methods - Shared > Exam #2 (2 of 2) > Flashcards

Exam #2 (2 of 2) Flashcards

1
Q

SAMPLING AND GENERALIZATION

A

SAMPLING AND GENERALIZATION

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

Population and Census

A

Population = the entire set of people the reseacher desires to learn about
Census = measures each person in the population (not feasible for the research we do)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

Sampling and Sample

A

Sampling = the selection of certain individuals to participate in the research
* The researcher uses information about these individuals to make inferences about the population from which they were drawn
* relates to inclusion/exclusion criteria, etc.

Sample = the people who actually participate in the research
* Researchers will not be able to know exactly the true characteristics of a population, but a representative sample helps facilitates these inferences
* representative samples are approximately the same as the population in every important respect

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

Probability sampling

A

Probability sampling is used to draw a representative sample from the population of interest
* –> each person in the population has a known and non-zero (though not necessarily equal) chance of being selected
* probability sampling often used in survey research/political polling

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

Types of probability sampling

A

Types of probability sampling:

  • Simple random sampling
  • Systematic random sampling
  • Stratified sampling
  • Disproportionate stratified sampling
  • Cluster sampling
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

Simple random sampling

A

Simple random sampling:
* Participants are randomly selected from a list of all members of the population of interest
* Every person on the list has the same chance of being selected
* Sampling frame = complete list of all the people in the population

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

Systematic random sampling

A

Systematic random sampling

  • With systematic random sampling, the names on the sampling frame are known to be listed in random sequence
  • A random starting point can be chosen
  • Ever nth name can be selected
  • Example: 5 is the random starting point chosen, then every 5th person chosen until we get 20 people
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

Stratified sampling

A

Stratified sampling

  • Stratified sampling is used when we want to make sure that some characteristic is represented in the sample in the same proportion as it is represented in the population
  • Involves drawing separate samples from a set of known subgroups called strata rather than sampling from the population as a whole
  • Example: Two strata of IUP graduate and undergraduate students
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

Disproportionate stratified sampling

A

Disproportionate stratified sampling
* Disproportionate stratified sampling is used when the strata differ in size and we want to include enough people from each one to allow us to compare the characteristics of the strata
* This approach involves a technique called oversampling = drawing a sample that includes a larger proportion of some strata than they are represented in the population
* Example: Inviting all students from a small department and a small # of students from a large department to attempt getting equal groups in each stratum

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

Cluster sampling

A

Cluster sampling
* Cluster sampling can be used when a complete sampling frame does not exist
* We break the population into a set of smaller groups, called clusters
* Then we randomly choose some of the clusters until we reach a level where there are sampling frames

Example: study about alcohol use and the way students spend their time on college campuses. Not possible to get students from every university. Instead, draw from 5 random states. Then draw from 5 universities from list of state’s universities. Then conduct research on 25% of students
* –> go through a randomized process to get sampling frames

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

Representative sampling & sampling bias

A

Representative sampling requires:
* The existence of one or more sampling frames listing the entire population of interest, AND
* All selected individuals must be sampled

Sampling bias occurs when either of these conditions is not met
* There is the potential that the sample is not representative of the population

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

Nonprobability sampling

A

Nonprobability sampling

  • When probability sampling is not possible, nonprobability samples must be used
  • Examples of nonprobability sampling include:
  • –Snowball sampling
  • –Convenience sampling
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

Snowball sampling

A

Snowball sampling is used when members of the population of interest are rare or difficult to reach
* One or more individuals from the population are contacted
* These individuals lead the researcher to other members of the population
* Example: New parents sharing study with other new parents they know

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

Convenience sampling

A

Convenience sampling
* When convenience sampling is used, the researcher samples individuals who are readily available without any attempt to draw a sample that is representative of a certain population
* Even though psych 101 students are all psych students, the researchers aren’t studying them for that specific characteristic. Hence, convenience sampling

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

Random sampling vs. Random assignment

A

Random sampling selects participants for the research on the basis of chance
* Random sampling strengthens EXTERNAL validity by drawing a representative sample

Random assignment places people into experimental conditions on the basis of chance
* Random assignment strengthens INTERNAL validity by evenly distributing participant characteristics across conditions and thus ruling out selection threat

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

GENERALIZED CAUSAL INFERENCE

A

GENERALIZED CAUSAL INFERENCE

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
17
Q

Relationships between generalizability and construct validity/external validity

A

Construct validity: validity of inferences about generalizing from specific operationalizations to higher order abstract constructs

External validity: validity of inferences about generalizing the cause-effect relationship to other persons, settings, treatments, and measurements

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
18
Q

Mnemonic for remembering the 5 principles of generalized causal inference

A

5 principles of generalized causal inference mnemonic:

Snakes rarely meet in caves

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
19
Q

5 Principles of Generalized Causal Inference

A

5 Principles of Generalized Causal Inference:

  1. Surface similarity
  2. Ruling out irrelevancies
  3. Making discriminations
  4. Interpolation and extrapolation
  5. Causal explanation
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
20
Q
  1. Surface similarity
A
  1. Surface similarity
  • How similar do the particulars of the study seem to be to the prototypical features of the target of generalization?
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
21
Q
  1. Surface similarity:
    Applications to construct validity
A
  • Face validity: do the study particulars appear to capture the intended constructs?
  • Content validity: do the study particulars capture all the prototypical features of the intended constructs?
  • Inadequate explication of constructs impedes these judgments
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
22
Q
  1. Surface similarity:
    * Applications to external validity
A
  • We often make judgments about likely generalizability of research findings based on how similar the study particulars are to the target of generalization (will the intervention work well in both Illinois and Rhode Island? Consider similarities of settings)
  • Campbell’s principle of proximal similarity = greater similarity leads to greater confidence about generalization
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
23
Q

2. Ruling out irrelevancies

A

2. Ruling out irrelevancies

  • What characteristics of the people, settings, treatments, and outcomes are irrelevant (because they do not limit generalization)?
  • They do not qualify the effect that we are finding –> don’t limit generalizability
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
24
Q

2. Ruling out irrelevancies

Applications to construct validity

A

Multiple operationalism: Across the operationalizations of a construct, there should be heterogeneity of irrelevant features. This rules out the mono-method and mono-operation threats to construct validity

  • Example: Social psych studies. Some involve computers, some ink blots, etc., These features are irrelevant as long as they are measuring the same construct
    Campbell: heterogeneity of irrelevancies help make sure the construct is not confounded even with apparently irrelevant features
  • Think about within context of mono-method and mono-operation: are we really getting at this bigger construct we’re interested in?
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
25
Q

2. Ruling out irrelevancies

Applications to external validity

A
  • Identifying variables that are NOT moderators (what other variables don’t matter?)
  • If a strong effect is found despite heterogeneity, those variables are unlikely to be moderators (e.g., we find a strong effect even though the individuals differ)
26
Q

3. Making discriminations

A

3. Making discriminations

  • What characteristics of the people, settings, treatments, and outcomes DO limit generalization?
27
Q

3. Making discriminations

Applications to construct validity

A
  • (Discriminant validity: distinguishing a measure of a construct from measures designed to assess other related constructs) (e.g., making sure an anxiety measure isn’t also measuring similar constructs like depression)
  • Similar process for persons, settings, and treatments. Defining by showing how they are different from other persons, settings, and treatments
  • –In addition to describing the setting, also want to describe what is different between settings
  • Example: Testing effectiveness of shorter versions of a treatment–explain what the differences are, what are we shortening, etc.
28
Q

3. Making discriminations

Applications to external validity

A
  • Identifying moderators of the causal effect
29
Q

4. Interpolation and Extrapolation

A

Interpolation: generalizing to unsampled values within the range of the sampled persons, settings, treatments, and outcomes
* Example: Medication tested at 50mg, 100mg, 150mg. Generalizing to 75mg (which wasn’t tested) would be interpolation–still within range

Extrapolation: generalizing beyond the range of the sampled persons, settings, treatments, and outcomes
* Same example: Generalize to 250mg? That’s extrapolation–outside of range

30
Q

4. Interpolation and Extrapolation

Applications to construct validity

A
  • Careful to avoid confounding constructs with levels of constructs, as this can lead us to extrapolate too far beyond the study particulars
  • Example: Sure, 10 sessions of therapy could reasonbly be expected to be superior to 5 sessions of therapy. But 1,000,000 sessions of therapy? –> can’t extrapolate too far. Keep it close to range of sampled values
31
Q

4. Interpolation and Extrapolation

Applications to external validity
(What are interpolation and extrapolation aided by?)

A

Interpolation and extrapolation are aided by:
1. sampling more values,
2. well-specified functional form of the relationship over the sampled range (e.g., curvi-linear relationship? linear relationship?)
3. making judgments about values close to those included in the study

32
Q

5. Causal explanation

A
  • What processes underlie/explain the relationship to be generalized?
33
Q
  1. Causal explanation
    Applications to construct validity
A
  • Deep/structural similarity: similarities in the underlying structural relations in different operational instances that give rise to a common concept that characterizes all instances
  • IOW: making sure all operationalizations are tapping into the bigger/underlying construct
  • Example: we want the experience of I-sharing to be ‘felt,’ however it’s operationalized
34
Q

5. Causal explanation

Applications to external validity

A

5. Causal explanation: Applications to external validity

  • Knowing the full causal system (including mediators) allows us to reproduce a causal effect with other people, in other settings, and using other treatments and outcomes
35
Q

Feasibility of the 5 principles of generalized causal inference

A

Feasibility of the 5 Principles of Generalized Causal Inference

Surface similarity and causal explanation are feasible in single studies

  • Example: Social program is effective. Can then examine the surface similarity between that program and where it was implemented and the target of generalization. How similar the populations, how similar the settings?

Ruling out irrelevancies, making discriminations, and interpolation/extrapolation are more feasible in research programs

  • Heterogeneity of irrelevant features–having multiple studies will help us find multiple irrelevancies, make more discriminations, etc.
36
Q

Random sampling (for generalizability)

A

Random sampling is sometimes a good way to increase the generalizability of research findings
* But mostly for generalization to persons (and even then, not always feasible)
Much more difficult to randomly sample treatments or outcomes
* We usually want to choose these based on theoretical reasons

37
Q

Purposive Sampling
(purposeful sampling)
PSI-Het

A

Purposive sampling of heterogeneous instances (PSI-Het)

  • Drawing a sample that captures the range of values present in the population of interest
  • Develop a sample that is heterogeneous and broad
  • –> helps identify irrelevancies, make discriminations, identify what makes a difference, interpolation and extrapolation
  • SCC: PSI-Het can reduce power due to the heterogeneity of units threat to stat. conclusion validity
38
Q

Purposive Sampling
(purposeful sampling)
PSI-Typ

A

Purposive sampling of typical instances (PSI-Typ)

  • Drawing a sample close to the central tendency of the population of interest
  • Develop a sample that is average or most typical
  • Results generalize to typical cases
    Not just applicable to people, but also setting, treatment type, and outcomes we measure
39
Q

Purposive sampling and surface similarity

A

Purposive sampling and surface similarity

Generalization is facilitated by similarity between the study particulars and the target of generalization

  • PSI-Typ and PSI-Het both useful, depending on the target of generalizations
  • Example: More interested in widely applying findings to other peoples/treatments? Use PSI-Het
  • Researchers should clearly define study particulars so that readers can assess surface similarity for themselves (e.g., who participated, what settings, what measures, what outcomes?)
40
Q

Purposive sampling and ruling out irrelevancies

A

Purposive sampling and ruling out irrelevancies
Generalization is facilitated by identifying attributes of persons, settings, treatments, and outcomes that are irrelevant
* PSI-Het allows testing those characteristics that are presumed to be irrelevant
Might want to make heterogenous those study particulars presumed irrelevant but that critics might argue with
* Example: If we don’t think it matters whether program targets 18–25 yo or 40-45yo, it’s helpful to confirm this

If PSI-Het is not used, there will still be variability in the study, which we can measure to determine whether those characteristics are truly irrelevant
* IOW: Even if we don’t purposefully try to recruit a heterogenous sample, we can still conduct analyses because there will still be some kinds of differences

41
Q

Purposive sampling and making discriminations

A

Purposive sampling and making discriminations:

Generalization is facilitated by distinguishing the people, settings, treatments, and outcomes for which generalization does or does not hold

  • Can include potential moderating variables
  • Can use stratification
  • Can add differentiating measures (nonequivalent DVs)
  • Can measure features of persons, settings, and treatments that may affect generalization

PSI-Het and PSI-Typ could both be useful here

  • Ex: PSI-Het and Big 5 –> continuum of extroversion, whereas PSI-Typ has those closer to central tendency (e.g., recruiting some people with high extroversion and some people with low extroversion –> high contrast)
  • PSI-Het and psychotherapy –> 10 different types of therapy
  • PSI-Typ and psychotherapy –> two most typical therapies/time spent in therapy
42
Q

Purposive sampling and interpolation/extrapolation

A

Purposive sampling and interpolation/extrapolation

Generalization is faciliated by having many levels of the study feature on which generalization is desired

  • Can often diversify people and outcomes (easy to find different people, add additional outcome measures, and add different measures that operationalize the same construct)

May not always be possible to diversify settings and treatments

  • But if unplanned variability in treatment occurs, the researcher should measure it (e.g., we expect participants to attend 10 sessions but they only come to 5)
  • And the researcher should measure/describe features of the setting(s)
43
Q

Purposive sampling and causal explanation

A

Purposive sampling and causal explanation

  • Relies less on sampling techniques
    Three techniques:

Qualitative methods (e.g., observation, interviews)

  • If we are trying to determine what the treatment does for people that leads to outcomes –> could observe what’s happening, interview subjects, etc.

Statistical methods that help establish mediation (e.g., SEM, bootstrapping)

Experiments that manipulate explanatory variables

  • E.g., directly manipulate the mediator and see if it has direct impact on the outcome variable
44
Q

GENERALIZED CAUSAL INFERENCE ACROSS MULTIPLE STUDIES

A

GENERALIZED CAUSAL INFERENCE ACROSS MULTIPLE STUDIES

45
Q

Directed programs of experiments

A

Directed programs of experiments = series of replications designed to systematically investigate treatments, outcomes, moderators, and mediators

How to synthesize this information?

46
Q

Narrative Reviews of Existing Literature

A

Narrative Reviews of Existing Literature:

Author synthesizes the results from the existing research about topic

Vote-counting = tally research with significant positive effect, negative effect, and no effect

  • This approach is limited. Many studies underpowered, not informative about effect size, may not consider sample size

A lot of information to keep track of, especially with many studies on a given topic

  • Can be subjective–influenced by author’s perspective
47
Q

Meta-Analysis

A

Meta-Analysis

  • Quantitative way of synthesizing the results of multiple studies on a given topic
  • Basic procedure: Compute common effect size for each study and average them
48
Q

Mnemonic for remembering 5 meta-analytic steps

A

5 meta-analytic steps mnemonic:

Incredible chefs cook amazing Italian

49
Q

What are the 5 meta-analytic steps?

A

5 meta-analytic steps:

  1. Identify problem and perform literature review
  2. Coding of studies
  3. Computing effect sizes
  4. Analysis
  5. Interpreting and presenting results
50
Q

Meta-Analysis steps
1. Identify problem and perform literature review

A

1. Identify problem and perform literature review:

Need clearly defined research question to determine criteria for inclusion

  • How broad/narrow is the scope of the project?
  • Include moderators/mediators or just main effect?
  • What participant populations to include?
  • Cultural/Linguistic range? (e.g., articles in other languages?)
  • Time frame?

The more uncertainty about the research question, about the articles to include, etc., the more room for subjectivity

Identify problem and perform literature review
* Find relevant literature (all studies that meet inclusion criteria)
* Try to avoid file-drawer problem (look for fugitive literature)
* (electronic databases, articles’ references, asking people who work in the relevant areas, email relevant listservs)

51
Q

Meta-Analysis steps
#2. Coding of studies

A

Meta-Analysis steps
#2. Coding of studies:

Must capture relevant characteristics of each study, such as:

  • Participant characteristics, sample size
  • Setting characteristics, year of study
  • Treatment methodology/delivery
  • Outcome measure

Missing or ambiguous information complicates the coding process

Can be very time consuming, especially as the number of studies increases
* Whether you need the same IV operationalization, methodology, etc., depends on the researcher, the topic, etc.

52
Q

Meta-Analysis steps
*#3. Computing effect sizes

A

Meta-Analysis steps
#3. Computing effect sizes

  • Many studies have different outcome measures, so it’s important to calculate a common effect size (like d or r) for each
53
Q

Meta-Analysis steps
*#4. Analysis

A

Meta-Analysis steps
#4. Analysis:

  • Calculate the average effect size (may weight studies according to sample size)
  • Homogeneity testing indicates whether variance among studies reflects only sampling error or systematic error that could be accounted for by other predictors
  • Sensitivity analysis can indicate how many nonsignificant studies would need to be included to alter the conclusions of the meta-analysis
54
Q

Meta-Analysis steps
#5. Interpreting and presenting results

A

Meta-Analysis steps
#5. Interpreting and presenting results:

  • Present the findings
    Explain the significance of the findings and how they add to the literature
55
Q

Meta-analysis assists SCC’s 5 principles of generalized causal inference.

What are those principles again?

A

5 principles of generalized causal inference:

  1. Surface similarity
  2. Ruling out irrelevancies
  3. Making discriminations
  4. Interpolation and extrapolation
  5. Causal explanation
56
Q

Meta-analysis and surface similarity

A

Meta analysis and surface similarity

  • Collection of studies increases the number of construct inferences that can be made (e.g., similar populations –> more generalizable)
  • Collection of studies increases the chance that one will present similar characteristics to a targeted generalization
  • If the literature is limited in some way, the meta-analysis makes this known

Meta-analyses help highlight whether and how the literature is limited

57
Q

Meta-analysis and ruling out irrelevancies

A

Meta analysis and ruling out irrelevancies:

  • Probably heterogeneous irrelevancies in the many studies collected for the meta-analysis
  • Can collapse across irrelevances (by calculating single effect size across studies)
  • If strong finding still emerges, indicates robust effect, and suggests those variables are irrelevant

If there are enough studies (and therefore enough power), we can analyze those variables as potential moderators to determine whether they are really irrelevant
* (e.g., gender of experimenter)

58
Q

Meta-analysis and making discriminations

A

Meta analysis and making discriminations

Many more study features varied across studies than within any given study

  • Helps to clarify the relevant part(s) of a treatment
  • Helps to clarify the boundaries of a treatment effect (e.g,. the treatment is really effective for people of a certain age range, or for a certain diagnosis)
59
Q

Meta-analysis and interpolation/extrapolation

A

Meta analysis and interpolation/extrapolation

Likely to be more values, and a wider range of values in the studies collected for a meta-analysis than in any single study

60
Q

Meta-analysis and causal explanation

A

Meta analysis and causal explanation

  • More information available from diverse studies to help elucidate relevant parts of causal effect and mediators
  • We’re looking for the process, mediators, etc. –> what explains how this treatment’s outcomes?

Applied Research Methods - Shared (2 decks)

Brainscape helps you reach your goals faster, through stronger study habits.
© 2024 Bold Learning Solutions. Terms and Conditions