Nov. 26th/28th - Chapter 14: Generalizing Results

Question 1

Challenges to generalizing results

Can results generalize to other populations?

Answer 1

• Research is generally conducted on samples rather than entire populations
• However, most psychological research relies on convenience samples - people selected simply because they’re available and willing
• The population most available to university students is university students

Question 2

Can results generalize to other populations?

Beyond university students

Answer 2

Represent a highly restricted population - AKA WEIRD

Question 3

Can results generalize to other populations?

Beyond volunteers

Answer 3

• Research on volunteers has found that those who agree to participate often differ in various ways from non-volunteers
• Different kinds of people volunteer for different kinds of things
• People participate based on financial compensation being offered

Question 4

Can results generalize to other populations?

Beyond gender

Answer 4

• Sometimes, researchers use either females or males, or only one gender
• Simply because this is convenient or better suited to one gender
• Individuals who identify as a gender minority might be excluded from analysis
• Gender-based findings might not generalize: If you wish to generalize, it’s necessary to replicate the study with participants of the gender in question
• Gender bias influencing the research process

Question 5

Can results generalize to other populations?

Beyond culture

Answer 5

• Somewhat recently, participants in most studies were North American university students who were primarily white
• But now, majorities have changed
• Much of cultural research has centered on identifying cross-cultural similarities and differences in responses to the same environments, along with personality and other characteristics
  Operational definitions of the constructs we study are grounded in particular cultural meanings

Question 6

Can results generalize beyond the specific study?

Beyond the experimenter

Answer 6

• The person who actually conducts the experiment can trigger another generalization problem
• In most studies, to reduce experimenter biases and influences on participants, only one experimenter is used
• As little attention is typically paid to the personal characteristics of experimenters, it’s possible that the results of a study using only one experimenter cannot be generalized to other types of experimenters

Question 7

Beyond the experimenter - characteristics:

Answer 7

Some important characteristics include:
* Gender
* Personality
* Amount of practice in the role of an experimenter

Question 8

Beyond the experimenter - 2 solutions:

Answer 8

• One solution is to use two or more experimenters with differing characteristics
• Or deliver instructions using a computer, which minimizes the amount of interaction between experimenters and participants

Question 9

Can results generalize beyond the specific study?

Beyond a pretest

Answer 9

• May limit the ability to generalize any results to populations that do not receive a pretest
• In reality, people are rarely given a pretest
• What can be done when a pretest is desired, but concern that it will affect later responses?
• Solomon four-group design: same experiment is conducted with/without the pretest; researcher can then examine whether there is an interaction between the IV and pretest variable

Question 10

Can results generalize beyond the specific study?

Beyond the laboratory:

Answer 10

• ** In experiments, the goal of high internal validity may sometimes conflict with the goal of external validity**
• Does the artificiality of the lab setting limit the ability to generalize results to real-life settings?
• One way to examine phenomena under more realistic circumstances: field experiments - thus increase external validity
• Comparing the results of lab and field settings can create controversy - can lab-based findings be generalized to real events?
• Do lab or field experiments that examine the same variables generally produce the same results? Do they differ in their estimation of the effect-size for the phenomenon?
• AFTER RESEARCH: while lab studies and field studies do tend to find similar effects, there is substantial variability in the degree to which this is true - based on the field of psychology and topic of study

Question 11

Solution to generalizing results:

Answer 11

• replicate the study: replication is an important way to overcome some of the questions of generalization that stem from the results of a single study

Question 12

2 types of replication

Answer 12

1. Direct
2. Indirect

Question 13

Direct Replication:

Answer 13

• Direct replication: an attempt to replicate the procedures of a study as closely as possible to see whether the same results are obtained
• Especially when results from initial studies are unexpected or based on small samples
• Crucial to determining whether an original finding can generalize to other samples drawn from the same population
• Can be done when embarking down a new area of research

Question 14

What happens when direct replication fails?

Answer 14

• This does not indicate that the original phenomenon doesn’t exist
• Failures to replicate share the same difficulties of interpretation as statistically non-significant results
• Failure to replicate meaning…
  ® OG results are invalid
  ® Replication attempt was flawed
• A well-designed replication attempt with a large sample that fails to detect an effect should also be considered evidence that the effect may not truly exist
• In some cases, only occurring under certain, limited circumstances
• In other cases, may turn out that the OG researcher misrepresented the results in some way, or engaged in questionable research practices in order to produce false evidence for a phenomenon

Question 15

Direct replication and disciplinary reform:

Answer 15

Many complex q’s have been raised about the way research in psychology and other disciplines is conducted

Consider:
* Whose responsibility is it to replicate results?
* To what extent can we trust past literature that no one has attempted to replicate?
* What is the best way to replicate other work?
* Which kinds of replications are most important?

Historically, have been difficult to publish
* Recent actions from journal editors involve accepting particular types of replication studies after adequate peer review
* Researchers developing practices to help ensure all direct replications are high quality and objective

Recommendations for convincing replication attempts:
* Ensuring high statistical power
* Following the OG procedures as closely as possible
* Making details public for verification

Become common for researchers from labs to collaborate in order to conduct simultaneous direct replications
* Not only does a multi-lab approach add to external validity, but promotes objectivity as researchers are less likely to have personal investment in the outcomes

Question 16

Conceptual Replication

Answer 16

The use of different procedures to replicate a research finding
* In conceptual replication, the IV is manipulated in a different way and/or the DV is measured in a different way
* A relationship that appears with one set of operationalizations should generalize to different ways of manipulating and measuring the same variables
* Examples:
- Alternative stimulus
- Alternative dependent measure
* When successful, conceptual replications of a case can prove that the relationship between variables generalizes beyond the original operationalizations

Question 17

What can conceptual replications tell us?

Answer 17

• Spark an out of the box interpretation of research findings: that they represent relationships between conceptual variables but are grounded in specific operationalizations
• A conceptual replication would demonstrate that the relationship between the conceptual, theoretical variables is present
• Can help develop theories of behaviour, but not a substitute for direct replications

Question 18

Conceptul Replication - Issue:

Answer 18

One issue of conceptual replications being the potential to promote Type I errors
* If a conceptual replication fails to find an effect, it is possible to toss that study aside as methodology problematic, and continue trying different operationalizations until the OG effect is conceptually replicated
* Thus, for greater confidence in the generalizeability of an effect, consider conducting direct replication first, to ensure that the OG results generalize beyond the original sample
* Then carefully substitute alternative operationalizations in a conceptual replication

Question 19

Consider different populations…by

Answer 19

Recognizing the limits of convenience samples and seek diverse samples
* Doesn’t mean you must scrap data, but we can question whether our results would generalize beyond this specific population, and seek broader samples whenever possible

Think about how results might relate/differ if a different population were investigated
* Rather than assuming that processes operate universally, actively consider how different populations might think or act differently; then use factorial designs to test those hypotheses and adapt the theory accordingly

However, it’s not always possible to collect data from diverse sources
* The internet makes it easily accessible
* Can recruit through social platforms, which tend to harvest samples of greater diversity with respect to socio-economic status, ethnicity, age, and work experience

Question 20

Examine the influence of group membership using factorial designs:

Answer 20

• Factorial designs allow us to study interactions, with an interaction occurring when an effect exists under one condition but not another
• Can be leveraged to study generalizability across groups directly
• EX: study examining relationship between crowding and aggression, based on rural participants; you question whether these results are generalizable to those who were raised in an urban setting, within a crowded city
• While the main effect of rural geography is present, it doesn’t prevent the generalization because the overall positive relationship between crowing and aggression is both present for urban/rural participants

Question 21

Rely on multiple studies to draw conclusions: literature reviews

Answer 21

In a literature review, researchers typically review a summary of existing results, indicating what findings are strongly or weakly supported in the literature and more analyses

Question 22

Rely on multiple studies to draw conclusions: meta-analysis

Answer 22

• A more prevalent approach to review paper is to use statistical techniques to combine the results from past studies, AKA meta-analysis
• In such, information from each past study is gathered and considered statistically, including effect-sizes, sample sizes, and other features of the studies that might influence the effect size for the outcome
• Compares the strength of a given finding across many different studies that tested the same or similar variables, in order to examine the best estimate of the effect size for a given effect

Question 23

Meta-analysis vs. literature review?

Answer 23

Meta-analysis:
* A single conclusion is evident, in the form of an estimation of the effect-size observed across many past studies
* Uses a quantitative approach, analyzing the data with statistics, to arrive at an estimate that integrates all this past information
* Can only summarize and integrate results from past quantitative studies; excluding qualitative
* By pooling results, can provide a more informative estimate of population values compared to relying on a single study

Question 24

Meta-analysis vs. literature review?

Answer 24

Lit review:
* Difficult to arrive at a clear conclusion
* Qualitative - helps identify trends in literature and directions for future study