Chapter 14: Replication, Transparency, & Real-World Importance

Question 1

What makes a study important?

Answer 1

1. To be important, a study must be replicated.
2. Does it have to be generalizable?
   a. Generalization mode? Then yes
   b. Theory-testing mode? Then no
3. Does it have to have real world applicability? It’s complicated . . .

Question 2

Replicable

Answer 2

Describing a study whose results have been reproduced when the study was repeated, or replicated. Gives a study credibility.

Question 3

Direct replication

Answer 3

A replication study in which researchers repeat the original study as closely as possible to see whether the original effect shows up in the newly collected dat.a

Question 4

Conceptual replication

Answer 4

A replication study in which researchers examine the same research question (the same conceptual variables) but use different procedures for operationalizing variables.

Question 5

Replication-plus-extension

Answer 5

A replication study in which researchers replicate their original study but add variables or conditions that test additional questions.

Question 6

Replication crisis

Answer 6

Journals prefer to publish new research and not
replications. So, replications are rarely published (particularly direct replications).
A large group of psychologists (Open Science
Collaboration) selected 100 studies from 3 major
psychology journals to attempt to replicate. Only 39% were success replications.

Question 7

Improvements to Scientific Practice

Answer 7

1. Larger sample sizes
2. Report all analyses and variables
3. Open Science Collaboration
4. Preregistration

Question 8

Meta-analysis

Answer 8

A way of mathematically averaging the effect sizes of all the studies that have tested the same variables to see what conclusion that whole body of evidence supports.

Question 9

File drawer problem

Answer 9

A problem relating to literature reviews and meta-analyses based only on published literature, which might overestimate the support for a theory because studies finding null effects are less likely to be published than studies finding significant results, and are thus less likely to be included in such reviews.

Question 10

Ecological validity

Answer 10

The extent to which the tasks in manipulations of a study are similar to real world context; an aspect of external validity. The focus is on whether a laboratory study
generalizes to real-world settings.

Question 11

Theory-testing mode

Answer 11

A researcher’s intent for a study, testing association claims or causal claims to investigate support for theory. External validity is not the priority!

Question 12

Generalization mode

Answer 12

The intent of researchers to generalize the findings from the samples and procedures in their study to other populations or contexts. External validity is
essential!

Question 13

Which mode?

Answer 13

Frequency claims: always in generalization mode

Association and causal claims: sometimes in generalization mode, more often theory-testing.

Question 14

Field-setting

Answer 14

A real-world setting for a research study. Has high external validity, but doesn’t mean the results would
generalize to all people, in all settings. Plus, researchers can create real situations in the laboratory
(experimental realism).

Question 15

Experimental realism

Answer 15

The extent to which a laboratory experiment is designed so that participants experience authentic emotions, motivations, and behaviors.

Question 16

Scientific literature

Answer 16

A series of related studies, conducted by various researchers, that have tested similar variables.

Question 17

HARKing

Answer 17

A questionable research practice in which researchers create an after-the-fact hypothesis about an unexpected research result, making it appear as if they predicted it all along. Stands for “hypothesizing after the results are known.”

Question 18

p-hacking

Answer 18

A family of questionable data analysis techniques, such as adding participants after the results are initially analyzed, looking for outliers, or trying new analyses in order to obtain a p-value of just under .05, which can lead to nonreplicable results. The practice of p-hacking is misleading when others are not told about all the different ways the data were analyzed and only the strongest version is reported.

Question 19

Underreporting null effects

Answer 19

Researchers mislead about the strength of the evidence by not reporting conditions or measures that did not support the hypothesis.

Question 20

Using small samples

Answer 20

In a small sample, a few chance values can influence the data set, so the study’s estimate is imprecise and less replicable.

Question 21

Open science

Answer 21

The practice of sharing one’s data, hypotheses, and materials freely so others can collaborate, use and verify the results.

Question 22

Open data

Answer 22

When psychologists provide their full data set on the Internet so other researchers can reproduce the statistical results or even conduct new analyses on it.

Question 23

Open materials

Answer 23

When psychologists provide their study’s full set of measures and manipulations on the Internet so others can see the full design or conduct replication studies.

Question 24

Pre-registration

Answer 24

A term referring to a study in which, before collecting any data, the researcher has stated publicly what the outcome is expected to be.

Question 25

Cultural psychology

Answer 25

A subdiscipline of psychology concerned with how cultural settings shape a person’s thoughts, feelings, and behavior, and how these in turn shape cultural settings.