Class 8: Research methods in cognition, levels of analysis, generalisability, control, and open science

Question 1

What was the finding in Dasgupta and Greenwald’s study that Brian Nosek spoke of?

Answer 1

The study was investigating whether exposing people to positive black individuals would reduce their implicit race bias – measured by IAT. Joy-Gaba and Nosek believed that it was likely associative between ‘good’ and ‘black’ exemplars and acted to reduce implicit race bias. They did not get the original result with replication due to failure of replicating an important moderator.

Question 2

The________ is the process of conducting a study again, with the goal of verifying the original findings.

Answer 2

Replication

Question 3

Is there is persuasive evidence that individuals know what things are more likely to replicate than others?

Answer 3

Yes. Nosek highlighted in his talk ‘What is Replication’ that there were prediction market priced bets placed based on probability of replication with the results indicating that individuals have some insight in likelihood of replication. Some prediction of probability of replication is said to help sharpen our theoretical expectations.

Question 4

Mook (1983) says that the representativeness of a sample is a) always the most important factor in designing an experiment, b) especially important if the research is not in survey form, or c) depends on the purpose of the research

Answer 4

c) He says the representativeness of the sample is more or less important depending on the purpose of the research. For instance if it is a survey, it is very important, if it is a case study then it may be less so.

Question 5

Fill in the blank: “External invalidity can occur when the specific conditions of the __________ do not match those of the real world.”

Answer 5

Study

Question 6

A common definition for replication is “repeating a study’s procedure and observing whether the prior results recur.” Nosek defines replication as “an attempt to reproduce a previously observed finding with no a priori reason to expect a different outcome”. What is the key difference in these definitions?

Answer 6

In Nosek’s definition, nothing about procedural similarity is included. His definition is centred around theoretical expectations as he suggests that claiming that something is a replication is itself a theoretical commitment.

Question 7

Why should we not discard all research due to external validity?

Answer 7

Misplaced preoccupation with external validity can result in dismissing good research for which generalisation to real life was not intended or meaningful.

Question 8

True/False Argyle dismissed Milgram’s work on obedience as absurd and unethical

Answer 8

FALSE

Question 9

According to Brian Nosek, replication is not necessary for scientific progress because it simply confirms what we already know.

Answer 9

False. According to Brian Nosek, replication is critical for scientific progress because it helps to establish the reliability and validity of research findings.

Question 10

Define the term “External Validity”.

Answer 10

External Validity - The extent to which the findings of a study can be generalized to other populations, settings, and times beyond the specific context of the study.

Question 11

True or False: Parents react to the form of a child’s speech, not its content.

Answer 11

Answer: False Explanation: Direct observation of parent-child interactions reveals that parents react to the content of a child’s speech, not to its form.

Question 12

What does Mook (1983) suggest as a more appropriate criterion for evaluating psychological research?

Answer 12

Instead of focusing on external validity, Mook (1983) proposes that researchers should emphasize internal validity and the contribution of their work to the understanding of psychological processes and theories. He believes that well-designed studies with strong internal validity can provide valuable insights into human behavior, even if they do not directly generalize to all populations or situations. By building a robust body of evidence through numerous studies, researchers can gradually enhance the external validity of their findings.

Question 13

How is ‘replication’ defined in Brian Nosek’s talk?

Answer 13

Repeating a study’s procedure and observing whether the prior finding recurs.

Question 14

What does Brian Nosek propose the revised definition of replication is?

Answer 14

Attempting to reproduce a previously observed finding with no a priori reason to expect a different outcome.

Question 15

The purpose of replication is to increase the reliability of research findings.

Answer 15

TRUE

Question 16

True or False: The concept of external validity was developed with a specific type of research in mind, namely experiments designed to evaluate the effectiveness of new teaching procedures or enrichment programs on culturally deprived populations.

Answer 16

TRUE

Question 17

Instead of predicting the real world from lab experiments and results, psychologists are instead predicting …?

Answer 17

Psychologists are testing what occurs in labs, then refining their theory to test if it will occur in lab settings

Question 18

What is the revised definition of Replication? (As mentioned in the video)

Answer 18

It is the process of attempting to reproduce a previously observed finding with no a priori reason to expect a different outcome.

Question 19

How can we try to reduce the impact of false negatives (or the likelihood)?

Answer 19

We can minimise false negatives by obtaining original materials.

Question 20

What are the reasons provided by Nosek for why procedural similarity is a pragmatic solution currently in replication research?

Answer 20

’- We lack theoretical specificity - There is a lack of understanding of the methods of studies and what modifications will produce differing results

Question 21

How can a study’s external validity be enhanced?

Answer 21

A study’s external validity can be enhanced by conducting the study in multiple settings, recruiting a diverse sample of participants, and varying the temporal characteristics of the study.

Question 22

What are some of the benefits of open science practices, as advocated by Nosek?

Answer 22

Some benefits include increased transparency in research, improved collaboration, and greater reproducibility of findings.

Question 23

When building evidence about theoretical expectations, generalisability is where we have low/high commitment. When committing to something as a replication, we have low/high commitment.

Answer 23

When building evidence about theoretical expectations, generalisability is where we have LOW commitment. When committing to something as a replication, we have HIGH commitment.

Question 24

What is external validity according to Mook’s article “In defense of external invalidity”?

Answer 24

Answer: External validity refers to the extent to which research findings can be generalized to real-world settings and populations beyond the sample used in the study.

Question 25

Mook (1983) suggests that external validity is not important in experimental research, as long as the findings are internally valid.

Answer 25

Mook (1983) argues that external validity is important in experimental research, and that researchers should aim to balance both internal and external validity in their studies. Mook states that sacrificing external validity for internal validity can limit the usefulness of findings in real-world settings, and suggests that researchers use a variety of research methods to increase the generalizability of their findings.

Question 26

What is external invalidity?

Answer 26

External invalidity refers to the inability to generalise the results of a study to other populations, settings, or time periods.

Question 27

True or False: Experiments that were conducted in real-life settings will still have some limits in their generalizability.

Answer 27

TRUE

Question 28

True or False? Justifications for limited external validity may include experimental control, repeatability, and the need for specific research contexts.

Answer 28

TRUE

Question 29

According to Brian Nosek, should replication be framed in conceptual or procedural terms?

Answer 29

Conceptual terms

Question 30

What does Mook state claim is the biggest source of confusion regarding the debate around External Validity?

Answer 30

The purpose of collecting data in the laboratory is to pred t real-life behaviour in the real world–this is not always the case.

Question 31

What did the original discussion of EV by Campbell and Stanley(1967) reveals?

Answer 31

They reveal that the experimental investigation they had in mind was a rather complex mixed design with pretests, a treatment and a posttest

Question 32

Which of the following statements best summarizes the main argument of Mook’s article “In defense of external invalidity”? A) External validity is a minor concern in research and can be overlooked in favor of internal validity. B) The concept of external validity is flawed and should be abandoned altogether. C) External validity is crucial for the practical application of research findings and should be given more attention. D) Researchers should prioritize the use of laboratory experiments over field studies to maximize external validity.

Answer 32

Answer: C) External validity is crucial for the practical application of research findings and should be given more attention.

Question 33

In what kind of research is representativeness important, according to Mook?

Answer 33

Survey research

Question 34

What must be considered in the checklist of decisions, and what does Mook mean when he talks about the specifics of the sample and setting?

Answer 34

(A) We Must consider what conclusion wants to be drawn and (B) whether the specifics of our sample or setting will prevent us from drawing it. Sample > Are you trying to (A) estimate sample characteristics to those of a particular population, or (B) trying draw conclusions not about a population but rather a theory that specifies what these subjects ought to do, or ( C ) if any subject does or can be made to do this or that Setting > is the intention to predict (A) what would happen in a real-life setting or (B) “target” class of such settings

Question 35

What are some strategies proposed by Mook (1983) to enhance external validity in research studies?

Answer 35

Mook suggests several strategies to enhance external validity, including the use of more realistic and diverse samples, the replication of studies across different settings and populations, and the examination of the generalizability of findings through meta-analyses and other methods. He also emphasizes the importance of reporting effect sizes and statistical power to evaluate the practical significance of study findings.

Question 36

What are some of the challenges associated with replicating studies in psychology?

Answer 36

Some of the challenges include differences in sample characteristics, variations in experimental procedures, and variations in measurement instruments.

Question 37

True or False: According to the study by Brown and Hanlon (1970), parents react to the content of a child’s speech rather than its form?

Answer 37

True.

Question 38

Ecological Validity

Answer 38

The realism with which a design of evaluation setup matches the user’s real work context; how accurately the design or experiment reflects the relevant characteristics of the ecology of interaction

Question 39

True or False: According to Mook (1983), context-specific research is less valuable than research that can be generalized to multiple settings and populations.

Answer 39

Answer: False Explanation: Mook argues that context-specific research can provide valuable insights into specific situations or populations. Rather than aiming for generalizability, these studies seek to gain a deeper understanding of a particular context. In some cases, Mook suggests that external invalidity can even be a strength, as it allows researchers to uncover unique insights that may not emerge from more generalizable studies.

Question 40

Mook argues that one origin of the misconceptions about EV, its goals and why we should absolutely reach it lies in the “feeling” that comes with words such as “validity” and “invalidity” a priori. True or False?

Answer 40

True. Cf “Warm Fuzzies and Cold Creepies” paragraph

Question 41

Why are some areas of psychology further ahead at addressing the issues surrounding replication compared to others?

Answer 41

Areas such as social psychology are further ahead as experiments are easier to replicate. Areas such as infant or clinical psychology are further behind because it is harder to replicate studies.

Question 42

What are the two reasons highlighted in the youtube video that experiments are conducted through procedural similarity rather than altering the procedure but ensuring the expected result is the same.

Answer 42

Lack of specificity Lack of understanding the methods

Question 43

True/False The assumption that laboratory settings should resemble real-life ones as much as possible is true.

Answer 43

FALSE

Question 44

Three questions to consider when interpreting a failure to replicate

Answer 44

is the original study a false positive, is the replication a false negative, are both true effects.

Question 45

Fill in the blank: Mook argues in his article, In Defense of External Invalidity, that a misplaced preoccupation with external ________ can lead us to dismiss good research for which generalization to real life is not intended or meaningful.

Answer 45

Validity

Question 46

Why are conclusions that cannot be generalized important?

Answer 46

They can contribute to the understanding of the process. It can reveal what can occur, not necessarily what happens in real-life conditions.

Question 47

Brian Nosek believes that all scientific studies should be replicated. True or False?

Answer 47

False : Brian Nosek believes that not all studies need to be replicated, and that replication efforts should be focused on studies that are particularly important, influential, or controversial.

Question 48

What are some potential limitations of using college students as research participants and how might these limitations affect the external validity of research findings?

Answer 48

College students may not be representative of the broader population and may limit the external validity of research findings. They are often more educated, younger, and more diverse than the general population, and they may have different life experiences and attitudes that affect their behaviour. This can limit the generalizability of research findings beyond the college student population.

Question 49

What is grandiose narcissism?

Answer 49

A subtype of narcissism characterized by an inflated sense of self-importance, a need for admiration, and a lack of empathy for others.

Question 50

In discussing external validity, Mook (1983) says that in experiments we are not ‘making generalisations’, but rather we are a) examining them b) testing them c) appraising them d) evaluating them

Answer 50

b) testing them

Question 51

How can experiments which lack EV (and also provide seemingly silly results) still provide value?

Answer 51

Findings, no matter its apparent value, can contribute towards an underlying theory of the world

Question 52

According to Mook 1983, external validity “asks a question and invites us to think of prior questions”, what is external validity NOT?

Answer 52

an automatic desideratum

Question 53

Fill in the blank: Every replication, because they’re never identical, is a test of _.

Answer 53

Generalizability

Question 54

What were cambell and Stanley (1967) aims

Answer 54

To evaluate the effect of an enrichment program for the culturally deprived

Question 55

Why is external invalidity sometimes considered to be less important than internal validity?

Answer 55

External invalidity is often considered less important than internal validity because it does not threaten the causal conclusions drawn from the study.

Question 56

To what kind of experiment does Mook (1983) acknowledge the importance of the external validity problem?

Answer 56

Applied experiments.

Question 57

Brian Nosek argues that procedural changes are necessary in replication studies so the definition of replication should focus on conceptual terms. How is this justified?

Answer 57

By conceptualising replication in the realm of theoretical expectations, changing what procedural differences are necessary with no prior expectation that such changes will produce a different result. Relies on researchers theoretical understanding of the concept being tested.

Question 58

What are three ways to minimise false negatives in studies?

Answer 58

Obtain original materials. Use high-powered designs. Peer review in advance.

Question 59

What is external invalidity, as defined by D.G. Mook in his article “In Defense of External Invalidity”?

Answer 59

External invalidity refers to the extent to which research findings can be generalized to real-world settings and situations outside of the laboratory.

Question 60

Why does Mook suggest that researchers should prioritise external validity?

Answer 60

To make research more relevant and applicable to real-world situations

Question 61

True or False: The goal of replication is to copy the methodology and procedure of a previous study and observe if we get the same results.

Answer 61

False. The goal is actually to attempt to reproduce the finding of a previous study with no a priori reason to expect a different result. I.e., with this new sample/context/setting, these are the adjustments I need to make to the procedure, in order to have no reason to think that it will produce a different result.

Question 62

Why is replication important in science?

Answer 62

Replication is important in science because it helps to confirm the results of studies and to build confidence in scientific findings.

Question 63

What is external validity?

Answer 63

External validity is the extent to which our findings can be generalised in other populations, settings, variables, etc.

Question 64

According the Brian Nosek, every replication, because they are not identical, is a test of ___?

Answer 64

Generalisability

Question 65

Why does the experiment by Harlow raised a threat of external validity?

Answer 65

One of the reasons is because the sample used was not deemed as representative of the population. In this case, Harlow’s baby monkeys were not representative of the population of monkeys in general as they were born in captivity and orphaned.

Question 66

Why is there a focus on Procedural Similarity?

Answer 66

There is a lack of Theoretical Specificity There is a lack of understanding the Methodology. Having the same procedure is a substitute for Theoretical & Methodological Understanding

Question 67

How does Brian Nosek view Replication?

Answer 67

Brain Nosek views Replication as a conceptual understanding. Replication provides us with an opportunity to confront current theoretical understandings of how we obtain research findings.

Question 68

Why is there a current focus on procedural similarity in psychological research?

Answer 68

A lack of theoretical specificity and of understanding of methods. Using procedural replication (rather than conceptual) is a substitute for theoretical and methodological understanding.

Question 69

Multiple Choice: According to the study by Brown and Hanlon (1970), what do parents react to in a child’s speech? a) Its form b) Its content c) Its grammar d) Its tone

Answer 69

Answer: b) Its content Explanation: Direct observation of parent-child interactions reveals that parents react to the content of a child’s speech, not to its form.

Question 70

According to Mook (1983), what is the importance of external validity in research?

Answer 70

External validity ensures that research findings have practical implications and can be applied to real-world situations.

Question 71

When considering internal and external validity, what does Mook suggest should be the sacrificed one if required?

Answer 71

Sacrifice internal validity

Question 72

What are the three ways Brian Nosek described to interpret a failure to replicate?

Answer 72

The three ways are (1) the original finding was a false positive, (2) the replication is a false negative, or (3) neither, that there are important substantive differences that explain the different results

Question 73

What is usually lacking when someone believes replication can only occur if it has the exact procedure of what it is replicating?

Answer 73

This often demonstrated a lack of theoretical understanding, and shortcomings in understanding the methods of the original study.

Question 74

Criticisms of external validity may include issues such as sample characteristics, ecological validity, and ________ of results.

Answer 74

generalization

Question 75

true/false Theoretical specificity doesn’t enable effective replication

Answer 75

false

Question 76

What are the two things that Mook suggests can lead to the compromise of a study when focusing on external validity?

Answer 76

Rigor and control

Question 77

What is the “agricultural” concept of the aims of research?

Answer 77

The assumption that the experiment is intended to be generalised to similar subjects, manipulations, and settings.

Question 78

Fill in the Blank: According to the article, the fact that eyeglasses can influence our judgments of intelligence says something about us as __________.

Answer 78

Answer: judges Explanation: The fact that eyeglasses can influence our judgments of intelligence, though it may be quite devoid of real-world application, surely says something about us as judges.

Question 79

True or False: Harlow concluded that wild monkeys in the jungle would choose terry-cloth over wire mothers.

Answer 79

False: Harlow did not generalise his findings of an artificial experiment involving monkeys born in captivity to monkeys in the wild. Harlow actually concluded that the hunger-reduction interpretation of mother love would not work

Question 80

What is the difference between statistical generalization and analytical generalization, and how do they relate to external validity?

Answer 80

Statistical generalization refers to the use of probability and sampling techniques to make inferences about a population, while analytical generalization involves the transfer of theoretical concepts and propositions to different contexts, and both are important for achieving external validity.

Question 81

True or False: According to Mook (1983), experiments that lack external validity are worthless.

Answer 81

FALSE

Question 82

How is generalisability explained in Mook’s (1983) research article?

Answer 82

To what populations, settings, treatment variables, and measurement variables can this effect be generalized.

Question 83

Why is it that claiming that a study is a replication is a theoretical claim?

Answer 83

No two studies can be identical; Even if the same sample, environment and context were the same, the conditions have changed.

Question 84

What is the main argument that Mook (1983) makes in his article “In Defense of External Invalidity”?

Answer 84

Mook (1983) argues that external invalidity is not necessarily a fundamental flaw in psychological research, as it is often criticized. He contends that the focus should be on the internal validity and theoretical implications of a study, rather than its direct generalizability to other populations or situations.

Question 85

TRUE/FALSE: An individual study with enough power, theoretical strength, and significant findings can act as conclusive evidence for or against a theoretical concept.

Answer 85

FALSE. A key part of Nosek’s definition of replication is that no individual study is ever conclusive evidence for or against our theoretical understanding. Evidence is accumulated slowly across many studies in order to get an understanding of a phenomenon.

Question 86

What was the goal of the Many Labs 2 replication work described by Nosek?

Answer 86

To evaluate the current assumption in much of research that when we don’t get the same finding during replications it is due to changes in the sample, setting, or context, and whether we do or should use this as a default explanation.

Question 87

What are the two types of validity that Mook discusses in his article?

Answer 87

Mook discusses internal and external validity. Internal validity refers to the extent to which an experiment’s results are due to the manipulation of the independent variable, whereas external validity refers to the generalizability of the experimental results to other situations or populations.

Question 88

In “In Defense of External Invalidity” by Douglas G. Mook, what were some benefits for the use of artificial and controlled laboratory conditions in research.

Answer 88

Some benefits include: - Showed what can occur, rather than what typically does happen - could illustrate the power of a phenomenon by demonstrating that it even happens under unnatural conditions that prevent it from happening - control and produce conditions that does not happen in real life

Question 89

True/False Milgram’s work on obedience was conducted in a laboratory setting.

Answer 89

TRUE

Question 90

True or False: According to Argyle (1969), wearing glasses has a significant effect on judgments of intelligence, regardless of the duration of exposure to the stimulus person.

Answer 90

False. According to Argyle, wearing glasses has a significant effect on judgments of intelligence when the stimulus person is seen for 15 seconds, but has no effect when seen for 5 minutes of conversation.

Question 91

True or false. Based on the video, Brian Nosek claims that there is no such thing as an exact replication because every time we do a replication its in a new context.

Answer 91

True.

Question 92

According to Brian Nosek, the process of __________ is a critical part of scientific research that involves reproducing findings from previous studies.

Answer 92

Replication

Question 93

What is Brian Nosek’s definition of replication?

Answer 93

Attempting to reproduce a previous observed finding with no a priori reason to expect a different outcome.

Question 94

What are three productive ways that one can reduce the chances of a false negative according to Brian Nosek?

Answer 94

1. Obtain original materials 2. High powered designs 3. Peer review to occur in advance

Question 95

The validity of these generalizations is tested by their success at prediction which has nothing to do with 1,________ 2,________3,_________

Answer 95

1, naturalness, 2, representativeness, 3, nonreactivity

Question 96

The external validity question asks, in short: are the sample, the setting, and the manipulation of a study so artificial that the class of “target” real-life situations to which the results can be generalized is likely to be trivially small? When the answer is yes, what is Mook’s general reasoning behind why, sometimes, this same research can still be considered well-done, meaningful, or informative?

Answer 96

He argues that generalisation to real life is sometimes not intended or meaningful. In these cases, a study’s external validity shouldn’t lead to the research being dismissed altogether.

Question 97

When the setting and manipulation of variables are too artificial in an experiment, what does this impact?

Answer 97

External Validity

Question 98

Why are experimental settings made artificial in the words of Mook?

Answer 98

To control for extraneous variables and to permit separation of factors that do not come separately in Nature-as-you-find-it.

Question 99

What is Mook’s Checklist of decisions?

Answer 99

What conclusion do we want to draw? Does our sample’s specifics prevent us from drawing it?

Question 100

What is external validity?

Answer 100

The extent to which research findings can be generalized to real-world settings and populations.

Question 101

Define external invalidity

Answer 101

The extent of how well the conclusion of the study can be generalised to other contexts

Question 102

What are the levels of analysis in psychology?

Answer 102

The levels of analysis refer to different perspectives from which psychologists can examine behaviour and mental processes, including biological, cognitive, social, and cultural.

Question 103

Because every replication is never identical, it is a test of generalisability. True or false?

Answer 103

True.

Question 104

True or False: Brian Nosek used the example of Schnabel and Nadler’s (2008) study and how it was revised for an American sample to argue for the original definition of holding the procedure constant when it comes to replication.

Answer 104

FALSE: In fact, this article was used by Brian Nosek to argue for a new definition regarding replication as he noted that holding the procedure constant would mean keeping the study replicated in the same language and context as the original study for an Israelian sample, which wouldn’t be applicable to an American sample; he argues that replication is in fact concerned with conceptual matters over procedural matters.

Question 105

Which of the following is NOT a way to minimise false negatives: (A) Obtain original materials (B) Change the original procedure to fit the new context of the replication (C) Peer review in advance (D) Use high-powered designed

Answer 105

(B) is not a way to minimise false negatives. Although procedural changes may be necessary for a replication, it does not minimise false negatives.

Question 106

True or False: Mook (1983) argues that external validity should always be prioritized over internal validity in psychological research.

Answer 106

Answer: False Explanation: Mook contends that the importance of external validity has been overemphasized in psychological research. He argues that internal validity should often be prioritized, as establishing causal relationships is crucial for scientific progress. Mook also suggests that the relevance of external validity depends on the type of research being conducted, with applied research benefiting more from generalizability than basic research.

Question 107

What are the three major conclusions that may be drawn by a failure to replicate according to Brian Nosek?

Answer 107

1. The original is a false positive 2. The replication is a false negative 3. Neither of the findings are true (violation of expectations)

Question 108

How did Mook use what was happening ‘in the sample’ to make predictions about a theoretical proposition?

Answer 108

Although Mook understood that sample bias gave rise to issues of generalisability. By considering the theoretical proposition ‘parents respond to the grammar of their children’s utterances (either through approval or correcting mistakes). They can predict that it ought to be observed that parents do this. However, as evidence showed parents reacted to content rather than grammar thus disconfirming the prediction. It was then taken further to show that these children went on to acquire the approved syntax even if these parents did not react to grammaticality, which disconfirmed the theory further illustrating that this theory was analysed by looking at what happened in the sample rather than trying to generalise from the sample to the population.

Question 109

What are some of the causes of narcissism discussed in the video?

Answer 109

Parental overvaluation, neglect or abuse, cultural values, and individual factors such as temperament and genetics.

Question 110

What are 2 reasons that many psychology experiments are accused of “failure to generalize to the real world”?

Answer 110

Because they have sample bias or artificiality of setting

Question 111

What is the reason that Brian Nosek justifies why we should think of replication as conceptual replication, not procedural replication?

Answer 111

Procedural replication, properly done, would just be the same study. We are not able to replicate the same procedure, with the same participants, at the same time, in the same context as the original study. Therefore, we have to consider replication as conceptual, because the theoretical concept is what we are attempting to replicate, not the procedure.

Question 112

True or False? Representativeness of sample is important for certain purposes, but for other purposes it is a trivial issue. Therefore, one must evaluate the sampling procedure in light of the purpose—separately, case by case.

Answer 112

TRUE

Question 113

True or false: Mook (1983) mentions that a possible cause for the confusion about EV regarding sampling bias stems from survey researchers.

Answer 113

True. For most survey researchers, one main goal is to generalize the sample findings to the population.

Question 114

What is the main argument presented by Mook in his article “In Defense of External Invalidity”?

Answer 114

Mook argues that external invalidity, or the limited generalizability of experimental results to other situations or populations, is not necessarily a problem. He believes that researchers should focus on understanding the underlying processes and mechanisms in a particular context, rather than trying to generalize their findings too broadly.

Question 115

Define External Invalidity?

Answer 115

The extent to which the findings of a study can be generalized to other populations, settings, or contexts beyond the specific conditions of the study.

Question 116

True or False: External validity refers to the extent to which research findings can be generalised to real-world settings and populations outside the research context.

Answer 116

True.

Question 117

According to Brian Nosek, what is replication?

Answer 117

The process of repeating an experiment to confirm its findings.

Question 118

What are some of the reasons why external validity is often neglected in research, according to Mook’s article “In defense of external invalidity”? A) It is difficult to achieve both internal and external validity in the same study. B) Researchers often prioritize theoretical significance over practical applicability. C) The use of convenience samples limits the generalizability of findings. D) All of the above.

Answer 118

Answer: D) All of the above.

Question 119

According to Mook (1983), which of the following is NOT a common way in which researchers sacrifice external validity for the sake of internal validity in experimental research? A) Using highly selected samples that are not representative of the population of interest. B) Using contrived settings and procedures that do not resemble real-world situations. C) Using multiple measures to increase the reliability of the results. D) Using simple and straightforward experimental designs that do not capture the complexity of real-world phenomena.

Answer 119

C) Using multiple measures to increase the reliability of the results. According to Mook (1983), using multiple measures to increase the reliability of the results does not necessarily sacrifice external validity for internal validity. However, using highly selected samples, contrived settings and procedures, and simple experimental designs can limit the generalizability of the findings to real-world settings and populations.

Question 120

what are four reasons someone may do an experiment, if not to predict real-life behaviour?

Answer 120

1. They May be asking whether something can happen, rather than whether it typically does happen. 2. their prediction may specify something that ought to happen in the lab, so they go to a lab to see whether it does. 3. They may want to demonstrate the power of a phenomenon by showing that it happens even under unnatural conditions that ought to preclude it. 4. To produce conditions that have no counterpart in real life at all, so that the concept of “generalising” to the real world” has no meaning

Question 121

According to Mook, why is external validity not a necessary condition for experimental research?

Answer 121

Mook argues that internal validity is more important than external validity because it is necessary to establish a cause-and-effect relationship between the independent and dependent variables.

Question 122

What is the definition of external validity?

Answer 122

External validity refers to the extent to which research findings can be generalized to other populations, settings, and conditions beyond the specific study.

Question 123

What are some potential benefits of developing theories that are broadly applicable and what are some potential challenges in developing such theories?

Answer 123

Broadly applicable theories allow researchers to understand how psychological principles work across different populations, settings, and contexts, which can lead to more effective interventions and policies. However, developing broadly applicable theories can be challenging, as it requires taking into account the complexity and diversity of human behaviour and psychology. It also requires a willingness to challenge assumptions and preconceived notions about human behaviour, and a willingness to incorporate diverse perspectives and experiences into psychological research.

Question 124

What is an interaction effect?

Answer 124

An interaction effect occurs when the effect of one variable on the outcome of an experiment depends on the level of another variable.

Question 125

Why are many psychological investigations accused of “failure to generalise to the real world”?

Answer 125

Because of sample bias or artificiality of setting.

Question 126

According to Brian Nosek’s talk, what is the defiinition of replication?

Answer 126

Repeating a study’s procedure and observing whether the prior finding recurs.

Question 127

According to Mook (1983), what can the pursuit of internal validity lead to?

Answer 127

The pursuit of internal validity can lead researchers to conduct studies that are artificial and disconnected from the real world.

Question 128

In the video “What is Replicability?” by Brian Nosek, what is the revised definition of replication that he provides?

Answer 128

Replication is attempting to reproduce a previously observed finding with no a priori reason to expect a different outcome.

Question 129

According to Mook what are the implications of a stripped-down setting?

Answer 129

Although tasks, variables and settings have no real-world counterparts it allows us to understand real-world phenomena for example how the visual system works. Through the processes dissected in the laboratory, there are found to also operate in the real world, such that the results apply to any setting in which it is relevant. Therefore, showing that the findings do not represent a class of real-world phenomena they define one.

Question 130

What is external validity in psychology research?

Answer 130

External validity refers to the extent to which research findings can be generalized beyond the specific context in which the research was conducted.

Question 131

Why is it impossible to perfectly replicate a study?

Answer 131

You can never exactly replicate the people were who did the original study.

Question 132

TRUE or FALSE: To ensure robustness and relevancy of a persons research, they should strongly focus on external validity.

Answer 132

False, researchers should aim for a balance between internal and external validity.

Question 133

How can researchers balance internal validity and external validity in their study designs?

Answer 133

Researchers can balance the trade-off between internal validity and external validity by using a multi-method approach that combines different types of evidence and research designs, conducting pilot studies to test and refine the methods, reporting effect sizes and confidence intervals, and considering the practical significance of the findings in addition to the statistical significance.

Question 134

Which of the below options is NOT an implication of conceptual replication according to Brian Nosek?

Answer 134

A) Theoretical specificity enables effective replication B) Replication promotes specificity C) Understanding whether a study is a replication can change over time D) Procedural replication Furthers the progress of scientific research.

Question 135

What are some threats to external validity, and how can they be minimized?

Answer 135

Some threats to external validity include selection bias, reactivity or measurement effects, experimenter bias, multiple-treatment interference, and situational specificity. These threats can be minimized by using random sampling, randomized treatment allocation, standardized measures, blinding procedures, replication across different settings, and ecological validity.

Question 136

minimise false negatives by

Answer 136

obtaining original materials, create high power designs, peer review in advance

Question 137

What is applied experiments

Answer 137

Research that is applied to a theory that relates to real life issue with a practical outcome

Question 138

Define generalisability

Answer 138

Generalisability is the extent in which the findings of a study can be applied in different contexts, populations and conditions.

Question 139

Why is it that some experiments that is “clearly applicable to the real world” have limits to its generalisability?

Answer 139

Unknown factors would be present and no single study is able to discover them all.

Question 140

What problem does Mook (1983) think that questioning the external validity of studies can help explore?

Answer 140

We focus too much on whether results found in studies are representative of the subjects/participants that we test, and not enough on whether the environments and how we brought about these findings represent what might be found in the real world

Question 141

True or false: According to Mook (1983), external invalidity is a serious problem in experimental research.

Answer 141

FALSE

Question 142

How does Brian Nosek define replication?

Answer 142

Attempting to reproduce a previously observed finding with no a priori reason to expect a different outcome.

Question 143

What does Brian Nosek believe should be the focus of replication in the context of psychological research?

Answer 143

Replication should be focused on the conceptual (not procedural) recreation of a study to test a “known” physiological phenomenon for its validity.

Question 144

Among all of these Douglas G . Mook’s view about external validit,which one is correct? A,EV is only relevant in certain types of research, specifically applied experiments aimed at evaluating the effectiveness of a particular intervention (e.g., a new teaching method or a fertilizer on crop growth). B,Got a tendency to focus too narrowly on subject representativeness and not enough on representativeness of settings and manipulations in experimental research. C, Not all experiments are intended to be generalized in this way. Some experiments are designed to test specific hypotheses rather than to make broad generalizations. In these cases, EV may not be as relevant a concern. D,ALL of these above

Answer 144

D,ALL of these above

Question 145

Why does Brian Nosek argue that we need replication?

Answer 145

Replication can provide an opportunity to challenge the current theoretical understanding of the way a result is obtained.

Question 146

Fill in the blank: “The generalization of experimental results to the real world depends on __________.”

Answer 146

External validity.

Question 147

What is external validity?

Answer 147

It refers to the extent in which research findings can be generalised/applied to real-world contexts or populations beyond that specific context of the study.

Question 148

Why do an experiment if we are not using our experiments to predict real-life behaviour?

Answer 148

1. Asking the question of CAN something happen rather than does it typically happen 2. Prediction may specify something that ought to happen in the lab, so this is then investigated in the lab 3. Show the power of a phenomenon if it were to happen under unnatural conditions that ought to prevent it 4. To simply contribute to an understanding of the processes going on.

Question 149

According to Mook, External Validity is a concept that should be banned from the scientific jargon since it is impossible to reach. True or False?

Answer 149

False. EV is useful for some applied contexts (agricultural conception)

Question 150

What is external validity?

Answer 150

The extent to which research findings can be generalised to real-world settings

Question 151

What is the main argument presented in “In defense of external invalidity” by Mook (1983)?

Answer 151

Mook argues that external validity is an essential component of scientific research and that it should not be neglected in favor of internal validity. He proposes that external validity should be viewed as a separate research goal and that studies should be designed to maximize both internal and external validity.

Question 152

true or false, The characteristics of the sample or population studied have no impact on external validity.

Answer 152

False. The characteristics of the sample or population studied can have an impact on external validity.

Question 153

What is external validity and why is it important in research?

Answer 153

External validity refers to the extent to which research findings can be generalized to other settings, populations, and situations beyond the one in which the research was conducted. It is important in research because it helps to ensure that the findings have real-world relevance and can be applied to practical situations.

Question 154

What is external invalidity

Answer 154

Validity of applying conclusions of a study to a population (ability to generalise)

Question 155

What is the importance of replication in experimental research?

Answer 155

Replication is important because it helps to establish the generalizability of experimental results and to verify the validity of the findings.

Question 156

true or false, External validity refers to the extent to which the findings of a study can be generalized to other settings or populations beyond the study.

Answer 156

TRUE

Question 157

True or false: Mook (1983) contends that generalizability is not a concern in experimental research.

Answer 157

FALSE

Question 158

True or False? Social processes observed to occur within a laboratory setting accurately reflect natural social settings

Answer 158

FALSE

Question 159

True or False: Mook argues that context plays a negligible role in understanding human behaviour and its relationship to external validity.

Answer 159

False. Mook emphasises the importance of context in understanding human behaviour and the impact it has on the external validity of research findings.

Question 160

What are the three components of experimental investigation as mentioned in the original discussion of external validity by Campbell and Stanley (1971)?

Answer 160

Pretests, treatment imposed/withheld, and posttest.

Question 161

True or False: according to a study by Mook (1983), an experiment conducted by Harlow raised a threat of external validity, in which the sample was not representative of the population of monkeys in general as Harlow’s baby monkeys were born in captivity and orphaned.

Answer 161

TRUE

Question 162

What is the recommended approach for researchers when it comes to prioritizing internal and external validity in their studies?

Answer 162

Researchers should prioritize both internal and external validity in their studies, as they are both important for advancing scientific knowledge and developing practical applications.

Question 163

What is external validity? Is it better to have higher or lower validity and why?

Answer 163

External validity refers to the extent to which the findings of a study can be generalized to other populations, settings, and contexts. It is better to have high external validity to ensure that the findings of a study have real-world implications.

Question 164

T or F: Mook (1983) argues that external validity is often prioritised over internal validity in experimental research.

Answer 164

False. In Mook’s (1983) article he argues that internal validity is often prioritised over external validity in experimental research.

Question 165

True or false: Mook (1983) suggests that researchers should focus on maximizing internal validity at the expense of external validity.

Answer 165

FALSE

Question 166

What are some ways researchers can enhance the external validity of their research?

Answer 166

Researchers can enhance the external validity of their research by using diverse samples that are representative of the population of interest, using real-world stimuli and settings, and using designs that allow for the manipulation of variables in ways that mimic real-world situations. For example, researchers can use field studies, naturalistic observations, and quasi-experimental designs to increase the external validity of their research.

Question 167

Why are many psychological investigations accused of “failure to generalise to the real world”?

Answer 167

The failure to generalise to the real world stems from sample bias or artificiality of the setting.

Question 168

By the study conducted by Higgins and Marlatt (1973) on the tension-reduction view of alcohol consumption. Dring alcohol can absolutely decrease people’s tension after they have been eclectric shocked,is this statement true or wrong?

Answer 168

Wrong

Question 169

True or false? A misplaced preoccupation with external validity can lead us to dismiss good research for which generalization to real life is not intended or meaningful.

Answer 169

TRUE

Question 170

TRUE or FALSE: Harlow’s monkey experiment applies to some criteria of external validity.

Answer 170

FALSE: Harlow’s experiment is deficient or unevaluable on every criterion of external validity

Question 171

According to Mook 1983, for what reason do researchers make experimental settings artificial?

Answer 171

We do it to control for extraneous variables and to permit separation of factors that do not come separately in nature-as-you-find-it.

Question 172

According to Mook (1983), what are 3 limits that can still be found in a study’s generalizability even if it had been conducted and proved applicable in the real world?

Answer 172

Cultural, historical, and age-group limits.

Question 173

Name at least one way (out of three) to reduce the likelihood of false negatives.

Answer 173

You could have said: * Obtaining original materials. * Developing high-powered designs. * Peer reviewing in advance.

Question 174

How does Mook (1983) address the criticism of laboratory experiments’ lack of external validity?

Answer 174

Mook (1983) acknowledges the limitations of laboratory experiments in terms of external validity but argues that it is more important to understand the underlying psychological principles and mechanisms in a controlled environment. He suggests that the primary goal of psychological research should be to develop and test theories, and that generalizability can be addressed through follow-up research and the accumulation of evidence across multiple studies.

Question 175

True or False: Mook believes that in psychological research it is always reasonable to sacrifice external validity for internal validity.

Answer 175

False. Mook questions the idea that it is always reasonable to sacrifice external validity for internal validity and argues for a trade-off between the two.

Question 176

What is external validity?

Answer 176

External validity refers to the extent to which the results of a study can be generalized to other settings, populations, and conditions beyond the specific context of the study.

Question 177

true or false: If one finds a significant improvement in the ex-perimental subjects as compared with the controls, one predicts that implementation of a similar ma-nipulation, in a similar setting with similar subjects, will be o benefit on a larger scale.

Answer 177

TRUE

Question 178

Brian Nosek and his team have been able to effectively reproduce the same findings for approximately ___% of studies they’ve replicated.

Answer 178

47%

Question 179

What are some limitations of relying on laboratory studies to draw conclusions about behaviour?

Answer 179

Relying solely on laboratory studies to draw conclusions about behaviour can limit the generalizability of research findings and reduce their practical relevance. Laboratory studies may not accurately reflect how people behave in the real world, and may miss important factors such as cultural, social, and environmental factors.

Question 180

Can internal validity be established independently of external validity in research, and what is the relationship between the two?

Answer 180

Internal validity cannot be established independently of external validity, as the two are interconnected and equally important for advancing scientific knowledge and developing practical applications.

Question 181

True or False - Brian Nosek argues that one of the key benefits of replication is that it helps identify whether a scientific finding is generalizable across different populations and contexts.

Answer 181

TRUE

Question 182

true or false, Internal validity refers to the extent to which a study accurately measures what it intends to measure within the study itself.

Answer 182

TRUE

Question 183

Fill in the blank. “The greatest weakness of laboratory experiments lies in their ………… Social processes observed to occur within a laboratory setting might not necessarily occur within more natural settings” - Babbie, 1975.

Answer 183

Artificiality.

Question 184

According to the article by Mook, one major problem with the accusation of external invalidity or failure to generalize a research finding to the real-world situation is…

Answer 184

generalizations are often not intended by researchers

Question 185

Mook argues that there is an assumption that the purpose of collecting data in the lab is to ‘predict real-life behaviour in the real world’. List other possible reasons.

Answer 185

Examples: - To predict whether something can happen - To specify something that would happen in the lab - To demonstrate the power of a phenomenon - To produce conditions that cannot happen in real life

Question 186

Describe four uses of an experiment other than to predict real-life behaviour according to Mook.

Answer 186

looking at whether something can happen rather than whether it typically does happen, confirming the direction of a prediction, demonstrating the power of phenomenon even under unnatural conditions, and producing conditions that have no counterparts in real life.

Question 187

fill in the blank: Surely one of the experiments that has had permanent impact on our thinking is the study of_____ in rhesus monkeys, elegantly con-ducted b y Harlow

Answer 187

“mother love”

Question 188

True or false: The article by Mook (1983) argues that external validity and generalizability refer to the same concept.

Answer 188

False. Studies can have high external validity, but still limited generalizability.

Question 189

True or False? An assumption put forward in the Mook paper was that the purpose of data collection in the laboratory was to predict real-life behaviour in the real world?

Answer 189

True. However, there are situations where lab findings are not meant to replicate the real world so generalisation would provide no meaning for determining validity. It depends on what kind of conclusions one wants to draw from the study.

Question 190

True or false? A highly artificial laboratory experiment cannot tell us anything we can generalise to the real world

Answer 190

False – As demonstrated in dark adaptation studies, highly artificial laboratory experiments can still tell us about processes that take place in the real world (in that case, that dark adaptation takes place in two phases as function of the cones and then the rods in our eyes)

Question 191

According to Mook, how can researchers improve external validity in their studies?

Answer 191

Mook suggests several strategies for improving external validity, including conducting studies in naturalistic settings, using more diverse samples that reflect the population of interest, and taking steps to ensure that the variables and conditions in the study are similar to those found in the real world.

Question 192

What is External Validity?

Answer 192

It asks the question of how generalisable the effect of populations, settings, treatments, & measurement variables are.

Question 193

True or false: the article by Mook (1983) argued that “generalisations” are often not the intention when somebody is conducting a research

Answer 193

TRUE

Question 194

Fill in the blank: Open science is an approach to conducting research that emphasises __________, collaboration, and sharing of data and findings.

Answer 194

Transparency

Question 195

True or False: Internal validity refers to the extent to which a study can establish causal relationships between variables.

Answer 195

TRUE

Question 196

What is an example where the effects of an independent variable studied in isolation may be exaggerated?

Answer 196

The example provided was wearing glasses increasing IQ. Another may be having tattoos influencing competence in academic work. When it is the only information, it may have a large negative influencing effect on judgement, but when other variables are considered, it is likely to have a much smaller effect on judgements of competence.

Question 197

Replication

Answer 197

Repeating a study’s procedure and observing whether the prior finding recurs

Question 198

According to Mook 1983, what are the two differences between lab and life?

Answer 198

the absence of punishment and the possibility of discounting the danger of obedience

Question 199

True of False: When we make generalizations of our findings to the real world, they may have added force because of artificiality of sample or setting.

Answer 199

TRUE

Question 200

Why should we consider external validity with less fervour?

Answer 200

Generalising findings from an experimental study may not always be the primary intent of the study. As such, making sure that all studies have adequate external validity might be going overboard when evaluating whether a study and its findings are sound or not.

Question 201

According to Mook, what is the “pragmatic fallacy”?

Answer 201

The “pragmatic fallacy” is the erroneous belief that because a study’s findings are useful or valuable in some way, the study must also have high external validity.

Question 202

What is the point that the paper by Mook, is making?

Answer 202

The author argues that even in artificial laboratory settings, experiments can provide valuable insights into real-world phenomena and that the focus should be on understanding the processes being studied rather than trying to replicate real-life situations.

Question 203

True or False: Mook (1983) believes that external validity is more important for applied research than for basic research.

Answer 203

Answer: True Explanation: Mook distinguishes between applied research, which aims to solve practical problems, and basic research, which seeks to advance theoretical understanding. He argues that external validity is more relevant to applied research, as it focuses on the generalizability of findings to real-world situations. In contrast, basic research prioritizes internal validity, as it is concerned with establishing causal relationships and developing new theories.

Question 204

What is external validity, and why is it important to psychological research?

Answer 204

External validity refers to the degree to which research findings can be generalized beyond the specific conditions, participants, and measures used in the study. It is important because it determines the extent to which the results can be applied to real-world settings and populations.

Question 205

According to Mook, what is an “Agricultural”?

Answer 205

The assumption that an experiment’s results are intended to be generalized.

Question 206

True or False: A misplaced preoccupation with external validity can lead us to dismiss good research for which generalization to real life is not intended or meaningful.

Answer 206

True

Question 207

Unrepresentative samples are of no value at all T/F?

Answer 207

False. Unrepresentative samples can provide valuable insight, e.g., exceptional/different cases can demonstrate what CAN occur