Lecture 11: Psychological science into the 21st century: The standard paradigm, the replication crisis, and Open Science Flashcards
Isn’t it remarkable that you come across a chapter in a psychology textbook entitled
“Is psychology a science?”
Points to…
- insecurity of psychologists with regard to their own scientific field
- healthy self-critical attitute (typical property of psychology, seen less in other areas)
the image of the psychologist is determined by the media
there, psychologists rarely act in a scientific role. much more often you see psychologists that have never done research
verschil tussen hoe kinderen scientists en psychologen zien, en een groot verschil tussen “perceived difference in knowledge between different kinds of scientists and the average person”
oke
what also plays a role
criticism from within
criticism from within =
- Psychoanalysis: Psychology is superficial
- Critical psychology: psychology wrongly treats man as a physical object
- Humanistic psychology: scientific psychology is cool and distant, when it should be concerned with people
- Feminist and postcolonial psychology: scientific psychology is a matter for Western white men
so psychology has a hard time with itself and with the outside world… voorbeelden
- Psychologists regularly have to defend themselves
- Partly against stereotypes, partly against justified objections
- Bad experiences with Popper and Freud
- So for psychology, the demarcation problem is very acute…
- Maybe methodology can come to the rescue?
methodolatry=
Worship of a method that employs it uncritically regardless of ever- changing particulars and steadfastly ignoring past negative results.
wat is er met methodology en methodolatry
- Methodology plays an important role in psychology
- General principles like striving for clarity, focus on testing, the provisional nature of science
- Plus specific attention to issues such as uncertainty (statistics), induction, confounds
- Strong orientation towards empirical data + statistical analysis
- This is all good. But…
- A prevailing viewpoint in psychology is that science if characterizes solely by method
- “Whoever follows the scientific method is
scientific” - But make no mistake! You can also follow the scientific method very superficially…
- Are you scientific just because you follow the
precepts?
methodology kan daardoor dus ook …. worden
methodolatry
wat is de analogie van Feynman
Cargo cult science, in WOII the Americans build bases on the Pacific islands, and a cariety of religious practices spontaneously arose from that (planes, chocolate etc)
-> he suggested that science can work the same way, it looks like science but it isnt.
wat was feynman zijn boodschap
science is not just doing experiments and statistical analyses, it is an attitude
quote van Feynman
[T]here is one feature I notice that is generally
missing in Cargo Cult Science. That is the idea
that we all hope you have learned in studying
science in school—we never explicitly say what
this is, but just hope that you catch on by all
the examples of scientific investigation. It is
interesting, therefore, to bring it out now and
speak of it explicitly. It’s a kind of scientific
integrity, a principle of scientific thought that
corresponds to a kind of utter honesty—a kind
of leaning over backwards.
The first principle is that you must not fool
yourself—and you are the easiest person to
fool.
mertons core values
- communalism
- universalism
- disinterestedness
- organized skepticism
communalism =
scientific products belong to no one
universalism =
truth claims are judged the same, no matter who makes them (je hoeft niet perse een phd te hebben etc)
disinterestedness=
scientists are not interested in the outcome of research
organized skepticism=
ideas are cracked down on and rigorously tested, regardless of who proposes them
hoe was de communalism tijdens 20th century
- articles were published behind a publisher paywall
- psychologists often regarded their data as property, did not share data or materials etc
universalism hoe zit dat nu
- no explicit favouring groups in psychology, however clearly power structures (students, editorial positions, leadership of scientific org)
- also underrepresentation of women, minorities, scientists from developing countries
- WEIRD demographic
- shared value is not automatically a reality
WEIRD demographic
westerd
educated
industrialized
rich
democratic
disinterestedness hoe zit dat nu
- psychologists and other researchers are interested in success
- the temptation to make things better than they are can be considerable
- fraud and questionable research practices
organized skepticism hoe zit dat nu
- scientific culture separates the person and the idea, therefore distrust is not personal (but sometimes understood as suspicion)
- however, much of psychology in 20th century appears to have been too gullible
the scientific attitude and mertons norms are not actualized all the time, this does not make them useless! why?
they are regulative ideals: things we strive for. striving for an ideal can be useful, even if this ideal is strictly unattainable.
science has room for.. not for …
skepticism
cynicism
The standard of psychological research in the 20th century:
- Create a theory (usually a verbal story)
- Come up with an experiment (usually small n)
- Test hypothesis (usually with a significance test)
- Publish results (usually without full disclosure and without data)
who introduced the notion of a scientific paradigm
Kuhn (1968)
Vul’s Voodoo correlations
- The neuroscientist Edward Vul lamented the small studies in neuroscience
- Identified extremely high correlations between e.g. personality and fMRI results (r>.8)
- In fact these values were higher than should be possible given the reliability of the measures used
- Argued that these results could not possibly be truthful
Bargh’s fantastic effects
- Some psychologists claimed fantastic effects
- Priming people with words like “old” and “elderly” leads people to walk more slowly?
- People who are lonely take hotter showers?
- These effects raised more than a few eyebrows
- The research results seemed too good to be true
so who pioneered behavioural priming research
John Bargh
4 types of questionable research practices
- p hacking
- publication bias
- HARKing
- Dropping experimental conditions that ‘dont work’
p-hacking
deleting outliers, adding covariates, splitting the sample, selecting variables until you find a significant effect
publication bias
only publishing positive results
HARKing
Hypothesizing after the Results are Known
Bem’s Feeling the Future
- Psychologist Daryll Bem purported to have shown that ESP exists
- Extraordinary claims that basically nobody believed
- But… Bem had hardly done anything out of the ordinary
- Most people drew the conclusion that this meant that something was wrong with our methods
wat was er mis met de methodes
- samples sizes varied across studies without a stated reason
- different studies appear to have been lumped together or split apart
- its not clear which analyses were planned in advance
- one tailed tests in the absence of a directional prediction
- p values very close to 0.05
- how many other studies were run but not reported?
the stapel case
- Social psychologist Diederik Stapel published heavily in leading journals
- Amazing effects that made their way into newspaper headlines
- Three brave Ph.D. students suspected something was wrong
- Data for 55 of his papers were entirely made up!
Diederik Stapel (1966).
Important difference of fraud with QRPs:
Fraud is intentional, QRPs are not
replication crisis
In 2013, a group of psychologists – the Open
Science Collaboration – decided to assess
replicability of 100 psychological studies
* The results were published in Science and suggested that many studies could not be replicated
* This marked the beginning of the replication crisis (now known as the crisis of confidence)
pillars of open science (3)
- Open Data: So that anybody can assess the
evidence - Open Materials: So that anybody can replicate the study
- Preregistration: So that a posterori tinkering with analyses becomes visible
relationship between mertonian norms and open science
- Wagenmakers et al. (2021) relate open
science to Merton’s norms - Argue that Openness and transparancy
instantiate these norms - Open science = science done right?
Open Science institutionalized
- Center for Open Science
- Open Science Framework
- Psychological Accelerator
- GitHub code sharing
- Findable, Accessible, Interoperable, Reusable (FAIR) data
- Transparency and Openness Promotion (TOP) guidelines
- Diamond Open Access journals
the Open Science Framework
- The OSF is a repository
- Allows researchers to make all of their materials available
- Now standard in psychology
the Manylabs project
- In five Manylabs projects, different labs collaborated to replicate psychological phenomena
- Bad news: some of psychology’s most sexy findings don’t replicate
- Good news: many findings replicate well, be it often with smaller effect sizes
- The Manylabs setup has been taken up in other fields (ManyBabies, Many Birds, etc.)
the Psychological Science Accelerator
- The Psychological Science Accelerator (PSA) aims to accelerate cumulative reliable knowledge
- Countering WEIRDness through diversity is a main aim
- PSA can rapidly employ huge distributed studies to evaluate research findings
criticisms of open science
- Increased bureaucracy and more work for researchers
- Most data are never downloaded, many preregistrations aren’t adhered to, badges aren’t checked
- Too much focus on replication, not enough on creativity
- Open Science is expensive and can compound inequalities
- Open Science is not always feasible or desirable
- Some approaches may decline (e.g. field studies, qualitative research)
- “How scientific is psychology?” remains an important question
- Clinging to methodological precepts risks degrading into methodolatry
- Scientific culture requires more than rule following: the broad norms of Merton, for instance
- In the 20th century, the psychological paradigm did not always adhere to these norms
- Result: the replication crisis
- Current transformation to Open Science is in progress, but of course comes with its own risks and pifalls…
oke
