critical perspectives Flashcards
1
Q
replication - the crisis
A
- Open Science Collaboration (2015)
- looked at hundred studies
–> taken from quite esteemed journal
–> arguably the best studies taken from the best journals at the time - only 36% of the studies
replicated when the collaboration replicated them again
–> only 23% from social psychology were replicable - scared psychologists
–> crisis of confidence
2
Q
cristea et al - emotion studies
A
- top 65 studies is emotion psychology
–> 40 cited observations
–> 25 experimental - showed greater effects than meta-analyses and large studies using the same questions
3
Q
what is meant by replication?
A
- doing the study again
- aim to see if the same findings are found
- more evidence we have that shows the same thing again and again, more likely we are to believe it
–> e.g. if one study finds ‘81% of people believe this’ it may be hard to believe, but if a second study finds ‘78% of believe this’ then they both become more believable
4
Q
define replication
A
repeatedly findings the same results
5
Q
benefits of replication
A
- Protects against false positives
–> e.g. sampling error - Controls for artifacts
–> maybe you had leading questions
–> maybe the look of the researcher impacted results - Addresses researcher fraud
- Test whether findings generalise to different populations
- Test the same hypothesis using a different procedure
6
Q
direct replication
A
- scientific attempt to recreate the critical elements of the original study
–> samples, questions, procedures, measures - The same or similar results are an indication that the findings are accurate and reproducible
- way of replicating the elements of the question you think are impacting results
- NOT EXACT replication
–> practically impossible in psychology
7
Q
conceptual replication
A
- test the same hypothesis using a different procedure
- same or similar results are an indication that the findings are robust to alternative research designs, operational definitions and samples
8
Q
registered replication reports - APA
A
- collection of independently conducted, direct replications of an original study, all of which follow a shared, predetermined protocol
- results of the replication attempts are published regardless of the outcome
9
Q
reasons for replication - faking
A
- Diederik Stapel
- found out he was faking and fabricated results
- hugely influential psychologist
- started off properly and fairly
–> found complicated and messy results across many variables
–> couldn’t bet his paper published
–> journal editors suggested cutting out the messy bits and added details - started to write more neat articles
–> made the data support the argument and create a narrative
10
Q
reasons for replication - sloppy science
A
- nine circles of scientific hell
1. limbo
–> seeing bad practice and not saying anything
2. overselling
–> focus on the bits of the study that worked
3. post-hoc storytelling
4. p-value fishing
–> outcome switching
5. creative outliers
–> deciding who to remove to make the data look better
6. plagiarism
7. non-publication
–> not publishing your papers
8. particle publication
9. inventing data - further down you are the worse it is
11
Q
outcome switching - sloppy science
A
- part of p-value fishing
- changing the outcomes of interest in the study depending on the observed results
- an example ‘p-hacking’
–> taking decisions to maximise likelihood of a statistically significant effect
–> rather than an objective or scientific grounds - if you do two tests, p value is now not 0.05 (5%) its actually 10%
–> if you found to be non-significant and one to be significant but you IGNORE the non-significant, you have changed you interested outcome due the results
12
Q
need for replication - small samples
A
- small samples and lack of statistical power can be a problem
- can say you have found an effect but if you find this in only a few number of people, that might not be the same in a larger sample
–> needs to be replicated in a larger group to be accepted
13
Q
need for replication - publication bias
A
- part of ‘non-publication’ and ‘partial publication’
- findings that are statistically significant are more likely to be published than those that are not
–> in general, there are good reasons for this - But could published studies represent the 5% of findings that occur by chance alone?
–> known as “the file drawer problem’ - quite scary to only see significant results
14
Q
how common is sloppy science?
A
- John et al (2012)
- Surveyed over 2,000 psychologists in the US about their involvement in questionable research practices
–> failing to report all the measures or conditions
–> deciding whether to collect more data after looking to see whether the results were significant
–> selectively reporting studies that “worked” - Concluded that the percentage of respondents who have engaged in questionable practices was surprisingly high
15
Q
peaking
A
- can’t stop study when you reach a peak
- have to decide on a set sample
- complete the entire study and the whole sample
- then calculate results
- data changes all the time, stopping at a point of your choosing doesn’t highlight the entire pattern
- might have just caught the data at a particular peak or stoop
16
Q
is sloppy science really a problem?
A
- Simmons, Nelson and Simonsohn (2011)
- Flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates
- Tested if listening to certain music makes you younger (2 songs)
- did find a significant effect that listening to a certain song made you 1.5 years younger
–> this is impossible
–> how? - they only reported the parts that gave them a significant effect
–> e.g. only 2 songs
–> controlling for father’s age - therefore appears impossible
17
Q
reporting using 0.05
A
- 0.05 means that there is a 1/20 chance you’ll find something
- if you test 20 things you will find an effect
- if you only report this, this looks like a significant effect
- if you reported all other 19 things and said they found no effect, people would question the significance of the 20th variable
–> but people didn’t use to do this
18
Q
moderators
A
- variables that influence the nature
–> e.g. direction and/or size of an effect - For example, country or culture
–> e.g. ”Reverse” ego-depletion - Identifying moderators is good because it improves our understanding.
–> “Second generation research”
–> moderators test when there is an effect –> 1st gen tests that there IS an effect, 2nd is WHEN
19
Q
poor replication - scientist error
A
- replication of a study done into elderly, priming and walking speeds (Bargh, 2012)
- Doyen replicated but didn’t find the same results
–> responded with ‘Doyen and her colleagues are “incompetent and ill informed” making “gross” methodological changes’ - replication needs to be reported whether it replicates results or not
20
Q
summarise replication
A
- replication is a cornerstone of science
- however, there are concerns that may findings in psychology may not be replicable
- there is, therefore, “substantial room for improvement with regard to research practices”
21
Q
one solution to poor science
A
open science
22
Q
open science
A
- the process of making the content and process of producing evidence and claims transparent and accessible to others
- without transparency, claims only achieve credibility based on trust in the confidence or authority of the originator
–> transparency is superior to trust
23
Q
open methodology
A
- documenting the methods and process by which those methods were developed / decided upon
24
Q
pre registration
A
- define research questions, methods and approach to analysis BEFORE observing the research outcomes
–> definitely before analysis - prevents HARKing
–> Hypothesizing After Results are Known
–> Hindsight bias
25
Q
chris chambers paper
A
- we must register results
- moves to uphold transparency not only make psychology more scientific but they harness our knowledge of the mind to strengthen science
26
Q
open science framework
A
- can upload all registrations
- they get time stamped and date checked
- can pre register what you want to do before you do it
- all get saved
- anyone can do it
- not peer reviewed
27
Q
registered reports
A
- not the same as a registered replication report
- you can split the peer review process into two stages
28
Q
two stages of the peer review process - registered reports
A
- Reviewers and editors assess a detailed protocol
–> study rationale, procedure and a detailed analysis plan - Following favourable reviews (and probably revision to meet methodological standards), the journal offers acceptance in-principle
–> publication of the findings is guaranteed provided that the authors adhere to the approved protocol, the study meets pre-specified quality checks, and conclusions are appropriately evidence-bound
29
Q
debate about the value of preregistration
A
- can slow things down
- can be constraining
- but they make it clear what is exploratory and what is not
–> if you do this it is, pre registering is fine - avoids chance discoveries
- when comparing standard reports with pre-registered reports
–> standard reports publish supported hypotheses 95% of the time
–> pre registration only publish supported hypotheses 50% of the time, they ALSO report 50% of the time unsupported hypotheses
30
Q
open source materials and code
A
- Use open source technology (software and hardware) and open your own technologies
–> e.g. the code used to programme the questionnaire or experiment - this allows for better replication
–> people can use the exact same code and programme
–> or they can adapt it
31
Q
open data
A
- make data set freely available:
–> allows other scientists to verify the (original) analyses
–> facilitates research beyond the scope of the original research
–> avoids duplication of data collection - there some issues with open data
32
Q
issues with open data
A
- where do we put it?
–> how will people find it - how do you prepare the data for submission?
–> data needs to be FAIR (findable, accessible, interoperable, reusable)
–> anonymity / confidentiality
33
Q
open access (publication of findings)
A
- traditional model of publication
–> researchers submit a paper to a scientific journal who decide whether or not to publish
–> researcher then signs copyright over to the journal who then charge universities / libraries for access - problems:
–> unfair
–> limits access to those who have funds to pay for subscriptions
34
Q
types of open access publishing
A
- gold open access
- green open access
35
Q
gold open acces
A
- researchers (or more likely the funders or host institution) pay the journal to publish the article
- the final (formatted) version is freely accessible and permanently accessible for everyone
36
Q
green open access
A
- Also referred to as self-archiving
- Put an (unformatted) version of a manuscript into a repository
–> e.g. psyarchive
37
Q
effects of open access publication
A
- Open access works are used more:
–> within academia, open access works are cited between 36% and 600% more than works that are not open access
–> outside of academia, open access works are given more coverage by journalists and discussed more in non-scientific settings (e.g., on social media) - Open access works facilitate meta-research
–> enable the use of automated text- and data-mining tools
38
Q
issues with open access
A
- very expensive
- can cost 1000s to publish a paper and grant access
–> penalizes more junior researchers with less grants and little money to publish
39
Q
Lisa Barrett - APS president
A
making open access complete an immediate is a great goal and a necessary element of any plan to democratize science
40
Q
amount of open science
A
- very little between 2014 and 2017
41
Q
ways to promote open access
A
- legislation
–> if funded by certain organisations, you have to do open science - digital badges to acknowledge open science practices
–> there is evidence that badges promote open science (or at least open data)
–> seeing the badges made student teachers and social scientists trust the paper more but not the public - societies
- groups
- guidelines
42
Q
Transparency and Openness promotion - TOP guidelines
A
- Citation standards
- Data transparency
- Analytic methods (code) transparency
- Research materials transparency
- Design and analysis transparency
- Preregistration of studies
- Preregistration of analysis plans
- Replication
43
Q
summarise open science
A
- open science and other practices that improve rigour increases the replicability of research
–> can lead to replication of up to 86% of studies (better than the previous 36%) - makes research more credible
–> better to be credible than trying to be incredible - its science done the right way
