Week 17 - Being skeptical/Publication + Replication Flashcards
What is skepticism?
- Asking why
- Applying a critical consideration of info that is presented to you
Do you need to walk 10,000 steps a day?
- No, 10,000 steps is arbitrary
- Company made up 10,000 steps to sell their watches with pedometers
- Walking more does have benefits
-> contextual - different for everyone
Is breakfast the most important meal of the day?
- Yes, it was started by Kellogg’s as a marketing tactic
- Skipping breakfast reduces overall energy levels and induces morning lethargy
-> changes depending on what you eat for breakfast
Do you have to have 5 fruit + vegetables a day?
- Complicated
- WHO recommended 400g fruits and veg a day
- Some estimate you should be eating 10-a-day
-> more important to eat different coloured fruits and veg
e.g. green fruits good for your heart, blood and cardiovascular disease
e.g yellow fruits and orange fruits are full or vitamin C which is good for muscles + bones
What are 4 questions you can ask to be skeptical?
1) How do you know that? - where have you got the info from?
2) What’s your evidence? - is it anecdotal or from a research paper?
3) Who told you that?
4) Why are you telling me this?
What is an example of misinformation?
Parasite drug prevents COVID-19
- Ivermectin (parasite drug) shown to reduce SARS-CoV-2 mortality rates in RCTs
- Prevents viral RNA reproduction after 48 hours -> stops virus spreading throughout entire body
- Increased doses required to achieve results -> doses were unsafe to humans
- No ivermectin vs placebo studies were conducted -> don’t know if effect ivermectin has was from ivermectin or from the process of taking medicine
What is an example of data misuse?
- 80% of dentists recommend Colgate
- Dentists were asked to recommend a series of brands and products
-> 80% of dentists included Colgate on the list - This was found to be criminally misleading
-> Colgate were misusing data in their advertisement to sell toothpaste
-> misleading you to encourage you to buy toothpaste to make profit
What is an example of malicious intent?
- Woman sued McDonalds for not realising her coffee would be served hot
-> she received 3rd degree burns after spilling coffee on her lap while driving - The coffee was served at 82 degrees (far higher than average)
-> this temp caused 3rd degree burns on skin in 3 seconds - McDonalds knew about the risks of serving coffee that hot
- McDonalds launched a PR campaign to discredit and disparage the victim
Why do we publish?
- Academics need places to publish work
-> Newspapers
-> Magazines
-> Books - Academic journals are preferred
Who is involved in the publication process and how is it done?
- Publications are peer reviewed by two other academics (anonymous)
- Need to remain skeptical - publication doesn’t mean perfect
Think about:
-> method issues
-> sample issues
-> scope of conclusions
Each publication is reviewed on a series of criteria, different for each journal
General 3 are:
- Novelty -> how new is the research
- Quality -> is it good quality science, is it robust?
- Relevance -> what’s the relevance to the real world + to the journal
Describe the publication process
1) Academic writes paper
2) Academic submits on. submission portal
3) Picked up by Action Editor who initially decide whether it fits the journal
4) Action editor assesses paper + decides which academics might review it
5) Academics asked to review paper (normally 2-3) - reviews are blind
6) Academics do review in 1-3 months for free
7) Action editor reads reviews and makes a considered judgement - REJECT / REVISE / ACCEPT
If rejected, revise paper and submit to another journal or STOP
If revise, then try to address as many issues as possible
-> resubmit arguing you have met as many of the comments as possible
1) Publication process: Academic writes paper
Where should you submit?
- Relevance -> pick a relevant journal to submit to
- Esteem -> Is it a big journal? Do a lot of people read it? - More likely to get cited
- Impact factor -> How impactful the research in the journal is - measure of how many times you’ve been cited + the real world implications
Journal guidelines:
- Length, format, word count
- Coverage -> type of research they will publish - some will only publish certain methods (e.g. qual research or quant research), certain topics, or type of paper written (e.g. research paper, meta analysis, systematic review)
2) Publication Process: Picked up by Action Editor and sent to review
Editor:
- Biases
-> might be opposed or in support of your research
-> might not understand your research
Desk rejection:
- Doesn’t meet requirements
- Doesn’t fit the journal
- No new findings/additions
Once decided if they will keep your research:
- They will pick 2 reviewers in a similar area
- Anonymous
- Authors can request/avoid certain people
Blind reviews
- Reviewers + authors don’t know each other
- Can sometimes be obvious
Reviewer input:
- Feedback on methods and writing
- May have biases/agendas
3) Publication process: Action editor reads reviews and makes a considered judgement
Editor response:
Reject:
-> try another journal
-> cut your losses
Revise:
-> make some changes
-> resubmit
Accept
-> you have been published
What are problems with the publication process?
Until recently only the paper was reviewed - no data, no analyses
System is based on honesty
- Incentives to publish
- REF
- Job descriptions
- Renown
Might play games that increase our chances of being published
What are 6 author biases?
Exaggeration:
- Significant data, small impact/effect
- More renown
-> taking a bit of data + blowing it out of proportion
Outliers:
-> people that don’t fit the data - extreme values at either end of system
- Either ignored or explained away
- Excluded from analysis to “improve” effect/significant
-> can sometimes remove them for good reasons
p-hacking:
- Gather lots of data and change analysis/variables until something works
- Run lots of tests + only report significant ones
Post-hoc storytelling:
- Data first, do analyses then write hypotheses after
Inventing/fabricating data:
- Adding, removing or editing data that doesn’t support a hypothesis
- Just making it up
Non-publication:
- Not reporting findings
- Publication bias -> only significant findings get published
What is replication?
- Rerunning or redesigning studies
-> conceptual or direct replication
Conceptual replication -> take a design and transcribe it into a different format
Direct replication -> taking the method section that someone has written and doing it again
Validates findings
-> increases reliability + validity
- Can test new contexts/populations using the same methods
- Tests applicability
How can you improve the publication process?
- Working with open science
-publish data
-publish analyses
-state hypotheses and objectives first
-> stops p-hacking + post-hoc narratives and prevents non publication of results - Allows for easier replication
- Encourages honesty
- Reduces publication bias
