being sceptical Flashcards
being sceptical
What do we know? – in conclusion • How do you know that? Should always ask • Source – what is the source of info? • Quality of the evidence • Control groups?
The publication process
- Academics need places to publish work
- Can write articles in newspapers/magazines
- Can write books
- Who decides what is published in these?
The publication process
- Important to think critically about all evidence!
- Just because some research is published in a peer reviewed (reviewed by other academics) journal, doesn’t make it true! It simply increases the chance of it being true.
o But all studies have their drawbacks. Methodological issues, sampling issues etc
o Their conclusions may be limited in scope
o Each article builds on a body of knowledge, that when taken as a whole is useful.
• The publication process is also a human endeavour
o The Scientist
o The Reviewers
o The Editing Team
o The journal and the type of articles it “accepts”
Where to submit
- Journal esteem / impact factor
- What’s the journal’s remit?
- What’s the word length of the articles?
- Which journal might be sympathetic – does that journal publish this ‘type’ of work?
- Who are the editorial board?
Who picks it up?
What is their bias?
Which people do they ‘know’ that might review
this given their knowledge of the literature?
- This itself is skewed by the current state of play of the literature
- Authors can request particular reviewers and request not to be reviewed by particular people
- Editor can decide whether to take that advice or not
The publication process
Blind reviews? - Maybe not. People tend to cite themselves, so sometimes very obvious - Might even say “we have previously shown that….. (Guest & Lamberts, 2011)”
Reviewers are also often not named – imagine a long form academic version of twitter, pre-twitter.
(see examples that follow)
What are the reviewers biases? Do they have a competing theory? Do they agree? Does this work confirm/extend their findings?
- What is reviewed is the written paper
- No data is provided – and so no way of checking data or analyses …
- A system based on honesty!
- There might be some issues here …
What are the incentives for academics?
- A job requirement to publish
- In UK REF
Neuroskeptic (2012) The Nice Circles of Scientific Hell
Problems in the publication method
• The article raises a variety of issues – most of these impact the ability to replicate findings – which is hugely problematic, as science can only develop IF findings have a decent degree of replicability
• Issues raised;
• Turning a blind eye.
• Exaggeration
o Often in psychology – papers find “significant” results, but actually the size of these effects might be small
o Motivated to do so to make a bigger splash and get published
Post hoc story telling
• Science is about developing and testing hypothesis. If the hypotheses come after the data…
P value fishing
• You should make a hypotheses and run the test that best tests this. If you run lots of tests of different kinds, then statistically you are likely to find something significant simply by chance alone …
Outliers
• Outliers are a significant problem in data analysis. If you have extreme scores they can seriously mess up things like means and standard deviations, which are things that are used in many statistical tests.
• So you need them out. But that means selecting what is an outlier based on a rule and sticking to that.
• E.g., in some of my tests RTs < 300ms are impossible and
RTs> 2000ms are very slow indeed and suggest someone has not done that particular trial.
• But removing can influence analysis – so say whether it does!
Non publication of data
- Most academics are sat on years worth of data…..
- Often many academics try to replicate existing studies (with project students) and may fail to, but do not attempt to publish data.
- This creates a bias in the field
Partial publication
• Conduct a large study in which lots of things are measured, but then publish different papers only on sub-sets of this overall data.
replication
- What is replication?
- It is an attempt to replicate a study and produce the same finding
- Fundamental principle of scientific development, that findings are replicable
- Direct V conceptual replication (same V different methods)
- The importance of a good methods section!
- But – vitally important – needed for ‘good science’
Studies focused just on replication used to be relatively rare because:
- Harder to publish
- a replication is not new and exciting if it replicates
- If it does not replicate, there may be many more reasons for this (methodological issues) which might require extensive investigating before being able to publish a lack of replication.
