critical perspectives 2 - open science

Question 1

what is open science

Answer 1

a solution to replication crisis

process of making the content and process of producing evidence and claims transparent and accessible to others

without transparency, claims only achieve credibility based on trust in the confidence or authority of the originator

transparency is superior to trust

can’t just trust a credible journal

Question 2

what is preregistration

Answer 2

define the research questions, methods, and approach to analysis BEFORE observing research outcomes (or before data collection)

prevents HARKing (hindsight bias)

registration revolution - idea that research is improving as a result of this, findings/protocol must be registered

protocol can be registered on open science framework - locked in so people can see if study does follow their protocol or change is later

Question 3

open methodology

Answer 3

documenting the methods and process by which those methods were developed/decided upon

method should give enough info for accurate replication

Question 4

what is HARKing

Answer 4

Hypothesising After Results are Known

hindsight bias

Question 5

registered reports

Answer 5

NOT same as registered replication report

peer review process in 2 stages:

1. reviewers and editors assess protocol
   - study rationale, procedure, and analysis plan
2. following favourable reviews (and prob revision to meet methodological standards), the journal gives acceptance in principle
   - journal says they will publish the results if you stick to the protocol and have good conclusions which are evidence bound
   - prevents them from only publishing significant results

Question 6

challenge of preregistration

Answer 6

slows things down

too constraining - have to do as you said you would, might change your mind - lots of discoveries in past were by chance

just need to make clear what was planned and what wasn’t planned

people can just publish on open science framework to not have to wait for peer reviewed approval - seek approval if you want critical analysis of methodology

Question 7

significance of standard vs registered reports

Answer 7

about 95% of standard reports found significant results

about 50% of registered reports found significance

as publication bias is more controlled for

unrealistic for scientists to be right 95% of the time - therefore registered reports are beneficial to the future of science

Question 8

open source materials and code

Answer 8

open source tech (software and hardware) and keep own tech open

e.g. the code you use to program questionnaire or experiment should be made open so others can replicate exactly the same

Question 9

open data

Answer 9

making dataset freely available

allows others to verify original analyses

facilitates research beyond scope of original research

avoids duplication of data collection - not wasting time and money for data that already exists

Question 10

challenges with open data

Answer 10

where do you share it - how can it be found

do people know it is there to be found - is there a point sharing if it cannot be found

Question 11

preparation of data for submission - FAIR

Answer 11

needs to be FAIR

• findable
• accessible
• interoperable (ability of computer systems to make use of data)
• reusable

also anonymous and confidential - ethics!

Question 12

open access to publication of findings - why it is important/why traditional is bad

Answer 12

traditional model of publishing:

• submit paper to journal
• they decide whether or not to publish
• researcher signs copyright over to journal who charge unis/libraries for access

problems:

• unfair
• limits access to those who have funds to pay for subscriptions

Question 13

types of open access publishing (2)

Answer 13

gold open access

• researcher/funder/institution pays journal to publish article
• final (formatted) version is freely and permanently accessible for everyone

green open access

• self-archiving
• unformatted version of manuscript is put into a repository (probs identical contents to if it was published paid for)

Question 14

effects of open access publication

Answer 14

open access = used more

• cited 36-600% more than others
• given more coverage by journalist and discussed more

facilitates meta-research

• enables AI to find info or text/data mining tools - can be missed often if not open access

Question 15

how much open science from 2014-17

Answer 15

found none of 250 random psych studies were transparent or reproducible in research

Question 16

how is open-science practices acknowledged/encouraged

Answer 16

digital badges on research papers from centre for open science

this acted as an incentive

3 badges:
preregistered, open data, open materials

evidence found that badges promoted open science (or at least open data)

also found coloured badges vs greyed out or control (no badges) made students and social scientists rate the integrity of the research more highly (not seen in members of public though)

Question 17

TOP guidelines

Answer 17

transparency and openness promotion

8 guidelines (don’t memorise prob)

1. citation standards
2. data transparency
3. analytic methods (code) transparency
4. research materials transparency
5. design and analysis transparency
6. preregistration of studies
7. preregistration of analysis plans
8. replication

Question 18

effect of open science on replicability of research

Answer 18

open science (and other practices that improve rigour)

increases replicability

86% of open science papers were replicable (compared to 36% found previously of many research papers)

shows that “research on research” needs to be done so things are completed to high standards and are accurate to reality

“credible or incredible” - transparency and scrutiny guarantees credible research