WEEK 7) Principles of research critical appraisal

Question 1

Why is it imp to study research design?

3 main things, hint minimise and maximise

Answer 1

• Minimize potential for bias or unclear interpretation
• Maximize efficiency of resource utilization (dont want wasted resources). e.g. draw conclusions which are contrary to whats really going on. Wasted resources, made conclusions about a treatment that could be harmful.
• Minimise ethical issues with subjects

Question 2

Distinguish btn basic research and applied research.

Answer 2

BASIC RESEARCH: you’re addressing a question but you
dont want to use the results of the study to relevant things in the real world

vs

APPLIED RESEARCH: attempt to apply to real world problems.

Question 3

Why do we study critical appraisal of research?

Answer 3

Application of scientific method to human studies not always easy

Cannot treat 􏰀research results􏰁 as a black box in which we have uncritical faith

Practicing psychologists are the experts and need to decide for themselves whether a given study is to be believed e.g. psychologists need to be able to critically evaluate articles and then maybe apply to real world therapy

Question 4

What is the scientific method steps for a normal experimental design?

Answer 4

Ask q 
Do background r
construct hypothesis 
test with experiment 
analyse results/ draw conclusions 
Decide whether hypothesis is true or partially true or false
Report results 
Think and try again (go back to construct h).

Question 5

What is good about the scientific method?

Answer 5

• Conclusions based on quantifiable and reproducible evidence
• evidence-based psychology
• Control of variables other than that of interest ! confidence in attribution of effect

Question 6

What is the scientific method steps for a non-experimental design?

Answer 6

Have idea 
do background r
pose q
collect data
Analyse results/ draw conclusions 
Generate a hypothesis 
report results 
think again and go back to pose q.

Question 7

What is Random sampling?

Answer 7

When every member of the population has an equal probability of selection

Question 8

What is good about random sampling ?

Answer 8

• then your sample is truly representative.

- And should be easily generalisable.

Question 9

If youve used random sampling, can you assume its randomised?

Why is this?

Answer 9

In most cases you cant ensure its completely random. Samples may be technically non-random but arbitrary.

In practice, samples are often obtained for convenience, higher chance of being non- representative e.g. all 300 level psych students.

(Applies to surveys as much as experimental studies)

Question 10

What are four key measurement issues?

List their importance from lowest to highest importance

Answer 10

• Most closely match research question / hypotheses
• Best (psychometric) properties – Reliability/validity
– Minimum variance (you want to minimise error v).
– Most responsive (to experimental factor)
• Most feasible
• Least cost

Question 11

This is what critical appraisal is not, tell me what it is then…

negative dismissal of any piece of research

Answer 11

balanced assessment of benefits and strengths.

Question 12

This is what critical appraisal is not, tell me what it is then…

Assessments of results alone

Answer 12

assessment of results and r process

Question 13

This is what critical appraisal is not, tell me what it is then…

based entirely on detailed statistical analysis

Answer 13

consideration of qual and quant aspects of r

Question 14

This is what critical appraisal is not, tell me what it is then…

to be undertaken by expert researchers /statisticians

Answer 14

to be undertaken by all health professionals as part of their work.

Question 15

True or false, good appraisal is inherently retrospective

Answer 15

true

Question 16

What is CONSORT in critical appraisal of r?

Answer 16

CONSORT: way of assessing areas of a research report.
Ensures various criteria are met. Authors should address
this checklist.. This is used in randomised trials.

Question 17

What are 10 questions one could ask to help you make sense of evidence / randomised controlled trials?

Answer 17

1. Did the study ask a clearly defined question
2. Is it a randomized controlled trial
3. Subjects appropriately allocated to groups
4. Staff and subjects blinded
5. All subjects who entered accounted for
6. All subjects followed-up consistently
7. Sample size adequate for power
8. How are the results presented and key message
9. How precise are the results
10. Were all important outcomes considered

Question 18

What are some biases in these appraisal guidelines (the 10 q ones on randomised controlled trials).

Answer 18

• Primarily geared to intervention studies; CBT, pharmacotherapy, …
• Not so much attention given to other forms of research
• Seeking scientific evidence of potential effects attributable to intervention
• RCTs considered the holy grail

Question 19

Your perspective will affect which aspects of appraisal you give most weight to. You MUST consider the three reasons why you are bothering to read the article. What could they be?

Answer 19

– Research direction

– Clinical practice
(implications for your own clinical practice.)

– Non-specific
(your purpose for reading article will affect the emphasis you place on parts. )

Question 20

Is it bad to not explicitly state your hypothesis in an intro?

Answer 20

no not necessarily, so long as after the intro you have a clear sense of where the report is going, it should be fine. Even if aims arent explicitly stated.

Question 21

It is a randomly controlled trial if

Answer 21

the participants are dbl blind =neither the participants nor the researchers know which participants belong to the control group, nor the test group.

e.g. random assignment.

(Double blind means both subject and investigator blinded to treatment allocation)

Question 22

Should an article explicitly state that all subjects are accounted for?

Answer 22

It should, otherwise it should at least mention drop outs etc. Otherwise its suspicious.

can infer that this is suspicious. They should report
the completion rate e..g if one or two partiicpants dropped out.
Not reported which suggests suspicious missing data.

Question 23

One must also state that the sample size is adequate for power. What should the report say?

Answer 23

Sample size specified which is good “there were 86 men as participants” but no reference to adequacy of statistical power

Question 24

True or false you will have more power with a repeated measures design rather than a btn subjects design.

Answer 24

True.

Question 25

Precision of the results in the practice example given?

Answer 25

• No confidence intervals given although they could be calculated from the statistics provided
• SDs are relatively small
• Cohen􏰂s d reported which makes comparison of effects across outcomes possible

Question 26

What could be the effect of having your paper funded by a certain industry that wants a particular outcome?

Was anything missed?

Answer 26

Report could be biased.

• Motivated by altruistic desire to ascertain the truth ?
• Unconscious biases ?
• Did the clinical investigators feel free to modify the protocol or analytical approach to satisfy themselves ?
• How independent were the independent biostatisticians ?

Question 27

What is impact factor (IF) of research?

Answer 27

a measure of the impact articles published in a given journal have in their field on average

Question 28

Does IF measure quality ?

Answer 28

Does the IF measure quality ?
– Consider discipline with small numbers of researchers, cannot compete with psychology !
– Many citations may just be the squeaky wheel

Question 29

Can the IF mislead?

Answer 29

– Some types of articles are more likely to be cited than others ! higher IF
– Journals can play games to increase IF

Question 30

What is a quality score?

Answer 30

Take appraisal criteria (e.g. CONSORT) and assign a score to each. Sum scores for an article to obtain overall score

Question 31

What are the pros and cons of a quality score?

Answer 31

• Pros:
– Quantifies quality
– Enables comparisons across studies – Regardless of journal IF or quality
• Cons:
– Assumes we all apply same weight to a given item
– A score could fail certain (key) items but still get an overall pass

(largely discarded scoring measure today).

Question 32

What are three ways a journal can be rated on?

Answer 32

• Rated by prestige
• Rated by reviewer difficulty
• Rated by impact factor

Question 33

Impact factor scores for common journals are?

Answer 33

– Am J. Psychiatry = 5.9
– Journal of adolescence = 1.2

•  By comparison
–  Nature = 27.1
–  Lancet = 17.5
–  Psychological bulletin = 9.75
–  Australian J. psychology = 0.64