Session 2: Methods and Evidence

Question 1

Consider the argument for evidence-based medicine

Answer 1

• Health service delivery should be based on best available evidence
• Best evidence = findings of rigorously conducted research rather than personal past experiences etc
• Evidence of: effectiveness (of drugs, practices, interventions) and cost-effectiveness (in a system with finite resources, where should money be spent to gain the maximum utility?)

Question 2

Consider previous practices and ineffective and inappropriate interventions

Answer 2

• Ineffective and inappropriate interventions waste resources that could be used more effectively
• Variations in treatment create inequities
• Previously practices were influenced too much by: professional opinion (big names in the field etc), clinical fashion (what peers are doing), historical practice and precedent, organisational and social culture
• Research has shown that clinicians have often persisted in using healthcare interventions that are ineffective, failed to take up other interventions known to be effective and tolerated huge variations in practice which can lead to all sorts of risks to the patients.
• Examples include doing ineffective things such as prophylactic use of lidocaine during MI and NOT doing things that ARE effective e.g. treatment of eclamptic seizures with MgSO4.

Question 3

Describe the origins of evidence-based healthcare

Answer 3

• Archie Cochrane’s book Effectiveness and efficiency: random reflections on health services (1972) set out principles of evidence-based practice and criticised medical profession for failing to take account of research
• Cochrane called for register of all RCTs, enabling sytematic reviews and meta-analyses of data and evidence produced by RCTs. Cochrane centres develop. If all evidence is put together in one place, it is much more accessible.

Question 4

Give the standard definition of Evidence-based practice

Answer 4

• Evidence-based practice involves the integration of individual clinical expertise WITH the best availble external clinical evidence from systematic research.
• “Evidence based medicine is not ‘cookbook’ medicine. Because it requires a bottom up approach that integrates the best external evidence with individual clinical expertise and patients’ choice, it cannot result in slavish, cookbook approaches to individual patient care. External clinical evidence can inform, but can never replace, individual clinical expertise.”
• Also known as evidence-based healthcare and evidence-based medicine

Question 5

Why are systematic reviews needed?

Answer 5

• Traditional, “narrative” literature reviews may be biased and subjective.
• Not easy to see how studies were identified for review (methodology not transparent)
• Quality of studies reviewed variable and sometimes poor.
• Systematic reviews are useful - can help address clinical uncertainty (can answer a question that needs to be answered).
• Systemaic reviews can also highlight gaps in research/poor quality research

Question 6

Why are systematic reviews useful to clinicans?

Answer 6

• By appraising and integrating findings, they offer both quality control and increased certainty (poor quality studies are excluded).
• They offer authoritative, generalisable and up-to-date conclusions.
• They save clinicans from having to locate and appraise the studies for themselves.
• They may reduce delay between research discoveries and implementation
• They can help to prevent biased decisions being made (depends on quality of review though)
• They can be relatively easily converted into guidelines and recommendations
• However, doctors need to be able to access systematic reviews and to appraise them to be satisfied about the quality of evidence.

Question 7

What does it mean to assess quality of evidence?

Answer 7

• Easiest and best done using a critical appraisal tool or instrument
• Critical appraisal tools suggest the things to look for, and the questions to ask of, research articles

Question 8

Where do you find the evidence?

Answer 8

• Medical journals: many reputable, peer-reviewed journals publish systematic reviews e.g. the Lancet, BMJ. EBP-specific journals are a more recent phenomenon, focusing on critical appraisal and systematic reviews.
• Cochrane Collaboration: the Cochrane library includes 4 databases (The Cochrane Database of Systematic Reviews, the Database of Abstracts of Reviews of Effectiveness, the Cochrane Controlled Trials Register, the Cochrane Review Methodology Database)
• The NHS Centre for Reviews and Dissemination: a national centre to lead the review, management and dissemination of research findings in Britain.
• NIHR Health Technology Assessment Programme: aims to produce high quality research information on the costs, effectiveness and broader impact of health technologies. Includes primary research AND systematic reviews.

Question 9

What are the types of critiques of evidence-based practice?

Answer 9

• ‘Practical criticisms’: around the possibility of evidence-based practice
• ‘Philosophical criticisms’: around the desirability of evidence-based practice

Question 10

What are the Practical Criticisms of EBM?

Answer 10

• May be an impossible task to create and maintain systematic reviews across all specialties.
• May be challenging and expensive to disseminate and implement findings
• RCTs are seen as the gold standard but not always feasible or even necessary/desirable (e.g. due to ethical considerations).
• Choice of outomes often very biomedical which may limit which interventions are trialled and thefore which are funded (e.g. NICE guidance). Some treatments are favoured over others e.g. complementary/alternative medicines focus on a holistic rather than a biomedical outcome hence lack of funding by pharmaceutical companies.
• Requires good faith on the part of pharmaceutical companies - potentially conflicting interests.

Question 11

What are the Philosophical Criticisms of EBM?

Answer 11

• Does not align with (most) doctors’ modes of reasoning (probabilistic versus deterministic causality) - shows us treatment works but no idea how treatment works (gap in knowledge).
• Aggregate, population-level outcomes don’t mean that an intervention will work for an individual.
• Potential of EBM (or its implementation, e.g. through NICE or clinical governance) to create ‘unreflective rule followers’ out of professionals - ‘you just have to follow guidelines to make patient better’
• Might be understood as a means of legitimising rationing, with potential to undermine trust in the doctor-patient relationship, and ultimately the NHS (patients may see doctors as accountants who focus on balancing the books - you deserve it, you don’t attitude)
• Professional responsibility/autonomy (Cookbook medics try to achieve best average outcomes for population as a whole which goes against principle of autonomy for each individual patient)

Question 12

Consider the problems of getting evidence into practice

Answer 12

• Evidence exists but doctors don’t know about it - dissemination ineffective? Doctors not incentivised to keep up-to-date?
• Doctors know about the evidence but don’t use it - why not? Habit? Organisational culture? Professional judgement?
• Organisational systems cannot support innovation - e.g. because managers lack ‘clout’ to invoke changes.
• Commissioning decisions to reflect different priorities e.g. what if patients say they want something else?
• Resources not available to implement change - financial or human resources: change management is a complicated and demanding process

Question 13

Describe getting evidence into practice

Answer 13

• Now widespread changes
• Partly due to policies such as clinical governance and the establishment of the care quality commission and NICE
• NHS organisations are now legally obliged to follow NICE guidance within 3 months of issue (individual clinicans not - yet?)
• Reluctance to fund things for which the evience is poor - may become even more prominent in a more austere NHS?

Question 14

Why do we need social research?

Answer 14

• So we can be more confident in answering questions about social life e.g. if women were told more about the dangers of smoking in pregnancy, they wouldn’t smoke

Why do doctors need to know more about social research methods?

• Policies and practices are based on social science research
• You have a responsibility as a doctor to be able to access, appraise and use this research
• Need to integrate and critically evaluate multiple resources (Tomorrow’s Doctors)

Question 15

What are the 2 main groups of methods of social research?

Answer 15

Quantitative

Qualitative

Question 16

What is meant by Quantitative Research? What are its strengths?

Answer 16

• Collection of numerical data e.g. X% of pregnant women smoke and Y% of these women would like to give up.
• Begins with idea/hypothesis (very prescriptive in approach) e.g. working-class women are more likely to smoke when pregnant than middle-class women
• By deduction allows conclusions to be drawn - about relationships between variables and sometimes about causal relationships.
• Strengths = reliability and repeatability.

Question 17

What are Quantitative Research Designs?

Answer 17

• Experimental study designs e.g. RCTs, cohort studies, case-control studies (all to isolate key factors of interest ‘signal’ from other contributory factors ‘noise’), cross-sectional surveys
• Secondary analysis of data from other sources:
• ​Official statistics e.g. census, Economic and Social Data Service
• Other national surveys: conducted by charities e.g. Joseph Rowntree Foundation, universities, think tanks, polling companies
• Local and regional surveys: e.g. those ocnducted by universities, NHS organisations and local councils (public health related)
• Questionnaires

Question 18

Describe questionnaires in quantitative research

Answer 18

Common method

Range of different applications:

• Measure of exposure to risk factors, effect of lifestyle and dietary factors on cancer
• Knowledge and attitudes e.g. sexual health
• Satisfaction with health services

Question 19

Describe questionnaire design

Answer 19

Questionnaires should be

• Valid: measure what they’re supposed to measure
• Reliable: measure things consistently (measuring same thing time after time again), differences in results come from differences between participants, not from inconsistencies in how items are understood or in how responses are interpreted

Question 20

How can you ensure validity and reliability in questionnaires?

Answer 20

• Published questionnaires may have been tested for validity and reliability - undergone peer review, published in journals
• Unpublished questionnaires are developed in specific contexts. Validity and reliability have to be established

Question 21

Describe the types of questions in questionnaires

Answer 21

Mainly closed questions e.g. Yes/No, strongly agree/agree/neither/disagree…, numeric scales, smiley faces

• Can be self-completed on paper or administered e.g. by telephone, interviewer, internet
• Offer ‘other - please specify’ option
• Can have open ended questions e.g. often free text box at end of questionnaire. You may need to give instructions, will take longer to complete. Plan in advance how to analyse - coding o free-text responses into new or existing categories, more qualitative analysis.
  *

Question 22

Describe pros and cons of quantitative methods

Answer 22

Good at

• Describing
• Measuring
• Finding relationships between things (e.g. ethnicity and smoking cessation)
• Allowing comparisons

BUT:

• May force people into inappropriate categories
• Don’t allow people to express things in the way they want
• May not access all important information
• May not be effective in establishing causality

Question 23

What is meant by Qualitative Research?

Answer 23

• Aims to make sense of phenomena in terms of meanings people bring to them e.g. why don’t people give up smoking? What is it like to live with rheumatoid arthritis?
• Need to listen to what people have to say, understand their perspective
• Emphasises meaning, experience and views of respondents
• Analysis emphasises the researcher’s interpretation, not measurement
• Can also provide insights into people’s behaviour (subject to caveats)

Question 24

What are different methods of qualitative research?

Answer 24

• Observation and ethnography
• Interviews
• Focus groups
• Documents

Question 25

Describe Observation and Ethnography

Answer 25

• Ethnography: studying human behaviour in its natural context
• Observe what people actually do, rather than relying on what they tell you (rationalisations?)
• Two forms: participant observation and non-participant observation
• The primary advantage of observational methods is to gain access to behaviour of which individuals themselves may provide biased accounts, or indeed be unaware. *People may be doing things without even being aware of doing them. *
• Observational methods allow the researcher to record the mundane and unremarkable (to participants) features of everyday life that interviewees might not feel were worth commenting on and the context within which they occur.
• Observation is labour intesive but can provide a valuable insight into what actually happens and can explosre aspects of health and healthcare that other methods cannot get at.
• Commonly combined with other more formal interviews and other soruces of data e.g. documents in ethnographic studies

Question 26

Describe Interviews

Answer 26

• Semi-structured
• Structured by prompt guide
• Clear agenda of topics - order not followed rigidly
• May seem conversational in style
• Emphasis on participants giving their perspective; interviewer should facilitate this.
• People can take the conversation in whatever direction they want - non-standardised.
• Detailed, focused accounts relating to an issue of interest - but gives someone’s professed views of or explanation of the issue, not an unproblematic description of the issue itself.

Question 27

Describe Focus Groups

Answer 27

• Flexible method - can be a quick method for establishing parameters (e.g. in order to develop questionnaires) or for accessing group-based collective understandng of an issue.
• Not so useful for individual experience
• May encourage people to participate
• Some topics may be too ‘sensitive’ for focus groups?
• Deviant views may be inhibited - may feel marginalised and not contribute => you lose their perspective
• Focus groups can be difficult to arrange. You have to consider membership of group e.g. consultants and healthcare assistants. Need a fairly homogenous group (for people to feel uninhibited and happy to contribute).
• Good facilitator is needed - managing group dynamics is not easy. Can be difficult to keep group on track!

Question 28

Describe Documentary and Media Analysis

Answer 28

• ​Independent evidence e.g. medical records, patient diaries But “artful reconstructions of the events they describe” - “somewhere between the worlds of observation and of interviewing”
• May provide historical context
• Useful for subjects difficult to investigate - ‘inside story’
• Can analyse television, newspaper and media stories e.g. media coverage of bottle and breast feeding.

Question 29

Describe analysis of qualitative data

Answer 29

• Very different from quantitative analysis
• Ongoing iterative (continuous, not waiting until you’ve collected all the data) - labour intensive

Offers an inductive approach:

• Close inspection of data (e.g. interview / focus group transcripts): reading and re-reading
• Try to identify themes
• Produce specification for themes
• Assign data to themes
• Constantly compare data analysis against the themes

Question 30

Describe the pros and cons of qualitative research

Answer 30

Good for

• Understanding the perspective of those in a situation
• Accessing information not revealed by quantitative approaches (depending on specific research, this extra information may not be needed)
• Explaining relationships between variables (e.g. why and how does ethnicity promote/discourage smoking cessation)

Less Good For:

• Finding consistent relationships between variables
• Generalisability (qualitative methods may be good at identifying a range of views on an issue but it would be dangerous to infer propensity of those views from a small sample that may not be statistically representative)

Question 31

Describe appraising qualitative research

Answer 31

• Like quantitative research, qualitative research should be carried out robustly.
• Some debate over most appropriate criteria for assessing the quality of qualitative research
• CASP (Critical Appraisal Skills Programme) offers one tool - rigour, credibility, relevance
• Transparency around sampling, methods and analysis is key.
• Good qualitative research will leave an audit trail

Question 32

How would you choose an appropriate study or design?

Answer 32

Depends upon

1. Topic under investigation and research question
2. Research team’s preferences/expertise
3. Time and money available
4. Funders and/or audience - what is this research expected to deliver? What is the funders’ agenda?

Different methods can be used in same study, especially if complementary