Evaluation of health services

Question 1

What is the definition of health service evaluation?

Answer 1

the assessment of whether a service achieves its objectives

Question 2

What framework do we use for health service evaluation?

Answer 2

Structure
Process
Outcome

Question 3

Give examples of structure

Answer 3

structure is what there is (buildings, staff, equipment)

• number of ICU beds per 1000 population
• number of vascular surgeons per 1000 population
• ratio of patients with diabetes to diabetic nurses

Question 4

Give examples of process

Answer 4

process is what is done

• number of patients seen in A+E
• number of operations performed
• waiting time in A+E
• how is priority assessed in A+E (in general, the process through which patients go through in A+E)

Question 5

Give examples of outcome

Answer 5

• mortality eg 30 day mortality rate
• morbidity eg complication rates
• quality of life/PROMs eg symptom scores, ADLs
• patient satisfaction

Question 6

In relation to CBT and mental health, give quantitative examples of structure, process and outcomes

Answer 6

structure: the ratio of patients with depression to available CBT appointments
Process: proportion of patients who complete the CBT course
outcome: percentage change in PHQ-9 questionnaire post CBT

Question 7

In relation to breast screening, give quantitative examples of structure, process and outcomes

Answer 7

Structure: the number centres offering breast cancer screening in South Yorkshire
Process: the proportion of patients accepting a screening offer who were invited to take part
Outcome: The number of breast cancer cases detected as a result of the screening programme

Question 8

In relation to hip replacement for OA, give quantitative examples of structure, process and outcomes

Answer 8

Structure: the ratio of orthopaedic surgeons to patients with OA needing hip replacement
Process: The number of hip replacements carried out per year in Sheffield
Outcome: complication rates of hip replacements in patients with BMI over 30

Question 9

In relation to management of diabetes in GP, give quantitative examples of structure, process and outcomes

Answer 9

Structure: the proportion of practices with a diabetic nurse specialist in Sheffield
Process: the proportion of patients who attend their diabetic reviews in one year
Outcome: the proportion of diabetic patients with foot ulcers in primary care in Sheffield

Question 10

What are the disadvantages of ‘outcomes’ (eg morbidity, mortality etc) in health service evaluation?

Answer 10

• link between cause and effect difficult to establish due to confounding factors
• lack of data
• poor data quality - completeness, relevance
• time lag between service provided and outcome may be long eg diabetic review and amputation, so difficult to measure

Question 11

Separate from structure, process and outcome in health services evaluation, we can also think of assessing the quality of care. What are Maxwell’s Dimensions of quality?

Answer 11

Effectiveness
Efficiency
Equity
Acceptability
Accessibility
Appropriateness (Relevance)

(3Es and 3As)

Question 12

How would you explain effectiveness?

Answer 12

Does the intervention / service produce the desired effect?

Question 13

How would you explain efficiency?

Answer 13

Is the output maximised for a given input (or is the input minimised for a given level of output)?

Question 14

How would you explain equity?

Answer 14

Are patients being treated fairly?

Question 15

How would you explain acceptibility?

Answer 15

How acceptable is the service offered to the people needing it?

Question 16

How would you explain accessibility?

Answer 16

Is the service provided? Geographical access; Costs for patients; Information available; Waiting times

Question 17

How would you explain appropriateness?

Answer 17

Is the right treatment being given to the right people at the right time?

Question 18

Outline some qualitative methodologies used to evaluate a service

Answer 18

Focus groups

Interviews

Question 19

Outline some sources of quantitative data for evaluating health services

Answer 19

Medical records
Routinely collected data - mortality, admissions
Surveys

Question 20

Overall, what is the general framework used for evaluating a health service, including structure, process and outcomes?

Answer 20

1. define what the service includes
2. aims/objectives of the service
3. structure, process and outcomes and Maxwell’s Dimensions of quality
4. methodology to be used (qualitative, quantitative)
5. results, conclusions and recommendations

Question 21

How would you evaluate one aspect of mental health services in Sheffield?

Answer 21

1. define - effectiveness of CBT
2. aims - return to work, reduction in antidepressant use, improve mood, save resources eg GP appointments, improve functioning in ADLs
3. structure - ratio of people with depression to CBT appointments; process - proportion of patients who complete the course of CBT; outcomes - percentage change in PHQ-9 post course
4. methodology: quantitative - compare percentage change in PHQ-9 to those who have CBT and those who don’t; qualitative - people’s experiences of CBT and effectiveness focus group
5. Maxwell’s dimensions - effectiveness - %change in PHQ9, efficiency - waiting times for CBT, equity - CBT access to deprived populations, acceptability - is CBT acceptable to pts; accessibility - where is it provided; appropriateness - are the right people receiving CBT or do they have complex mental health problems

Question 22

Other examples of structure, process and outcome from my lecture notes…

Answer 22

Quantitiave- needs a denominator in all cases

structure
- ratio of diabetic nurses to patients

Process

• Proportion of patients with type 2 diabetes that attend their diabetic review
• proportion of patients with type 2 diabetes going to DESMOND
• proportion of patients who are offered breast screening that turn up to their appointment

Outcome
- proportion of patients with type 2 diabetes with foot ulcers

Qualitative

• patient satisfaction with diabetic review
• interviewing pts on the helpfulness of district nurse support

Question 23

In terms of qualitative research, what is the difference between participant observation and non-participation observation?

Answer 23

Participant observation - people who provide the service are doing the evalulation

Non-participant observation - a researcher who does not provide the service is doing the evaluation

Think of it as - are the study participants also participants in the researchers service or not