Health-Related Evaluation Methods

Question 1

What is the formative realist case evaluation example

Answer 1

Involved an evaluation of a difficult to describe large scale “transformation” of health services and its related report.

Question 2

What are the overall questions considered in all realist evaluations?

Answer 2

What works, for whom, under what circumstances?

What are transferable lessons about effective change?O

Question 3

Theory or logic theory whose aim is to explain how an intervention is understood to work. It used mixed data sources and collection methods of ethnographic observation, semi-structured interviews and document analysis

Answer 3

Realist context-intervention-mechanism theory

Question 4

Summarize the “lessons” that were reported related to Mechanism 1 from the formative realist case evaluation example

Answer 4

1. Relationships between organizations are trusting
2. Approaches to integration are imaginative, locally responsive, negotiable and supported by technology rather than rigid and driven by tech
3. External incentives are designed to reward collaborative performance and do not make organizations compete
4. The strategy for integration includes both “soft” and “hard” approaches
5. Solutions are participatory rather than developed by one party and imposed on others

Question 5

The generalizability of the findings in this type of formative realist case evaluation example can be increased by ___________________.

Answer 5

Providing details of the change actions and of the context

Question 6

The description of the mechanisms or change principles and how they were triggered by the intervention in the setting enables others to try to ________ the mechanism in their different settings, possibly using different ___________.

Answer 6

Reproduce

Interventions

Question 7

How does this type of formative realist case differ from many other forms of evaluation, particularly experimental evaluations?

Answer 7

Gives a narrative understanding of the process of change in an environment rather than showing a few measured outcomes attributed to a change

Question 8

What difficulties are involved with formative realist case evaluations?

Answer 8

1. Distinguishing between context and mechanism
2. Is there an end point?
3. When to “close”/
4. Time-consuming
5. Can realist evaluation studies be replicated?
6. How does realist evaluation differ from traditional mediator/moderator analysis?

Question 9

Describe the reasons to evaluate a reform before it starts, during its implementation and after its implementation

Answer 9

Assess whether it is feasible, to predict problems and to help plan how to implement it

Question 10

During a reform, an evaluation is to help make corrections _________ implementation

Answer 10

During

Question 11

After the reform, an evaluation helps to explain the lessons for _______ changes or reforms and ________ to scientific knowledge that could help the reformers explain, understand or predict the effects of _____________ changes.

Answer 11

Future
Contributes
Large-scale

Question 12

List and explain the usual steps for evaluation a reform

Answer 12

1. Describing the reform “instrument”
2. Gathering data about changes in health service performance and possibly also in health
3. Assessing whether these data really are outcomes of reform

Question 13

The aim of the reform is often to do the following

Answer 13

1. Present evidence of the results of the reform
2. Describe the process of implementing the reform
3. Assess the strengths and weaknesses of the reform as implemented and judge the value of the results
4. Recommend improvements

Question 14

In most reforms, the intention is to change the organization, procedures, financing methods and other “things”.

Answer 14

Immediate subject or target

Question 15

Reform that is often aimed to improve patients health or well-being

Answer 15

Long term aim

Question 16

Intervention that does not work directly on the _________ _________ (patients or the population) but works through changing the immediate subject (health organization) in order to achieve the ultimate benefits or better health.

Answer 16

Ultimate beneficiaries

Question 17

4 designs typically utilized to evaluate health reforms

Answer 17

1. Implementation designs
2. Comparing achievements with reform objectives
3. Before/after design
4. Comparing one site with one of more others implementing the reform

Question 18

Designs that report on strengths and weaknesses of the process and make explicit the criteria used to assess the process. For example, one criterion might be whether all stakeholder groups were involved.

Answer 18

Implementation designs

Question 19

Compares the intended goals of the reform with the extent to which they are achieved

Answer 19

Comparing achievements with reform objectives

Question 20

This design describes the reform implementation process but also gathers data before the reform and then after the reform has been proceeding for enough time to have an effect.

Common criteria are costs, efficiency, access, equity, quality and service comprehensiveness.

Answer 20

Before/after design

Question 21

This comparative design gathers data about different sites or areas implementing the same reform. It can be used cross-nationally to compare similar types of reform.

Answer 21

Comparing one site with one or more others implementing the reform