Case 6: Evaluation of policy measures: how to go about it?

Question 1

How can we identify which goals need to be met?

Answer 1

When interventions are developed often times a needs assessment is made to identify what goals need to be met → these goals need to be clearly defined

Question 2

What research designs & data collection techniques are used to evalute policy?

Answer 2

• RCTs
• Quasi-experimental studies
• Survey studies
• Interviews
• ETC?

Question 3

What practical issues do policy evaluations need to take into consideration?

Answer 3

• Resource scarcity
• Organisational complexity
• Stakeholder participation,
• Transparency
• Context
• Skill of the researchers involved

Question 4

What is evaluation?

Answer 4

• Process of analysing policies & context in which they occur to determine whether changes need to be made in implementation
• to assess intended & unintended consequences of programs & policies

Question 5

What are the methods of evaluation?

Answer 5

• strongest approaches include a mixed methods approach as this provides a comprehensive review of evaluation

there is more info here i think

Question 6

Why is evaluation relevant?

Answer 6

• forms basis for making choices when resources are limited
• helps in making midcourse corrections & improving programs

Question 7

What is a policy evaluation?

Answer 7

Assess operation & impact of (public) policies & action programmes introduced

Question 8

What is the difference between research & evaluation?

Answer 8

Research is a systematic process for generating new knowledge and relating it to existing knowledge in order to improve understanding about the natural and social world → this is different to evaluation even though evaluation uses research methods

Question 9

What is evidence based policy?

Answer 9

movement within public policy to give evidence greater weight in the shaping of policy decisions → integrates experience, judgement and expertise

Question 10

What is tool 45 of the EU?

Answer 10

1. What is an evaluation
2. Principles of objectivity & independence

Question 11

Explain box 1 (what is an evaluation) of tool 45

Answer 11

Evaluation –> evidence-based judgement of the extent to hwich an exisiting intervention is:
- effective in fulfilling expectations & meeting objectives
- efficient in cost-effectiveness & proportionality of acutal costs to benefits
- relevant to current & emerging needs
- Coherent internally & externally (with other EU interventions/international agreements)
- has EU added value (produces results beyond what would have been achieved by MS acting alone)

Question 12

What is the aim of evaluation?

Answer 12

aims to draw conclusions about the causal effects of the (EU) intervention on the actual outcome/results

Question 13

What is relevant about tool 45?

Answer 13

Goes beyond factual assessment of what has happened & considers:
- why something has happened
- how much change can be attributed to EU intervention
- to what extent this change meets original expectations/projections

Question 14

What is tool 46?

Question 15

What is important according to tool 46 when designing an evaluation?

Answer 15

1. Clarify purpose of evaluation
2. Define the scope
3. Explain intervention logic
4. Draft good evaluation questions
5. Identify appropriate points of comparison
6. consider appropriate data collection & analytical methods

Question 16

Explain “clarify the purpose of the evaluation” when designing an evaluation from tool 46

Answer 16

Deciding & clearly describing what the evaluation will deliver & how its findings will be used.

Question 17

Explain “define the scope” when designing an evaluation

Answer 17

• setting out clearly what will be evaluated.
• Can be in terms of interventions, time period, geographical coverage, particular effects, etc
• Should understand what will be covered by evaluation & what won’t & why.

Question 18

Explain “explain the intervention logic” when designing an evaluation

Answer 18

• summarising how the intervention was expected to work (i.e. at the time of adoption by the Commission or later by the co-legislators, or at the time of implementation), including the underlying assumptions.
• The intervention logic can draw from any prior impact assessment or other documentation such as the explanatory memorandum, which justified the initial policy action.

Question 19

Explain “draft good evaluation questions” when designing an evaluation

Answer 19

• They should address the five evaluation criteria and any other aspect as relevant, also considering the feedback on the ‘call for evidence’ to the extent possible.
• Questions should cover all issues that are known to be of interest to the stakeholders.

Question 20

Explain “consider approriate data collection & analytical methods” when designing an evaluation

Answer 20

it’s important that the evaluation is set up to collect and analyse a range of different data, using the appropriate data and methodologies to fill existing data gaps and to robustly answer the evaluation questions

Question 21

**What is tool 47?

incomplegte

Answer 21

Instrumental
Conceptual
Symbolic

Question 22

Explain box 1 of tool 45

Answer 22

• box 1 = What is an evaluation:
• Goes beyond assessment of what’s happened & considers why something has happened, how much change can be attributed to EU intervention & to what extent this change meets original expectations/projections
  ● Evaluation aims to draw conclusions about the causal effects of the (EU) intervention on the actual outcome/results

Question 23

Explain box 2 of tool 45

Answer 23

• box 2 = principles of objectivity and independence
• DGs may focus on objectivity and independence