Monitoring and evaluation

Question 1

How does OECD define evaluation?

Answer 1

The systematic and objective assessment of an ongoing or completed project, programme or policy, its design, implementation and results.

Question 2

What are the aims of an evaluation?

Answer 2

To determine:

• the relevance and fulfilment of objectives
• development efficiency
• effectiveness
• impact
• sustainability

Question 3

Why is an evaluation important?

Answer 3

• It is a useful tool to gather info to improve ongoing and future projects and the services offered to stakeholders and beneficiaries, as well as for generating ideas and innovation for future projects.
• Can provide evidence-based assessments of the worth or significance of a project
• Can provide project management with recommendations for improving the performance and for best use of resources
• Indicates not only that information on the performance of a particular project will be available, but also that the agency is committed to improve performance and maximize results
• Promotes transparency and accountability which, in turn, assist donors in their decision-making and about future project funding and can increase stakeholder buy-in

Question 4

What types of evaluation are there and how are they defined?

Answer 4

They are defined by the timing of the evaluation - the purpose - who conducts it - and the methodology used.

Timing: ex-ante, real-time (early stages of implementation), midterm, final, ex-post.

Actors involved: internal vs external / Individually vs Jointly

Question 5

What is a formative evaluation?

Answer 5

It is an evaluation conducted during implementation for the purposes of improving performance. It is intended to help Project Managers adjust and improve project implementation based on the evaluator’s findings and stakeholders’ suggestions and needs.

Real-time evaluations are linked to formative evaluations - similar to performance monitoring.

Question 6

What is a summative evaluation?

Answer 6

It is conducted at the end of a project, for the benefit of stakeholders not directly involved in the management of the implementation (e.g. donors). It provides insights about the effectiveness of the project and gives the stakeholders the opportunity to use best practices identified during the evaluation.

Question 7

What is an impact evaluation?

Answer 7

It attempts to determine the entire range of the project, including the positive and negative, primary and secondary long-term effects produced by the project - directly or indirectly, intended or unintended. Often ex-post.

Question 8

What is an outcome evaluation?

Answer 8

An attempt to assess the extent to which a project was successful in producing change (e.g. if the indented institutional changes were produced by a project)

Question 9

What is a process evaluation?

Answer 9

It assess how the results of a project were achieved and the different systems put in place for implementation (e.g. if participatory methods were used; if the organisation implementing the project responded effectively to change)

Question 10

When are evaluations planned?

Answer 10

1. During project development: the purpose, timing, intended use and methodology of the evaluation must be planned + a budget line for evaluation in the proposed budget, which means that consultants fees or staff costs, travel, data collection costs and data analysis costs must be included.
2. Prior to conducting the evaluation.

Question 11

Who can do an evaluation?

Answer 11

1. An external individual or firm who is recruited to conduct the evaluation.
2. An internal independent evaluator, who is a staff member from outside of the implementing office, not involved in the design or implementation of the project.
3. Internal self-evaluation from staff involved with the project.

Question 12

What does planning involve when it comes to external evaluators?

Answer 12

Estimating how many days the evaluator will require for:

• initial document and literature review
• travel
• initial meetings with the management team
• preparation of the inception report
• data collection and analysis
• presentation of initial findings
• preparation of the draft report
• revisions and finalization of the evaluation report

Question 13

What is an inception report?

Answer 13

An inception report of an evaluation is prepared by an evaluator after an initial review of relevant documentation. It sets out the conceptual framework to be used in an evaluation, the key evaluation questions and methodology, including information on data sources and collection, sampling and key indicators. The inception report also includes a timeline for the evaluation project and drafts of data collection instruments.

Question 14

What is important in the planning stage when using an internal independent evaluator?

Answer 14

1. Travel expenses
2. Salary
3. Duration of assignment, and budget

Question 15

What is the key planning tool prior to the implementation of the evaluation?

Answer 15

TOR - terms of reference.

It should build on the work done by the project developer during the initial planning stage and elaborate it in greater detail.
It should provide clear and detailed specifications of the roles and responsibilities of the parties involved in the evaluation.
In case of external evaluators, the TOR is important element of the contract.

Question 16

What should the TOR include?

Answer 16

• Evaluation context: political, economic, social, environmental, legal context + project itself.
• Evaluation purpose: why is the evaluation conducted, why is it conducted at that time, who are the intended audience, how will it be used.
• Evaluation scope: What will the evaluation cover, which time period will be assessed, which components or phases, which geographical area, should something be excluded.
• Evaluation criteria: the criteria that will be addressed.
• Methodology: the method for data collection and analysis.
• Deliverables: the products that should be generated at various stages of the evaluation process + responsibility, e.g. inception report, initial presentation of findings, draft evaluation report, final evaluation report.
• Time schedule: the dates by which tasks have to be done or products have to be handed over + responsibility
• Role specification of the various parties involved
• Budget: details the resources available to conduct the evaluation + the cost of each item.

Cross-cutting themes should be addressed throughout the TOR.

Question 17

Which evaluation criteria are widely adopted in development- or humanitarian oriented projects?

Answer 17

1. Relevance: the extent to which a project’s objective and intended results remain valid and pertinent. Focused on intentions of the project rather than the actual achievements. Examines the relationships between inputs, activities, outputs, objectives + if the hypothesis and assumptions remain valid.
2. Efficiency: how well human, physical and financial resources are used to undertake activities and how well these resources are converted into outputs. HOW the outputs are realized, not their effects.
3. Effectiveness: assess the extent to which a project achieves its results. If activities have led to outputs which have led to outcomes which has realized objective. Considers if beneficiaries have used the services and products provided and if this has led to intended outcomes. All internal and external factors that influenced the achievement of outputs and outcomes.
4. Impact: the positive, negative, primary, secondary long-term effects produced by a project, directly or indirectly, intentionally or unintentionally.
5. Sustainability: The durability of the project’s results or the continuation of the projects benefits once external support ceases. Ex-post.

In humanitarian interventions:

connectedness: extent to which activities of short-term emergency nature are conducted in a manner that takes longer-term and interconnected problems into account
coverage: extent to which humanitarian assistance reaches all major populations affected and the impacts of humanitarian actions on different population and subgroups
coherence: extent to which there is a consistency across policies (security, developmental, trade etc), and the extent to which all policies take into account humanitarian and HR considerations.

Question 18

What does the evaluator do?

Answer 18

1. Document review of e.g. reports, deliverables, national strategies and plans, project documents, and any other documentation deemed relevant.
2. He or she submits a draft inception report, or evaluation matrix, which provides a detailed description of their understanding of the TOR and how they will conduct the evaluation
3. Data collection
4. Analyse data

Question 19

Quality control during an evaluation

Answer 19

1. After receipt of the inception report; assess work, correct understanding
2. During data collection; periodically monitor, check up on meetings and surveys and target numbers, ensure logistical arrangements
3. During the presentation of initial findings; identify and address misinterpretations or gaps
4. After receipt of the draft evaluation report; it should be ensured that the report describes the project and context, the eval. purpose and scope, the methodology and data sources and limitations, that findings are presented clearly and based on evidence, that conclusions and recommendations are stated clearly and based on evidence

Question 20

How to use evaluations?

Answer 20

1. Disseminate evaluation findings; learn from them, circulate reports widely
2. Incorporate findings into practice; review to identify lessons learned and good practices

Question 21

Stages in the implementation phase

Answer 21

1. Project planning
2. Project start-up
3. Project execution
4. Project closure

Question 22

Planning for project implementation

Answer 22

Plan and monitor:
activities
results
budget
risks

Lack of planning and monitoring –> extensions, underspending or overspending, unachieved or underachieved results, poor reporting, negative evaluations, risk of relationships and reputation

Question 23

Detailed work plan

Answer 23

A simple tool to plan and monitor a project’s activities. Developed already at the project proposal stage by the project developer. The PM then takes the same workplace and makes it more detailed. Should include not only the project activities identified in the results matrix but all activities and tasks related to all phases of implementation + a section for ongoing monitoring, evaluation and reporting activities + additional sub-activities/tasks, assigned to staff member, so detailed concrete levels of each activity.
Balance in level of detail.

So: Activity + sub-activities/tasks
Responsible person
Status
Comments
Time plan & sequence

Question 24

What are activities or tasks that need to be completed before others can be undertaken called?

Answer 24

Sequential or linear.

Question 25

Extensions and budget revisions

Answer 25

• If P.M. identifies the need of extension, budget reallocation, revision to the results matrix or workplan –> approval from staff and donor.
• If granted, additional or revised documentation are to be shared with all parties.

Question 26

What happens in project start-up phase?

Answer 26

1. The team is established + recruitment
2. Agree on structure (organigram), coordination, communication mechanisms - internal and external
3. For large-scale multicomponent multiagency projects, a steering committee might be useful
4. Launching of project

Question 27

How can project managers support staff development and training?

Answer 27

1. Supporting member’s participation in development and learning activities
2. Encouraging active commitment to continuous learning
3. Budgeting development and learning initiatives in project proposals
4. Discussing and agreeing on learning and development plans, incl. identifying new responsibilities for professional growth
5. Ensuring that members apply and share the skills gained

Question 28

Why is donor reporting important?

Answer 28

1. Enhance relationship
2. Donors become more willing to accept budget reallocations, extensions, requests
3. Transparency and accountability
4. Important for auditors and evaluators to assess whether the project has followed regulations and rules + its efficiency and effectivity
5. Agencies are contractually obliged to report
6. Facilitates monitoring
7. Encourages analysis of challenges encountered

Question 29

Reporting requirements

Answer 29

• It’s normally set out in the donor agreement
• Frequency and deadlines decided or discussed
• Format of the narrative and financial reports; either required by donor, or agencies own templates
• Language

Question 30

What are the two main types of donor reports?

Answer 30

Interim reports and final reports

Question 31

What is narrative description and financial report?

Answer 31

Narrative description: the activities undertaken, an analysis of data presented, an assessment of the results achieved, and a description of the challenges faced in the implementation of a project and the corrective actions undertaken

financial report: project expenditures, funds received, and any remaining balance, as well as interest income if specified in the agreement. They include, where relevant, a description of in-kind contributions received by the project and report, where possible, their monetary value

Question 32

Steps in drafting donor report

Answer 32

1. Documents review; make sure the most recent version of the results matrix is used, any changes are adhered to, review previous reports and donor agreements
2. Drafting narrative report: use correct template, do not change template, 5-10 pages for interim report, no more than 15 pages for final reports.

Question 33

Common section in narrative reports

Answer 33

1. Summary table with info such as: Project title, duration, budget, geographical coverage, burn rate, reporting period, partners involved.
2. Summary of key achievements during the reporting period
3. Progress made towards realizing outcomes and outputs;
   - Details on implementation of activities leading to progress or realization
   - Clear indication of which activities were conducted jointly or entirely by partners, stakeholders, consultants or beneficiaries
   - Description of coordination mechanisms between partners, stakeholders, consultants, beneficiaries
   - Description of any revisions made to original project documents, e.g. the Results Matrix
4. Progress achieved compared with the indicators in the results matrix: a tab similar to the logical framework, but with two additional columns; progress made during the reporting period and cumulative project progress
5. Challenges encountered and action taken, including analysis of assumptions that did not hold true,
6. Conclusion. Summarize. Focus on the future.
7. Annexes.