Assessment

Question 1

1 - Which of the following statements is true with regards to the application of teaching and learning instruments?

• A trainee requires three satisfactory OSATS to be deemed competent to undertake a procedure
• Effective teaching, effective learning and effective assessment should be seen as part of the same educational process
• Oral assessments are best used for formative rather than summative purposes
• Oral assessments do not provide an assessment of the quality of the behaviour being assessed
• All of the above

2 - Which of the following statements is true with regards to workplace based assessments?

• They are mainly learner-led but can also be trainer-driven, depending on the circumstances
• They are opportunistic, not needs driven, unless there is a proper understanding between the trainer and trainee
• They are sufficiently reliable to stand alone as an assessment tool and do not need to be used together with endpoint high stakes ‘know how’ and ‘show how’ assessments of learning
• Weaker trainees will more likely seek feedback and therefore it is essential that WPBAs are done regularly and repeatedly
• None of the above

3 - Which of the following statements is true with regards to summative assessments?

• Either a summative or a formative OSATS form can be used to document a judgement on a trainee’s competence
• The primary purpose of a summative assessment is to judge a trainee as ‘able to perform competently’ or ‘working towards competence’
• The purpose of a summative assessment is to establish whether a trainee is competent to perform a procedure
• The RCOG suggests that any registrar or consultant can act as assessors in WPBA encounters
• Unlike formative OSATS forms, summative OSATS require a judgement relating to the competence of the trainee
• Which of the following statements is true with regards to formative assessments?

4 - A formative assessment of learning is known as a Supervised Learning Experience (SLE)

• A judgement regarding a trainee’s competence is required and its sole purpose is the provision of feedback and development of a learning plan
• Fomative assessments allow the trainer to deliver more honest feedback and for the trainee to reflect more readily in a positive manner
• Trainee learning at the end of a procedure is evaluated by comparing it against a standard or benchmark
• None of the above

Answer 1

1 - Correct answer:
All of the above

The answer is all of the above. OSCEs do not provide a complete profile of an individual’s level of competency or the quality of behaviour being assessed.

Formative OSATS are best used to provide practise and obtain feedback for a given procedure and to guide decisions about next steps in the learning process.

Trainees require a minimum of three procedures deemed competent per core procedure and by more than one assessor.

2 - They are mainly learner-led but can also be trainer-driven, depending on the circumstances

The answer is they are mainly learner-led but can also be trainer-driven, depending on the circumstances. Although in principle WPBA are learner-led (with the trainee identifying areas for observations and feedback), it is also important to appreciate that they can also be trainer-driven

3 - Unlike formative OSATS forms, summative OSATS require a judgement relating to the competence of the trainee

The answer is unlike formative OSATS forms, summative OSATS require a judgement relating to the competence of the trainee. Summative OSATS allow a trainee to demonstrate their competence in a procedure and progress in their training. Formative OSATS give a trainee the opportunity to practise and get feedback for a given procedure.

4 - Fomative assessments allow the trainer to deliver more honest feedback and for the trainee to reflect more readily in a positive manner

The answer is fomative assessments allow the trainer to deliver more honest feedback and for the trainee to reflect more readily in a positive manner.

Question 2

1 - With reference to the AMEE guide on setting standards in assessment (link is external), select the correct answer for the following questions.

• In a norm-references orientation, the standard is based on:
• What knowledge and skills the candidate is expected to have
• The knowledge and skills of selected appropriate panellists
• The mean performance of the candidates who took the last instance of the assessment
• The performance of an external large representative sample equivalent to the candidates taking the test
• The mean performance of the candidates who took this assessment

2 - When using standards setting to determine a passing score, the modified Angoff approach:

• Produces a cut-off score based on the proportion of borderline candidates who are expected to answer a question correctly
• Looks at the performance of borderline candidates
• Is a judgemental approach
• Requires an understanding of borderline characteristics
• All of the above

3 - When using a conjunctive standard, which of the following is false?

• None of the above
• Reliability of individual test components needs to be considered
• Candidates cannot compensate for poor performance in one test area with good performance in other areas
• The standard is set for individual components of a test
• Candidates do not get diagnostic feedback about their performance

4 - In a criterion-referenced orientation (also called standards based assessment), the standards are based on:

• The mean performance of an external large representative sample equivalent to the panellists setting the standards
• The content and the competence level under consideration
• The mean performance of the candidates who took the last instance of the assessment
• The performance of an external large representative sample equivalent to the candidates taking the test
• The mean performance of the candidates who took this assessment

Answer 2

1 - The answer is the performance of an external large representative sample equivalent to the candidates taking the test.

2 - The answer is all of the above.

3 - The answer is candidates do not get diagnostic feedback about their performance.

4 - The answer is the content and the competence level under consideration.

Question 3

1 - Regarding Bloom’s classification of learning, select the correct answer for the following questions.

• The domains of learning are:
• Cognitive, affective and skills
• Cognitive, affective and psychomotor
• Cognitive, psychomotor and skills
• Affective, psychomotor and skills
• None of the above

2 - The cognitive domain assesses:

• Skills
• Attitudes
• Knowledge
• Memory
• All of the above

3 - The affective domain assesses:

• Skills
• Attitudes
• Knowledge
• Responsiveness
• All of the above

3 - The psychomotor domain assesses:

• Skills
• Attitudes
• Knowledge
• Perception
• All of the above

Answer 3

1 - The answer is cognitive, affective and psychomotor.
2 - The answer is knowledge.
3 - The answer is attitudes.
4 - The answer is skills.

Question 4

Cognitive domain

This domain and its hierarchy relates to factual information and the way in which knowledge is applied.

Answer 4

Example

You see a woman in the outpatient gynaecology clinic who has a raised BMI of 45 who is troubled with menorrhagia.

The following hierarchy explains different steps of applying the cognitive domain to this woman.

1. Knowledge: Acquisition of the theoretical knowledge regarding the causes of menorrhagia and the assessment and management of cases with menorrhagia and a raised BMI.
2. Comprehension of concept: Understanding the science behind menorrhagia and how it could be associated with or be more common in women with raised BMI, endometrial pathology or secondary to fibroids.
3. Application: Applying this knowledge and comprehension to this particular woman in view of her risks, symptoms and clinical picture.
4. Analysis: Analysing this woman’s complaint regarding the degree of bleeding, pattern and associated pain, psychological impact etc. to try and reach a provisional diagnosis.
5. Synthesis: Formulation of a plan for investigating the woman, such as performing an ultrasound scan or an endometrial biopsy, and a proposal of management options.
6. Evaluation: Assessing how much this management plan would suit this woman and whether any alternatives might be more suitable for her.

Question 5

Affective domain

This domain relates to attitudes towards learning.

Answer 5

Example

You face a situation where a woman makes a complaint against you for miscommunication and being dissatisfied with the service you provided her. The following hierarchy explains the different steps of applying the affective domain to this situation.

1. Awareness of learning: Accepting the complaint and being open to the perception and views of the woman regarding the service she received versus the one she expected.
2. Responding: Going through the complaint with care and the intention of responding to it.
3. Valuing: Appreciating the woman’s experience and caring about what has been written by her.
4. Organisation: Translating the value into an action plan to address the woman’s concerns and solving them.
5. Acting consistently: Modifying your communication skills to avoid similar situations in the future, for example by attending communication courses.

Question 6

Psychomotor domain
This domain relates to skills – from initial perception through to mastery of the skill and, ultimately, to the development of new skills and techniques.

Answer 6

Example

You attend a skills lab on how to insert a Rusch balloon for the management of atonic postpartum haemorrhage. The following hierarchy explains different steps of applying the psychomotor domain for acquisition of such a skill.

1. Perceiving a skill: Attending the skills lab and attentively receiving the theoretical component and the demonstration for performing a Rusch balloon catheter insertion.
2. Appreciate the skill: Understanding how valuable the skill is in an emergency situation with uncontrollable postpartum haemorrhage and how important gaining the skill is.
3. Guided response under supervision: Attempting to perform the insertion of the balloon catheter into a mannequin under supervision and demonstrating how to place it in the uterus and inflate it with the appropriate amount of fluid.
4. Habitual skill: Performing the skill independently several times to gain enough confidence.
5. Mastery: Having enough confidence for applying this in a real clinical situation independently.
6. Creation: Applying knowledge and individualising management to each case and feeling confident to modify management as required for each case of postpartum haemorrhage.

Question 7

Workplace-based assessment in practice

Answer 7

Formative assessments (SLEs)
CbD
Mini-CEX
OSATS

Summative assessments (AoPs)
OSATS

Question 8

Examples of formative and summative

Answer 8

formative:
1 - Annual job appraisal
2 - Peer observation
3 - GCSE mock examination
4 - Part 2 MRCOG OSCE course station

Summative
1 - Driving theory test
2 - Part 1 MRCOG
3 - TOG article CPD questions
4 - IQ test
5 - Job interview

Question 9

True and false
1 - Quality feedback should focus strongly on what did not go well during an observed performance, so the trainee can work on their errors

2 - A trainee requests a formative (SLE) assessment encounter in opening and closing the abdomen, which goes better than expected. Therefore, they ask you to document it as a successful summative (AoP) encounter instead. You are able to accept the request as the trainee has demonstrated the requisite competence level

3 - All WPBA tools can be used in both formative and summative encounters, as long as the specific purpose of the encounter is agreed beforehand by the trainer and the trainee

4 - The changes to WPBA tools ensure that trainees have access to more detailed, documented feedback regarding their observed performance

Answer 9

1 - The answer is false. Quality feedback in the context of WPBA encounters identifies what went well and what could have gone better. It also requires a detailed learning plan highlighting the steps required for improvement.

2 - The answer is false. It is important to maintain the distinction between formative (SLE) and summative (AoP) assessments, as assessors will be looking for different things during each encounter. In this scenario, the trainee should be informed that the encounter will be documented as a SLE, but encouraged to request an AoP for the procedure in the near future.

3 - The answer is false. Only the OSATS tool will be used in distinct formative and summative encounters. The CbD and Mini-CEX tools will be used in formative encounters only.

4 - The answer is true. The principal justification for the trainees is to improve the provision of quality feedback, which is documented and can be reflected upon as the trainee’s progressively improves in a given area.

Question 10

Assessing knowledge (cognitive domain)

Answer 10

The following pages describes the types of assessments you can use to assess knowledge:

single best answer
multiple-choice and extended matching questions
essay questions
modified essay questions
oral assessment
OSCE examination.

Question 11

Grading of reflective ability (ATTITUDES)

Answer 11

This instrument has been developed to assess the student’s ability to reflect and to give a pre-oral grading based on the evidence presented in the portfolio.

The student’s ability to reflect can be assessed on three levels of increasing complexity:

level 1 – describes relevant evidence of progress from experience (should be achievable by the majority of students)
level 2 – evaluates relevant experience as evidence of progress (demonstrates a more complex level of achievement)
level 3 – identifies and re-evaluates one’s own learning needs (is the highest level of complexity of reflective ability).

Question 12

Assessing skills (psychomotor domain)

Answer 12

OSATS demonstration video
mini-CEX demonstration video
case-based discussion video
team observation/multisource feedback video
workplace-based assessment: a new approach to existing tools.