AlphaPlus

At AlphaPlus we have extensive experience of designing and developing OSCEs across a range of clinical and healthcare contexts. In this article, Assessment Manager Gemma O’Brien discusses some of the key principles underpinning effective design for these type of assessments.

What is an OSCE?

Objective Clinical Structured Examinations (OSCEs) are a common method of assessment for medical and healthcare students and professionals. They are typically used to assess clinical competence.

An OSCE assesses performance in a simulated clinical environment. Typically, candidates complete a series of timed activities (OSCE stations) in a circuit. This assessment format aims to ensure a consistent (or ‘objective’) experience for candidates. Each station has a trained assessor who assesses performance against standardised assessment criteria to ensure a consistent assessment experience for candidates.

• Some examples of OSCE use:
  • as a test of the minimum competence or accepted standards for entry to a medical profession (e.g. PLAB 2, NMC Test of Competence)
  • as a high stakes assessment tool for undergraduate and postgraduate medical qualifications (e.g. Royal College of Physicians in the UK)
  • as a formative assessment tool to provide feedback to medical and healthcare students (e.g. in undergraduate medical education)

What does an OSCE assess?

OSCEs mainly focus on the assessment of practical clinical skills such as taking a clinical history or carrying out an injection. However theoretical knowledge also plays an important role.

Miller’s (1990¹) pyramid of clinical competence is a framework used in medical education to describe the progression of a healthcare professional’s skills and abilities. It consists of four levels, arranged in a hierarchy, with each level building upon the one below.

Miller’s pyramid is used to emphasise the importance of progression from knowledge acquisition (‘knows’) to practical application (‘shows how’) and ultimately achieving clinical proficiency (‘does’).

At the lower levels of the pyramid, learners understand the theory that is the foundation of clinical competence. At the upper levels, learners integrate theory, psychomotor skills and professional attitudes, to perform as health professionals in different contexts. An example of this might be a scenario in which candidates have to assess a ‘patient’ with minimal information, and ask appropriate questions in order to elicit the diagnosis and show they know how to manage it appropriately.

Khan et al (2013²) draw a distinction between clinical ‘competency’ and ‘competence’. Competency is the combination of appropriate cognitive, psychomotor and affective skills, whilst competence is an attribute of a person. Khan et al (2013) argue that the performance of an individual on identical clinical tasks can vary considerably depending on the context of the assessment, and that the performance of a candidate in an OSCE might not be the same as their performance in the workplace on identical tasks. OSCEs should therefore be considered as a tool to provide a snapshot of candidates’ demonstrated performance in a particular area in a simulated environment. In real life, non-clinical skills such as leadership and team working play an important role in determining overall performance. An OSCE therefore assesses that a candidate is able to ‘show how’ (Miller, 1990) a candidate would perform in a simulated clinical environment.

While it may not be possible to simulate all aspects of a real clinical environment, OSCEs allow for the practical assessment of clinical skills, problem-solving abilities and knowledge whilst avoiding the inherent variation in real-life clinical contexts. The structured nature of the assessment goes some way towards avoiding bias in the assessment and ensures a consistent and ultimately, fair assessment experience for candidates.  OSCEs also allow candidates to be assessed on skills which may not occur in a predictable way in real life, such as the resuscitation of cardiac arrest. 

Validity

Validity refers to the extent to which an assessment measures what it is intended to measure. In other words, whether the OSCE effectively evaluates the clinical skills, knowledge and competencies it is designed to assess and is fair to candidates.

In the context of OSCEs, the following are crucial for effective assessment design:

• The OSCE must adequately cover the relevant content and clinical scenarios it is supposed to evaluate. The assessment must be as authentic as possible and representative of the domain it seeks to assess (such as the relevant curriculum or professional standards).
• The OSCE must be highly reliable, for example the marking must be standardised so outcomes are replicable and hence fair for candidates.
• The OSCE must effectively distinguish between candidates with different levels of clinical competence, especially ‘safe’ and ‘unsafe’ to practice.
• The results of the OSCE must align with other established measures of clinical competence.
• The consequences of the assessment must be sound, for example, the assessment should not promote negative learning effects.

Potential validity issues for OSCE assessments

A well designed OSCE can drive learning and have a positive educational impact. However, if OSCE stations are not designed to authentically recreate clinical scenarios or the tasks are compartmentalised or driven by checklist scoring, then this creates a risk that performance is not assessed holistically. Candidates can become overly focused on learning skills to pass examinations, rather than increasing their clinical competence (Miller, 1990; Shumway and Harden, 2003³; Khan et al 2013).

To avoid this, scoring approaches for OSCEs can use global rating approaches that focus on broad categorisations such as communication and patient safety alongside the specific aspects of clinical procedures being assessed (Khan et al, 2013). In addition, the supporting materials provided to candidates to prepare for the assessment should also be carefully considered to avoid rote learning of specific clinical scenarios.

An important facet of validity is reliability, that is, the extent to which an OSCE provides consistent or dependable results. Reliability must be carefully considered in the design of OSCE circuits as well as individual stations. Standardisation and training of markers and robust quality assurance and monitoring processes must be in place to ensure consistency of marking across candidates. Another aspect of reliability and consistency relates to the use of actors in OSCE stations. Actors can increase the authenticity of assessment by allowing candidates to interact with a real person in order to take a patient history, but actors must be carefully prepared and trained so that they respond to every candidate in a consistent way and do not inadvertently provide information that could alter the demand of the assessment.

Testing candidates across a large sample of clinical cases can provide insight into the breadth of their clinical competence and can increase the reliability of measurement. However, this must be carefully balanced against the need to ensure that the assessment duration is manageable for candidates and to ensure that success in the assessment does not become unduly time-bound.

We have wide experience of working on medical assessments for a variety of different organisations including designing and creating OSCE assessments for the NMC Test of Competence.

For more information and advice about how we can help you with OSCE assessments, please click here to contact us.

---

[1] Miller’s Pyramid of Professional Competence with examples of assessment… | Download Scientific Diagram (researchgate.net)

[2] Khan, K.Z. et al. (2013). The Objective Structured Clinical Examination (OSCE): AMEE Guide No. 81. Part II: Organisation & Administration. Medical teacher, 35(9), pp.e1447–e1463.

[3] Shumway JM, Harden RM (2003). AMEE Guide No. 25: The assessment of learning outcomes for the competent and reflective physician. Med Teach 25:569–584.