Woolf, KVM; Page, M; (2018) Academic support for the Assessment and Appraisal Workstream of Health Education England’s review of the ARCP. Research Department of Medical Education, UCL Medical School: London, UK.

Preview

Text 2017-10-04-UCL-HEE-ARCP-review-FinalReportSubmitted.pdf - Published Version Download (1MB) | Preview

Abstract

1. Executive summary 1.1 Background This report was prepared for Health Education England (HEE) by the Research Department of Medical Education at UCL Medical School. Its purpose is to inform the recommendations of HEE’s Annual Review of Competency Progression (ARCP) Review Assessment and Appraisal Workstream in relation to a number of aspects of the ARCP process. This report aims to answer the following questions set by the Assessment & Appraisal Workstream:  How can the summative components of the ARCP process be standardised, and what measures can be taken to ensure that the process is robust, reliable and valid?  How can the appraisal elements of the ARCP process be standardised, ensuring that appropriate formative feedback is provided to all trainees?  How can ARCP processes be aligned to GMC revalidation requirements?  What understanding do trainees and supervisors have of the educational principles that underpin the summative and formative elements of ARCPs? In terms of the summative elements of the ARCP we considered: the criteria against which panels make decisions, which evidence panels use to make decisions, the ability of the ARCP to distinguish between different levels of trainee performance and to identify patient safety concerns, the reliability and fairness of the ARCP, and the impact of the ARCP on trainee learning. In terms of the appraisal elements of the ARCP we focused on the feedback given to trainees by the panel, and how trainees are prepared for the ARCP panel. We also considered the relationship between the ARCP and medical revalidation, and trainee and supervisor perceptions of the ARCP. 1.2 Methods We started with a scoping review of the peer-reviewed and grey literature on the ARCP to identify evidence regarding the validity of the summative and appraisal aspects of ARCP, the ARCP and revalidation, and trainee and supervisor perspectives on the ARCP. We then reviewed the official national guidance relating to ARCPs, and some other local or specialty-specific guidance that we were aware of or had found via non-systematic Google searches. We then drew together the literature findings and guidance with evidence from the wider medical education and mainstream education literature, to provide suggestions for increasing the validity, reliability, standardisation, and robustness of the ARCP as a tool to assess and drive trainee progress, to revalidate trainees, and to improve trainee and supervisor understanding of the ARCP. 1.3 Results The overarching suggestion from this report is that the ARCP panel is recognised as a high-stakes assessment that is likely to have a significant impact on patient care. The ARCP panel should therefore be subject to the same scrutiny and psychometric considerations as other high-stakes assessments, such as medical school finals and postgraduate examinations. 5 Report findings and suggestions are summarised under headings relating to the research questions. Major findings and suggestions are presented first, followed by a summary of the more detailed findings and suggestions, as appropriate. 1.3.1 How can the summative components of the ARCP process be standardised, and what measures can be taken to ensure that the process is robust, reliable and valid? 1.3.1.1 Major findings and suggestions Major finding: The guidance lacks consistency and detail about how panels should make decisions. This is a threat to the standardisation, reliability, and validity of the ARCP. Major suggestion: The guidance to provide more detailed and practicable information about the information panels should use to make decisions, and how panels should go about making decisions in practice (including that panel decisions should be made without the trainee present). Decision-aids to be developed that are applicable nationally, and consistent in quality across specialties. Major finding: There are concerns that ARCPs only identify very poorly performing trainees and fail to identify other trainees with specific or less serious performance issues. This may result partly from the ‘failure to fail’ phenomenon. Major suggestion: Minimise failure to fail by having panels provide constructive feedback for all trainees post-decision; by providing time for panels to discuss trainee performance; by having panels consider multiple sources of evidence not just the Educational Supervisor’s Report; by improving support for trainees who are failed; and by introducing lower-stakes formative pre-ARCP reviews. Major finding: There is a lack of empirical research and published evaluation of the ARCP, and ARCP data provided for research has sometimes been of poor quality. This makes it difficult to assess the quality of the ARCP as an assessment. Major suggestion: Collect and provide good quality data for rigorous research and evaluation. 1.3.1.2 Summary of more detailed findings and suggestions Finding: The guidance is unclear about the basis upon which panels should make decisions, leading to inconsistency and threatening validity. Suggestion: In order to develop decision-aids that are applicable nationally, comparable across specialties, and that help panels make valid and robust decisions, the following needs to be clarified in the guidance:  Expectations around trainee progression and performance.  How panels should weight different pieces of information (research is needed to discover what panels are doing currently, and the impact that weighting can have on outcomes).  How panels should take into account any contextual and environmental factors that affected a trainee’s performance, whether positively or negatively.  The purpose of the additional information sought and the processes to be followed when a trainee is anticipated to receive an unsatisfactory outcome; and how the additional information should be used by panels to improve the validity of their decision-making.  Expectation that assessments submitted in the portfolio are sampled across the curriculum and assessors. 6  Requirement that Educational Supervisors be absent when panels are making decisions. If a trainee is present at panel, this will not influence panel decision-making (i.e. decisions to be made in absentia). Consider having all trainees submit written evidence stating whether they support or disagree with the Educational Supervisor’s report to lessen the conflict between the Educational Supervisor’s dual roles as mentor and assessor. Suggestion: To aid the development of decision-aids, it may be helpful to map all UK local and specialty-specific ARCP guidance to draw out the similarities and differences in the competencies required and the information provided about how panels should make judgements. Following this, a Delphi process (or similar) could be performed to achieve consensus about how panels should make decisions in practice. This could be done by specialty or by families of specialities. Suggestion: Newly-developed decision aids to be compared and reviewed to ensure consistency across specialties and locations, and then piloted to ensure they are practicable. Finding: There is a lack of evidence about whether the ARCP is able to reliably distinguish between satisfactory and unsatisfactory performance, or between different levels of unsatisfactory performance. Trainees and trainers are sceptical about the ARCP’s ability to identify anything other than extreme poor performance, feeling it is not able to identify patient safety concerns but may be overly harsh to trainees with protected characteristics. Medical education research from outside the UK demonstrates reliable judgements about a trainee’s competence can be made on a basis of a several workplace based assessments sampled across curriculum areas and assessors. Suggestion: Collect and provide good quality data for rigorous research and evaluation regarding the validity and reliability of the ARCP. Ensure research is published. Suggestion: Put in place training and support for supervisors and panels to fail trainees if necessary. This includes: normalising ‘remedial’ training by providing constructive feedback for all trainees (including stretching feedback for trainees performing well); better information for supervisors, panels, and trainees about support provided to trainees receiving unsatisfactory outcomes, especially for those receiving an Outcome 4; providing sufficient time for panels to discuss each trainee’s performance; ensuring panels consider multiple sources of evidence not just the Educational Supervisor’s Report; having formative relatively low-stakes pre-ARCPs for all trainees and/or using other tools to help the early identification of problems. Finding: Panels make more reliable judgements than individuals, but steps need to be taken to guard against problems that can arise from group decision-making. Suggestion: Panels to agree on their roles and their expectations of trainees at the start (potentially including clearer role expectations and Terms of Reference for panel members); proformas to aid structured decision-making; sufficient time for panel members to discuss each trainee, with panel chairs regularly summarising and ensuring panel members contribute and share as much information as possible. More panel members are recommended (with the proviso that all should be fully engaged) with three being a minimum. Finding: On average IMGs, male doctors, older doctors, and doctors from black and minority ethnic (BME) backgrounds are more likely to have an unsatisfactory ARCP outcome. It is unclear whether 7 this reflects panel decision-making or other factors. Unconscious bias training may not be sufficient to combat the potential for unfair bias. Suggestion: Panels to explicitly state their commitment to ensuring decision-making is fair during the ARCP. Panel membership to be monitored, efforts made to ensure diversity, and monitoring undertaken to check for any unfair bias. Panels to have greater consideration of the impact of training environments or other external factors on trainee progression and performance. Finding: Many trainees feel demotivated by the fact that it is impossible to achieve an ARCP outcome higher than ‘Satisfactory’ and see it as discouraging excellence. Suggestion: Consideration of how the ARCP process can recognise excellence. Constructive feedback for all trainees, including ‘stretching’ feedback for those performing well. 1.3.2 How can the appraisal elements of the ARCP process be standardised, ensuring that appropriate formative feedback is provided to all trainees? 1.3.2.1 Major findings and suggestions Major finding: At present, only trainees expected to receive an unsatisfactory outcome are required to talk to the panel to discuss their performance, although the limited literature suggests in practice most trainees receive some kind of feedback from the panel, and many receive this in person. There is little guidance about the format in which feedback should be provided or what feedback should be about (e.g. past performance or future performance). There appears to be considerable variation between specialties and geographic locations, which is likely to hinder the validity of the ARCP. Major suggestion: Ensure all trainees receive constructive feedback to improve their learning and performance, including trainees who are progressing satisfactorily. Provide guidance to standardise the way in which trainees receive feedback across specialties and locations. Ensure trainees and panels know panel decision-making will not be influenced by a trainee’s attendance. Major finding: There is no national guidance relating to preparing trainees for ARCPs. In practice, it seems different LETB’s/Deaneries provide different types of pre-ARCP meetings for different groups of trainees. This lack of standardisation is likely to hinder the validity of the ARCP. Major suggestion: Ensure all trainees have a pre-ARCP meeting with their Educational Supervisor and another person, possibly an ARCP panel member, to check progress and provide feedback. Provide guidance to standardise the pre-ARCP meeting process. 1.3.2.2 Summary of detailed findings and suggestions Finding: All trainees are likely to benefit from feedback. Research on appraisal suggests the benefits depend on appraiser and appraisee factors, but it is uncertain how relevant this research is to panels providing feedback. ARCP prioritises summative assessment in the form of the Educational Supervisor Report and assessments of trainees by seniors, but peer- and self-assessment is important for learning. Suggestion: All trainees to receive constructive feedback post-ARCP panel, including stretching feedback for those performing well. Standardised guidance on feedback to be developed and evaluated. Panel members to be trained to provide feedback. Panels to consider peer and self-assessment as well as assessor by seniors. 8 Finding: Neither the Gold Guide nor the Foundation Programme Guidance describe in any detail how the trainee should be prepared for the ARCP. The literature provides various ways to prepare trainees for the panel, such as ensuring trainee portfolios meet up-to-date curriculum requirements and having interim/pre-ARCP panels. Suggestion: Formative interim/pre-ARCPs for all trainees, which focus on providing constructive feedback and feedforward, and on identifying any problems a trainee is having, including any contextual or environmental factors affecting their progression. A degree of externality, possibly from a panel representative, is likely to be beneficial. Suggestion: Explore the potential for developing a standardised tool (adaptable to local contexts), to help trainees and supervisors track achievement over the course of the year and map these to curricular requirements. Finding: Lack of standardised support for trainees receiving an Outcome 4, despite many of these doctors continuing to practice medicine. Potentially risky for patient safety. Also likely to increase failure to fail. Suggestion: Greater educational and career support for trainees receiving an Outcome 4. Trainees, panels, and supervisors to have more information about the support for trainees who are failed in order to combat failure to fail. 1.3.3 How can ARCP processes be aligned to GMC revalidation requirements? 1.3.3.1 Major findings and suggestions Major finding: Lack of clarity about how revalidation decisions and progression decisions relate to one another. Lack of consistency between revalidation for trainees and non-trainees (e.g. consultants, staff grade doctors). The former is based on a summative ARCP and progression against a curriculum, the latter is based on multiple formative appraisals and considered against Good Medical Practice. Major suggestion: Clarification in the guidance as to how the processes for determining progression and revalidation relate to one another. Trainees could receive feedback from the ARCP panel that is aligned with the major domains of Good Medical Practice, to help align revalidation during and after training. 1.3.4 What understanding do trainees and supervisors have of the educational principles that underpin the summative and formative elements of ARCPs? 1.3.4.1 Major findings and suggestions Major finding: The limited literature suggests many trainees feel the an annual review is valuable in principle, but have serious criticisms of the ARCP, perceiving that it does not provide meaningful feedback, that it can discourage excellence, and that it is not sensitive enough to pick up anything but very poor performance. Major suggestion: Communicate to trainees how the ARCP review aims to:  ensure all trainees receive constructive feedback in preparation for the ARCP and after the ARCP, including feedback to stretch trainees who are already progressing satisfactorily to encourage excellence, and by finding ways for process to recognise excellence; and by providing 9 interim/pre-ARCPs and developing tools to help trainees and supervisors track a trainee’s progression and identify problems early.  improve the reliability and validity of the ARCP (and thereby contribute to patient safety) by standardising panel decision-making across specialties, grades, and locations; by increasing the panel’s consideration of environmental and contextual impacts on a trainee’s progression; by reducing the impact of a single Educational Supervisor’s report on outcomes; by ensuring panels make decisions without the Educational Supervisor or the trainee present; by ensuring rigorous and transparent evaluation of the ARCP. 1.4 Conclusions We found relatively little published research on the ARCP, and much of the evidence was small-scale. This reflects the fact that, despite its high-stakes nature, the ARCP panel is not officially an assessment and does not receive the same scrutiny as other high-stakes assessments in medical education. Our primary suggestion therefore that is the ARCP panel is officially recognised as a high-stakes assessment likely to have a significant impact on patient care and subject to the same scrutiny as other high-stakes assessments. The lack of standardisation in ARCP processes is concerning, and we have suggested much greater clarity in the guidance in a number of areas to address this. To prevent the proliferation of local guidance and the concomitant threat to validity, we have suggested guidance is standardised and then piloted to determine feasibility in practice. We have also suggested that the relationship between the ARCP and revalidation, and the appraisal elements of the ARCP, are both more clearly defined and communicated. The ARCP is generally perceived negatively by trainees, who feel it does not provide meaningful feedback, that excellence is not rewarded, and only the poorest performance is identified. We have suggested a number of changes designed to improve the validity of the ARCP by making it more meaningful. Communicating with trainees about the changes being made to the ARCP and the rationale for those changes is likely to be crucial to restore confidence in the assessment. Finally, we firmly believe in the principle that ‘assessment is not easy to develop and is only as good as the time and energy put into it’ (1) (p.707). We recognise that investment in the ARCP is particularly challenging in the current circumstances, but we believe it is worthwhile. As Eva et al (2) point out: ‘It seems antithetical to the very reasoning behind assessment (the protection of patients) to suggest that we should not think about how to improve current assessment practices, not only in terms of their role in gatekeeping but also in terms of their opportunities for shaping further learning’ (p.907). We suggest investment in undertaking proper and continual evaluation of the process and outcomes of the ARCP, including any changes made, is essential to ensure the validity, reliability, robustness, and defensibility of the ARCP and its role in postgraduate training.

Download activity - last month

Export as CSVJSONXML

Download activity - last 12 months

Export as JSONCSVXML

Downloads by country - last 12 months

Export as XMLJSONCSV

Archive Staff Only

View Item