Newman, M; Kwan, I; Schucan Bird, K; Hoo, HT; (2021) The impact of Feedback on student attainment: a systematic review. Education Endowment Foundation: London, UK.

Preview

Text Systematic-Review-of-Feedback-EPPI-2021.pdf Download (4MB) | Preview

Abstract

Abstract Meta-syntheses have reported positive impacts of feedback for student achievement at different stages of education and have been influential in establishing feedback as an effective strategy to support student learning. However, these syntheses combine studies of a variety of different feedback approaches, combine studies where feedback is one of a number of intervention components and have several methodological limitations. For example the lack of quality appraisal of the included studies. There is also still more research needed to investigate the impact of different types of feedback on different students in different settings. Objective This systematic review was conducted at the request of the Education Endowment Foundation to provide more precise estimates of the impact of different types of feedback in different contexts for different learners aged between 5 and 18. The review analysis sought to explore potential variations in the impact of feedback through subgroup analysis of the characteristics of the feedback, the educational setting, the learners and the subject. This review provides evidence that can be used to support the development of guidance for teachers and schools about feedback practices. Methods design A systematic review was undertaken in two stages. First, a systematic map identified and characterised a subset of studies that investigated the attainment impacts of feedback. Second, an in-depth review comprising of a meta-analysis was performed to answer the review questions about the impact of interventions that comprised of feedback only and to explore the variety of characteristics that may influence the impact of feedback. Methods search We used the Microsoft Academic Graph (MAG) dataset hosted in EPPI-Reviewer to conduct a semantic network analysis to identify records related to a set of pre-identified study references. The MAG search identified 23,725 potential studies for screening. Methods study selection Studies were selected using a set of pre-specified selection criterion. Semi-automated priority screening was used to screen the title and abstract of studies using bespoke systematic review software EPPI-Reviewer. The title and abstract screening was stopped after 3,028 studies and 745 were identified for full-text screening. Reviewers carried out a moderation exercise, all screening a selection of the same titles to develop consistency of screening. Thereafter, single reviewer screening was used with referral for a second reviewer opinion in cases of uncertainty. Methods data collection Studies were coded using a bespoke data extraction tool developed by the EEF Database Project. Study quality was assessed using a bespoke risk of bias assessment adapted from the ROBINS-I tool. The review team undertook a moderation exercise coding the same set of studies to develop consistency. Thereafter, single reviewer coding was used, based on the full text with referral for a second opinion in cases of uncertainty. Methods synthesis Data from the studies was used to calculate standardised effect sizes (Standardised Mean Difference- Hedge’s g). Effect sizes from each study were combined to produce a pooled estimate of effect using Random Effects Meta-analysis. Statistical Heterogeneity tests were carried out for each synthesis. Sensitivity analysis was carried out for assessed study quality. Subgroup analysis was completed using meta-analysis to explore outcomes according to the different characteristics of feedback, context and subjects. Main results The full text screening identified 304 studies to include in the initial systematic map, of which 171 studies investigated feedback only. After applying final selection criteria, 43 papers with 51 studies published in and after the year 2000 were included. The 51 studies had approximately 14,400 students. Forty studies were experiments with random allocation to groups and 11 were prospective quantitative experimental design studies. The overall ecological validity was assessed as moderate to high in 40 studies and the overall risk of bias assessed as low to moderate in 44 studies. The interventions took place in curriculum subjects including literacy, mathematics, science, social studies, and languages, and tested other cognitive outcomes. The source of feedback included teacher, researcher, digital, or automated means. Feedback to individual students is reported in 48 studies and feedback to group or class is reported in four studies. Feedback took the form of spoken verbal, non-verbal, written verbal, and written non-verbal. Different studies investigated feedback that took place immediately after the task, during the task and up to one week after the task (delayed feedback). Most of the feedback interventions gave the learner feedback about the outcome and the process/strategy. Some provided feedback on outcome only and two provided feedback about task/strategy only. On the main research question, the pooled estimate of effect of synthesis of all studies with a low or moderate risk of bias indicated that students who received feedback had better performance than students who did not receive feedback or experienced usual practice (g = 0.17, 95% C.I. 0.09 to 0.25). However, there is statistically significant heterogeneity between these studies (I2 = 44%, Test for Heterogeneity: Q(df = 37) = 65.92, p = 0.002), which suggests that this may not be a useful indicator of the general impact of feedback on attainment when compared to no feedback or usual practice. The heterogeneity analysis suggested considerable heterogeneity between studies in the main synthesis and all the subgroup synthesis, and in the majority of the cases the heterogeneity is statistically significant. This means caution is required when considering the results of the synthesis. The results of the subgroup synthesis suggest that a variety of student and context factors may have an effect on the impact of feedback. Conclusions The results of the review may be considered broadly consistent with claims made on the basis of previous synthesis and meta-synthesis, suggesting that feedback interventions, on average, have a positive impact on attainment when compared to no feedback or usual practice. The limitations in the study reports and the comparatively small number of studies within each subgroup synthesis meant that the review was not able to provide very much more certainty about the factors that affect variation in the impact of single component feedback interventions within different contexts and with different students. More research is needed in this area to consider what may moderate the impact of feedback. However, the findings further support the conclusion made by previous studies that feedback, on average, has a positive impact on attainment; moreover, this is based on a more precise and robust analysis than previous syntheses. This suggests that feedback may have a role to play in raising attainment alongside other effective interventions. Findings were further interpreted by a panel of expert practitioners and academics to produce the EEF’s Teacher feedback to improve pupil learning guidance report. 1. Background and review rationale Feedback can be defined as information communicated to the learner that is intended to modify the learner’s thinking or behaviour for the purpose of improving learning. Meta-syntheses have reported positive impacts of feedback, with effect sizes ranging from d = 0.70 to d = 0.79 for student achievement at different stages of education and have been influential in establishing feedback as highly effective with regards to student learning. For example, the EEF Teaching and Learning Toolkit meta-synthesis suggests that feedback may have ‘very high’ impact (equivalent to eight months’ additional progress) for relatively low cost. However, caution is necessary when interpreting the findings of these meta-syntheses for a number of reasons. Firstly, the average effect size reported in the EEF Toolkit is based on combining the estimates from existing meta-analyses of individual studies, which may contain limitations of various kinds (see the list below for examples) that may mean that average effect sizes identified are overestimates. Second, some studies included in syntheses (such as Kluger and DeNisi’s meta-analysis ) suggest that some feedback interventions may, in fact, negatively impact pupils. Third, previous meta-syntheses have not explored in detail the impact of potential moderating factors, such as different types of feedback. As Ekecrantz has argued, there is still a need to better understand how and under what circumstances teacher feedback on student performance promotes learning as well as, to question the generalised claim (that feedback improves attainment) itself. For example, a recent meta-analysis that re-analysed studies included in the original synthesis by Hattie and Timperley revised down the average effect size from the estimates of the effects of feedback from their originally published Standardised Mean Difference of d = 0.79 to d = 0.48. In the revised meta-analysis, 17% of the effect sizes from individual studies were negative. The confidence interval ranged from d = 0.48 to d = 0.62, and the authors found a wide range of effect sizes. Different moderators were also investigated to explore the impact of different characteristics of context and feedback. Whilst this meta-analysis offers improvements over previous meta-syntheses, it has a number of limitations, including: • It only included studies drawn from 36 existing meta-analyses, the most recent of which was published in 2015. Eligible studies published after 2015 or not included in these meta-analyses would not have been included. • All comparative study designs were included. Less robust study designs may have overestimated the positive effect of feedback. • There was no reported study quality assessment/moderation or sensitivity analysis, which may have led to an overestimation of the pooled effect sizes. • The meta-analyses included studies with high levels of heterogeneity, I2 = 80% or more (in the main and moderator analysis). This suggests that the synthesis may be combining studies/comparing feedback practices inappropriately. • The meta-analysis did not consider all potentially relevant moderating factors. It may also be the case that the impact of feedback depends on factors other than those analysed, including the ability of the learner, the learning context, and/or the frequency, duration, timing, and type of feedback. This systematic review was conducted at the request of the EEF to try and provide more accurate and precise estimates of the impact of different types of feedback in different schooling contexts. The review examines the impact of single component feedback, in different contexts, and for different learners with a greater degree of granularity and precision than is currently available via the EEF Teaching and Learning Toolkit strand on ‘Feedback’. For EEF, the purpose of the systematic review is to provide evidence that can be used to inform guidance for teachers and schools about effective feedback practices. The systematic review methods and processes were developed and carried out conterminously with the EEF Database project with a view to facilitating the future use of the produced resources and supporting the ongoing work of the Database project. 1.1 Domain being studied: Feedback approaches This review focuses on interventions that provide feedback from teachers to learners in mainstream educational settings. Feedback is defined in accordance with the EEF toolkit definition: ‘Feedback is information given to the learner and/or teacher about the learner’s performance relative to learning goals or outcomes. It should aim to produce (and be capable of) producing improvement in students’ learning. Feedback redirects or refocuses either the teacher’s or the learner’s actions to achieve a goal, by aligning effort and activity with an outcome. It can be about the output of the activity, the process of the activity, the student’s management of their learning or self-regulation, or them as individuals. This feedback can be verbal or written or can be given through tests or via digital technology. It can come from a teacher or someone taking a teaching role, or from ‘peers’.’ This initial broad definition, whilst conceptually coherent, does create challenges both in practice for teachers and in terms of identifying and distinguishing between practices when considering research evidence. For example, what is the difference between small group learning and ‘peer feedback’? It seems perfectly reasonable to assume that small group learning must contain conversations between students about their work and the task they have been asked to complete and thus is ‘feedback’. However, in practice, this may not be what teachers think of as ‘feedback’ and in the research literature, ‘small group learning’ is investigated both as a unique pedagogical strategy and as a component of a number of other pedagogical strategies. As the development of the understanding of the scope of the review evolved, the working definition of feedback for the review became modified practically through the exclusion of certain categories of intervention, even though they may contain an element of feedback practice. The inclusion criteria in the methods section outlines the revised definition that the review team used. 1.2 Conceptual framework/Theory of Change There are several ways in which feedback is conceptualised as improving learner performance—i.e. as a Theory of Change. The ‘Feedback’ strand in the EEF Teaching and Learning Toolkit draws most explicitly on the conceptualisation of Hattie and Timperley’s (2007) model. This model emphasises the importance of systems of feedback where the teacher provides feedback to the specific needs of individual students. The searching processes used in this review are consistent with this model as the studies used in the Feedback strand of the EEF Teaching and Learning Toolkit were used to ‘seed’ the search. However, they did not preclude the inclusion of studies that may draw on other ‘models’ of feedback which, though similar to Hattie and Timperley (2007), may be argued to place more emphasis on, for example: developing learner self-regulation (Nicole and Macfarlane-Dick, 2006); students’ intrinsic motivation (Dweck, 2016); and/or are subject specific—for example, ‘Thinking Mathematically’ (Mason, Burton and Stacey, 2010). The coding tools used in the review were informed by the model (in terms of coding about the source and content of the feedback; see Appendix 3). 1.3 Review design A systematic review approach was used to investigate the research questions. The review was undertaken in two stages. First, a systematic map identified and described the feedback characteristics of a subset of studies that investigated the attainment impacts of feedback. The map was used to make decisions about focusing the analysis in the second in-depth systematic review stage. At the second stage an in-depth review, including meta-analysis, was performed on a subset of the studies identified in the map to answer the review questions and explore the variety of intervention and context characteristics that may influence the impact of feedback. This systematic review was designed to complement the work of the EEF Database project. The EEF Database project is currently undertaking a programme to extract and code the individual studies from the meta-synthesis used in the EEF Teaching and Learning Toolkit. The search strategy used in this review was ‘seeded’ from studies identified as being about ‘feedback’ in the database, and this systematic review used the coding tools developed by the Database team (see Appendix 3). The studies newly identified in this review will be subsequently included in the EEF Database. This systematic review was also designed to provide additional research evidence for use in guidance on feedback developed for schools produced by the EEF, and therefore to fit with a particular time window for the review’s production. The results of the meta-analyses were presented to an advisory panel of academics and teaching practitioners, who used the results , their own expertise, a review of practice undertaken by the University of Oxford, and conceptual models (such as Hattie and Timperley) to draft recommendations for practice.

Download activity - last month

Export as JSONCSVXML

Download activity - last 12 months

Export as JSONCSVXML

Downloads by country - last 12 months

Export as CSVXMLJSON

Archive Staff Only

View Item