Isaacs, T; Thomson, RI; (2013) Rater experience, rating scale length, and judgments of L2 pronunciation: Revisiting research conventions. Language Assessment Quarterly , 10 (2) pp. 135-159. 10.1080/15434303.2013.769545.

Preview

Text Isaacs_rater_experience_rating_scale_length.pdf Download (421kB) | Preview

Abstract

This mixed-methods study examines the effects of rating scale length and rater experience on listeners' judgments of second-language (L2) speech. Twenty experienced and 20 novice raters, who were randomly assigned to 5-point or 9-point rating scale conditions, judged speech samples of 38 newcomers to Canada on numerical rating scales for comprehensibility, accentedness, and fluency. Results yielded high Cronbach's alpha coefficients and no group differences for rating scale length or rater experience. However, Rasch category probability plots revealed that raters had difficulty differentiating between scale steps, particularly in mid-scale range—a challenge that was exacerbated in the 9-point scale condition. Evidence from verbal protocols and posttask interviews suggested that experienced and novice raters adopted strategies to either draw on or offset their perceived experience with L2 speech in conducting their ratings. Implications for L2 pronunciation research are discussed, including the need to draw on raters' perceptions to develop a greater understanding of major listener perceptual constructs in L2 pronunciation and to move beyond adherence to the native speaking standard in data collection procedures.

Download activity - last month

Export as XMLJSONCSV

Download activity - last 12 months

Export as CSVXMLJSON

Downloads by country - last 12 months

Export as XMLJSONCSV

Archive Staff Only

View Item