Warner, Mark;
Strohmayer, Angelika;
Higgs, Matthew;
Coventry, Lynne;
(2025)
A critical reflection on the use of toxicity detection algorithms in proactive content moderation systems.
International Journal of Human-Computer Studies
, 198
, Article 103468. 10.1016/j.ijhcs.2025.103468.
Preview |
Text
Warner_1-s2.0-S1071581925000254-main.pdf Download (2MB) | Preview |
Abstract
Toxicity detection algorithms, originally designed for reactive content moderation systems, are being deployed into proactive end-user interventions to moderate content. Yet, there has been little critique on the use of these algorithms within this moderation paradigm. We conducted design workshops with four stakeholder groups, asking participants to embed a toxicity detection algorithm into an imagined mobile phone keyboard. This allowed us to critically explore how such algorithms could be used to proactively reduce the sending of toxic content. We found contextual factors such as platform culture and affordances, and scales of abuse, impacting on perceptions of toxicity and effectiveness of the system. We identify different types of end-users across a continuum of intention to send toxic messages, from unaware users, to those that are determined and organised. Finally, we highlight the potential for certain end-user groups to misuse these systems to validate their attacks, to gamify hate, and to manipulate algorithmic models to exacerbate harm.
| Type: | Article |
|---|---|
| Title: | A critical reflection on the use of toxicity detection algorithms in proactive content moderation systems |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1016/j.ijhcs.2025.103468 |
| Publisher version: | https://doi.org/10.1016/j.ijhcs.2025.103468 |
| Language: | English |
| Additional information: | This is an Open Access article published under a Creative Commons Attribution 4.0 International (CC BY 4.0) Licence (https://creativecommons.org/licenses/by/4.0/). |
| Keywords: | Proactive moderation, Moderation, Hate speech, Context, Toxicity-detection, Abusability |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10205333 |
Loading...
Loading...Loading...Loading...Archive Staff Only
 |
View Item |
