TY  - JOUR
KW  - Proactive moderation
KW  -  Moderation
KW  -  Hate speech
KW  -  Context
KW  -  Toxicity-detection
KW  -  Abusability
A1  - Warner, Mark
A1  - Strohmayer, Angelika
A1  - Higgs, Matthew
A1  - Coventry, Lynne
JF  - International Journal of Human-Computer Studies
SN  - 1071-5819
UR  - https://doi.org/10.1016/j.ijhcs.2025.103468
PB  - Elsevier BV
ID  - discovery10205333
N2  - Toxicity detection algorithms, originally designed for reactive content moderation systems, are being deployed into proactive end-user interventions to moderate content. Yet, there has been little critique on the use of these algorithms within this moderation paradigm. We conducted design workshops with four stakeholder groups, asking participants to embed a toxicity detection algorithm into an imagined mobile phone keyboard. This allowed us to critically explore how such algorithms could be used to proactively reduce the sending of toxic content. We found contextual factors such as platform culture and affordances, and scales of abuse, impacting on perceptions of toxicity and effectiveness of the system. We identify different types of end-users across a continuum of intention to send toxic messages, from unaware users, to those that are determined and organised. Finally, we highlight the potential for certain end-user groups to misuse these systems to validate their attacks, to gamify hate, and to manipulate algorithmic models to exacerbate harm.
N1  - This is an Open Access article published under a Creative Commons Attribution 4.0 International (CC BY 4.0) Licence (https://creativecommons.org/licenses/by/4.0/).
AV  - public
Y1  - 2025/04//
VL  - 198
TI  - A critical reflection on the use of toxicity detection algorithms in proactive content moderation systems
ER  -