Liu, Lu; Vlachidis, Andreas; Crymble, Adam; Lee, Deborah; Humbel, Marco; (2025) Towards Comparable Historical NER: Building a Shared Evaluation Corpus for 18th-Century Historical Texts. In: Anthology of Computers and the Humanities. (pp. pp. 968-982). Anthology of Computers and the Humanities

Preview

Text Towards Comparable Historical NER Building a Shared Evaluation Corpus for 18th-Century Historical Texts.pdf - Published Version Download (286kB) | Preview

Abstract

Named Entity Recognition (NER) is increasingly applied to historical text analysis. However, differences in evaluation materials, metrics, and annotation guidelines across existing NER projects make it difficult to systematically compare different approaches to historical NER. This study addresses this issue by constructing an evaluation corpus through the normalization of four annotated datasets from the long 18th century. We evaluate the performance of the Edinburgh Geoparser, spaCy and BERT-based tool on this corpus using five evaluation modes. Results show that even under the most lenient criteria, the highest F1-score remains below 70%, highlighting the challenges of applying existing NER systems to historical texts. Through detailed error analysis, we identify common challenges such as spelling and formatting issues. These findings demonstrate the limitations of NER tools in historical documents. We argue that future work should involve collaboration with historians to ensure that evaluation corpus align with real user needs.

Download activity - last month

Export as CSVXMLJSON

Download activity - last 12 months

Export as CSVXMLJSON

Downloads by country - last 12 months

Export as XMLJSONCSV

Archive Staff Only

View Item