Automatic normalization for linguistic annotation of historical language data

  • This paper deals with spelling normalization of historical texts with regard to further processing with modern part-of-speech taggers. Different methods for this task are presented and evaluated on a set of historical German texts from the 15th–18th century, and specific problems inherent to the processing of historical data are discussed. A chain combination using word-based and character-based techniques is shown to be best for normalization, while POS tagging of normalized data is shown to benefit from ignoring punctuation marks. Using these techniques, when 500 manually normalized tokens are used as training data for the normalization, the tagging accuracy of a manuscript from the 15th century can be raised from 28.65% to 76.27%.

Export metadata

Additional Services

Share in Twitter Search Google Scholar
Author:Marcel Bollmann
Parent Title (German):Bochumer linguistische Arbeitsberichte ; 13
Series (Serial Number):Bochumer linguistische Arbeitsberichte : BLA (13)
Publisher:Ruhr-Universität Bochum, Sprachwiss. Inst.
Place of publication:Bochum
Document Type:Working Paper
Year of Completion:2013
Year of first Publication:2013
Publishing Institution:Universitätsbibliothek Johann Christian Senckenberg
Release Date:2013/11/14
GND Keyword:Korpus <Linguistik>; Annotation
Page Number:86
Dewey Decimal Classification:4 Sprache / 41 Linguistik / 410 Linguistik
Linguistik-Klassifikation:Linguistik-Klassifikation: Korpuslinguistik / Corpus linguistics
Licence (German):License LogoDeutsches Urheberrecht