Corpus Annotation Pipeline for Non-standard Texts

This publication doesn't include Faculty of Arts. It includes Faculty of Informatics. Official publication website can be found on



Year of publication 2018
Type Article in Proceedings
Conference Text, Speech, and Dialogue, 21st International Conference, TSD 2018
MU Faculty or unit

Faculty of Informatics

Keywords Non-standard language; Interlingual homographs; Corpora annotation
Description According to some estimations (e.g. [9]), web corpora contain over 6% of foreign material (borrowings, language mixing, named entities). Since annotation pipelines are usually built upon standard and correct data, the resulting annotation of web corpora often contains serious errors. We studied in depth annotation errors of the web corpus czTenTen 12 and proposed an extension to the tagger desamb that had been used for czTenTen annotation. First, the subcorpus was made using the most problematic documents from czTenTen. Second, measures were established for the most frequent annotation errors. Third, we established several experiments in which we extended the annotation pipeline so it could annotate foreign material and multi-word expressions. Finally, we compared the new annotations of the subcorpus with the original ones.
Related projects: