Low Inter-Annotator Agreement = An Ill-Defined Problem?

Warning

This publication doesn't include Faculty of Arts. It includes Faculty of Informatics. Official publication website can be found on muni.cz.

Authors

KOVÁŘ Vojtěch RYCHLÝ Pavel JAKUBÍČEK Miloš

Year of publication 2014
Type Article in Proceedings
Conference Eighth Workshop on Recent Advances in Slavonic Natural Language Processing
MU Faculty or unit

Faculty of Informatics

Citation
Web https://nlp.fi.muni.cz/raslan/2014/13.pdf
Field Informatics
Keywords NLP; inter-annotator agreement; low inter-annotator agreement; evaluation
Description nnotation tasks where the inter-annotator agreement is low are usually considered ill-defined and not worth attention. Such tasks are also considered unsuitable for algorithmic solution and for evaluation of computer programs that aim at solving them. However, there is a lot of problems (not only) in the natural language processing field that are practically defined and do have this nature, and we need computer programs that are able to solve them. The paper illustrates such problems on particular examples and suggests methodology that will enable training and evaluating tools using data with low inter-annotator agreement.
Related projects:

You are running an old browser version. We recommend updating your browser to its latest version.