Fast Construction of a Word-Number Index for Large Data

Warning

This publication doesn't include Faculty of Arts. It includes Faculty of Informatics. Official publication website can be found on muni.cz.

Authors

JAKUBÍČEK Miloš ŠMERK Pavel RYCHLÝ Pavel

Year of publication 2013
Type Article in Proceedings
Conference RASLAN 2013 Recent Advances in Slavonic Natural Language Processing
MU Faculty or unit

Faculty of Informatics

Citation
Web https://nlp.fi.muni.cz/raslan/2013/paper10.pdf
Field Informatics
Keywords word to number index; number to word index; finite state automata; hat-trie
Description The paper presents a work still in progress, but with promising results. We offer a new method of construction of word to number and number to word indices for very large corpus data (tens of billions of tokens), which is up to an order of magnitude faster than the current approach. We use HAT-trie for sorting the data and Daciuk’s algorithm for building a minimal deterministic finite state automaton from sorted data. The latter we reimplemented and our new implementation is roughly three times faster and with smaller memory footprint than the one of Daciuk. This is useful not only for building word-number indices, but also for many other applications, e.g. building data for morphological analysers.
Related projects:

You are running an old browser version. We recommend updating your browser to its latest version.