1710.02187

#1 BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages [PDF] [Copy] [Kimi¹] [REL]

Authors: Benjamin Heinzerling, Michael Strube

We present BPEmb, a collection of pre-trained subword unit embeddings in 275 languages, based on Byte-Pair Encoding (BPE). In an evaluation using fine-grained entity typing as testbed, BPEmb performs competitively, and for some languages bet- ter than alternative subword approaches, while requiring vastly fewer resources and no tokenization. BPEmb is available at https://github.com/bheinzerling/bpemb

Subject: Computation and Language

Publish: 2017-10-05 19:24:07 UTC

---