1902.04094

#1 BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model [PDF⁸] [Copy] [Kimi²]

Authors: Alex Wang ; Kyunghyun Cho

We show that BERT (Devlin et al., 2018) is a Markov random field language model. This formulation gives way to a natural procedure to sample sentences from BERT. We generate from BERT and find that it can produce high-quality, fluent generations. Compared to the generations of a traditional left-to-right language model, BERT generates sentences that are more diverse but of slightly worse quality.

Subjects: Computation and Language ; Machine Learning

Publish: 2019-02-11 19:02:27 UTC