2309.16354

Total: 1

#1 Transformer-VQ: Linear-Time Transformers via Vector Quantization [PDF60] [Copy] [Kimi14]

Author: Lucas D. Lingle

We introduce Transformer-VQ, a decoder-only transformer computing softmax-based dense self-attention in linear time. Transformer-VQ's efficient attention is enabled by vector-quantized keys and a novel caching mechanism. In large-scale experiments, Transformer-VQ is shown highly competitive in quality, with strong results on Enwik8 (0.99 bpb), PG-19 (26.6 ppl), and ImageNet64 (3.16 bpb). Code: https://github.com/transformer-vq/transformer_vq

Subjects: Machine Learning ; Computation and Language ; Computer Vision and Pattern Recognition

Publish: 2023-09-28 11:26:52 UTC