Total: 1

#1 Learning deep representations by mutual information estimation and maximization [PDF5] [Copy] [Kimi4]

Authors: R Devon Hjelm ; Alex Fedorov ; Samuel Lavoie-Marchildon ; Karan Grewal ; Phil Bachman ; Adam Trischler ; Yoshua Bengio

In this work, we perform unsupervised learning of representations by maximizing mutual information between an input and the output of a deep neural network encoder. Importantly, we show that structure matters: incorporating knowledge about locality of the input to the objective can greatly influence a representation's suitability for downstream tasks. We further control characteristics of the representation by matching to a prior distribution adversarially. Our method, which we call Deep InfoMax (DIM), outperforms a number of popular unsupervised learning methods and competes with fully-supervised learning on several classification tasks. DIM opens new avenues for unsupervised learning of representations and is an important step towards flexible formulations of representation-learning objectives for specific end-goals.