JGO8CvG5S9@OpenReview

Total: 1

#1 Universal Approximation Under Constraints is Possible with Transformers [PDF1] [Copy] [Kimi2] [REL]

Authors: Anastasis Kratsios, Behnoosh Zamanlooy, Tianlin Liu, Ivan Dokmanic

Many practical problems need the output of a machine learning model to satisfy a set of constraints, K. Nevertheless, there is no known guarantee that classical neural network architectures can exactly encode constraints while simultaneously achieving universality. We provide a quantitative constrained universal approximation theorem which guarantees that for any non-convex compact set K and any continuous function f:RnK, there is a probabilistic transformer ˆF whose randomized outputs all lie in K and whose expected output uniformly approximates f. Our second main result is a ``deep neural version'' of Berge's Maximum Theorem (1963). The result guarantees that given an objective function L, a constraint set K, and a family of soft constraint sets, there is a probabilistic transformer ˆF that approximately minimizes L and whose outputs belong to K; moreover, ˆF approximately satisfies the soft constraints. Our results imply the first universal approximation theorem for classical transformers with exact convex constraint satisfaction. They also yield that a chart-free universal approximation theorem for Riemannian manifold-valued functions subject to suitable geodesically convex constraints.

Subject: ICLR.2022 - Spotlight