2508.02762

Total: 1

#1 Context-Adaptive Multi-Prompt Embedding with Large Language Models for Vision-Language Alignment [PDF] [Copy] [Kimi1] [REL]

Authors: Dahun Kim, Anelia Angelova

We propose Context-Adaptive Multi-Prompt Embedding, a novel approach to enrich semantic representations in vision-language contrastive learning. Unlike standard CLIP-style models that rely on a single text embedding, our method introduces multiple structured prompts, each containing a distinct adaptive token that captures diverse semantic aspects of the input text. We leverage a pretrained LLM as the text encoder within the CLIP framework, processing all prompts jointly in a single forward pass. The resulting prompt embeddings are combined into a unified text representation, enabling semantically richer alignment with visual features. To further promote semantic diversity and representation quality, we incorporate a diversity regularization loss and a negation-aware loss, encouraging specialization across prompts and improving contrastive discrimination. Our method achieves consistent improvements on both image-text and video-text retrieval benchmarks.

Subjects: Machine Learning , Artificial Intelligence

Publish: 2025-08-03 20:48:43 UTC