2606.11961

Total: 1

#1 Categorical Prior Lock-in: Why In-Context Learning Fails for Structured Data [PDF] [Copy] [Kimi] [REL]

Authors: Antonio Pelusi, Stefano Braghin, Alberto Trombetta

Large language models (LLMs) are increasingly used as conditional generators for structured data, relying on in-context learning (ICL) to adapt to new distributions without parameter updates. We investigate the limits of ICL for structured generation under distribution mismatch, using high-cardinality tabular data as a controlled test case, and identify a structural failure mode we term \textit{categorical prior lock-in}: the inability of ICL to update the model's prior over token distributions inherited from pre-training. Across two 7B-parameter open-weight models, ICL improves numerical fidelity with additional examples but exhibits a sharp ceiling on categorical distributions, failing to reproduce rare classes entirely. Parameter-efficient fine-tuning (LoRA) overcomes these limitations but introduces measurable memorization risk and, in some cases, destabilizes structured output generation, highlighting a fundamental trade-off between adaptability and privacy.

Subjects: Machine Learning , Artificial Intelligence

Publish: 2026-06-10 11:41:13 UTC