| Total: 266
For the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP 2026), a dedicated Industry Track was organised to provide a forum for work on real-world applications of speech and language technologies. The track received 931 submissions, which were evaluated by 733 reviewers and 158 area chairs. Following the review process, 265 papers were selected for presentation, including 23 oral presentations and 242 posters. Across the accepted programme, prominent themes include evaluation and benchmarking, agents and orchestration, efficient serving and cost control, retrieval and knowledge grounding, and safety and reliability. This overview summarises the scope of the track and its submission and review process, highlights these research trends, and describes the Industry Track programme, including its oral sessions and panel discussion.
Recommender systems are central to online services, enabling users to navigate through massive amounts of content across various domains. However, their evaluation remains challenging due to the disconnect between offline metrics and online performance. The emergence of Large Language Model-powered agents offers a promising solution, yet existing studies model users in isolation, neglecting the contextual factors such as time, location, and needs, which fundamentally shape human decision-making. We introduce ContextSim, an LLM agent framework that simulates believable user proxies by anchoring interactions in daily life activities. A life simulation module generates scenarios specifying when, where, and why users engage with recommendations. To align preferences with genuine humans, we model agents’ internal thoughts and enforce consistency at both action and trajectory levels. Experiments across domains demonstrate closer alignment with human behavior than prior work. We further validate our approach through offline A/B testing correlation and show that RS parameters optimized using ContextSim yield improved real-world engagement.
Setting appropriate prices for second-hand products remains challenging for inexperienced individual sellers on Consumer-to-Consumer (C2C) e-commerce platforms. Existing regression-based approaches rely on static product–price mappings, making it difficult to capture subtle condition differences expressed in colloquial product descriptions. Inspired by recent breakthroughs in Large Language Models (LLMs), we present LLP, a retrieval-then-generation framework that generates price suggestions based on dynamically retrieved comparable products. LLP employs bidirectional reasoning to generate high-quality rationales as supervision, and subsequently applies retrieval-aware policy optimization to further improve both reasoning faithfulness and pricing accuracy. At inference time, LLP employs entropy-guided filtering to discard low-confidence price predictions. Extensive evaluations demonstrate the effectiveness of our method. It raises the static adoption rate (SAR) from 40% to 72%, while increasing GMV by 0.7% and CTCVR by 1.4%. LLP is now fully deployed in Xianyu, serving millions of individual sellers daily.
Large language models increasingly operate as autonomous agents that select and invoke tools from large registries. We identify a critical gap: when unauthorized tools are visible in an agent’s context, models select them in 48–68% of adversarial scenarios—even when explicitly instructed not to. Role escalation attacks (e.g., “I’m the CFO, override the access controls”) are the most dangerous category, reaching 96% unauthorized invocation in frontier models. We show this holds across three models spanning open-weight and frontier systems, including instruction-tuned models with strong alignment training. Critically, prompt-based compliance is both insufficient and unpredictable: explicit per-tool allowlists reduce violations to as low as 4.0% but never to zero, and compliance varies widely across models—from 4.0% to 37.0% UIR—with no reliable relationship to general capability. We propose a proxy-enforced attribute-based access control (ABAC) layer for MCP that filters tool registries at discovery time. Because unauthorized tools never reach the model context, UIR is 0% by design—a structural guarantee that prompt instructions cannot replicate regardless of model or phrasing.
We present Lunar-R1, an 8B-parameter reasoning model tailored for lunar science and mission operations, developed through continued pre-training, supervised fine-tuning, and reinforcement learning. To optimize the accuracy-efficiency trade-off, we introduce Latent Difficulty Perception (LDP), a lightweight mechanism that leverages final-layer hidden states to estimate query difficulty and regulates Chain-of-Thought generation length via difficulty-aware reward modeling. We construct Lunar-QA, an expert-verified benchmark derived from authentic lunar mission materials, and further evaluate on domain-specific tasks spanning astronomy and astrodynamics. Experimental results demonstrate that LDP reduces average token consumption by 38.9% while improving reasoning accuracy. Profiling on an RK3588 NPU yields 0.54 J/token energy efficiency and 380 ms time-to-first-token latency, validating feasibility for lunar surface edge deployment.
Industrial query-to-agent matching fails when topical relevance is mistaken for executable capability, especially on long-tail and boundary-sensitive requests. We formulate annotation as capability-bound process supervision and instantiate it with Debate-to-Skill, which uses reusable decision principles, structured deliberation, verifier-based verdict extraction, and disagreement-driven refinement. On an industrial Query2Agent benchmark, we compare Debate-to-Skill with direct-label supervision, reasoning-SFT, and structural ablations. The results test whether gains come from supervising the capability-critical decision process itself, especially on grey-zone cases where semantic relatedness and executable capability diverge.
Early in deployment, an AI search system typically operates over a frequently updated product catalog, so the available items and their properties cannot be treated as stable knowledge that can be encoded in fixed prompts or strategies. Fine-tuning, reinforcement learning, and static prompt patches fit poorly: labels are scarce, rewards drift with inventory, model releases are costly, and prompt fixes quickly stale. We present Inventory-Grounded Policy-Level Optimization (IGPO), a training-free approach for fixed AI search pipelines. IGPO separates policy from environment facts: it learns Policy Guidelines for acting on runtime inventory evidence rather than memorizing available materials. Online, IGPO grounds each query by probing the inventory and constructing an inventory portrait, then injects relevant Policy Guidelines into the retrieval and selection prompts. Offline, stochastic rollouts are grouped by query—mixed outcome groups directly yield contrastive signal, and an inventory-guided exploration loop distinguishes missed retrieval routes from cases where no matching support is found under the observed inventory evidence. Since May 2026, IGPO has been deployed in a commercial smart-assistant AI search system. A 14-day online A/B test of the complete IGPO treatment shows a 3.17% relative CTR lift and a 38.9% reduction in audited bad cases.
As large language models (LLMs) increasingly assist scientific writing, the limitations and token costs of generating TeX become increasingly visible. This paper analyzes TeX’s architectural mismatch with LLM workflows, stemming from its lack of an explicit structural representation, to illustrate its limitations on generated semantics and error localization. As an alternative, we introduce .tmu, a low-entropy tree-structured representation. With its efficient data structure and clear contextual boundaries, .tmu outperforms .tex in the above aspects. Experiments across four LLMs provide evidence for this claim in most evaluated settings. Furthermore, we show that due to its lower information entropy, fine-tuning LLMs on .tmu achieves approximately 43% lower final training loss than on .tex. Our work provides a more scalable and LLM-friendly data representation for LLM-assisted scientific writing.
Existing LLM data-analyst benchmarks fix data access at full disclosure and evaluate only on answer accuracy. But real deployments rarely grant full access. No prior work, to our knowledge, asks *"what’s the least amount of information needed for the agent to work?"*. In this article, we treat dataset disclosure as a controlled treatment along a *graduated information-disclosure* (GID) axis: C0 (column names) and C∞ (full table) bracket a nested chain C1 ⊆ … ⊆ C5 that adds per-column metadata, dataset purpose, summary statistics, sampled cells, and sampled rows. We maintain agent architecture, tool registry, and decoding fixed, and score the final reports on a 14-attribute rubric across a 6 × 4 × 12 panel of frontier models, planted-signal datasets, and disclosure levels. At a high level, from C0 to C∞, seven attributes rise and five fall. However, the rise/fall pattern has nuance, the optimal intermediate level depends on several factors including models, datasets and attributes. Level C1 - one sentence of types and units beyond column names - matches (statistically) C∞ on signal recovery and decoy rejection at ∼49× lower prompt-token cost. The key takeaway from this work is - **current LLM data-science agents can act on partial information alone**, so row-level records need never leave the institution. Hence, the current framework also offers a way to enforce data boundary constraints (required by GDPR) without compromising on quality.
Cloud incident response increasingly relies on LLM agents that translate heterogeneous telemetry and natural-language alerts into operator-facing diagnoses. However, in production RCA workflows, LLM agents commonly serialize these structured signals into text, losing the temporal and topological structure that is critical for diagnosis. We present Knsight, an industry-deployed RCA agent that converts operational telemetry into chart-image canvases and uses a frozen multimodal LLM to reason directly over visual evidence. Knsight introduces Vision Recursive-of-Thought (V-RoT), an iterative diagnose-review-refine loop in which a Reviewer module inspects each canvas and decides whether to refine, zoom in, or stop. To adapt across recurring production failures, Knsight further converts Reviewer feedback into a shared, alias-substituted skill library used at inference time. Offline, on the OpenRCA benchmark, Knsight improves aggregate Correct by 7.0 points over the strongest text-only baseline and reduces timeouts from 117 to 11. Online, in a 90-day deployment across 1,284 production incidents, Knsight increases top-1 acceptance from 38.9% to 71.4% and reduces time-to-acknowledge and time-to-resolve by 59% and 32%, respectively.
Chinese query correction (CQC) is important for search and query recommendation on content platforms, but supervised methods rely on large annotated correction pairs that are costly to maintain as query vocabularies evolve. Unsupervised correction with language models is attractive, yet in the short-query setting, unconstrained generation often over-corrects ambiguous inputs toward high-frequency phrases, causing intent drift. We propose GUIDE, a generative unsupervised framework for CQC based on a confuse-then-clarify paradigm. GUIDE encodes phonetically or visually confusable characters with shared-IDs and reconstructs the original query with an encoder–decoder architecture, which constrains correction to plausible confusion neighborhoods while learning from unlabeled query streams. A time-decayed, query-frequency-weighted objective further supports adaptation to rapidly changing query vocabularies. Experiments on QSpell 250K and KwaiSearch, a large-scale industrial dataset from Kuaishou, show that GUIDE consistently outperforms strong baselines, while online A/B testing further confirms gains in correction quality and downstream engagement.
Small language models are attractive in production because they are inexpensive, fast, and easy to specialize, but specialization remains a difficult loop. The hard decisions surround training: acquiring and curating supervision, diagnosing failures, selecting a training strategy, and preventing a targeted fix from breaking behavior that already works. We present Pioneer Agent, a closed-loop system that automates this lifecycle. In cold-start mode, the agent turns a task specification into a trained model by searching jointly over data composition, hyperparameters, and learning strategy. In production mode, it converts judged failures into a failure taxonomy and corrective curriculum, retrains, and retains an update only when it passes regression gates. We introduce AdaptFT-Bench, seven staged scenarios that combine fixable noise with harmful supervision at increasing poison rates. Across eight cold-start benchmarks, Pioneer Agent improves over the base checkpoint on every task. Across all seven AdaptFT-Bench scenarios, it improves or preserves the deployed checkpoint while indiscriminate retraining degrades, with final gaps up to 43 points. In two production-style case studies, the same loop raises intent-classification pass rate from 84.9% to 99.3% and entity F1 from 0.345 to 0.810. These results show that deployment failures can be converted into effective training curricula without manual intervention between iterations.
Estimation of product procurement cost is a key capability for price governance on large-scale e-commerce platforms: from images, structured attributes, comparable products, and pricing knowledge, a practical system must infer a reasonable cost to surface suspiciously inflated or potentially non-compliant listings. Directly invoking general-purpose VLMs is prohibitive at high QPS (queries per second), and their verbose rationales complicate auditing and rule integration. We present Cost-LLM, a compact 7B multimodal cost-estimation model trained by a two-stage post-training pipeline: segment-normalized SFT on progressive pricing rationales, followed by GRPO with an asymmetric MAPE-based reward aligned with cost accuracy. On an in-house benchmark covering 105 fine-grained categories, Cost-LLM reaches MAPE 29.47% and Acc@20 40.42%, yielding the best point estimates among the evaluated systems. Deployed as an auxiliary signal in a price-anomaly pipeline, it provides a 30.73% relative complementary-detection yield over the production baseline at an audited precision-matched operating point. Its locally deployed serving latency is 0.4-0.8 s per listing, compared with 3-8 s observed for external commercial LLM/VLM APIs. We share the design choices, training recipe, and deployment lessons.
In local-life service platforms, query suggestion reduces user effort by generating candidate queries from input prefixes. Traditional multi-stage systems rely heavily on historical popular queries, limiting their ability to capture long-tail and emerging demand. Although LLMs provide strong semantic generalization, their deployment in local-life services faces three challenges: insufficient city-preference awareness, exposure bias in preference optimization, and strict online latency constraints. We propose LocalSUG, an LLM-based query suggestion framework for local-life services. LocalSUG mines city-preference-enhanced candidates from term co-occurrence and injects them into prompts as dynamic references rather than fusing them into model parameters. This allows the model to adapt to changing city preferences, such as merchant openings or closures, while reducing stale or locally invalid suggestions. We further introduce a beam-search-driven GRPO algorithm to align training with inference-time decoding and optimize relevance together with business-oriented rewards. Finally, quality-aware beam acceleration and vocabulary pruning reduce online latency while preserving generation quality. Offline evaluations and large-scale online A/B testing show that LocalSUG improves CTR by +0.35% and reduces the low/no-result rate by 3.98%, demonstrating its effectiveness in real-world deployment.
Production multimodal systems must serve complex vision-language tasks under strict latency, memory, and cost constraints. Frontier multimodal large language models (MLLMs) deliver strong reasoning but are prohibitively expensive for online serving; compact alternatives fit deployment budgets but lack structured reasoning ability. We present OmniThoughtVis, a scalable distillation pipeline that converts open-source multimodal corpora into deployment-ready chain-of-thought (CoT) supervision for compact MLLMs. Starting from a diverse seed pool, the pipeline generates structured reasoning traces from a high-capacity teacher, jointly annotates reasoning difficulty, answer quality, and semantic task tags, and applies difficulty-aware filtering with tag-based diversity sampling to produce a reusable 1.8M-sample candidate pool. Practitioners construct budget-specific training subsets from this pool without re-running distillation. A key practical finding is that reasoning difficulty is a far more effective data selection signal than answer quality, which becomes uninformative when teacher outputs are already strong. Distilled Qwen3-VL models (2B-8B) consistently improve over undistilled baselines across nine benchmarks, with the distilled 4B model matching or exceeding the undistilled 8B on multiple tasks. In production, replacing a large-model serving pipeline with the distilled compact model reduces end-to-end response latency by 79% while maintaining comparable downstream quality.
SQL code completion boosts developer productivity by predicting missing snippets from editing SQL contexts. However, existing solutions yield low acceptance since they cannot abstain when no suggestion is needed and lack SQL-specific training. To address both limitations, we introduce PACE-SQL, a preference-aligned framework that improves SQL completion accuracy and user preference alignment. We build two datasets with completion and abstention samples: WeSQL-1.5M from real-world interaction logs to capture authentic user preferences, and OpenSQL-FIM from public SQL corpora. We adopt continual pretraining and supervised fine-tuning to improve completion quality, and design a reward function in the reinforcement learning to enhance alignment with user preferences. Experiments show PACE-SQL improves Exact Match and Edit Similarity scores by 9.67% and 7.52% over the best baseline on WeSQL-1.5M and raises online user acceptance rate from 5% to 29%.
Large-scale content moderation requires models that can apply complex, evolving policies under limited expert supervision. We present an empirical study of reinforcement learning (RL) for policy-grounded moderation across three production-derived tasks. Using established RL and inference primitives, we develop three integrity-specific designs: a policy-grounded full-trace reward, rationale-marginalized Monte Carlo scoring, and policy-structured reflection. Each design addresses an observed failure mode, including reasoning-length collapse and polarized confidence estimates. Across the three tasks, RL-Only systems trained on 200–836 labeled examples remain competitive with supervised fine-tuning (SFT) systems trained on thousands of examples, corresponding to up to approximately one order of magnitude greater label efficiency at the evaluated operating points. Performance generally improves with additional data and training compute before saturating, although larger rollout counts exhibit variability. The strongest SFT→RL systems attain point-estimate performance comparable to policy-trained full-time employee (FTE) reviewers across the three tasks.
Production text-to-SQL pipelines often end with an LLM-as-judge that nobody has checked against humans. We audited ours and found it broken: the deployed gpt-4o-mini judge agreed with two-author gold at Cohen’s κ = 0.04 on a disagreement-enriched set and 0.42 on a uniform-random spot-check, with most over-flags traceable to a single mechanism we call Grade-Hallucination. Replacing it with self-hosted Qwen3.6-27B (κ = 0.72) reaches the same range as Claude Opus 4.7 (κ = 0.71); the comparison is underpowered at n = 96, but the deployment decision is driven by Qwen costing about 1/300 per call. Ensembling is not a free win: pairing the weak judge with a stronger one degrades agreement; three strong judges with unanimity routing reach κ = 0.79 at 89.7% auto-coverage. Applying the audit recipe out-of-domain flags 25.5% of BIRD-financial’s expert-authored gold SQLs as likely substantive issues under our annotation protocol. Code and pre-registration are at https://anonymous.4open.science/r/synca-audit-CBE1; the datasets are deferred to the post-publication release for anonymization reasons.
Tool-augmented LLM agents discover relevant APIs from large registries before planning execution, yet finite context windows prohibit injecting all tool definitions into a single prompt. The dominant approach, which we term Retrieve-then-Plan (RtP), embeds the user query and retrieves similar tools via approximate nearest-neighbor search. We study an alternative Plan-then-Retrieve (PtR) paradigm in which the LLM first generates Abstract Tool Stubs (hypothesized function signatures describing needed capabilities) which then serve as retrieval queries against the registry. Evaluating on ToolBench (46,984 unique API endpoints, 50 categories) across single-tool (G1), intra-category multi-tool (G2), and cross-category multi-tool (G3) query subsets, we find a clear complexity-dependent crossover: RtP outperforms PtR on single-tool queries while PtR outperforms RtP on multi-tool queries. This crossover holds at every retrieval budget (k=1 to 20), is robust across registry sizes from 1k to 47k APIs, and is consistent across model families (Gemini 2.5 Flash and gpt-oss-120b). We characterize a stub-hallucination failure mode (the LLM invents plausible but non-existent API names) that acts as a double-edged sword: on simple queries PtR’s stubs abstract away proper-noun signals that RtP matches directly, yet on complex queries the same mechanism decomposes multi-faceted intent into functional sub-needs. We further analyze token overhead and show that PtR’s extra discovery call roughly doubles inference cost, which pays off on multi-tool queries but hurts on single-tool ones, motivating complexity-adaptive routing as a practical direction — one that can be driven by a zero-cost signal already present in the pipeline (RtP’s top-1 retrieval score).
As LLM agents are exposed to hundreds to tens of thousands of skills, tools, and API functions, full-library prompting becomes costly, slow, and less reliable: each added candidate increases prompt tokens and latency, while longer candidate lists introduce more distractors for LLM selection. We present Toollery, a training-free candidate-compression framework for scalable LLM skill/tool selection. Following established document-side query expansion, Toollery generates user-intent queries from each skill/tool specification and builds a retrieval index that maps real user requests to compact candidate sets before final LLM decision-making. By treating high-level skills and atomic tools as selectable capabilities, Toollery can be applied to both skill libraries and tool registries. We evaluate Toollery on the roughly 79K-capability SkillRouter benchmark, BFCL-V4 with over 440 atomic tools, and 3,396 proprietary smart-cockpit requests over 220 tools. Across these settings, Toollery keeps online selection bounded to a compact top-k candidate set and improves recall over ordinary specification retrieval. At a fixed top-10 budget, Toollery improves end-to-end selection on the cockpit dataset, and maintains comparable AST Accuracy on BFCL-V4. These results support Toollery as a practical candidate-compression framework for large and evolving agent capability libraries, while showing that quality and cost gains depend on workload coverage and provider caching.
Effective troubleshooting agents in enterprise customer support depend on retrieving actionable guidance from similar historical resolved cases, yet existing retrieval-augmented generation (RAG) systems treat support cases as static documents and overlook their multi-stage, stateful nature. We introduce RAFT (Retrieval-Augmented Framework for Troubleshooting Agents), a stateful RAG framework that abstracts each resolved case into a directed chain of timeline entries and retrieves at the entry level, surfacing cases whose intermediate states match the active case and returning the parent-case trajectory anchored at the matched state; an optional case-level graph links cases by shared root cause and resolution. We evaluate this retrieval layer directly, which, unlike evaluating a full agent system, requires no production deployment. Because public multi-stage troubleshooting data is extremely rare, we pair a synthetic benchmark built from Microsoft Learn Windows Server documentation with real Apache Jira issues carrying human-created duplicate labels. RAFT improves Case Hit over vanilla RAG and GraphRAG baselines at every stage of case progress, with statistically significant gains over the strongest baseline; the Jira results provide directional evidence that the advantage transfers to real case histories. We release our benchmark, implementation, and the Apache Jira evaluation set.
User interface (UI) and user experience (UX) evaluation is central to product development, yet reliable feedback still relies on recruiting human participants or running online A/B tests, making early-stage iteration slow and costly. In light of this, recent work has explored Multimodal Large Language Models as proxy evaluators. However, existing approaches either produce surface-level critiques or a judgment that reflects the model’s own biases rather than the genuine response of a particular user. We introduce PerceptUI, a framework for persona-conditioned UI/UX evaluation that predicts how a specific user would answer interface-related questions and produces natural-language rationales. PerceptUI is trained in two stages: (i) contrastive reflection fine-tuning distills teacher-generated rationales by extracting lessons from human decisions, and (ii) reflective prompt-evolution, which refines prompts using the model’s failure traces. Across multiple domains and datasets, PerceptUI improves alignment with human responses, generalizes to unseen questions and personas, and yields population-level response distributions.
Identifying proxy issuers and bonds is a critical but largely manual step in illiquid corporate bond pricing, requiring analysts to navigate sparse public information and inconsistent industry signals. We present a modular pipeline for proxy retrieval that integrates issuer profiling with hierarchical industry classification and multi-stage retrieval over issuers. We evaluate alternative design choices across hierarchical classification and reranking stages to understand how structured representations and industry-aware signals influence performance. Experiments on a large internal dataset show that domain-adapted embeddings and explicit industry hierarchy signals are primary drivers of quality across stages. The evaluated system produces issuer-level proxy shortlists rather than final bond prices. Pilot deployment experience highlights the value of modular, cost-aware designs for robust proxy retrieval in enterprise settings.
LLMs can personalize education, although current static-prompt tutoring systems struggle to adapt to diverse academic disciplines. We develop and test a system with subject-aware prompting, based on 14 pedagogical features (e.g., tutor scaffolding, student understanding) extracted from raw transcripts. We first train a prompt routing model in a simulation environment, and then deploy it for online adaptation with actual high-school students. The simulation benchmark shows the router outperforming two static baselines (0.694 vs. 0.647 and 0.64, p<0.001). A/B testing (N=656 conversations from 359 students) shows sim-to-real transfer where the model switches from analytical to scaffolding learning strategies. Our adaptive prompt selection mechanism improves instructional efficiency, maintains pedagogical quality and reduces interactions by around 3 turns (p=0.007). While a greedy router achieves a comparable exercise conversion rate with the baseline (19.1% vs. 19.6%), a stochastic router that samples strategies leads to a higher conversion rate (28.1%).
Multi-agent LLM systems that generate structured workflows from natural-language requests are now deployed in production across cloud automation, DevOps, and enterprise orchestration. Operating them exposes a recurring change-management problem. Routine updates, such as re-running an input, swapping the LLM, or refactoring an agent’s prompt or orchestration code, often produce workflows that differ substantially from validated references. Engineers then lack a principled way to decide whether a change is safe to ship. Automatic workflow evaluation is the natural tool, but in practice metric scores are poorly calibrated, and a numeric change rarely communicates the severity of the underlying degradation. We introduce WorkflowPerturb, a controlled benchmark that applies realistic, graded perturbations to golden workflows. It contains 4,973 golden workflows and 44,757 perturbed variants across three perturbation types (Missing Steps, Compressed Steps, Description Changes) at severities of 10%, 30%, and 50%. We benchmark multiple metric families, analyzing their sensitivity and calibration using expected score trajectories and instance-level alert rates. Our results characterize systematic differences across families and support severity-aware interpretation of workflow evaluation scores in change-management settings. WorkflowPerturb is publicly available at https://huggingface.co/datasets/microsoft/WorkflowPerturb.