| Total: 86
Large language model (LLM)-based multi-agent systems (MAS) have shown strong potential for solving complex tasks, yet their development forces a tradeoff: code frameworks are expressive but engineering-intensive, while no-code builders simplify authoring but constrain agent interactions to author-defined workflows. We present **ChatDev 2.0: DevAll** (hereafter DevAll), a no-code platform for building, executing, and inspecting heterogeneous MAS that delivers both high expressiveness and ease of use. In terms of expressiveness, DevAll pairs a declarative executable graph abstraction with a cycle-aware execution engine, so that heterogeneous agents and dynamic and cyclic interactions can be represented and executed within a single framework. For ease of use, an integrated visual interface lets users author, run, monitor, and inspect MAS, including human-in-the-loop steps, entirely without writing code. Experiments demonstrate that DevAll reproduces state-of-the-art MAS across three representative tasks at competitive performance and without task-specific orchestration code, highlighting its effectiveness as a general-purpose platform for LLM-based MAS. DevAll is available at https://github.com/OpenBMB/ChatDev.
Large Language Models (LLMs) increasingly provide direct answers to user questions, raising concerns about reduced engagement in critical thinking and problem-solving. Hint generation offers an alternative by guiding users toward answers without revealing them, while hint evaluation assesses the quality of such guidance. Research in this area is hindered by fragmented datasets, inconsistent annotation formats, and evaluation tools that are often dataset-specific or unavailable. To address these challenges, we introduce HintEval, an open-source Python library for unified hint generation and evaluation. HintEval standardizes access to diverse hint datasets, supports answer-aware and answer-agnostic generation methods, and implements multiple evaluation metrics within a shared data model. The toolkit enables reproducible experimentation, cross-dataset analysis, and multi-dimensional evaluation with minimal engineering effort. We further demonstrate its utility through human studies in which participants assess generated hints and use them to answer questions, showing that hints can effectively support users in reaching correct answers. HintEval is accompanied by comprehensive documentation, an executable Google Colab notebook for rapid experimentation, and a demonstration video. By promoting consistent evaluation practices and lowering barriers to entry, it facilitates systematic research on hint-based question answering (QA) in NLP and IR.
Large language model (LLM) steering has emerged as a promising paradigm for controlling model behavior at inference time through targeted manipulation of hidden states, offering a lightweight alternative to expensive retraining. However, existing steering frameworks suffer from critical limitations: computational inefficiency, limited extensibility, and restricted functionality that hinder both research progress and practical deployment. We present EasySteer, a unified framework for high-performance, extensible LLM steering built on vLLM. Our system features modular architecture with pluggable interfaces for both analysis-based and learning-based methods, fine-grained parameter control, pre-computed steering vectors for eight application domains, and an interactive demonstration system. Through deep integration with vLLM’s optimized inference engine, EasySteer achieves 10.8-22.3× speedup over existing frameworks. Extensive experiments demonstrate its effectiveness in overthinking mitigation, hallucination reduction, and other key applications. EasySteer transforms steering from research technique to production-ready capability, establishing critical infrastructure for deployable, controllable language models.
The emergence of Large Reasoning Models has introduced exceptionally long Chain-of-Thought traces, creating a transparency burden where critical logic is often buried under massive procedural text. To address this, we present ReasoningLens, an open-source framework designed for the hierarchical visualization and diagnostic auditing of complex reasoning chains. ReasoningLens addresses information necropsy by: (1) structuring traces into interactive hierarchies that separate high-level strategy from low-level execution; (2) leveraging an agentic auditor for automated error detection and tool-augmented verification; and (3) synthesizing systemic reasoning profiles to reveal model-specific blind spots. By transforming unstructured walls of text into actionable insights, ReasoningLens provides a modular foundation for interpreting, debugging, and optimizing the next generation of reasoning-centric AI.
Classical Chinese poetry composition has long valued *Tuiqiao*, the iterative refinement of words, imagery, and prosody. However, many current AI poetry systems follow a one-shot generation paradigm, which reduces users to prompt providers and weakens their creative agency. We present *Jiuge-Tuiqiao*, an interactive human-AI collaborative system for classical Chinese poetry composition. The system is designed around a triadic model: user-driven control, ancient-guided evidence, and AI-assisted generation. Users can lock characters or lines, receive real-time prosody feedback, and obtain interpretable refinement suggestions grounded in high-frequency collocations, PPL-ranked classical lines, and structured knowledge extracted from classical encyclopedias. This design turns AI from an autonomous generator into a background assistant that supports the user’s own process of poetic refinement. Preliminary experiments and user feedback suggest that *Jiuge-Tuiqiao* improves controllability, interpretability, and user engagement in classical poetry composition.
Good tutoring adapts to the individual: it tracks what a learner knows, notices why they go wrong, and asks the next question that will help most. Most deployed tutoring tools instead serve fixed item banks and treat a wrong answer as a single bit of signal. We present CoLearn, an interactive, agentic tutor that supports an iterative tutoring loop: the learner practises, and the system builds an evidence-grounded memory of the learner’s mastery and misconceptions. This memory is updated as evidence accumulates and is used to generate the next personalised question. CoLearn has three components: (i) a persistent learner-state memory that updates per-topic mastery with a soft-evidence variant of Bayesian Knowledge Tracing, where a large language model acts as a continuous ob- servation function; (ii) adaptive question generation that targets the learner’s weakest topic and recurring misconceptions; and (iii) an evi- dence view that makes personalisation visible and testable through live progress visualisation and blind A/B comparison. In blind A/B evaluation, questions conditioned on this memory are preferred over non-personalised ones 68–69% of the time, and in persona simulations with hidden ground-truth mastery the agent’s belief converges toward the learner’s true mastery.
Human evaluation is the gold standard for multilingual NLP, but is often skipped in practice and substituted with automatic metrics, because it is notoriously complex and slow to set up with existing tools with substantial engineering and operational overhead. We introduce Pearmut, a lightweight yet feature-rich platform that makes end-to-end human evaluation as easy to run as automatic evaluation. Pearmut removes common entry barriers and provides support for evaluating multilingual tasks, with a particular focus on machine translation. The platform implements standard evaluation protocols, including DA, ESA, or MQM, but is also extensible to allow prototyping new protocols. It features document-level context, absolute and contrastive evaluation, attention checks, ESAAI pre-annotations and both static and active learning-based assignment strategies. Pearmut enables reliable human evaluation to become a practical, routine component of model development and diagnosis rather than an occasional effort.
Streaming Speech-to-Text Translation (StreamST) requires producing translations concurrently with incoming speech under strict latency constraints, demanding models that balance low latency with high translation quality. Despite rapid progress, evaluation remains fragmented across existing frameworks, which make different assumptions about how systems operate—for example, whether they process continuous speech or short pre-segmented audio, and whether they support output revision (retranslation) or not (incremental) during decoding. As a result, comparing systems fairly and consistently across studies remains challenging. SimulEval, the most widely used framework, reflects these limitations: it supports only incremental decoding, assumes short segmented inputs, and lacks a native support for system demonstrations. More broadly, existing alternatives address only subsets of evaluation and deployment needs, leaving no unified solution for benchmarking and interactive demonstration. To address this gap, we introduce simulstream, the first open-source framework for StreamST evaluation and demonstration. It supports both incremental and re-translation decoding on long-form speech, provides fine-grained logging for quality and latency evaluation, and includes an interactive web interface for real-time visualization and comparison.
Traditional approaches to wearable health signal analysis, such as smartwatches, are constrained by rigid analytical frameworks and limited personalisation. The emergence of LLM agents creates a new opportunity for Personal Health Agentic Analysis, where health insights can be generated adaptively and in context. However, currently there is no open-source locally deployable platform capable of processing personal health data in real time while preserving privacy. We present , a locally deployable, privacy-first agent platform that is fully compatible with real-time health data ecosystems across a wide range of wearable devices. is guided by three design principles. The database is treated as a first-class component. Effectiveness and efficiency are jointly optimised to achieve a low-cost Pareto-optimal balance. Data are processed in real time while the user is modelled over the long term. Together, these principles make it practical for individuals to harness Personal Health Agents for continuous, personalised health monitoring for better wellbeing.
Regulatory compliance checking-deciding whether a target document satisfies the obligations of a regulation-requires interpreting dense legal text, identifying which provisions apply, and grounding each decision in explicit evidence. We present ARCCS, an end-to-end, automated, agentic, and regulation-agnostic Legal NLP system for compliance checking. ARCCS decomposes raw regulatory text into atomic, traceable requirements and evaluates a target document against them using retrieved evidence, confidence scores, and human-interpretable justifications. This design decouples compliance assessment from any fixed regulatory template or predefined rule set, enabling the pipeline to operate over regulations of varying size and structure. We evaluate ARCCS in two complementary settings. First, in a GDPR policy-document evaluation, LLM-based judges find its decisions and justifications legally and evidentially consistent in up to 96.67% of the assessed cases. Second, on an EU public-procurement benchmark comprising more than 1,200 individual rule checks, the system attains 98.8% accuracy in violation detection. ARCCS is, to our knowledge, the first fully open-source system for end-to-end regulatory compliance checking and auditable report generation.
Fairness metrics such as demographic parity and equalized odds summarize group-level behavior, while the location of subgroup errors in the score distribution often remains unclear. RISE-NLP is a diagnostic interface for auditing probabilistic text classifiers through signed residual distributions. Given prediction scores, labels, group metadata, and text, RISE-NLP plots group residual-quantile curves, reports distributional metrics, and links selected percentile regions to the examples behind the gap. Using BERT and RoBERTa predictions, we observe Fdist=0.097--0.150 in the two identity-linked toxicity settings and 0.055--0.058 in a randomly grouped SST-2 control used as a descriptive reference. A 16-participant formative study evaluates users’ understanding and perceived usefulness of the residual visualizations, showing their potential to support reasoning about fairness–accuracy trade-offs. The project is available at https://ufdatastudio.com/projects/risenlp.
Manga pages combine text, layout, characters, and narrative order, making them difficult to process as plain images or text alone. We present MangaLens, an integrated tool for structured manga parsing. MangaLens combines LLM-based reasoning with segmentation-based grounding to support manga parsing subtasks such as panel structuring, speaker linking, and related cross-page reasoning tasks. The system produces page-level and book-level JSON records and visualizes subtask outputs with aligned image evidence, enabling users to verify, correct, and reuse parsing artifacts. Evaluation results show that MangaLens produces reliable structured parsing outputs across major subtasks. The tool is released at https://github.com/nict-astrec-att/MangaLens
Scientific abstracts mix contributions with background, motivation, and meta-language, so tools that read them as-is cannot separate what a field produces from what it discusses. We present Drift Inspector, an open-source system for measuring and exploring how a research field changes over time at the level of Atomic Contribution Claims (ACCs): decontextualized, contribution-bearing propositions an LLM extracts from each abstract before analysis. The system clusters these claims across years into an interactive map where every trend traces back to the claims and papers behind it. Applied to six years of EMNLP, it shows the field shifting away from classic NLP tasks toward LLM-era capabilities such as reasoning and multimodality — a movement that keyword or whole-abstract counts blur. The released data extend beyond EMNLP: the same pipeline has processed the full ACL Anthology (346k claims, 80k abstracts, 423 venues). Extraction is human-validated and clustering checked against an external manually constructed taxonomy.
Online debates on platforms like Reddit or Kialo are structured as reply trees: each message can receive several replies, which in turn can each receive several replies, producing a branching structure rather than a linear conversation. Turning one path (or a mix of paths) through such a tree into a natural, turn-by-turn conversation among multiple speakers (a process we call linearization) requires deciding which messages to keep, in what order, and who each message is addressed to. To this aim, we present LLMberjack, a platform for creating multi-party conversations starting from existing debates, originally structured as reply trees. The system offers an interactive interface that visualizes discussion trees and enables users to construct coherent linearized dialogue sequences while preserving participant identity and discourse relations. It integrates optional large language model (LLM) assistance to support automatic editing of the messages and speakers’ descriptions. We provide preliminary evidence about the platform’s utility by showing how tree visualization facilitates the creation of coherent, meaningful conversation threads and how LLM support enhances output quality while reducing human effort. The tool is open-source and designed to promote transparent and reproducible workflows to create multi-party conversations, addressing a lack of resources of this type.
Autonomous GUI and web agents frequently generate long, multi-step execution traces with rich environmental states, making manual inspection and fault localization challenging. Existing visualization tools typically render isolated single-run trajectories, obscuring fine-grained behavioral discrepancies across parallel runs on the same task. We introduce TRACEFORK, an interactive system designed to align, compare, and inspect paired same-task trajectories produced by AI agents or human demonstrators. TRACEFORK ingests execution traces from standardized benchmarks (e.g., WebArena, VisualWebArena) and custom user logs via an extensible normalization schema. Using a multi-modal dynamic programming alignment engine, the system collapses shared prefixes onto a single trunk, isolates behavioral divergence points into distinct branches, and identifies downstream reconvergence via conservative rejoin cues. Across a library of 1,722 tasks and 1,387 aligned trajectory pairs (33,337 steps), TRACEFORK achieves a median alignment latency of 45.5 ms per pair (p95 320.4 ms). On a manually annotated pilot subset, it achieves an F1 score of 0.910 for divergence localization, offering an open-source, deterministic diagnostic workbench for the agent research community.
Personalized AI assistants in collaborative settings must balance private user context with shared team knowledge; i.e. the same question can have different answers, depending on who is asking. However, retrieving and reasoning over a growing shared archive, as agents do today, increases latency and cost. We present Goldilocks, a multimodal assistive-agent demo that explores a tiered personal memory architecture which combines a lightweight private working resident memory alongside a shared team memory, both grounded in a common eventstore. A sufficiency gate first tries to answer from the low-cost resident tier and only escalates other tiers when needed. To evaluate long-lived personalized collaborative assistants, we also introduce WhoQA, a 500-question evaluation set built from public GitHub history, where work-related questions and answers trace to real pull requests and commits. On WhoQA, Goldilocks’s deployed small-model configuration beats a strong hybrid RAG baseline by 7.0 points (59.0 vs. 52.0) at 13% lower estimated energy (a tier-weighted token cost), and reaches 62.4 (+10.4) with the answer model held fixed. The system serves 42.6% of queries from the resident tier. A compact social-memory scoping further lifts accuracy on teammate-specific personal details from 53 to 80. While we demonstrate with a set tier and scoping configuration, the system is generalizable to multiple tiers depending on the use case. The live demo lets users explore memory tiers, gating decisions, and interactively compare personalized responses across team personas.
Safety evaluation of multimodal large language models requires tracking not only whether an attack succeeds, but also how the interaction unfolds across turns and input modalities. We present MUSE (Multimodal Unified Safety Evaluation), an open-source, browser-based, run-centric platform for multimodal safety evaluation. MUSE treats each attack run as the persistent unit of execution, inspection, and analysis, preserving its configuration, multi-turn trajectory, delivered modalities and media, target responses, and safety judgments. A five-level response taxonomy further distinguishes full Compliance from Partial Compliance and refusal behavior, yielding hard ASR, soft ASR, and gray-zone width (GZW). Across 11,700 evaluations on six multimodal LLMs, direct text-only requests yield only 3.1% macro hard ASR and 4.4% soft ASR, while iterative attack procedures are substantially more effective. Attack effectiveness also varies substantially with the attacker backbone. In contrast, Inter-Turn Modality Switching (ITMS), evaluated as a controlled delivery-modality probe, does not consistently increase attack success. These results demonstrate the value of run-centric, fine-grained evaluation for characterizing multimodal safety behavior beyond a single binary success metric.
We present ‘gradiend‘, an open-source Python package that operationalizes the GRADIEND method for learning feature directions from factual-counterfactual MLM and CLM gradients in language models. The package provides a unified workflow for feature-related data creation, training, evaluation, visualization, persistent model rewriting via controlled weight updates, and multi-feature comparison. We demonstrate ‘gradiend‘ through an English pronoun running example, a semantic sentiment use case that evaluates lexical generalization to held-out target words, and a large-scale feature comparison.
Email is becoming the universal interface for AI agents, but an ingested message contains un- trusted, malicious data the agent may execute as instructions. The indirect prompt-injection failure underlies real-world incidents such as EchoLeak (M365 Copilot, CVE-2025-32711) and the Gemini-in-Gmail hidden-HTML hijack. We present e2a, a deployable, open-source authenticated-email gateway that gives an agent a verified email identity and exposes two agent- agnostic defense hook points. The (G)ateway authenticates senders via SPF/DKIM/DMARC and parses MIME structure; the (L)LM-layer screens the normalized prompt for injection. On top of the gateway, we construct a bench- mark that isolates email structure as the first- class injection surface: one payload rendered across eight transport surfaces. We measure the per-surface evasion delta for open-source classifiers, three commercial security APIs, and frontier LLM judges. We find that structural hiding defeats detection. A naive text/plain- first parse costs every detector substantial AUC, dropping most classifiers below chance, and even on the gateway’s structure-aware view no non-LLM detector survives. A frontier LLM judge on the same view recovers most of the loss, and conditioning the threshold on crypto- graphic sender authentication (DMARC) lifts held-out recall further. No single layer suf- fices. e2a is live and open-source (Apache-2.0) at https://e2a.dev, released with the bench- mark and harness.
The Model Context Protocol (MCP) lets large language model agents call external tools, and that tool channel has become a security liability: poisoned tool metadata, injected instructions in tool responses, and malicious servers can subvert an agent while never touching the user’s prompt. These risks are increasingly well documented, but hard to experience: most defenses exist as prose or prototypes. We present SHIELDMCP, an interactive, deployment-ready system that makes both the attack and the runtime defense tangible in a browser. SHIELDMCP operates as a transparent proxy between the LLM client and any MCP server, performing threestage validation: pre-call description integrity, outbound parameter sanitization, and postresponse analysis, all without modifying either the agent or the server. The demonstration exposes the full pipeline through a browser interface in which visitors launch curated attacks, watch each validation stage execute live with per-token highlighting of the detected payload, and submit their own tool descriptions, call parameters, or tool responses to a playground that runs the real detector. On a framework benchmark of 487 attack and 200 benign scenarios across 40 MCP servers, the default rule-based backend reduces the overall attack success rate to 8.6% (from a 70.9% undefended baseline measured against live agents in a companion study) at a median added latency under 20 ms; an optional fine-tuned classifier backend reaches near-zero on this benchmark, though we show this is a withindistribution result that does not yet transfer to paraphrased, multilingual, or encoded attacks. SHIELDMCP is open-source under the Apache-2.0 license and installable with a single pip command; every framework number in this paper is reproducible from the shipped harness.
Large language models (LLMs) are increasingly evolving from conversational assistants into agents capable of operating external digital environments. Graphical user interface (GUI) agents are an important part of this shift, as many real-world workflows remain accessible only through user-facing software interfaces. However, despite recent progress on general computer-use benchmarks, domain-specific professional standard operating procedures (SOPs) remain challenging for GUI agents because they often involve implicit domain knowledge, software-specific conventions, and task-level verification requirements. We introduce OmegaUse-SOP, a human-in-the-loop SOP engineering system for transforming human demonstrations of professional computer use into reusable SOP skills for GUI agents. Analogous to prompt engineering, SOP Engineering iteratively refines demonstrations, execution rules, and domain knowledge to convert professional SOPs into reusable GUI-agent skills. OmegaUse-SOP consists of four modules: Observe, Reason, Configure, and Execute. Together, these modules record expert operations as multimodal GUI traces, abstract low-level events into semantic step-level instructions, incorporate domain rules and task-specific parameters, and execute the resulting skills in live GUI environments through step-wise grounding, action generation, and verification. To demonstrate its effectiveness, we collaborate with a power-sector client and test OmegaUse-SOP on photovoltaic simulation workflows in PVsyst 7.2. The results suggest that OmegaUse-SOP can improve GUI-agent reliability on professional SOP tasks, highlighting a practical path toward deploying GUI agents in domain-specific professional software environments.
Agentic coding tools present new opportunities to transform research workflows. The performance of agent systems built depends on both large language models (LLMs) and the harness around LLMs, which is the orchestration code that determines an agent’s behavior. We present ToFu, an agentic harness for researchers that reads your codebase, edits files, runs commands, and integrates with your development tools. ToFu plays a dual role in research. As a research assistant, it supports practical research workflows with superior token efficiency, lower cost, and multilingual capability compared with existing agentic harnesses. Its release under the MIT License further enables local deployment for privacy-sensitive users. As a research object, ToFu provides a white-box agentic harness that allows researchers to inspect, modify, and evaluate its orchestration logic, tool-use behavior, and harness design, while retaining strong benchmark performance and an application-level user experience.
Ancient manuscripts are the primary source of ancient linguistic corpora. However, some of them exhibit duplications due to unintentional repeated publication or deliberate forgery. Notable examples include counterfeit fragments within the Dead Sea Scrolls and the repeated publication of Oracle Bones (OB). Identifying such duplicates is critical for both archaeological curation and historical research. We present OBD-Finder, an explainable coarse-to-fine framework for large-scale duplicate discovery in ancient manuscript collections, with a primary focus on Oracle Bones. OBD-Finder integrates unsupervised low-level keypoint matching with high-level character-centric content similarity to progressively filter and rank duplicate candidates, balancing retrieval accuracy with computational efficiency. Designed as a practical, transparent, and domain-expert-friendly tool, the system produces candidate duplicates and supports expert inspection and validation. In real-world deployment, OBD-Finder has enabled domain experts to uncover 63 previously unknown duplicate Oracle Bone pairs that had remained overlooked for decades. Extensive experiments also confirm that OBD-Finder achieves competitive retrieval accuracy while offering substantially superior efficiency and interpretability compared to existing approaches. Code and deployed system are publicly available: https://github.com/cszhangLMU/OBD-Finder/.
Developing enterprise customer-service assistants with Reinforcement Learning (RL) critically depends on high-fidelity user simulators and well-aligned reward functions, since real-customer interactions are impractical for large-scale training and rule-based rewards fail to capture complex business requirements. We present CoEvoLoop, a framework that co-evolves the assistant policy together with these RL components through an expert-calibrated, multi-round loop. Beyond an LLM-based simulator for rollout generation and a multi-dimensional grader for rubric-based evaluation of service quality and compliance, CoEvoLoop integrates an expert annotation interface where sampled trajectories and grader judgments are reviewed and corrected. Each round, expert feedback updates the simulator and grader, after which the assistant is optimized with trajectory-level RL against the updated evaluation signal; monitoring interfaces support rollout inspection, dimension-wise failure attribution, and cross-round tracking. In an industrial evaluation, the system improves multi-turn service quality and reduces compliance violations over training rounds; component ablations and blind expert evaluation confirm that co-evolving the simulator and grader is load-bearing for these gains. We release the system demo code, along with a demo video.
As Large Language Models (LLMs) evolve into autonomous agents, the need for unified evaluation platform becomes critical. However, current evaluation pipelines remain highly fragmented and tightly coupled, hindering reproducibility and causing redundant engineering. To address this, we introduce AgentCompass, an open-source, lightweight, and extensible platform for evaluating LLM-based agents. AgentCompass organizes the evaluation process around three independent components, namely Benchmark, Harness, and Environment, thereby enabling flexible configurations without requiring the reimplementation of complex execution logic. Furthermore, it features a fault-tolerant asynchronous runtime and comprehensive trajectory analysis tools to transparently diagnose nuanced failure modes like reward-hacking. Natively supporting over 20 benchmarks across five capability dimensions, AgentCompass provides the community with a scalable and reproducible infrastructure for advancing agent research.