| Total: 438
Given a source image of a clothed person (an image subject), AI-based nudification applications can produce nude (undressed) images of that person. Moreover, not only do such applications exist, but there is ample evidence of the use of such applications in the real world and without the consent of an image subject. Still, despite the growing awareness of the existence of such applications and their potential to violate the rights of image subjects and cause downstream harms, there has been no systematic study of the nudification application ecosystem across multiple applications. We conduct such a study here, focusing on 20 popular and easy-to-find nudification websites. We study the positioning of these web applications (e.g., finding that most sites explicitly target the nudification of women, not all people), the features that they advertise (e.g., ranging from undressing-in-place to the rendering of image subjects in sexual positions, as well as differing user-privacy options), and their underlying monetization infrastructure (e.g., credit cards and cryptocurrencies). We believe this work will empower future, data-informed conversations—within the scientific, technical, and policy communities—on how to better protect individuals' rights and minimize harm in the face of modern (and future) AI-based nudification applications. redContent warning: This paper includes descriptions of web applications that can be used to create synthetic non-consensual explicit AI-created imagery (SNEACI). This paper also includes an artistic rendering of a user interface for such an application.
The rapid growth of mobile apps has provided convenience and entertainment, including adult-oriented apps for users 18 and older. Despite various strategies to prevent minors from accessing such content, the effectiveness of these measures remains uncertain. This paper investigates these mechanisms and proposes a novel detection solution: GUARD (Guarding Underage Access Restriction Detection). GUARD determines relevant components (e.g., those that can accept the user's age or birthdate) based on the spatial relationships of the components in a layout and tracks the data flows through taint analysis. Recognizing static analysis limitations, GUARD also dynamically interacts with apps to identify age-related input components, which are then used for precise taint analysis. Our analysis of 31,750 adult-only apps (out of 693,334 apps on Google Play) reveals that only 1,165 (3.67%) implement age verification, with the majority relying on the weakest method, the age gate (which simply asks users if they are over 18). Even apps with stronger age verification (e.g., document uploads, online ID verification) can be bypassed using simple methods like false IDs or fake documents. They can also be circumvented through accounts from services without age checks (e.g., OAuth abuse) or by exploiting regional differences via VPNs. This paper also proposes countermeasures to enhance the effectiveness of age verification methods, which received positive feedback from Google through our email exchanges.
Automation apps such as iOS Shortcuts and Android Tasker enable users to "program" new functionalities, also called recipes, on their smartphones. For example, users can create recipes to set the phone to silent mode once they arrive at their office or save a note when an email is received from a particular sender. These automation apps provide convenience and can help improve productivity. However, these automation apps can also provide new avenues for abuse, particularly in the context of intimate partner violence (IPV). This paper systematically explores the potential of automation apps to be used for surveillance and harassment in IPV scenarios. We analyze four popular automation apps—iOS Shortcuts, Samsung Modes & Routines, Tasker, and IFTTT—evaluating their capabilities to facilitate surveillance and harassment. Our study reveals that these tools can be exploited by abusers today to monitor, impersonate, overload, and control their victims. The current notification and logging mechanisms implemented in these automation apps are insufficient to warn the victim about the abuse or to help them identify the root cause and stop it. We therefore built a detection mechanism to identify potentially malicious Shortcuts recipes and tested it on 12,962 publicly available Shortcuts recipes. We found 1,014 recipes that can be used to surveil and harass others. We then discuss how users and platforms mitigate such abuse potential of automation apps.
LLM-based Conversational AIs (CAIs), also known as GenAI chatbots, like ChatGPT, are increasingly used across various domains, but they pose privacy risks, as users may disclose personal information during their conversations with CAIs. Recent research has demonstrated that LLM-based CAIs could be used for malicious purposes. However, a novel and particularly concerning type of malicious LLM application remains unexplored: an LLM-based CAI that is deliberately designed to extract personal information from users. In this paper, we report on the malicious LLM-based CAIs that we created based on system prompts that used different strategies to encourage disclosures of personal information from users. We systematically investigate CAIs' ability to extract personal information from users during conversations by conducting a randomized-controlled trial with 502 participants. We assess the effectiveness of different malicious and benign CAIs to extract personal information from participants, and we analyze participants' perceptions after their interactions with the CAIs. Our findings reveal that malicious CAIs extract significantly more personal information than benign CAIs, with strategies based on the social nature of privacy being the most effective while minimizing perceived risks. This study underscores the privacy threats posed by this novel type of malicious LLM-based CAIs and provides actionable recommendations to guide future research and practice.
Many software products are composed of components integrated from other teams or external parties. Each additional link in a software product's supply chain increases the risk of the injection of malicious behavior. To improve supply chain provenance, many cybersecurity frameworks, standards, and regulations recommend the use of software signing. However, recent surveys and measurement studies have found that the adoption rate and quality of software signatures are low. We lack in-depth industry perspectives on the challenges and practices of software signing. To understand software signing in practice, we interviewed 18 experienced security practitioners across 13 organizations. We study the challenges that affect the effective implementation of software signing in practice. We also provide possible impacts of experienced software supply chain failures, security standards, and regulations on software signing adoption. To summarize our findings: (1) We present a refined model of the software supply chain factory model highlighting practitioner's signing practices; (2) We highlight the different challenges–technical, organizational, and human–that hamper software signing implementation; (3) We report that experts disagree on the importance of signing; and (4) We describe how internal and external events affect the adoption of software signing. Our work describes the considerations for adopting software signing as one aspect of the broader goal of improved software supply chain security.
In recent years there has been significant interest from policymakers in addressing ransomware through policy and regulations, yet this process remains far more of an art than a science. This paper introduces a novel method for quantitatively evaluating policy proposals: we create a simulated game theoretic agent-based economic model of security and use it as a testbed for several policy interventions, including a hands-off approach, mandatory minimum investments, and mandatory cyber insurance. Notably, we find that the bottleneck for better security outcomes lies not in better defender decision-making but in improved coordination between defenders: using our model, we find that a policy requiring defenders to invest at least 2% of resources into security each round produces better overall outcomes than leaving security investment decisions to defenders even when the defenders are "perfect play" utility maximizers. This provides evidence that security is a weakest-link game and makes the case for mandatory security minimums. Using our model, we also find that cyber insurance does little to improve overall outcomes. To make our tool accessible to others, we have made the code open source and released it as an online web application.
To combat the deluge of enterprise breaches, government agencies have developed and published a wealth of cybersecurity guidance for organizations. However, little research has studied this advice. In this paper, we conduct the first systematic analysis of government guidance for enterprise security. We curate a corpus of prominent guidance documents from 41 countries and analyze the availability of advice, the coverage provided by the advice, and the consistency of advice across countries. To facilitate detailed analysis and comparisons, we develop a tree-based taxonomy and quantitative comparison metric, and then apply these tools to analyze "essential" enterprise best practice documents from ten countries. Our results highlight a lack of consensus among the governments' frameworks we analyzed—even among close allies—about what security measures to recommend and how to present guidance.
Advances in generative models have transformed the field of synthetic image generation for privacy-preserving data synthesis (PPDS). However, the field lacks a comprehensive survey and comparison of synthetic image generation methods across diverse settings. In particular, when we generate synthetic images for the purpose of training a classifier, there is a pipeline of generation-sampling-classification which takes private training as input and and outputs the final classifier of interest. In this survey, we systematically categorize existing image synthesis methods, privacy attacks, and mitigations along this generation-sampling-classification pipeline. To empirically compare diverse synthesis approaches, we provide a benchmark with representative generative methods and use model-agnostic membership inference attacks (MIAs) as a measure of privacy risk. Through this study, we seek to answer critical questions in PPDS: Can synthetic data effectively replace real data? Which release strategy balances utility and privacy? Do mitigations improve the utility-privacy tradeoff? Which generative models perform best across different scenarios? With a systematic evaluation of diverse methods, our study provides actionable insights into the utilty-privacy tradeoffs of synthetic data generation methods and guides the decision on optimal data releasing strategies for real-world applications.
Information-based attacks on social media, such as disinformation campaigns and propaganda, are emerging cybersecurity threats. The security community has focused on countering these threats on social media platforms like X and Reddit. However, they also appear in instant-messaging social media platforms such as WhatsApp, Telegram, and Signal. In these platforms, information-based attacks primarily happen in groups and channels, requiring manual moderation efforts by channel administrators. We collect, label, and analyze a large dataset of more than 17 million Telegram comments and messages. Our analysis uncovers two independent, coordinated networks that spread pro-Russian and pro-Ukrainian propaganda, garnering replies from real users. We propose a novel mechanism for detecting propaganda that capitalizes on the relationship between legitimate user messages and propaganda replies and is tailored to the information that Telegram makes available to moderators. Our method is faster, cheaper, and has a detection rate (97.6%) 11.6 percentage points higher than human moderators after seeing only one message from an account. It remains effective despite evolving propaganda.
In this paper, we introduce GradEscape, the first gradient-based evader designed to attack AI-generated text (AIGT) detectors. GradEscape overcomes the undifferentiable computation problem, caused by the discrete nature of text, by introducing a novel approach to construct weighted embeddings for the detector input. It then updates the evader model parameters using feedback from victim detectors, achieving high attack success with minimal text modification. To address the issue of tokenizer mismatch between the evader and the detector, we introduce a warm-started evader method, enabling GradEscape to adapt to detectors across any language model architecture. Moreover, we employ novel tokenizer inference and model extraction techniques, facilitating effective evasion even in query-only access. We evaluate GradEscape on four datasets and three widely-used language models, benchmarking it against four state-of-the-art AIGT evaders. Experimental results demonstrate that GradEscape outperforms existing evaders in various scenarios, including with an 11B paraphrase model, while utilizing only 139M parameters. We have successfully applied GradEscape to two real-world commercial AIGT detectors. Our analysis reveals that the primary vulnerability stems from disparity in text expression styles within the training data. We also propose a potential defense strategy to mitigate the threat of AIGT evaders. We open-source our GradEscape for developing more robust AIGT detectors.
Large Language Models (LLMs) have demonstrated remarkable capabilities of generating texts resembling human language. However, they can be misused by criminals to create deceptive content, such as fake news and phishing emails, which raises ethical concerns. Watermarking is a key technique to address these concerns, which embeds a message (e.g., a bit string) into a text generated by an LLM. By embedding the user ID (represented as a bit string) into generated texts, we can trace generated texts to the user, known as content source tracing. The major limitation of existing watermarking techniques is that they achieve sub-optimal performance for content source tracing in real-world scenarios. The reason is that they cannot accurately or efficiently extract a long message from a generated text. We aim to address the limitations. In this work, we introduce a new watermarking method for LLM-generated text grounded in pseudo-random segment assignment. We also propose multiple techniques to further enhance the robustness of our watermarking algorithm. We conduct extensive experiments to evaluate our method. Our experimental results show that our method achieves a much better tradeoff between extraction accuracy and time complexity, compared with existing baselines. For instance, when embedding a message of length 20 into a 200-token generated text, our method achieves a match rate of 97.6%, while the state-of-the-art work Yoo et al. only achieves 49.2%. Additionally, we prove that our watermark can tolerate edits within an edit distance of 17 on average for each paragraph under the same setting.
Large Language Models (LLMs) have raised increasing concerns about their misuse in generating hate speech. Among all the efforts to address this issue, hate speech detectors play a crucial role. However, the effectiveness of different detectors against LLM-generated hate speech remains largely unknown. In this paper, we propose HateBench, a framework for benchmarking hate speech detectors on LLM-generated hate speech. We first construct a hate speech dataset of 7,838 samples generated by six widely-used LLMs covering 34 identity groups, with meticulous annotations by three labelers. We then assess the effectiveness of eight representative hate speech detectors on the LLM-generated dataset. Our results show that while detectors are generally effective in identifying LLM-generated hate speech, their performance degrades with newer versions of LLMs. We also reveal the potential of LLM-driven hate campaigns, a new threat that LLMs bring to the field of hate speech detection. By leveraging advanced techniques like adversarial attacks and model stealing attacks, the adversary can intentionally evade the detector and automate hate campaigns online. The most potent adversarial attack achieves an attack success rate of 0.966, and its attack efficiency can be further improved by 13-21x through model stealing attacks with acceptable attack performance. We hope our study can serve as a call to action for the research community and platform moderators to fortify defenses against these emerging threats.
Large language models (LLMs) nowadays have attracted an affluent user base due to the superior performance across various downstream tasks. Yet, recent works reveal that LLMs are vulnerable to backdoor attacks, where an attacker can inject a specific token trigger to manipulate the model's behaviors during inference. Existing efforts have largely focused on single-trigger attacks while ignoring the variations in different users' responses to the same trigger, thus often resulting in undermined attack effectiveness. In this work, we propose EmbedX, an effective and efficient cross-trigger backdoor attack against LLMs. Specifically, EmbedX exploits the continuous embedding vector as the soft trigger for backdooring LLMs, which enables trigger optimization in the semantic space. By mapping multiple tokens into the same soft trigger, EmbedX establishes a backdoor pathway that links these tokens to the attacker's target output. To ensure the stealthiness of EmbedX, we devise a latent adversarial backdoor mechanism with dual constraints in frequency and gradient domains, which effectively crafts the poisoned samples close to the target samples. Through extensive experiments on four popular LLMs across both classification and generation tasks, we show that EmbedX achieves the attack goal effectively, efficiently, and stealthily while also preserving model utility.
Recent advances in Large Language Models (LLMs) have led to impressive alignment—where models learn to distinguish harmful from harmless queries through supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF). In this paper, we reveal a subtle yet impactful weakness in these aligned models. We find that simply appending multiple end-of-sequence (eos) tokens can cause a phenomenon we call "context segmentation", which effectively shifts both "harmful" and "benign" inputs closer to the refusal boundary in the hidden space. Building on this observation, we propose a straightforward method to BOOST jailbreak attacks by appending eos tokens. Our systematic evaluation shows that this strategy significantly increases the attack success rate across 8 representative jailbreak techniques and 16 open-source LLMs, ranging from 2B to 72B parameters. Moreover, we develop a novel probing mechanism for commercial APIs and discover that major providers—such as OpenAI, Anthropic, and Qwen—do not filter eos tokens, making them similarly vulnerable. These findings highlight a hidden yet critical blind spot in existing alignment and content filtering approaches. We call for heightened attention to eos tokens' unintended influence on model behaviors, particularly in production systems. Our work not only calls for an input-filtering based defense, but also points to new defenses that make refusal boundaries more robust and generalizable, as well as fundamental alignment techniques that can defend against context segmentation attacks.
Utilizing Trusted Execution Environments (TEEs) to protect Large Language Models (LLMs) on users' devices is a practical solution for model owners. To alleviate the computation burden on TEEs, researchers have proposed TEE-Shielded LLM Partition (TSLP) to offload heavy computation layers to co-operating untrusted GPUs, while lightweight layers are shielded in TEE. TSLP utilizes various lightweight obfuscation schemes to protect offloaded weights from various attacks meanwhile not introducing large computation overhead. However, existing lightweight obfuscation algorithms have one vital vulnerability in common: the direction similarity of obfuscated vectors. In this paper, we propose a novel attack, ArrowMatch, that utilizes direction similarity to recover obfuscated private weights. To achieve this, ArrowMatch compares direction distances between obfuscated model weights and public pre-trained model weights. To mitigate this vulnerability, we propose a novel obfuscation scheme, ArrowCloak, which leverages lightweight matrix-vector multiplication to protect vector directions and private weights. We evaluate ArrowMatch and ArrowCloak on four representative LLMs, using seven datasets, along with five obfuscation schemes. The results show that ArrowMatch can break the protection of all existing lightweight obfuscation schemes with high accuracy (similar to no protection) and effectively recover the private weights (with over 98% accuracy). In addition, ArrowCloak can effectively defend against ArrowMatch (6.5X better than state of the art) and protect direction information by increasing the direction distance over 900X. We also evaluate the performance of ArrowCloak on a real-world Intel SGX device and show that ArrowCloak can reduce total overhead by 2.83X compared to shield-the-whole baseline.
We introduce LLMmap, a first-generation fingerprinting technique targeted at LLM-integrated applications. LLMmap employs an active fingerprinting approach, sending carefully crafted queries to the application and analyzing the responses to identify the specific LLM version in use. Our query selection is informed by domain expertise on how LLMs generate uniquely identifiable responses to thematically varied prompts. With as few as 8 interactions, LLMmap can accurately identify 42 different LLM versions with over 95% accuracy. More importantly, LLMmap is designed to be robust across different application layers, allowing it to identify LLM versions —whether open-source or proprietary— from various vendors, operating under various unknown system prompts, stochastic sampling hyperparameters, and even complex generation frameworks such as RAG or Chain-of-Thought. We discuss potential mitigations and demonstrate that, against resourceful adversaries, effective countermeasures may be challenging or even unrealizable.
Safety alignment has become an indispensable procedure to ensure the safety of large language models (LLMs), as they are reported to generate harmful, privacy-sensitive, and copy-righted content when prompted with adversarial instructions. Machine unlearning is a representative approach to establishing the safety of LLMs, enabling them to forget problematic training instances and thereby minimize their influence. However, no prior study has investigated the feasibility of adversarial unlearning—using seemingly legitimate unlearning requests to compromise the safety of a target LLM. In this paper, we introduce novel attack methods designed to break LLM safety alignment through unlearning. The key idea lies in crafting unlearning instances that cause the LLM to forget its mechanisms for rejecting harmful instructions. Specifically, we propose two attack methods. The first involves explicitly extracting rejection responses from the target LLM and feeding them back for unlearning. The second attack exploits LLM agents to obscure rejection responses by merging them with legitimate-looking unlearning requests, increasing their chances of bypassing internal filtering systems. Our evaluations show that these attacks significantly compromise the safety of two open-source LLMs: LLaMA and Phi. LLaMA's harmfulness scores increase by an average factor of 11 across four representative unlearning methods, while Phi exhibits a 61.8× surge in the rate of unsafe responses. Furthermore, we demonstrate that our unlearning attack is also effective against OpenAI's fine-tuning service, increasing GPT-4o's harmfulness score by 2.21×. Our work identifies a critical vulnerability in unlearning and represents an important first step toward developing safe and responsible unlearning practices while honoring users' unlearning requests. Our code is available at https://doi.org/10.5281/zenodo.15628860.
Large Language Models (LLMs) have showcased remarkable capabilities across various domains. Accompanying the evolving capabilities and expanding deployment scenarios of LLMs, their deployment challenges escalate due to their sheer scale and the advanced yet complex activation designs prevalent in notable model series, such as Llama, Gemma, Mistral. These challenges have become particularly pronounced in resource-constrained deployment scenarios, where mitigating inference bottlenecks is imperative. Among various recent efforts, activation approximation has emerged as a promising avenue for pursuing inference efficiency, sometimes considered indispensable in applications such as private inference. Despite achieving substantial speedups with minimal impact on utility, even appearing sound and practical for real-world deployment, the safety implications of activation approximations remain unclear. In this work, we fill this critical gap in LLM safety by conducting the first systematic safety evaluation of activation approximations. Our safety vetting spans seven state-of-the-art techniques across three popular categories (activation polynomialization, activation sparsification, and activation quantization), revealing consistent safety degradation across ten safety-aligned LLMs. To overcome the hurdle of devising a unified defense accounting for diverse activation approximation methods, we perform an in-depth analysis of their shared error patterns and uncover three key findings. We propose QuadA, a novel safety enhancement method tailored to mitigate the safety compromises introduced by activation approximations. Extensive experiments and ablation studies corroborate QuadA's effectiveness in enhancing the safety capabilities of LLMs after activation approximations.
Streaming services like Netflix, Prime Video, and HBO Max rely on DRM solutions to ward off piracy. By enabling the distribution of encrypted content, DRM systems prevent subscribed users from downloading the streamed content, as well as unauthorized users from having access to it. Google Widevine, one of the most deployed DRMs, provides a fully software-based solution on desktop platforms to ensure portability. In this paper, we empirically investigate the security protections implemented by Widevine to counter an attacker tampering with its interactions within its environment, namely with the operating system and the hosting browser. Focusing on randomness and time, we uncover new flaws in the Widevine license acquisition process, particularly targeting the freshness and expiration of the licenses. To demonstrate the effectiveness of our findings, we develop Narrowbeer, a practical replay attack allowing legitimate users to generate never-expiring licenses, and enabling unauthorized users to reuse these licenses to access premium content without subscription. Finally, we validate our attack against real-world streaming services by succeeding in repeatedly playing the same license on different desktop devices.
Vulnerability and exploit analysis are at the heart of soft- ware security research and practice. However, a formalization framework for dissecting the cause, development, and impact of common software errors has been missing. To address this gap, we introduce Lancet, a formalization framework that reliably tracks three distinct types of ownership within its operational semantics that can be used to identify and differ- entiate between various types of vulnerabilities and exploit primitives even in the presence of memory corruption. Addi- tionally, we developed two downstream tools, FCS and EPF, to demonstrate how security analysts can use Lancet for de- tailed crash and exploit analysis. FCS serves as a fast crash triaging tool, aiding patch synthesis in our winning system in the DARPA AIxCC semi-final, while EPF fingerprints the transition of exploitation primitives to facilitate exploit analy- sis. Experiment results show that both tools are efficient and effective.
We present a new method for automatically synthesizing code-reuse attacks—for example, using Return Oriented Programming—based on mechanized formal logic. Our method reasons about machine code via abstraction to the p-code intermediate language of Ghidra, a well-established software reverse-engineering framework. This allows it to be applied to binaries of essentially any architecture, and provides certain technical advantages. We define a formal model of a fragment of p-code in propositional logic, enabling analysis by automated reasoning algorithms. We then synthesize code-reuse attacks by identifying selections of gadgets that can emulate a given p-code reference program. This enables our method to scale well, in both reference program and gadget library size, and facilitates integration with external tools. Our method matches or exceeds the success rate of state-of-the-art ROP chain synthesis methods while providing improved runtime performance.
Data-oriented programming (DOP) is a methodology for embedding malicious programs into fixed executable vulnerable binaries. DOP is effective for implementing code reuse attacks that exploit memory corruptions without violating many defence techniques, such as non-execute, address space layer randomisation, control flow and code point integrity. Existing approaches for automated exploit generation for DOP follow the program synthesis approach: given a description of an attack phrased as a program, they perform extensive constraint-based search to identify the required payload for the corrupted memory. The program synthesis-inspired approaches come with three major shortcomings regarding (a) efficiency: attack generation often takes prohibitively large amount of time, (b) soundness: they provide no formal guarantees whatsoever that a particular user-described attack is feasible in a particular vulnerable program with suitable payloads, and (c) capability visibility: they do not make clear to users what attack capabilities are admitted by the vulnerable program. In this work, we propose a novel approach to synthesise code reuse attacks via DOP by casting this task as an instance of the previously unexplored programming language synthesis idea. Given a vulnerable program and an exploit (e.g., buffer overflow), our approach derives a grammar of a programming language for describing the available attacks. Our approach addresses the issue (a) by shifting the cost of synthesising individual attacks to synthesising the entire attack language: once the grammar is generated, the compilation of each attack takes negligible time. The issues (b) and (c) are addressed by establishing correctness of our grammar synthesis algorithm: any attack expressible in terms of a generated grammar is realisable. We implement our approach in a tool called DOPPLER—an end-to-end compiler for DOP-based attacks. We evaluate DOPPLER against available state-of-the art techniques on a set of 17 case studies, including three recent CVEs, demonstrating its improved effectiveness (it generates more attacks) and efficiency (it does so much faster).
ZIP is one of the most popular archive formats. It is used not only as archive files, but also as the container for other file formats, including office documents, Android applications, Java archives, and many more. Despite its ubiquity, the ZIP file format specification is imprecisely specified, posing the risk of semantic gaps between implementations that can be exploited by attackers. While prior research has reported individual such vulnerabilities, there is a lack of systematic studies for ZIP parsing ambiguities. In this paper, we developed a differential fuzzer ZipDiff and systematically identified parsing inconsistencies between 50 ZIP parsers across 19 programming languages. The evaluation results show that almost all pairs of parsers are vulnerable to certain parsing ambiguities. We summarize our findings as 14 distinct parsing ambiguity types in three categories with detailed analysis, systematizing current knowledge and uncovering 10 types of new parsing ambiguities. We demonstrate five real-world scenarios where these parsing ambiguities can be exploited, including bypassing secure email gateways, spoofing office document content, impersonating VS Code extensions, and tampering with signed nested JAR files while still passing Spring Boot's signature verification. We further propose seven mitigation strategies to address these ambiguities. We responsibly reported the vulnerabilities to the affected vendors and received positive feedback, including bounty rewards from Gmail, Coremail, and Zoho, and three CVEs from Go, LibreOffice, and Spring Boot.
Disassembly is a crucial yet challenging step in binary analysis. While emerging neural disassemblers show promise for efficiency and accuracy, they frequently generate outputs violating fundamental structural constraints, which significantly compromise their practical usability. To address this critical problem, we regularize the disassembly solution space by formalizing and applying key structural constraints based on post-dominance relations. This approach systematically detects widespread errors in existing neural disassemblers' outputs. These errors often originate from models' limited context modeling and instruction-level decoding that neglect global structural integrity. We introduce Tady, a novel neural disassembler featuring an improved model architecture and a dedicated post-processing algorithm, specifically engineered to address these deficiencies. Comprehensive evaluations on diverse binaries demonstrate that Tady effectively eliminates structural constraint violations and functions with high efficiency, while maintaining instruction-level accuracy.
Reproducibility has been an increasingly important focus within the Security Community over the past decade. While showing great promise for increasing the quantity and quality of available artifacts, reproducibility alone only addresses some of the challenges to establishing experimental validity in scientific research and is not enough to move forward our discipline. Instead, replicability is required to test the bounds of a hypothesis and ultimately show consistent evidence to a scientific theory. Although there are clear benefits to replicability, it remains imprecisely defined, and a formal framework to reason about and conduct replicability experiments is lacking. In this work, we systematize over 30 years of research and recommendations on the topics of reproducibility, replicability, and validity, and argue that their definitions have had limited practical application within Computer Security. We address these issues by providing a framework for reasoning about replicability, known as the Tree of Validity (ToV). We evaluate an attack and a defense to demonstrate how the ToV can be applied to threat modeling and experimental environments. Further, we show two papers with Distinguished Artifact Awards and demonstrate that true reproducibility is often unattainable; however, meaningful comparisons are still attainable by replicability. We expand our analysis of two recent SoK papers, themselves replicability studies, and demonstrate how these papers recreate multiple paths through their respective ToVs. In so doing, we are the first to provide a practical framework of replicability with broad applications for, and beyond, the Security research community.