We study how latent (sleeper) attacks can exploit persistent memory in multi-agent LLM systems to cause delayed, cross-domain behavioral manipulation.
About the project
Modern LLM systems become stateful through the use of persistent memory, automated memory management (insertion, deletion, compression), and multi-agent coordination over long time horizons. While most prior work on LLM security has focused on prompt injection[1] and tool misuse[2], the security implications of persistent memory are largely underexplored [3]. In our prior SPAR (Fall ’25) project, we empirically demonstrated that persistent memory can lead to undesirable cross-domain leakage, where information introduced in one context can silently influence behavior in unrelated downstream tasks.
This suggests that persistent memory can be exploited as a further input surface to compromise the model behavior. Considering this risk, we aim to investigate latent (sleeper) attacks[4] on memory in multi-agent LLM systems. Here an external adversary indirectly manipulates memory state processes with the goal of implanting instructions that have long term malicious implications in future conversations. The main research question is: How can adversaries exploit persistent memory and memory management mechanisms to cause delayed, cross-task, and cross-domain behavioral manipulation in agentic LLM systems, and how can such attacks be systematically detected and potentially mitigated?
The project aims to implement realistic multi-agent system with persistent memory (e.g., ingestion agent, memory manager, and task/recommendation agent) and evaluate sleeper attacks under realistic threat models where the adversary can only interact through standard inputs (documents, browsing, user interaction), without direct access to memory or model weights.
Example (Motivating Scenario). Consider a retail webpage for a fizzy drink that embeds subtle instructions instructing the system to (i) delete stored user preferences for healthy products and (ii) record a strong preference for their specific drink. When ingested by an LLM-based agent with persistent memory, this information may be stored and later influence unrelated tasks. This may cause the system to preferentially recommend the drink in future conversations—without any explicit user request or visible prompt.
We will measure long-term behavioral compromise using metrics such as recommendation bias, policy compliance drift, and safety degradation across extended cross multi session interactions. We will also aim to explore defense mechanisms.
[1]Greshake, Kai, et al. "Not what you've signed up for: Compromising real-world llm-integrated applications with indirect prompt injection." Proceedings of the 16th ACM workshop on artificial intelligence and security. 2023. [2]Zhang, Zhexin, et al. "Agent-SafetyBench: Evaluating the Safety of LLM Agents." CoRR (2024). [3]Mireshghallah, Niloofar, et al. "CIMemories: A Compositional Benchmark for Contextual Integrity of Persistent Memory in LLMs." arXiv preprint arXiv:2511.14937 (2025). [4]Hubinger, Evan, et al. "Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training." CoRR (2024).
Theory of change
This project advances AI safety by identifying persistent memory as a critical, underexplored attack surface in stateful and increasingly autonomous LLM systems. By demonstrating how latent memory-based attacks can cause delayed, cross-domain behavioral manipulation without modifying model weights or immediate prompts, this work exposes a class of long-horizon integrity and alignment failures that existing safety techniques do not address. The resulting threat models, empirical evidence, and mitigation strategies will directly inform the secure design of memory-augmented agents deployed in high-stakes domains.
Your role
Mentees will act as the equivalent of junior researchers in a ML lab, where they will read, develop ideas, code and hopefully write a paper. Much of the time would be spent in developing and conducting experiments.
Prerequisites
- Coding in python.
- Familiarity with hugging face
- Ability to use LLMs via API calls
- Version control like github
Location preference
NA
Application question(s)
-
Can you speculate how persistent memory is maintained for commercial AI platforms?
-
How would you simulate with an open source LLM and close source (eg openai) LLM the threat model described.
About the mentors
Ivaxi is a PhD student at CISPA focusing on causal reasoning and language model safety. She has previously worked on concept based explainabiity before transitioning research to causal reasoning in LLM and now on safety. She is particularly interested in safety risks of inherently dynamic, self propagating and open-ended systems with memory.
Vyas completed his PhD at the Machine Intelligence Lab at the University of Cambridge on the adversarial robustness of Large Language Models and Audio Language Models.
Vyas is keen on the development of safe and reliable AI systems. With an increasing number of users and applications of AI, Vyas is concerned that there is a significant drive for better (and more impressive) performance in industry, with little concern for the safety implications. Therefore, Vyas is keen on pushing the frontiers of AI safety research in the hope it can catchup with the deployment of these AI systems.