Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

Real AI Agents with Fake Memories: Fatal Context Manipulation Attacks on Web3 Agents

Created by
  • Haebom

Author

Atharv Singh Patlan, Peiyao Sheng, S. Ashwin Hebbar, Prateek Mittal, Pramod Viswanath

Outline

This paper investigates the security vulnerabilities of AI agents integrated into Web3 under the context of adversarial threats in real-world scenarios. In particular, we introduce “context manipulation”, a comprehensive attack vector that exploits unprotected context surfaces such as input channels, memory modules, and external data feeds. We demonstrate memory injection, a more stealthy and persistent threat than traditional prompt injection, and empirically demonstrate that malicious injection can cause asset transfers and protocol violations using a decentralized AI agent framework called ElizaOS. By evaluating more than 150 blockchain tasks and 500 attack test cases using CrAIBench, a Web3-centric benchmark, we confirm that AI models are more vulnerable to memory injection, and show that prompt injection defenses and detectors as defensive strategies provide only limited protection, while fine-tuning-based defenses significantly reduce the attack success rate.

Takeaways, Limitations

Takeaways:
We systematically analyzed the security vulnerabilities of AI agents in a Web3 environment and proposed a new attack vector called context manipulation.
We experimentally demonstrated the severity of memory injection attacks and exposed the limitations of existing defense techniques.
We demonstrate that fine-tuning-based defense techniques are effective and suggest directions for future research.
It emphasized the need for developing safe and reliable AI agents in a blockchain environment.
Limitations:
Since this is an evaluation result for a specific framework called ElizaOS, further research is needed to determine whether it can be equally applied to other AI agent frameworks.
The scope of the CrAIBench benchmark may be limited, and further evaluation of more diverse and complex scenarios is needed.
Further analysis is needed on the long-term effectiveness and maintenance costs of fine-tuning-based defenses.
👍