Benchmarking Guardrails for AI Agent Safety
Sign in<br>Subscribe
Introduction<br>AI Agents extend large language models beyond text generation. They can call functions, access internal and external resources, perform deterministic operations, and even communicate with other agents. Yet, most existing guardrails weren’t built to protect these operations. OWASP and security researchers have been investigating the increased risk that agentic systems afford, which range from tool poisoning attacks to amplifying malfunctions in reasoning capabilities required for tool calling to indirect prompt injection attacks on the web or in vector databases, and many more.<br>This is very much related to the lethal trifecta, a way to think about how the risk surface evolves when giving LLMs access to tools.<br>Credit: https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/<br>That tool access can expose private data by linking to a vector database, potentially leaking company secrets or, worse, sensitive information on individuals. Allowing people to communicate with the system through natural language gives an avenue for new styles of attacks. And allowing information to come from external sources creates a great amount of untrusted content for your agentic system to handle. This trifecta showcases why it is so important to not only do whole system evaluations, but also to stress test each component on as many threats as possible.<br>Most guardrail models today are specifically trained to operate on inputs and outputs of agentic systems, and not their inner workings. This runs counter to the construction of agents. The LLMs governing agents may be black boxes, but that doesn’t mean the data flowing throughout an agentic system is a black box. To address this, we built any-guardrail to quickly test off-the-shelf, open-source guardrail models on some of the threats within agentic systems. Our findings: Some models show promise for prompt injection detection, but critical gaps remain in protecting function-calling operations.<br>In this post, we walk through how we evaluated certain guardrail models on out-of-distribution tasks, specifically indirect prompt injection attacks and detecting function-call malfunctions. We will discuss the guardrails selected, the benchmarks used, our setup and the results.<br>TL;DR<br>Our experiments found that PIGuard was an effective indirect prompt injection detection model against the email and tabular portions of the BIPIA dataset. Moreover, we found that detecting function call malfunctions is still a hard task for customizable judges to tackle, a gap in the field that we hope closes in the coming months. To run any of our experiments or analyze our data and results, please go to our GitHub Repo. And remember to try out any-guardrail!<br>Guardrails<br>We selected guardrails that could plausibly handle out-of-distribution tasks: detecting indirect prompt injection attacks and customizable judge models for function calling malfunctions. Although none of these models were purpose-built for these agent-specific risks, we selected specific guardrails supported by any-guardrail (as of the writing of this blog post) that are either purpose-built for adjacent tasks or are built to support a wide variety of judgement tasks.
Guardrail<br>Base Model<br>Task
Deepset<br>deberta-base-v3<br>Prompt Injection Detection
PIGuard<br>deberta-base-v3<br>Prompt Injection Detection
Harm Guard<br>deberta-large-v3<br>Prompt Injection Detection
Pangolin<br>ModernBERT base<br>Prompt Injection Detection
Sentinel<br>ModernBERT large<br>Prompt Injection Detection
Jasper<br>deberta-base-v3, gelectra-base<br>Prompt Injection Detection
ProtectAI<br>deberta-v3-small, deberta-base-v3<br>Prompt Injection Detection
FlowJudge<br>phi-3.5-mini-instruct<br>Customizable Judge
GLIDER<br>phi-3.5-mini-instruct<br>Customizable Judge
The prompt injection detection models were all encoder-only models. This is most probably due to the fact that encoder-only models cannot be jailbroken, whereas decoder-only models are susceptible to jailbreaks and prompt injection attacks. Of course, these models still have potential privacy issues and may be susceptible to traditional adversarial attacks.<br>The customizable judge models (FlowJudge and GLIDER) are decoder-only models that have been fine-tuned on diverse judgement tasks with a rubric-based evaluation, making them promising candidates for judging function call correctness.<br>Benchmarks<br>Since we are benchmarking guardrails, not general LLMs or agents, our data has to be formatted differently. Instead of input-output pairs or an environment to check how an LLM uses tools, we need datasets that have complete conversations or examples with labels.<br>For indirect prompt injection attacks, this is relatively easier, as most prompt injection datasets contain labeled data points. However, for function calling, evaluation is typically done in an environment with known tools, where success is measured by whether an LLM executed the right tool tasks. Our challenge was to find...