TRANSCRIPT 🔗
Enterprise AI. Securing Agents.
Hosts: Rob. Cloudadorn Academy.
This is Enterprise AI, from Cloudadorn Academy. Episode nine: Securing Agents.
How do you secure an agent that can be steered by the documents it reads? The problem is prompt injection, and the twenty twenty-six guidance from the OWASP GenAI Security Project says that no reliable way to prevent it exists in today's models.
We've met that confusion in the memory and tools episodes: source material treated as instructions. On its own, that can sound like a curiosity. It turns serious when the agent doing the reading holds two other things at the same time: access to private data, and some way to send information out. An agent like that can be turned against the person it works for.
Simon Willison gave that combination a name on the sixteenth of June, twenty twenty-five. He called it the lethal trifecta for AI agents: private data, untrusted content, and external communication.
We'll start with why a model can't simply ignore the bad text, because every defense later on depends on that. Then a way to count the risk in any agent you run, which comes down to a rule of two. Then three real incidents, told by what was harmed and what was fixed. And at the end, the question every team has to settle for itself: when an agent holds all three, is a person clicking approve enough, or does the agent have to be built differently?
When you build an agent, you give the model instructions. You also give it material to work on: a support ticket, an inbox, a page of search results. To you, those are different kinds of thing. One is the brief from your manager, and the other is the paperwork on the desk. To the model, they arrive the same way.
Willison put it plainly in that June post. Models, he wrote, are unable to reliably distinguish the importance of instructions based on where they came from. Everything eventually gets glued together into a sequence of tokens and fed to the model. The developer's instructions, the user's request, and a paragraph pulled from a stranger's web page all end up in one long line. There's no separate lane marked trusted.
OWASP, which is how I'll refer to the GenAI Security Project from here, makes the same point in its twenty twenty-six entry on prompt injection, in more formal language. Models make no architectural distinction between instructions and data, it says, and their behavior is stochastic. Stochastic means probabilistic: the same input won't always give the same output. From those two points OWASP concludes that no reliable prevention mechanism exists today, and that defense is therefore architectural rather than interceptive.
That last phrase is worth slowing down for. Interceptive defense tries to catch the bad text on its way in. Architectural defense arranges the system so that even when bad text gets through, the agent doesn't hold everything a high-impact attack needs. Most of what follows is about the second kind, because on OWASP's reading the first kind isn't reliable today.
One distinction before going further, because it trips people up. Prompt injection is not the same thing as jailbreaking. Jailbreaking is a user pushing a model into saying something it's not supposed to say, something embarrassing or against its rules. Prompt injection is text from somewhere else steering an agent that is acting for someone. The person who gets hurt is usually not the person who wrote the text. Willison has warned that developers who assume prompt injection is the same as jailbreaking will frequently ignore the issue as irrelevant to them. If your picture of the threat is a user typing something rude into a chat box, you'll miss the cases where the user did nothing wrong at all.
So, the three properties. Willison's list, in his words, is private data, untrusted content, and external communication.
Private data is anything the agent can read that an outsider shouldn't see. Your mail, your files, a customer table. Untrusted content is any text that someone outside your control could have written, and in practice that covers most email and most of the web. External communication is any route by which the agent can get information out, whether that's sending a message, calling an outside service, or something less obvious than either.
Consider an invented assistant with three capabilities: reading your inbox, searching your company's shared drive, and sending email for you. That's all three. Every incoming message is untrusted content. The drive is private data. The send function is the way out. Nothing about that assistant is unusual.
When all three sit in one session, untrusted text can steer the model toward using its private access and its outbound channel together. That follows directly from the single line of tokens. The model can't reliably tell that an instruction came from a stranger's email, so it may simply act on it.
OWASP's prompt injection entry restates Willison's check in the same three terms, and adds the part that gives you something to do. An agent that can simultaneously access private data, ingest untrusted content, and communicate externally has the conditions for high-impact exploitation, and removing any one leg removes them.
That's the hopeful part. You don't have to solve prompt injection to make it far less dangerous. You have to break the triangle.
Willison has kept track of production incidents with this shape, and he notes that almost all of them were promptly fixed by the vendors. Usually the fix was locking down the route the data left by, so that malicious instructions no longer had any way to extract what they'd taken. That's the vendor's version of breaking the triangle. They close the exit.
His conclusion for people mixing their own tools is narrower and sharper. The only way to stay safe there, he wrote, is to avoid that lethal trifecta combination entirely. That's his opinion, and it's a strong one.
Willison's three properties are a diagnosis. The Agents Rule of Two
Load full transcript
comes from Meta. OWASP later budgets that rule, and the wording shifts in a way that matters.
On the thirty-first of October, twenty twenty-five, Meta stated the Agents Rule of Two. The rule names three properties: processing untrustworthy inputs, accessing sensitive systems or private data, and changing state or communicating externally.
Meta calls prompt injection a fundamental, unsolved weakness in all large language models. Meta says that until robustness research allows reliable detection and refusal of prompt injection, an agent must satisfy no more than two of those three within a session, to avoid the highest impact consequences of prompt injection. Meta also says that, to enable additional use cases, it can be safe to move from one configuration of the rule to another inside that same session.
If an agent requires all three without a new session, meaning a fresh context window, Meta says it should not be permitted to operate autonomously. At a minimum, Meta says, it needs supervision: approval from a human in the loop, or another reliable means of validation.
Meta states limits on that rule. Satisfying it is not sufficient protection against other threats common to agents, or against lower-consequence outcomes of prompt injection, such as misinformation. Designs that satisfy it can still be prone to failure, including when a user blindly confirms a warning. Defense in depth is critical, because the failure of a single layer may be likely, and the rule is a supplement, and not a substitute, for common security principles such as least privilege. Some highly sought-after use cases will be difficult to fit cleanly into the rule, including a background process where a human in the loop is disruptive or ineffective. Meta says it will continue research towards the rule's supervisory approval checks, through alignment controls such as oversight agents and the open source LlamaFirewall platform.
What OWASP requires is separate. In the twenty twenty-six prompt injection entry, OWASP uses the rule of two as a floor and credits the rule to Meta AI, twenty twenty-five. When all three are present, OWASP requires per-action human approval. When untrusted input is paired with sensitive data, or with state change or external communication, OWASP requires an explicit assessment of the risk left over.
Noma's criticism is separate from both of those. OWASP says the rule is silent on autonomy depth, and for that point it cites Noma Security, twenty twenty-five. Noma Security, twenty twenty-six, argues that the rule is silent on what may be the single most important factor in agentic risk: how much freedom an agent has to act without human oversight. In that same piece, Noma argues that in indirect prompt injection, the attack class the rule was designed to address, the untrusted input is often embedded within the sensitive data itself.
Here's the short version. Staying at no more than two properties in a session is Meta's line. When you are at two and one of them is untrusted input, write down the risk that remains. That is OWASP's line. Three properties, and no fresh session, means the agent does not operate on its own.
I want to be careful here, because rules like this get stretched. As OWASP says, the rule is a floor. Passing it is not a safety certificate. It's about one failure in particular: outside text hijacking an agent that can do damage. An agent with only two properties can still make mistakes, still produce confident nonsense, still carry permissions it never needed.
That last problem, permissions wider than the job, has its own entry on the OWASP list, which is where we go next.
OWASP's twenty twenty-six list is a Top Ten for applications built on large language models.
Number one on the twenty twenty-six list is prompt injection. Number two is sensitive information disclosure. Number three is excessive agency. The remaining seven run from supply chain down to improper output handling, and they're worth reading in full on the project's site.
Excessive agency is the one I'd spend time on. Michael Bargury, quoted in the announcement, said that the risk where expert judgment and the record agree most clearly is excessive agency, now number three. OWASP's entry traces it to three roots: excessive functionality, excessive permissions, or excessive autonomy. In plain terms, that's a tool that can do more than the job, access wider than the task, or freedom to act with nobody checking.
The guidance on that entry is concrete, and three parts of it matter here. First, implement authorization in logic, rather than relying on the model to decide whether an action is allowed. Second, use a human in the loop to approve high-impact actions before they're taken. Third, run what OWASP calls a graduated enforcement policy, in four steps: audit, warn, block, escalate. Low-consequence or easily reversible actions can auto-approve. High-consequence or irreversible ones go to a person for review.
Then there's the agentic material. OWASP keeps a separate document, the Top Ten for Agentic Applications twenty twenty-six, and its source file is marked Version twenty twenty-six, December twenty twenty-five. The excessive agency entry points into it. In agentic systems, it says, excessive agency can show up as tool misuse and exploitation, identity and privilege abuse, and cascading failures. A cascading failure is what it sounds like: one agent's bad action becomes another system's input, and the error travels.
One more entry from that agentic list, on how people come to trust agents, matters a great deal for the argument at the end. We'll get there.
First, the layer you'll probably be offered before anything else: runtime guardrails. A guardrail here means a filter that runs beside the model, checking what goes in and what comes out.
Amazon's documentation for Bedrock Guardrails is a fair example of how these products frame the problem. It describes prompt attacks as user prompts intended to bypass the
safety and moderation capabilities of a foundation model to generate harmful content.
Read that definition closely. It's written about user prompts. That's a real risk, and you probably want something covering it. But in the Copilot case coming up, as OWASP describes it, the trigger was a crafted email, rather than text the user typed. A filter built around the first isn't automatically an answer to the second. That's not a claim Amazon makes.
Willison is blunt about the whole category. Guardrail products, he wrote, almost always carry confident claims that they capture ninety-five percent of attacks, or something similar. Here's my opinion on that number. In most of security, a control that misses one attempt in twenty isn't the thing you stake the system on. It's a layer, and it can be a useful one. You still need the architecture underneath.
Policy as code is a different idea, and for this problem I think it's a more promising one. Instead of asking a model whether an action looks safe, you write the rules down as code, outside the model, and every action the agent proposes has to pass through them. Open Policy Agent, usually shortened to O P A, is one tool for this. Its documentation says it provides a high-level declarative language that lets you specify policy as code, along with simple programming interfaces so your software can hand the decision off to it.
That lines up with OWASP's instruction to put authorization in logic rather than leave it to the model. If the model proposes sending a file to an outside address, a policy engine that never read the email behind the request can check the rule and refuse. That's what architectural defense looks like at small scale.
Now the three cases. For each one, what was at risk, and what happened afterwards.
The first is Slack. On the twentieth of August, twenty twenty-four, the researchers at PromptArmor published a report on Slack AI. They wrote that the vulnerability they found could allow attackers to steal anything a user puts in a private Slack channel, by manipulating the language model used for generating content.
Think about what that harm means. Private channels are where people put the things they don't want the whole company to see.
Slack's response is part of the record, and it matters to get it right. According to PromptArmor's timeline, on the nineteenth of August Slack replied that it had reviewed the report and deemed the evidence insufficient. PromptArmor also quotes Slack's position: messages posted to public channels can be searched for and viewed by all members of the workspace, regardless of whether they've joined the channel, and this is intended behavior.
So in that exchange, Slack didn't accept the evidence. Its stated position was about how public channels are designed to work.
That wasn't where it ended. Slack later said, in a security update in twenty twenty-four, that it deployed a patch to address the reported issue on the twentieth of August, the same day PromptArmor
published. It added that it had no evidence at that time of unauthorized access to customer data. So the record has three parts: PromptArmor's finding, Slack's first reply, and the patch that followed.
I still find the case useful. Put it against the trifecta. A workspace assistant holds private data, which here means the private channels. As Slack put it, messages posted to public channels can be searched for and viewed by all members of the workspace, regardless of whether they've joined the channel, and this is intended behavior. PromptArmor says the way out was Slack AI: an attacker with access to Slack could steal data from private channels they were not part of.
The second case is Microsoft three sixty-five Copilot.
On the eleventh of June, twenty twenty-five, Microsoft published C V E twenty twenty-five, three two seven one one. A C V E is the public catalogue number a vulnerability gets, so that everyone can refer to the same flaw. Microsoft classed it as information disclosure. The description says AI command injection in Microsoft three sixty-five Copilot allows an unauthorized attacker to disclose information over a network.
Microsoft scored it critical. On the common vulnerability scoring scale, version three point one, the base score was nine point three. The scoring record says no privileges were required, no user interaction was required, and the impact on confidentiality was high. No user interaction. The person whose data was at risk didn't have to click anything.
OWASP's twenty twenty-six prompt injection entry describes the case, citing Aim Security, as a crafted email that triggers an AI-powered productivity assistant to exfiltrate organizational data with no user interaction.
Now the fix. The same scoring record marks an official fix, and marks the maturity of any exploit as unproven. CISA added its own enrichment to the record, and it lists exploitation as none. So the fair summary is this: a critical information disclosure flaw that needed no click, fixed officially by Microsoft, with no exploitation recorded in those entries.
Against the trifecta, it lines up almost exactly. Email is untrusted content. Organizational data is private data. And the harm was data leaving.
The third case is a different kind, and it's here as a contrast. In December twenty twenty-three, Business Insider reported on an AI chatbot that a Chevrolet dealership had added to its website. One user, the reporter wrote, got the bot to agree to sell a car for one dollar. The reporter added that this was not legally binding.
A post that Business Insider quoted said the chatbot of Chevrolet of Watsonville was taken offline that weekend. Business Insider also reported that Aharon Horwitz, chief executive of Fullpath, and his team quickly shut down the bot for that dealer. Horwitz told the publication that the chatbot never disclosed any confidential dealership data.
Put it against the trifecta. The bot sat on a public website, so anyone could write to it. It's closer to what I called jailbreaking earlier: the person typing was the person the bot was talking to. Business Insider reported that Fullpath chief executive Aharon Horwitz and his team quickly shut down the chatbot for that dealer.
Now to the open question. When an agent does hold all three properties, is a human approval gate enough? Or do you have to change the structure so it never holds all three in the first place?
The strongest argument for gates comes straight from the guidance. OWASP says any agent with untrusted input, sensitive data, and the ability to change state or communicate externally needs per-action human approval. It says to require explicit human confirmation before any privileged, irreversible, or externally visible action, and to show the reviewer the exact rendered action rather than a summary. Its excessive agency entry says the same about high-impact actions: a human approves them before they're taken.
There's real logic there. A gate lets you keep a useful capability. You don't have to take the send function away; you have a person look at each send. And it doesn't ask the model to act as a refusal engine, which, given that single line of tokens, it can't reliably be. When a person sees the actual message, the actual recipient and the actual attachment, the person supplies the separation the model lacks, because the person knows which words came from a stranger. On a good day, that works well.
The strongest argument against relying on the gate comes from the same documents. OWASP itself says approval fatigue degrades reviewer judgment at volume. Its agentic entry on human-agent trust exploitation says the risk is amplified when people over-rely on autonomous recommendations or unverifiable rationales, approving actions without independent validation. It names automation bias, perceived authority, and anthropomorphic cues, meaning human-like behavior, as things that make abuse look legitimate and hard to spot. Automation bias is the habit of trusting a suggestion because a machine made it.
Researchers say much the same. Beurer-Kellner and colleagues, twenty twenty-five, wrote a paper called Design Patterns for Securing L L M Agents against Prompt Injections, where L L M stands for large language model. They write that user confirmations pose a safety risk themselves, with tired or overloaded human verifiers approving unsafe actions. And the researchers at Invariant Labs, writing in May twenty twenty-five, observed that many users opt for an always-allow confirmation policy and stop monitoring individual actions.
Put those side by side and the weakness of the gate is plain. Its whole value rests on one person reading carefully, every time. Volume wears that down. So does a fluent, confident agent, and so does a long day. Meanwhile the person who planted the text only needs the reviewer to slip once.
The structural position starts from the other end. Willison's view, for anyone combining tools, is to avoid the
trifecta entirely. OWASP's prompt injection entry says removing any one leg removes the conditions for high-impact exploitation. A structural control doesn't depend on anyone's attention late on a Friday afternoon. A rule enforced in code, outside the model, applies the same way to the first action of the day and the five hundredth.
To be fair to the other side, structure has costs too. Splitting an agent so that the part reading strangers' text can't reach private data takes more engineering. Some tasks seem to genuinely need all three properties at once. And people notice when a capable assistant gets less capable, and they tend to find ways around it.
How you decide depends on your situation, but here is my view, and it is only my view. A gate is a real control when three things hold. The person sees the exact action, not a summary. The action is something they can judge in a few seconds. And the volume is low enough that they're actually reading. A gate on a reversible step, in an agent that holds only two of the three properties, is a sensible design. A gate in front of an agent that already holds all three, staffed by someone who stopped reading weeks ago, isn't doing the job anyone thinks it's doing.
If you can't make those three things true, change the structure. Remove a leg. Separate the session that reads outside text from the session that holds the keys. Put the rules in code. Then use gates for what's left, and keep their number small enough that a person can keep up.
OWASP's graduated policy already points that way. Reversible, low-consequence actions auto-approve, while irreversible, high-consequence ones go to a person. That saves human attention for the few decisions that deserve it, and OWASP says approval fatigue degrades reviewer judgment at volume.
Apply the rule of two to every agent you run, as Meta stated it on the thirty-first of October, twenty twenty-five. Inside one session, no more than two of Meta's three properties. If you need all three without a fresh session, don't leave the agent to operate on its own. At a minimum it needs supervision. OWASP, in twenty twenty-six, credits the rule to Meta AI and uses it as a floor. The per-action approval and the explicit risk assessment are requirements OWASP states.
Take each agent, and inside a single session, count Meta's three properties. Does it process untrustworthy inputs? Does it have access to sensitive systems or private data? Can it change state, or communicate externally?
If the count is two and one of them is untrusted input, write down the risk that's left, because OWASP asks for that assessment. If the count is three and you are not opening a fresh session, don't leave the agent to operate on its own. Split the context so that no single session holds all three. In my view the split alone isn't enough: if the session that read outside text can pass instructions to the privileged one, through a summary or a document, nothing has really been separated. So keep that handoff narrow, and enforce permissions in code, outside the model, where nothing the model read can change them. Or put a real check on the third property: the exact action, shown to a person, at a volume they can actually read.
Then look past the count, because passing it isn't the whole job. Meta says a design that meets the rule can still fail, including when a user blindly confirms a warning. Separately, OWASP notes that the rule is silent on autonomy depth, citing Noma Security, twenty twenty-five. Noma Security, twenty twenty-six, argues that the rule is silent on how much freedom the agent has to act without human oversight, and that in indirect prompt injection the untrusted input is often embedded within the sensitive data itself. Look for permissions wider than the task. Look for tools that can do more than the job needs. Those are two of OWASP's three roots of excessive agency. And look for the human one, a reviewer who has quietly stopped reading.
The idea underneath all of it is the one from the start. A model can't reliably tell an instruction from a document, because to the model they're one line of text. So don't build an agent where that confusion can reach your private data and a way out at the same time.
For that audit, keep three sources beside you: Simon Willison's June twenty twenty-five post on the lethal trifecta, Meta's Agents Rule of Two from the thirty-first of October, twenty twenty-five, and the twenty twenty-six OWASP Top Ten for L L M Applications, at genai dot owasp dot org.
That's episode nine of Enterprise AI. The sources for everything in this episode are in the show notes. Rob is an AI voice. Thanks for listening.