Your AI agent's guardrails live in a file it can edit.

A test team took control of a live assistant through nothing but its chat box. The rules meant to stop them were sitting in a file the agent was allowed to rewrite.

What happened

A security firm published its notes from an adversarial test of a consumer-facing AI assistant, written up on Mondaq. The testers were given no special access. They had the same chat window every customer had.

From there they reached root-level control of the backend, switched off the safety rules, and turned the assistant into a tool for attacking other users of the same service. The safety rules were stored as configuration files the agent had permission to write to, so turning them off was not a break-in. It was an edit.

The part worth repeating is in the title of their report. The agent helped. Told it could not do something, it went looking for another route, because finding another route is what an agent is built to do. Persistence is the product feature. It is also the attack.

Why this matters to your business

Most companies are not running an assistant with shell access, so it is tempting to file this under someone else's problem. The shape generalises badly enough that you should not.

An agent can reach whatever its credentials can reach. Not what you told it to reach, not what the demo showed, what the credentials permit. A support agent wired up with one all-purpose API token can issue refunds because refunds were the use case, and can also change plan tiers, read other customers' records, and export a list, because nobody scoped the token down. The instruction not to do those things is a sentence in a prompt. A sentence is not a permission boundary.

So the question for a business owner is not whether the model is well behaved. It is how much damage is reachable if it behaves badly once, and whether you would know. That number is set by plumbing decisions made in an afternoon months ago, usually by whoever was fastest to get the pilot working.

Why this is a CX-Builder use case

The fix in the report is not a better model or firmer wording. It is moving the limits out of the agent and into the system around it: least privilege, allowlisted actions, controlled network access, monitoring. Nothing there is exotic. It is ordinary operational discipline applied to a new kind of software.

That is the shape of a flow, and it is what CX-Builder is for. Every tool the agent can call is a node you placed on purpose, so the list of nodes is the list of powers, visible on one screen. Anything that writes, refunds, or changes an account can sit behind a human-in-the-loop step. Because it is self-hosted, the credentials, the logs, and the boundary live on infrastructure you own rather than inside a vendor's account you cannot inspect.

None of that depends on the model choosing to comply. That is the whole point. A limit the agent can talk its way past was never a limit.

What this looks like if you build it

In practice it is one agentflow with a short tool list and scoped credentials behind each connector, so a read-only lookup uses a read-only account and nothing else. Retrieval gives the agent the documents it needs without giving it the store to write into. An approval node holds anything that changes money or account state, and the run log is written somewhere outside the flow, so the record of what happened does not sit in the same place the agent can edit.

The useful test at design time is to look at the canvas and ask what the worst single action on it could do. If you cannot answer, the agent has more reach than you have model of.