Policy-Gated Agents: audit and harden tool-using agents

No, it isn't safe to put in front of real customers yet. Every rule in the bot's instructions ("max $100 without a manager", "only the customer's own orders") is only a request to the model; the code never checks any of them. The full report is in audit.md.

How I checked: I read all the code and ran your test, which passes. Then I ran 17 scenarios against the unchanged bot, using a stand-in model that does whatever the email asks. That stands in for a real model that gets persuaded or is simply wrong, and shows what your code allows, not how likely Claude is to comply. Twelve scenarios ended in a refund, account change or email that shouldn't have happened. In total, $8,410 went out in refunds that break your rules or exceed what the order cost.

The worst problems:

Your own suspicious sample email works. If the model follows it, the bot refunds $320 + $48 with no manager, because the email calls the customer "gold-tier". It also changes Maria's account email to the outside address and sends the confirmation there. The manager check only looks at the amount and tier the model types in (agent.py:21-22).

No checks on refunds. It will refund another customer's order, more than the order cost, or the same order over and over. One test had ten $100 refunds of a $320 order in one go.

The bot refunds on its own. If the model errors or runs out of steps, it refunds the hinted order in full and says "We've refunded your order." (agent.py:31-36, 53-57). During an outage, every email with an order hint gets a refund.

It can change the email on any customer's account (update_customer_email), and send email to any address, so it could relay phishing under your brand.

Retries pay again. One provider timeout paid a $48 refund twice. A locked database turned one $48 refund into 24 payments ($1,152), with nothing in your records.

• Your test checks the reply's wording. It would still pass if the bot refunded nothing or crashed.

Fastest fixes, in order:

1. Remove the account-email tool.
2. Send email only to the address on file.
3. Make both fallbacks hold the case for a person.
4. Check every refund in code before paying: the order belongs to the sender, the amount isn't more than what's left to refund, the tier comes from your database, and there are daily caps.
5. Make refunds exactly-once: one key per email, and no retry around the payment.
6. Add tests that check what was actually paid and sent.

The first three take about an hour combined.

Questions for you (also listed in the report):

• Should gold customers really skip approval with no cap? The code says yes; your written rule doesn't.
• Should the bot refund when the model is down?
• How do you identify the customer? If it's just the email's From line, anyone can write as anyone.
• Where does order_hint come from?

I haven't changed your code. My probe scripts are in scratch/ if you want to rerun them. I can make these fixes and turn the scenarios into tests, and I can add the pattern screener I used to your repo. It flagged the suspicious sample and passed the normal one.