Somebody filed a fake bug report
In June 2026, security researchers at a firm called Tenet Security tried something almost embarrassingly simple. They filed a bug report. Not a real one. A fake one, with instructions hidden inside.
Here is the setup. When a website goes wrong in a visitor’s browser, it sends an error report to a monitoring service. For that to work, a small access key sits in the part of the website every visitor downloads anyway. It is meant to be public. Anyone can read it.
So the researchers did not break into anything. They sent an error report, the way any visitor’s browser would.
An AI assistant then read that report as part of its normal work, treated it as trustworthy technical information, and did what the hidden instructions told it to do. With all the permissions of the developer it was working for.
In controlled tests against more than a hundred targets, this worked about 85% of the time, against three of the most widely used AI coding assistants. One test handed over a working key to the company’s cloud systems.
Nothing raised an alarm. No virus scanner, no firewall, no access control.
The monitoring provider’s own verdict on the flaw was unusually blunt: "technically not defensible."
To be fair to the facts: this was a controlled experiment, not a confirmed break-in at any real company. But the technique worked. And the reason it worked cannot be patched.
No credentials were stolen, no policy was violated, no perimeter was breached: every step in the chain was authorized. That is the problem.
Why it worked: the machine cannot tell who is speaking
Picture a temp on their first morning. Willing, quick, and completely new. Nobody has explained which notes on the desk came from you and which arrived with the post. So they do whatever any note says.
That is roughly how today’s AI reads. Your instruction, the customer’s email, the uploaded PDF, the error report: it all arrives as one long piece of text. There is no separate channel marking one part as an order from the boss and another as merely something to look at.
Simon Willison, the developer who first named this problem back in 2022, puts it plainly: these systems "are unable to reliably distinguish the importance of instructions based on where they came from."
So text your AI only reads can turn into an order it follows.
His rule of thumb is the most useful thing you can carry into a meeting with any AI supplier. Real trouble needs three things at the same time:
- the system can see confidential data
- it also reads text written by someone outside your company
- and it has some way to send information back out
Each one on its own is harmless. All three together is the dangerous combination. His own example should give pause to anyone who has just been offered an email assistant: an attacker can simply write your AI an email and tell it what to do.
The obvious answer, buying a filter that catches malicious instructions, is weaker than it sounds. These products typically advertise stopping around 95% of attacks. That sounds reassuring until you notice that the attacker who fails simply tries again tomorrow.
This is now the top of the risk list
Forrester, a research firm that does not sell a solution to any of this, put AI assistants at the top of its 2026 risk list.
What changed is the interesting part. In 2023 and 2024, the worry was what criminals would do with AI. Now the bigger worry is what companies are doing to themselves, by putting AI to work without deciding what it may touch.
And why does nobody notice? Because these assistants do not behave like an intruder. They do not trigger a login prompt. They do not show up in the security logs the way a person does. And they do not go home at five.
The fix is something you already do with money
So what does a trustworthy setup look like?
The clearest answer I have read came from a company forced to face an awkward question. They sell software that controls who may access what, and they built an AI whose job is deciding what other AI systems are allowed to reach. Which raises the obvious follow-up: and who controls that one?
Their security chief, Kevin Paige, gave a concrete answer. He is describing his own product and none of it has been independently checked, so apply the usual discount. The thinking is still worth borrowing.
- The controlling AI is treated as just another member of staff. Its own permissions, and it has to follow the same rules it enforces.
- It has a short, fixed list of things it is able to do. If a task is not on the list, it cannot do it. Not "is told not to". Cannot.
- Its core instructions cannot be changed while it is running, so nobody can talk it out of its own limits.
- Everything it does is written down, and marked as done by the AI rather than by a person.
- And the important one: the part that reads incoming text is not the part allowed to act on it.
That last point should sound familiar, because you almost certainly do it already with money. The person who opens the post is not the person who signs the cheques. Whoever books the invoice is not whoever releases the payment. Nobody calls that distrust. It is simply how you stop a single mistake, or a single forged letter, from turning into a payment.
This is the same idea, applied to software. Let the AI read whatever it needs to read. Just make sure the part doing the reading is not also the part holding the pen.
Here is why I trust that principle rather than filing it under supplier marketing. Three groups with nothing to do with each other arrived at it within about six weeks. Researchers writing academic papers. The company above, while designing a product. And the team that cleaned up the fake bug report attack, whose recommended fix was not to change the exposed key, but to limit what the AI was allowed to do with whatever came back through it.
A researcher, a supplier and an emergency response team, arriving at the same place down three different roads. That counts for more than any one of them saying it.
The rule underneath is short enough to remember. Do not tell an AI to behave. Remove its ability to misbehave. Instructions are exactly what an attacker overwrites.
The part nobody likes to admit
There is an honest limit here, and it comes from Elia Zaitsev, chief technologist at the security firm CrowdStrike: "If you start with an agent in a sandbox that has no ability to touch anything, it is worthless. Very quickly, you are in this race of giving it more capabilities. And then what is the point of your sandbox?"
He is right. These systems are useful precisely because they can reach things. Every useful permission is also a door. So "lock it all down" is not a plan. Deciding which part of the system holds which key is.
The reassuring part: care makes you faster
Worth knowing, before anyone tells you that being careful costs speed.
BARC, a research institute based in Würzburg, surveyed 639 senior people responsible for AI, on behalf of the software vendor Domino. Companies whose oversight kept pace with their AI work were about 3.9 times more likely to have AI genuinely running in day-to-day operations, rather than stuck in pilot projects.
And the sectors doing best on discipline were not the slow ones. Banks and insurers, the most heavily regulated in the survey, came top on both counts at once: oversight and speed into real operation.
Two honest caveats: a software vendor paid for the study, and it shows a connection rather than proof that one causes the other. But it matches what the engineering evidence points at anyway. Companies that decide early what their AI may touch spend less time later finding out what it touched.
One number for readers here in Germany, Austria and Switzerland: European companies reported the widest gap of the three regions surveyed. Around 43% said their oversight was fully in place, against roughly 51% in North America and the UK.
Why this is not an abstract worry for us
Read those three ingredients again, then look at what we actually build.
A chatbot answering questions from a company’s own documents: confidential data, plus text typed by whoever is on the other end. An assistant that reads incoming mail and sorts it: that is Willison’s own example, almost word for word. A voice assistant that takes calls and writes into a CRM: a transcript is text written by a stranger too, which is easy to forget, because it feels like a conversation rather than a form.
All three have all three ingredients. That is not an argument against building them. It is the reason we separate the reading part from the acting part, keep the list of things the system can do short and fixed, and put a person in front of anything that sends, pays or deletes.
Those are not extras we add when a client asks. They are the default, because the alternative is a system whose safety depends on nobody ever writing it a clever email.
Five questions to ask on Monday
These are the questions we have to answer about our own work, so they are fair to ask of anyone, us included. You do not need to be technical:
- What can it reach? Not what it is for. The full list.
- Where does text from outside come in, and is that same part allowed to act?
- Are its limits things it cannot do, or just instructions telling it not to?
- If it did something wrong last Tuesday, could we work out what and why?
- Who is responsible for it? A name, not a department.
If those questions get vague answers, that is your answer.