The latest AI security problem is not a model that says the wrong thing. It is a system that sits inside trusted workflows, sees too much, and can be pushed into actions its designers did not intend. Recent incidents around frontier model release decisions and enterprise assistant governance show that the boundary between useful automation and operational risk is already dissolving. two of the recent examples -
Anthropic’s withholding frontier model. Anthropic chose not to broadly release Claude Mythos Preview because it could autonomously identify and exploit zero-day vulnerabilities, which is a very strong real-world signal that autonomous cyber capability is already a release-gating issue.
Microsoft 365 Copilot Chat’s DLP bypass bug. A bug let the assistant summarize confidential emails for weeks despite enterprise sensitivity controls, which is a clean example of how AI systems can fail inside trusted workplace environments.
The most consequential security question in AI is beginning to shift. It is no longer only whether a model can produce unsafe text, write exploit code, or be persuaded to ignore its instructions. It is whether an AI system, connected to real organisational tools and data, can be manipulated into taking actions that a human never intended.
That distinction matters.
A chatbot that answers a question badly may create confusion. An agent that can search internal documents, access a CRM, send emails, execute code, update a ticketing system, or call a procurement API has a materially different risk profile. It does not need to “break in” to an organisation. It may already have the keys, the identity, and the access permissions required to operate.
The security challenge is therefore not simply the model. It is the entire system around it: its tools, memory, access tokens, retrieval sources, approval flows, audit logs, and the assumptions embedded in its deployment.
From generation to action
Generative AI was primarily introduced as an interface for producing text, code, images, and summaries. The user remained responsible for deciding what to do next.
Agentic AI changes that relationship.
The system can be asked to pursue an objective, break it into tasks, inspect information, choose tools, and take a sequence of actions. In practice, this might mean researching suppliers, drafting and sending outreach, triaging support tickets, updating records, monitoring infrastructure, producing reports, or coordinating workflows across several applications.
This is why the language around “AI assistants” can be misleading. Many systems are becoming operational participants rather than passive productivity tools.
The attraction is obvious.
Organisations are under pressure to reduce administrative workload, improve service response times, and make better use of fragmented information. An agent that can navigate documents, systems, and workflows promises a way to turn AI from an interface into a productive layer of the organisation.
But the same capabilities change the attack surface.
A system that can read an inbox can encounter a malicious email. A system that can browse the web can ingest hostile instructions embedded in a page. A system that retrieves knowledge from shared drives can encounter adversarial content inside a document. If it can also send messages, modify data, invoke APIs, or trigger physical processes, a piece of untrusted text can become an operational security event.
The trusted-insider problem
Traditional cybersecurity assumes that an attacker must obtain access, escalate privileges, or exploit a vulnerability. Agentic systems create another route: influence a trusted actor that already has legitimate access.
This is the practical significance of indirect prompt injection. Rather than asking the model directly to do something harmful, an attacker places instructions in content the agent is likely to read: a webpage, document, calendar invitation, support ticket, spreadsheet, email, or tool output. NIST has described this as “agent hijacking,” where malicious instructions within ingested data can cause an agent to take unintended actions.nist
The risk becomes serious when three conditions exist at once:
The agent can access sensitive information.
The agent can consume untrusted external or user-provided content.
The agent can take consequential actions beyond producing a response.
That might include sending external communications, changing a record, transferring files, creating accounts, authorizing a transaction, or interacting with industrial and operational systems.
The key point is not that every agent will be compromised. It is that organisations should not treat an agent as a reliable security boundary simply because it has been instructed to behave well. Natural-language instructions are not equivalent to deterministic access controls.
Prompt injection remains the top-listed risk in OWASP’s 2025 guidance for LLM applications, and its recommended mitigations point toward a familiar security principle: minimize privileges, isolate untrusted content, validate outputs, and require human approval for high-risk actions.
The permission layer matters most
The useful question for organisations is not, “Which model is safest?”
It is: what is this system authorised to do, with which information, under what conditions, and how quickly can that authority be withdrawn?
A capable model with no access to sensitive systems is mainly an information-quality and reputational risk. A less capable model connected to sensitive data, automation tools, and poorly governed credentials may be a much larger operational risk.
This reframes AI security as a permissions architecture problem.
An agent should not inherit a human employee’s broad access merely because it is working on that employee’s behalf. It should receive the narrowest possible permissions, for the shortest necessary time, through credentials that are distinct, observable, and revocable.
That means treating agents as their own class of digital identity. Each should have defined boundaries: which systems it can access, which actions it can perform, what data it may retain, and when it must pause for approval.
The important distinction is between assistance and authority. It is usually sensible to let an agent draft an email. It is more consequential to let it send one. It may be reasonable for it to recommend a supplier. It is another matter for it to amend a supplier record or initiate a purchase workflow.
As authority rises, controls should become stronger and more explicit.
Runtime governance, not paperwork
Much current AI governance is centered on model selection, acceptable-use policies, risk assessments, and pre-deployment review. These remain necessary. But they are not sufficient for systems whose behaviour is shaped continually by live context, changing tools, external content, and user requests.
The critical controls are runtime controls.
A practical deployment approach includes:
Least-privilege access: Give agents only the data and tools required for a specific task.
Sandboxed execution: Separate agent environments from sensitive networks and restrict what they can communicate with; the UK’s NCSC has advised that agents should run in sandboxed environments controlling local and network access.
Human approval gates: Require a human decision before irreversible, financial, external, or safety-relevant actions.
Clear trust boundaries: Mark retrieved, uploaded, and web-derived material as untrusted rather than allowing it to blend invisibly into an agent’s instructions.
Tool-level enforcement: Check permissions in deterministic software at the API or workflow layer, not through natural-language policy alone.
Action logging: Record what the agent saw, which tools it used, what it attempted, and which identity authorised each action.
Rapid revocation: Make it possible to suspend an agent, rotate its credentials, or remove tool access quickly when behaviour is anomalous.
This is not bureaucracy around AI. It is what turns a promising prototype into a system that can survive contact with real data, real users, and real adversaries.
Physical systems raise the stakes
The argument becomes more important when agents move beyond knowledge work.
Industrial, logistics, transport, energy, aerospace, and public-service environments increasingly combine AI with sensor data, operational dashboards, maintenance records, digital twins, and control-adjacent workflows. In these settings, the value of an agent may lie in detecting an anomaly, recommending an intervention, prioritizing a work order, or coordinating a response.
That can deliver genuine operational benefit. But it also means the governance question is no longer only whether an AI system generated inaccurate text. It is whether a compromised or poorly designed workflow can affect maintenance decisions, availability, safety, cost, or resilience.
The appropriate principle is not to prohibit autonomy. It is to match autonomy to consequence.
An agent can be highly autonomous in collecting evidence, identifying patterns, drafting recommendations, and preparing options. Its authority should narrow where actions are expensive, irreversible, safety-critical, externally communicative, or difficult to audit.
The organisational test
The near-term winners in enterprise AI will not necessarily be those deploying the most autonomous agents first. They will be the organisations that can explain how those agents are governed after deployment.
A useful test is simple:
Can the organisation show what an agent was allowed to do, what information shaped its actions, what it actually did, who remained accountable, and how it could be stopped?
If the answer is unclear, the system is not ready for consequential work, however impressive its demonstration may be.