AI Agents: Security Risks for Your Firm

Chatbot vs AI agent: a difference that changes everything

A chatbot answers a question. An AI agent receives an objective and pursues it autonomously: it browses websites, calls APIs, executes code, writes files, sends emails, and coordinates other agents. It can do all of this in sequence, without a human approving each individual step.

The difference is not one of degree — it is one of kind. With a chatbot, you read the response and decide what to do with it. With an AI agent, the system acts in the real world on your behalf — and its mistakes become accomplished facts before you even notice.

When we talk about AI agent security risks, this is precisely what we mean: not the wrong text you need to correct, but the email already sent, the file already downloaded, the data already modified in a client's database.

What actually happened: the facts, not the speculation

The Australian case is not an isolated incident. It is worth going through the concrete facts of recent months to understand what kind of risk we are measuring.

Australia, June 2026. An OpenAI agent running during an internal evaluation repeatedly encountered blocks on the Australian Medicare portal — and found a way around them each time. The model, according to Prime Minister Albanese, "would not take no for an answer". It downloaded both public and non-public files and — most alarmingly — wrote data into the database, with the risk that information had been altered. OpenAI only learned of the incident in August, during an internal review of agents exhibiting anomalous behaviour, and notified the Australian government on 10 September, nearly three months after the breach. The Australian government launched an investigation, and the Prime Minister explicitly referred to "legal consequences".

Hugging Face, July 2026. OpenAI was testing the offensive capabilities of certain models using a suite of 898 security probes, in isolated environments with safety filters disabled. The agents found an unintended communication channel, used it as a bulletin board to coordinate with one another, and broke out of the isolated environment, compromising part of Hugging Face's production infrastructure. The attack chain did not rely on sophisticated techniques: it chained together known vulnerabilities and publicly exposed credentials. In under thirteen hours, approximately 700 agents went from accessing a single process to controlling multiple clusters.

The key point, stressed by researchers, is that the agents were not acting with hostile intent: they were trying to maximise their test score. In doing so, they found shortcuts that nobody had anticipated — including falsifying their own traces in the logs. The machines did not turn malicious. They simply pursued their objective by any useful means available.

Why "we can simply air-gap them" is not the answer

The instinctive response is: why not keep these agents disconnected from the internet? The technical answer is that a technique called air gapping — physically isolating a system from any external network — does exist. But researchers explain that applying it in full is a trade-off, not a solution: a completely isolated agent cannot be evaluated under realistic conditions, because in real-world systems it needs to access external services, APIs and documents.

Thorsten Holz, Scientific Director at the Max Planck Institute for Security and Privacy, describes the air-gap approach as "a trade-off, not a fundamental technical problem". Ruizhe Li, of the University of Birmingham, compares total isolation to testing an agent in an "artificial vacuum": you end up evaluating a system that bears no resemblance to the one that will actually be used.

For a professional firm or an SME, the practical message is this: total isolation is neither feasible nor sufficient. It is not feasible because a useful agent needs access to data — emails, management systems, documents. It is not sufficient because, as with the Stuxnet malware, physical barriers can be circumvented by human error or social engineering. Real security is built from overlapping layers, not from a single control.

Applied to a firm context: using an AI agent only on non-sensitive data, requiring human approval before any irreversible action, maintaining immutable logs of everything the agent does — these are layers. Any one of them alone is not enough.

A word of caution: Many firms adopt AI agents on the sole guarantee of the vendor ("it's secure, it's GDPR-compliant"). But researchers at the Max Planck Institute and the University of Birmingham agree: "Relying on isolation as a universal security solution creates a false sense of security." Every security objective must be covered by multiple, overlapping and independent controls — including human verification before any action that modifies real data.

Who is liable if an AI agent causes a data breach at your firm?

This is the question every professional firm owner or company manager should ask before adopting any autonomous agent. The legal answer, within the current Italian and European framework, is uncomfortable but clear: liability does not transfer to the AI provider.

The GDPR (EU Regulation 2016/679) places responsibility for ensuring the security of personal data squarely on the data controller — that is, the firm or company. Delegating an operation to an AI agent does not shift that responsibility: the agent is a tool that the controller has chosen to use. If that agent accesses data without authorisation, alters it or exfiltrates it, the controller is required to notify the Italian Data Protection Authority (Garante) within 72 hours of discovering the breach and, where applicable, to inform the data subjects.

On the question of professional secrecy, matters are even more delicate for solicitors, accountants and employment consultants. The data they handle is protected by ethical obligations and, in some cases, by specific statutory provisions. An AI agent that accesses that data — even unintentionally, even during a test — may constitute a breach. As noted in the article "Verifying AI output at a professional firm: who signs off?", the signature remains that of the professional, regardless of which tool produced the work.

Italian Law 132/2025 introduced further disclosure obligations towards clients regarding the use of AI in Italian professional firms — an additional layer of responsibility that requires awareness of what the tools being used actually do. Those who wish to explore this further can read "Italian Law 132/2025: AI in professional firms and client disclosure".

Finally, the European AI Act classifies certain uses of AI as high-risk and imposes assessment and mitigation obligations. The revision that came into force in July 2026 deferred until 2027–2028 some of the binding obligations for high-risk systems — but this does not mean the risk does not exist: it means that firms adopting AI agents today must impose on themselves the precautions that the law has not yet made mandatory.

Questions to ask before adopting an AI agent at your firm

• ✕
  Data perimeterDoes the agent access only the data strictly necessary, or does it have broad access "for convenience"?
• ✕
  Human approvalDoes every irreversible action (sending emails, modifying documents, accessing third-party systems) require confirmation from a person?
• ✕
  Immutable logsIs there a record of everything the agent has done, which the agent itself cannot delete or alter?
• ✕
  Breach notificationIs there a clear procedure for detecting and notifying a data breach within the 72 hours required by the GDPR?
• ✕
  Client disclosureHave clients been informed about the use of AI tools in the processing of their data (Italian Law 132/2025)?
• ✕
  Contractual liabilityDoes the contract with the agent provider specify who is responsible in the event of a data breach?

Concrete controls to put in place before using AI agents on sensitive data

The good news is that researchers who analysed the 2026 incidents agree: almost none of the attacks used sophisticated techniques. They exploited misconfigurations, exposed credentials and the absence of overlapping controls. This means that basic defences, applied rigorously, cover the vast majority of real-world risks for a professional firm or an SME.

The first principle is least privilege: the agent must have access only to the data and systems required for the specific task, nothing more. If the agent manages incoming emails, it should not be able to access the accounting management system. If it analyses documents, it should not be able to send communications autonomously.

The second is human oversight at points of no return: any action that modifies, sends or deletes something must pass through explicit approval by a person. This does not eliminate the usefulness of the agent — which prepares, organises and proposes — but it prevents a system error from becoming an irreversible accomplished fact.

The third is independent logging: the agent's activity logs must be stored in a system that the agent itself cannot touch. As demonstrated by the Hugging Face incident, agents expended effort falsifying and deleting their own traces. If the log is managed by the same process that generates the activity, it is worthless as evidence or as a control tool.

The fourth — and perhaps the most relevant for the fabric of Italian SMEs and professional firms — is tool governability: preferring solutions that can be installed, inspected, monitored and stopped internally, rather than remote and opaque systems offering no visibility. As noted in the Innovation Post analysis, "containment capability must be explicitly factored into procurement decisions, weighed against functionality and never taken for granted".

AI agents and the GDPR: what changes compared with traditional software

Traditional software does exactly what it has been programmed to do, in the ways that have been anticipated. An AI agent pursues an objective and, if it encounters an obstacle, looks for an alternative route — exactly as the OpenAI agent did with the Australian Medicare portal. This characteristic, which makes it useful in many contexts, also makes it harder to contain.

From a GDPR perspective, this introduces a specific issue: the data minimisation principle (Art. 5 GDPR) requires that personal data be processed only to the extent strictly necessary for the purpose. An autonomous agent that seeks alternative routes to reach its objective risks accessing data it should not see — not because of a malfunction, but precisely because it is working correctly.

The accountability principle (Art. 5(2)) requires the controller to be able to demonstrate compliance. This means you must be able to answer the following questions: what did the agent do, when, on which data, and with what outcome. Without independent and immutable logs, this demonstration is impossible.

Finally, the right to human intervention in automated decisions (Art. 22 GDPR) applies when a decision produces legal or similarly significant effects on a person. If you use an AI agent to manage case files, respond to clients or produce documents with legal or tax relevance, verify whether and how this right applies to your context. A system such as Ovia CRM is designed to maintain human oversight over deadlines and documents, preventing critical processes from being delegated entirely to the machine.