Automation Bias: the Risk of Checking AI Less

What is automation bias and why does it matter for those working with AI

Automation bias is the tendency to over-rely on the indications of an automated system, progressively reducing one's critical attention as the system proves itself reliable. This is not laziness: it is a cognitive adaptation. The human brain stops allocating attentional resources to what it perceives as already resolved.

The paradox, well known in human-factors research on automated systems, is that this bias grows precisely when the system is working well. The more accurate the AI, the less vigilant the human reviewer. The problem does not surface often, but when it does — a contract with the wrong clause, an incorrect tax calculation, a missed deadline — the damage is already done, and the signature at the bottom is yours.

For an accountant, a lawyer or a labour-law consultant, a signature is not a formality. It is an assumption of responsibility. Delegating the production of a document to AI does not transfer that responsibility: it leaves it exactly where it was, with the added risk that the error went unnoticed.

Who is liable if the AI produces an error you did not verify?

The short answer is: you. The longer answer does not change the substance, but adds a few important nuances on the regulatory side.

Italian Law 132/2025, which governs the use of artificial intelligence in professional activities in Italy, requires professionals to inform clients when they use AI tools and to maintain human oversight of outputs. This means that adopting a reliable tool is not enough: you must document that oversight actually took place. An AI-generated output signed without traceable review is not merely an operational error — it is a potential breach of professional obligations.

On the European AI Act front, AI systems used in high-impact contexts — such as the production of legal or tax documentation — require transparency in processes and verifiability of decisions. Saying "the AI produced a correct output" is not sufficient: you must be able to demonstrate how it was verified and by whom. In the absence of this documentation, liability falls on the person who used the tool.

Add GDPR to the picture: if an AI output containing clients' personal data is signed without verification, any error in the processing of that data constitutes a breach for which the data controller is responsible — that is, the firm. See also our in-depth article on AI hallucinations and professional liability.

The emergency brake: what Nadella teaches Italian professionals

Satya Nadella, CEO of Microsoft, recently used a precise metaphor to describe how organisations should relate to advanced AI models: the emergency brake on a train. Someone must be able to pull it at any moment, even while the train is in motion.

His argument is that the most reliable AI system will not be the one we trust the most, but the one that allows us to trust the model less. This is a position that inverts the way many people think about AI adoption: the goal is not to find the perfect model, but to build around any model a control system that works independently of the model's quality.

For a professional firm, the "emergency brake" translates into very practical terms: who has the mandate to stop an AI process? How often are already-signed outputs checked on a sample basis? Is there a review log? Nadella is explicit: human oversight cannot be reduced to a button if there are no procedures, defined responsibilities and periodic checks. This is not a technology question. It is a question of AI process governance in your firm.

How to organise AI output control without slowing your firm down

The most common concern is that a structured review process will consume the time that AI was supposed to save. This is a legitimate concern, but it stems from a binary view: either you check everything or you check nothing. The reality is that control should be calibrated to the risk level of the output, not to the volume of outputs.

A reply to a generic email carries a very different risk profile from a contractual clause or a tax return. The former can be reviewed in thirty seconds. The latter requires careful, documented reading. The strategy is not "check more": it is "check thoroughly the things that matter, and record everything".

In practice, a firm can build a three-tier review matrix: low-risk outputs (quick check, no formal record required), medium-risk outputs (professional's review with date and initials), high-risk outputs (documented review, filed in the client's dossier). This structure does not slow down the work: it organises it. And in the event of a dispute, it proves that supervision genuinely took place.

A system such as Ovia Second Brain can support this structure by capturing review decisions, linking them to the right client and making them retrievable on demand — without the professional having to manage a manual archive.

Checklist: does your firm have an AI control strategy?

• ✓
  Output risk matrixHave you classified AI outputs by professional risk level and defined a review procedure for each level?
• ✓
  Named responsible for every workflowDoes every AI process have a nominated professional who signs off the review and can halt the workflow?
• ✕
  Review logDo you keep a record of who verified what and when, in a form that can be produced in the event of a dispute?
• ✕
  Sample audit frequencyAt least once a month, do you re-check a sample of already-approved outputs to verify that the system is not degrading?
• ✓
  Client disclosure on AI useDo clients know you use AI tools and that supervision is human, as required by Italian Law 132/2025?
• ✕
  Written procedures, not just practiceAre the control rules set out in an internal document accessible to all staff, not just held in the principal's head?

How often should I verify AI outputs in my firm?

There is no universal frequency: it depends on the volume of outputs, the risk level and the maturity of the system. What the regulations require — and what good professional practice suggests — is that the frequency be decided in advance, not managed on a case-by-case basis.

A reasonable starting point: full review for all high-risk outputs (deeds, contracts, returns); monthly sample review of already-approved medium-risk outputs; quarterly audit of the entire AI workflow to verify that the instructions given to the system are still appropriate.

The quarterly audit is often underestimated. An AI system does not degrade in the same way as a physical machine, but its instructions can become outdated if regulations, internal procedures or client types change. As the most recent discussions on AI highlight, the risk is not only that the AI makes a mistake: it is that it stops adapting to a context that has meanwhile changed, while we continue to rely on past results.

Does the AI Act require professionals to document AI controls?

The answer is yes, with nuances that depend on the classification of the system being used. The European AI Act distinguishes AI systems by risk level: those used in high-impact professional contexts — such as the production of legal, tax or employment documentation — fall into categories requiring transparency, process documentation and verifiable human supervision.

This does not mean that every firm must produce a formal certification for every output. It means that, in the event of an inspection or dispute, it must be possible to answer three questions precisely: which AI system was used, who supervised the output, and how that supervision is documented. Those who cannot answer these questions are exposed, regardless of the quality of the tool they adopted.

Italian Law 132/2025 adds a further layer: the obligation to disclose AI use to the client is not a bureaucratic formality, but a tool for protecting both the client and the professional. You can read more in our dedicated article on Italian Law 132/2025 and client disclosure. Documentation of control is, at the same time, proof of compliance and proof of professional diligence.