Zum Hauptinhalt springen
Misuse pattern · AI Act

MU-02 Adopting outputs unchecked

Updated on 6 min Reviewed by: Cenedril Editorial
EU AI ActISO/IEC 42001

An insurer has a language model draft replies to customer complaints. The plan is for case handlers to read each draft, compare it with the file and adjust it. After a few months, almost every draft goes out unchanged. One of them promises a customer a refund his contract does not provide for, and the customer insists on the written promise. The review was part of the process on paper, and in daily work it had shrunk to a single click.

This pattern covers results that are passed on or acted upon without the intended review. For the AI Act (Regulation (EU) 2024/1689), it is a textbook case of reasonably foreseeable misuse under Article 3(13): hardly any human behaviour is as predictable as trusting a machine that is right most of the time.

What is behind it?

Many AI systems are built on the assumption that a person checks the result before it takes effect. The review is part of the safety concept: providers refer to it in the instructions for use, deployers write it into their workflows. Once the review disappears, the system runs without the safeguard it was designed around. The model’s error rate stays the same, yet now every error reaches its target.

How it happens in daily work

  • Habit. After a hundred drafts were fine, the hundred and first looks just as reliable. This trust in automated results is called automation bias and is well documented.
  • Time pressure and metrics. Whoever is measured by throughput saves time where nobody measures: in the review.
  • Fluent language. Generative systems write smoothly and confidently even when the content is wrong. The form suggests a quality the content lacks.
  • Missing means. The reviewer has no access to the file, the source or the expertise needed to judge the result. Waving it through is all that remains.
  • Blurred responsibility. “The AI suggested it” becomes an excuse, although responsibility stays with the organization and with the person who uses the result.

Why it is risky

Hallucinated facts, wrong calculations, biased assessments and third-party material in texts pass unfiltered into the outside world or into decisions. A second kind of harm follows: approval records and logs claim a control that never happened. In an audit or a dispute, the organization then evidences a level of care it never applied.

Impact

The impact follows the reach of the result. An unchecked internal summary costs, at worst, one misguided meeting. An unchecked promise to a customer, a flawed medical letter or a wrong figure in an annual report has financial, health or legal consequences. The more people a result reaches and the harder it is to recall, the more expensive the skipped review becomes.

Practical examples

Code assistant in development. A development team adopts suggestions from a code assistant, and the tests pass. Nobody notices that a database query forwards user input without protection. The gap is found only during a penetration test, after the code had been running in production for months.

Summaries of hospital letters. A medical practice has incoming discharge letters summarized, and the doctor signs off the summary. In one case it leaves out a newly diagnosed allergy. She did not open the full letter, since the summary was there.

Press release with an invented quote. The communications team of an industry association has a press release drafted and publishes it after a quick look. The model invented the quote from an expert it contains. The correction reaches fewer people than the original release.

Effective controls

The following ISO/IEC 42001 controls counter this pattern.

Prevention:

  • A.3 Internal organization: Roles and responsibilities set out who reviews which outputs and who may reject them. Responsibility stays with named people.
  • A.8 Information for interested parties: Users receive information about typical errors, limitations and how to read the outputs. People who know the weak spots review more precisely.
  • A.9 Use of AI systems: The processes and objectives for responsible use determine where people check results and with what authority they may override the system.

Detection:

  • A.6.2 AI system life cycle: Monitoring of operation and performance, including the accuracy of outputs, reveals errors that slipped through review. Event logs show whether outputs were edited or adopted unchanged.

Response:

  • A.3 Internal organization: Through the channel for reporting concerns, staff can flag that a review cannot be done properly under the given conditions. That is often the first warning of this pattern.

The AI Usage Policy treats every AI output as a proposal (Section 4.1) and sets out in Section 4.4 what the review looks at: accuracy, bias, rights in the material and documentation for consequential uses. Section 6 requires a competent person to review a consequential result before anyone acts on it.

For high-risk AI systems, the provider has to estimate the risks of foreseeable misuse under Article 9(2)(b) and name the circumstances that may lead to risks in the instructions for use (Article 13(3)(b)(iii)). Article 14(4) requires that the people assigned to oversight remain aware of automation bias (point (b)), interpret the outputs correctly (point (c)) and are able to disregard or override them (point (d)). Under Article 26(2), the deployer assigns oversight to people with the necessary competence, training and authority and gives them the support they need (→ EU AI Act).

Two further provisions apply regardless of risk class. Under Article 4, providers and deployers take measures that support the AI literacy of their staff, and that includes knowing how error-prone the systems are. And under Article 50(4), labelling of AI-generated text on matters of public interest may be dropped only where the text has undergone human review or editorial control and someone holds editorial responsibility. A review that exists only on paper does not support that exception.

Sources

Frequently asked questions

How thoroughly do I have to check an AI output?

That depends on its effect. For an internal brainstorming list a critical look is enough; for texts to customers, figures in reports or statements about people, facts, sources and calculations are checked against reliable records. Define for each use case what the review covers and when it is documented.

Is it enough for someone to click ‘Approve’?

Only if the person was able to judge the content and actually did so. That takes expertise, time, access to the source data and the authority to reject an output. If any of these is missing, the click documents a review that never took place.

How do I notice that reviews have become a formality?

Typical signs are a very high share of unchanged outputs, strikingly short handling times, no rejection at all over long periods and errors that customers or affected people spot first. Spot checks by a second person make the pattern visible before damage occurs.

No Cookies!

This wiki collects nothing, bakes nothing and leaves nothing behind. There's nothing to consent to. Privacy doesn't get better than this.