Zum Hauptinhalt springen
Misuse pattern · AI Act

MU-05 Circumventing human oversight

Updated on 6 min Reviewed by: Cenedril Editorial
EU AI ActISO/IEC 42001

A payment service provider uses an AI system that holds suspicious transfers until a person has checked and released them. Before Christmas the review queue backs up, and merchants complain about delayed payments. A team leader sets up a rule that releases all held payments automatically after two hours. On paper, human oversight still exists, while in the system a timer has taken it over.

This pattern describes cases in which the intended intervention or stop option is bypassed or switched off. For the AI Act (Regulation (EU) 2024/1689), it is reasonably foreseeable misuse under Article 3(13): every safeguard that costs time creates the wish to skip it.

What is behind it?

Human oversight is built in where errors of the system can become expensive. The provider defines the arrangements: either built into the system or described so that the deployer can implement them. They include confirmation steps, review queues, thresholds above which a person decides, alerts and a stop function. Each of these arrangements slows the workflow, and that is exactly why it comes under pressure.

How it happens in daily work

  • Pressure on throughput. The review queue becomes a bottleneck, and the fastest fix is to bypass it.
  • Alert fatigue. A system reports too many anomalies that turn out to be harmless. Eventually the alerts are switched off or acknowledged unread.
  • Technical shortcuts. IT connects the system directly through an interface, and the confirmation step of the user interface gets lost on the way.
  • Unrestricted configuration. Administrator rights allow oversight functions to be deactivated without approval, and nobody notices the change.
  • Macros and scripts. A small helper program clicks ‘Confirm’ automatically so nobody has to click any more.

Why it is risky

Human oversight is often the last safeguard for risks that remain despite all other measures. Article 14(2) of the AI Act describes it in just that way. Once it is gone, errors take effect at once: payments go to fraudsters, posts are deleted, machines keep running. If the circumvention interferes technically with the system and the provider did not foresee it, it may amount to a substantial modification under Article 3(23). For a high-risk system, the deployer is then itself treated as the provider under Article 25(1)(b).

Impact

The impact of this pattern is typically high, because oversight is planned precisely for systems with significant risk. Assets, health and safety can be affected, and so can fundamental rights such as freedom of expression or protection from discrimination. Switching off oversight removes the safeguard at exactly the point that was previously identified as the most dangerous.

Practical examples

Moderation on a platform. A forum has an AI system flag posts, which moderators then review. To reduce the backlog, the team configures posts above a certain score to be deleted at once. Critical yet permissible posts are caught as well, and the authors never learn that no human ever saw them.

Person detection on a production line. A camera with person detection stops a robot arm as soon as someone enters the work area. Because it also triggers on shadows and reflective clothing, the maintenance team deactivates detection for the night shift. A worker is injured while handling the machine.

Shift planning through an interface. An AI system proposes shift plans that a team lead has to confirm. When connecting it to the time recording system, IT builds in automatic confirmation. Plans that violate rest periods go to staff without review.

Effective controls

The following ISO/IEC 42001 controls counter this pattern.

Prevention:

  • A.3 Internal organization: Roles and responsibilities set out who may change oversight functions and who is accountable for the system. Anyone who notices a circumvention can raise it through the channel for reporting concerns.
  • A.6.2 AI system life cycle: Requirements, release criteria for deployment and the rules for operation anchor oversight in the system. Event logs record when oversight functions were changed or bypassed.
  • A.9 Use of AI systems: The objectives for responsible use determine where people check results and with what authority they may override the system. Use follows the provider’s documents, including the oversight they foresee.

Detection:

  • A.6.2 AI system life cycle: Analysing the event logs reveals approvals by technical users, unusually fast confirmations and deactivated alerts.

Response:

  • A.9 Use of AI systems: When a circumvention is discovered, the organization restores oversight or suspends use until that is done, and reviews the decisions taken during the period without oversight.

The AI Usage Policy states in Section 6 that a consequential AI result never takes effect automatically without a person being able to check and correct it, and that the person exercising oversight can stop the system. Section 10 requires, for high-risk systems, that this person receives the support needed to intervene.

Article 14 requires that high-risk AI systems can be effectively overseen by people while in use. The provider determines the arrangements and either builds them in or describes them for implementation by the deployer (Article 14(3)). The people exercising oversight must be able to disregard, override or reverse an output (Article 14(4)(d)) and to bring operation to a safe halt with a stop button or a similar procedure (point (e)). Under Article 14(2), this also serves to limit risks from foreseeable misuse.

The provider evaluates misuse under Article 9(2)(b) and names the oversight measures in the instructions for use (Article 13(3)(b)(iii) and (d)). The deployer uses the system in line with those instructions under Article 26(1), assigns oversight to people with authority and support under paragraph 2 and monitors operation under paragraph 5. It keeps the automatically generated logs for at least six months under paragraph 6 (→ EU AI Act).

Sources

Frequently asked questions

Is adjusting thresholds already a circumvention?

Not automatically. Many providers allow thresholds to be set within the limits stated in the instructions for use. An adjustment becomes a circumvention when it removes cases from human review that the oversight concept is meant to cover, or when it happens outside the defined change route and without a risk assessment.

What do we do when human review becomes a bottleneck?

Make the bottleneck visible and decide formally: more capacity, better pre-sorting, thresholds adjusted after a risk assessment or a narrower use. The responsible person makes the decision, and it is documented. A silent shortcut only hides the risk.

Who may trigger the stop function of an AI system?

For high-risk systems, every person assigned oversight, because Article 14(4)(e) of the AI Act requires that they can interrupt operation. The organization also defines who restarts the system. The stop function must be easy to reach and must not disappear behind administrator rights.

No Cookies!

This wiki collects nothing, bakes nothing and leaves nothing behind. There's nothing to consent to. Privacy doesn't get better than this.