A health insurer has applications for medical aids pre-checked by an AI system. The system recommends approval or rejection, and according to the procedure the case handler makes the decision. When applications pile up, the department configures rejection recommendations to flow straight into a rejection notice. Formally a person still decides, in reality a calculated result does, and the insured never learn of it.
This pattern describes cases in which an output decides a case on its own although a human decision is required. It is the sharpest form of misuse in dealing with results and reasonably foreseeable misuse under Article 3(13) of the AI Act (Regulation (EU) 2024/1689), because every backlog creates a wish for automation.
What is behind it?
Many AI systems are built as decision aids. The decision itself stays with a person, because the law requires it, because the risk analysis assumes it or because the system lacks the context that only a look at the individual case provides. The pattern begins the moment a recommendation turns into an effect without any step in between. The distinction from MU-02: there, a review takes place, only superficially. Here, there is none for the case at all.
How it happens in daily work
- Batch processing. Thousands of cases are processed at once, and the results go out together.
- Creeping expansion. At first only clear-cut cases are handled automatically, then the line moves step by step.
- Chained workflows. The output triggers the next step in the workflow, such as sending a letter, and nobody has built in a stop in between.
- Misread recommendation. The interface shows ‘Recommendation: reject’, and the team treats it as an instruction.
- Collective sign-off. A person signs off a list of hundreds of results without opening a single case.
Why it is risky
Without individual review, special circumstances are not considered, and errors of the system become decisions. Legally, the pattern touches a sensitive point: under Article 22(1) GDPR, people have the right not to be subject to a decision based solely on automated processing that produces legal effects concerning them or similarly significantly affects them. Exceptions exist only under paragraph 2 and with safeguards such as the right to obtain human intervention, to express one’s point of view and to contest the decision (paragraph 3).
Impact
The people affected are denied benefits, have applications rejected, accounts blocked or contracts terminated. Because automated decisions are issued in large numbers, an error repeats itself a hundredfold at once. The organization faces legal consequences as well: invalid decisions, GDPR fines, damages and the effort of reopening every affected case. The larger the number of cases, the more expensive every error that runs through without a person.
Practical examples
Automatic rejections in recruitment. An applicant tracking system ranks applications by fit. HR activates the function that automatically rejects all applications below a certain rank, although the internal rule requires a decision by the responsible person. Qualified applicants with an unusual CV receive a rejection within minutes.
Settlement of minor claims. An insurer automatically rejects minor claims when a fraud score exceeds a threshold. Customers receive a standard letter without reasons. Only a wave of complaints reveals that the score is conspicuously high for certain postcodes.
Account suspension at a delivery service. A delivery service suspends rider accounts automatically when a system reports anomalies in delivery times. Riders who were slower because of roadworks or illness lose their income before any person knows the circumstances.
Effective controls
The following ISO/IEC 42001 controls counter this pattern.
Prevention:
- A.6.1 Management guidance for AI system development: The development processes contain requirements for human oversight, including workflows and tools, particularly where natural persons are affected.
- A.6.2 AI system life cycle: Requirements and release criteria set out where a person decides, and the system is built and deployed so that this step cannot be dropped.
- A.8 Information for interested parties: Users understand that an output is a proposal, and affected people learn what role the system plays in the decision.
- A.9 Use of AI systems: The objectives for responsible use include the question whether automated decision-making is appropriate for the use case at all. Reviewers receive the authority to override results.
Detection:
- A.6.2 AI system life cycle: Event logs show decisions without any human action, and monitoring reports conspicuous clusters.
Response:
- A.9 Use of AI systems: Affected decisions are reviewed individually after the fact, the people concerned are informed and errors are corrected.
The AI Usage Policy prohibits in Section 4.2 using the output of an AI system without competent review as the sole basis for a decision with legal or similarly significant effect. Section 6 governs human oversight, Section 8 the right to explanation under Article 86.
Link to the AI Act
For high-risk AI systems, the provider evaluates foreseeable misuse under Article 9(2)(b) and names risks and oversight measures in the instructions for use (Article 13(3)(b)(iii) and (d)). Article 14(4)(d) requires that the person exercising oversight can decide in a given situation not to use the system or to disregard its output. For remote biometric identification under Annex III point 1(a), action may in principle only be taken on an identification result once two competent people have verified it separately (Article 14(5)). The deployer uses the system accordingly under Article 26(1) and (2), informs affected people under paragraph 11 and explains the role of the system on request under Article 86 (→ EU AI Act).
Alongside the AI Act, Article 22 of the GDPR applies to all systems that process personal data, regardless of their risk class.
Related patterns and threats
- MU-02 Adopting outputs unchecked: The preliminary stage, in which the review still exists yet withers.
- MU-05 Circumventing human oversight: The technical route by which this pattern often arises.
- T03 Excessive automation (automation bias): The threat that feeds trust in automation.
- T12 Bias and discrimination: What flows systematically into decisions without individual review.
Sources
- Regulation (EU) 2024/1689 (AI Act), Article 3(13), Articles 9, 13, 14, 26 and 86
- Regulation (EU) 2016/679 (GDPR), Articles 13, 14 and 22
- Court of Justice of the European Union, judgment of 7 December 2023, case C-634/21 (SCHUFA Holding, scoring)
- ISO/IEC 42001:2023, Annex B.6.1.3, B.9.3 and B.9.4
- AI Usage Policy of the AI Compliance Starter Kit, Sections 4.2, 6 and 8