Zum Hauptinhalt springen
Misuse pattern · AI Act

MU-07 Automated decision without individual review

Updated on 6 min Reviewed by: Cenedril Editorial
EU AI ActISO/IEC 42001GDPR

A health insurer has applications for medical aids pre-checked by an AI system. The system recommends approval or rejection, and according to the procedure the case handler makes the decision. When applications pile up, the department configures rejection recommendations to flow straight into a rejection notice. Formally a person still decides, in reality a calculated result does, and the insured never learn of it.

This pattern describes cases in which an output decides a case on its own although a human decision is required. It is the sharpest form of misuse in dealing with results and reasonably foreseeable misuse under Article 3(13) of the AI Act (Regulation (EU) 2024/1689), because every backlog creates a wish for automation.

What is behind it?

Many AI systems are built as decision aids. The decision itself stays with a person, because the law requires it, because the risk analysis assumes it or because the system lacks the context that only a look at the individual case provides. The pattern begins the moment a recommendation turns into an effect without any step in between. The distinction from MU-02: there, a review takes place, only superficially. Here, there is none for the case at all.

How it happens in daily work

  • Batch processing. Thousands of cases are processed at once, and the results go out together.
  • Creeping expansion. At first only clear-cut cases are handled automatically, then the line moves step by step.
  • Chained workflows. The output triggers the next step in the workflow, such as sending a letter, and nobody has built in a stop in between.
  • Misread recommendation. The interface shows ‘Recommendation: reject’, and the team treats it as an instruction.
  • Collective sign-off. A person signs off a list of hundreds of results without opening a single case.

Why it is risky

Without individual review, special circumstances are not considered, and errors of the system become decisions. Legally, the pattern touches a sensitive point: under Article 22(1) GDPR, people have the right not to be subject to a decision based solely on automated processing that produces legal effects concerning them or similarly significantly affects them. Exceptions exist only under paragraph 2 and with safeguards such as the right to obtain human intervention, to express one’s point of view and to contest the decision (paragraph 3).

Impact

The people affected are denied benefits, have applications rejected, accounts blocked or contracts terminated. Because automated decisions are issued in large numbers, an error repeats itself a hundredfold at once. The organization faces legal consequences as well: invalid decisions, GDPR fines, damages and the effort of reopening every affected case. The larger the number of cases, the more expensive every error that runs through without a person.

Practical examples

Automatic rejections in recruitment. An applicant tracking system ranks applications by fit. HR activates the function that automatically rejects all applications below a certain rank, although the internal rule requires a decision by the responsible person. Qualified applicants with an unusual CV receive a rejection within minutes.

Settlement of minor claims. An insurer automatically rejects minor claims when a fraud score exceeds a threshold. Customers receive a standard letter without reasons. Only a wave of complaints reveals that the score is conspicuously high for certain postcodes.

Account suspension at a delivery service. A delivery service suspends rider accounts automatically when a system reports anomalies in delivery times. Riders who were slower because of roadworks or illness lose their income before any person knows the circumstances.

Effective controls

The following ISO/IEC 42001 controls counter this pattern.

Prevention:

Detection:

Response:

  • A.9 Use of AI systems: Affected decisions are reviewed individually after the fact, the people concerned are informed and errors are corrected.

The AI Usage Policy prohibits in Section 4.2 using the output of an AI system without competent review as the sole basis for a decision with legal or similarly significant effect. Section 6 governs human oversight, Section 8 the right to explanation under Article 86.

For high-risk AI systems, the provider evaluates foreseeable misuse under Article 9(2)(b) and names risks and oversight measures in the instructions for use (Article 13(3)(b)(iii) and (d)). Article 14(4)(d) requires that the person exercising oversight can decide in a given situation not to use the system or to disregard its output. For remote biometric identification under Annex III point 1(a), action may in principle only be taken on an identification result once two competent people have verified it separately (Article 14(5)). The deployer uses the system accordingly under Article 26(1) and (2), informs affected people under paragraph 11 and explains the role of the system on request under Article 86 (→ EU AI Act).

Alongside the AI Act, Article 22 of the GDPR applies to all systems that process personal data, regardless of their risk class.

Sources

Frequently asked questions

When is a decision based solely on automated processing under Article 22 GDPR?

When no person with authority and expertise reviews the individual case and can change the outcome. A purely formal approval changes nothing. Following the judgment of the Court of Justice of the EU in case C-634/21 (SCHUFA) of 2023, even an automatically determined score can fall under it if a third party's decision draws decisively on it.

May clear-cut cases be decided automatically?

Possibly: where it is legally permitted, for instance under the exceptions in Article 22(2) GDPR with the safeguards of paragraph 3, and where the instructions for use and the risk assessment allow it. A common approach is to grant approvals automatically and reserve adverse decisions for people. This line is drawn deliberately and documented.

What do I have to tell the people affected?

Under Article 13(2)(f) and Article 14(2)(g) GDPR, the existence of automated decision-making under Article 22, with meaningful information about the logic involved and the consequences. Deployers of high-risk systems under Annex III inform affected people of the use under Article 26(11) of the AI Act, and Article 86 grants a right to an explanation of the system's role.

No Cookies!

This wiki collects nothing, bakes nothing and leaves nothing behind. There's nothing to consent to. Privacy doesn't get better than this.