Zum Hauptinhalt springen
Misuse pattern · AI Act

MU-14 Use in fallback operation without the usual controls

Updated on 5 min Reviewed by: Cenedril Editorial
EU AI ActISO/IEC 42001

After a cyberattack, an insurer’s case management system is down for two weeks. So that claims do not pile up, the head of department has the AI system for claims assessment connected directly to the mailbox. The review steps, the logging and the approval by a second person depend on the failed system and are dropped. For two weeks, assessments go out unchecked. In the crisis, the very system whose oversight counted as indispensable in normal operation ran without any oversight at all.

This pattern describes AI systems used during a disruption while the accompanying checks are suspended. For the AI Act (Regulation (EU) 2024/1689), it is reasonably foreseeable misuse under Article 3(13): under pressure, every organization reaches for the tool that still works.

What is behind it?

The controls around an AI system rarely sit in the system itself. Approvals run through a workflow, logs through a central platform, reviews through people who have to be available at the time. If one of these building blocks fails, the AI system often keeps running unchanged, just without its safeguards. Sometimes it is even extended in the emergency, because it is supposed to take over work that people would otherwise do.

How it happens in daily work

  • Dependent controls. Logging, approval steps or monitoring depend on another system that has failed.
  • Staff shortage. Waves of illness, strikes or holiday periods leave reviewers missing.
  • Improvisation. Makeshift integrations, private devices or public AI tools stand in for failed services.
  • Contingency plans without AI. The plan covers servers and telephony, and AI systems and their controls do not appear in it.
  • Forgotten catch-up. After recovery, nobody reviews the decisions made during fallback operation.

Why it is risky

Disruptions are exactly the situation in which unusual inputs occur and errors are most likely. Without review they go undetected, without logs they can no longer be investigated later. Attackers deliberately exploit crises, because attention lies elsewhere. And the obligations of deployers of high-risk systems under Article 26 of the AI Act do not pause during a disruption.

Impact

The impact is high, because errors accumulate over days or weeks and because the organization has little capacity for corrections in the crisis. Customers and patients are affected, and with safety-related systems, life and limb as well. Fallback operation aggravates the risks that controls are meant for and weakens the controls themselves at the same time.

Practical examples

Emergency call triage during a flood. A control centre uses a system that pre-sorts incoming emergency calls by urgency, and a dispatcher checks each rating. During a flood, calls multiply and the check is suspended. Several calls with unclear descriptions end up far back in the queue, although people are in mortal danger.

Public tool as a substitute. A company’s internal, approved AI service is down for three days. Staff switch to free public tools and paste customer data into them, because the work has to go on. Nobody had settled this case in advance.

Monitoring during a move. A company relocates its data centres. Monitoring of model performance is switched off for the duration of the move and set up again only weeks later. In the meantime the model has degraded because of changed data, and nobody noticed.

Effective controls

The following ISO/IEC 42001 controls counter this pattern.

Prevention:

  • A.2 Policies related to AI: The AI policy is aligned with the organization’s other policies, including those on business continuity, and expressly applies during a disruption too.
  • A.3 Internal organization: Roles define who may approve restricted operation and who is accountable for returning to full controls.
  • A.6.2 AI system life cycle: The rules for ongoing operation cover monitoring, repair and support. A proven approach is to describe operation during a disruption there as well.
  • A.9 Use of AI systems: The processes for responsible use apply regardless of the operating situation, and human oversight remains part of the deployment.

Detection:

  • A.6.2 AI system life cycle: Event logs and monitoring show when controls had failed or were switched off, provided they run independently of the disrupted systems.

Response:

The ISO/IEC 27001 controls on information security during disruption (A.5.29) and ICT readiness for business continuity (A.5.30) apply in addition. The AI Usage Policy remains binding in fallback operation, in particular Section 3.3 on shadow AI, Section 6 on human oversight and Section 10 on suspending high-risk systems.

For high-risk AI systems, the provider evaluates foreseeable misuse under Article 9(2)(b) and states in the instructions for use the circumstances that may lead to risks (Article 13(3)(b)(iii)) as well as maintenance and care (point (e)). Article 15(4) requires resilience against errors and faults and names technical redundancy with backup or fail-safe plans as a possible means. Oversight must be able to bring the system to a safe state under Article 14(4)(e).

The deployer uses the system in line with the instructions under Article 26(1), assigns oversight to capable people under paragraph 2, monitors operation under paragraph 5 and suspends use if there is a risk. It keeps the logs under paragraph 6. Article 26 contains no exception for operational emergencies (→ EU AI Act).

Sources

Frequently asked questions

May I drop individual controls in an emergency?

The AI Act provides no relief from deployer obligations for operational disruptions. What is possible is to define in advance which controls are replaced by equivalent substitute measures during a disruption, for example a manual approval list for a failed approval system. If a control cannot be replaced, stopping the system is usually the safer choice.

Do AI systems belong in the contingency plan?

Yes, as soon as important workflows depend on them. The plan answers two questions: how does the workflow continue if the AI system fails? And what happens if the AI system runs while the controls around it have failed? The second question is often forgotten.

What happens to decisions taken during fallback operation?

They are reviewed after the return to normal operation, at least by sampling and in full for consequential decisions. Errors are corrected and the people affected informed. This review belongs in the contingency plan as a fixed step, or it gets lost in the bustle of recovery.

No Cookies!

This wiki collects nothing, bakes nothing and leaves nothing behind. There's nothing to consent to. Privacy doesn't get better than this.