After a cyberattack, an insurer’s case management system is down for two weeks. So that claims do not pile up, the head of department has the AI system for claims assessment connected directly to the mailbox. The review steps, the logging and the approval by a second person depend on the failed system and are dropped. For two weeks, assessments go out unchecked. In the crisis, the very system whose oversight counted as indispensable in normal operation ran without any oversight at all.
This pattern describes AI systems used during a disruption while the accompanying checks are suspended. For the AI Act (Regulation (EU) 2024/1689), it is reasonably foreseeable misuse under Article 3(13): under pressure, every organization reaches for the tool that still works.
What is behind it?
The controls around an AI system rarely sit in the system itself. Approvals run through a workflow, logs through a central platform, reviews through people who have to be available at the time. If one of these building blocks fails, the AI system often keeps running unchanged, just without its safeguards. Sometimes it is even extended in the emergency, because it is supposed to take over work that people would otherwise do.
How it happens in daily work
- Dependent controls. Logging, approval steps or monitoring depend on another system that has failed.
- Staff shortage. Waves of illness, strikes or holiday periods leave reviewers missing.
- Improvisation. Makeshift integrations, private devices or public AI tools stand in for failed services.
- Contingency plans without AI. The plan covers servers and telephony, and AI systems and their controls do not appear in it.
- Forgotten catch-up. After recovery, nobody reviews the decisions made during fallback operation.
Why it is risky
Disruptions are exactly the situation in which unusual inputs occur and errors are most likely. Without review they go undetected, without logs they can no longer be investigated later. Attackers deliberately exploit crises, because attention lies elsewhere. And the obligations of deployers of high-risk systems under Article 26 of the AI Act do not pause during a disruption.
Impact
The impact is high, because errors accumulate over days or weeks and because the organization has little capacity for corrections in the crisis. Customers and patients are affected, and with safety-related systems, life and limb as well. Fallback operation aggravates the risks that controls are meant for and weakens the controls themselves at the same time.
Practical examples
Emergency call triage during a flood. A control centre uses a system that pre-sorts incoming emergency calls by urgency, and a dispatcher checks each rating. During a flood, calls multiply and the check is suspended. Several calls with unclear descriptions end up far back in the queue, although people are in mortal danger.
Public tool as a substitute. A company’s internal, approved AI service is down for three days. Staff switch to free public tools and paste customer data into them, because the work has to go on. Nobody had settled this case in advance.
Monitoring during a move. A company relocates its data centres. Monitoring of model performance is switched off for the duration of the move and set up again only weeks later. In the meantime the model has degraded because of changed data, and nobody noticed.
Effective controls
The following ISO/IEC 42001 controls counter this pattern.
Prevention:
- A.2 Policies related to AI: The AI policy is aligned with the organization’s other policies, including those on business continuity, and expressly applies during a disruption too.
- A.3 Internal organization: Roles define who may approve restricted operation and who is accountable for returning to full controls.
- A.6.2 AI system life cycle: The rules for ongoing operation cover monitoring, repair and support. A proven approach is to describe operation during a disruption there as well.
- A.9 Use of AI systems: The processes for responsible use apply regardless of the operating situation, and human oversight remains part of the deployment.
Detection:
- A.6.2 AI system life cycle: Event logs and monitoring show when controls had failed or were switched off, provided they run independently of the disrupted systems.
Response:
- A.3 Internal organization: After the disruption, decisions from fallback operation are reviewed, errors corrected and the lessons fed into the contingency plan.
The ISO/IEC 27001 controls on information security during disruption (A.5.29) and ICT readiness for business continuity (A.5.30) apply in addition. The AI Usage Policy remains binding in fallback operation, in particular Section 3.3 on shadow AI, Section 6 on human oversight and Section 10 on suspending high-risk systems.
Link to the AI Act
For high-risk AI systems, the provider evaluates foreseeable misuse under Article 9(2)(b) and states in the instructions for use the circumstances that may lead to risks (Article 13(3)(b)(iii)) as well as maintenance and care (point (e)). Article 15(4) requires resilience against errors and faults and names technical redundancy with backup or fail-safe plans as a possible means. Oversight must be able to bring the system to a safe state under Article 14(4)(e).
The deployer uses the system in line with the instructions under Article 26(1), assigns oversight to capable people under paragraph 2, monitors operation under paragraph 5 and suspends use if there is a risk. It keeps the logs under paragraph 6. Article 26 contains no exception for operational emergencies (→ EU AI Act).
Related patterns and threats
- MU-05 Circumventing human oversight: In fallback operation, oversight is often dropped on grounds of urgency.
- MU-08 Continued use despite a known malfunction: Related when a disrupted system keeps running despite recognizable faults.
- MU-03 Entering confidential or personal data: Makeshift solutions with public tools quickly lead to data leaks.
- T06 Life cycle and maintenance gaps: Missing rules for maintenance and operation are what make fallback operation dangerous.
Sources
- Regulation (EU) 2024/1689 (AI Act), Article 3(13), Articles 9, 13, 14, 15 and 26
- ISO/IEC 42001:2023, Annex B.2.3, B.3.2, B.6.2.6, B.6.2.8 and B.9.3
- AI Usage Policy of the AI Compliance Starter Kit, Sections 3.3, 6 and 10
- Business Continuity Plan of the Starter Kit