Zum Hauptinhalt springen
Misuse pattern · AI Act

MU-08 Continued use despite a known malfunction

Updated on 6 min Reviewed by: Cenedril Editorial
EU AI ActISO/IEC 42001

An online retailer uses an AI system to detect fraud in orders. After an update, it classifies a striking number of orders from one particular region as suspicious. Customer service notices the cluster after two days and posts it in the IT team chat. The answer is that the provider is working on a fix, and until then the system keeps running. For three weeks, customers from that region are turned away. The fault was known, nobody decided on continued operation, and that was exactly the decision.

This pattern describes a system that stays in use although faulty behaviour has already been observed. For the AI Act (Regulation (EU) 2024/1689), it is reasonably foreseeable misuse under Article 3(13): hardly any organization likes to switch off a system its workflows depend on.

What is behind it?

Every AI system makes errors, and not every error calls for shutting it down. Once a malfunction is known, however, a conscious decision is needed: continue with restrictions, roll back to the previous version or suspend. The pattern arises when that decision is never taken or, despite a recognizable risk, comes down on the side of continued operation. Often a long time passes between the first observation and any reaction, during which harm accumulates.

How it happens in daily work

  • Dependency. The workflow no longer works without the system, and there is no fallback procedure.
  • Unclear responsibility. Nobody knows who may suspend a system, and nobody wants to do it on their own authority.
  • Informal reports. Observations end up in a chat or a hallway conversation, never in the incident process.
  • Hope in the provider. The fix has been announced, and until then things are supposed to work somehow.
  • Getting used to workarounds. Staff correct the errors by hand, and the fault becomes the normal state.
  • Small appearance. The fault concerns only certain cases and therefore looks harmless.

Why it is risky

The harm continues and grows every day. It often hits one particular group again and again, as in the example above. Knowledge adds to this: from the first report onwards the organization acts knowingly, which aggravates liability questions. And without a formal report, the provider learns of the fault too late or not at all, so other deployers face the same problem.

Impact

The impact ranges from lost revenue and dissatisfied customers to harm to health, for instance where medical or safety-related systems are affected. From the moment a fault is known, every further harm becomes a decision of the organization. For high-risk systems, a breach of the duty to suspend under Article 26(5) comes on top.

Practical examples

Predictive maintenance after a sensor replacement. A plant uses a system that predicts pump wear. After the sensors are replaced, it reports ‘normal’ throughout. Maintenance knows the values have been wrong ever since and lets the system run anyway, because switching to fixed intervals means extra work. A pump fails and damages an entire installation.

Chatbot with outdated tariffs. A municipal utility changes its prices, and the chatbot keeps quoting the old ones. The team has known for weeks and refers complaints to the website. Customers sign contracts on the basis of the wrong information and later demand the quoted prices.

Speech recognition in medical documentation. After an update, a hospital’s speech recognition confuses similar-sounding drug names. Doctors correct the errors when they notice them. Nobody reports to the provider or rolls back to the previous version until a wrong medication appears in a discharge letter.

Effective controls

The following ISO/IEC 42001 controls counter this pattern.

Prevention:

  • A.3 Internal organization: Roles define who may suspend a system. The channel for reporting concerns provides for timely escalation to management, and people who report are protected from reprisals.
  • A.6.2 AI system life cycle: Monitoring, repair, updates and support are defined for operation, including a procedure for responding to errors.
  • A.9 Use of AI systems: If the deployment gives rise to concern, the organization communicates this internally and to the supplier.

Detection:

Response:

The AI Usage Policy requires in Section 7 that malfunctions are reported without delay and without waiting for certainty. Section 9 sets the support route and repair procedure per system, Section 10 the suspension of high-risk systems in case of risk. For reports to authorities, the Starter Kit offers a serious incident report template.

For high-risk AI systems, the provider evaluates foreseeable misuse under Article 9(2)(b) and names the circumstances that may lead to risks in the instructions for use (Article 13(3)(b)(iii)). The people exercising oversight must be able to detect and address anomalies, malfunctions and unexpected performance (Article 14(4)(a)) and to stop the system (point (e)).

For the deployer, Article 26(5) is the central provision: it monitors operation on the basis of the instructions for use. If it has reason to believe that use may present a risk within the meaning of Article 79(1), it informs the provider or distributor and the market surveillance authority without undue delay and suspends use. It reports serious incidents under Article 3(49) immediately, first to the provider. The provider takes corrective action under Article 20 and reports serious incidents to the authorities under Article 73 (→ EU AI Act).

Sources

Frequently asked questions

Do I have to switch off an AI system at every error?

No. What matters is that someone with authority makes a conscious decision: continue with restrictions, roll back to the previous version or suspend. For high-risk systems the decision is partly prescribed: if there is reason to believe there is a risk within the meaning of Article 79(1), the deployer suspends use under Article 26(5) and informs the provider and the market surveillance authority.

What is a serious incident under the AI Act?

Under Article 3(49), an incident or malfunction that directly or indirectly leads to grave consequences, such as death or serious harm to a person's health, a serious and irreversible disruption of critical infrastructure, an infringement of Union law obligations intended to protect fundamental rights, or serious harm to property or the environment. Deployers of high-risk systems then immediately inform the provider first.

How do I deal with a purchased system whose provider does not fix a fault?

Document the fault and the report to the provider, set a deadline and demand the correction. ISO/IEC 42001 foresees that the organization requires corrective action from a supplier whose system does not work as intended. Until the fix, you decide on restrictions, a fallback procedure or suspension and record the decision.

No Cookies!

This wiki collects nothing, bakes nothing and leaves nothing behind. There's nothing to consent to. Privacy doesn't get better than this.