Zum Hauptinhalt springen
Misuse pattern · AI Act

MU-12 Use outside the operating limits

Updated on 6 min Reviewed by: Cenedril Editorial
EU AI ActISO/IEC 42001

A logistics company reads delivery notes with an AI system. The promised accuracy applies to German and English documents, scanned at good resolution, up to a fixed number of pages per hour. In the Christmas season, more than twice as many pages arrive, many as phone photos, some in Polish. Errors pile up, and nobody connects them with the limits. Every single condition under which the promised accuracy applies was violated at the same time.

This pattern describes systems used outside the agreed volume, response time, language or data quality. For the AI Act (Regulation (EU) 2024/1689), it is reasonably foreseeable misuse under Article 3(13): peaks in load, new markets and poorer data are part of every organization’s daily routine.

What is behind it?

Every performance figure of an AI system holds under conditions: certain input formats and languages, a minimum data quality, a value range, a load and sufficient computing power. Once these conditions are left behind, there is no reliable statement on the quality of the results. Unlike use for other tasks (MU-01), the purpose stays the same. What changes are the circumstances in which the system is supposed to work.

How it happens in daily work

  • Peaks in load. Seasonal business, promotions or disruptions in other systems drive volumes far beyond what was agreed.
  • New languages and markets. The system is used for customers whose language it barely knows.
  • Poorer data. Phone photos where scans were planned, incomplete records, new formats from another upstream system.
  • Reduced resources. To save costs, the model runs on smaller hardware or in a cheaper performance tier.
  • Time pressure. A workflow demands answers faster than the system can reliably deliver them.
  • Unknown limits. The figures are in the documentation, and nobody in operation has read them.

Why it is risky

Performance usually drops quietly. Results look as always and are wrong more often. On timeouts, many integrations fall back on default values that nobody recognizes as errors. Quality also often drops unevenly: whoever speaks a less supported language or sends poorer photos gets poorer results. And warranty or contractual claims against the provider weaken, because the promised conditions were not met.

Impact

The impact grows with volume: when a system makes errors during a peak, it makes them in many cases at once. Wrongly recorded deliveries, missed appointments and flawed decisions pile up before anyone sees the connection. Outside the operating limits, the organization carries the risk alone, because nobody has promised any performance any more.

Practical examples

Phone assistant at a medical practice. A practice has appointment calls taken by a voice assistant designed for German. Many patients speak Turkish or Arabic, or German with a strong accent. The assistant books wrong appointments and misjudges urgency, and the practice notices only from empty and double-booked slots.

Forecast beyond the value range. A municipal utility uses a model that forecasts electricity demand. It was trained on temperatures of a temperate climate. During a record heat wave, it delivers forecasts that are far off, and purchasing relies on them.

Fraud check with a time limit. A payment workflow expects the fraud model’s answer within a few hundred milliseconds. On the busiest day of the year, the service responds more slowly, the integration runs into a timeout and approves payments by default. Numerous fraudulent transactions are among them.

Effective controls

The following ISO/IEC 42001 controls counter this pattern.

Prevention:

  • A.6.2 AI system life cycle: Requirements and specification describe the conditions under which the system is to work. The deployment plan takes into account the environment in which it really runs.
  • A.8 Information for interested parties: The documents for users state the system’s limits in an understandable way, including for the people who operate it day to day.
  • A.10 Third-party and customer relationships: Supplier contracts set out service commitments and conditions, and the organization monitors whether they are met.
  • A.9 Use of AI systems: Use follows the documents, and the data the system is applied to matches what those documents describe.

Detection:

  • A.6.2 AI system life cycle: System and performance monitoring in operation captures errors and failures and checks whether the system works with real data as expected.

Response:

The AI Usage Policy sets out in Section 9 the support route, the responsible unit and the promised response time for each system, for purchased systems together with the provider’s commitments. Section 7 requires malfunctions to be reported, and Section 10 takes up the duty of suitable input data for high-risk systems.

For high-risk AI systems, describing the limits is part of the instructions for use. They state the level of accuracy, robustness and cybersecurity together with the known and foreseeable circumstances that may affect it (Article 13(3)(b)(ii)), the circumstances of foreseeable misuse (point (iii)), the input data specifications (point (vi)) and the computational and hardware resources needed (point (e)). Article 15 requires accuracy figures in the instructions and resilience against errors and faults from the environment. The provider evaluates misuse under Article 9(2)(b).

The deployer uses the system in line with the instructions under Article 26(1), ensures under paragraph 4 that input data under its control is relevant and sufficiently representative, and monitors operation under paragraph 5 (→ EU AI Act).

Sources

Frequently asked questions

Where do I find the operating limits of an AI system?

In the instructions for use, the technical documentation and the contract with the provider. For high-risk systems, the instructions must state the accuracy, the circumstances that affect it, the input data specifications and the computational and hardware resources needed. If information is missing, ask before deployment and record the answer.

Is use just above the limit already a problem?

Not every overshoot causes errors at once, yet outside the limits there is no reliable statement on performance any more. That is why it has to be defined in advance what happens when a limit is exceeded: throttle, queue, hand over to people or reject.

Who is responsible if a system delivers wrong results outside its limits?

First of all the deployer, because it used the system in a way that departs from the instructions for use. High-risk systems are to be used in line with the instructions under Article 26(1) of the AI Act. Claims against the provider are also on shaky ground if the promised conditions were not met.

No Cookies!

This wiki collects nothing, bakes nothing and leaves nothing behind. There's nothing to consent to. Privacy doesn't get better than this.