A logistics company reads delivery notes with an AI system. The promised accuracy applies to German and English documents, scanned at good resolution, up to a fixed number of pages per hour. In the Christmas season, more than twice as many pages arrive, many as phone photos, some in Polish. Errors pile up, and nobody connects them with the limits. Every single condition under which the promised accuracy applies was violated at the same time.
This pattern describes systems used outside the agreed volume, response time, language or data quality. For the AI Act (Regulation (EU) 2024/1689), it is reasonably foreseeable misuse under Article 3(13): peaks in load, new markets and poorer data are part of every organization’s daily routine.
What is behind it?
Every performance figure of an AI system holds under conditions: certain input formats and languages, a minimum data quality, a value range, a load and sufficient computing power. Once these conditions are left behind, there is no reliable statement on the quality of the results. Unlike use for other tasks (MU-01), the purpose stays the same. What changes are the circumstances in which the system is supposed to work.
How it happens in daily work
- Peaks in load. Seasonal business, promotions or disruptions in other systems drive volumes far beyond what was agreed.
- New languages and markets. The system is used for customers whose language it barely knows.
- Poorer data. Phone photos where scans were planned, incomplete records, new formats from another upstream system.
- Reduced resources. To save costs, the model runs on smaller hardware or in a cheaper performance tier.
- Time pressure. A workflow demands answers faster than the system can reliably deliver them.
- Unknown limits. The figures are in the documentation, and nobody in operation has read them.
Why it is risky
Performance usually drops quietly. Results look as always and are wrong more often. On timeouts, many integrations fall back on default values that nobody recognizes as errors. Quality also often drops unevenly: whoever speaks a less supported language or sends poorer photos gets poorer results. And warranty or contractual claims against the provider weaken, because the promised conditions were not met.
Impact
The impact grows with volume: when a system makes errors during a peak, it makes them in many cases at once. Wrongly recorded deliveries, missed appointments and flawed decisions pile up before anyone sees the connection. Outside the operating limits, the organization carries the risk alone, because nobody has promised any performance any more.
Practical examples
Phone assistant at a medical practice. A practice has appointment calls taken by a voice assistant designed for German. Many patients speak Turkish or Arabic, or German with a strong accent. The assistant books wrong appointments and misjudges urgency, and the practice notices only from empty and double-booked slots.
Forecast beyond the value range. A municipal utility uses a model that forecasts electricity demand. It was trained on temperatures of a temperate climate. During a record heat wave, it delivers forecasts that are far off, and purchasing relies on them.
Fraud check with a time limit. A payment workflow expects the fraud model’s answer within a few hundred milliseconds. On the busiest day of the year, the service responds more slowly, the integration runs into a timeout and approves payments by default. Numerous fraudulent transactions are among them.
Effective controls
The following ISO/IEC 42001 controls counter this pattern.
Prevention:
- A.6.2 AI system life cycle: Requirements and specification describe the conditions under which the system is to work. The deployment plan takes into account the environment in which it really runs.
- A.8 Information for interested parties: The documents for users state the system’s limits in an understandable way, including for the people who operate it day to day.
- A.10 Third-party and customer relationships: Supplier contracts set out service commitments and conditions, and the organization monitors whether they are met.
- A.9 Use of AI systems: Use follows the documents, and the data the system is applied to matches what those documents describe.
Detection:
- A.6.2 AI system life cycle: System and performance monitoring in operation captures errors and failures and checks whether the system works with real data as expected.
Response:
- A.10 Third-party and customer relationships: If the provider’s commitments do not cover the real demand, they are renegotiated or the use is adjusted.
The AI Usage Policy sets out in Section 9 the support route, the responsible unit and the promised response time for each system, for purchased systems together with the provider’s commitments. Section 7 requires malfunctions to be reported, and Section 10 takes up the duty of suitable input data for high-risk systems.
Link to the AI Act
For high-risk AI systems, describing the limits is part of the instructions for use. They state the level of accuracy, robustness and cybersecurity together with the known and foreseeable circumstances that may affect it (Article 13(3)(b)(ii)), the circumstances of foreseeable misuse (point (iii)), the input data specifications (point (vi)) and the computational and hardware resources needed (point (e)). Article 15 requires accuracy figures in the instructions and resilience against errors and faults from the environment. The provider evaluates misuse under Article 9(2)(b).
The deployer uses the system in line with the instructions under Article 26(1), ensures under paragraph 4 that input data under its control is relevant and sufficiently representative, and monitors operation under paragraph 5 (→ EU AI Act).
Related patterns and threats
- T01 Complexity of environment: The threat behind it: the environment has more variations than the tests.
- T04 Poor data quality: Poorer input data is the most common way limits are exceeded.
- T05 System hardware failures: Scarce or disrupted computing resources push the limits down.
- MU-06 Use on untested groups: Related when the limit concerns a language or a group of people.
Sources
- Regulation (EU) 2024/1689 (AI Act), Article 3(13), Articles 9, 13, 15 and 26
- ISO/IEC 42001:2023, Annex B.6.2.2, B.6.2.5, B.6.2.6, B.9.4 and B.10.3
- AI Usage Policy of the AI Compliance Starter Kit, Sections 7, 9 and 10
- ISO/IEC 42001 in the Cenedril wiki: Annex A and the controls for operating AI systems