Zum Hauptinhalt springen
AI threat · ISO/IEC 42001 Annex C

T07 Immature technology / model drift

Updated on 5 min Reviewed by: Cenedril Editorial
ISO/IEC 42001EU AI Act

A payment service provider detects fraud with a model that shows excellent figures at launch. Within a year, fraudsters change their approach and customers pay more and more often with their smartphones. The detection rate drops month after month without the system ever reporting an error. A model is a snapshot of the world at the time of its training, and the world keeps moving.

ISO/IEC 42001 treats technology readiness as a risk source in Annex C. The standard sees two sides. Less mature technology brings unknown factors, such as unclear limits and boundary conditions or performance that drifts over time. Mature technology, for its part, tempts people into complacency because they have grown used to it.

What is behind it?

The threat combines two closely linked problems. With young technology, nobody knows exactly where its limits are. And every technology, young or old, loses accuracy when the conditions it learned under change. The performance of an AI system therefore has to be confirmed again and again in operation.

Immature technology

New model classes such as generative AI or AI agents that carry out actions on their own spread faster than test methods and practical experience can develop. Performance claims often come from benchmarks that have little in common with your own use. Early adopters meet failure modes nobody has described yet.

Drift in operation

  • Data drift. The inputs shift: new products, new customer groups, other devices, seasonal effects.
  • Concept drift. The relationship between input and correct result changes, for example in fraud patterns, buying behaviour or through a new legal situation.
  • Drift through continued learning. Systems that keep learning from production data can develop in an unwanted direction, especially when their own outputs flow back as new training data.
  • Drift caused by the provider. With hosted models, the provider can swap the model version. The same inputs then produce different results.

Growing used to mature technology

A system that has run reliably for years drops out of sight. Nobody looks at the metrics any more, review dates get postponed, doubts are brushed aside with “it has always worked”. The standard names exactly this habituation as a risk source.

Impact

Drift produces a growing number of wrong decisions, each of which hardly stands out. The loss of performance often hits a particular group first, such as new customers whose behaviour did not occur in training. A quality problem thus quickly turns into a question of fairness. Drift is creeping damage: no single day looks like an incident, yet the sum of the errors certainly does.

Practical examples

Demand forecast after a range change. A grocery retailer introduces an organic own brand with new item numbers. The forecasting model does not know these items and estimates their demand from weak similarities. For weeks, the stores hold too much fresh produce, and write-offs rise.

Silent update of a hosted language model. A company extracts invoice fields through the interface of a language model. The provider switches to a new model version that changes the output format slightly. Downstream processing leaves fields empty without raising an alarm, and the error only shows at month-end closing.

Young technology in a pilot. An insurer tests an AI agent that sends follow-up questions to customers on its own during claims handling. In edge cases it sends duplicate or contradictory messages. Nobody had expected this failure mode, because there was hardly any experience with the still young technology.

Effective controls

The following ISO/IEC 42001 controls counter this threat.

Prevention:

  • A.5 Assessing impacts of AI systems: The assessment looks at consequences across the entire life cycle and shows how much depends on the system’s performance.
  • A.6.2 AI system life cycle: Verification and validation work with defined criteria, and the essentials of operation are defined in advance, from watching the system and its performance to fixing faults, rolling out updates and providing help.

Detection:

  • A.6.2 AI system life cycle: Performance monitoring on production data reveals drift and the need for retraining.
  • A.9 Use of AI systems: The objectives for responsible use include watching the accuracy of outputs and reporting concerns when the system gets worse on production data.

Response:

Article 15 of the AI Act (Regulation (EU) 2024/1689) requires high-risk systems to remain consistently accurate and robust throughout their life cycle. For systems that keep learning after being placed on the market, Article 15(4) demands measures against feedback loops in which biased outputs influence future inputs. Under Article 72, providers collect and analyse performance data from operation to check continued compliance, and deployers monitor operation under Article 26(5). Learning changes that were predetermined and described in the technical documentation do not count as a substantial modification under Article 43(4) (→ EU AI Act).

Sources

Frequently asked questions

What distinguishes data drift from concept drift?

With data drift, the inputs change, for instance because new customer groups or devices come in. With concept drift, the relationship between input and correct result changes, for instance when fraudsters adapt their methods. Both reduce performance, and concept drift is harder to spot because the inputs can stay inconspicuous.

How often does a model need retraining?

There is no fixed rule. What counts is how fast the operating environment changes and how much performance measurably declines. A proven approach is to set thresholds and tie retraining to crossing them, complemented by regular review dates.

Why can mature technology be a risk as well?

ISO/IEC 42001 points out that technology becomes a risk once it is taken for granted. A system that has run well for years gets less attention, and monitoring and reviews fall asleep. That is exactly when a change goes unnoticed for a long time.

No Cookies!

This wiki collects nothing, bakes nothing and leaves nothing behind. There's nothing to consent to. Privacy doesn't get better than this.