A payment service provider detects fraud with a model that shows excellent figures at launch. Within a year, fraudsters change their approach and customers pay more and more often with their smartphones. The detection rate drops month after month without the system ever reporting an error. A model is a snapshot of the world at the time of its training, and the world keeps moving.
ISO/IEC 42001 treats technology readiness as a risk source in Annex C. The standard sees two sides. Less mature technology brings unknown factors, such as unclear limits and boundary conditions or performance that drifts over time. Mature technology, for its part, tempts people into complacency because they have grown used to it.
What is behind it?
The threat combines two closely linked problems. With young technology, nobody knows exactly where its limits are. And every technology, young or old, loses accuracy when the conditions it learned under change. The performance of an AI system therefore has to be confirmed again and again in operation.
Immature technology
New model classes such as generative AI or AI agents that carry out actions on their own spread faster than test methods and practical experience can develop. Performance claims often come from benchmarks that have little in common with your own use. Early adopters meet failure modes nobody has described yet.
Drift in operation
- Data drift. The inputs shift: new products, new customer groups, other devices, seasonal effects.
- Concept drift. The relationship between input and correct result changes, for example in fraud patterns, buying behaviour or through a new legal situation.
- Drift through continued learning. Systems that keep learning from production data can develop in an unwanted direction, especially when their own outputs flow back as new training data.
- Drift caused by the provider. With hosted models, the provider can swap the model version. The same inputs then produce different results.
Growing used to mature technology
A system that has run reliably for years drops out of sight. Nobody looks at the metrics any more, review dates get postponed, doubts are brushed aside with “it has always worked”. The standard names exactly this habituation as a risk source.
Impact
Drift produces a growing number of wrong decisions, each of which hardly stands out. The loss of performance often hits a particular group first, such as new customers whose behaviour did not occur in training. A quality problem thus quickly turns into a question of fairness. Drift is creeping damage: no single day looks like an incident, yet the sum of the errors certainly does.
Practical examples
Demand forecast after a range change. A grocery retailer introduces an organic own brand with new item numbers. The forecasting model does not know these items and estimates their demand from weak similarities. For weeks, the stores hold too much fresh produce, and write-offs rise.
Silent update of a hosted language model. A company extracts invoice fields through the interface of a language model. The provider switches to a new model version that changes the output format slightly. Downstream processing leaves fields empty without raising an alarm, and the error only shows at month-end closing.
Young technology in a pilot. An insurer tests an AI agent that sends follow-up questions to customers on its own during claims handling. In edge cases it sends duplicate or contradictory messages. Nobody had expected this failure mode, because there was hardly any experience with the still young technology.
Effective controls
The following ISO/IEC 42001 controls counter this threat.
Prevention:
- A.5 Assessing impacts of AI systems: The assessment looks at consequences across the entire life cycle and shows how much depends on the system’s performance.
- A.6.2 AI system life cycle: Verification and validation work with defined criteria, and the essentials of operation are defined in advance, from watching the system and its performance to fixing faults, rolling out updates and providing help.
Detection:
- A.6.2 AI system life cycle: Performance monitoring on production data reveals drift and the need for retraining.
- A.9 Use of AI systems: The objectives for responsible use include watching the accuracy of outputs and reporting concerns when the system gets worse on production data.
Response:
- A.6.2 AI system life cycle: Defined update procedures govern retraining, a return to the previous version and informing users.
- A.10 Third-party and customer relationships: If a supplier’s system no longer works as intended, the organization demands corrections and clarifies who is responsible for changes to the model.
Link to the AI Act
Article 15 of the AI Act (Regulation (EU) 2024/1689) requires high-risk systems to remain consistently accurate and robust throughout their life cycle. For systems that keep learning after being placed on the market, Article 15(4) demands measures against feedback loops in which biased outputs influence future inputs. Under Article 72, providers collect and analyse performance data from operation to check continued compliance, and deployers monitor operation under Article 26(5). Learning changes that were predetermined and described in the technical documentation do not count as a substantial modification under Article 43(4) (→ EU AI Act).
Related threats
- T01 Complexity of environment: What is a rare situation today becomes the normal case tomorrow through drift.
- T06 Life cycle and maintenance gaps: Missing monitoring and untested retraining are life cycle gaps.
- T04 Poor data quality: Outdated training data is a quality flaw that grows over time.
Sources
- ISO/IEC 42001:2023, Annex C.3.7: Technology readiness, together with Annex B.6.2.6
- Regulation (EU) 2024/1689 (AI Act), Articles 15, 26, 43 and 72
- ISO/IEC 42001 in the Cenedril wiki: Annex C and the risk sources for AI systems