A provider of software that predicts the risk of sepsis in hospitals built its processes for classic models: a fixed training data set, a fixed test set, release of the version. Then the team adds an analysis of discharge letters through a large language model from a third-party provider. The release procedure still checks the provider’s own application version. That the model provider updates its model behind the interface is foreseen in no step at all. The processes carefully check what stays the same and overlook what changes.
EN 18286 requires providers, in clause 6.1.3, to consider the AI technologies in use when they deal with risks to the quality management system (QMS). The standard mentions different kinds of machine learning, rule-based expert systems, logic-based methods and dedicated chips as examples. Each of them places different demands on testing, release and monitoring.
What is behind it?
Processes are built for the technology in the house when they are planned. When a new technology arrives, the old test steps often only appear to fit. They run through, yet they miss the properties that make up the risk of the new technology.
Three typical gaps
- Continuously learning systems. A system that keeps learning after deployment changes its behaviour without a new release. A process that only releases versions does not see these changes. EN 18286 therefore requires the predetermined changes to be defined in the design, verified and validated, and all changes to be logged (clause 8.8).
- Third-party models. A model integrated through an interface can change without the provider of the high-risk system doing anything. If you do not track model version and source separately from your own application, you notice the switch only in the results.
- New model types. For generative models with open-ended outputs, an accuracy figure on a fixed test set hardly suffices. Additional tests for invented content, evasive behaviour and susceptibility to manipulated inputs are common.
Why the gap goes unnoticed
The test steps are formally met, and the reports look familiar. Often the knowledge to spot the gap is missing as well: EN 18286 requires the nature of the AI technologies in use to be considered in the competence requirements (clause 7.2.3). Someone who has only ever tested classic models will not necessarily ask the right questions about a language model.
Impact
Untested changes in behaviour reach the people the system affects. In continuously learning systems, biased results can reinforce themselves through feedback loops, which is exactly what Article 15(4) of the AI Act aims to prevent. A substantial modification that nobody recognizes as such is especially delicate in legal terms: under Article 43(4) it requires a new conformity assessment, and without one the system in use is no longer covered by the earlier assessment. The technical documentation then describes a system that no longer runs in that form.
Practical examples
Silent update. A provider of a system that ranks job applications by suitability uses a general-purpose AI model for text analysis. The model provider releases a new version and the interface stays the same. Three weeks later the team notices that applications with certain wording are rated differently. Nobody had tested the new version.
Learning in operation. A provider of a system that flags suspected fraud in social benefits lets the model keep learning from caseworkers’ decisions. Because caseworkers mainly review flagged cases, the model learns from its own suggestions. The flagging rate for one postcode area rises month after month.
New hardware. A provider moves a model for defect detection on pressure vessels onto a device right next to the plant and compresses it for this purpose. The release procedure checks the software, and the compressed model on the new hardware is not validated separately.
Effective measures
The AI Provider Policy from the AI Compliance Starter Kit addresses this risk in these places:
- Section 4 (Strategy for regulatory compliance): Continuously learning systems are treated separately. Their outputs are not fed back unchecked, their predetermined changes are set out in the design and in the technical documentation, and a change of the underlying model counts as a substantial modification.
- Section 12 (Development and quality assurance): Model version and source of a purchased model are tracked separately from the application version. When a new system comes under the QMS or an existing one is substantially modified, the provider checks whether the processes need revising.
- Section 13 (Examination, test and validation): Test types, test data, metrics and acceptance thresholds are recorded per system and updated with every model version.
- Section 23 (Resources and security of supply): Supplied models are accepted, verified and validated before use, and every new version is checked again.
These ISO/IEC 42001 controls contribute as well:
- A.4 Resources for AI systems: Documenting tooling resources records which model types and algorithms a system uses.
- A.6.1 Management guidance for AI system development: The processes for responsible development are adapted to the technology in use.
- A.6.2 AI system life cycle: Verification, validation and monitoring in operation capture how the technology in use actually behaves.
- A.10 Third-party and customer relationships: Supplier management includes models and components provided by third parties.
Link to the AI Act
Article 17(1) of Regulation (EU) 2024/1689 requires procedures for design, development and quality assurance as well as for examination, testing and validation together with their frequency (points (b) to (d)). These procedures have to fit the technology in use, or they do not secure conformity. For continuously learning systems, Article 15(4) requires the risk of biased feedback loops to be eliminated or reduced as far as possible. Article 43(4) exempts predetermined changes set out at the initial conformity assessment from counting as substantial modifications. For supplied tools, services and components, Article 25(4) requires a written agreement on information and technical access. EN 18286 anchors these duties in the QMS; for the reach of the presumption of conformity, see EN 18286: Quality Management System for AI Providers.
Related QMS risks
- QMS-05 Missing information or support from third parties: If you do not understand a third-party model because the provider’s information is missing, you cannot test it properly either.
- QMS-06 Missing resources or competences: New technology calls for new knowledge in testing and monitoring.
- T07 Immature technology / model drift: The threat at system level when behaviour shifts over time.
Sources
- EN 18286:2026, clauses 6.1.3 b, 7.2.3, 8.8, 9.4 and 10.2
- Regulation (EU) 2024/1689 (AI Act), Article 3(23), Article 15(4), Article 17(1), Article 25(4) and Article 43(4)
- AI Provider Policy, Sections 4, 12, 13 and 23
- EN 18286: Quality Management System for AI Providers in the Cenedril wiki