Zum Hauptinhalt springen
QMS risk · EN 18286

QMS-04 AI technologies in use overstretch the processes

Updated on 6 min Reviewed by: Cenedril Editorial
EN 18286EU AI ActISO/IEC 42001

A provider of software that predicts the risk of sepsis in hospitals built its processes for classic models: a fixed training data set, a fixed test set, release of the version. Then the team adds an analysis of discharge letters through a large language model from a third-party provider. The release procedure still checks the provider’s own application version. That the model provider updates its model behind the interface is foreseen in no step at all. The processes carefully check what stays the same and overlook what changes.

EN 18286 requires providers, in clause 6.1.3, to consider the AI technologies in use when they deal with risks to the quality management system (QMS). The standard mentions different kinds of machine learning, rule-based expert systems, logic-based methods and dedicated chips as examples. Each of them places different demands on testing, release and monitoring.

What is behind it?

Processes are built for the technology in the house when they are planned. When a new technology arrives, the old test steps often only appear to fit. They run through, yet they miss the properties that make up the risk of the new technology.

Three typical gaps

  • Continuously learning systems. A system that keeps learning after deployment changes its behaviour without a new release. A process that only releases versions does not see these changes. EN 18286 therefore requires the predetermined changes to be defined in the design, verified and validated, and all changes to be logged (clause 8.8).
  • Third-party models. A model integrated through an interface can change without the provider of the high-risk system doing anything. If you do not track model version and source separately from your own application, you notice the switch only in the results.
  • New model types. For generative models with open-ended outputs, an accuracy figure on a fixed test set hardly suffices. Additional tests for invented content, evasive behaviour and susceptibility to manipulated inputs are common.

Why the gap goes unnoticed

The test steps are formally met, and the reports look familiar. Often the knowledge to spot the gap is missing as well: EN 18286 requires the nature of the AI technologies in use to be considered in the competence requirements (clause 7.2.3). Someone who has only ever tested classic models will not necessarily ask the right questions about a language model.

Impact

Untested changes in behaviour reach the people the system affects. In continuously learning systems, biased results can reinforce themselves through feedback loops, which is exactly what Article 15(4) of the AI Act aims to prevent. A substantial modification that nobody recognizes as such is especially delicate in legal terms: under Article 43(4) it requires a new conformity assessment, and without one the system in use is no longer covered by the earlier assessment. The technical documentation then describes a system that no longer runs in that form.

Practical examples

Silent update. A provider of a system that ranks job applications by suitability uses a general-purpose AI model for text analysis. The model provider releases a new version and the interface stays the same. Three weeks later the team notices that applications with certain wording are rated differently. Nobody had tested the new version.

Learning in operation. A provider of a system that flags suspected fraud in social benefits lets the model keep learning from caseworkers’ decisions. Because caseworkers mainly review flagged cases, the model learns from its own suggestions. The flagging rate for one postcode area rises month after month.

New hardware. A provider moves a model for defect detection on pressure vessels onto a device right next to the plant and compresses it for this purpose. The release procedure checks the software, and the compressed model on the new hardware is not validated separately.

Effective measures

The AI Provider Policy from the AI Compliance Starter Kit addresses this risk in these places:

  • Section 4 (Strategy for regulatory compliance): Continuously learning systems are treated separately. Their outputs are not fed back unchecked, their predetermined changes are set out in the design and in the technical documentation, and a change of the underlying model counts as a substantial modification.
  • Section 12 (Development and quality assurance): Model version and source of a purchased model are tracked separately from the application version. When a new system comes under the QMS or an existing one is substantially modified, the provider checks whether the processes need revising.
  • Section 13 (Examination, test and validation): Test types, test data, metrics and acceptance thresholds are recorded per system and updated with every model version.
  • Section 23 (Resources and security of supply): Supplied models are accepted, verified and validated before use, and every new version is checked again.

These ISO/IEC 42001 controls contribute as well:

Article 17(1) of Regulation (EU) 2024/1689 requires procedures for design, development and quality assurance as well as for examination, testing and validation together with their frequency (points (b) to (d)). These procedures have to fit the technology in use, or they do not secure conformity. For continuously learning systems, Article 15(4) requires the risk of biased feedback loops to be eliminated or reduced as far as possible. Article 43(4) exempts predetermined changes set out at the initial conformity assessment from counting as substantial modifications. For supplied tools, services and components, Article 25(4) requires a written agreement on information and technical access. EN 18286 anchors these duties in the QMS; for the reach of the presumption of conformity, see EN 18286: Quality Management System for AI Providers.

Sources

Frequently asked questions

What is a predetermined change?

A change to a continuously learning system that is already defined in the design and described in the technical documentation. If it was set out in this way at the initial conformity assessment, it does not count as a substantial modification under Article 43(4) of the AI Act. Every execution is still verified, validated and logged.

Is switching to a different third-party model a substantial modification?

The Regulation does not settle this across the board. A modification is substantial if it was not planned and affects compliance with the requirements or changes the intended purpose (Article 3(23)), and EN 18286 requires the rationale for the classification to be recorded. The AI Provider Policy explicitly counts a change of the underlying model among the substantial modifications.

When do I have to adapt my processes to a new technology?

EN 18286 requires a check as soon as a new AI system comes under the QMS or an existing one is substantially modified. Based on the characteristics of the system, the provider then decides whether the processes need revising. It has proven useful to ask the same question when a new model type is introduced in an existing system.

No Cookies!

This wiki collects nothing, bakes nothing and leaves nothing behind. There's nothing to consent to. Privacy doesn't get better than this.