Zum Hauptinhalt springen
QMS risk · EN 18286

QMS-01 The QMS misses its intended outcomes

Updated on 7 min Reviewed by: Cenedril Editorial
EN 18286EU AI ActISO/IEC 42001

A provider develops an AI system that ranks incoming emergency calls by urgency in dispatch centres. Its manual is complete: test plan, release procedure, monitoring plan with a monthly evaluation. Then a customer pushes for a new model version. The team ships it after a shortened test, the person who tested it also signs the release, and the monitoring evaluation is skipped two months in a row because the colleague in charge is tied up in another project. When the market surveillance authority asks for documents after a complaint, the room goes quiet. For the current version, there is no evidence at all that the described procedures were applied.

EN 18286 requires providers, in clause 6.1, to determine the risks to the quality management system (QMS) itself when they plan it. The first of these risks is the most obvious one: the system does not deliver what it was set up for. That means the ongoing evidence that the provider’s high-risk AI systems meet the requirements of the AI Act, from design to retirement.

What is behind it?

A QMS is a chain of processes. It only works if every link is actually performed and its result is recorded. When one link fails, the rest of the chain still looks tidy, yet the statement about the system’s conformity loses its basis. The standard therefore asks providers to consider, already at the planning stage, what could endanger the intended outcomes. For these risks the provider plans actions, builds them into the QMS processes and decides how to evaluate whether they work (clause 6.1.2).

Where processes typically break

  • Skipped tests. Deadline pressure shortens test runs or drops them entirely. This often affects small model updates that nobody classifies as a change.
  • Missed evaluations. The monitoring plan names a cycle, yet nobody checks whether the evaluation took place. The data is there, a judgement on continued compliance is missing.
  • Merged roles. The person who develops, tests and releases ends up being the same person. The release loses its value as an independent control.
  • Missing records. The test took place, but its result sits in a chat thread or a local file. As evidence, that is weak.

QMS risk or system risk?

The standard keeps the two strictly apart. Risk management for the individual AI system under Article 9 of the AI Act looks at dangers to health, safety and fundamental rights that come from the system. The risks under clause 6.1 concern whether the management system functions. Separate does not mean unrelated, though: a QMS that misses its outcomes leaves system risks undetected, because the very tests that would have found them are missing.

Impact

Without evidence there is no robust conformity assessment. In the internal control procedure under Annex VI, the provider itself verifies that its QMS meets Article 17 and that development and monitoring match the technical documentation. If it later turns out that a system on the market does not conform, corrective actions under Article 20 follow, in the worst case withdrawal or recall. Under Article 16 the QMS is one of the provider obligations whose breach can lead to fines under Article 99.

Then there are the consequences for the people the system affects. An error that a skipped test would have found hits them directly: an emergency rated too low, an application rejected without reason. It gets most expensive when a failed process only comes to light through a request from the authority, because by then there is no time left to close the gap properly.

Practical examples

Medical image analysis. A provider updates the model of software that pre-sorts skin images for review by doctors. Because the change counts as a “minor improvement”, only part of the test cases are run. The new version is worse at detecting a rare skin lesion. The full test run that the test plan requires for every model version would have caught this.

Scoring for banks. A software company supplies banks with a creditworthiness assessment system and keeps a monitoring plan with a quarterly evaluation for it. After a staff change, the evaluation is left undone twice. During this time the mix of applicants shifts, and the rejection rate for one age group rises sharply. The data showing this was available all along.

Candidate pre-selection. A provider sells a tool that pre-sorts job applications. Releases run through a ticket system. While preparing for an audit, the team notices that a third of all releases were tested and approved by the same person. The tests were fine, yet nobody confirmed them independently.

Effective measures

The AI Provider Policy from the AI Compliance Starter Kit addresses this risk in several places:

  • Section 3 (Processes of the quality management system): The process table lists the results of each process and, in a separate column, how its monitoring shows that it works. Every process thus becomes something you can check.
  • Section 8 (Life cycle processes): Every phase has entry and exit criteria. Verification and release lie with different roles, and a process that misses its criteria is handled as a corrective action.
  • Section 13 (Examination, test and validation): Tests run for every model version and before every release against defined acceptance thresholds.
  • Section 16 (Risk management): The risks of the QMS are kept in the AI Risk Register under the category “Quality management system”. Their actions are built into the processes and evaluated for effectiveness in the management review.
  • Section 17 (Post-market monitoring): The monitoring plan sets the cycle and the person responsible, and each evaluation is recorded.

These ISO/IEC 42001 controls contribute as well:

The topic comes up on schedule in the management review: EN 18286 lists the actions against QMS risks among the items reviewed there (clause 10.1).

Article 17(1) of Regulation (EU) 2024/1689 requires providers of high-risk AI systems to have a QMS that ensures compliance with the Regulation and is laid down in written policies, procedures and instructions. Its minimum content includes examination, test and validation procedures together with their frequency (point (d)) and a post-market monitoring system in line with Article 72 (point (h)). A QMS whose processes do not work misses exactly this purpose. The duty to determine the risks of the QMS itself comes from EN 18286; Article 17 does not name it explicitly. If you apply the standard, you can rely on the presumption of conformity for Article 17(1), to the extent defined by Annex ZA of the standard (→ EN 18286: Quality Management System for AI Providers).

Sources

Frequently asked questions

How does QMS-01 differ from a risk of the AI system?

QMS-01 concerns the management system: processes that should secure and demonstrate conformity do not work. A risk of the AI system concerns harm caused by the system itself, such as wrong results for a group of people. EN 18286 keeps the two apart, which is why the AI risk register lists QMS risks under a category of their own.

How do I notice that my QMS misses its outcomes?

By missing traces. If a released version has no complete test report, no approval by a second person or no monitoring evaluation in the planned cycle, a process has failed. Internal audits and the management review are the places where such gaps are meant to surface on schedule.

Is it enough to assess this risk once?

No. EN 18286 requires you to evaluate how effective the actions are, and the management review explicitly takes up the actions against QMS risks. A reassessment with every management review and after every finding that shows a failed process has proven useful.

No Cookies!

This wiki collects nothing, bakes nothing and leaves nothing behind. There's nothing to consent to. Privacy doesn't get better than this.