An engineering company trains a model to predict bearing damage in pumps, using five years of sensor data. Only after rollout does it turn out that some sensors were miscalibrated for years and that failures were often entered into the maintenance software days late. The model has learned to predict the wrong thing reliably. A learning system can never be better than the data it learned from.
ISO/IEC 42001 lists risk sources related to machine learning in Annex C. At the top are the quality of the data and the way it is collected. Both act directly on objectives such as safety and robustness, and in the product fairness is added as the third casualty of bad data.
What is behind it?
For a learning system, data is textbook and exam material at once. Whatever is missing or wrong in the data, the model learns along with everything else, and it passes the errors on with great confidence. Data quality is more than the absence of errors. It has several sides, and each one alone can make a system unusable.
Dimensions of data quality
- Completeness. Missing values, missing periods, missing groups of cases. A model can only learn patterns it gets to see.
- Correctness. Measurement errors, typos, records assigned to the wrong entity and wrongly assigned labels. Labels are especially delicate, because people often set them under time pressure and with unclear rules.
- Representativeness. The data reflects the situations and groups of people the system meets in operation, and in sufficient numbers.
- Timeliness. Data from before a product change or a change in the law describes a world that no longer exists in that form.
- Consistency. Different sources use different units, formats or definitions for the same feature.
Where the flaws come from
Problems often start at collection, for instance when only data from one site or one customer group was available. When data is taken over from external sources, origin and collection rules remain unclear. During preparation, well-meant cleaning can remove exactly the rare cases that matter. And in operation, users feed in inputs that look different from the training data. Deployers are responsible for this last part.
Impact
Bad data degrades accuracy, robustness and fairness at the same time. An inaccurate model produces wrong decisions and rework, a model that lacks robustness fails at small deviations, and unbalanced data disadvantages groups that are poorly represented in it. Data flaws are especially costly because they often surface only after rollout. By then a patch rarely suffices: data has to be collected or labelled again and the model retrained.
Practical examples
Document classification at an insurer. A model is supposed to sort incoming mail by request type. The training labels come from several teams, and one team files as a “complaint” what another records as an “enquiry”. The model takes over the contradictions, and urgent complaints end up in the slow queue.
Sales forecasting in retail. A retailer learns from its sales data how much to reorder. In some branches, sales of zero appear for weeks, although the goods were simply not on the shelf. The model infers low demand, orders less, and the shelves stay empty again.
Wrong inputs at the deployer. A law firm uses a purchased system that extracts data from contracts. The instructions for use require scans at a certain resolution, and the firm scans at a lower one to save storage. The recognition errors only come to light when a deadline is entered incorrectly.
Effective controls
The following ISO/IEC 42001 controls counter this threat.
Prevention:
- A.7 Data for AI systems: The organization defines data quality requirements and makes sure that data for development and operation meets them. It documents how data is acquired and selected, where it comes from and by which criteria it is prepared.
- A.5 Assessing impacts of AI systems: The assessment asks who would be hit by faulty or incomplete data and draws attention to groups that are poorly represented in the data.
- A.6.2 AI system life cycle: Verification and validation check whether the test data reflects the intended domain of use, and release criteria hold back a model that falls short.
Detection:
- A.6.2 AI system life cycle: Monitoring in operation shows whether the system performs as expected on real production data.
Response:
- A.7 Data for AI systems: Records of data provenance make it possible to trace an error back to its source and to clean up every affected dataset.
- A.9 Use of AI systems: Deployers use the system only with data that matches its documentation and pass concerns on to the responsible people and to the supplier.
Link to the AI Act
For high-risk systems, Article 10 of the AI Act (Regulation (EU) 2024/1689) is the central provision. Training, validation and test data are subject to data governance that covers, among other things, collection and origin, preparation steps such as labelling and cleaning, the assumptions made, the suitability of the data and identified gaps. The datasets have to be relevant for the purpose, sufficiently representative and as far as possible free of errors and complete, and they have to take into account the geographical, behavioural or functional setting of the use. Under Article 26(4), deployers make sure that the input data under their control is relevant and representative. Article 15 asks for an appropriate level of accuracy and robustness. For personal data, the accuracy principle of Article 5(1)(d) GDPR applies as well (→ EU AI Act).
Related threats
- T12 Bias and discrimination: Unbalanced data is the most common cause of discriminatory results.
- T08 Data poisoning: What happens here by accident happens there on purpose.
- T07 Immature technology / model drift: Even good data ages when the world changes.
Sources
- ISO/IEC 42001:2023, Annex C.3.4: Risk sources related to machine learning, together with Annex B.7
- Regulation (EU) 2024/1689 (AI Act), Articles 10, 15 and 26
- Regulation (EU) 2016/679 (GDPR), Article 5
- ISO/IEC 42001 in the Cenedril wiki: Annex C and the risk sources for AI systems