When Defect Samples Are Scarce: What Actually Works

A process under control produces few defects. That is the goal of manufacturing, and it is also the reason defect detection is hard: the failure you most need to catch is the one with the fewest examples, and frequently with none. Project plans routinely assume that a few weeks of production will supply a training set. It will not, because a defect occurring at a rate of one in ten thousand needs a production volume the site may not reach for months, and because the first occurrence is often the one nobody thought to photograph.

The archive is a biased sample

The least obvious and most damaging problem is selection bias. Most defect archives are assembled from parts that were rejected, by inspectors, using the current inspection method. That means the images in the archive show the defects the current system already catches, seen the way the current system sees them. The defect that escapes today is not in the folder, because by definition it was not rejected. Training on that archive produces a model that reproduces the existing detection performance and improves nothing about the escapes — which were the reason for the project.

The bias also reinforces itself. If the model is then used to decide what to archive, the archive converges on the model own view of what a defect looks like, and the disagreements between the model and a human are exactly the data that would have improved it. The remedy is deliberate: sample released and good parts at random on a defined schedule, have a person review them periodically, and archive the review outcome, not the model verdict. It costs inspection time and it is the only way the archive contains information the current system does not already hold. For the collection side of this, the stratified approach described in training data collection is the same discipline applied from the start.

What to do when there are almost no examples

• Define the boundary of normal, and collect that instead. If the defect cannot be collected, collect good parts across the full range of natural variation: supplier, batch, season, shift, surface finish, part position. A model that knows the boundary of acceptable can flag anything outside it, and abnormal parts are generated by production without any effort on your part.
• Use anomaly detection rather than classification. It trains on known-good material only and reports deviation from it, so it needs no defect examples at all. It cannot tell you what the defect is, and its false-positive rate needs managing, but it is the correct tool when the failure is rare and unpredictable.
• Manufacture samples deliberately. Where it is safe and honest to do so, seeded defects on sacrificial parts produce controlled variation: a scratch of a known depth, a missing component, a deliberate misplacement. These are not real production defects, and the resulting model should be validated on real ones, but they establish the response much earlier than waiting would.
• Mine the historical record. Rework, repair and customer return records, warranty claims and line-stop logs often contain defects that were never photographed at the station, sometimes with a returned part still available to image.

Choosing the operating point when defects are rare

When defects occur at one in ten thousand, the arithmetic of the operating point becomes counter-intuitive. A detector with an excellent-sounding false-positive rate of one per cent will, on ten thousand good parts, produce a hundred false alarms to catch perhaps one defect. The financial case is not decided by accuracy at all but by the relative cost of a miss and a false alarm: if a miss reaches a customer and costs a recall, a hundred false alarms costing a minute each is a bargain; if a false alarm stops the line for ten minutes, the same ratio is unsustainable.

The practical resolution is usually a two-stage arrangement rather than one threshold. A sensitive first stage flags everything suspicious, and it is allowed to be generous because it only routes parts. A second stage — a human review of the flagged frames and a re-inspection of the part — makes the actual disposition. This keeps the sensitive setting where a miss is prevented and places the cost of the false positive where it is cheap, and it is a design that also produces the reviewed samples that improve the model. The metrics that describe the two stages, and which of them acceptance should look at, are set out in precision and recall.