Collecting and Labelling Shopfloor Data Without Building In Bias

When judgement uses deep learning, data quality matters far more than data volume, and the problem is almost never that there is too little. It is that the data is skewed. Collected over two days, on a single shift, on one machine, with parts placed by one person, the resulting model memorises those conditions precisely and falls apart as soon as the batch, the shift or the operator changes. Avoiding that has to happen at collection time, not afterwards. The second risk lives in labelling, and it is not mislabelling so much as inconsistency: the same person calls something a pass on Monday and a fail on Tuesday, and the training set gains a batch of self-contradictory samples from which the model can only learn noise.

Five dimensions collection has to cover

• Batch: different suppliers and different arrival ages produce different appearances; cover the common incoming states.
• Shift: ambient light, machine temperature and operator experience all differ across day, evening and night. Do not collect from one shift only.
• Lighting: shop lighting changes through the day, machine doors open and close, and neighbouring equipment casts its own light into the frame.
• Station and operator: the same action looks different at different stations and in different hands, particularly in placement attitude.
• Operating condition: empty, loaded, mid-changeover, just started and after hours of running all have different image characteristics.

These five do not need equal volume, but all of them need representation. A practical approach is to build a collection schedule across dimensions before shooting, spreading times and shifts deliberately; compared with shooting whenever there is a free moment, a schedule removes a great deal of skew. One further point that gets missed: capture the metadata with each image — batch, shift, station, timestamp — because without it, neither stratified sampling during labelling and training nor diagnosis when the model misbehaves is possible. This is the same requirement as write-back field design, one stage earlier.

Borderline samples: where models are won and lost

Borderline means two things: parts that are just inside tolerance but look marginal, and parts that look acceptable but are actually out. They occur naturally at low rates and are the main source of both misses and false alarms, because the model wobbles exactly here. They cannot be collected by waiting. They have to be hunted: from historical rework, from alarm records, and from the experienced eye of somebody saying this batch looks off. Once found, they should form their own group, split between training and acceptance, with no overlap between the two. If there genuinely are too few, it is better to tighten the criterion and state explicitly that this region is not yet covered than to go live with a model that behaves randomly at the boundary. The trade-off between misses and false alarms is set out in weighing misses against false alarms.

Situations the labelling rules must cover

The unjudgeable label matters and should not be dropped. Forcing a labeller into a binary choice when the case is genuinely ambiguous injects random noise into the training set, and the model learns that this kind of image is sometimes right and sometimes wrong — which is exactly the flickering behaviour you then see in production. Reporting unjudgeable separately has additional value: if its share is high, either the imaging conditions are insufficient or the criterion itself is too vague, and both of those are design problems rather than things to solve by hoping the model copes.

Tools and effort estimation

For tools, classification labelling — which class does this image belong to — costs a fraction of detection or segmentation labelling. If the criterion can be reduced to classification, design it that way first; the same outcome can cost several times less in labelling effort. Rough rules of thumb: a simple classification label takes seconds, while a bounding box or outline takes tens of seconds to minutes. Work backwards from that to size the effort — how many images, and what does each cost — before committing people. Allow for review effort too, typically twenty to thirty percent of labelling volume. There is no need to start with a heavyweight platform; in early exploration, a lightweight tool and a working rule set beats a platform built before the problem is understood.