How to Sample for Human Review After a Vision Verdict

Automating judgement does not remove the need for human review, it relocates it. Somebody still has to check whether the system is judging correctly, and the question is what they should look at. The intuitive answer — review every nth item, or a fixed percentage — is the weakest of the available options, because it samples uniformly while the risk is never uniform. A fixed percentage spends most of its effort on ordinary conditions that the system already handles, and almost none on the periods and check items where it is genuinely uncertain. A risk-weighted sample produces more useful information from fewer reviews.

Stratify, then sample within strata

The confidence-based row is the one that makes this practical. If the system records how confident it was in each verdict, low-confidence decisions can be pulled for review automatically, which concentrates human attention exactly where errors are most likely. That presupposes confidence being written into the record, which is another argument for the field design in MES write-back field design. Where confidence is not available, prioritise by alarm and borderline status plus batch and time stratification, which recovers most of the benefit.

How much to sample

A commonly workable starting point is all alarms, all borderline cases, and a light daily sample of conforming parts stratified by shift and batch — a few dozen items, not a fixed percentage. The shift from percentage to count matters: at low volume a percentage yields too few items to mean anything, and at high volume it consumes more labour than the information justifies. A fixed daily count produces a comparable sample every day. Adjust from there. Early in a system's life, sample more heavily so that weaknesses surface quickly; once performance is stable, reduce. And where the escape cost is high, sample permanently more heavily regardless of how well the system appears to be doing, because that is precisely the case where an undetected failure is most expensive.

• Record the review result: what was reviewed, what was found, what was concluded. Unrecorded review is indistinguishable from no review.
• Close the loop: a finding must change something — a criterion, a threshold, a training sample, an imaging condition. Otherwise review is pure cost.
• Count the disagreements: the disagreement rate between system and reviewer is the single most informative number the process produces.
• Review the reviewer occasionally: two people looking at the same sample independently is the only way to know whether the review standard itself is drifting.

The feedback loop is the point

Review without feedback is a cost with no return, and it is the most common way these processes die: people spend an hour a day checking images, nothing changes, and within a few months the review is being ticked without being done. The loop is straightforward when designed deliberately. A disagreement between system and reviewer becomes a candidate training sample, or triggers a criterion review. A borderline case that recurs becomes a reason to look at the imaging conditions. A growing disagreement rate becomes a reason to examine whether the criterion itself has drifted from what production actually needs. What this requires is that somebody owns the loop, meeting perhaps monthly, with the review records in front of them. The measured version of this — miss rate and false alarm rate tracked over time — is the subject of misses versus false alarms.