How to Run Acceptance for a Vision SOP Project

Acceptance for a vision SOP project is never really about one number called accuracy. It is about agreeing, in advance, on what sample the number is measured against, how the judgement is defined, and who decides that it is good enough. Where the threshold sits, how the sample is composed, how many misses and how many false alarms are tolerated, who responds to an alarm and within how long, how long records are kept — if none of this is written as measurable terms, the two sides will end up with different definitions of acceptable, almost guaranteed. The practical answer is to split acceptance into two stages: one for whether the system judges correctly, and one for whether it holds up in production. The first uses an offline image set; the second uses hours of real takt. Their criteria are completely different, and mixing them is how discussions go in circles.

Three things to freeze before anything is installed

Before hardware arrives, three things need to be frozen in writing, and moving any of them means running acceptance again. The first is the judgement object: which items this one station is asked to judge. This sounds redundant, and it is where most rework starts — the line asks whether the camera could "also check the screw angle while it is there", a request that never appeared in any requirements list and has no imaging condition behind it. The second is the judgement boundary: for the same action, where exactly does the system draw the line between pass and fail. Tightening a screw can reasonably be judged as "a torque signal was received", "the head sits flush against the workpiece", or "the head is within tolerance of a reference angle". Those three have different imaging needs, different false alarm behaviour and different costs. The third is the sample source: which images acceptance will run on, who supplies them, and where genuine defects come from. With those frozen, the two sides finally share a basis for discussion. For how to break a vague requirement down this way, see motion normativity boundaries.

How to write the metrics so they can be measured

Misses and false alarms must be written separately, because their tolerable ranges are very different. A miss is an alarm that should have fired and did not, and its cost lands directly on defects leaving the station. A false alarm is an alarm that should not have fired, and its cost lands on operator trust and line takt. Compressing the two into one accuracy figure leaves both sides without a basis for judgement, because the same ninety-five percent can mean five percent misses and no false alarms, or zero misses and a fifty percent false alarm rate — the second would never survive on a real line. Once separated, each needs a precondition attached: which sample the figure was computed on. The same system can look near-perfect on good parts and fall apart on borderline parts that are just within tolerance, and borderline parts are exactly what the line cares about most.

Sample selection: good parts always pass and prove nothing

Many projects take whatever came off the line that afternoon, which produces a set made entirely of conforming parts. A system that answers "pass" to everything scores brilliantly. Such an acceptance proves neither that the system catches problems nor that it behaves sensibly on edge cases. A usable set contains at least three kinds of sample: genuine conforming parts, including visually varied ones from different batches and incoming conditions; genuine defects, taken from real production or rework records rather than staged for the camera; and a group of borderline parts that sit just inside the boundary. The third kind is the hardest to assemble and usually needs process and quality to dig through historical rework together, but it is the group that determines whether false alarms will run away after go-live. The set should also be split into a training portion and an acceptance portion — accepting against images the model trained on is an open-book exam.

• Seal the set: fix it at project start and do not substitute samples later, or the two sets of figures will not be comparable.
• Label the set: record which item each image belongs to, which process, and what the manual verdict was. Without this, statistics run on memory.
• Classify the set: label the defect type as well, so acceptance can show whether the system is uniformly weak or blind to one specific defect.
• Size the set: with very few defect samples, no percentage carries statistical meaning. Fewer well-populated metrics beat many hollow ones.

Alarm response: the clause that is always missing

A system that judges correctly has not necessarily solved anything. Between the alarm firing and somebody acting on it sits a shopfloor process that usually appears in no acceptance document. At minimum, three things need naming: who owns the response — operator, team leader or process engineer — what response time is allowed, and what happens if nobody responds in that time. The last one matters most, because without an escalation path an alarm degrades into an unread notification within days. Equally important is the response shape: does the station stop, prompt and continue, or merely log? That single choice decides whether operators route around the system, a pattern covered in alarm fatigue. Whether the corrective action itself gets recorded is also an acceptance item: logging that an alarm fired without logging what was done about it still leaves the process data incomplete.

Accept, then observe

Splitting acceptance into provisional and final stages prevents most disputes. Provisional acceptance happens once installation and commissioning are complete, uses the offline sample set, and tests whether the system judges correctly. Final acceptance happens after a defined period of continuous running — typically four to eight weeks, covering changeovers, different shifts and different incoming batches — and tests whether the system is liveable: false alarm frequency, downtime caused by alarms, and whether any defect escaped without an alarm. Because the two are separate, a project does not deadlock when provisional acceptance passes but the line complains about daily false alarms, and the supplier gets a legitimate window to fine-tune thresholds against real data. What happens if acceptance fails also needs stating in advance: return for re-tuning, expand the sample, or go live on a subset of stations first. Each conclusion carries a different cost. The implementation timeline sets out where these stages fall.