Defining False-Alarm and Miss Rates Without Fooling Yourself

False-alarm and miss rates usually look better than they are because of the denominator, not the numerator. Write the formulas out before anything else: false-alarm rate is the count of events the system flagged as anomalous that were actually normal, divided by all events the system flagged; miss rate is the number of anomalies missed, divided by the number that actually occurred. Which denominator you pick, whether you count per station or across the line, and whether idle time is included, will all change the number for the same system — so agree the conventions before acceptance, not after.

Write the formulas and name the denominator

The denominator decides the conclusion. Change the false-alarm denominator to all inspections or all units and the figure looks tiny, because most judgements never raise an alert. Change it to all alerts and it looks alarming. Neither is wrong; they answer different questions, so say which one you are using before comparing. Any report that states a false-alarm rate of 0.5% should say within a line what the denominator was, over what period, and across which stations. When two reports that both claim to pass cannot be compared, it is usually the convention, not the data. Lowering false alarms is an engineering job — the levers are in reducing false alarms. The job here is measuring the result honestly.

Per station, or across the line?

• Line-wide average. Looks good and hides the worst station. On a line of twenty stations, one noisy station can vanish into the average.
• Per station, then aggregate. Locates the problem, at the cost of more data and more reporting — but it is the only convention that drives improvement.
• Split by shift. Lighting, staffing and cycle time differ between shifts; pooling them dilutes each. Read them separately.
• Split by part number. Essential on mixed-model lines, or a systematic false alarm on one part number is buried under normal data from the others. Getting all these views out assumes the system records station, shift and part number — see data traceability.

Ways you end up fooling yourself

• Counting only confirmed false alarms. Alerts an operator dismissed without confirming never enter the numerator, so the rate drops — and those are the alerts that most damage trust. A manual sampling check is a useful cross-reference; see manual inspection versus visual SOP.
• Dropping idle and stoppage time. Counting time when the line was not running inflates the denominator and dilutes the error rate with periods that produced nothing.
• Measuring misses only via downstream catches. A miss that is never discovered never enters the numerator. Miss rate is therefore optimistic by construction and can only be bounded, not measured — say so plainly.
• Reporting the average, never the worst station. Averages hide tail risk, and the tail is where complaints come from. A good line average still hides a station that keeps skipping a step.
• Changing the convention between acceptance and production. One formula at acceptance and another afterwards makes the numbers incomparable, and the argument never ends. Once written into the acceptance document, a convention change should go through the same sign-off.

Writing acceptance criteria you can actually judge