Why a Solution That Worked on the Pilot Line Fails in Production

A pilot line is not a smaller version of the production line. It is usually cleaner, lit better, run more slowly, staffed by more experienced people, and watched closely by everyone involved — because it is a pilot. Every one of those differences flatters the system, and when it later underperforms on the real line the usual explanation offered is that production is simply messier. That is true but unhelpful, because the gaps were visible throughout the pilot to anyone looking for them. The differences are enumerable, which means they are testable before committing to a rollout rather than discovered afterwards.

The seven differences that matter

The fifth row does the most damage and is the least discussed. During a pilot there is usually an engineer within reach, who can nudge a threshold, restart a process, or interpret an ambiguous verdict. In production there is nobody, and the system has to handle every situation on its own, including the ones nobody anticipated. Running at least one full shift with no engineer present during the pilot is the single most informative test available, and it regularly surfaces problems that never appeared while somebody was standing there. The same logic applies to records: the pilot team often collects data by hand alongside the system, which masks the fact that the system's own records are incomplete.

Testing without building two systems

You do not need to deploy a second system to test these things; you need to vary the conditions of the one you have. Three tests recover most of the value. First, run at real takt for a full shift, including a changeover — this exercises timing, operator pressure and the transition between products at the same time. Second, run with unselected incoming parts and operators who were not part of the pilot, which tests whether the system has learned the pilot's particular conditions rather than the process. Third, leave it alone for a shift and read the records afterwards, which tells you what the system does when nobody is helping. Each of these can be done in the last weeks of a pilot at modest cost, and each has a habit of producing an uncomfortable finding. The rollout path itself is covered in from pilot to full rollout.

Setting a rollout threshold that means something

A threshold worth having has four properties: it is numeric, it was agreed before the pilot, it is measured under production-like conditions, and it has a consequence attached. "The pilot went well" has none of them. Something like "at production takt across a full shift, with unselected parts, fewer than five false alarms per shift and no misses on the standard sample set for two consecutive weeks, then we roll out" is a threshold — and if it is not met, the correct response is to fix the cause rather than to proceed on optimism. Where the pilot genuinely is limited, say so explicitly and set a smaller first rollout, at two or three stations rather than the whole line, with the same numeric test applied again.