Inference Time Optimisation
Latency on a vision station is not the time the model takes. It is the whole path from the trigger arriving to the result leaving: image transfer, preprocessing, inference, post-processing, publication. Breaking it down is the first step and the one most often skipped, because the intuition that the model is the slow part is frequently wrong. Resizing, colour conversion and normalisation on a general purpose processor can cost as much as the model itself.
Key takeaways
- Measure before optimising: profile acquisition, preprocessing, inference and post-processing separately.
- Input resolution is the strongest lever and the one that most directly costs detection capability.
- Quantisation saves time and memory; verify accuracy on a fixed sample set, not on impressions.
- Preprocessing and post-processing frequently outweigh inference, so do not optimise only the model.
The reflex when something is too slow is to reach for a smaller model. That is often the wrong first move, and it is expensive: a smaller model costs accuracy, and if the bottleneck was elsewhere, it costs accuracy for nothing. The correct order is to instrument the pipeline — timestamp the trigger, the end of acquisition, the end of preprocessing, the end of inference, the end of post-processing, the publication of the result — and run it for a few hundred cycles. What comes back is usually a surprise.
Where the time actually goes
In practice the model is often a minor part of the total. Moving an image from the camera into host memory takes time, particularly with several cameras running. Preprocessing — resizing, colour space conversion, normalisation, tensor layout — is done pixel by pixel, and at high resolution on a general purpose processor it can match or exceed a small model. Post-processing, including thresholding, contour extraction and connected component analysis, grows with resolution and can be substantial. If only the model is optimised, none of that improves.
The other common finding is time spent waiting rather than computing: waiting for the frame to finish transferring, waiting for a previous result, waiting for a record to be written. This does not appear in a single frame benchmark, because the compute in the previous stage hides it. It appears when several hundred cycles are measured end to end, which is why the metric that matters is the distribution of end-to-end latency, not the mean duration of the inference call.
| Measure | Typical gain | Price |
|---|---|---|
| Reduce input resolution | Scales roughly with pixel count; the largest single lever | Small defects may vanish; must be verified against the sample set |
| Choose a lighter architecture | Substantial | Accuracy may drop; requires retraining, not just swapping a file |
| Quantise (float to integer) | Time and memory, best on dedicated accelerators | Verify accuracy on a fixed sample set |
| Add an accelerator | Large | Cost, heat and lead time, plus a re-integration effort |
| Optimise preprocessing and post-processing | Depends on the current implementation; can be large | Work in the imaging library, but often better value than a new model |
| Cut cross-process and network transfers | Moderate | An architectural change that affects maintainability |
Resolution first, and its trap
Going from 1024 to 512 inputs removes three quarters of the pixels and cuts inference roughly in proportion. Nothing else returns as much for as little work. But it also changes what can be detected: a defect occupying a handful of pixels at full resolution may be one pixel, or merged into the background, after downsampling. So resolution reduction must be validated together with the detection rate, using the fixed sample set, with the specific question of which defects stopped being found. It is a change to the criteria, not just to the implementation.
A more careful approach is to work at two resolutions: a fast low resolution pass over the whole field to reject the clear majority, then a high resolution look only at the regions the first pass flagged. The compute goes where it is needed rather than everywhere. The cost is added logic and a need for a reliable coarse screen; but on a tight cycle where defects can appear anywhere, it usually beats a uniform reduction in resolution.
Common questions about inference latency
Is a GPU always faster?
No. A GPU wins on large batches and high throughput, but on single frames with a low latency requirement the transfer and launch overheads can consume the gain. Vision stations usually have modest compute demands, and a dedicated accelerator or a processor with inference instructions can serve better. The basis for deciding is measured end-to-end latency, not a specification sheet.
What if quantisation costs accuracy?
First establish whether the loss is inside the tolerance. Quantisation often shifts results less than the natural variation between samples, which is acceptable. If it genuinely exceeds the tolerance, consider quantising only part of the network or using a method with less loss, rather than abandoning the idea. Whichever route is taken, the comparison must be made on the same fixed sample set, which then becomes the ongoing monitoring baseline.
How do several cameras sharing one processor change the sum?
Budget for the worst case — the moment when several cameras have frames pending at once. If the triggers are staggered, the average load may be comfortable, but the peak decides whether frames are dropped. Where the process allows it, deliberately offsetting the trigger times so that stations do not image in the same millisecond is a cheap and effective way to flatten the peak.
Is your model missing the cycle?
Tell us the resolution, the model and the takt time. We will say which stage is likely to be the bottleneck and which fix is worth trying first.
Related articles
-
Alarm Fatigue: Why the Floor Turns the System Off
Many visual SOP systems are not beaten by technical problems — they are muted by…
Read more → -
Visual SOP Camera Selection and Mounting
Camera selection for visual SOP is not the same as for product inspection. The s…
Read more → -
Vision Monitoring at the Semiconductor Die Bonder
Semiconductor packaging stations are characterised by high precision, fast cycle…
Read more → -
Wrong-Assembly Alarms Without the False Alarms
The most common reason a visual SOP project fails is not poor recognition — it i…
Read more →
Need specific advice for your production line?
Our engineers can share practical approaches to process error prevention and SOP compliance inspection for your scenario.
Request a free trial plan