A vision pilot usually fails long before anyone looks at a model. It fails at a camera mounted three metres too high, on a shop floor where the light changes at 4pm, against labels that three different people interpreted three different ways.
The pilot was run in conditions production will never repeat
Pilots get the good camera, the clean corner and the cooperative shift. Production gets the dusty lens, the backlit doorway and the night shift. If your pilot footage was collected over two clear afternoons, you have measured the easy case.
Collect across the full range before committing: different shifts, weather, seasons and the days when things go wrong. A model that holds up on your worst hour is worth more than one that excels on your best.
Camera placement is doing most of the work
Angle, height, distance and lens choice determine what is even visible. No amount of training recovers detail that was never captured. Before anyone buys a GPU, walk the site and check that the thing you want detected is actually resolvable in the frame.
- Can a person identify the target in a single still from that camera?
- Is the target visible at the worst light of the day, not the best?
- Does anything routinely block the view — a pallet, a vehicle, an open door?
- Does the camera stay in position, or does it drift after a wash-down?
Labels disagree more than models do
Ask three people to label the same hundred frames and compare. Where they disagree is exactly where your model will be unreliable, and no architecture change fixes an inconsistent definition. Write the definition down, test it on a sample, then label.
Nobody agreed what success meant
"Improve quality" is not a target. "Catch surface defects above 2mm on line 3 with fewer than five false alarms per shift" is. The second can be tested; the first will be argued about at the end of the pilot.
If a pilot cannot state, in one sentence, what number it is trying to move and what false-alarm rate is acceptable, it is not a pilot. It is a demonstration.
There was no plan for the day after
Production means someone owns retraining, someone gets the alerts, and something happens when the model is wrong. If the answer to "who watches this on Sunday night" is nobody, the deployment will quietly stop being used.
A checklist worth running first
- One sentence stating the metric and the acceptable false-alarm rate
- Footage covering the worst conditions, not the best
- A written label definition that two people apply the same way
- A named owner for alerts, and a defined action when one fires
- An agreed decision point where the pilot is stopped if it misses the target
None of this is glamorous, and all of it decides the outcome. Model selection is the last ten percent of the problem, not the first.