A model with high accuracy in the lab can perform worse than expected once it is deployed. The main reason is the gap between the training data and real-world data. A different sensor, a different season or different user behaviour changes the inputs the model sees.
Before deployment, define three things: the metric used to judge success and the acceptance threshold; a mechanism that monitors whether the input distribution drifts over time; and the conditions under which the model is retrained, and who approves the new version. If the need is not real, a rule-based solution is often more robust and easier to explain in a first release. If the model's complexity makes the system hard to maintain, it adds no value.
Planning a similar project?
We can talk through how the topic in this article applies to your product.
