The notebook is the natural habitat of the data scientist. It is interactive, forgiving, and designed for exploration. Production is none of these things. It is scheduled, unforgiving, and designed for reliability. Bridging these two worlds requires more than wrapping a model in a REST API—it requires a fundamentally different engineering discipline.
What kills production ML. The top causes of production ML failure are not model accuracy problems. They are: data schema drift (the production pipeline receives data in a format the model never saw during training), feature store inconsistency (the features computed at training time differ from those computed at inference time), and monitoring blindness (the team cannot tell whether the model is degrading because there are no baseline metrics to compare against).
The pipeline is the product. In production ML, the model is one component of a larger system. The system includes data ingestion, validation, feature computation, model serving, prediction logging, monitoring, alerting, and retraining triggers. Each of these components must be versioned, tested, and deployed independently. The model is the kernel; the pipeline is the operating system.
Government deployment requirements. ML systems deployed in public-sector contexts face additional constraints. EU AI Act high-risk system requirements mandate human oversight, explainability, and audit trails. SAM.gov contract vehicles for AI services require similar provisions under the NIST AI RMF. These are not optional features—they are deployment prerequisites that must be built into the pipeline from the start.
The retraining problem. Models degrade. Distribution shifts. Edge cases emerge. The production system must handle retraining without downtime, without data leakage between training and evaluation sets, and without the silent errors that occur when a retrained model produces outputs that no one validates before deployment. Automated retraining without automated validation is a production incident waiting to happen.
In our practice, we approach production ML the same way we approach any critical system: define the failure modes first, then build the pipeline to detect and recover from each one. The model comes last—not because it is unimportant, but because a reliable pipeline with a mediocre model outperforms an unreliable pipeline with a perfect model.