cd ~/
MLOpsProductionInfrastructureDeep LearningMeta

Production ML Systems: Beyond Model Training

What it takes to run ML in production at scale - monitoring, versioning, deployment, and all the things that aren't training.

Evyatar Bluzer
4 min read

Training a good model is maybe 20% of production ML. The other 80% is data pipelines, serving, monitoring, and the unglamorous plumbing that papers skip.

The Full ML System

Data Pipeline → Feature Engineering → Training → Evaluation →
Deployment → Monitoring → Feedback → Data Pipeline

Most ML research focuses on the training box. In production, every other box has to work too, and any one of them can take the model down.

Data Pipeline

Production data does not behave like a benchmark dataset. It arrives continuously, its quality varies with upstream bugs and schema changes, it is too large to fit in memory, and freshness matters because stale training data means a stale model. Our system handles this with streaming ingestion, automated quality checks that monitor distributions, incremental dataset creation, and lineage tracking so we always know which data trained which model. Six months in, the data pipeline was the bottleneck; most of this section is the response.

Feature Engineering

VPS features come in three kinds: image features (learned descriptors), geometric features (camera poses), and context features (time, weather, device type). The production concern that dominates everything else is that feature computation must be identical in training and serving - hence feature stores for consistency, and feature versioning to keep V1 and V2 features from mixing. Training/serving skew is a top cause of production issues.

Training Infrastructure

At Meta scale this means distributed training across hundreds of GPUs, hyperparameter search across thousands of configs, automatic retraining on new data, and experiment tracking to compare it all. We retrain VPS models weekly with the latest data.

What makes a model good enough to ship?

Research evaluation asks about benchmark accuracy. Production evaluation asks whether the model will improve the product, which pulls in latency (is it fast enough), memory (does it fit), fairness (does it work equally across regions), and robustness (does it handle edge cases). Eval gates on all of these keep bad models from shipping.

Deployment

Rollout is gradual by design, the same dogfood-then-small-percentage sequence the learned features went through. Shadow mode comes first, with the new model running alongside the old one and results compared. Then a canary at 1% of traffic, then incremental steps through 10%, 50%, and 100% over days, with a holdback group kept on the old model for comparison. Any regression triggers automatic rollback.

Model rollout ladderSix boxes in a row connected by arrows from eval gates through shadow mode, canary, 10 percent, 50 percent and 100 percent; highlighted stubs from the last five drop to a collector line, and one highlighted arrow from it points down into a muted old-model box spanning the row below. steps advance over days Eval gates latency, memory, fairness, robustness Shadow mode beside the old model, results compared Canary 1% of traffic 10% 50% 100% any regression at any step: automatic rollback Old model runs beside the new model in shadow mode; a holdback group is kept on it for comparison
The rollout ladder for a new VPS model: eval gates, shadow mode, a 1% canary, then 10%, 50% and 100% over days, while the old model keeps serving underneath with a holdback group, and any regression at any step rolls back onto it automatically.

Monitoring

Nobody can manually check every prediction in production, so the monitors do it: prediction distribution monitoring (is the output changing), error rate monitoring (are we failing more), latency monitoring (are we slowing down), and data drift detection (is the input distribution changing). Alerts fire before users complain, which is the whole point of having them.

The Feedback Loop

User feedback, implicit and explicit, feeds future models. Localization success and failure become labels for hard examples, user corrections become ground truth for retraining, and engagement metrics signal what users actually value. Closing this loop is what makes continuous improvement real.

Lessons Learned

Invest in infrastructure early; retraining, deployment, and monitoring save more time than model tweaks ever will. Keep everything reproducible, because you will need to recreate a historical model at the worst possible moment. Monitor everything, since you cannot fix what you cannot see. And plan for failure: models will misbehave, and fast detection plus fast rollback is what limits the damage.

Comments