ML Production Readiness Checklist: 50 Things to Verify Before You Ship
A ML Production Readiness checklist earns its place on a page only if someone actually runs it before shipping, not after an incident review. This one comes with code attached, because a checklist nobody can verify against their own repo is just a list of good intentions with a nicer font.
Twenty of the 50 items below are checked automatically by a static scanner built specifically for this article. It runs against two small sample projects: one built to pass, one built to look like a typical pre-checklist repo. The other 30 items cannot be seen from a file system scan at all. They need a human to sign off, and this article says so plainly instead of pretending a script can cover everything.
TL;DR
- A dependency-free static scanner (ast and pathlib only, same approach as the Article 14 auditor) checks 20 of the 50 checklist items directly against a project’s files.
- Run against a sample repo built to this checklist’s standard, it passed 19 of 20 automated checks (95%). The single miss was a deliberately unpinned dependency, left in on purpose, because an honest audit should still find something.
- Run against a sample repo built the way a rushed pre-launch project actually looks (unpinned base image, no tests, one hardcoded model file, a one-line README), it passed 1 of 20 (5%).
- The remaining 30 items, sign-off, on-call rotation, compliance review, a rollback runbook that has actually been rehearsed, cannot be automated. This article names them anyway instead of quietly dropping them because they are hard to check.
- Every number below came from running the code. The full scanner, both sample repos, and the exact commands are in the companion repository, linked in Resources & References.
Want to run the scanner yourself? The full checker, both sample repos, and the exact commands used for every number below are in the companion repository: https://github.com/Emmimal/ml-production-readiness-checklist
Why Most ML Systems Ship Before They Are Ready (And the Cost)
A model that scores well in a notebook and a model that is ready to serve real traffic are two different claims. The gap between them is everything the previous 14 articles in this series covered: a pipeline that survives a retrain (Article 3), a registry that lets you roll back cleanly (Article 4), monitoring that catches drift instead of watching only uptime (Articles 9 and 10), a rollout that does not bet the whole system on day one (Article 11), latency that holds under real load (Article 12), an A/B test that actually proves the new model is better (Article 13), and debt that gets paid down instead of accumulating quietly underneath a stable-looking accuracy number (Article 14).
None of that shows up in a validation score. That is the real cost of shipping before ready. It is rarely a dramatic outage on day one. It is a slow accumulation of the exact failure modes those 14 articles described, each one invisible until the moment it is not.
The scanner built for this article turns that gap into two numbers instead of a feeling. Below is what it actually found, laid out as a section-by-section comparison between the two sample repos it scanned.
That single [#...] in Section 4 for the legacy repo is not a mistake. It passed the “no undeclared consumers” check for an unglamorous reason: the repo only has two Python files, so there are not enough files for the same artifact string to get read twice. The check is honest about what it can and cannot see, and this article tries to be too.
How to Use This Checklist (Team Workflow)
This is not a document to read once and forget. It works best as a gate: nobody merges a model to production without walking every item across all 5 sections, either verified automatically by the scanner or signed off by a named person.
A workflow that has held up in practice:
- Run the scanner against the repo. It covers items 1, 4, 6, 10, 11, 12, 14, 15, 23, 24, 25, 29, 31, 33, 38, 40, 41, 42, 43, and 44 without any human input.
- Walk the remaining 30 items as a team, out loud, in a short meeting. Each one gets a named owner, not a checkbox nobody claims.
- Treat any “no” as a blocker, not a note for later. A checklist that allows exceptions by default has already stopped being a checklist.
- Re-run the scanner on every release, not only the first one. Pinning rot and configuration drift happen quietly, the same way the technical debt in Article 14 does.
When a team disagrees about which failed item to fix first, the scoring framework from Article 14 (severity times blast radius, divided by effort) is the right tool. Not every “no” deserves the same urgency, and that framework exists specifically to rank them by cost instead of by whoever raises their voice first in the room.
Section 1: Data Quality and Pipeline Checks (Items 1–10)
1. Input schema is validated before training and before serving. [Automated check] A schema check catches a renamed column or a silently changed type before it reaches the model, not after. The scanner looks for pandera, pydantic, or an equivalent validation library actually being used in the code, not merely imported and forgotten.
2. Missing-value handling strategy is defined and documented per feature. Silent imputation with a default value is still a decision. It should be a deliberate one, written down somewhere, not whatever .fillna(0) happened to do during exploration and never got revisited.
3. Train/test split has no leakage. No future information baked into a feature, no duplicate rows split across train and test, no target-derived column that would not exist at prediction time. This single failure mode is the most common reason a validation score looks better than production ever will.
4. Random seed is fixed everywhere randomness enters. [Automated check] Split, shuffle, weight initialization, all of it. Every number in this article, and every one in this series before it, ran with seed=42. The scanner checks the target project for the same discipline: random_state=, seed=, or an equivalent fixed-seed pattern.
5. Class balance and label distribution have been checked and are understood. Not necessarily corrected. Understood. A model trained on a 99:1 class split behaves very differently from one trained on a balanced set, and the team should know which one it has before either ships.
6. Training data is versioned. [Automated check] Any model in production should trace back to an exact data snapshot, not “whatever was in the table that week.” The scanner looks for DVC tracking files or an equivalent data-version pointer in the repo.
7. Outlier and anomaly handling is a deliberate decision, not a silent one. Dropped rows, capped values, flagged-and-kept: any of the three is defensible on its own. Not knowing which one the pipeline actually does is not.
8. Pipeline is idempotent. Running the same input through it twice should produce the same output both times. A pipeline that depends on wall-clock time, an unseeded shuffle, or an external API’s mutable state fails this quietly, and it usually only surfaces at the worst possible moment: mid-retrain.
9. Feature definitions live behind one declared interface. Multiple scripts independently reading the same raw file is exactly the undeclared-consumer pattern documented in Article 14. The automated version of this check runs later, in Section 4, item 40, because that is where the audit tool built to catch it actually lives.
10. Automated data-quality tests run before training, not only at ingestion. [Automated check] The scanner looks for a test_data*.py file. Its presence does not guarantee the tests inside it are good. Its absence guarantees nobody is checking this at all.
Section 2: Model Validation and Testing (Items 11–20)
11. Model beats a trivial baseline on the same test set. [Automated check] A majority-class classifier or a mean predictor, whichever fits the problem. If the real model cannot clear that bar with room to spare, nothing downstream matters yet. The scanner looks for a DummyClassifier, DummyRegressor, or an equivalent baseline that is actually scored, not just imported for show. On the sample project built for this article, that comparison is real: baseline accuracy came out at 0.513, cross-validation mean at 0.997, and held-out test accuracy at 0.997, all from one run of train.py with seed=42.
12. Validation uses cross-validation or multiple splits. [Automated check] A single train/test split can flatter or punish a model by luck alone. The scanner checks for cross_val_score, KFold, or a similar pattern in the training code, and the 0.997 cross-validation mean quoted above came from exactly that pattern, five folds, shuffled, seeded.
13. Test set was never touched during hyperparameter tuning. This is the hardest leak to catch after the fact, because the code that causes it usually looks completely reasonable in review. The only real defense is a validation set that hyperparameter search is allowed to see, and a test set that it never does.
14. A test suite exists for the model and feature code itself. [Automated check] Not the training script running once and printing a number. Actual tests, the kind a characterization test from Article 14 would build on top of. The scanner checks for a test_model*.py file, and running that suite against the sample project produced a clean 2 passed from pytest.
15. A model card documents intended use, training data, and known limitations. [Automated check] One page is enough. What it should never be is missing. The scanner looks for a model_card.md file in the repo root.
16. Performance has been checked across key segments, not only in aggregate. An aggregate accuracy number can hide a model that fails badly for one segment of users while performing fine everywhere else. Whether that segment is defined by account tenure, geography, or product tier depends on the domain, but someone on the team should have actually looked.
17. The model has been tested against edge-case and malformed inputs. Empty strings, out-of-range values, unexpected nulls, whatever the serving API’s schema check does not already reject upstream. This is exactly where an entangled model, the CACE effect from Article 14, tends to surface unexpected behavior first.
18. Inference latency and memory footprint are benchmarked at expected load. Article 12 covers the full profiling methodology in depth. The number that matters here is not “how fast on my laptop” but “how fast under the traffic pattern this will actually see in production.”
19. A rollout plan exists: shadow, canary, or A/B, not a straight cutover. Articles 11 and 13 cover the mechanics of each. The plan itself just needs to exist in writing before launch day, not get improvised during it.
20. A previous model version is tagged and ready to serve if this one needs rolling back. Article 4 covers the registry mechanics behind this. The check here is simpler: can someone name the exact version they would roll back to, right now, without searching for it first.
Section 3: Deployment and Infrastructure (Items 21–30)
21. Model artifact is saved to a versioned registry, not a loose file on someone’s laptop. The registry pattern from Article 4 exists specifically so “which model is actually serving traffic right now” has one answer, not three competing guesses from three different people.
22. Serving API validates request payloads and rejects malformed input before it reaches the model. The schema check from item 1 protects training. This is the same idea, applied to protect inference.
23. A health or readiness endpoint exists, separate from the prediction endpoint. [Automated check] Load balancers and orchestrators need something to poll that does not also run a model inference on every check. The scanner looks for a /health, /healthz, or /ready route defined in the serving code.
24. Base image and dependencies are pinned to specific versions, not latest. [Automated check] A latest tag means the exact same Dockerfile can build a different image next week, silently, with no diff to review. The scanner flags any FROM image:latest line it finds.
25. Every dependency in requirements.txt is version-pinned. [Automated check] Same failure mode as item 24, one layer higher in the stack. This is the one item the “ready” sample repo built for this article actually failed: a single dependency, requests, left unpinned on purpose. An honest scanner should find something, and this is exactly what it found, at 19 of 20 rather than a suspiciously perfect 20 of 20.
26. Concurrency limits and autoscaling policy are configured for expected traffic. Article 12’s load-testing methodology is what tells a team what “expected traffic” actually means in numbers. This item just checks that the infrastructure config reflects that number instead of a guess made once and never revisited.
27. Timeouts and retries are set for any downstream call the serving path depends on. A feature-store lookup or a third-party enrichment call without a timeout turns one slow dependency into a full outage for every model that calls it.
28. Secrets and credentials come from environment variables or a secret manager, never hardcoded. This barely needs a citation. It stays on the list because it still gets missed, most often in the serving script that started life as a quick notebook experiment.
29. CI pipeline runs the test suite and blocks deploy on failure. [Automated check] Article 2 covers the full CI/CD setup for deployment. The scanner just confirms a workflow file exists and actually references running tests, not merely building and pushing an image.
30. Load has been tested at expected peak traffic, not only average. Average load tells a team almost nothing about the moment that actually breaks a system. Peak, ideally with some margin above it, is the number that matters here.
Section 4: Monitoring, Alerting, and Observability (Items 31–40)
31. Every prediction, with its inputs, is logged to a persistent store. [Automated check] Without this, drift detection and post-hoc debugging both have nothing to work from later. The scanner checks for logging calls inside the actual serving path, not a stray print statement left over from development.
32. A ground-truth or delayed-label capture pipeline exists. Accuracy in production cannot be measured without eventually knowing what actually happened. For a fraud model that might be a chargeback signal weeks later; for a churn model, whether the account actually churned. Whatever the domain’s version of that signal is, it needs a defined capture path, not a hope that someone remembers to check later.
33. Data drift detection is running against live traffic, not only checked once before ship. [Automated check] Article 9 covers the KS test, PSI, and MMD approaches in full. The scanner checks for the same statistical pattern actually present in the codebase: a ks_2samp call or an equivalent drift-detection import. Run against the sample project’s own monitor.py, that exact check returned drift_detected=True with a p-value of 0.0000 on a deliberately shifted distribution, which is the expected result for a detector working correctly.
34. A monitoring dashboard shows drift and performance, not only uptime. Article 10 builds this dashboard in full. The check here is simply that it exists, and that someone besides its original builder knows how to read it.
35. Alert thresholds are set on drift and accuracy, not only on server errors. This is the exact gap Article 14’s entanglement finding exposed: a monitoring setup that watches only accuracy can miss a real change happening underneath a nearly-unchanged number. Alerting needs to watch more than whether the server is up.
36. An on-call runbook exists for model-specific incidents, not only infra incidents. A 500 error and a model quietly returning wrong predictions behind a 200 status code are very different failures. Only one of them shows up in a standard infrastructure runbook.
37. Inference latency and throughput are monitored continuously, not benchmarked once. Article 12’s benchmark captures the state on the day it ran. Production conditions change after that day, and the monitoring needs to keep watching once the benchmark report is filed away.
38. Retraining triggers are implemented and tied to drift, time, or performance thresholds. [Automated check] Article 3 covers the retraining pipeline this feeds into. The scanner checks for an actual trigger function, something like should_retrain(), rather than a comment saying retraining should probably happen at some point.
39. Inference cost is tracked, not discovered on the first cloud bill. Autoscaling that works perfectly from a latency standpoint can still be a budget surprise if nobody is watching the cost side of the same dashboard that watches latency.
40. A static audit for undeclared consumers and configuration debt has been run against the codebase. [Automated check] This reuses the exact AST-based auditor built for Article 14. The scanner walks every .py file for shared artifact strings referenced by more than one file, the same undeclared-consumer pattern flagged back in item 9, now checked mechanically instead of by memory or by hoping someone notices.
Section 5: Documentation, Handoff, and Team Readiness (Items 41–50)
41. README documents setup and usage clearly enough for someone new to run it. [Automated check] The scanner checks for both a setup section and a usage section, not just a project name and a one-line description at the top.
42. Architecture is documented, even as a simple diagram. [Automated check] It does not need to be elaborate. A short markdown file showing how data flows from feature store to serving to monitoring is enough, and the scanner just confirms a file like it exists in the repo.
43. Model registry entry is complete: owner, version, and training data pointer. [Automated check] Article 4 covers the registry pattern in depth. The scanner checks that a registry file exists at all. Whether it is actually filled in correctly and kept current is still a human check.
44. A rollback runbook exists and has actually been tested, not only written. [Automated check] The scanner can confirm the file is there. It cannot confirm anyone has ever run the steps inside it. That gap is exactly why this item also belongs in the manual review, not only the automated pass, and why “exists” and “tested” are two different claims that this checklist deliberately does not conflate.
45. On-call rotation is assigned and knows this model exists. A model nobody on the rotation has heard of is a model nobody can actually respond to at 2 a.m.
46. Stakeholders have signed off on acceptable failure modes and their business impact. Not every wrong prediction costs the same. A recommendation model showing a slightly worse suggestion and a credit model wrongly declining a loan are not the same conversation, and whoever owns that business impact should have had the conversation before launch, not after.
47. Compliance or privacy review is complete if the model touches regulated or personal data. This is a gate, not a suggestion, in any domain where it applies. Skipping it does not remove the requirement. It just moves the conversation to after something has already gone wrong.
48. A post-launch review is scheduled, not left to happen only if something breaks. The absence of an incident is not the same as confirmation that everything is working as intended. A scheduled review is what catches the quiet failures the alerting missed.
49. At least one other engineer has been walked through the system. A model that only one person understands is a single point of knowledge failure, and that person taking a vacation should never be a production risk.
50. A deprecation or sunset plan exists for when this model eventually gets replaced. Every model in this series eventually gets replaced by a better one. Deciding now how that transition happens, rather than improvising it later under pressure, is what keeps the next launch from repeating all 49 items above from scratch.
![Terminal-style grid titled "ALL 50 ITEMS, AT A GLANCE" displaying a 50-item MLOps deployment checklist. The items are split across 5 sections, with 20 items marked in green as "[A]" (automated by scanner) and 30 items marked in yellow as "[M]" (manual sign-off required).](https://emitechlogic.com/wp-content/uploads/2026/08/ALL-50-ITEMS-AT-A-GLANCE-1024x485.png)
Downloadable PDF Version and GitHub Template
The full 50-item list, formatted as a one-page printable checklist, plus the scanner and both sample repos used to generate every number in this article, live in the companion repository: https://github.com/Emmimal/ml-production-readiness-checklist Clone it, point the scanner at an actual repo, and see which of the 20 automated items pass before the team sits down to walk the remaining 30 by hand.
ML Production Readiness Checklist FAQ
Can a checklist like this actually be automated?
Part of it. This article automates 20 of the 50 items with a dependency-free static scanner, the same ast-based approach used for the Article 14 auditor. The other 30 are judgment calls, sign-off, review, and rehearsal, that no file scan will ever be able to see.
Is a 95% score on the automated checks enough to ship?
No. The 20 automated items are necessary, not sufficient. A project can pass every one of them and still fail on stakeholder sign-off, compliance review, or a rollback runbook nobody has actually rehearsed. The score is a floor, not a finish line.
How is this different from the Article 14 technical debt audit?
Article 14’s auditor looks backward, at debt already accumulated in a system that is already live. This checklist looks forward, at whether a system is ready to become live in the first place. The undeclared-consumer check, item 40 here, is literally the same code, reused because the exact same failure mode threatens both a running system and one that has not shipped yet.
What is the ML Test Score, and how does it relate to this checklist?
Google’s 2017 paper by Breck, Cai, Nielsen, Salib, and Sculley proposed a 28-test scoring rubric for ML production readiness, organized around data tests, model tests, infrastructure tests, and monitoring tests: the same four broad areas this checklist’s 5 sections cover in more implementation detail [1]. This checklist is a practical, code-verified companion to that idea, not a replacement for reading the original paper.
Why does the “ready” sample project fail one check instead of passing all 20?
Because a spotless pass right after building a project specifically to demonstrate a checklist would not prove anything. It would just mean the scanner agrees with whoever built the fixture. Article 14 made the same point about its own before/after refactor: it still found 3 leftover magic numbers after the fix, and that is what made the audit worth trusting. This article’s one deliberate miss, an unpinned
requestsdependency, does the same job here.
Key Takeaways
A checklist that cannot be run against a real repo is a wish list with good intentions. Twenty of these 50 items can be checked by a script, and running that script against two real sample projects showed exactly what the gap looks like: 19 of 20 passed on a project built to this standard, 1 of 20 passed on a project built the way most pre-launch repos actually look.
The remaining 30 items need a name attached to each one, not a checkbox. Sign-off, rehearsal, and review are the parts a static scanner will never be able to see, and pretending otherwise is how “the runbook exists” quietly turns into “the runbook has never once been tested.”
This closes the Production ML Engineering series. Article 1 laid out the 5 pillars of a production-ready ML system. This checklist is the gate that confirms all 5 actually got built, not just planned on a whiteboard.
What’s Next
This is the final article in the 15-part Production ML Engineering series. For the complete series map, from deployment (Article 2) and retraining (Article 3) through monitoring (Articles 9 and 10), see the Production ML Engineering guide. The four articles this checklist draws on most directly: Shadow Deployment and Canary Testing for Machine Learning Models (Article 11), How to Debug ML Inference Latency and Throughput Issues (Article 12), How to A/B Test Machine Learning Models the Right Way (Article 13), and ML Technical Debt: How to Identify, Measure, and Pay It Down (Article 14), whose static auditor this checklist’s scanner reuses directly for item 40.
Resources & References
[1] Breck, E., Cai, S., Nielsen, E., Salib, M., & Sculley, D. (2017). The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction. 2017 IEEE International Conference on Big Data, 1123-1132. https://doi.org/10.1109/BigData.2017.8258038 (open-access version: https://research.google/pubs/the-ml-test-score-a-rubric-for-ml-production-readiness-and-technical-debt-reduction/)
[2] Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F., & Dennison, D. (2015). Hidden Technical Debt in Machine Learning Systems. Advances in Neural Information Processing Systems 28 (NeurIPS 2015), 2503-2511. https://proceedings.neurips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html
[3] Python Software Foundation. ast (Abstract Syntax Trees), Python standard library documentation. Used to build the readiness scanner with no third-party dependency. https://docs.python.org/3/library/ast.html
[4] scikit-learn developers. sklearn.dummy.DummyClassifier documentation, used for the baseline comparison in item 11. https://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyClassifier.html
[5] SciPy developers. scipy.stats.ks_2samp documentation, used for the drift check in item 33. https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ks_2samp.html
[6] Emmimal P Alexander. (2026). ML Technical Debt: How to Identify, Measure, and Pay It Down. EmiTechLogic, Article 14 of the Production ML Engineering series. https://emitechlogic.com/ml-technical-debt-how-to-identify-measure-and-pay-it-down/
[7] Emmimal P Alexander. (2026). Shadow Deployment and Canary Testing for Machine Learning Models: A Practical Guide. EmiTechLogic, Article 11 of the Production ML Engineering series. https://emitechlogic.com/shadow-deployment-and-canary-testing-for-machine-learning-models-a-practical-guide/
[8] Emmimal P Alexander. (2026). How to Debug ML Inference Latency and Throughput Issues. EmiTechLogic, Article 12 of the Production ML Engineering series. https://emitechlogic.com/debug-ml-inference-latency/
[9] Emmimal P Alexander. (2026). How to A/B Test Machine Learning Models the Right Way. EmiTechLogic, Article 13 of the Production ML Engineering series. https://emitechlogic.com/how-to-a-b-test-machine-learning-models-the-right-way/
Full code for the scanner, both sample repos in their ready and legacy states, and the exact commands used to reproduce every number in this article are available in the companion repository: https://github.com/Emmimal/ml-production-readiness-checklist
Disclosure
All code referenced in this article is original work, built and executed specifically for this piece: the 20-check static scanner, both sample repos, and the training, drift-check, and test-suite runs against the “ready” sample project. There are no pretrained weights and no external or scraped datasets involved anywhere in it.
Every number quoted above is the direct output of running that code, not an estimate: the 19/20 and 1/20 automated-check scores, the 0.513 baseline / 0.997 cross-validation / 0.997 test-accuracy scores from the sample project’s training run, the drift p-value of 0.0000, and the 2 passed test-suite result. All of it ran on Python 3.12.3, scikit-learn 1.8.0, pandas, scipy, pandera, and pytest 9.1.1, with seed 42 fixed wherever randomness enters the picture.
No affiliate relationships, no sponsorships, and no tool named above (scikit-learn, pandas, scipy, pandera, pytest) paid for placement. They are referenced because they are what the code actually runs on.
This is Article 15, the final article in the Production ML Engineering series.
Leave a Reply