On a quiet Tuesday, an operations analyst at a mid-sized hospital group opens a shared drive and finds a forecasting model nobody remembers approving. It predicts how many beds’ worth of linen the laundry should plan for, it has been running for two years, and the person who built it has changed jobs. The numbers look reasonable. The analyst has no idea whether to trust them. That uneasy moment is what an MLOps audit is for, and the checklist below is written for exactly that person: a practical, human walk-through you can complete in an afternoon.
Everything here concerns administration and software, never patient care. The framework is adapted from a reference called the Ops Atlas; its MLOps article treats the post-launch care of trained models as an engineering job in its own right, one with its own habits and gates. Take a pen and tick as you go.
Section A: Can anyone rebuild it?
- Is there a record of the code version that produced the live model?
- Can you point to the exact snapshot of data it learned from, ideally with a fingerprint rather than a folder name?
- Are the settings and software environment written down somewhere other than someone’s memory?
- If you retrained today, would the results land within a tolerance you have agreed in advance?
The analyst in our story could answer none of these. In the Atlas’s mental picture, the model is a finished build whose ingredients include the data, so if either the code or the data is lost, the model becomes an orphan. Fixing this section first costs little and pays back everywhere else. A one-page run record, filled in at the end of every training session, is usually enough, and it takes about as long as writing a handover note for a shift.
Section B: Who depends on it?
- Is there a list of the teams and reports that read this model’s output?
- Does anyone control access to those outputs?
- If you retired the model next month, who would complain first?
The analyst discovered that three separate spreadsheets quietly consumed the laundry forecast, one of them feeding a purchasing decision. The Atlas lists undeclared consumers among its anti-patterns because they make retirement impossible. Write the consumers down, and the model becomes manageable.
Section C: How would you know it was going wrong?
- Does anyone track server health, input data and answer quality as three separate questions?
- When real outcomes arrive late, is there a stand-in signal, such as the shape of recent predictions?
- Are alert levels tuned on your own history, rather than copied from a tool’s defaults?
- When two periods are compared, are they alike? A holiday week set against an ordinary week will look like drift when it is only the calendar.
Here the analyst found a dashboard with a bright green indicator that only measured whether the server answered. It said nothing about whether the answers still made sense. The Atlas is clear that a vendor default is not a truth, and that a lone dashboard cannot say whether a shift is a pipeline bug, an upstream defect or a real change in the world.
Section D: What happens when it changes?
- Is each new version compared with the live one on a frozen test set, including important sub-groups?
- Is it released gradually, with the old version still ready?
- Is going back a matter of repointing, with a non-learned fallback standing by?
- When was the last time anyone rehearsed it?
The Atlas cautions against two opposite mistakes. One is thinking only about the launch and ignoring everything after it. The other is automating everything, such as a nightly retrain that promotes itself, which can create feedback loops that cause outages. A human gate in the right place is not backwards; it is deliberate design.
Section E: Does the change in one thing disturb another?
- If you adjust one input, do you know which other outputs shift?
- Is there a note about which quick fixes were made, and why?
The Atlas borrows a phrase from research on hidden technical debt: changing anything changes everything. That is a reason to be modest about quick wins, because each one adds an entanglement that someone will pay for later. In practice, keep a short change log next to the model, and when a tweak is made to one input, ask aloud which downstream reports might notice. Five minutes of that conversation beats a week of detective work.
Scoring your afternoon
Count your ticks. Fewer than half means the model is running on goodwill, and the first target is Section A. Half to three-quarters suggests a sound base with gaps, usually in monitoring. More than that, and your time is better used on rehearsals. Whatever the total, give every unticked line an owner and a date; an audit that ends as a list nobody owns is just another file on the drive.
What the analyst did next
By the end of the week our analyst had written a one-page record, listed the three consumers, and booked a quarterly rollback rehearsal with the IT desk. The laundry forecast itself did not change at all. What changed was that the team could now explain it. If you want a lighter companion read, this site has a friendly piece on keeping smart machines under control without the stress, and a broader look at building dependable infrastructure for modern IT work.
The story is invented, written for software and operations staff, and offers no clinical advice.
