Book Appointment

A Practical ObservabilityOps Audit for Hospital IT Teams

Hospital IT

If your hospital runs its own dashboards, log search and alert routing, there comes a day when someone asks whether the monitoring setup is actually healthy. This checklist is for that day. It treats ObservabilityOps as an audit target: the system that watches your administrative software, such as scheduling, billing exports and staff portals. It is deliberately non-clinical, and you can walk through it in an afternoon with a coffee and a shared document.

Before you start

Grab three things: a list of the services your team supports, access to the alerting configuration, and someone who carries the pager. The Ops Atlas, in its ObservabilityOps entry, frames the field as looking after the monitoring stack as a service, so every question below is about the platform rather than about the applications it observes. Score each line green, amber or red and jot a line of proof beside it.

Section A: Ownership and inventory

  • Can you name the person accountable for the platform, and their backup?
  • Is there a current list of data sources, destinations and consumers?
  • Is there a note on who may add new metrics, logs or dashboards?
  • Does every telemetry stream have a team that would miss it if it vanished?

The Atlas places a named owner and a basic heartbeat at the first rung of its maturity ladder, so these four items are the minimum bar. If more than one fails, stop and fix ownership before continuing.

Section B: Cardinality and limits

  • Do metric labels exclude unbounded values such as user or appointment identifiers?
  • Is there a limit on series or labels per scrape target?
  • Do you know your busiest ten metrics by series count?
  • Is there a check on the share of data labelled with an unknown service name?

A target that breaches its limits can cause the whole scrape to be rejected, so limits must be set deliberately, ideally with a clear message to the owning team. A healthy share of unknown service names is close to zero, because names come from instrumentation standards.

Section C: Alert quality

  • Are pages tied to symptoms people feel, such as failed bookings or slow portal logins?
  • In the last quarter, how many pages led to action? The Atlas example is 15 of 42, or 35.7 percent; what is your ratio?
  • Is there a list of alerts unused for ninety days, and a date to retire them?
  • Does a heartbeat alert prove that the alert route is alive?
  • Has anyone staged an artificial fault and confirmed that a human actually received the page?

Be honest with the actionability number. A low figure does not mean the team is careless; it usually means thresholds were copied from defaults. The Atlas quotes a plain rule from the reliability literature: signals no one reads should be weighed for deletion.

Section D: Logs and traces

  • Do structured logs carry trace and span identifiers and severity as top-level fields?
  • Are multi-line stack traces recombined into a single record?
  • Is there explicit handling for logs whose severity could not be parsed?
  • Do traces stay connected across queues, gateways and any older header formats?
  • Are extra context values allow-listed and stripped before calls leave your network?

This section matters for privacy as well as debugging. Context carried along with a request can travel to outside services if nobody filters it, so confirm that only approved keys leave the building and that public-facing entry points begin a fresh trace.

Section E: Retention and cost

  • Is retention tiered into hot, warm and archive, with a written reason for each period?
  • Have you estimated disk from retention time, samples per second and bytes per sample?
  • Is there a unit-cost figure, for example monthly spend divided by millions of requests?
  • Does any team receive a monthly showback of what its telemetry costs?

Disk needs can be calculated rather than guessed: take seconds of retention, times the samples arriving each second, times one to two bytes for each sample. If you want practical neighbours to this audit, the article on building resilient IT operations and the one on keeping technology teams sustainable cover the people and process side.

Section F: People and process

  • Do service teams still decide which of their own symptoms deserve a page, or has everything been handed to the platform owner?
  • Is there a written way to request a new dashboard, log source or alert rule?
  • When the platform itself degrades, is there a status note staff can read?
  • Has anyone confirmed that outside vendors’ marketing claims about pipelines or automated analysis were tested against your own data?

The last question is a gentle one. Claims made in sales material are only evidence about the vendor; a short trial on a sample of your own telemetry is evidence about you.

Scoring and next steps

Count the fails. Four or more in a section means that section becomes the next thirty-day project; two or three means schedule a review; fewer is cause for a quiet moment of pride. The Atlas’s three-block, ninety-day outline makes a handy template: ownership, inventory and heartbeat first, standards and limits second, pruning and a first cost report last.

If you only have time for part of this, do sections A and C first, since ownership and alert quality are where most departments discover their biggest gaps. Section B follows naturally, because label discipline is the cheapest way to avoid an outage in the platform itself.

A final tip: repeat this audit every quarter and keep the completed sheets. A shelf of dated audits becomes the best evidence that the platform is improving, and it makes the next budget conversation far less stressful than it would be with only opinions to offer.