Book Appointment

Nine Steps to Make Hospital Event Functions Safe: A ServerlessOps Plan

Hospital IT

This is a no-frills procedure for hospital IT staff who inherit event-driven functions and need them dependable by the end of a quarter. The practice has a name, ServerlessOps, though you can follow the steps without ever using it. We cover only administrative workflows, things like bed-board notifications, appointment updates and discharge paperwork routing, and leave anything clinical to clinicians.

Allow roughly ninety days. Do the steps in order, because each one makes the next safer.

Step 1: List every trigger and every owner

Open a spreadsheet. One row per function. Columns: what starts it, what it changes outside itself, who is called when it breaks, and whether the caller is waiting live or the work is queued. Include timers and file-drop triggers; those are the ones people forget. Expect the first pass to uncover functions nobody can explain. Do not delete them yet. Tag them “unknown” and assign a deadline for somebody to claim them.

The Ops Atlas uses the same starting point in its first thirty days, as set out in the ServerlessOps entry at xOps School: triggers, failure queues and the ratio of timeouts first, improvements afterwards.

A practical tip from experience with shared hospital systems: ask the finance team which functions appear on the cloud invoice, then match the list against your spreadsheet. Anything billed but unlisted is a lead, and anything listed but never billed is probably dead and a candidate for retirement at the end of the quarter.

Step 2: Give every background path a failure destination

Any function triggered by a queue or an event bus needs a place for failed items to land. Then give that place three attachments: somebody accountable by name, a notification wired to a real inbox or pager, and a storage window that outlasts whatever the source keeps. If the source forgets after four days, keep failures for two weeks. Check each destination by pushing a harmless test item through and confirming that the alarm reaches a human.

Step 3: Align the timeouts

Write down three numbers for each function: its own time limit, the limit on any outbound call it makes, and, if a queue feeds it, the queue’s visibility window. Rules of thumb that hold up: outbound calls time out earlier than the function, and the queue window is a healthy multiple of the function limit. The Atlas suggests six times. A mismatch here produces ghost duplicates, because the queue offers the same item to another worker before the first has finished with it.

Step 4: Stop duplicates doing damage

  1. Pick the five functions with the most serious side effects, meaning payments, bookings and patient messages.
  2. For each, define a business key. A booking number works; a message number does not, because replays assign new ones.
  3. Record the key before the side effect and check it first.
  4. Set the record’s lifetime to cover your longest realistic outage. The Atlas warns that a library default of an hour can be far too short for that.
  5. If a downstream service offers its own duplicate protection, pass your key through to it and check how long it remembers.

This is the highest-value step in the whole plan. Do it carefully.

Step 5: Handle batches and streams properly

Functions that read from queues or streams in batches should be configured to report partial success, so one bad record does not force the whole batch around again. For stream sources, cap the number of retries and how old a record may get before it is discarded. The Atlas cautions that stock settings can retry forever, so a single poisoned record ends up blocking everything queued behind it. Where the platform supports splitting a failing batch in half to isolate the culprit, switch that on.

Step 6: Put a ceiling on parallelism

Find the slowest or most fragile downstream system your functions call. Agree with its owners on how many simultaneous calls it can tolerate, then cap the function’s concurrency below that figure. Check the account-wide allowance too: if your busiest hour uses most of it, other teams’ functions will be throttled when yours spikes. Reserve capacity for the critical ones, and remember that reserving takes capacity out of the shared pool.

Step 7: Deploy so that a bad release is small

Roll out new versions gradually, shifting a small share of traffic first and watching the numbers before shifting the rest. Deployment tooling such as AWS SAM supports aliases and canary shifts for this. One practical trap from the Atlas: some parameter-only changes reach only the unversioned latest copy unless a specific template extension is switched on, so confirm that your release really lands where traffic goes.

Step 8: Choose and publish a short list of measures

Pick four: the real success rate of the business action, duplicate side effects, age of the oldest failed item, and cost per million events. Put them on one page, reviewed monthly by IT and finance together. Alarm on item age and failed-item counts, not just the error total.

Keep the list short on purpose. A dashboard of thirty figures gets ignored; a page of four gets argued about, which is exactly what you want in a monthly meeting.

Step 9: Rehearse

In days sixty to ninety, run a game day. Replay a block of test events on purpose, kill a dependency, then confirm nothing happened twice and the failure queue behaved. Record what you found. Passing that exercise for your critical paths is the Atlas’s third maturity level.

Where this fits among other IT work

These steps sit beside wider infrastructure work rather than replacing it. If you want the broader view of keeping hospital systems resilient, our overview of resilient IT operations for hospitals covers the surrounding ground, and the earlier note on keeping automated systems manageable without burnout deals with the human side of looking after them.

Finish by scheduling the first monthly review before you close the project. Procedures without a calendar date tend to rot. This guide is general technical information and not clinical, legal or procurement advice.