An assertion is not a control. Any operations team can say “our chat bot is locked down”; the review that matters asks to see the proof, item by item, the way an evidence-based clinician asks for the study rather than the anecdote. This checklist is built on that habit. It walks through a ChatOps setup — shared chat as the operating room, where bots broadcast events and only vetted commands ever act — and for each area names the artefact that demonstrates the control actually exists. The technical baseline for every item comes from the full ChatOps field notes on the Ops Atlas; read them once before scoring, then let the checklist do the talking. Teams that have already worked through structured selections elsewhere — the discipline behind choosing an orthopedic hospital step by step — will find the same skeleton: criteria first, evidence second, verdict last.
How to Run a ChatOps Evidence Review
Gather four artefacts before the session: the inventory of commands the bot may execute, the permission policy that governs them, the destination and format of the audit record, and the last month of refusal logs. Score each area below as evidenced, partial, or absent. Be strict about the difference between “we could produce that” and “here it is” — in controls, as in medicine, the chart is the reality.
Origin and Identity: Proving Who Sent What
- Every inbound request is verified against cryptographic proof of origin before any command processing begins. Slack signs every request with HMAC-SHA256; verification compares that signature and discards anything older than the five-minute replay limit, while still answering the platform inside 3,000 milliseconds — speed is part of correctness. On Teams it means validating a signed JWT. A static token, as Mattermost deployments use, does not cover the request body — treat it as identification, not authentication, and add compensating checks.
- Each verified request maps to an immutable user identifier. Display names appear nowhere in the authorisation path.
- Forged or replayed requests have a documented fate: rejected, logged, and visible to a human the same day.
Permissions: What the Bot May Do, and for Whom
- The command inventory exists as a written artefact, and every entry carries a risk grade: pure information, read-only, reversible write, or high-impact.
- Reversible writes require three simultaneous conditions — a qualifying role, an approved channel, and an audit record. Actions graded high-impact require two distinct people, and the gateway enforces that the approver differs from the requester. This dual-authorisation pattern is formalised in security standards as an access control, so borrow the wording from the standard and make it configuration, not culture.
- A break-glass override exists, is deliberately inconvenient, and every activation lands in a mandatory next-day review.
- Free text never reaches a shell. Commands match fixed patterns; everything else is refused in public.
Audit and Review: The Record That Outlives the Room
- The audit trail lives outside the chat platform, in an append-only store, following a defined schema so that events can be queried years later. Chat scrollback is conversation, not evidence; retention and integrity guarantees differ, and reviewers should never depend on the platform’s goodwill.
- Refusal logs are reviewed on a published cadence — weekly works — with the mix of reasons tracked, not just the total.
- Allow-list entries are re-justified quarterly; anything without an owner or a recent use is removed in the open.
Attention: The Budget Nobody Invoices
Evidence-led teams measure attention because attention is the resource the channel consumes first. Three numbers do most of the work. The actionable-notification ratio — what share of bot posts led a human to do something — has been observed in the twenties of percent in real deployments, with something nearer half as a fair target. Approval latency shows whether the two-person rule protects or paralyses; minutes are healthy, days are a design fault. And notification load per shift turns “the channel is noisy” into a figure that can be halved by grouping and deduplication. A worked example of the honesty this produces: one published deployment reports audited-change coverage around ninety-four percent and command success just above ninety-eight — numbers modest enough to believe, which is what makes them useful.
Adaptability: The Evidence Only Time Provides
One control cannot be photographed in a single review, only demonstrated across years: the operations logic must survive the chat platform it runs on. The industry keeps setting exam dates. Hipchat and Stride are already gone. Slack retired its legacy custom bots on 31 March 2025. Microsoft switches off Office 365 Connectors for Teams in May 2026. Opsgenie, home to many teams’ alerting, closes on 5 April 2027. Each announcement stranded the teams that had welded their workflows to the adapter, and spared the teams that kept decisions, policies and records in components they own. The review question is therefore not “does it work today?” but “show me what would break tomorrow” — and accept only one answer: a replaceable adapter, with nothing of consequence living inside it.
Scoring Your ChatOps Setup, and What Comes Next
Count the evidenced items. A full score is rare and should not be the goal of a first pass; the goal is a list of absences with owners. Partial scores usually cluster — teams strong on permissions tend to be weak on attention metrics, and vice versa — so fix clusters, not items. Re-run the review quarterly, after any platform migration, and after every break-glass use. For the wider architectural picture the checklist sits inside — where a chat adapter fits among gateways, executors and the rest of a modern operations stack — the guide to building resilient modern IT infrastructure on this site is a natural next read, and the closing argument is simple: in operations, the evidence you can produce on request is the difference between a practice and a habit with good PR.
