Adding a new system’s logs to a security pipeline looks easy until the first quiet failure. This guide describes a repeatable procedure for bringing one source on board under SecDataOps, the practice of engineering security telemetry so that evidence stays complete, trustworthy and affordable. It is written for hospital and clinic technology teams and stays strictly administrative: nothing here concerns clinical systems, patients or advice. Follow the steps in order and keep a dated note of each.
Step 1: Register the source and name its owner
Create a row in the coverage register. Record the system, the team responsible, the kind of events it can emit and where they would be sent. If nobody will accept ownership, stop here, because an unowned feed is the first thing to rot. A human name or team alias in the owner column is the requirement; a shared inbox nobody reads does not count.
Step 2: Define what “good” means
Write a short telemetry contract. According to the Ops Atlas, a contract lists the owner, schema, required fields, freshness, tiers, retention, a canary, cost and known gaps. Keep it to a page. Choose numbers you can measure: for example, the Atlas sample sets freshness at five minutes for the ninety-fifth percentile, a parse rate of at least 0.995 and completeness of at least 0.98. These are examples. Pick figures that match what this system can realistically deliver.
Step 3: Choose the transport carefully
Delivery guarantees differ. Old-fashioned syslog has no way to confirm receipt, which lets a lost message disappear without a trace. If the source offers an agent or an API with acknowledgement, prefer it. When your collector keeps events in memory only, remember that the Atlas describes such buffers as not durable, and configure a disk buffer so a restart does not lose data. Record the choice in the contract.
Step 4: Parse into a shared schema
Map the raw fields to the layout your pipeline uses. Open schemas exist, and the Atlas names OCSF as one, with a numbering rule where the type identifier equals the class identifier times one hundred plus the activity identifier. Whatever layout you adopt, keep two timestamps, the moment the event happened and the moment you processed it, using UTC and ISO 8601 notation down to the millisecond. Store a stable unique identifier too, because replays will otherwise count the same event twice.
Step 5: Treat the parser as code
Put the parsing rules in version control. Add sample inputs, called fixtures, and expected outputs, and run them automatically whenever the rules change. Guard against clever shortcuts: the Atlas describes a strict conversion that turned a blank into zero, so rubbish text looked like a blocked attempt on port zero. A safer pattern is an anchored expression that sends anything unparseable to a quarantine area for inspection rather than forcing it through.
Step 6: Keep the raw copy and decide the tiers
Send the untouched original to the cheapest long-term tier. Send a normalised copy forward into searchable storage for as long as the people who query it truly need. Resist the urge to discard raw data after normalising; parsers err and schemas evolve. Check the cost per day and write it into the contract so later reviews have a baseline.
Step 7: Set retention on purpose
Look at what your obligations and risks require rather than accepting a default. The Atlas cites several published reference points: a 2021 American federal memo with graded maturity tiers and long minimums, 2024 Australian-led advice proposing twelve months as a floor to begin with for admin and security records, and less for informational ones, and an Indian rule that says half a year. Pick a number, record who chose it and note the date it should be reviewed.
Step 8: Plant a canary and add a health monitor
Create a small recurring test event and verify that it appears at the far end. Add a per-source monitor that pages after a defined period of silence, not a global graph. Averages hide the single stream that has stopped.
Step 9: Prove it with a silence test
In a lab or with the owner’s agreement, stop the feed on purpose and time the gap before an alert fires. Write down the result. Repeat monthly for important sources and after any significant upgrade or certificate rotation, since changes made outside the change board have been known to reset logging quietly.
Step 10: Review and record
Update the register, mark the date of onboarding and add the source to the next maturity review. The Atlas ladder runs from reactive to inventoried, contracted, measured and optimising; completing these steps moves a source along it. For background on the term and the definitions used here, see the Ops Atlas page for SecDataOps.
Allow a realistic amount of time. A simple source with clean documentation can be onboarded in a day, while an awkward one with free-text messages and inconsistent fields may need a week of parser work. Do not skip steps to hit a date; the skipped step tends to resurface as a silent gap months later, when somebody needs the evidence and finds a hole where it should be.
A quick checklist
- Owner named, row created in the register.
- Contract written with measurable targets.
- Delivery method chosen and buffers made durable.
- Parser versioned with fixtures and automated tests.
- Raw copy retained cheaply; tiers and cost recorded.
- Retention set deliberately with a review date.
- Canary, monitor and first silence test completed.
If you enjoy seeing how the many operations disciplines fit together, try our overview, a unified look at modern IT, data and cloud operations, and the practical article on making infrastructure resilient through everyday operations habits. This article is general technology information, not medical, legal or compliance advice.
