Halfway through buying conversations this year, a pattern settled in: every security vendor now says some version of “AI”, most mean something different by it, and the teams signing the cheques have started asking for proof instead of slides. AIOps for Security is the phrase that has gathered around this moment — models that spot anomalies and rank alerts, assistants that investigate with language models, and a young discipline of evaluation that decides what such systems may be allowed to do. Here is where the field actually stands, without the brochure adjectives: what shipped, what the evidence says, which standards arrived, and where buyers keep getting caught.
The label, and the lack behind it
Start with honesty about the name. AIOps for Security is an editorial label — there is no standards body behind it, no founding milestone, no certification. It exists because the market needed a shelf. On that shelf sit several overlapping product families: behaviour-based anomaly detection sold as UEBA, machine-learned alert scoring, and the newest arrivals, agents that read alerts and investigate them with large language models. Gartner tracks the agent wave under its own name — “AI SOC agents” — and for 2026 has placed the category at its Peak of Inflated Expectations, the firm’s signal that enthusiasm is running well ahead of demonstrated results. Gartner also has a word for capability claims that outrun evidence: AI washing. Both warnings belong in the procurement file from day one. Scope matters when the shelf is this loosely defined: anomaly detection, alert scoring, model-assisted investigation, evaluation with base rates and miss bounds, and autonomy levels are all in; IT event correlation, telemetry plumbing, containment playbooks, and vulnerability prioritisation are different shelves with different labels, however much the brochures blur them.
One clarification saves whole meetings: this is not the “AIOps” that has been an IT-operations term for years. The older AIOps correlates events and trims noise in infrastructure monitoring. The security variant faces an adversary that adapts to the detector, and a single miss can cost a breach — two differences that change how any accuracy figure should be read.
What actually shipped in AIOps for Security this year
Three vendor facts define the current state of play. First, Microsoft’s Phishing Triage Agent: it begins auto-closing mail from the moment it is enabled, and as of October 2026 its documentation offers no observe-only mode — buyers who want a watch period before closing starts should confirm in writing whether one exists for their tenancy. Second, Google’s Triage and Investigation Agent, which the company grades at autonomy level one — recommend, not act — and times at roughly a minute per triage. That is the more conservative posture, and the more useful precedent: a big vendor explicitly labelling its own product below full autonomy. Third, a deadline that has nothing to do with AI but lands on the same desks anyway: Sentinel’s Fusion engine is on the way out, replaced by Defender XDR correlation, and every workspace has until 31 March 2027 to finish the move. Teams planning agent rollouts on top of Sentinel should sequence that migration first, or their evaluation baselines will move underneath them.
The evidence base, read plainly
Independent benchmarks now exist, and their numbers deserve reading twice. ITBench’s evaluation, presented at ICML 2025, ended with agents having cleared 25.2% of its compliance-and-security scenario set. CyberSOCEval — Meta and CrowdStrike’s joint 2025 benchmark — scored 23-34% when the task was malware analysis and 43-53% when it was reasoning over threat intelligence, against coin-flip baselines of 0.63% and 1.7%. Read that as: the systems clearly beat random, and clearly cannot run a response function alone. Neither benchmark measures your mail flow, your identity infrastructure, or your adversaries. The reasonable conclusion is neither dismissal nor deference — it is “useful floor, useless ceiling”.
The research record adds a caution about vendor-grade numbers. Between 2019 and 2022, the TESSERACT researchers re-examined published security-ML results that quoted F1 near 0.99 and traced most of the inflation to spatial and temporal bias in how the tests were constructed. If near-perfect scores can be manufactured by accident in academic settings, a slide deck can manufacture them without breaking a sweat. Scores approaching 1.0 on adversarial problems are a finding about the test, not the model. The instinct to distrust such numbers is old: Sommer and Paxson observed in 2010 that machine learning saw little operational security use precisely because errors carry a steep price there — a caution that predates the current agent wave by sixteen years and still holds.
Standards are arriving late, but arriving
The governance scaffolding now exists in draft-and-revision form, and procurement teams can cite it without embarrassment:
- NIST’s AI Risk Management Framework — version 1.0, issued 2023, revision under way — sets the risk vocabulary most regulators now echo.
- The Generative AI Profile, NIST AI 600-1 (2024), extends it to the model class these agents actually use.
- AI 100-2 E2025 (March 2025) covers adversarial evaluation — the part your purple team will care about.
- MITRE ATLAS, updated to the 2026.09 release on data format 6.0.0, catalogues attacks against AI systems themselves.
- OWASP’s Top 10 for LLM Applications (2025) does the same at application level.
- The UK NCSC has three directly relevant papers: Deploying AI Systems Securely (April 2024), AI and Data Security (May 2025), and the December 2025 note on prompt injection.
None of these documents tells a buyer “sign here”. Together they tell a buyer what to ask, which is nearly as useful.
Where AIOps for Security buyers keep getting caught
Three traps account for most of the regret we hear about. The first is residency fine print. Microsoft’s Security Copilot defaults customer-data sharing to on unless the tenant licences at E5 or E7 tier; Google’s Security Operations processes Gemini requests on global endpoints, though the company states prompts are excluded from model training. Neither fact is a dealbreaker by itself — both are facts that belong in writing before signature rather than after. The second trap is upload leakage: investigation assistants that push attachments to external analysis services, where VirusTotal’s own terms acknowledge that uploaded content may be seen by its premium subscribers. An attachment under investigation is evidence in an active matter, and every upload is a disclosure choice. The third trap is the oldest: buying “percent automated” as the KPI. Numbers that reward closing volume reward closing carelessly, and the misses become invisible because no human ever sees the ticket.
A fourth trap deserves its own line: prompt injection. An agent that reads inbound mail is reading attacker-controlled text, and the NCSC’s December 2025 guidance argues such a model is an “inherently confusable deputy” whose confusability may never be fully mitigated. Deterministic guards and least-privilege scopes are the control set; clever filtering is not.
What a pragmatic team does this quarter
Strip the field to its working parts and the to-do list is short:
- Count the base rate for two alert categories — the fraction of alerts that are genuine — because rarity caps every tool’s precision on your data.
- Run any candidate in observe mode against blind-labelled cases before it touches a live decision; if the product lacks an observe mode, that is the evaluation result.
- Grant autonomy by action class, with written gates: recommend first, auto-close only on a proven miss bound, reversible actions last — and never irreversible ones.
- Plant canary cases and retire the closing privilege the moment the plants get closed.
- Report per category, never pooled, so a healthy average cannot launder a failing category.
Teams that already operate complex monitoring will recognise most of this posture; the unified XOps approach to modern operations maps the surrounding territory, and one operator’s notes on keeping smart machines useful without the chaos translate directly to agent supervision. When the platform side of the house needs attention after all this, start with how engineering support strengthens delivery and cloud foundations.
For the reference version of everything above — the autonomy rungs, the gate arithmetic, the evaluation drills — keep the Ops Atlas listing for AIOps for Security open beside the vendor scorecards. Status checks age, but evidence requirements do not, and this quarter’s evidence requirements are written above.
