{"id":896,"date":"2026-11-02T03:40:00","date_gmt":"2026-11-02T03:40:00","guid":{"rendered":"https:\/\/planespart.com\/blog\/?p=896"},"modified":"2026-11-02T03:40:00","modified_gmt":"2026-11-02T03:40:00","slug":"aiops-for-security-nine-buying-mistakes","status":"publish","type":"post","link":"https:\/\/planespart.com\/blog\/aiops-for-security-nine-buying-mistakes\/","title":{"rendered":"Nine Buying Mistakes in AIOps for Security"},"content":{"rendered":"<p>Anyone who sources aircraft parts knows the drill: no certificate, no install. A part arrives with paperwork traceable to its origin, and you inspect it anyway, because the paperwork describes the part, not your aeroplane. Tooling decisions in security deserve identical discipline, especially now that software sold as AIOps for Security \u2014 machine scoring of alerts, language models doing investigation, agents closing tickets \u2014 arrives wrapped in the smoothest sales talk in the industry. What follows are the nine mistakes that show up again and again when teams buy this category of tool, arranged so you can run the checklist before the money moves.<\/p>\n<h2>Before You Sign: Nine Mistakes in AIOps for Security Purchases<\/h2>\n<ol>\n<li>Buying an automation percentage instead of an error boundary.<\/li>\n<li>Accepting a single blended accuracy score.<\/li>\n<li>Falling for near-perfect F1 figures.<\/li>\n<li>Allowing the tool to close from day one.<\/li>\n<li>Trusting a spotless random spot-check.<\/li>\n<li>Reading benchmark scores as predictions for your queue.<\/li>\n<li>Letting software take irreversible actions.<\/li>\n<li>Feeding it attacker content without channel limits.<\/li>\n<li>Skipping the data-residency fine print.<\/li>\n<\/ol>\n<h3>Mistake One: Buying a Percentage Instead of a Boundary<\/h3>\n<p>&#8220;Seventy percent automated&#8221; is the industry&#8217;s favourite slide, and it measures nothing that matters. Automation share says how much of the queue the machine touched, not how often it was right, and not what a wrong touch cost. The useful question is about boundaries: what is the worst-case share of genuine threats it passes silently, and what proof stands behind that number? A tool that closes 95% of the queue while quietly burying one real attack in a hundred is worse than a tool that closes nothing, because the burying is invisible. Procurement language should therefore ask for bounds and sample sizes, never for a coverage figure. When a representative answers with the percentage again, ask it a third time, slowly.<\/p>\n<h3>Mistake Two: Accepting One Blended Score<\/h3>\n<p>Aggregate accuracy flatters every deployment. Pool all alert categories together and a single healthy-looking figure can conceal a disaster inside one of them \u2014 the documented case had 42.3% of cloud-identity alerts missed while the phishing category glowed at 9.1%. Any evaluation that reports one overall percentage while ignoring its categories is, practically speaking, hiding its worst result. Demand the breakdown by alert family, and weigh the categories by how much damage a miss in each would actually cause.<\/p>\n<h3>Mistake Three: Falling for Near-Perfect F1<\/h3>\n<p>When a vendor quotes an F1 score of roughly 0.99, read it as a confession about their test set rather than a boast about their model. Research through the TESSERACT effort, spanning 2019 to 2022, demonstrated that such glittering figures can be inflated by spatial and temporal bias baked into benchmark construction. A broader audit by Arp&#8217;s team, covering thirty leading security ML papers, met sampling bias in 90% of them and data snooping in 73%. Peer review is no shield here; the flaw travels through the literature into the marketing.<\/p>\n<h3>Mistake Four: Allowing Close-From-Day-One<\/h3>\n<p>Observation before action is the safety feature buyers keep forgetting to ask for. Microsoft&#8217;s Phishing Triage Agent starts dismissing mail the moment it runs, and as of October 2026 no observe-only mode is documented for it. That transfers the whole burden of caution onto your deployment: build the shadow phase yourself, with blind labels applied by your people, before a single automatic closure earns belief. Log every machine verdict during those weeks and have two people grade a slice of them independently, so disagreements surface early. A vendor who cannot offer a watch mode is telling you their calibration ran on somebody else&#8217;s mail.<\/p>\n<h3>Mistake Five: Trusting a Spotless Spot-Check<\/h3>\n<p>The audit that feels rigorous and proves little: dip at random into a twentieth of the quarter&#8217;s closes \u2014 around 90 items out of roughly 1,845 \u2014 and find nothing wrong. Zero errors in a small sample still leaves the underlying miss rate bounded near 38%, courtesy of the rule of three. The stronger instrument is planted evidence: seed about fifty canaries a quarter, every one a confirmed genuine threat, under a blunt contract that strips auto-close the moment the agent closes three of them.<\/p>\n<h3>Mistake Six: Confusing Benchmark Scores With Your Queue<\/h3>\n<p>Public benchmarks now exist, which is progress, and their numbers are humbling, which is the point. ITBench, presented at ICML 2025, set agents compliance-and-security scenarios and watched them resolve 25.2%. CyberSOCEval \u2014 Meta and CrowdStrike&#8217;s 2025 effort \u2014 put agents at roughly 23 to 34% for malware analysis work and 43 to 53% for threat-intel reasoning, while pure random guessing scored 0.63% and 1.7% respectively. Better than coin flips, nowhere near competent. And a score earned on someone else&#8217;s scenarios says little about your mail flow, your identity stack, or your attackers. Treat benchmarks as a floor for expectations, never as a prediction.<\/p>\n<h3>Mistake Seven: Letting It Take Irreversible Action<\/h3>\n<p>Trust should be granted per action class, and the top of the ladder stays human. Reversible steps \u2014 hold a message, quarantine a file \u2014 can go to software once a false-action ceiling of 1% or better has been proven across 300 or more verdicts on items it would have acted upon, with a rollback actually rehearsed at least once and rate limits in force. The irreversible rung is never granted to software in careful programmes, whatever the demo implies. If a vendor&#8217;s roadmap ends at &#8220;full autonomy&#8221;, the roadmap ends the conversation.<\/p>\n<h3>Mistake Eight: Feeding It Attacker Content Unsupervised<\/h3>\n<p>An investigation agent reads what attackers wrote \u2014 that is its job and its weakness. Indirect prompt injection, identified back in 2023, lets hostile text steer a model that processes it, which turns any outbound channel the agent holds into a liability. The UK&#8217;s NCSC has called the model an inherently confusable deputy and cautions that the problem may never be fully mitigated, steering operators toward deterministic guards and least privilege instead of hopeful filtering, since deny-lists unravel under infinite rephrasing. Before signing, map every channel the agent can reach and cut the ones it cannot justify.<\/p>\n<h3>Mistake Nine: Skipping the Residency Fine Print<\/h3>\n<p>Three lines worth reading before purchase. Without M365 E5 or E7 licensing, Security Copilot leaves customer-data sharing switched on by default. Gemini in Google SecOps processes on global endpoints, though prompts are excluded from training. File uploads have their own footnote \u2014 VirusTotal notes that submitted content &#8220;may also be shared&#8221; with its premium customers. None of these lines is hidden, exactly; all of them change what leaves your environment, and all of them belong in the contract conversation, not the retrospective.<\/p>\n<h2>The AIOps for Security Buying Habit, Inspected<\/h2>\n<p>Notice the pattern running through all nine: every failure comes from accepting a claim where an inspection belonged. Parts people never do that, and the same reflex serves here. Ask for the boundary, the category split, the watch phase, the canary contract, and the channel map \u2014 in writing, with numbers attached. The <a href=\"https:\/\/www.xopsschool.com\/aiops-for-security\/\">Ops Atlas reference that tracks AIOps for Security<\/a> collects the gates and definitions this checklist leans on. And when the sourcing conversation widens to tooling in general, the lessons transfer: the same inspection-first mindset guides readers through <a href=\"https:\/\/planespart.com\/blog\/control-system-parts-buying-mistakes-to-avoid\/\">control system parts buying mistakes<\/a>, <a href=\"https:\/\/planespart.com\/blog\/avionics-parts-buying-mistakes-to-avoid\/\">avionics purchasing traps<\/a>, and the wider craft of <a href=\"https:\/\/planespart.com\/blog\/sourcing-enterprise-technology-a-strategic-manual-for-application-selection\/\">sourcing enterprise technology strategically<\/a>. Buy the way you install: papers first, trust second, and a documented path for pulling the part if it misbehaves.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Nine buying mistakes that sink AIOps for Security tools: blended scores, spotless spot-checks, benchmark overreach, missing watch phases, and residency terms.<\/p>\n","protected":false},"author":6,"featured_media":897,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[617],"tags":[],"class_list":["post-896","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-aviation-maintenance"],"_links":{"self":[{"href":"https:\/\/planespart.com\/blog\/wp-json\/wp\/v2\/posts\/896","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/planespart.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/planespart.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/planespart.com\/blog\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/planespart.com\/blog\/wp-json\/wp\/v2\/comments?post=896"}],"version-history":[{"count":1,"href":"https:\/\/planespart.com\/blog\/wp-json\/wp\/v2\/posts\/896\/revisions"}],"predecessor-version":[{"id":913,"href":"https:\/\/planespart.com\/blog\/wp-json\/wp\/v2\/posts\/896\/revisions\/913"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/planespart.com\/blog\/wp-json\/wp\/v2\/media\/897"}],"wp:attachment":[{"href":"https:\/\/planespart.com\/blog\/wp-json\/wp\/v2\/media?parent=896"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/planespart.com\/blog\/wp-json\/wp\/v2\/categories?post=896"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/planespart.com\/blog\/wp-json\/wp\/v2\/tags?post=896"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}