Microsoft drops realistic benchmark for cybersecurity AI

ExCyTIn-Bench is Microsoft’s newly open-sourced benchmark designed to test how well AI agents handle real-world cybersecurity investigations.

Most existing security evals act like multiple-choice trivia quizzes. ExCyTIn-Bench takes a completely different route by dropping AI agents into a simulated Azure Security Operations Center (SOC) loaded with 57 noisy log tables pulled from Microsoft Sentinel and related services.

How it actually works

Instead of inspecting static text snippets, agents inside ExCyTIn-Bench have to act like real SOC analysts. They query live-style tables, jump between data sources, formulate advanced search queries, and hunt down indicators of compromise (IoCs).

The ground truth is derived from threat investigation graphs—bipartite alert-entity graphs created by human analysts. This setup gives the benchmark a step-by-step, fine-grained reward mechanism, allowing defenders to score how well the model reasons, uses tools, and synthesizes evidence rather than just checking if it guessed the right final answer.

Microsoft is already running the suite internally to stress-test in-house security models and audit features across Microsoft Security Copilot, Sentinel, and Defender. The team is also preparing personalized benchmark generation tailored to specific customer tenant threats.

What the early results show

Explicit reasoning turns out to be everything. In initial evaluations, GPT-5 with High Reasoning led the pack with a 56.2% average reward, almost matching high-end chain-of-thought methods like ReAct, reflection, and BoN at 56.3%.

Dial back the reasoning depth on GPT-5, however, and performance drops by nearly 19%. Step-by-step deliberation is clearly essential when sorting through noisy security data.

Smaller models equipped with effective chain-of-thought, such as GPT-5-mini, are punching well above their weight and rivaling larger models at a fraction of the cost. Meanwhile, open-source models are rapidly closing the gap with closed proprietary systems.

How to get started

ExCyTIn-Bench is free and fully open-source. The source code is available on GitHub, and the accompanying dataset is hosted directly on Hugging Face. Security teams and model builders can clone it right now to evaluate custom agents or contribute new benchmark runs.

Why it matters

For CISOs and security leads, this is a much-needed reality check against vendor hype. Static benchmarks rarely translate to real SOC survival, and ExCyTIn-Bench gives teams a reproducible way to see if an AI agent can actually triage multitable log data before handing it network keys.

We think shifting evaluation to step-by-step reasoning rewards is a smart move. The main friction point will be integration overhead, as setting up and interpreting these complex, multistep runs will still require serious SOC engineering time.