Cosine's new AI coder Genie just crushed Devin on SWE-bench

Back in March, Devin shook the tech world by claiming the title of the first fully autonomous AI software engineer. Just five months later, a new challenger didn't just catch up—it blew past Devin's benchmark.

San Francisco-based startup Cosine, backed by Y Combinator, just unveiled Genie, an autonomous AI software engineer. On SWE-Bench, the gold standard for evaluating real-world software engineering capabilities, Genie scored 30%. That easily tops Devin's 13.8%, as well as rival tools like Amazon Q and Factory's Code Droid, which both sat around 19%.

Cosine co-founder and CEO Alistair Pullen posted on X that Genie is far more than a benchmark score, claiming it was trained from day one to behave and think like a human engineer.

Built like a coworker, not a chat box

Instead of simply wrapping a standard base model with basic terminal tools, Cosine spent nearly a year compiling a custom dataset of real human software development activities. Pullen explained that high-quality data of this kind simply didn't exist, forcing the team to build a complex data pipeline involving static analysis, self-play, and step-by-step verification over billions of tokens.

Under the hood, Genie runs on an unreleased GPT-4o variant provided through OpenAI's experimental access program. This model features massive token output capabilities—likely related to OpenAI's recently teased long-output GPT-4o variant capable of spitting out 64,000 tokens compared to the standard 4,000.

Genie handles 15 programming languages, ranging from Python and TypeScript to Rust, Go, and C++. It doesn't just write code; it integrates directly into Slack and GitHub. Genie can ask clarifying questions when requirements are vague and respond directly to review comments on its own pull requests. All code lives inside the user's repository, meaning Cosine doesn't store private codebase data.

For now, Genie is available only via an invite-only preview.

Why it matters

We think Genie marks a critical turning point for the autonomous agent space: specialized synthetic datasets and ultra-long output context can beat brute-force prompting.

For developers and engineering leads, the bar for AI tools is rapidly shifting from code completion to end-to-end task execution. If Genie delivers on its promise, a developer could assign a bug ticket or feature request before going to bed and wake up to a well-tested pull request ready for review. The Slack integration is a clever touch that acknowledges how software gets built in the real world.

For the AI landscape, Cosine's launch puts immense pressure on Cognition. Devin's early spotlight is fading fast as vertical startups partner directly with frontier labs to fine-tune specialized models. Cosine feeding its training learnings back to OpenAI highlights a mutually beneficial playbook that smaller startups will likely copy.

Still, a 30% score on SWE-Bench means Genie still fails 70% of the time on complex, multi-file problems. Fixing broken code written by an AI can often take longer than writing it from scratch. Genie looks undeniably impressive on paper, but we will see how it handles messy legacy codebases once real users get their hands on it.