xAI unleashes Grok-2 to take on Claude and GPT-4

For weeks, an anonymous model named "sus-column-r" had been quietly climbing the LMSYS Chatbot Arena leaderboard, casually beating Claude 3.5 Sonnet and GPT-4-Turbo. Now the curtain is officially pulled back.

XAI officially launched the beta of Grok-2 alongside its smaller sibling, Grok-2 mini. That mystery model was indeed Grok-2, and the leap over Grok-1.5 is massive.

Frontier scores and an unexpected image partnership

On standard academic benchmarks, Grok-2 proves it can trade blows with the best frontier systems available.

It scored 56.0% on GPQA, up from Grok-1.5's 35.9%. In MATH, it reached 76.1%, alongside an 88.4% on HumanEval coding tasks. Vision benchmarks tell a similar story, putting up 69.0% on MathVista and 93.6% on DocVQA, matching or exceeding GPT-4o in several categories.

Beyond raw intelligence benchmarks, the big everyday upgrade on the 𝕏 app is visual generation. xAI teamed up with Black Forest Labs to integrate their state-of-the-art FLUX.1 model directly into Grok.

Currently, both Grok-2 and Grok-2 mini are rolling out to 𝕏 Premium and Premium+ subscribers. The app features a redesigned interface, real-time post retrieval, and faster reasoning.

Later this month, both models will hit developers via a dedicated enterprise API, equipped with multi-region low-latency deployment, mandatory multi-factor authentication (including Yubikey and TouchID), and full team analytics.

Why it matters

If you are paying for 𝕏 Premium, this is an instant win. You are getting access to top-tier reasoning and coding alongside what is arguably the most capable text-to-image model on the market right now via FLUX.1, all hooked into real-time social data.

For the AI landscape, xAI has officially entered the heavyweight title fight. Moving from Grok-1 in November 2023 to beating GPT-4-level benchmarks in under a year is a staggering engineering feat, validating the massive compute power and talent density Elon Musk has been assembling.

Partnering with Black Forest Labs rather than spending months building an in-house diffusion model was a brilliant, pragmatic move. That said, letting a minimally restricted frontier image generator loose on an uncurated social feed is already sparking deepfake and copyright concerns. We think xAI will face a much harder test when pitching this platform to risk-averse enterprise API clients.