Grok goes multimodal with Grok-1.5V
Grok-1.5V just dropped as a first-generation multimodal model, expanding beyond text to handle documents, diagrams, and real-world photos. The team also released RealWorldQA, a new spatial reasoning benchmark where Grok scored 68.7%, edging out rival frontier models. It will roll out soon to early testers and existing users.
If an AI can only read text, it is basically blind to the real world.
xAI just officially unveiled Grok-1.5 Vision (Grok-1.5V), its first-generation multimodal model. Alongside its text capabilities, Grok can now process documents, scientific diagrams, charts, screenshots, and everyday photographs.
The preview will roll out soon to early testers and current Grok subscribers.
Reading diagrams and road views
xAI showed off Grok-1.5V turning a logic flowchart of a guessing game directly into working Python code without hesitation.
On the benchmark board, Grok-1.5V trades blows with heavyweights like GPT-4V, Claude 3 Opus, and Gemini Pro 1.5. In Mathvista, it scored 52.8%, slightly nudging past GPT-4V. In diagram reasoning on AI2D, it clocked 88.3%.
All these scores were evaluated in a zero-shot setting without chain-of-thought prompting, according to xAI.
Alongside the model, xAI also dropped a brand-new benchmark dubbed RealWorldQA. It is designed to test basic spatial understanding in physical environments, featuring over 700 images with questions and answers, many sourced from vehicle cameras. Grok-1.5V scored 68.7% on it, leading GPT-4V and Gemini Pro 1.5.
Why it matters
We think this is xAI finally bridging the gap to become a serious multimodal contender.
For everyday users and developers, Grok is no longer just a witty chatbot on social media. Being able to digest diagrams, UI screenshots, and real-world snaps brings it closer to a practical daily assistant, though you will still need early access or an existing subscription to take it for a spin.
The real story here is the emphasis on physical space. While rivals mostly brag about reading corporate earnings charts, xAI is putting vehicle camera feeds into its benchmarks. That screams synergy with Musk's broader ecosystem, particularly Tesla and Optimus.
Catching up to GPT-4V on zero-shot visual reasoning in this timeframe is impressive. Still, basic spatial Q&A is a long way from real-world robotic interaction. But by shifting the benchmark battle toward physical environments, xAI is quietly changing what people expect frontier vision models to do.
Sources (1)
- x.ai(opens in a new tab)Originalx.ai