OpenAI rolls out upgraded GPT-4 Turbo with vision

OpenAI just dropped what it called a "majorly improved" GPT-4 Turbo, but coders running the benchmarks are seeing a very different story.

The company pushed the new model to its API and began rolling it out to ChatGPT. The biggest highlight is that GPT-4 Turbo with Vision is now generally available, finally supporting JSON mode and function calling directly within visual workflows.

While developers building multimodal apps rejoiced, AI coding tool aider quickly published benchmark results showing noticeable regressions.

More visual power, worse code

In the official announcement, OpenAI highlighted the unified API experience. Developers can now feed images into GPT-4 Turbo and reliably get structured JSON responses or trigger function calls, eliminating tedious multi-step workarounds.

But when put to the test on pure coding tasks, the new model stumbled.

Aider ran the vision-enabled GPT-4 Turbo through its code editing benchmark of 133 Exercism Python tasks. The model scored just 62%, landing at the bottom of the entire GPT-4 family.

While previous GPT-4 models hovered between 63% and 66%—a gap that aider notes may be statistically marginal—the direction of travel was still down, not up.

The return of "lazy coding"

The bigger headache for engineers is that the model's notorious laziness has gotten worse.

Earlier GPT-4 Turbo previews caught flak for skipping implementations and leaving placeholder comments like "implement method here" for the user to finish.

To measure this, aider uses a dedicated benchmark of 89 Python refactoring tasks designed to bait models into cutting corners.

The new GPT-4 Turbo with Vision scored a dismal 34%. According to aider, that makes it the laziest coder of any GPT-4 model tested by a wide margin.

As a result, while aider added full support for the new release, it decided to keep gpt-4-1106-preview as its default engine, citing its superior coding reliability.

Why it matters

This release shows that the "all-in-one frontier model" dream still comes with painful trade-offs.

If you need multimodal capabilities with structured outputs—like extracting structured data from receipts or UI mockups—the new API is an immediate upgrade worth adopting.

However, if your primary workflow is heavy software engineering, jumping on OpenAI's latest model right now looks like a downgrade.

We think this highlights an emerging friction in foundation model training. Squeezing vision, function calling, and instruction following into a single Turbo checkpoint seems to dilute its precision on edge-case programming logic.

For enterprise teams and dev tools, it reinforces an essential lesson: never auto-upgrade your production model without your own rigorous evals.