GPT-4 is quietly closing its speed gap with GPT-3.5

If you've been pinging OpenAI's API lately, you might have noticed GPT-4 feeling unusually snappy.

Turns out, it wasn't just in your head. Developer platform Portkey has been tracking real-world response times for both GPT-3.5 and GPT-4 over the past three months, and the data shows GPT-4 is rapidly closing the latency gap with its lighter sibling.

Shaving the worst-case lag in half

Breaking down what actually happens during an API call, Portkey separated total latency into three distinct buckets: round-trip network time (usually 100 to 200ms), system queuing time (100 to 500ms), and actual model processing time.

Processing time isn't just about output length. Portkey noted that token count is often a poor predictor of speed. A simple prompt with 204 tokens cruised through in 4.5 seconds, while a gnarly, complex prompt of just 33 tokens choked the model for a full 32 seconds.

When looking at median requests, both GPT-3.5 and GPT-4 already perform consistently well, clocking in at under 1ms per token.

The dramatic breakthrough happened in the tail latency. For the 99th percentile of requests—the sluggish queries that usually leave users staring at blank loading screens—GPT-4's latency has more than halved in just a single quarter.

Why it matters

We think this shifts the practical math for anyone building interactive AI products.

Until recently, many developers avoided GPT-4 for conversational bots or search interfaces because the worst-case wait times completely wrecked user experience. Now that tail latencies have collapsed by half, the speed penalty is mostly gone for the majority of everyday tasks. If you can afford the bill, GPT-4 is suddenly viable in places where you used to have to settle for GPT-3.5.

It also highlights how much silent infrastructure work OpenAI is pouring into serving efficiency. Running frontier models isn't just about benchmark scores; it's about hardware allocation and inference optimization under heavy load.

The lingering catch is cost and reliability near the ceiling. Portkey pointed out they are still investigating whether OpenAI throttles response speeds as developers approach their rate limits. GPT-4 may not be slow anymore, but keeping your infrastructure costs from exploding is still on you.