Baidu's Ernie 3.5 beats ChatGPT on Chinese benchmarks

When OpenAI rolled out GPT-4, the big question was how fast Chinese rivals could close the gap. Baidu's answer took three months.

Baidu announced that the artificial intelligence model powering its Ernie Bot has outpaced OpenAI's ChatGPT on several key benchmarks. Citing the China Science Daily, Baidu said its newly minted Ernie 3.5 even topped GPT-4 in Chinese language evaluations.

Baidu began public testing of Ernie Bot back in March, and this update marks its quickest leap in foundational model development.

Testing Ernie against the giants

The benchmark breakdown shows exactly where each model shines.

In a test published by Microsoft based on standard admission and qualification exams, like college entrance and bar exams, Ernie 3.5 beat both ChatGPT and GPT-4 on the Chinese questions. When it came to English, Ernie 3.5 beat ChatGPT but still trailed GPT-4.

Another benchmark focused specifically on Chinese subjects, spanning more than 13,000 multiple-choice questions across 50 fields. Ernie 3.5 came out ahead of both ChatGPT and GPT-4.

Things changed on a third benchmark developed by a group of U.S. universities covering general science and humanities. On that home turf for Western models, Ernie 3.5 fell behind both ChatGPT and GPT-4.

Baidu CTO Haifeng Wang noted that just three months after Ernie Bot's beta rollout, Ernie 3.5 managed broad gains across creative writing, Q&A, reasoning, and code generation, alongside improvements in training and inference speed.

Plugins and faster iterations

Beyond raw scores, Baidu highlighted plugins as a defining feature for Ernie 3.5.

Plugins essentially serve as apps that the chatbot can call to complete specific tasks. Baidu has already hooked up its core search engine via a plugin, letting Ernie Bot fetch live web data rather than relying solely on frozen training data.

Why it matters

For users working primarily in Chinese, whether drafting documents or studying for local professional exams, Ernie 3.5 looks like a capable alternative to OpenAI's lineup, especially with real-time search natively baked in.

In our view, Baidu played a smart tactical move. Rather than chasing GPT-4 across the board, it focused heavily on dominating Chinese-specific tasks, where it holds a natural data advantage and needs to keep domestic rivals like Alibaba at bay.

That said, the results show that Ernie still has clear limits. In English reasoning and Western university-level exams, GPT-4 remains comfortably ahead. Baidu is also following OpenAI's playbook with plugins, but as OpenAI learned, keeping users engaged with third-party extensions is easier said than done.