Alibaba turns its Qwen LLM into a self-driving brain

Teaching AI to drive has always forced a messy trade-off: build a custom autonomous driving stack from the ground up, or shove a general vision-language model into the car and hope it somehow understands 3D space.

Alibaba is trying a cleaner middle ground. On August 31, the company revealed Qwen-Drive-1.0, a vision-language foundation model designed to handle driving scene perception and path planning without rewriting the underlying foundation model.

Plugging experts into an untouched core

Instead of altering the architecture of its natively multimodal Qwen3.5-4B, Alibaba kept the core network completely untouched. Single-view cameras, multi-view driving setups, and sequential video frames all feed into a shared visual pathway.

From there, two external components tap into the representations.

The first is a BEV (Bird's-Eye-View) perception head. Acting as an inspectable 3D probe, it processes multi-view single frames to simultaneously tackle 3D object detection, semantic occupancy prediction, and map segmentation. Its training losses flow right back into the shared visual encoder.

The second is a Planning Expert powered by a diffusion transformer. Using flow matching, it predicts a 5-second ego trajectory at 10 Hz. Crucially, it can take an optional written explanation of the driving plan as a conditioning prompt.

Pure open-data pedigree

The whole system was trained across staged phases on 2.83 million publicly available samples. Alibaba standardized differing annotation styles across public datasets to ensure stable trajectory outputs.

It works surprisingly well. In driving visual question-answering tests, Qwen-Drive-1.0-SFT posted an average score of 69.43, topping general-purpose VLMs as well as specialized embodied and driving models.

There is still a catch, though. Alibaba admitted that what the model reasons in text does not always match the physical trajectory it generates—a consistency gap they are still working to fix.

Why it matters

We think Alibaba's modular approach is the most practical way forward for bringing VLMs into vehicles. Rebuilding an entire foundation model specifically for driving is expensive and throws away general reasoning capabilities. By keeping a 4B model intact and using dedicated 3D heads, they get real-world spatial precision without losing common sense.

Using only public datasets is another win. It shows researchers and smaller automakers that you do not need proprietary vehicle fleets to build functional driving foundation models.

However, the gap between textual rationale and planned trajectory is a serious safety hurdle. In an autonomous vehicle, a model explaining that it sees a pedestrian while simultaneously drawing a path straight through the crosswalk is catastrophic. Until that reasoning alignment is rock-solid, this remains a promising blueprint rather than road-ready software.