PaLM-E brings 562B multimodal power to real-world robots

What happens when you plug a massive 562-billion-parameter language model straight into a real robot's sensors?

Google researchers just revealed PaLM-E, an embodied multimodal language model designed to control robots directly using continuous visual and sensory data alongside natural language.

Feeding images as words

Traditional approaches to robotics usually separate perception from reasoning. A computer vision pipeline tags what it sees into text labels, and a language model tries to figure out what to do next. That handoff loses critical spatial context and breaks easily on unfamiliar items.

PaLM-E skips that handoff entirely. The researchers built encoders that transform continuous robot observations, like raw camera feeds and joint state estimates, into dense vectors that match the exact dimension of text token embeddings.

In practice, an image of an open drawer and an actual English word look identical to the transformer. They are packed together into what the authors call multimodal sentences, fed into a decoder-only model that spits out textual action plans autoregressively.

The largest version, PaLM-E-562B, fuses Google's 540B PaLM with a 22B Vision Transformer. Without any task-specific fine-tuning, it set a new state-of-the-art on the OK-VQA visual question answering benchmark, all while keeping PaLM's full generalist reasoning, coding, and math skills intact.

Zero-shot navigation around toy turtles

Once general vision and language are baked together, the physical payoff becomes obvious.

In real-world tests, a mobile manipulator received the command: "bring me the rice chips from the drawer." The robot had to parse the long-horizon request, find the cabinet, pull the handle, visually spot the chips, and deliver them, dynamically adjusting its next step using live camera frames.

The standout achievement is zero-shot generalization. In tabletop experiments, the robot was told to "push green blocks to the turtle." It executed the trajectory cleanly, despite having never encountered that toy turtle in its physical training demonstrations. In another test requiring it to push red blocks to a coffee cup, the dataset contained only three past demos with a cup, none of which had red blocks.

This is what the authors call positive knowledge transfer. Real-world robotics data is notoriously scarce, but internet-scale text and images are abundant. PaLM-E proves that internet commonsense can directly subsidize physical action.

Why it matters

Robotics has always had a crippling data wall. Collecting physical robot trajectories costs infinitely more time and money than scraping billions of web pages. Google's clever maneuver here was leveraging web-scale multimodal training to make up for the data deficit in the physical lab.

For developers and hardware builders, PaLM-E lays out the blueprint for embodied intelligence. Instead of training isolated models for narrow picking tasks, you can use a massive multimodal model as a high-level planner that commands lower-level motor policies. It gives robots genuine semantic reasoning about objects they have never physically handled before.

That said, we think the real-world friction is hard to ignore. A 562B model cannot run onboard a mobile robot without a massive cloud inference pipeline. That introduces significant latency, making millisecond-level reactive control impossible for fast industrial settings. PaLM-E tells the robot what to do next, but specialized motor controllers still have to handle the delicate physics. Even with those caveats, this marks a decisive turning point: frontier AI models are finally stepping out of the chatbot window and into the physical room.