Meet JAT, an open-source Transformer agent that plays games and moves robots
Researchers just open-sourced JAT, a multi-purpose Transformer agent built to reproduce and improve upon Gato. Running on a single network, it tackles 157 tasks across Atari games, navigation, and robotic control, matching 65.8% of expert performance while supporting basic text and vision tasks. Why it matters: It opens up generalist embodied AI research with fully public models and datasets.
JAT, short for Jack of All Trades, is an open-source multi-purpose Transformer agent designed to handle sequential decision-making, physical control, and multimodal tasks using a single network.
Built as an open reproduction and improvement upon DeepMind's Gato, the project aims to take generalist embodied AI out of proprietary labs and into open-source hands.
What it can do
JAT is trained across 157 distinct tasks spanning games, navigation, and continuous control.
It can play Atari video games, control simulated robots in MuJoCo, manipulate robotic arms to assemble objects in Meta-World, and follow natural language instructions to navigate BabyAI grid worlds.
On top of reinforcement learning tasks, the agent accepts text and images as inputs, demonstrating foundational vision and language capabilities through a unified interface.
How it works
Under the hood, JAT is built on EleutherAI's GPT-Neo architecture, featuring an embedding mechanism tailored for sequential decision-making.
It interleaves observation embeddings, rewards, and action tokens into a causal sequence. The model routes different modalities through specialized encoders: CNNs for pixel observations, linear layers for continuous vectors and discrete values, GPT-2 tokenizers for text, and ViT encoders for images.
Instead of just predicting the next action, JAT also learns to predict future observations. The researchers found that tuning the weight of this auxiliary prediction objective significantly boosts the agent's learning efficiency.
Across all 157 evaluation tasks, JAT achieved an average of 65.8% of expert performance. It reached 99.0% of the expert baseline in BabyAI, 84.8% in MuJoCo, and 65.5% in Meta-World. In Atari 57, it hit 14.1% of expert scores, matching 37.6% of human performance and beating humans outright on 21 games.
Availability
The entire suite is publicly hosted on the Hugging Face Hub. This includes the JAT model weights, the underlying code, individual expert RL policies, and the JAT dataset containing hundreds of thousands of expert trajectories.
Why it matters
DeepMind showed the potential of multi-game, multi-robot generalist agents with Gato back in 2022, but never released the weights or the training data.
JAT fills that void by offering researchers a fully open baseline to study cross-domain transfer in embodied AI. Being limited to basic behavioral cloning means the model cannot yet surpass its expert teachers, but having an open-source Transformer that plays games and moves robots is a huge foundation for the community to build on.
Sources (1)
- huggingface.co(opens in a new tab)Originalhuggingface.co