Shanghai AI Lab drops InternLumina-U2

InternLumina-U2 brings image generation, instruction-based editing, and omni-visual understanding into a single model architecture. Instead of piping prompts back and forth between a visual LLM and a standalone diffusion model, it handles text QA, image generation, editing, video analysis, and 3D object comprehension through one shared backbone and a discrete token interface.

Shanghai AI Laboratory open-sourced the model weights, SFT training pipeline, and accelerated inference code on Hugging Face and GitHub in October 2026.

What it can do

On the generation and editing side, it generates diverse styles ranging from photorealistic scenes to oil paintings, scoring 89 on GenEval. Its prompt comprehension during editing is remarkably sharp: tell it to "rotate only the orange block in the image" or turn a photograph into a charcoal sketch, and it modifies the target elements while leaving the rest intact.

On the comprehension side, it functions as a heavyweight visual language model rather than a simple renderer. You can feed it a complex ER database diagram and ask detailed schema questions, hand it a medical scan to identify specific neurological anomalies, or upload dense financial graphics, where it achieves an 86.2 on ChartQA.

It also natively handles motion and geometry. It can process a 64-frame video to summarize what happened from start to finish, and it can analyze GLB or GLTF 3D assets by using an integrated Blender pipeline to render multi-angle color and depth maps before reasoning over them.

How it works and specs

Previous unified models struggled because a single visual codebook cannot hold enough high-frequency detail, while hybrid continuous-discrete systems split understanding and generation across different pipelines. InternLumina-U2 bypasses this with a fully discrete, multi-codebook approach.

Using an AToken visual tokenizer, each spatial visual position is represented by 8 complementary discrete codebooks that capture geometry, text, and texture without ballooning sequence length. On input, these 8 embeddings are concatenated into a single backbone token. On output, the backbone denoises spatial positions in parallel, while an autoregressive head predicts the 8 codes per position in depth order.

Under the hood sits LLaDA2.0-mini, a masked diffusion language model converted from an autoregressive mixture-of-experts model. It has 16.9 billion total parameters, but only about 1.4 billion are active per token. Across its 20 Transformer layers, the first uses a dense feed-forward network, while the remaining 19 are MoE layers containing 256 routed experts and one shared expert, routing to 8 experts per token.

For inference, the repository includes an optional, training-free sampling method called Plateau-Halton with V-verify, which speeds up image generation by 4.45x.

To run it locally, you need Python 3.11, PyTorch 2.6, and CUDA 12.4 with flash-attn. Checkpoints are available for both NVIDIA GPUs and Huawei Ascend NPUs. Keep in mind that Apple's AToken tokenizer weights cannot be redistributed directly and must be downloaded separately from their repository, and 3D tasks require running the bundled script to set up Blender 4.5.

Why it matters

Unifying generative diffusion and multimodal reasoning into a single clean architecture has been a major bottleneck in open-source AI. Most production systems still daisy-chain a reasoning LLM to an external image generator, leaving the generator blind to true multimodal reasoning.

InternLumina-U2 demonstrates that a masked diffusion MoE can handle both directions natively while keeping runtime compute down to 1.4B active parameters. For developers building visual agents that need to inspect diagrams, make precise edits, and output fresh visual assets in one loop, this is one of the most flexible unified models available right now. Setting up the dependencies takes some effort, but having both NVIDIA and Ascend weights fully open makes it a standout foundation for future agentic workflows.