GAE puts 3D geometry inside the latent space

GAE (Geometry Native Autoencoder) is an open-source visual generation framework built to solve the consistency problem in AI video. Instead of just producing flat pixels that warp when the camera moves, GAE generates video alongside an accurate 3D depth map, camera trajectory, and point cloud from the very same representation.

Most video generators treat geometry as an afterthought, relying on appearance-only latents that easily drift. GAE bakes 3D structure directly inside the latent space from day one, forcing the generator to stay grounded in physical space.

What it does

GAE generates 3D-consistent scenes across indoor and outdoor environments, supporting both first-person and third-person viewpoints.

Starting from a single conditioning frame and a metric camera path, it jointly outputs an 81-view RGB video sequence at 672 x 378 resolution along with its underlying 3D geometry state. Users can watch the video while simultaneously viewing the depth sequence and progressive 3D reconstruction.

It also supports text-to-image workflows. From a single text prompt, GAE decodes the final latent into an RGB picture, a matching depth map, and an interactive 3D point cloud.

In controlled benchmarks on RealEstate10K, replacing standard latents with GAE cut camera trajectory error in half. Video quality improved as well, with FVD dropping by 12.7% on RealEstate10K and 23.1% on DL3DV.

How it works

GAE reparameterizes multi-level features from a frozen geometry foundation model into a compact latent space. It compresses heavy 3,072-channel geometric features into a structured 128-channel state without losing decodability.

The team designed two variants: GAE-64 prioritizes compact generation and camera path consistency, while GAE-128 preserves fine reconstruction details and cross-view correspondence.

A conditional flow matching model generates latents directly within this geometric space. Because the latent naturally encodes cross-view correspondence, dedicated output heads can decode appearance, depth, camera pose, and point clouds straight from the final state.

Under identical DiT training protocols, GAE outperformed representations like Stable Diffusion VAE, Wan2.1 video VAE, and RAEv2 in both visual quality and 3D geometric stability.

Hardware specs and code access

GAE is fully open source. The entire model weight footprint sits under 12GB, meaning you can run it locally on mid-range consumer GPUs.

The project page includes the complete paper, local installation and inference guides, along with full training and evaluation scripts.

Developers can clone the repository, download the weights directly via the project link, and reproduce the walkthrough demos locally.

Why it matters

If you work on game environments, robotics simulation, or 3D world creation, GAE is an immediate download. Traditional 3D generation pipelines require chaining multiple clunky tools together, whereas GAE packs end-to-end geometry decoding into a sub-12GB footprint that fits on normal consumer hardware.

We think the real breakthrough here is architectural. Instead of throwing more compute at cosmetic video diffusion, GAE fixes the representation problem at the latent level. Turning geometry into a native primitive shows that a compact latent can serve as a shared bridge between perception and generative AI.