Edge0 runs 35B models in 2.5 GB of RAM by streaming from SSD

Edge0 is an open-source streaming MoE inference framework that treats your device's solid-state drive like active memory. Instead of cramming an entire massive model into RAM, it lets you run a 35B parameter mixture-of-experts model on consumer hardware like Macs and iPhones using only around 2.5 GB of active memory.

The project supports macOS, iOS, Android, and Windows right out of the gate. It ships with two tuned model tiers on Hugging Face and ModelScope: a 35B tier based on Qwen3.6-35B-A3B and an 8B tier based on Ling 3.0.

How it works

Standard inference forces you to load every single weight into system memory upfront. Sparse MoE models, however, only use a handful of expert networks at any given token. In a 35B model, only around 3B parameters are actually active during a calculation.

Edge0 capitalizes on this by streaming expert weights off storage on demand via mmap. Peak memory consumption is determined purely by the active expert set rather than the total parameter count.

To prevent the forward pass from stalling while waiting for disk reads, Edge0 introduces a lightweight prerouter head. This module predicts the routing decision one step ahead, allowing expert loads to overlap with live computation and boosting decode throughput by up to 59%.

The base checkpoints are quantized to INT4, cutting the 35B model's footprint on disk to roughly 23 GB. To patch the accuracy loss from 4-bit quantization, Edge0 loads unmerged Recover-LoRA adapters trained via distillation. In OpenCompass benchmarks, the 35B model dropped just 3.9 points on average compared to its original FP16 base, while actually scoring slightly higher on MMLU-Pro.

Specs and performance

Tested on an M4 Pro Mac mini with 24 GB of RAM, the edge0-35b tier delivers 14.9 to 17.7 tokens per second during decode, while peak MLX memory allocation sits at around 2.9 GB.

The smaller edge0-8b model requires just 4.2 GB of disk storage and hits 23.9 to 25.3 tokens per second, consuming only 1.0 GB of peak memory.

The repository already includes platform-native engines across macOS (Rust), iOS (Swift and MLX Swift), Android (Kotlin), and Windows (C++ and Vulkan). A unified cross-platform inference framework is scheduled for Q4 2026.

For now, the Python framework runs on Apple Silicon Macs using MLX and Python 3.10 or higher. A dedicated CUDA backend remains on the roadmap.

How to run it

Getting started on a Mac is straightforward. Clone the Edge0-AI/Edge0 repository on GitHub, set up a Python 3.12 virtual environment, and install the package with its dependencies.

You can pull the models directly using the repository's helper script or the Hugging Face CLI. Each model folder includes the base checkpoint alongside its dedicated LoRA and prerouter weights.

Once downloaded, run edge0 demo edge0-35b to start an immediate chat session in your terminal, or launch edge0 serve edge0-35b to spin up a local OpenAI-compatible API endpoint.

Why it matters

Hardware vendors have long charged massive premiums for the unified memory required to host capable local models. Edge0 shows that clever predictive routing can bridge the latency gap of consumer storage, making high-quality models usable on machines previously locked out by memory limits.

If you want to run capable reasoning models offline on everyday laptops or phones, this is definitely worth testing today. Keep in mind that long context windows will still inflate your KV cache, but as a blueprint for local inference efficiency, Edge0 is one of the smartest releases we have seen recently.