Run 100B+ models on a 12GB GPU with Strata

Strata is a new open-source local inference engine designed to do one specific job: let you run massive 100B+ parameter models locally on a regular consumer PC with just 12GB of VRAM.

Released under the MIT license, Strata runs fully offline without any accounts, offering one-click installers for Windows, Linux, and macOS.

How the tiered engine works

Running models past 100B parameters usually demands enterprise-grade hardware clusters. Strata bypasses standard VRAM limits using a three-tier offloading architecture.

For mixture-of-experts architectures, Strata keeps the most frequently used expert weights pinned in GPU memory, stages the wider model in system RAM, and reads remainder layers directly from NVMe solid-state storage.

Your GPU, system RAM, and SSD work together in real time to handle execution and speculative decoding across the pipeline.

Supported models and hardware speeds

Strata prioritizes models like the 125B parameter Qwen3.8-Flash-Next, Qwen3.6-35B-A3B, and DeepSeekV4.1-Flash-1017.

Running the full Qwen3.8-Flash-Next requires 12GB VRAM combined with 64GB of RAM. Computers with 32GB of RAM can run the dedicated Coder variant using IQ1_M quantization.

For DeepSeekV4.1-Flash-1017, unallocated weights stream directly from your drive, requiring between 245GB and 280GB of free SSD storage depending on your quant selection.

Hardware compatibility is extensive. It supports NVIDIA RTX 20, 30, 40, and 50 series GPUs, AMD Radeon RX 7000, 9000, and AI PRO R9700 cards, Intel Arc on Linux, and Apple Silicon Macs with 64GB or more RAM.

In official benchmarks running DeepSeek at 2.00 bpw, an everyday RTX 3060 with 46GB RAM outputs 9 to 11 tokens per second. An RTX 5070 with 64GB RAM clocks 4 to 6 tokens per second, while an AMD Ryzen AI Max Strix Halo laptop hits roughly 19 to 21 tokens per second for coding tasks.

Setup and third-party tooling

Getting started takes minimal effort. Strata offers standalone installers for Windows, Linux, and macOS, alongside simple curl and PowerShell setup scripts for the terminal.

Running strata launch guides you through model selection and runtime downloads. A built-in dashboard monitors memory, temperatures, and hardware load in real time.

Crucially, Strata exposes local endpoints fully compatible with OpenAI and Anthropic APIs. Running strata connect links your local engine straight into workflow tools like Claude Code, Codex CLI, OpenCode, and Hermes Agent.

Why it matters

For developers who need private code generation or continuous agentic workflows without racking up API charges, Strata turns sub-$1000 desktop components into viable local inference servers.

It is not going to replace a full H100 datacenter rack for blistering speed, and your storage throughput will ultimately dictate response times. But by making 100B+ class intelligence genuinely usable on everyday hardware, Strata pushes local computing firmly past the lightweight model barrier.