ISTA-DASLab fits 763B DeepSeek-V4.1 onto a 12GB GPU

The open-source DeepSeek-V4.1-Flash-1017 GSQ-RCO quantizations from ISTA-DASLab let you run a 763B-parameter MoE model locally on a single consumer GPU with just 12GB of VRAM.

The repository provides standard GGUF builds at three precision targets: IQ1_M (1.75 bits per weight), IQ2_XXS (2.00 bpw), and IQ2_XS (2.25 bpw). Together with the Strata inference engine, local frontier-class intelligence no longer requires a rack of datacenter accelerators.

Squeezing 763B into 12GB VRAM

Traditional uniform quantization applies the same bit-width across all layers, which usually results in severe quality collapse around 2 bits.

To avoid this cliff, the team combined two methods. Gumbel-Softmax Quantization (GSQ) optimizes per-coordinate scalar grids while keeping the outputs compatible with standard GGUF layouts. Riemannian Constrained Optimization (RCO) then treats the total parameter budget as a manifold constraint, running a gradient search directly on task loss to allocate individual bit-widths across tensors based on their sensitivity.

Because DeepSeek-V4.1-Flash places most of its 763B parameters into 384 routed experts per layer, RCO pushes the experts down into aggressive IQ1 and IQ2 formats while preserving sensitive dense layers at higher precisions like Q5 and Q8.

Memory management does the rest of the heavy lifting. The model's 111GB Engram lookup table is packaged as a shared standalone file. Because it only requires sparse row lookups per token, it can remain memory-mapped directly on an SSD, using zero VRAM and zero system RAM. Paired with Strata and 46GB to 64GB of system memory, the model runs on a single 12GB VRAM card. Community reports note inference speeds of around 8 to 10 tokens per second on an RTX 3060 running the 2.00 bpw build.

Zero-loss performance at 2 bits

Despite the aggressive compression, benchmark numbers barely budge. Evaluated at reasoning effort 50 against the official FP8/FP4 base checkpoint, the models show remarkable retention across AIME25, GPQA-Diamond, and LiveCodeBench v6.

The recommended IQ2_XXS build scores 100.00 on AIME25, 90.91 on GPQA-Diamond, and 89.71 on LiveCodeBench v6. It matches the base model on all three evaluations, achieving a 100.0% task recovery score. Even the 1.75 bpw IQ1_M build preserves a 100.00 on AIME25 and retains 98.7% overall average performance.

The release also provides a multimodal vision projector file (mmproj) and embeds reasoning effort controls alongside verified Jinja chat templates.

Getting it running

All weights and conversion tools are publicly available on Hugging Face as standard GGUF files.

The easiest path is using the Strata engine. After running the one-line installer script, launching with the ist-das-2.00bpw flag downloads the roughly 249GB payload and spins up an OpenAI-compatible endpoint on localhost.

Users running llama.cpp will need build b11870 or newer. Download your chosen build shards along with the shared Engram file, create the required symlink for shard 6, and start llama-server or llama-cli as usual. For users who fetched earlier files prior to October 27, a provided script converts existing weights in place without requiring redownloads.

Why it matters

Running a 700B-plus parameter flagship locally used to require tens of thousands of dollars in server hardware. Non-uniform quantization combined with smart disk paging shatters that barrier, dragging elite reasoning capabilities down to everyday GPUs like the RTX 3060.

For independent developers and privacy-conscious builders, this makes genuine frontier experimentation possible on ordinary workstations. Disk space demands and memory-bandwidth speed trade-offs still apply, but this release proves that extreme low-bit compression no longer has to come at the expense of frontier reasoning.