HRM-Text pretrains a 1B reasoning model for $1,000

Pretraining a capable foundation model from scratch usually requires millions of dollars in compute, making it a playground reserved exclusively for deep-pocketed tech giants. But what if you could train a high-performing 1B reasoning model for roughly the cost of a flagship smartphone?

Sapient Intelligence open-sourced HRM-Text on GitHub under an Apache 2.0 license. The repository delivers a complete pretraining framework that allows developers to train a 1-billion-parameter text reasoning model from scratch for around $1,000 in compute budget.

Cutting compute by 600x with recurrent reasoning

The project skips brute-force data ingestion in favor of an architecture inspired by the human brain. HRM-Text uses a hierarchical recurrent latent structure designed to perform multi-step task completion directly in latent space.

Sapient claims this approach slashes required pretraining compute by 130x to 600x and shrinks the necessary training data by 150x to 900x. The model was trained on roughly 40 billion structured tokens, which is about 1/1,000th of the data footprint seen in typical contemporary base models.

The math on cloud compute is straightforward. Based on an estimated $2 per H100 hour, the 0.6B version runs on a single node of 8 H100 GPUs for 50 hours, costing about $800. The 1B version requires two nodes with 16 H100 GPUs running for 46 hours, tallying roughly $1,472.

Performance does not seem compromised by the lower cost. In reference runs, the 1B model achieved an 84.7% score on GSM8k, 56.5% on MATH, 82.3% on DROP, and 60.7% on MMLU. For a model with only 1 billion parameters trained on such a compact corpus, these reasoning numbers punch well above its weight class.

Solid tooling with a strict hardware catch

The engineering pipeline is surprisingly turnkey. It ships with PrefixLM sequence packing, FlashAttention 3 kernels, PyTorch FSDP2 distributed training, evaluation engines, checkpoint converters, and a prebuilt Docker environment.

However, there is an unavoidable hardware catch. Because the attention mechanism depends directly on FlashAttention 3, the training pipeline requires Hopper-class GPUs like NVIDIA's H100. Consumer hardware and older datacenter chips such as the A100 are left out of the pretraining flow.

Evaluating the checkpoints also typically requires an 80GB GPU. On the ecosystem side, native integration into Hugging Face Transformers is merged for the upcoming release, while native vLLM support is currently work in progress.

Why it matters

We think HRM-Text matters because it turns base model pretraining from an institutional privilege into an accessible weekend project for small labs and independent developers.

Up to now, smaller teams were essentially forced to rely on fine-tuning someone else's base weights. If HRM-Text's recipe holds true, any team with specialized, high-quality structured data can pretrain a domain-specific reasoning model for under two thousand dollars without asking for VC checks or computing grants.

For the wider industry, this project is a refreshing challenge to the pure scaling law dogma. It suggests that architectural creativity, specifically trading raw sequence length for recurrent latent depth, can extract massive reasoning capabilities out of a microscopic 40-billion-token dataset.

The real test will be whether this hierarchical recurrent design can scale gracefully to 7B or 70B parameters without hitting architectural bottlenecks. Its tight dependence on Hopper GPUs also limits broad hobbyist experimentation right now. Still, proving that a $1,000 bill can produce an 84.7% GSM8k model changes the baseline conversation for what efficient pretraining looks like.