Character.ai reveals early pretraining secrets, including 6-bit Squinch

What happens when your AI training cluster only has a quarter of the network bandwidth of state-of-the-art systems?

Back when Character.ai was still pretraining its own foundation models, cofounder Noam Shazeer and his team had to get absurdly creative. Now that the startup has pivoted toward fine-tuning open-source models, it just opened the vault on five core pretraining techniques that kept them competitive.

The Squinch hack and precision tricks

The headline innovation is Squinch, a 6-bit gradient compression algorithm invented by Shazeer. Running on severely bandwidth-constrained hardware, the team quantized gradients into 48-bit blocks containing eight values each, effectively cutting transmission down to 6 bits per element.

Because Squinch was tuned specifically to Transformer gradient distributions, it preserved bfloat16-level accuracy while slashing communication overhead. The concept later inspired similar techniques elsewhere, including DeepSeek's logfmt.

To keep numbers from breaking, the team developed Attention Z-Reg. Since bfloat16 loses precision at large magnitudes, they applied regularization directly to attention backward passes to keep logits near zero.

They also introduced Dynamic Clamping for quantization-aware training. Instead of static clamping thresholds that collapse small activations to zero, the model dynamically scales limits based on weight RMS values to keep training stable.

Packed attention and cheaper distillation

For chat workloads, Character.ai built Visibility Mask, using just two boundary tensors to map which tokens can attend to each other. This allowed native handling of tree-structured conversations, causal masks, and inference beam searches in single packed batches.

Finally, for knowledge distillation, storing full teacher probability distributions across giant vocabularies was unfeasible. The team applied Gumbel top-k subsampling to save offline teacher targets without biasing the student model during training.

Why it matters

For engineering teams training models on budget clusters or across distributed nodes, these techniques are pure gold. Network bandwidth is usually the hardest ceiling in distributed training, and Squinch proves you can compress gradients aggressively without sacrificing accuracy.

It also provides a fascinating look into the early days of generative AI startups. Before Google rehired Shazeer and Character.ai abandoned frontier pretraining, extreme engineering allowed a small team to compete with tech giants despite having far worse infrastructure.

We think releasing this playbook now is a smart nod to the community. While frontier pretraining is increasingly centralized among trillion-dollar labs, tricks like gradient compression and dynamic clamping will remain essential for researchers running MoE architectures and open-source models on modest hardware.