Character.ai reveals early pretraining secrets, including 6-bit Squinch
Character.ai just made public the internal engineering techniques that powered its early pretraining cluster, originally spearheaded by co-founder Noam Shazeer. The headliner is Squinch, a 6-bit gradient compression algorithm that matched bfloat16 fidelity despite operating on a fraction of modern network bandwidth. Why it matters: These bandwidth-saving tricks offer valuable architectural blueprints for anyone training sparse MoE models or fine-tuning across bandwidth-constrained clusters.