Apple open-sources DCLM models beating Mistral and Hugging Face

Apple's ML team, alongside researchers from the University of Washington, Tel Aviv University, and Toyota Research Institute, just dropped DCLM—a suite of open-source language models that hit way above their weight class.

The release features two primary models: a 7B parameter version and an ultra-compact 1.4B variant. Unlike many releases that keep the secret sauce hidden, Apple and its collaborators went truly open source by sharing the model weights, training code, and the entire pretraining dataset on Hugging Face.

How they pulled it off

The project revolves around a smart data curation method called model-based filtering. Instead of just throwing raw web data into the training pipeline, machine learning models automatically scrubbed and cherry-picked the highest-quality tokens from massive datasets, creating the DCLM-Baseline set.

The 7B model was trained from scratch on 2.5 trillion tokens using the OpenLM framework, sporting an initial 2K context window. On the standard 5-shot MMLU benchmark, it notched an impressive 63.7% accuracy.

That puts it well ahead of the prior open-data champion MAP-Neo by 6.6 percentage points, all while consuming 40% less compute during training. It also outperforms Mistral-7B-v0.3 (62.7%) and inches dangerously close to closed-data giants like Google's Gemma (64.3%) and Llama 3 8B (66.2%). The team even pushed its context window to 8K through an additional 100B token training run.

The tiny 1.4B packs a punch

The smaller 1.4B version, developed with the Toyota Research Institute, was trained on 2.6 trillion tokens and punched equally hard. It scored 41.9% on MMLU, easily surpassing Hugging Face's recent SmolLM 1.7B (39.97%), Qwen-1.5B (37.87%), and Microsoft's Phi-1.5 (35.90%).

In terms of licensing, the 1.4B model is shipped under Apache 2.0, making it ready for commercial deployment, while the 7B version comes under Apple's Sample Code License. Both models and an instruction-tuned 7B build can be grabbed directly on Hugging Face.

Keep in mind that this is foundational research rather than polished Apple Intelligence software for consumer hardware, so expect typical raw model quirks and potential training biases.

Why it matters

For AI researchers and data engineers, DCLM is a goldmine. While tech giants usually treat pretraining datasets like state secrets, Apple literally handed over the entire blueprint along with the recipe that trims 40% off compute costs.

It proves that data quality beats sheer brute force, even in the highly competitive 7B arena. If you are building lean models or studying pretraining efficiency, checking out DCLM's pipeline is a no-brainer.