EleutherAI drops massive 8TB dataset of licensed and public domain text
EleutherAI just released The Common Pile v0.1, an 8-terabyte AI training dataset built entirely from licensed and public domain text over two years. Alongside it, the team dropped two 7B Comma models that perform on par with early Llama models, pushing back on the industry narrative that frontier AI requires unpermitted scraping.
The Common Pile v0.1 is an 8-terabyte open dataset built specifically for pretraining large language models without copyright liability. If you need a clean, legally vetted foundation to train models from scratch, this is one of the largest open collections of licensed and public domain text available today.
The dataset took roughly two years to curate. EleutherAI led the initiative in collaboration with Hugging Face, Poolside, the University of Toronto, and several other academic and industry partners.
Where the 8TB of data comes from
The team built the dataset in consultation with legal experts, restricting content strictly to public domain works and explicitly licensed sources. A core pillar includes 300,000 public domain books digitized by the Library of Congress and the Internet Archive.
Beyond digitized print, EleutherAI tapped OpenAI's open source Whisper speech-to-text model to transcribe audio archives into high-quality training text.
The project marks a deliberate shift from EleutherAI's earlier dataset, The Pile, which included unpermitted copyrighted text and later became a frequent target of copyright lawsuits against downstream AI developers.
How the models hold up on benchmarks
To prove that clean data does not require a performance penalty, the team trained two 7-billion-parameter models, Comma v0.1-1T and Comma v0.1-2T, on only a slice of the Common Pile.
Across standard benchmarks for coding, math, and image understanding, both 7B Comma models match the performance of Meta's original Llama 7B. EleutherAI executive director Stella Biderman noted that the prevailing belief that unpermitted scraping is essential for strong benchmark performance simply does not hold up.
Both the complete The Common Pile v0.1 dataset and the Comma model weights are available to download right now on Hugging Face and GitHub. EleutherAI plans to ship open datasets more frequently alongside its infrastructure partners going forward.
Why it matters
For enterprise teams, labs, and solo builders who cannot stomach the legal risk of unverified web scrapes, this release delivers an immediate, drop-in training corpus. You get clear provenance without spending millions on content licensing deals.
Matching first-generation Llama at 7B is not beating today's state-of-the-art models, but that is missing the point. It completely demolishes the excuse that fair use violations are the only way to build competitive foundation models, giving the open source community a clean slate to build upon.
Sources (1)
- techcrunch.com(opens in a new tab)Originaltechcrunch.com