Training Data

Training data shapes what a model learns. We build and document open corpora so researchers can inspect that foundation and study how changing it changes a model.

The Pile brought together diverse sources of text in an 800GB language-modeling dataset. The Common Pile extends our work on open training data with an 8TB collection of public domain and openly licensed text.

Releasing a corpus also means documenting where its contents came from and how they were selected. This connects dataset construction to research on filtering, licensing, and the effects of data composition.

Selected Papers

Newest first. Showing 12 of 41.

Training Data

Selected Papers

Can Data Attribution Filter Out Subliminal Learning? Not Reliably

Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs

Bergson: An Open Source Library for Data Attribution

Where Does Social Reasoning Come From? Capability Provenance in Language Models

The Unequal Influence of Bad Advice: Using Training Data Attribution to Modulate Emergent Misalignment

Last Translation Benchmark

Quantifying the Effect of Test Set Contamination on Generative Evaluations

CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data

Concept Influence: Leveraging Interpretability to Improve Performance and Efficiency in Training Data Attribution

Mitigating Emergent Misalignment with Data Attribution

The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text

Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenon