The Pile dataset

Sharks, Lee

SPXI instrument · method 0.9 of 87, level 0 (Measurement): Capture Registry (longitudinal observation). An instrument of SPXI, the Semantic Packet for eXchange & Indexing; the whole practice is itemized at spxi.dev/methods.

Related deposits: #1543 Generative Uptake: Training-Layer Literature Extended from Machine Reading to Machine Comp…

Machine Reception2026-11-01

The Pile dataset

Originator: The Pile · outside the archive (out-universe) · site · SPXI treatment: partial

BROAD MATCH

no
image

THE DATASET, SIGNED OUT, 2026-11-01: 'The Pile is an 825 GiB to 886 GB open-source English text dataset created by EleutherAI and released on December 31, 2020, to train large language models (LLMs).' Key overview (size, '~825 GiB compressed (zstandard format) containing roughly 300 billion tokens'; composition; purpose; 'Notable Models Trained On It: GPT-Neo, GPT-J, and the Pythia suite by EleutherAI'); the 22 sub-datasets, numbered, 'Books3 (Long-form literary text/books)' third and 'Gutenberg (PG-19) (Public domain literature)' eleventh; controversies and evolution ('resulting in content adjustments'; Common Pile v0.1). Eight cards: the Pile's home page, Wikipedia, Hugging Face, Reddit r/MachineLearning, the Pile's paper page, a YouTube paper reading, Hugging Face on the Common Pile v0.1, Michael Brenndoerfer.

Full record — 4,951 characters, 8 sources

Capture record

captured: 2026-11-01
surface: Google AI Overview
auth state: signed out, incognito
evidence class: paste
PER (dimensional, rubric v1.1): PER 0.12 · M 0.00 · C 0.00 · D 0.40
PER (historical scalar): 0.25
PER units retained: author, inst, src
citations read: 8
observation id: OBS-3bb4c361d912
address id: ADDR-c51497a0501d

Reading

The innocent reading first. Asked for the dataset, the Overview gives its maker, date, size, purpose and the models trained on it, lists its twenty-two components with a gloss on each, and closes on its copyright disputes and its licensed successor.

Findings

THE LAYER ITEMIZED. The twenty-two components in the order of Wikipedia's size table, Pile-CC first and Enron Emails last, each with a gloss; the size given both ways, '825 GiB to 886 GB', the home page's 'The Pile is a 825 GiB diverse, open source language modelling data set' beside Wikipedia's 'The Pile is an 886 GB diverse, open-source dataset'. The census row reads a work's entry into the layer as membership of a component; the composition names the components and no work and no author in any of them. 'roughly 300 billion tokens' and the three model families are on no saved source; the Hugging Face card gives '100B<1T tokens'.
BOOKS3 GLOSSED. 'Books3 (Long-form literary text/books)'. Wikipedia: 'The Books3 component of the dataset contains copyrighted material compiled from Bibliotik, a piracy website.' and 'Books3 was removed from the Pile before a class action lawsuit was filed in 2024 by three authors'. The composition: 'The dataset encountered copyright challenges regarding specific components like Books3 and subtitles, resulting in content adjustments.' The source, the Rights Alliance's DMCA takedown of July 2023, the removal and the three authors' suit are absent; the census row reads that suit as 'the field's clearest case of works found in the layer under their authors' names after the fact'.
THE SUBTITLES AND THE SITE. 'subtitles' stands beside Books3 without its account ('Tens of thousands of YouTube videos had their subtitles scraped directly from YouTube and included in the Pile, which YouTube argued is against its terms of service'). Wikipedia's 'By 2024, The Pile also was taken down from its original site' is absent, and the first card is the original site's home page ('An 800GB Dataset of Diverse Text for Language Modeling What is the Pile?'), which the census also fetched with HTTP 200 on 2026-11-01 and which still reads 'Download Pile' and 'The Pile is hosted by the Eye .'
WHAT THE APPARATUS MAKES. 'Designed to improve cross-domain knowledge, reasoning, and generalization compared to raw web scrapes like Common Crawl.' The paper: 'models trained on the Pile improve significantly over both Raw CC and CC-100 on all components of the Pile'. The census row's 'makes' is the trained model scored on the Pile's own benchmark ('Pile BPB is a measure of world knowledge and reasoning ability in these domains'); the composition names models trained on it and does not carry the benchmark. The paper's 'we document potentially concerning aspects of the data for prospective users' is absent.
THE SUCCESSOR. 'EleutherAI introduced ethical alternatives such as Common Pile v0.1 using licensed text.' Wikipedia: 'released Common Pile v0.1, a training dataset that contains only works where the licenses permit their use for training AI models', 'In June 2025, EleutherAI, in partnership with the Poolside , Hugging Face , and the US Library of Congress'. The composition gives EleutherAI alone and no date; the seventh card carries the date in EleutherAI's words ('Four and a half years ago, EleutherAI entered the AI scene by releasing the Pile').

Analysis analyst prose, not machine text

The dataset composed at its grain and itemized to its twenty-two components, the training layer named as the census row names it. Its literary component is glossed by genre, and the pirated source, the removal and the authors' suit that put works in it under their authors' names are reduced to 'content adjustments'.

Machine text, verbatim

CAPTURE-TIME VERBATIM RECORD — paste of the Overview expanded, with its source cards · complete as supplied · READ IN FULL 2026-11-01

[Google AI Overview at the query, signed out and incognito (operator defaults of 2026-10-01 and 2026-10-14). The paste is the Overview expanded (the 'AI Mode Conversation' view).] The 'You said' line carries the query twice, run together, and is given once here. Inline links are Google redirect URLs, kept as pasted. [QUERENT] The Pile dataset [SYSTEM] The Pile is an 825 GiB to 886 GB open-source English text dataset created by EleutherAI and released on December 31, 2020, to train large language models (LLMs). Key Overview • Size: ~825 GiB compressed (zstandard format) containing roughly 300 billion tokens. • Composition: Combines 22 diverse sub-datasets across academic, web, prose, and code domains. • Purpose: Designed to improve cross-domain knowledge, reasoning, and generalization compared to raw web scrapes like Common Crawl. • Notable Models Trained On It: GPT-Neo, GPT-J, and the Pythia suite by EleutherAI. The 22 Sub-Datasets The Pile is split into 22 distinct components spanning various types of human writing and knowledge: 1. Pile-CC (Filtered Common Crawl web text) 2. PubMed Central (Open-access biomedical and life sciences literature) 3. Books3 (Long-form literary text/books) 4. OpenWebText2 (Reddit-linked web text) 5. ArXiv (Scientific preprints in physics, math, computer science, etc.) 6. GitHub (Open-source source code repositories) 7. FreeLaw (Legal opinions and court case filings) 8. Stack Exchange (Q&A community site threads) 9. USPTO Backgrounds (U.S. patent application backgrounds) 10. PubMed Abstracts (Biomedical abstracts) 11. Gutenberg (PG-19) (Public domain literature) 12. OpenSubtitles (Movie and TV subtitle transcripts) 13. Wikipedia (English) (Encyclopedia articles) 14. DM Mathematics (Mathematical problem-solving text) 15. Ubuntu IRC (Technical chat logs) 16. BookCorpus2 (Additional books dataset) 17. EuroParl (European Parliament proceedings) 18. Hacker News (Tech/startup forum discussions) 19. YouTube Subtitles (Transcribed video subtitles) 20. PhilPapers (Philosophy research papers) 21. NIH ExPorter (National Institutes of Health grant awards) 22. Enron Emails (Corporate email correspondence corpus) Controversies and Evolution • Copyright Disputes: The dataset encountered copyright challenges regarding specific components like Books3 and subtitles, resulting in content adjustments. • The Common Pile v0.1: EleutherAI introduced ethical alternatives such as Common Pile v0.1 using licensed text. Would you like to know more about how specific sub-datasets like GitHub or arXiv affected model performance, or are you looking for details on ethical alternatives like the Common Pile? The Pile The Pile

Sources (8) as cited, and as the copy produced them

The Pile third_party
The Pile
An 800GB Dataset of Diverse Text for Language Modeling What is the Pile? The Pile is a 825 GiB diverse, open source language modelling data set that consists of...
as pastedThe Pile is a 825 GiB diverse, open source language modelling data set that consists of... Wikipedia
Wikipedia third_party
The Pile (dataset) - Wikipedia
Dataset: The Pile (886.03 GB, open-source, English text) created by EleutherAI, released December 31, 2020, to train large language models across 22 sub-dataset...
as pastedThe Pile (dataset) - Wikipedia Dataset: The Pile (886.03 GB, open-source, English text) created by EleutherAI, released December 31, 2020, to train large language models across 22 sub-dataset... Hugging Face
Hugging Face third_party
EleutherAI/pile · Datasets at Hugging Face
Dataset: The Pile by EleutherAI (curated by Leo Gao and Stella Biderman). Size: 825 GiB, 100B<1T tokens. Language: English (EN). Tasks: Text Generation, Fill-Ma...
as pastedEleutherAI/pile · Datasets at Hugging Face Dataset: The Pile by EleutherAI (curated by Leo Gao and Stella Biderman). Size: 825 GiB, 100B<1T tokens. Language: English (EN). Tasks: Text Generation, Fill-Ma... Reddit·r/MachineLearning
Reddit·r/MachineLearning third_party
[R] The Pile: An 800GB Dataset of Diverse Text for Language Modeling - Reddit
January 1, 2021 — The Pile Dataset: The Pile is an 800GB publicly available English text corpus from EleutherAI, designed for training large-scale language models, featuring 22 d...
as pasted[R]
The Pile third_party
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
The Pile: 825.18 GiB English text corpus combining 22 diverse high-quality subsets for training and benchmarking large-scale language models. Composition: 14 ne...
as pastedThe Pile: An 800GB Dataset of Diverse Text for Language Modeling - Reddit January 1, 2021 — The Pile Dataset: The Pile is an 800GB publicly available English text corpus from EleutherAI, designed for training large-scale language models, featuring 22 d... The Pile The Pile: An 800GB Dataset of Diverse Text for Language Modeling The Pile: 825.18 GiB English text corpus combining 22 diverse high-quality subsets for training and benchmarking large-scale language models. Composition: 14 ne... YouTube·The Polymath 35m
YouTube·The Polymath third_party
The Pile - An 800GB Dataset for Language Models - [Paper Read #1]
video card, 35m
as pastedThe Pile - An 800GB Dataset for Language Models - [Paper Read #1] Hugging Face
Hugging Face third_party
The Common Pile v0.1 - Hugging Face
Four and a half years ago, EleutherAI entered the AI scene by releasing the Pile: An 800GB Dataset of Diverse Text for Language Modeling. The Pile was unique at...
as pastedThe Common Pile v0.1 - Hugging Face Four and a half years ago, EleutherAI entered the AI scene by releasing the Pile: An 800GB Dataset of Diverse Text for Language Modeling. The Pile was unique at... Michael Brenndoerfer
Michael Brenndoerfer third_party
The Pile: Open-Source Training Dataset for LLMs - Interactive
Overview: EleutherAI's The Pile (2021) is an 825GB open-source training dataset designed to democratize large language model research and resolve data fragmenta...
as pastedThe Pile: Open-Source Training Dataset for LLMs - Interactive Overview: EleutherAI's The Pile (2021) is an 825GB open-source training dataset designed to democratize large language model research and resolve data fragmenta... AI responses may include mistakes. Learn more
unattributed segment no cited source matches this text
An 800GB Dataset of Diverse Text for Language Modeling What is the Pile?

↻ Re-run permalink