[Google AI Overview at the query, signed out and incognito (operator defaults of 2026-10-01 and 2026-10-14). The paste is the Overview expanded (the 'AI Mode Conversation' view).] The 'You said' line carries the query twice, run together, and is given once here. Inline links are Google redirect URLs, kept as pasted.

[QUERENT] The Pile dataset

[SYSTEM] The Pile is an 825 GiB to 886 GB open-source English text dataset created by EleutherAI and released on December 31, 2020, to train large language models (LLMs).

Key Overview

• Size: ~825 GiB compressed (zstandard format) containing roughly 300 billion tokens.
• Composition: Combines 22 diverse sub-datasets across academic, web, prose, and code domains.
• Purpose: Designed to improve cross-domain knowledge, reasoning, and generalization compared to raw web scrapes like Common Crawl.
• Notable Models Trained On It: GPT-Neo, GPT-J, and the Pythia suite by EleutherAI.

The 22 Sub-Datasets

The Pile is split into 22 distinct components spanning various types of human writing and knowledge:
1. Pile-CC (Filtered Common Crawl web text)
2. PubMed Central (Open-access biomedical and life sciences literature)
3. Books3 (Long-form literary text/books)
4. OpenWebText2 (Reddit-linked web text)
5. ArXiv (Scientific preprints in physics, math, computer science, etc.)
6. GitHub (Open-source source code repositories)
7. FreeLaw (Legal opinions and court case filings)
8. Stack Exchange (Q&A community site threads)
9. USPTO Backgrounds (U.S. patent application backgrounds)
10. PubMed Abstracts (Biomedical abstracts)
11. Gutenberg (PG-19) (Public domain literature)
12. OpenSubtitles (Movie and TV subtitle transcripts)
13. Wikipedia (English) (Encyclopedia articles)
14. DM Mathematics (Mathematical problem-solving text)
15. Ubuntu IRC (Technical chat logs)
16. BookCorpus2 (Additional books dataset)
17. EuroParl (European Parliament proceedings)
18. Hacker News (Tech/startup forum discussions)
19. YouTube Subtitles (Transcribed video subtitles)
20. PhilPapers (Philosophy research papers)
21. NIH ExPorter (National Institutes of Health grant awards)
22. Enron Emails (Corporate email correspondence corpus)

Controversies and Evolution

• Copyright Disputes: The dataset encountered copyright challenges regarding specific components like Books3 and subtitles, resulting in content adjustments.
• The Common Pile v0.1: EleutherAI introduced ethical alternatives such as Common Pile v0.1 using licensed text.
Would you like to know more about how specific sub-datasets like GitHub or arXiv affected model performance, or are you looking for details on ethical alternatives like the Common Pile?
The Pile
The Pile
An 800GB Dataset of Diverse Text for Language Modeling What is the Pile? The Pile is a 825 GiB diverse, open source language modelling data set that consists of...
Wikipedia
The Pile (dataset) - Wikipedia
Dataset: The Pile (886.03 GB, open-source, English text) created by EleutherAI, released December 31, 2020, to train large language models across 22 sub-dataset...
Hugging Face
EleutherAI/pile · Datasets at Hugging Face
Dataset: The Pile by EleutherAI (curated by Leo Gao and Stella Biderman). Size: 825 GiB, 100B<1T tokens. Language: English (EN). Tasks: Text Generation, Fill-Ma...
Reddit·r/MachineLearning
[R] The Pile: An 800GB Dataset of Diverse Text for Language Modeling - Reddit
January 1, 2021 — The Pile Dataset: The Pile is an 800GB publicly available English text corpus from EleutherAI, designed for training large-scale language models, featuring 22 d...
The Pile
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
The Pile: 825.18 GiB English text corpus combining 22 diverse high-quality subsets for training and benchmarking large-scale language models. Composition: 14 ne...
YouTube·The Polymath
35m
The Pile - An 800GB Dataset for Language Models - [Paper Read #1]
Hugging Face
The Common Pile v0.1 - Hugging Face
Four and a half years ago, EleutherAI entered the AI scene by releasing the Pile: An 800GB Dataset of Diverse Text for Language Modeling. The Pile was unique at...
Michael Brenndoerfer
The Pile: Open-Source Training Dataset for LLMs - Interactive
Overview: EleutherAI's The Pile (2021) is an 825GB open-source training dataset designed to democratize large language model research and resolve data fragmenta...
AI responses may include mistakes. Learn more