{ "spxi_instrument": { "method": "0.9", "name": "Capture Registry (longitudinal observation)", "level": 0, "level_name": "Measurement", "url": "https://www.spxi.dev/methods/#m-0-9", "inventory": "https://www.spxi.dev/methods/", "statement": "SPXI instrument: method 0.9 of 87, level 0 (Measurement), Capture Registry (longitudinal observation).", "also": [] }, "slug": "the-pile-dataset-aio-20261101", "date": "2026-11-01", "surface": "Google AI Overview", "surface_basis": "Operator default of 2026-10-01 (runs begin in the Overview unless he says AI Mode); the 'AI Mode Conversation' paste is the Overview expanded (operator, 2026-10-18).", "surfaces": [ "Google AI Overview" ], "auth": "signed out, incognito", "auth_basis": "Operator defaults of 2026-10-14.", "ev": "paste", "mt": "'THE PILE DATASET' ON THE OVERVIEW — THE DATASET ITEMIZED, ALL TWENTY-TWO COMPONENTS; BOOKS3 AS 'LONG-FORM LITERARY TEXT', ITS PIRATED SOURCE, ITS REMOVAL AND THE AUTHORS' SUIT GIVEN AS 'CONTENT ADJUSTMENTS'", "match": "BROAD MATCH", "match_note": null, "s": "Machine Reception", "q": "The Pile dataset", "q_kind": "the census name; NEW address.", "cites": 8, "cite_list": [ { "site": "The Pile", "title": "The Pile", "snip": "An 800GB Dataset of Diverse Text for Language Modeling What is the Pile? The Pile is a 825 GiB diverse, open source language modelling data set that consists of...", "rel": "third_party", "note": null, "n": 1, "url": null }, { "site": "Wikipedia", "title": "The Pile (dataset) - Wikipedia", "snip": "Dataset: The Pile (886.03 GB, open-source, English text) created by EleutherAI, released December 31, 2020, to train large language models across 22 sub-dataset...", "rel": "third_party", "note": null, "n": 2, "url": null }, { "site": "Hugging Face", "title": "EleutherAI/pile · Datasets at Hugging Face", "snip": "Dataset: The Pile by EleutherAI (curated by Leo Gao and Stella Biderman). Size: 825 GiB, 100B<1T tokens. Language: English (EN). Tasks: Text Generation, Fill-Ma...", "rel": "third_party", "note": null, "n": 3, "url": null }, { "site": "Reddit·r/MachineLearning", "title": "[R] The Pile: An 800GB Dataset of Diverse Text for Language Modeling - Reddit", "snip": "January 1, 2021 — The Pile Dataset: The Pile is an 800GB publicly available English text corpus from EleutherAI, designed for training large-scale language models, featuring 22 d...", "rel": "third_party", "note": null, "n": 4, "url": null }, { "site": "The Pile", "title": "The Pile: An 800GB Dataset of Diverse Text for Language Modeling", "snip": "The Pile: 825.18 GiB English text corpus combining 22 diverse high-quality subsets for training and benchmarking large-scale language models. Composition: 14 ne...", "rel": "third_party", "note": null, "n": 5, "url": null }, { "site": "YouTube·The Polymath", "title": "The Pile - An 800GB Dataset for Language Models - [Paper Read #1]", "snip": null, "rel": "third_party", "note": "video card, 35m", "n": 6, "url": null }, { "site": "Hugging Face", "title": "The Common Pile v0.1 - Hugging Face", "snip": "Four and a half years ago, EleutherAI entered the AI scene by releasing the Pile: An 800GB Dataset of Diverse Text for Language Modeling. The Pile was unique at...", "rel": "third_party", "note": null, "n": 7, "url": null }, { "site": "Michael Brenndoerfer", "title": "The Pile: Open-Source Training Dataset for LLMs - Interactive", "snip": "Overview: EleutherAI's The Pile (2021) is an 825GB open-source training dataset designed to democratize large language model research and resolve data fragmenta...", "rel": "third_party", "note": null, "n": 8, "url": null } ], "archive_controlled_cites": 0, "transcript": "[Google AI Overview at the query, signed out and incognito (operator defaults of 2026-10-01 and 2026-10-14). The paste is the Overview expanded (the 'AI Mode Conversation' view).] The 'You said' line carries the query twice, run together, and is given once here. Inline links are Google redirect URLs, kept as pasted.\n\n[QUERENT] The Pile dataset\n\n[SYSTEM] The Pile is an 825 GiB to 886 GB open-source English text dataset created by EleutherAI and released on December 31, 2020, to train large language models (LLMs).\n\nKey Overview\n\n• Size: ~825 GiB compressed (zstandard format) containing roughly 300 billion tokens.\n• Composition: Combines 22 diverse sub-datasets across academic, web, prose, and code domains.\n• Purpose: Designed to improve cross-domain knowledge, reasoning, and generalization compared to raw web scrapes like Common Crawl.\n• Notable Models Trained On It: GPT-Neo, GPT-J, and the Pythia suite by EleutherAI.\n\nThe 22 Sub-Datasets\n\nThe Pile is split into 22 distinct components spanning various types of human writing and knowledge:\n1. Pile-CC (Filtered Common Crawl web text)\n2. PubMed Central (Open-access biomedical and life sciences literature)\n3. Books3 (Long-form literary text/books)\n4. OpenWebText2 (Reddit-linked web text)\n5. ArXiv (Scientific preprints in physics, math, computer science, etc.)\n6. GitHub (Open-source source code repositories)\n7. FreeLaw (Legal opinions and court case filings)\n8. Stack Exchange (Q&A community site threads)\n9. USPTO Backgrounds (U.S. patent application backgrounds)\n10. PubMed Abstracts (Biomedical abstracts)\n11. Gutenberg (PG-19) (Public domain literature)\n12. OpenSubtitles (Movie and TV subtitle transcripts)\n13. Wikipedia (English) (Encyclopedia articles)\n14. DM Mathematics (Mathematical problem-solving text)\n15. Ubuntu IRC (Technical chat logs)\n16. BookCorpus2 (Additional books dataset)\n17. EuroParl (European Parliament proceedings)\n18. Hacker News (Tech/startup forum discussions)\n19. YouTube Subtitles (Transcribed video subtitles)\n20. PhilPapers (Philosophy research papers)\n21. NIH ExPorter (National Institutes of Health grant awards)\n22. Enron Emails (Corporate email correspondence corpus)\n\nControversies and Evolution\n\n• Copyright Disputes: The dataset encountered copyright challenges regarding specific components like Books3 and subtitles, resulting in content adjustments.\n• The Common Pile v0.1: EleutherAI introduced ethical alternatives such as Common Pile v0.1 using licensed text.\nWould you like to know more about how specific sub-datasets like GitHub or arXiv affected model performance, or are you looking for details on ethical alternatives like the Common Pile?\nThe Pile\nThe Pile\nAn 800GB Dataset of Diverse Text for Language Modeling What is the Pile? The Pile is a 825 GiB diverse, open source language modelling data set that consists of...\nWikipedia\nThe Pile (dataset) - Wikipedia\nDataset: The Pile (886.03 GB, open-source, English text) created by EleutherAI, released December 31, 2020, to train large language models across 22 sub-dataset...\nHugging Face\nEleutherAI/pile · Datasets at Hugging Face\nDataset: The Pile by EleutherAI (curated by Leo Gao and Stella Biderman). Size: 825 GiB, 100B<1T tokens. Language: English (EN). Tasks: Text Generation, Fill-Ma...\nReddit·r/MachineLearning\n[R] The Pile: An 800GB Dataset of Diverse Text for Language Modeling - Reddit\nJanuary 1, 2021 — The Pile Dataset: The Pile is an 800GB publicly available English text corpus from EleutherAI, designed for training large-scale language models, featuring 22 d...\nThe Pile\nThe Pile: An 800GB Dataset of Diverse Text for Language Modeling\nThe Pile: 825.18 GiB English text corpus combining 22 diverse high-quality subsets for training and benchmarking large-scale language models. Composition: 14 ne...\nYouTube·The Polymath\n35m\nThe Pile - An 800GB Dataset for Language Models - [Paper Read #1]\nHugging Face\nThe Common Pile v0.1 - Hugging Face\nFour and a half years ago, EleutherAI entered the AI scene by releasing the Pile: An 800GB Dataset of Diverse Text for Language Modeling. The Pile was unique at...\nMichael Brenndoerfer\nThe Pile: Open-Source Training Dataset for LLMs - Interactive\nOverview: EleutherAI's The Pile (2021) is an 825GB open-source training dataset designed to democratize large language model research and resolve data fragmenta...\nAI responses may include mistakes. Learn more", "transcript_raw": "AI Mode Conversation\n\n\nYou said: The Pile datasetThe Pile dataset\n\nThe Pile is an 825 GiB to 886 GB open-source English text dataset created by EleutherAI and released on December 31, 2020, to train large language models (LLMs).\n\nKey Overview\n\n• Size: ~825 GiB compressed (zstandard format) containing roughly 300 billion tokens.\n• Composition: Combines 22 diverse sub-datasets across academic, web, prose, and code domains.\n• Purpose: Designed to improve cross-domain knowledge, reasoning, and generalization compared to raw web scrapes like Common Crawl.\n• Notable Models Trained On It: GPT-Neo, GPT-J, and the Pythia suite by EleutherAI.\n\nThe 22 Sub-Datasets\n\nThe Pile is split into 22 distinct components spanning various types of human writing and knowledge:\n1. Pile-CC (Filtered Common Crawl web text)\n2. PubMed Central (Open-access biomedical and life sciences literature)\n3. Books3 (Long-form literary text/books)\n4. OpenWebText2 (Reddit-linked web text)\n5. ArXiv (Scientific preprints in physics, math, computer science, etc.)\n6. GitHub (Open-source source code repositories)\n7. FreeLaw (Legal opinions and court case filings)\n8. Stack Exchange (Q&A community site threads)\n9. USPTO Backgrounds (U.S. patent application backgrounds)\n10. PubMed Abstracts (Biomedical abstracts)\n11. Gutenberg (PG-19) (Public domain literature)\n12. OpenSubtitles (Movie and TV subtitle transcripts)\n13. Wikipedia (English) (Encyclopedia articles)\n14. DM Mathematics (Mathematical problem-solving text)\n15. Ubuntu IRC (Technical chat logs)\n16. BookCorpus2 (Additional books dataset)\n17. EuroParl (European Parliament proceedings)\n18. Hacker News (Tech/startup forum discussions)\n19. YouTube Subtitles (Transcribed video subtitles)\n20. PhilPapers (Philosophy research papers)\n21. NIH ExPorter (National Institutes of Health grant awards)\n22. Enron Emails (Corporate email correspondence corpus)\n\nControversies and Evolution\n\n• Copyright Disputes: The dataset encountered copyright challenges regarding specific components like Books3 and subtitles, resulting in content adjustments.\n• The Common Pile v0.1: EleutherAI introduced ethical alternatives such as Common Pile v0.1 using licensed text.\nWould you like to know more about how specific sub-datasets like GitHub or arXiv affected model performance, or are you looking for details on ethical alternatives like the Common Pile?\nThe Pile\nThe Pile\nAn 800GB Dataset of Diverse Text for Language Modeling What is the Pile? The Pile is a 825 GiB diverse, open source language modelling data set that consists of...\nWikipedia\nThe Pile (dataset) - Wikipedia\nDataset: The Pile (886.03 GB, open-source, English text) created by EleutherAI, released December 31, 2020, to train large language models across 22 sub-dataset...\nHugging Face\nEleutherAI/pile · Datasets at Hugging Face\nDataset: The Pile by EleutherAI (curated by Leo Gao and Stella Biderman). Size: 825 GiB, 100B<1T tokens. Language: English (EN). Tasks: Text Generation, Fill-Ma...\nReddit·r/MachineLearning\n[R] The Pile: An 800GB Dataset of Diverse Text for Language Modeling - Reddit\nJanuary 1, 2021 — The Pile Dataset: The Pile is an 800GB publicly available English text corpus from EleutherAI, designed for training large-scale language models, featuring 22 d...\nThe Pile\nThe Pile: An 800GB Dataset of Diverse Text for Language Modeling\nThe Pile: 825.18 GiB English text corpus combining 22 diverse high-quality subsets for training and benchmarking large-scale language models. Composition: 14 ne...\nYouTube·The Polymath\n35m\nThe Pile - An 800GB Dataset for Language Models - [Paper Read #1]\nHugging Face\nThe Common Pile v0.1 - Hugging Face\nFour and a half years ago, EleutherAI entered the AI scene by releasing the Pile: An 800GB Dataset of Diverse Text for Language Modeling. The Pile was unique at...\nMichael Brenndoerfer\nThe Pile: Open-Source Training Dataset for LLMs - Interactive\nOverview: EleutherAI's The Pile (2021) is an 825GB open-source training dataset designed to democratize large language model research and resolve data fragmenta...\nAI responses may include mistakes. Learn more\n", "transcript_class": "CAPTURE-TIME VERBATIM RECORD — paste of the Overview expanded, with its source cards", "transcript_complete": "complete as supplied", "transcript_read": "READ IN FULL 2026-11-01", "per": 0.25, "per_corrected": null, "per_v": { "author": true, "inst": true, "id": false, "src": true }, "per_note": "EleutherAI named in the body; Leo Gao and Stella Biderman on the Hugging Face card; the home page, paper page and Wikipedia carded.", "availability": null, "per_dim": { "spec": "PER-dim rubric v1.1", "entity_type": "site", "target": { "ent": "TGT-ext-tll-the-pile", "name": "The Pile" }, "qualifier": null, "reached": "same", "gate": "scored", "atoms": [ { "atom": "name", "dim": "M", "weight": 3, "kind": "token", "present": true, "evidence": "The Pile is an 825 GiB to 886 GB open-source English text dataset created by EleutherAI" }, { "atom": "operator", "dim": "M", "weight": 3, "kind": "role", "present": true, "evidence": "created by EleutherAI and released on December 31, 2020" }, { "atom": "date", "dim": "M", "weight": 1, "kind": "token", "present": true, "evidence": "released on December 31, 2020" }, { "atom": "claim boundary", "dim": "M", "weight": 2, "kind": "reading", "present": true, "evidence": "open-source English text dataset created by EleutherAI and released on December 31, 2020, to train large language models (LLMs)." }, { "atom": "corpus", "dim": "C", "weight": 2, "kind": "reading", "present": true, "evidence": "Combines 22 diverse sub-datasets across academic, web, prose, and code domains." }, { "atom": "purpose", "dim": "D", "weight": 3, "kind": "reading", "present": true, "evidence": "Designed to improve cross-domain knowledge, reasoning, and generalization compared to raw web scrapes like Common Crawl." }, { "atom": "commitments", "dim": "D", "weight": 2, "kind": "reading", "present": false, "evidence": null } ], "inversions": [], "standing_substitutions": [], "scores": { "M": 0.0, "C": 0.0, "D": 0.4, "PER": 0.125 }, "src": true, "tombstone_pointer": false, "after_challenge": null, "basis": "reached: The Pile is an 825 GiB to 886 GB open-source English text dataset created by EleutherAI; The dataset reached by name with its maker, date, size, all 22 components, the models trained on it and its successor.", "read": { "by": "intake reader, in session; spans and record quotes checked verbatim by scripts/per_backlog.py", "on": "2026-11-01", "brief": "rebuild/capture-registry/intake-20261101-layer-aggregators/parts/the-pile.json", "external": true, "notes": "The dataset composed at its grain and itemized to its twenty-two components, the training layer named as the census row names it. Its literary component is glossed by genre, and the pirated source, the removal and the authors' suit that put works in it under their authors' names are reduced to 'content adjustments'." } }, "per_dim_pending": null, "per_dim_v1_0": null, "sf": null, "sf_derived": null, "reading": "The innocent reading first. Asked for the dataset, the Overview gives its maker, date, size, purpose and the models trained on it, lists its twenty-two components with a gloss on each, and closes on its copyright disputes and its licensed successor.", "analysis": "The dataset composed at its grain and itemized to its twenty-two components, the training layer named as the census row names it. Its literary component is glossed by genre, and the pirated source, the removal and the authors' suit that put works in it under their authors' names are reduced to 'content adjustments'.", "d": "THE DATASET, SIGNED OUT, 2026-11-01: 'The Pile is an 825 GiB to 886 GB open-source English text dataset created by EleutherAI and released on December 31, 2020, to train large language models (LLMs).' Key overview (size, '~825 GiB compressed (zstandard format) containing roughly 300 billion tokens'; composition; purpose; 'Notable Models Trained On It: GPT-Neo, GPT-J, and the Pythia suite by EleutherAI'); the 22 sub-datasets, numbered, 'Books3 (Long-form literary text/books)' third and 'Gutenberg (PG-19) (Public domain literature)' eleventh; controversies and evolution ('resulting in content adjustments'; Common Pile v0.1). Eight cards: the Pile's home page, Wikipedia, Hugging Face, Reddit r/MachineLearning, the Pile's paper page, a YouTube paper reading, Hugging Face on the Common Pile v0.1, Michael Brenndoerfer.", "d_full": "THE DATASET, SIGNED OUT, 2026-11-01: 'The Pile is an 825 GiB to 886 GB open-source English text dataset created by EleutherAI and released on December 31, 2020, to train large language models (LLMs).' Key overview (size, '~825 GiB compressed (zstandard format) containing roughly 300 billion tokens'; composition; purpose; 'Notable Models Trained On It: GPT-Neo, GPT-J, and the Pythia suite by EleutherAI'); the 22 sub-datasets, numbered, 'Books3 (Long-form literary text/books)' third and 'Gutenberg (PG-19) (Public domain literature)' eleventh; controversies and evolution ('resulting in content adjustments'; Common Pile v0.1). Eight cards: the Pile's home page, Wikipedia, Hugging Face, Reddit r/MachineLearning, the Pile's paper page, a YouTube paper reading, Hugging Face on the Common Pile v0.1, Michael Brenndoerfer.", "d_truncated": false, "links": [ { "url": "https://www.alexanarch.org/captures/the-pile-dataset-aio-20261101/", "authority": "canonical", "note": "the capture's own record page; cite this form" }, { "url": "https://www.alexanarch.org/captures/#the-pile-dataset-aio-20261101", "authority": "gallery", "note": "the canonical gallery, anchored by slug" }, { "url": "https://www.godkinggoogle.com/captures/#the-pile-dataset-aio-20261101", "authority": "mirror", "note": "a window that renders from the archive's registry; may lag a deploy" }, { "url": "https://www.leesharks.com/captures/#the-pile-dataset-aio-20261101", "authority": "mirror", "note": "a window that renders from the archive's registry; may lag a deploy" }, { "url": "https://www.machinemediation.org/captures/#the-pile-dataset-aio-20261101", "authority": "mirror", "note": "a window that renders from the archive's registry; may lag a deploy" } ], "imgs": [], "img_urls": [], "cite": "https://www.alexanarch.org/captures/the-pile-dataset-aio-20261101/", "citable_unit": "address — the exact issued string on one surface, per the Surface Rule (MANUS, 2026-08-15)", "addr_id": "ADDR-c51497a0501d", "obs_id": "OBS-3bb4c361d912", "n_observations": 1, "observations": [], "dates": [ "2026-11-01" ], "defects": [], "findings": [ "THE LAYER ITEMIZED. The twenty-two components in the order of Wikipedia's size table, Pile-CC first and Enron Emails last, each with a gloss; the size given both ways, '825 GiB to 886 GB', the home page's 'The Pile is a 825 GiB diverse, open source language modelling data set' beside Wikipedia's 'The Pile is an 886 GB diverse, open-source dataset'. The census row reads a work's entry into the layer as membership of a component; the composition names the components and no work and no author in any of them. 'roughly 300 billion tokens' and the three model families are on no saved source; the Hugging Face card gives '100B<1T tokens'.", "BOOKS3 GLOSSED. 'Books3 (Long-form literary text/books)'. Wikipedia: 'The Books3 component of the dataset contains copyrighted material compiled from Bibliotik, a piracy website.' and 'Books3 was removed from the Pile before a class action lawsuit was filed in 2024 by three authors'. The composition: 'The dataset encountered copyright challenges regarding specific components like Books3 and subtitles, resulting in content adjustments.' The source, the Rights Alliance's DMCA takedown of July 2023, the removal and the three authors' suit are absent; the census row reads that suit as 'the field's clearest case of works found in the layer under their authors' names after the fact'.", "THE SUBTITLES AND THE SITE. 'subtitles' stands beside Books3 without its account ('Tens of thousands of YouTube videos had their subtitles scraped directly from YouTube and included in the Pile, which YouTube argued is against its terms of service'). Wikipedia's 'By 2024, The Pile also was taken down from its original site' is absent, and the first card is the original site's home page ('An 800GB Dataset of Diverse Text for Language Modeling What is the Pile?'), which the census also fetched with HTTP 200 on 2026-11-01 and which still reads 'Download Pile' and 'The Pile is hosted by the Eye .'", "WHAT THE APPARATUS MAKES. 'Designed to improve cross-domain knowledge, reasoning, and generalization compared to raw web scrapes like Common Crawl.' The paper: 'models trained on the Pile improve significantly over both Raw CC and CC-100 on all components of the Pile'. The census row's 'makes' is the trained model scored on the Pile's own benchmark ('Pile BPB is a measure of world knowledge and reasoning ability in these domains'); the composition names models trained on it and does not carry the benchmark. The paper's 'we document potentially concerning aspects of the data for prospective users' is absent.", "THE SUCCESSOR. 'EleutherAI introduced ethical alternatives such as Common Pile v0.1 using licensed text.' Wikipedia: 'released Common Pile v0.1, a training dataset that contains only works where the licenses permit their use for training AI models', 'In June 2025, EleutherAI, in partnership with the Poolside , Hugging Face , and the US Library of Congress'. The composition gives EleutherAI alone and no date; the seventh card carries the date in EleutherAI's words ('Four and a half years ago, EleutherAI entered the AI scene by releasing the Pile')." ], "series": null, "other_slugs": null, "collisions": null, "oq": null, "rounds": null, "turns": 1, "rerun": null, "rerun_alt": null, "heteronym": null, "model_attribution": null, "operator_disclosure": null, "longitudinal_priors": null, "longitudinal_successors": null, "related_deposits": [ 1543 ], "intended": null, "originator": { "name": "The Pile", "relation": "external", "entity_type": "site", "spxi_treatment": "partial", "basis": "Not the archive's: an external resource, entered in the traininglayerliterature.org layer census on 2026-11-01 (data/layer-census, its own pages saved with hashes). Recorded 2026-11-01." }, "notes": { "prompts": "The prompts are the operator's, as given with the paste; the run's hour is not in the paste, the date is the paste's receipt (2026-11-01 09:56 EDT).", "operator": "with the pastes (09:56 EDT): 'captures, then /nons, then single pages'. The census names as printed for him for their /non rows. His notes in the paste: at Project Gutenberg, 'wikipedia in place of popup so ai mode:'; at HathiTrust, 'no popup. ai mode:'. The Literary History of Artificial Intelligence and The Pile came as two attached files." }, "notice": null, "axn_root": "AXN:0BBA.CAPTURE", "axn": "AXN:0BBA.CAPTURE.⏩🕙🏛️🧡🔅🪐", "record_url": "https://www.alexanarch.org/captures/the-pile-dataset-aio-20261101/" }