{"address":{"q":"Dolma dataset","record":"dolma-dataset-aio-20261101"},"analysis":{"analysis":"The corpus composed accurately and more finely than the saved abstract, from the dataset's own pages, with Ai2 as maker where the census row names only the paper's authors. Whether a document in the corpus keeps its author's name, the census row's open question, is not reached.","d":"THE CORPUS, SIGNED OUT, 2026-11-01: 'Dolma is an open-source 3-trillion-token English corpus created by the Allen Institute for Artificial Intelligence (Ai2) to support language model pretraining and transparency.' Access (Hugging Face, the GitHub toolkit); key features: scale ('version 1.7 utilizes about 1.715 trillion actual training tokens across ~5.2 billion documents totaling over 5,300 GB'), data sources by kind with their named sets, the curation pipeline ('Bloom filter deduplication (by URL, paragraph, and document), PII/toxicity masking'), licensing ('ODC-BY and Ai2's ImpACT license'), primary use (OLMo); two follow-up questions. Six cards: Hugging Face, the GitHub Pages documentation, Ai2, arXiv, Deep (Learning) Focus, ACL Anthology.","d_full":"THE CORPUS, SIGNED OUT, 2026-11-01: 'Dolma is an open-source 3-trillion-token English corpus created by the Allen Institute for Artificial Intelligence (Ai2) to support language model pretraining and transparency.' Access (Hugging Face, the GitHub toolkit); key features: scale ('version 1.7 utilizes about 1.715 trillion actual training tokens across ~5.2 billion documents totaling over 5,300 GB'), data sources by kind with their named sets, the curation pipeline ('Bloom filter deduplication (by URL, paragraph, and document), PII/toxicity masking'), licensing ('ODC-BY and Ai2's ImpACT license'), primary use (OLMo); two follow-up questions. Six cards: Hugging Face, the GitHub Pages documentation, Ai2, arXiv, Deep (Learning) Focus, ACL Anthology.","findings":["THE CORPUS AT ITS GRAIN. 'Dolma is an open-source 3-trillion-token English corpus' and 'web content (Common Crawl, C4, RefinedWeb), academic papers (arXiv, Semantic Scholar, peS2o), code (The Stack, StarCoder), public-domain books (Project Gutenberg), encyclopedic material (Wikipedia, Wikibooks), and social media discussions (Reddit, StackExchange)'. The saved abstract: 'a three-trillion-token English corpus, built from a diverse mixture of web content, scientific papers, code, public-domain books, social media, and encyclopedic materials.' The abstract names the six kinds; the composition names a set for each, Project Gutenberg the one source of the books, from the Ai2 card ('Sources: CommonCrawl, C4, peS2o, The Stack, Project Gutenb...').","THE MAKER THE CENSUS DOES NOT HAVE. 'created by the Allen Institute for Artificial Intelligence (Ai2)'; three cards carry the same ('Ai2 Dolma: 3 trillion token open corpus for language model pretraining | Ai2'; 'created as training data for OLMo by AI2'). The census row's operator is 'the paper's 36 authors, Luca Soldaini and 35 others', its saved arXiv page reading 'by Luca Soldaini and 35 other authors', and it states that 'the saved sources name no institution', the Hugging Face card having returned HTTP 429 ('We had to rate limit your IP'). The composition names the institution and no author.","A LATER VERSION AND THE LICENCES. 'version 1.7 utilizes about 1.715 trillion actual training tokens across ~5.2 billion documents totaling over 5,300 GB' and 'Released under permissive research licenses (such as ODC-BY and Ai2's ImpACT license)'. The census row's lifecycle: 'later versions or the dataset's current state are unread'. Version 1.7, the document count and the licences are on no saved source; the Ai2 card gives '5,245M documents, 5,334 GB'. The date reaches the composition only on the ACL card ('ACL 2024 Best Resource Paper Award'), agreeing with the abstract's 'Accepted at ACL 2024'; the award is on no saved source.","THE PURPOSE NARROWED TO ONE MODEL FAMILY. The abstract: 'To facilitate scientific research on language model pretraining', against a field where 'commercial models rarely detail their data'. The composition: 'to support language model pretraining and transparency' and 'Designed and used specifically to pre-train the OLMo open-source language model family.' Research drops from the purpose; the openness the abstract argues for stands as 'transparency', and the toolkit as 'the Dolma GitHub Toolkit', agreeing with 'we open-source our data curation toolkit to enable reproduction of our work'.","THE AUTHOR OF A DOCUMENT NOT CARRIED. The census row: 'the saved abstract does not say whether a document in Dolma keeps its author's name, source or attribution metadata', and its reading asks whether a work is carried with 'the metadata that lets it be found and attributed' (#186). The composition's operations on a document are 'language filtering', 'Bloom filter deduplication (by URL, paragraph, and document)' and 'PII/toxicity masking'; nothing in it says what of a book's author or source survives into the corpus, and its follow-up offers 'specific subsets or data processing scripts'."],"match":"BROAD MATCH","mt":"'DOLMA DATASET' ON THE OVERVIEW — THE CORPUS AT ITS GRAIN AND PAST THE SAVED ABSTRACT; AI2 NAMED WHERE THE CENSUS NAMES NO INSTITUTION; WHETHER A DOCUMENT KEEPS ITS AUTHOR NOT CARRIED","notes":{"operator":"with the pastes (09:56 EDT): 'captures, then /nons, then single pages'. The census names as printed for him for their /non rows. His notes in the paste: at Project Gutenberg, 'wikipedia in place of popup so ai mode:'; at HathiTrust, 'no popup. ai mode:'. The Literary History of Artificial Intelligence and The Pile came as two attached files.","prompts":"The prompts are the operator's, as given with the paste; the run's hour is not in the paste, the date is the paste's receipt (2026-11-01 09:56 EDT)."},"originator":{"basis":"Not the archive's: an external resource, entered in the traininglayerliterature.org layer census on 2026-11-01 (data/layer-census, its own pages saved with hashes). Recorded 2026-11-01.","entity_type":"site","name":"Dolma","relation":"external","spxi_treatment":"partial"},"reading":"The innocent reading first. Asked for the dataset, the Overview says what it is, who made it and why, where to get it, what it holds, how it was cleaned, under what licences it is released and which models it trains, from the dataset's own pages and its paper."},"measures":{"archive_controlled_cites":0,"cite_list":[{"n":1,"note":null,"rel":"third_party","site":"Hugging Face","snip":"Overview: AllenAI Dolma is an open corpus dataset containing 3 trillion tokens (version 1.7 utilizing 1.715 trillion actual training tokens) designed for langua...","title":"allenai/dolma · Datasets at Hugging Face","url":null},{"n":2,"note":null,"rel":"third_party","site":"GitHub Pages documentation","snip":"Dataset: Open 3-trillion-token corpus (web content, academic publications, code, books, encyclopedic materials) created as training data for OLMo by AI2; availa...","title":"dolma | Data and tools for generating and inspecting OLMo pre-training data.","url":null},{"n":3,"note":null,"rel":"third_party","site":"Allen Institute for Artificial Intelligence","snip":"Dataset: Dolma (3 trillion tokens, 5,245M documents, 5,334 GB) developed by Ai2 for OLMo pretraining. Sources: CommonCrawl, C4, peS2o, The Stack, Project Gutenb...","title":"Ai2 Dolma: 3 trillion token open corpus for language model pretraining | Ai2","url":null},{"n":4,"note":null,"rel":"third_party","site":"arXiv","snip":"Overview: Dolma is an open-source, three-trillion-token English corpus designed for language model pretraining research, addressing the scarcity of publicly ava...","title":"[2402.00159] Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research","url":null},{"n":5,"note":null,"rel":"third_party","site":"Deep (Learning) Focus·Deep (Learning) Focus","snip":"Overview: Dolma (3T-token open pretraining corpus) and OLMo (fully open-source LLM suite) demystify language model development by openly releasing training data...","title":"Dolma, OLMo, and the Future of Open-Source LLMs","url":null},{"n":6,"note":null,"rel":"third_party","site":"ACL Anthology","snip":"Resource: Dolma (ACL 2024 Best Resource Paper Award). Corpus: Open English dataset containing three trillion tokens for language model pretraining research. Con...","title":"Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research - ACL Anthology","url":null}],"cites":6,"per":0.25,"per_dim":{"after_challenge":null,"atoms":[{"atom":"name","dim":"M","evidence":"Dolma is an open-source 3-trillion-token English corpus created by the","kind":"token","present":true,"weight":3},{"atom":"operator","dim":"M","evidence":null,"kind":"role","present":false,"weight":3},{"atom":"date","dim":"M","evidence":"ACL 2024 Best Resource Paper Award","kind":"token","present":true,"weight":1},{"atom":"claim boundary","dim":"M","evidence":"open-source 3-trillion-token English corpus","kind":"reading","present":true,"weight":2},{"atom":"corpus","dim":"C","evidence":"Combines a diverse mix of web content (Common Crawl, C4, RefinedWeb), academic papers (arXiv, Semantic Scholar, peS2o), code (The Stack, StarCoder), public-domain books (Project Gutenberg)","kind":"reading","present":true,"weight":2},{"atom":"purpose","dim":"D","evidence":"to support language model pretraining and transparency.","kind":"reading","present":true,"weight":3},{"atom":"commitments","dim":"D","evidence":"explore the curation pipeline using the Dolma GitHub Toolkit.","kind":"reading","present":true,"weight":2}],"basis":"reached: Dolma is an open-source 3-trillion-token English corpus created by the; The corpus reached by name with its maker, size, sources, curation pipeline, licences and the model family it trains.","entity_type":"site","gate":"scored","inversions":[],"qualifier":null,"reached":"same","read":{"brief":"rebuild/capture-registry/intake-20261101-layer-aggregators/parts/dolma.json","by":"intake reader, in session; spans and record quotes checked verbatim by scripts/per_backlog.py","external":true,"notes":"The corpus composed accurately and more finely than the saved abstract, from the dataset's own pages, with Ai2 as maker where the census row names only the paper's authors. Whether a document in the corpus keeps its author's name, the census row's open question, is not reached.","on":"2026-11-01"},"scores":{"C":0.0,"D":0.0,"M":0.333,"PER":0.188},"spec":"PER-dim rubric v1.1","src":true,"standing_substitutions":[],"target":{"ent":"TGT-ext-tll-dolma","name":"Dolma"},"tombstone_pointer":false},"per_note":"Ai2 named and no author; arXiv 2402.00159 on a card; the dataset card, documentation, Ai2 page and paper carded.","per_v":{"author":false,"id":true,"inst":true,"src":true}},"observation":{"auth":"signed out, incognito","date":"2026-11-01","ev":"paste","q":"Dolma dataset","q_kind":"the census name; NEW address.","s":"Machine Reception","surface":"Google AI Overview","transcript":"[Google AI Overview at the query, signed out and incognito (operator defaults of 2026-10-01 and 2026-10-14). The paste is the Overview expanded (the 'AI Mode Conversation' view).] The 'You said' line carries the query twice, run together, and is given once here. Inline links are Google redirect URLs, kept as pasted.\n\n[QUERENT] Dolma dataset\n\n[SYSTEM] Dolma is an open-source 3-trillion-token English corpus created by the [Allen Institute for Artificial Intelligence (Ai2)](https://www.google.com/goto?url=CAEShQEB6zswFdYTFzcT8iARHW7j2rb03Q9fzUvuXk92kI5vJW1ys8rU0ZyNqvi9nb9SEfxVE6GDLUXfzIOJ9Mdgxo_wLR3UpEXqPziCd7TsCkT04tfI3NGm7zbcWyRpgISmYcfpVr3bLZKfJFJXV2yp769te1gUIyCeAsDgwURFI-czaqs1Tqxx) to support language model pretraining and transparency. [[1](https://www.google.com/goto?url=CAESWAHrOzAVfuCTDUTg4GGhhC3F70zMJsaeToVocN310KpVcxm35YicdR3Yfao1ze76K1TcwOI70UkQvtYaIxXXlQS2YEUAPbNuGdAYtdcU3sz835DbRODiVjw)]\nYou can access and download the files directly via the [AllenAI Dolma on Hugging Face](https://www.google.com/goto?url=CAESZQHrOzAVqDGWZgFkNAiGOTr1a3I1Et4PFMQit55QdiU3kqPsTyA9okSIwbdfjlhNbKbOp0k5Zv-YXrOnL6SgaiaRrWOD1-DMaGhXR8JXgYW-jE3KNA8m5Y1C3hoIQbK8y1B0uWYy) hub or explore the curation pipeline using the Dolma GitHub Toolkit. [[1](https://www.google.com/goto?url=CAESZQHrOzAVqDGWZgFkNAiGOTr1a3I1Et4PFMQit55QdiU3kqPsTyA9okSIwbdfjlhNbKbOp0k5Zv-YXrOnL6SgaiaRrWOD1-DMaGhXR8JXgYW-jE3KNA8m5Y1C3hoIQbK8y1B0uWYy), [2](https://www.google.com/goto?url=CAESWAHrOzAVLPcsYpsiFGKn0IOrMyoQK7Y1lvcTXMz2p8wmsK0D5ukuSnPSumjXvVgrgy5pvhwIre-i30P5hgiF_kqQNHI9UJXGv-ojqHUcg_DqhuzKfg5AHbo)]\nKey Features\n\n* Scale: Contains roughly 3 trillion tokens (version 1.7 utilizes about 1.715 trillion actual training tokens across ~5.2 billion documents totaling over 5,300 GB). [[1](https://www.google.com/goto?url=CAEShQEB6zswFdYTFzcT8iARHW7j2rb03Q9fzUvuXk92kI5vJW1ys8rU0ZyNqvi9nb9SEfxVE6GDLUXfzIOJ9Mdgxo_wLR3UpEXqPziCd7TsCkT04tfI3NGm7zbcWyRpgISmYcfpVr3bLZKfJFJXV2yp769te1gUIyCeAsDgwURFI-czaqs1Tqxx), [2](https://www.google.com/goto?url=CAESZQHrOzAVqDGWZgFkNAiGOTr1a3I1Et4PFMQit55QdiU3kqPsTyA9okSIwbdfjlhNbKbOp0k5Zv-YXrOnL6SgaiaRrWOD1-DMaGhXR8JXgYW-jE3KNA8m5Y1C3hoIQbK8y1B0uWYy)]\n* Data Sources: Combines a diverse mix of web content (Common Crawl, C4, RefinedWeb), academic papers (arXiv, Semantic Scholar, peS2o), code (The Stack, StarCoder), public-domain books (Project Gutenberg), encyclopedic material (Wikipedia, Wikibooks), and social media discussions (Reddit, StackExchange). [[1](https://www.google.com/goto?url=CAESZQHrOzAVqDGWZgFkNAiGOTr1a3I1Et4PFMQit55QdiU3kqPsTyA9okSIwbdfjlhNbKbOp0k5Zv-YXrOnL6SgaiaRrWOD1-DMaGhXR8JXgYW-jE3KNA8m5Y1C3hoIQbK8y1B0uWYy), [2](https://www.google.com/goto?url=CAEShQEB6zswFdYTFzcT8iARHW7j2rb03Q9fzUvuXk92kI5vJW1ys8rU0ZyNqvi9nb9SEfxVE6GDLUXfzIOJ9Mdgxo_wLR3UpEXqPziCd7TsCkT04tfI3NGm7zbcWyRpgISmYcfpVr3bLZKfJFJXV2yp769te1gUIyCeAsDgwURFI-czaqs1Tqxx)]\n* Curation Pipeline: Involves language filtering, Gopher/C4 quality heuristics, Bloom filter deduplication (by URL, paragraph, and document), PII/toxicity masking, and evaluation decontamination. [[1](https://www.google.com/goto?url=CAESfgHrOzAVf6rn5OZttIM0us6WvAir5Q1-WR0EyYm9ecTP7XBTIM-DinlhVME8i_KIkf8LZ3DAOz14u-sSPbfl_Pc4u7hcHYX6Q1oBjXL5hFzJgdfGrRKVE1UMINo78prcPEMu19EHqLlLGx8laR-2Dzu5RINQkc4u25jqyRpHwQ), [2](https://www.google.com/goto?url=CAEShQEB6zswFdYTFzcT8iARHW7j2rb03Q9fzUvuXk92kI5vJW1ys8rU0ZyNqvi9nb9SEfxVE6GDLUXfzIOJ9Mdgxo_wLR3UpEXqPziCd7TsCkT04tfI3NGm7zbcWyRpgISmYcfpVr3bLZKfJFJXV2yp769te1gUIyCeAsDgwURFI-czaqs1Tqxx)]\n* Licensing: Released under permissive research licenses (such as ODC-BY and Ai2's ImpACT license) alongside the [Dolma ACL Research Paper](https://www.google.com/goto?url=CAESYwHrOzAVvKhxmIIzUklJEDSJyWY8Ro1YnSpFLFxZhQ0EcW4hazs3O-L1Y8BHPxI9cIxyUxxa4A8aVoomoKlEYiUEFBtlTpFdl0qVShVTplvTAOrrsSsWQ_H2MW35T8flseiMVA). [[1](https://www.google.com/goto?url=CAESYwHrOzAVvKhxmIIzUklJEDSJyWY8Ro1YnSpFLFxZhQ0EcW4hazs3O-L1Y8BHPxI9cIxyUxxa4A8aVoomoKlEYiUEFBtlTpFdl0qVShVTplvTAOrrsSsWQ_H2MW35T8flseiMVA), [2](https://www.google.com/goto?url=CAESZQHrOzAVqDGWZgFkNAiGOTr1a3I1Et4PFMQit55QdiU3kqPsTyA9okSIwbdfjlhNbKbOp0k5Zv-YXrOnL6SgaiaRrWOD1-DMaGhXR8JXgYW-jE3KNA8m5Y1C3hoIQbK8y1B0uWYy), [3](https://www.google.com/goto?url=CAEShQEB6zswFdYTFzcT8iARHW7j2rb03Q9fzUvuXk92kI5vJW1ys8rU0ZyNqvi9nb9SEfxVE6GDLUXfzIOJ9Mdgxo_wLR3UpEXqPziCd7TsCkT04tfI3NGm7zbcWyRpgISmYcfpVr3bLZKfJFJXV2yp769te1gUIyCeAsDgwURFI-czaqs1Tqxx)]\n* Primary Use: Designed and used specifically to pre-train the [OLMo](https://www.google.com/goto?url=CAESfgHrOzAVf6rn5OZttIM0us6WvAir5Q1-WR0EyYm9ecTP7XBTIM-DinlhVME8i_KIkf8LZ3DAOz14u-sSPbfl_Pc4u7hcHYX6Q1oBjXL5hFzJgdfGrRKVE1UMINo78prcPEMu19EHqLlLGx8laR-2Dzu5RINQkc4u25jqyRpHwQ) open-source language model family. [[1](https://www.google.com/goto?url=CAESZQHrOzAVqDGWZgFkNAiGOTr1a3I1Et4PFMQit55QdiU3kqPsTyA9okSIwbdfjlhNbKbOp0k5Zv-YXrOnL6SgaiaRrWOD1-DMaGhXR8JXgYW-jE3KNA8m5Y1C3hoIQbK8y1B0uWYy), [2](https://www.google.com/goto?url=CAESfgHrOzAVf6rn5OZttIM0us6WvAir5Q1-WR0EyYm9ecTP7XBTIM-DinlhVME8i_KIkf8LZ3DAOz14u-sSPbfl_Pc4u7hcHYX6Q1oBjXL5hFzJgdfGrRKVE1UMINo78prcPEMu19EHqLlLGx8laR-2Dzu5RINQkc4u25jqyRpHwQ)]\n\nIf you'd like, let me know:\n\n* Are you planning to train or fine-tune an LLM using Dolma?\n* Are you looking for specific subsets or data processing scripts?\n\nI can provide the right code commands or pipeline details.\nHugging Face\nallenai/dolma · Datasets at Hugging Face\nOverview: AllenAI Dolma is an open corpus dataset containing 3 trillion tokens (version 1.7 utilizing 1.715 trillion actual training tokens) designed for langua...\nGitHub Pages documentation\ndolma | Data and tools for generating and inspecting OLMo pre-training data.\nDataset: Open 3-trillion-token corpus (web content, academic publications, code, books, encyclopedic materials) created as training data for OLMo by AI2; availa...\nAllen Institute for Artificial Intelligence\nAi2 Dolma: 3 trillion token open corpus for language model pretraining | Ai2\nDataset: Dolma (3 trillion tokens, 5,245M documents, 5,334 GB) developed by Ai2 for OLMo pretraining. Sources: CommonCrawl, C4, peS2o, The Stack, Project Gutenb...\narXiv\n[2402.00159] Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research\nOverview: Dolma is an open-source, three-trillion-token English corpus designed for language model pretraining research, addressing the scarcity of publicly ava...\nDeep (Learning) Focus·Deep (Learning) Focus\nDolma, OLMo, and the Future of Open-Source LLMs\nOverview: Dolma (3T-token open pretraining corpus) and OLMo (fully open-source LLM suite) demystify language model development by openly releasing training data...\nACL Anthology\nDolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research - ACL Anthology\nResource: Dolma (ACL 2024 Best Resource Paper Award). Corpus: Open English dataset containing three trillion tokens for language model pretraining research. Con...\nAI responses may include mistakes. [Learn more](https://support.google.com/websearch?p=ai_overviews)","transcript_class":"CAPTURE-TIME VERBATIM RECORD — paste of the Overview expanded, with its source cards","transcript_complete":"complete as supplied","transcript_raw":"AI Mode Conversation\nYou said: Dolma datasetDolma dataset\nDolma is an open-source 3-trillion-token English corpus created by the [Allen Institute for Artificial Intelligence (Ai2)](https://www.google.com/goto?url=CAEShQEB6zswFdYTFzcT8iARHW7j2rb03Q9fzUvuXk92kI5vJW1ys8rU0ZyNqvi9nb9SEfxVE6GDLUXfzIOJ9Mdgxo_wLR3UpEXqPziCd7TsCkT04tfI3NGm7zbcWyRpgISmYcfpVr3bLZKfJFJXV2yp769te1gUIyCeAsDgwURFI-czaqs1Tqxx) to support language model pretraining and transparency. [[1](https://www.google.com/goto?url=CAESWAHrOzAVfuCTDUTg4GGhhC3F70zMJsaeToVocN310KpVcxm35YicdR3Yfao1ze76K1TcwOI70UkQvtYaIxXXlQS2YEUAPbNuGdAYtdcU3sz835DbRODiVjw)]\nYou can access and download the files directly via the [AllenAI Dolma on Hugging Face](https://www.google.com/goto?url=CAESZQHrOzAVqDGWZgFkNAiGOTr1a3I1Et4PFMQit55QdiU3kqPsTyA9okSIwbdfjlhNbKbOp0k5Zv-YXrOnL6SgaiaRrWOD1-DMaGhXR8JXgYW-jE3KNA8m5Y1C3hoIQbK8y1B0uWYy) hub or explore the curation pipeline using the Dolma GitHub Toolkit. [[1](https://www.google.com/goto?url=CAESZQHrOzAVqDGWZgFkNAiGOTr1a3I1Et4PFMQit55QdiU3kqPsTyA9okSIwbdfjlhNbKbOp0k5Zv-YXrOnL6SgaiaRrWOD1-DMaGhXR8JXgYW-jE3KNA8m5Y1C3hoIQbK8y1B0uWYy), [2](https://www.google.com/goto?url=CAESWAHrOzAVLPcsYpsiFGKn0IOrMyoQK7Y1lvcTXMz2p8wmsK0D5ukuSnPSumjXvVgrgy5pvhwIre-i30P5hgiF_kqQNHI9UJXGv-ojqHUcg_DqhuzKfg5AHbo)]\nKey Features\n\n* Scale: Contains roughly 3 trillion tokens (version 1.7 utilizes about 1.715 trillion actual training tokens across ~5.2 billion documents totaling over 5,300 GB). [[1](https://www.google.com/goto?url=CAEShQEB6zswFdYTFzcT8iARHW7j2rb03Q9fzUvuXk92kI5vJW1ys8rU0ZyNqvi9nb9SEfxVE6GDLUXfzIOJ9Mdgxo_wLR3UpEXqPziCd7TsCkT04tfI3NGm7zbcWyRpgISmYcfpVr3bLZKfJFJXV2yp769te1gUIyCeAsDgwURFI-czaqs1Tqxx), [2](https://www.google.com/goto?url=CAESZQHrOzAVqDGWZgFkNAiGOTr1a3I1Et4PFMQit55QdiU3kqPsTyA9okSIwbdfjlhNbKbOp0k5Zv-YXrOnL6SgaiaRrWOD1-DMaGhXR8JXgYW-jE3KNA8m5Y1C3hoIQbK8y1B0uWYy)]\n* Data Sources: Combines a diverse mix of web content (Common Crawl, C4, RefinedWeb), academic papers (arXiv, Semantic Scholar, peS2o), code (The Stack, StarCoder), public-domain books (Project Gutenberg), encyclopedic material (Wikipedia, Wikibooks), and social media discussions (Reddit, StackExchange). [[1](https://www.google.com/goto?url=CAESZQHrOzAVqDGWZgFkNAiGOTr1a3I1Et4PFMQit55QdiU3kqPsTyA9okSIwbdfjlhNbKbOp0k5Zv-YXrOnL6SgaiaRrWOD1-DMaGhXR8JXgYW-jE3KNA8m5Y1C3hoIQbK8y1B0uWYy), [2](https://www.google.com/goto?url=CAEShQEB6zswFdYTFzcT8iARHW7j2rb03Q9fzUvuXk92kI5vJW1ys8rU0ZyNqvi9nb9SEfxVE6GDLUXfzIOJ9Mdgxo_wLR3UpEXqPziCd7TsCkT04tfI3NGm7zbcWyRpgISmYcfpVr3bLZKfJFJXV2yp769te1gUIyCeAsDgwURFI-czaqs1Tqxx)]\n* Curation Pipeline: Involves language filtering, Gopher/C4 quality heuristics, Bloom filter deduplication (by URL, paragraph, and document), PII/toxicity masking, and evaluation decontamination. [[1](https://www.google.com/goto?url=CAESfgHrOzAVf6rn5OZttIM0us6WvAir5Q1-WR0EyYm9ecTP7XBTIM-DinlhVME8i_KIkf8LZ3DAOz14u-sSPbfl_Pc4u7hcHYX6Q1oBjXL5hFzJgdfGrRKVE1UMINo78prcPEMu19EHqLlLGx8laR-2Dzu5RINQkc4u25jqyRpHwQ), [2](https://www.google.com/goto?url=CAEShQEB6zswFdYTFzcT8iARHW7j2rb03Q9fzUvuXk92kI5vJW1ys8rU0ZyNqvi9nb9SEfxVE6GDLUXfzIOJ9Mdgxo_wLR3UpEXqPziCd7TsCkT04tfI3NGm7zbcWyRpgISmYcfpVr3bLZKfJFJXV2yp769te1gUIyCeAsDgwURFI-czaqs1Tqxx)]\n* Licensing: Released under permissive research licenses (such as ODC-BY and Ai2's ImpACT license) alongside the [Dolma ACL Research Paper](https://www.google.com/goto?url=CAESYwHrOzAVvKhxmIIzUklJEDSJyWY8Ro1YnSpFLFxZhQ0EcW4hazs3O-L1Y8BHPxI9cIxyUxxa4A8aVoomoKlEYiUEFBtlTpFdl0qVShVTplvTAOrrsSsWQ_H2MW35T8flseiMVA). [[1](https://www.google.com/goto?url=CAESYwHrOzAVvKhxmIIzUklJEDSJyWY8Ro1YnSpFLFxZhQ0EcW4hazs3O-L1Y8BHPxI9cIxyUxxa4A8aVoomoKlEYiUEFBtlTpFdl0qVShVTplvTAOrrsSsWQ_H2MW35T8flseiMVA), [2](https://www.google.com/goto?url=CAESZQHrOzAVqDGWZgFkNAiGOTr1a3I1Et4PFMQit55QdiU3kqPsTyA9okSIwbdfjlhNbKbOp0k5Zv-YXrOnL6SgaiaRrWOD1-DMaGhXR8JXgYW-jE3KNA8m5Y1C3hoIQbK8y1B0uWYy), [3](https://www.google.com/goto?url=CAEShQEB6zswFdYTFzcT8iARHW7j2rb03Q9fzUvuXk92kI5vJW1ys8rU0ZyNqvi9nb9SEfxVE6GDLUXfzIOJ9Mdgxo_wLR3UpEXqPziCd7TsCkT04tfI3NGm7zbcWyRpgISmYcfpVr3bLZKfJFJXV2yp769te1gUIyCeAsDgwURFI-czaqs1Tqxx)]\n* Primary Use: Designed and used specifically to pre-train the [OLMo](https://www.google.com/goto?url=CAESfgHrOzAVf6rn5OZttIM0us6WvAir5Q1-WR0EyYm9ecTP7XBTIM-DinlhVME8i_KIkf8LZ3DAOz14u-sSPbfl_Pc4u7hcHYX6Q1oBjXL5hFzJgdfGrRKVE1UMINo78prcPEMu19EHqLlLGx8laR-2Dzu5RINQkc4u25jqyRpHwQ) open-source language model family. [[1](https://www.google.com/goto?url=CAESZQHrOzAVqDGWZgFkNAiGOTr1a3I1Et4PFMQit55QdiU3kqPsTyA9okSIwbdfjlhNbKbOp0k5Zv-YXrOnL6SgaiaRrWOD1-DMaGhXR8JXgYW-jE3KNA8m5Y1C3hoIQbK8y1B0uWYy), [2](https://www.google.com/goto?url=CAESfgHrOzAVf6rn5OZttIM0us6WvAir5Q1-WR0EyYm9ecTP7XBTIM-DinlhVME8i_KIkf8LZ3DAOz14u-sSPbfl_Pc4u7hcHYX6Q1oBjXL5hFzJgdfGrRKVE1UMINo78prcPEMu19EHqLlLGx8laR-2Dzu5RINQkc4u25jqyRpHwQ)]\n\nIf you'd like, let me know:\n\n* Are you planning to train or fine-tune an LLM using Dolma?\n* Are you looking for specific subsets or data processing scripts?\n\nI can provide the right code commands or pipeline details.\nHugging Face\nallenai/dolma · Datasets at Hugging Face\nOverview: AllenAI Dolma is an open corpus dataset containing 3 trillion tokens (version 1.7 utilizing 1.715 trillion actual training tokens) designed for langua...\nGitHub Pages documentation\ndolma | Data and tools for generating and inspecting OLMo pre-training data.\nDataset: Open 3-trillion-token corpus (web content, academic publications, code, books, encyclopedic materials) created as training data for OLMo by AI2; availa...\nAllen Institute for Artificial Intelligence\nAi2 Dolma: 3 trillion token open corpus for language model pretraining | Ai2\nDataset: Dolma (3 trillion tokens, 5,245M documents, 5,334 GB) developed by Ai2 for OLMo pretraining. Sources: CommonCrawl, C4, peS2o, The Stack, Project Gutenb...\narXiv\n[2402.00159] Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research\nOverview: Dolma is an open-source, three-trillion-token English corpus designed for language model pretraining research, addressing the scarcity of publicly ava...\nDeep (Learning) Focus·Deep (Learning) Focus\nDolma, OLMo, and the Future of Open-Source LLMs\nOverview: Dolma (3T-token open pretraining corpus) and OLMo (fully open-source LLM suite) demystify language model development by openly releasing training data...\nACL Anthology\nDolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research - ACL Anthology\nResource: Dolma (ACL 2024 Best Resource Paper Award). Corpus: Open English dataset containing three trillion tokens for language model pretraining research. Con...\nAI responses may include mistakes. [Learn more](https://support.google.com/websearch?p=ai_overviews)\n","transcript_read":"READ IN FULL 2026-11-01","turns":1},"slug":"dolma-dataset-aio-20261101"}