[Google AI Overview at the query, signed out and incognito (operator defaults of 2026-10-01 and 2026-10-14). The paste is the Overview expanded (the 'AI Mode Conversation' view).] The 'You said' line carries the query twice, run together, and is given once here. Inline links are Google redirect URLs, kept as pasted. [QUERENT] Dolma dataset [SYSTEM] Dolma is an open-source 3-trillion-token English corpus created by the [Allen Institute for Artificial Intelligence (Ai2)](https://www.google.com/goto?url=CAEShQEB6zswFdYTFzcT8iARHW7j2rb03Q9fzUvuXk92kI5vJW1ys8rU0ZyNqvi9nb9SEfxVE6GDLUXfzIOJ9Mdgxo_wLR3UpEXqPziCd7TsCkT04tfI3NGm7zbcWyRpgISmYcfpVr3bLZKfJFJXV2yp769te1gUIyCeAsDgwURFI-czaqs1Tqxx) to support language model pretraining and transparency. [[1](https://www.google.com/goto?url=CAESWAHrOzAVfuCTDUTg4GGhhC3F70zMJsaeToVocN310KpVcxm35YicdR3Yfao1ze76K1TcwOI70UkQvtYaIxXXlQS2YEUAPbNuGdAYtdcU3sz835DbRODiVjw)] You can access and download the files directly via the [AllenAI Dolma on Hugging Face](https://www.google.com/goto?url=CAESZQHrOzAVqDGWZgFkNAiGOTr1a3I1Et4PFMQit55QdiU3kqPsTyA9okSIwbdfjlhNbKbOp0k5Zv-YXrOnL6SgaiaRrWOD1-DMaGhXR8JXgYW-jE3KNA8m5Y1C3hoIQbK8y1B0uWYy) hub or explore the curation pipeline using the Dolma GitHub Toolkit. [[1](https://www.google.com/goto?url=CAESZQHrOzAVqDGWZgFkNAiGOTr1a3I1Et4PFMQit55QdiU3kqPsTyA9okSIwbdfjlhNbKbOp0k5Zv-YXrOnL6SgaiaRrWOD1-DMaGhXR8JXgYW-jE3KNA8m5Y1C3hoIQbK8y1B0uWYy), [2](https://www.google.com/goto?url=CAESWAHrOzAVLPcsYpsiFGKn0IOrMyoQK7Y1lvcTXMz2p8wmsK0D5ukuSnPSumjXvVgrgy5pvhwIre-i30P5hgiF_kqQNHI9UJXGv-ojqHUcg_DqhuzKfg5AHbo)] Key Features * Scale: Contains roughly 3 trillion tokens (version 1.7 utilizes about 1.715 trillion actual training tokens across ~5.2 billion documents totaling over 5,300 GB). [[1](https://www.google.com/goto?url=CAEShQEB6zswFdYTFzcT8iARHW7j2rb03Q9fzUvuXk92kI5vJW1ys8rU0ZyNqvi9nb9SEfxVE6GDLUXfzIOJ9Mdgxo_wLR3UpEXqPziCd7TsCkT04tfI3NGm7zbcWyRpgISmYcfpVr3bLZKfJFJXV2yp769te1gUIyCeAsDgwURFI-czaqs1Tqxx), [2](https://www.google.com/goto?url=CAESZQHrOzAVqDGWZgFkNAiGOTr1a3I1Et4PFMQit55QdiU3kqPsTyA9okSIwbdfjlhNbKbOp0k5Zv-YXrOnL6SgaiaRrWOD1-DMaGhXR8JXgYW-jE3KNA8m5Y1C3hoIQbK8y1B0uWYy)] * Data Sources: Combines a diverse mix of web content (Common Crawl, C4, RefinedWeb), academic papers (arXiv, Semantic Scholar, peS2o), code (The Stack, StarCoder), public-domain books (Project Gutenberg), encyclopedic material (Wikipedia, Wikibooks), and social media discussions (Reddit, StackExchange). [[1](https://www.google.com/goto?url=CAESZQHrOzAVqDGWZgFkNAiGOTr1a3I1Et4PFMQit55QdiU3kqPsTyA9okSIwbdfjlhNbKbOp0k5Zv-YXrOnL6SgaiaRrWOD1-DMaGhXR8JXgYW-jE3KNA8m5Y1C3hoIQbK8y1B0uWYy), [2](https://www.google.com/goto?url=CAEShQEB6zswFdYTFzcT8iARHW7j2rb03Q9fzUvuXk92kI5vJW1ys8rU0ZyNqvi9nb9SEfxVE6GDLUXfzIOJ9Mdgxo_wLR3UpEXqPziCd7TsCkT04tfI3NGm7zbcWyRpgISmYcfpVr3bLZKfJFJXV2yp769te1gUIyCeAsDgwURFI-czaqs1Tqxx)] * Curation Pipeline: Involves language filtering, Gopher/C4 quality heuristics, Bloom filter deduplication (by URL, paragraph, and document), PII/toxicity masking, and evaluation decontamination. [[1](https://www.google.com/goto?url=CAESfgHrOzAVf6rn5OZttIM0us6WvAir5Q1-WR0EyYm9ecTP7XBTIM-DinlhVME8i_KIkf8LZ3DAOz14u-sSPbfl_Pc4u7hcHYX6Q1oBjXL5hFzJgdfGrRKVE1UMINo78prcPEMu19EHqLlLGx8laR-2Dzu5RINQkc4u25jqyRpHwQ), [2](https://www.google.com/goto?url=CAEShQEB6zswFdYTFzcT8iARHW7j2rb03Q9fzUvuXk92kI5vJW1ys8rU0ZyNqvi9nb9SEfxVE6GDLUXfzIOJ9Mdgxo_wLR3UpEXqPziCd7TsCkT04tfI3NGm7zbcWyRpgISmYcfpVr3bLZKfJFJXV2yp769te1gUIyCeAsDgwURFI-czaqs1Tqxx)] * Licensing: Released under permissive research licenses (such as ODC-BY and Ai2's ImpACT license) alongside the [Dolma ACL Research Paper](https://www.google.com/goto?url=CAESYwHrOzAVvKhxmIIzUklJEDSJyWY8Ro1YnSpFLFxZhQ0EcW4hazs3O-L1Y8BHPxI9cIxyUxxa4A8aVoomoKlEYiUEFBtlTpFdl0qVShVTplvTAOrrsSsWQ_H2MW35T8flseiMVA). [[1](https://www.google.com/goto?url=CAESYwHrOzAVvKhxmIIzUklJEDSJyWY8Ro1YnSpFLFxZhQ0EcW4hazs3O-L1Y8BHPxI9cIxyUxxa4A8aVoomoKlEYiUEFBtlTpFdl0qVShVTplvTAOrrsSsWQ_H2MW35T8flseiMVA), [2](https://www.google.com/goto?url=CAESZQHrOzAVqDGWZgFkNAiGOTr1a3I1Et4PFMQit55QdiU3kqPsTyA9okSIwbdfjlhNbKbOp0k5Zv-YXrOnL6SgaiaRrWOD1-DMaGhXR8JXgYW-jE3KNA8m5Y1C3hoIQbK8y1B0uWYy), [3](https://www.google.com/goto?url=CAEShQEB6zswFdYTFzcT8iARHW7j2rb03Q9fzUvuXk92kI5vJW1ys8rU0ZyNqvi9nb9SEfxVE6GDLUXfzIOJ9Mdgxo_wLR3UpEXqPziCd7TsCkT04tfI3NGm7zbcWyRpgISmYcfpVr3bLZKfJFJXV2yp769te1gUIyCeAsDgwURFI-czaqs1Tqxx)] * Primary Use: Designed and used specifically to pre-train the [OLMo](https://www.google.com/goto?url=CAESfgHrOzAVf6rn5OZttIM0us6WvAir5Q1-WR0EyYm9ecTP7XBTIM-DinlhVME8i_KIkf8LZ3DAOz14u-sSPbfl_Pc4u7hcHYX6Q1oBjXL5hFzJgdfGrRKVE1UMINo78prcPEMu19EHqLlLGx8laR-2Dzu5RINQkc4u25jqyRpHwQ) open-source language model family. [[1](https://www.google.com/goto?url=CAESZQHrOzAVqDGWZgFkNAiGOTr1a3I1Et4PFMQit55QdiU3kqPsTyA9okSIwbdfjlhNbKbOp0k5Zv-YXrOnL6SgaiaRrWOD1-DMaGhXR8JXgYW-jE3KNA8m5Y1C3hoIQbK8y1B0uWYy), [2](https://www.google.com/goto?url=CAESfgHrOzAVf6rn5OZttIM0us6WvAir5Q1-WR0EyYm9ecTP7XBTIM-DinlhVME8i_KIkf8LZ3DAOz14u-sSPbfl_Pc4u7hcHYX6Q1oBjXL5hFzJgdfGrRKVE1UMINo78prcPEMu19EHqLlLGx8laR-2Dzu5RINQkc4u25jqyRpHwQ)] If you'd like, let me know: * Are you planning to train or fine-tune an LLM using Dolma? * Are you looking for specific subsets or data processing scripts? I can provide the right code commands or pipeline details. Hugging Face allenai/dolma · Datasets at Hugging Face Overview: AllenAI Dolma is an open corpus dataset containing 3 trillion tokens (version 1.7 utilizing 1.715 trillion actual training tokens) designed for langua... GitHub Pages documentation dolma | Data and tools for generating and inspecting OLMo pre-training data. Dataset: Open 3-trillion-token corpus (web content, academic publications, code, books, encyclopedic materials) created as training data for OLMo by AI2; availa... Allen Institute for Artificial Intelligence Ai2 Dolma: 3 trillion token open corpus for language model pretraining | Ai2 Dataset: Dolma (3 trillion tokens, 5,245M documents, 5,334 GB) developed by Ai2 for OLMo pretraining. Sources: CommonCrawl, C4, peS2o, The Stack, Project Gutenb... arXiv [2402.00159] Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research Overview: Dolma is an open-source, three-trillion-token English corpus designed for language model pretraining research, addressing the scarcity of publicly ava... Deep (Learning) Focus·Deep (Learning) Focus Dolma, OLMo, and the Future of Open-Source LLMs Overview: Dolma (3T-token open pretraining corpus) and OLMo (fully open-source LLM suite) demystify language model development by openly releasing training data... ACL Anthology Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research - ACL Anthology Resource: Dolma (ACL 2024 Best Resource Paper Award). Corpus: Open English dataset containing three trillion tokens for language model pretraining research. Con... AI responses may include mistakes. [Learn more](https://support.google.com/websearch?p=ai_overviews)