AI Mode Conversation
You said: Dolma datasetDolma dataset
Dolma is an open-source 3-trillion-token English corpus created by the [Allen Institute for Artificial Intelligence (Ai2)](https://www.google.com/goto?url=CAEShQEB6zswFdYTFzcT8iARHW7j2rb03Q9fzUvuXk92kI5vJW1ys8rU0ZyNqvi9nb9SEfxVE6GDLUXfzIOJ9Mdgxo_wLR3UpEXqPziCd7TsCkT04tfI3NGm7zbcWyRpgISmYcfpVr3bLZKfJFJXV2yp769te1gUIyCeAsDgwURFI-czaqs1Tqxx) to support language model pretraining and transparency. [[1](https://www.google.com/goto?url=CAESWAHrOzAVfuCTDUTg4GGhhC3F70zMJsaeToVocN310KpVcxm35YicdR3Yfao1ze76K1TcwOI70UkQvtYaIxXXlQS2YEUAPbNuGdAYtdcU3sz835DbRODiVjw)]
You can access and download the files directly via the [AllenAI Dolma on Hugging Face](https://www.google.com/goto?url=CAESZQHrOzAVqDGWZgFkNAiGOTr1a3I1Et4PFMQit55QdiU3kqPsTyA9okSIwbdfjlhNbKbOp0k5Zv-YXrOnL6SgaiaRrWOD1-DMaGhXR8JXgYW-jE3KNA8m5Y1C3hoIQbK8y1B0uWYy) hub or explore the curation pipeline using the Dolma GitHub Toolkit. [[1](https://www.google.com/goto?url=CAESZQHrOzAVqDGWZgFkNAiGOTr1a3I1Et4PFMQit55QdiU3kqPsTyA9okSIwbdfjlhNbKbOp0k5Zv-YXrOnL6SgaiaRrWOD1-DMaGhXR8JXgYW-jE3KNA8m5Y1C3hoIQbK8y1B0uWYy), [2](https://www.google.com/goto?url=CAESWAHrOzAVLPcsYpsiFGKn0IOrMyoQK7Y1lvcTXMz2p8wmsK0D5ukuSnPSumjXvVgrgy5pvhwIre-i30P5hgiF_kqQNHI9UJXGv-ojqHUcg_DqhuzKfg5AHbo)]
Key Features

* Scale: Contains roughly 3 trillion tokens (version 1.7 utilizes about 1.715 trillion actual training tokens across ~5.2 billion documents totaling over 5,300 GB). [[1](https://www.google.com/goto?url=CAEShQEB6zswFdYTFzcT8iARHW7j2rb03Q9fzUvuXk92kI5vJW1ys8rU0ZyNqvi9nb9SEfxVE6GDLUXfzIOJ9Mdgxo_wLR3UpEXqPziCd7TsCkT04tfI3NGm7zbcWyRpgISmYcfpVr3bLZKfJFJXV2yp769te1gUIyCeAsDgwURFI-czaqs1Tqxx), [2](https://www.google.com/goto?url=CAESZQHrOzAVqDGWZgFkNAiGOTr1a3I1Et4PFMQit55QdiU3kqPsTyA9okSIwbdfjlhNbKbOp0k5Zv-YXrOnL6SgaiaRrWOD1-DMaGhXR8JXgYW-jE3KNA8m5Y1C3hoIQbK8y1B0uWYy)]
* Data Sources: Combines a diverse mix of web content (Common Crawl, C4, RefinedWeb), academic papers (arXiv, Semantic Scholar, peS2o), code (The Stack, StarCoder), public-domain books (Project Gutenberg), encyclopedic material (Wikipedia, Wikibooks), and social media discussions (Reddit, StackExchange). [[1](https://www.google.com/goto?url=CAESZQHrOzAVqDGWZgFkNAiGOTr1a3I1Et4PFMQit55QdiU3kqPsTyA9okSIwbdfjlhNbKbOp0k5Zv-YXrOnL6SgaiaRrWOD1-DMaGhXR8JXgYW-jE3KNA8m5Y1C3hoIQbK8y1B0uWYy), [2](https://www.google.com/goto?url=CAEShQEB6zswFdYTFzcT8iARHW7j2rb03Q9fzUvuXk92kI5vJW1ys8rU0ZyNqvi9nb9SEfxVE6GDLUXfzIOJ9Mdgxo_wLR3UpEXqPziCd7TsCkT04tfI3NGm7zbcWyRpgISmYcfpVr3bLZKfJFJXV2yp769te1gUIyCeAsDgwURFI-czaqs1Tqxx)]
* Curation Pipeline: Involves language filtering, Gopher/C4 quality heuristics, Bloom filter deduplication (by URL, paragraph, and document), PII/toxicity masking, and evaluation decontamination. [[1](https://www.google.com/goto?url=CAESfgHrOzAVf6rn5OZttIM0us6WvAir5Q1-WR0EyYm9ecTP7XBTIM-DinlhVME8i_KIkf8LZ3DAOz14u-sSPbfl_Pc4u7hcHYX6Q1oBjXL5hFzJgdfGrRKVE1UMINo78prcPEMu19EHqLlLGx8laR-2Dzu5RINQkc4u25jqyRpHwQ), [2](https://www.google.com/goto?url=CAEShQEB6zswFdYTFzcT8iARHW7j2rb03Q9fzUvuXk92kI5vJW1ys8rU0ZyNqvi9nb9SEfxVE6GDLUXfzIOJ9Mdgxo_wLR3UpEXqPziCd7TsCkT04tfI3NGm7zbcWyRpgISmYcfpVr3bLZKfJFJXV2yp769te1gUIyCeAsDgwURFI-czaqs1Tqxx)]
* Licensing: Released under permissive research licenses (such as ODC-BY and Ai2's ImpACT license) alongside the [Dolma ACL Research Paper](https://www.google.com/goto?url=CAESYwHrOzAVvKhxmIIzUklJEDSJyWY8Ro1YnSpFLFxZhQ0EcW4hazs3O-L1Y8BHPxI9cIxyUxxa4A8aVoomoKlEYiUEFBtlTpFdl0qVShVTplvTAOrrsSsWQ_H2MW35T8flseiMVA). [[1](https://www.google.com/goto?url=CAESYwHrOzAVvKhxmIIzUklJEDSJyWY8Ro1YnSpFLFxZhQ0EcW4hazs3O-L1Y8BHPxI9cIxyUxxa4A8aVoomoKlEYiUEFBtlTpFdl0qVShVTplvTAOrrsSsWQ_H2MW35T8flseiMVA), [2](https://www.google.com/goto?url=CAESZQHrOzAVqDGWZgFkNAiGOTr1a3I1Et4PFMQit55QdiU3kqPsTyA9okSIwbdfjlhNbKbOp0k5Zv-YXrOnL6SgaiaRrWOD1-DMaGhXR8JXgYW-jE3KNA8m5Y1C3hoIQbK8y1B0uWYy), [3](https://www.google.com/goto?url=CAEShQEB6zswFdYTFzcT8iARHW7j2rb03Q9fzUvuXk92kI5vJW1ys8rU0ZyNqvi9nb9SEfxVE6GDLUXfzIOJ9Mdgxo_wLR3UpEXqPziCd7TsCkT04tfI3NGm7zbcWyRpgISmYcfpVr3bLZKfJFJXV2yp769te1gUIyCeAsDgwURFI-czaqs1Tqxx)]
* Primary Use: Designed and used specifically to pre-train the [OLMo](https://www.google.com/goto?url=CAESfgHrOzAVf6rn5OZttIM0us6WvAir5Q1-WR0EyYm9ecTP7XBTIM-DinlhVME8i_KIkf8LZ3DAOz14u-sSPbfl_Pc4u7hcHYX6Q1oBjXL5hFzJgdfGrRKVE1UMINo78prcPEMu19EHqLlLGx8laR-2Dzu5RINQkc4u25jqyRpHwQ) open-source language model family. [[1](https://www.google.com/goto?url=CAESZQHrOzAVqDGWZgFkNAiGOTr1a3I1Et4PFMQit55QdiU3kqPsTyA9okSIwbdfjlhNbKbOp0k5Zv-YXrOnL6SgaiaRrWOD1-DMaGhXR8JXgYW-jE3KNA8m5Y1C3hoIQbK8y1B0uWYy), [2](https://www.google.com/goto?url=CAESfgHrOzAVf6rn5OZttIM0us6WvAir5Q1-WR0EyYm9ecTP7XBTIM-DinlhVME8i_KIkf8LZ3DAOz14u-sSPbfl_Pc4u7hcHYX6Q1oBjXL5hFzJgdfGrRKVE1UMINo78prcPEMu19EHqLlLGx8laR-2Dzu5RINQkc4u25jqyRpHwQ)]

If you'd like, let me know:

* Are you planning to train or fine-tune an LLM using Dolma?
* Are you looking for specific subsets or data processing scripts?

I can provide the right code commands or pipeline details.
Hugging Face
allenai/dolma · Datasets at Hugging Face
Overview: AllenAI Dolma is an open corpus dataset containing 3 trillion tokens (version 1.7 utilizing 1.715 trillion actual training tokens) designed for langua...
GitHub Pages documentation
dolma | Data and tools for generating and inspecting OLMo pre-training data.
Dataset: Open 3-trillion-token corpus (web content, academic publications, code, books, encyclopedic materials) created as training data for OLMo by AI2; availa...
Allen Institute for Artificial Intelligence
Ai2 Dolma: 3 trillion token open corpus for language model pretraining | Ai2
Dataset: Dolma (3 trillion tokens, 5,245M documents, 5,334 GB) developed by Ai2 for OLMo pretraining. Sources: CommonCrawl, C4, peS2o, The Stack, Project Gutenb...
arXiv
[2402.00159] Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
Overview: Dolma is an open-source, three-trillion-token English corpus designed for language model pretraining research, addressing the scarcity of publicly ava...
Deep (Learning) Focus·Deep (Learning) Focus
Dolma, OLMo, and the Future of Open-Source LLMs
Overview: Dolma (3T-token open pretraining corpus) and OLMo (fully open-source LLM suite) demystify language model development by openly releasing training data...
ACL Anthology
Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research - ACL Anthology
Resource: Dolma (ACL 2024 Best Resource Paper Award). Corpus: Open English dataset containing three trillion tokens for language model pretraining research. Con...
AI responses may include mistakes. [Learn more](https://support.google.com/websearch?p=ai_overviews)
