[2402.00159] Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research 

 Skip to main content 

 Search

 Submit 
 Donate 

 Log in 

 Search arXiv 

 Press Enter to search · Advanced search 

--> 

 Computer Science > Computation and Language

 arXiv:2402.00159 (cs)

 [Submitted on 31 Jan 2024 ( v1 ), last revised 6 Jun 2024 (this version, v2)]

 Title: Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research

 Authors: Luca Soldaini , Rodney Kinney , Akshita Bhagia , Dustin Schwenk , David Atkinson , Russell Authur , Ben Bogin , Khyathi Chandu , Jennifer Dumas , Yanai Elazar , Valentin Hofmann , Ananya Harsh Jha , Sachin Kumar , Li Lucy , Xinxi Lyu , Nathan Lambert , Ian Magnusson , Jacob Morrison , Niklas Muennighoff , Aakanksha Naik , Crystal Nam , Matthew E. Peters , Abhilasha Ravichander , Kyle Richardson , Zejiang Shen , Emma Strubell , Nishant Subramani , Oyvind Tafjord , Pete Walsh , Luke Zettlemoyer , Noah A. Smith , Hannaneh Hajishirzi , Iz Beltagy , Dirk Groeneveld , Jesse Dodge , Kyle Lo 
 View a PDF of the paper titled Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research, by Luca Soldaini and 35 other authors

 View PDF 
 HTML (experimental) 

 Abstract: Information about pretraining corpora used to train the current best-performing language models is seldom discussed: commercial models rarely detail their data, and even open models are often released without accompanying training data or recipes to reproduce them. As a result, it is challenging to conduct and advance scientific research on language modeling, such as understanding how training data impacts model capabilities and limitations. To facilitate scientific research on language model pretraining, we curate and release Dolma, a three-trillion-token English corpus, built from a diverse mixture of web content, scientific papers, code, public-domain books, social media, and encyclopedic materials. We extensively document Dolma, including its design principles, details about its construction, and a summary of its contents. We present analyses and experimental results on intermediate states of Dolma to share what we have learned about important data curation practices. Finally, we open-source our data curation toolkit to enable reproduction of our work as well as support further research in large-scale data curation.

 Comments:

 Accepted at ACL 2024; Dataset: this https URL Code: this https URL 

 Subjects:

 Computation and Language (cs.CL) 

 Cite as:

 arXiv:2402.00159 [cs.CL] 

 (or 
 arXiv:2402.00159v2 [cs.CL] for this version)

 https://doi.org/10.48550/arXiv.2402.00159 

 Focus to learn more 

 arXiv-issued DOI via DataCite

 Submission history
 From: Luca Soldaini [ view email ] 
 [v1] 
 Wed, 31 Jan 2024 20:29:50 UTC (7,715 KB)

 [v2] 
 Thu, 6 Jun 2024 18:46:40 UTC (6,680 KB)

 Full-text links: 
 Access Paper:

View a PDF of the paper titled Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research, by Luca Soldaini and 35 other authors
 View PDF 
 HTML (experimental) 
 TeX Source

 view license 

 Current browse context:

 cs.CL

 < prev 

   |   
 next > 

 new 
 | 
 recent 
 | 2024-02 

 Change to browse by:

 cs 

 References & Citations

 NASA ADS 
 Google Scholar 

 Semantic Scholar 

 export BibTeX citation 
 Loading... 

 BibTeX formatted citation

 × 

 loading... 

 Data provided by: 

 Bookmark

 Bibliographic Tools 

 Bibliographic and Citation Tools

 Bibliographic Explorer Toggle 

 Bibliographic Explorer ( What is the Explorer? ) 

 Connected Papers Toggle 

 Connected Papers ( What is Connected Papers? ) 

 Litmaps Toggle 

 Litmaps ( What is Litmaps? ) 

 scite.ai Toggle 

 scite Smart Citations ( What are Smart Citations? ) 

 Code, Data, Media 

 Code, Data and Media Associated with this Article

 alphaXiv Toggle 

 alphaXiv ( What is alphaXiv? ) 

 Links to Code Toggle 

 CatalyzeX Code Finder for Papers ( What is CatalyzeX? ) 

 DagsHub Toggle 

 DagsHub ( What is DagsHub? ) 

 GotitPub Toggle 

 Gotit.pub ( What is GotitPub? ) 

 Huggingface Toggle 

 Hugging Face ( What is Huggingface? ) 

 ScienceCast Toggle 

 ScienceCast ( What is ScienceCast? ) 

 Demos 

 Demos

 Replicate Toggle 

 Replicate ( What is Replicate? ) 

 Spaces Toggle 

 Hugging Face Spaces ( What is Spaces? ) 

 Spaces Toggle 

 TXYZ.AI ( What is TXYZ.AI? ) 

 Related Papers 

 Recommenders and Search Tools

 Link to Influence Flower 

 Influence Flower ( What are Influence Flowers? ) 

 Core recommender toggle 

 CORE Recommender ( What is CORE? ) 

 Author 

 Venue 

 Institution 

 Topic 

 About arXivLabs

 arXivLabs: experimental projects with community collaborators

 arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

 Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

 Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

 Which authors of this paper are endorsers? |
 Disable MathJax ( What is MathJax? )

 We gratefully acknowledge support from
 our major funders ,
 member institutions , ,
 and all contributors.

 About 
 · 
 Help 
 · 
 Contact 
 · 
 Subscribe 
 · 
 Copyright 
 · 
 Privacy 
 · 
 Accessibility 
 · 
 Operational Status (opens in new tab) 

 Major funding support from