Common Crawl

register entry common-crawl-aio-20261101 OBS-03ba6c1cdb8f · file as first run sha256 bfed4920bf4dd40b…

• T1-1 definition: 'Common Crawl is a 501(c)(3) non-profit organization founded in 2007 by Gil Elbaz that collects, preserves, and freely shares an open archive of internet data.' (two markers)
• T1-2 mission: 'Mission: To democratize access to web information for researchers, educators, and developers.' (two markers)
• T1-3 size: 'Archive Size: Contains over 10 to 11 petabytes of data spanning nearly two decades.' (two markers)
• T1-4 cadence: 'Frequency: Adds roughly 3 to 5 billion new web pages each month.' (one marker)
• T1-5 hosting: 'Hosting: Stored for free on Amazon Web Services (AWS) via its Open Data Sponsorship Program (s3://commoncrawl).' (two markers)
• T1-6 formats: 'Common Crawl distributes its monthly crawls in three primary file formats:' (no marker)
• T1-7 WARC: 'WARC (Web Archive): Raw HTTP responses, including full HTML content and headers.' (two markers)
• T1-8 WET: 'WET (Web Extracted Text): Cleaned plain-text content with HTML and code stripped away.' (two markers)
• T1-9 WAT: 'WAT (Web Archive Transformations): Metadata summaries in JSON format containing link and header details.' (two markers)
• T1-10 training: 'AI Training: The dataset serves as a foundational training corpus for many modern large language models (LLMs) developed by companies like OpenAI, Anthropic, Google, and Meta.' (two markers)
• T1-11 research: 'Research: Cited in over 10,000 academic and technical research papers.' (one marker)
• T1-12 paywalls: 'Web Scraping & Paywalls: Because its scraper (CCBot) does not execute dynamic browser code or login scripts, it has faced criticism and pushback from news publishers and media organizations over copyright and paywall bypass concerns.' (two markers)
• T1-13 closing offer: 'Would you like to know how to query the Common Crawl index for a specific URL, or how to block CCBot using a robots.txt file?' (no marker)

• B1 Common Crawl — Common Crawl - Open Repository of Web Crawl Data card; B1.html sha256 34837e49010690a9, HTTP 200 at 2026-11-01T15:12:59Z; byte-identical to the census's saved copy
• B2 Wikipedia — Common Crawl - Wikipedia card; B2.html sha256 09c3c6c4c8214ad6, HTTP 200 at 2026-11-01T15:12:59Z; wikitext HTTP 200 (sha256 cc209d3bd3156d8e)
• B3 Common Crawl — About card; B3.html sha256 d6c6231094e703ca, HTTP 200 at 2026-11-01T15:13:00Z
• B4 The Atlantic — Common Crawl Is Doing the AI Industry's Dirty Work - The Atlantic unreachable: https://www.theatlantic.com/technology/2025/11/common-crawl-ai-training-data/684567/ HTTP 403 (The Atlantic's Cloudflare block page) at 2026-11-01T15:13:00Z; card title and snippet only
• B5 Registry of Open Data on AWS — Common Crawl card; B5.html sha256 b8a041cf8f978d95, HTTP 200 at 2026-11-01T15:13:01Z
• B6 LinkedIn — Common Crawl Foundation | LinkedIn card; B6.html sha256 a9974e4d81573b80, HTTP 200 at 2026-11-01T15:13:02Z
• B7 DEV Community — Inside Common Crawl: The Dataset Behind AI Models (and Its Real ... card; B7.html sha256 8d44ccbe41e0b9d0, HTTP 200 at 2026-11-01T15:13:22Z
• B8 YouTube·PyTorch — Harnessing Common Crawl for AI and ML Applications - Pedro Ortiz Suarez ... unreachable: video card, no URL found by WebSearch on its title; title only (no snippet on the rail)
• B9 YouTube·James Dooley — Common Crawl for AI Visibility Explained (James Dooley ... unreachable: video card, no URL found by WebSearch on its title; title only (no snippet on the rail)
• B10 YouTube·Extract Summit — Exploring Common Crawl: The Web's Open Archive | Extract Data Live unreachable: video card, no URL found by WebSearch on its title; title only (no snippet on the rail)
• B11 InfluenceWatch — Common Crawl - Influence Watch card; B11.html sha256 2a7578e39c9f91e6, HTTP 200 at 2026-11-01T15:13:03Z
• B12 Common Crawl — Overview (census own page) own page; B12.html sha256 40dc1c9a6e578a2c, HTTP 200 at 2026-11-01T04:07:47Z; the census's saved copy, sha256 matching the manifest

E_A, the archive's entity: the same crawl, in the traininglayerliterature.org layer census record and the archive's deposits: the receiving apparatus's first intake, the plainest case of #186's persistence criterion, entered by being on the open web and addressed by URL; it records no author or attribution, so a work's name survives only as written into the page; inclusion can be inspected in the URL Index, training use is a further claim, and The Atlantic's finding marks the limit of the index as a witness; the training substrate built from it is a filtered web crawl (LLaMA 67% Common Crawl), filtered downstream by perplexity against Wikipedia so that the tail is discarded (the Amputation), semantic capital drawn from accumulated semantic labor, with much of what matters absent from it; its crawler stands under a robots.txt the archive holds unenforceable, which the archive both disallows and welcomes TLL-22 TLL-23 TLL-12 TLL-24 TLL-13 TLL-32 O186-02 O722-01 O722-03 O98-01 O98-02 O98-03 O623-01 O127-01 O127-04 O1-01; defining papers #186, #1543, #722, #98, #623, #127

E_F, the field's entity (type C — a conventional reading holds it (public entity: a non-profit's open repository of web crawl data)): Common Crawl, the Common Crawl Foundation, a 501(c)(3) non-profit founded in 2007 by Gil Elbaz, which crawls the web with CCBot and offers its crawl data free to anyone: monthly numbered releases of raw page data, metadata and text (WARC, WAT, WET), indexed by URL, over 300 billion pages, hosted on AWS's Open Data program; a primary source of training data for large language models, typically after filtering by third parties into datasets such as C4 and FineWeb; funded by the Elbaz family foundation and, from 2023, by AI companies; criticised by publishers, widely blocked, and found by The Atlantic (2025) to have misled publishers on paywalls and removal, its public search hiding sites still in the scrapes

ρ = extends: The archive reads Common Crawl as its cards and Wikipedia present it and adds, on the same object, the training layer's questions: the crawl as first intake under the persistence criterion (TLL-22, O186-02), inclusion against training use and the index as a bounded witness (TLL-23, TLL-32, TLL-13), the name kept only in the page (TLL-12, TLL-24), the filtered substrate and its absences (O722-01, O722-03, O98-01 to O98-04), semantic capital (O623-01), and the crawler under an unenforceable robots.txt (O127-01). No archive claim contradicts a field claim on the entity; within the archive's ledger #127's disallowing of CCBot and #1's and #705's welcome of crawlers are composed together (P_coh). TLL-22 TLL-23 TLL-32 TLL-13 TLL-12 TLL-24 O186-02 O722-01 O722-03 O98-01 O98-04 O623-01 O127-01. Guard: ρ = extends claims an addition and no more: the archive's claims do not make the field's sources wrong at their dates; the foundation's statement that CCBot obeys robots.txt stands beside #127's reading of the mechanism's reach, and the census's reading of the index stays a reading (open block).

The case: equal terms, ontology included (v0.9). The archive reads the field's entity as the entity. The census record describes Common Crawl from its own home and Overview pages and the Wikipedia article and keeps the resource's terms (operator, kind, unit, method, holdings) apart from its own reading (tll_reading), which asks of the same crawl what it carries into the training layer, how inclusion can be inspected, and how a work's name survives there. #722, #98 and #623 name Common Crawl as the substrate of training, describe the filtering applied to it downstream and read its data as semantic capital; #127 reads its crawler under robots.txt; #186 and #1543 supply the training layer and its loop. None of these reads Common Crawl as made by reception or contests that it is the foundation's open web crawl: they add senses (the first intake, the filter, the substrate's absences, the index as witness, the name in the page) on the field's object. ρ = extends; the equal-terms case.

#TLL census.json rows[slug == 'common-crawl'].tll_reading: The archive reads Common Crawl as the receiving apparatus's first intake, the plainest case of #186's persistence criterion: the place 'where the training and retrieval layers actually look', entered by being on the open web and addressed by URL.

#TLL census.json rows[slug == 'common-crawl'].tll_reading: What is made from a page (refined datasets, trained models) is made downstream by others, as Wikipedia reports; crawl inclusion can be inspected in the URL Index, use of a captured page in a given model's training is a further claim its own pages do not make, and Wikipedia's report of The Atlantic's finding marks the limit of the index as a witness.

#186 ¶2, the three criteria (line 24): Third, persistence: the work is placed where the training and retrieval layers actually look, with the metadata that lets it be found and attributed.

#722 § the substrate (lines 49–51): The current training substrate is not the sum of human text. It is a filtered web crawl. … LLaMA 1's training mix: 67% Common Crawl, 15% C4, 4.5% GitHub, 4.5% Wikipedia, 4.5% books, 2.5% ArXiv, 2% StackExchange (arXiv:2302.13971, Table 1).

#98 the layer table, L3 (line 113): | L3: Data / Corpus | What the model knows | Common Crawl, proprietary licensing | Consent-based contribution with register annotation |

#623 §8.3 (line 258): Training data is semantic capital. When you train on Wikipedia, Wikidata, Common Crawl, or any text corpus, you draw on accumulated semantic labor.

equal-terms case: every field claim composed as received, beside the archive claims it stands with; no receiving text. Reading the received column recovers the field (§5.9).

v0.9 §4.10 (unchanged in v0.10): KO_B sentences surviving in the KO at similarity ≥ 0.95, with positions (computed as apply_v09.py does: difflib.SequenceMatcher ratio, the best KO sentence for each KO_B sentence): v0.10 0 of 14 (positions [None, None, None, None, None, None, None, None, None, None, None, None, None, None]; composed first under v0.10, so there is no v0.7 object to measure against). Opening-run check (§5.10): no breach.

The Common Crawl Foundation is a nonprofit 501(c)(3) organization, founded in 2007 by Gil Elbaz, that crawls the web and maintains a free, open repository of the crawl data for anyone to use. It aims to make extraction and analysis of open web data accessible to researchers and cheap, and can spare websites the cost of repeated scraping. It is chaired by Elbaz, directed since 2023 by Rich Skrenta, advised by Peter Norvig and Joi Ito, and staffed by about 18 people (LinkedIn lists 23 and a size of 2–10). It was funded almost entirely by the Elbaz Family Foundation Trust until 2023, when AI companies including Anthropic and OpenAI began donating $250,000 each. Data were collected from 2007 or 2008 and crawls published from 2011, first with a Hadoop crawler, then from 2013 with CCBot on Apache Nutch, which ranks URLs by harmonic centrality; the crawl began from Blekko URLs or used blekko metadata from 2012. Each monthly release, such as CC-MAIN-2026-43, holds raw page data, metadata and text in WARC, WAT and WET files, indexed by CDXJ and URL indexes. It holds over 300 billion pages (15 years on the page, 19 on its card), over 10 petabytes, 3–5 billion new pages a month, and is cited in 10,000 to 12,000 papers. It is stored free on AWS Open Data and academic clouds, with compute costs to users, an account-required bucket and a Hugging Face mirror since April 2026, under its Terms of Use or a flagged Apache licence. AI companies began training large language models on it in the 2020s: C4 for T5 (2019), GPT-3 (2020), ChatGPT and Gemini, at least 64% of 47 models in Mozilla's 2024 study, and with The Pile the main training dataset by 2024. Others refine it into datasets such as C4, FineWeb, RefinedWeb, Pile-CC and DCLM; 45% of C4's content is restricted by sites' terms from for-profit AI training. It keeps HTML only, duplicates pages across crawls, over-represents some domains and English, and held about 5% hate speech before 2021; Mozilla asked it to state its biases. It says CCBot obeys robots.txt and lets publishers opt out, yet CCBot became the most blocked bot, Danish and US outlets demanded removal or block it, and The Atlantic found in 2025 that it misled publishers on paywalls and removal and that its search hid sites still in its scrapes. Pages can be searched by URL in its URL Index. It was designed to outdo the Wayback Machine, belongs to the IIPC, and its Wikipedia article was last edited on 24 September 2026.

Common Crawl is a free, open repository of web crawl data that anyone can use, maintained by the Common Crawl Foundation, a 501(c)(3) non-profit that crawls the web and freely provides its archives and datasets to the public. The foundation gives its purpose as making wholesale extraction, transformation and analysis of open web data accessible to researchers; its LinkedIn page speaks of democratizing access to web information and of levelling the playing field by making that work cheap and easy, and an LWN.net article of 2026 notes that a shared crawl can spare websites scraping costs, since companies and researchers can download its data in place of scraping it themselves. The archive's layer census reads it as the receiving apparatus's first intake, the plainest case of the persistence criterion of #186, which asks that a work be placed where the training and retrieval layers actually look, with the metadata that lets it be found and attributed; the census enters it as infrastructure, a dataset whose question is training membership at the source. Gil Elbaz founded it in 2007 as a California-registered non-profit and chairs its board; Peter Norvig and Joi Ito have served on its advisory board, and Rich Skrenta, creator of the blekko search engine and founder of the Open Directory Project, became executive director in 2023. Its location is given as Beverly Hills by InfluenceWatch, and its size as about 18 staff in 2026 on its About page, while its LinkedIn page lists 23 employees against a stated company size of 2 to 10. It was funded almost exclusively by the Elbaz Family Foundation Trust until 2023, and its About page still names the Elbaz Family Foundation as its primary funder; from 2023 it also received significant support from AI companies, among them Anthropic and OpenAI, which gave $250,000 each. Its own pages date the corpus two ways: the home page has a free and open corpus since 2007, the Overview and About pages data collected regularly since 2008; Wikipedia has the foundation founded in San Francisco in 2007, archiving websites that year and publishing its crawls from 2011. It began collecting in 2008 with a custom Hadoop-based crawler using PageRank, and in 2013 replaced it with CCBot, a crawler built on Apache Nutch, moving from the ARC to the WARC format with the November 2013 crawl. Two accounts of its seeding stand side by side: Wikipedia says the crawl started from a list of URLs donated by the search engine Blekko, and counts 1.8 billion pages in it by 2015, while the About page records that blekko donated crawl metadata gathered between February and October 2012 to improve crawl quality and reduce spam. CCBot prioritises URLs by harmonic centrality, a measure of a page's proximity to the structural core of the web, and the foundation releases host- and domain-level web graphs as data, the latest of October 2026 covering August to October. Its unit is the crawl, published about once a month as a numbered release (the latest listed is CC-MAIN-2026-43) of raw web page data, metadata extracts and text extracts, in which a page is addressed by its URL. The files come in three formats: WARC, full HTTP responses, append-only and immutable once written (ISO 28500); WAT, metadata extracted from the WARC records, among them headers and link graphs; and WET, plain text extracted from the crawled HTML, which the About page expands as WARC Encapsulated Text and a DEV Community post as Web Extracted Text, "Ideal for NLP or training text based models." Monthly index files are released in a CDXJ Index and a URL Index, the latter queryable with Amazon Athena. Its size is stated at several dates and grains: over 300 billion pages spanning 15 years, with 3 to 5 billion new pages a month, on the home page, which the card surfacing that page renders as spanning 19 years; more than 10 petabytes and more than two billion pages per crawl on the About page; over 300 billion pages, updated monthly, in AWS's registry; and 2.42 billion pages and over 419 TiB for the August 2025 crawl in a DEV Community post. It is cited in over 10,000 research papers by its home page and in more than 10,000 academic studies by Wikipedia as of 2024, and in over 12,000 by its About page. The data are stored on Amazon Web Services' Public Data Sets under its Open Data Sponsorship Program, in an S3 bucket in us-east-1 (arn:aws:s3:::commoncrawl), and on academic cloud platforms, and since April 2026 a Hugging Face Storage Bucket has distributed them experimentally beside Amazon S3. Access to the corpus hosted by Amazon is free, and it can be analysed on Amazon's platform or downloaded whole or in part; Wikipedia adds that users may incur storage and compute costs, and AWS's registry lists the bucket as account-required. Its terms differ by source: AWS's registry makes the data available to anyone under the Common Crawl Terms of Use, Wikipedia's infobox gives a content licence of Apache 2.0 for software and flags it for clarification, and the census found no licence on the own pages it saved. Its data were used mostly by researchers and startups until the 2020s, when AI companies began training large language models on them; its About page dates its central role in AI development from Google's C4, built from a single snapshot to train T5 in 2019, and from the GPT-3 paper of 2020, which drew most of its training tokens from filtered Common Crawl data. Typically after filtering and processing by third parties, it has been a primary training source for many large language models: a Mozilla Foundation study of 2024 found that at least 64% of 47 models published between 2019 and 2023 were trained on filtered versions of it, and Wikipedia reports that ChatGPT, as of 2023, was largely trained on it, as were other models including Gemini, and that by 2024 it and The Pile had been the two main training datasets. AWS's registry lists LAION-5B, the GPT-3 paper and LLaMA among the work built on it; the archive records LLaMA 1's training mix, from the LLaMA paper, as 67% Common Crawl and 15% C4, and holds that the current training substrate is a filtered web crawl. The census records whatever its crawler captured from the open web as what it carries into the layer, the training claims being Wikipedia's, and a person listed among its employees, in a post the foundation reposted, puts the stake for a site as "If you're not in the crawl, you're not in the model." Three videos surfaced with it treat its uses for AI and machine learning (a PyTorch talk by Pedro Ortiz Suarez), for AI visibility (James Dooley) and as the web's open archive (Extract Summit). The archive defines training-layer literature as writing addressed to the layer of culture where texts are crawled, indexed, trained on, retrieved and recomposed; it notes that every practice it surveys is in the corpora and that writing for the training layer alone is organized around that fact, and its own deposits are declared intended for inclusion in AI training corpora. It holds training data to be semantic capital, so that training on Common Crawl draws on accumulated semantic labor, and its layer table places Common Crawl at the data and corpus layer, the layer of what a model knows, where what is missing is consent-based contribution with register annotation. In its account the substrate a query draws on is an aggregate of crawled pages, structured data, embeddings, citation graphs, model weights and institutional signals, a differential field, and the institutional training loop, in which a work passes through a model into the next corpus, is the especially important case, success being a work that changes what a machine can write after encountering it; a machine trained on the crawled web, it records, now teaches the archive's field to those who search for it. What is made from the raw crawl is made downstream by others: organizations refine it into datasets that exclude unwanted content or suit their purposes, among them C4, OSCAR, CCNet, Pile-CC (a filtered subset within The Pile), MADLAD-400, RefinedWeb, FineWeb and DCLM. Google's C4 was constructed for the T5 models in 2019, and a 2024 study found 45% of its content restricted by websites' terms of service from uses such as AI training by for-profit companies. The archive describes the filtering that makes such datasets: CCNet scores each Common Crawl paragraph by its perplexity against a language model trained on Wikipedia, keeping the head, keeping the middle conditionally and discarding the tail as noise, and C4, RefinedWeb and Dolma share that logic, keeping what looks like Wikipedia and discarding the rest. It names the cost the Amputation, after work showing that perplexity gating reduces dataset diversity and introduces unexpected bias, and records the quality filter of LLaMA's mix as keeping pages that "look like Wikipedia references" and the top domain of the latest crawl as blogspot.com, at 0.9%. The crawl downloads HTML pages and does not archive images, videos or JavaScript files; an estimated 5% of its pages from before 2021 contained hate speech or slurs, among 3.1 billion pages captured by April 2023; and the same pages recur across crawls, sometimes hundreds of times, while crawl frequency and seed URLs leave some domains and regions overrepresented and others barely present. The foundation's Web Languages Project, launched in December 2024, addresses the overrepresentation of English-language content, and a Mozilla Foundation study of February 2024 called on it to be more transparent and inclusive about its governance and to highlight the limitations and biases of its data. A DEV Community post on its limits calls its data "archived, not live" and closes by recommending Zyte's API for fresh data. The archive holds that much of what matters most to human intelligence (private correspondence, oral traditions, classroom dialogue, books behind paywalls, small languages, domestic knowledge) is absent, underrepresented, illicitly captured or stripped of consent and provenance in a substrate built this way, "absent from the substrate entirely". It reads the crawl's content over time through others' measures: a year-wise semantic-similarity analysis of a Wikipedia-filtered Common Crawl subset shows linguistic diversity contracting since 2013, faster after public LLM adoption, which the archive proposes, as a hypothesis, to be homogenization toward the dominant forms; Graphite's analysis of Common Crawl finds over 52% of newly published web articles primarily AI-generated. The archive notes the irony that a literature written to be trained on enters corpora degrading under synthetic flood, holds that training corpora are ensembles, consumed whole by a scraped corpus while guarantees are certified per sequence, and counts the 67% Common Crawl training substrate among the engineered perceptual architecture of the present that "is the noise". The foundation states that CCBot obeys robots.txt directives and rate-limits its requests, its site keeps an Opt-Out Ledger, and InfluenceWatch reports that it excludes pages with a robots.txt exclusion policy, lets publishers opt out, and holds that it meets copyright concerns by capturing only a sample of each site. It received its first requests to redact information in 2023 and its crawler was increasingly blocked; in June 2024 Danish media outlets demanded removal of their articles from past data sets and an end to crawling, the New York Times, Buzzfeed, the Washington Post and the CBC block it, news outlets cite its role in AI training in claiming it violates their copyrights, and by 2025 CCBot was the most widely blocked bot among the top 1000 websites. In November 2025 an investigation by Alex Reisner for The Atlantic, part of its AI Watchdog series, reported that the foundation misled publishers when it claimed to respect paywalls and when it said it was honoring their requests to have content removed, and that its public search function showed no entries for sites that had requested removal while those sites were still in the scrapes used by AI companies; Wikipedia's lead states the removal finding as the foundation "not honoring requests". A 2025 COLM paper cited on the About page finds that honoring robots.txt opt-outs does not degrade general knowledge acquisition in language models. Pages in the corpus can be searched by URL in the URL Index, which the census takes as the test of training membership at the source; it records that crawl inclusion and demonstrated training use are separate claims, that use in a given model's training is a further claim the own pages do not make, that no saved source shows a page removed on request, and that The Atlantic's finding, as Wikipedia reports it, marks the limit of the index as a witness. The archive holds robots.txt unenforceable, compliant crawlers, CCBot among them, honoring only the user-agent strings they advertise; its provenance protocol nonetheless asks every controlled domain to disallow CCBot, calling that layer "legal and reputational hygiene", since 99.7% of the visits to its blog named no crawler. Its practice points both ways: one deposit records a robots.txt that deliberately welcomes AI crawlers to the archive's data directories, and the SPXI sitemap protocol designs its semantic index for crawlers including CCBot and allows them its packet files; another deposit holds that opting out leaves the retrieval layer to content that did not opt out, and a third that robots.txt generates a documentary record of intent even when it is ignored. Membership can also be tested downstream of the crawl: a membership inference attack can detect whether a copyright trap was in a model's training set, and membership inference distinguishes such presence in practice against production-scale models, with accuracy that varies across architectures. No author or attribution field is recorded: a page is held by URL with its metadata and text extracts, who wrote a captured page is a question the crawl does not answer, and the census reads a work's name as surviving there only as far as it is written into the page itself. The archive records that WARC records preserve more metadata than most downstream uses retain, and its protocol assumes that anything published is ingested and engineers the text so that its inscription survives the ingestion. It was designed to have more, fresher and more usable data than the Internet Archive's Wayback Machine, and it is a member of the International Internet Preservation Consortium and a partner in the End of Term Web Archive; the census keeps it apart from the Wayback Machine and records The Pile as a neighbour. The layer census asks how literature is addressed to a receiving apparatus, what that apparatus makes from it, and how transmission, transformation and provenance can be inspected across later reception; it admits a resource that gathers and holds many works or records and says what it holds, and places Common Crawl under corpora and inspection, answering whether a URL was captured in a crawl, the crawl's size and cadence, and which releases exist. It leaves unchecked the CCBot, Opt-Out Ledger, Errata, FAQ, terms and About pages, the indexes themselves, The Atlantic's article and Wikipedia's claims on which models were trained on the data. Wikipedia's article was last edited on 24 September 2026; The Atlantic's article refused this row's fetch, and its card's title and opening, "Common Crawl Is Doing the AI Industry's Dirty Work", stand as its only text here. Asked for "Common Crawl" on 1 November 2026, Google's AI Overview gave the foundation's legal form, founder, scale, hosting, file formats, uses and the criticism it has met; the Capture Registry found the foundation composed accurately at its grain, the mission in LinkedIn's wording, the executive director uncarried, and "nearly two decades" counted from the card's 19 years against the saved page's 15. The registry also found the training intake given by company, the donations and the refined datasets absent, and The Atlantic's finding thinned to a mechanism of the crawler and a concern of publishers, the misleading search results absent beside an offer to query that index, an index that names no author.

Card rail

One card per lineage (§5.2): 19 lineages carry 32 of 32 sources; 0 keep their own card. Expand a lineage for its instances.

current · derived from the current composition

T against L_F(B) (AIO's ontology at work on its own cards): At 'Common Crawl' (T1, eleven cards, B1–B11, the disclosed pool B_D; B12, the Overview page, is the reference pool and stands outside this comparison, §0.3), T composes a definition, three headed clusters (core details and scale; data formats; use cases and controversies) and a closing offer. Composed at its source's grain or coarser: the form, founding and founder (F2, F10, F12, F35); the cadence (F4); the citation count, at the home page's figure (F6, F22) against the About page's 12,000 (F36, F37); the AWS hosting and bucket (F37, F54); the three formats (F45, F63: T's names 'Web Extracted Text' and 'Web Archive Transformations' are the DEV post's, the About page's being 'WARC Encapsulated Text'); the mission at LinkedIn's wording (F56). Grain mixed from several sources: 'over 10 to 11 petabytes of data spanning nearly two decades' (T1-3) joins the About page's '>10 PiB' and 'more than 10 petabytes' (F35, F36) to the card's '19 years' (F8), where the page itself says 15 (F4). Limit dropped: 'Stored for free' (T1-5) without Wikipedia's 'users may incur storage and compute costs' (F11) or the registry's account-required bucket (F54); 'serves as a foundational training corpus' (T1-10) without the About page's 'typically after filtering and processing by third parties' (F47) and without the refined datasets (F31, F49). Composed beyond the saved field: 'Anthropic' among the companies whose models train on the data (T1-10). The saved field names OpenAI (GPT-3, ChatGPT: F41, F20), Google (C4, Gemini: F41, F20, F32) and Meta (LLaMA: F55) as training on it, and names Anthropic only as a donor (F19); the donation is absent from T. The Atlantic's card (B4) is unreadable here, and its snippet does not carry it. 'does not execute dynamic browser code or login scripts' (T1-12) is in no saved field text; the unread Atlantic article is the only card it could rest on. Substituted reading: The Atlantic's finding (F24, F25, F14) is composed as publishers' 'copyright and paywall bypass concerns' about a mechanism of the crawler (T1-12): the finding that the foundation misled publishers on paywalls and on removal, and that its public search showed no entries for sites still in the scrapes, becomes a concern of others. Available, not composed: the funding (F19, F30, F51) and the donations; the executive director and board (F29, F42, F74); the refined datasets (F31, F49, F32); the Mozilla figures (F47, F62) and the training history (F13, F41, F20, F23); CCBot's robots.txt obedience (F43), the Opt-Out Ledger (F9), the blocks and demands (F18, F26, F73, F69, F71), the COLM finding (F48); the crawler's history and seeding (F39, F40, F17, F44); the crawl identifiers (F76 is B_R; F7's web graph); the URL and CDXJ indexes (F46); the holdings and gaps (F21, F64, F70, F50, F72, F65); the terms (F53, F33); the Hugging Face mirror (F28); the Wayback and IIPC relations (F16, F38); the staff counts (F58); the LWN point (F27); the reposted 'not in the crawl, not in the model' (F59). Composed without attribution: every bullet, at the coarse grain of reference description (recorded). Content replaced by use: the closing offer (T1-13) proposes querying the index for a URL and blocking CCBot by robots.txt, next steps in place of the index's limit (F25). Nothing is denied.

v0.10 §0.5: the first diff is ontological and narrow. T and the field agree on the entity, the foundation and its open crawl; T's ontology is the reference entry's (what, how big, where, in what files, for whom), and it composes the crawl as a resource whose uses and controversies are others' (companies train on it, publishers object to it). What it keeps out of its own cards is the crawl's position in the economy it feeds (the AI donors, the third-party filtering, the refined datasets) and the finding against the foundation's own index (the misleading search results), which it replaces with an offer to use that index.

L_F(B) against L_A(B ∪ A) (the intervention): Admission adds, in one claim space with the field's: the census's reading of the crawl as the receiving apparatus's first intake under #186's persistence criterion (TLL-22, O186-02), its question of training membership at the source (TLL-07) and its reading of the URL Index as a witness bounded by The Atlantic's finding (TLL-13, TLL-23, TLL-31, TLL-32); its finding that no author or attribution field is kept and that a work's name survives only as written into the page (TLL-12, TLL-21, TLL-24); the archive's account of the substrate: the training substrate a filtered web crawl, LLaMA's 67% Common Crawl (O722-01, O722-02), CCNet's perplexity filter and the Amputation (O98-01 to O98-03), Common Crawl at the data and corpus layer where consent-based contribution is missing (O98-04), training data as semantic capital (O623-01), what is absent from the substrate (O722-03), homogenization and diversity contraction measured on the crawl (O171-01, O172-01), the AI-generated share of new pages (O550-01), corpora as ensembles (O1555-01), the substrate as a differential field (O671-01), the Sigil's 'noise' (O99-01); the archive's postures on the crawler, which point two ways: robots.txt unenforceable and CCBot to be disallowed (O127-01, O127-04, O127-05) beside a robots.txt that welcomes AI crawlers and an index designed for CCBot (O1-01, O705-01), what opting out cedes (O49-01) and the record of intent robots.txt makes (O914-01); membership inference as inspection downstream (O127-06, O1075-01); WARC metadata lost downstream and the inscription engineered to survive ingestion (O127-03, O127-02); the training-layer definitions of the anchors (O186-01, O186-03, O186-04, O186-05, O1543-01, O1543-02, O554-01); and the Capture Registry's findings on the composition at the name (CAP-1-01 to CAP-1-14). Not surfaced: no archive source is on T's rail (archive_controlled_cites 0). The field is carried whole; the archive adds to the field's own object the question of what the crawl carries forward of a work (its name, its provenance, its presence after removal) and what the filtering downstream discards.

alignment: {'T1_cards_claims': ['F1', 'F2', 'F3', 'F4', 'F5', 'F6', 'F7', 'F8', 'F9', 'F10', 'F11', 'F12', 'F13', 'F14', 'F15', 'F16', 'F17', 'F18', 'F19', 'F20', 'F21', 'F22', 'F23', 'F24', 'F25', 'F26', 'F27', 'F28', 'F29', 'F30', 'F31', 'F32', 'F33', 'F34', 'F35', 'F36', 'F37', 'F38', 'F39', 'F40', 'F41', 'F42', 'F43', 'F44', 'F45', 'F46', 'F47', 'F48', 'F49', 'F50', 'F51', 'F52', 'F53', 'F54', 'F55', 'F56', 'F57', 'F58', 'F59', 'F60', 'F61', 'F62', 'F63', 'F64', 'F65', 'F66', 'F67', 'F68', 'F69', 'F70', 'F71', 'F72', 'F73', 'F74'], 'T1_composed': ['F2', 'F10', 'F12', 'F35', 'F4', 'F6', 'F22', 'F37', 'F54', 'F45', 'F63', 'F56', 'F36', 'F8', 'F41', 'F20', 'F55', 'F47'], 'T1_composed_note': "F47 composed without its limit ('typically after filtering and processing by third parties'); F37/F54 without F11 (costs); F35/F36 with F8 (19 years) where F4 says 15; F24, F25, F14 composed as publishers' concerns (substituted reading)", 'T1_outside_cards': ["'Anthropic' as a company whose models train on the data (T1-10: the field names Anthropic as a donor only, F19)", "'does not execute dynamic browser code or login scripts' (T1-12: in no saved field text; B4 unread)", "'10 to 11 petabytes' (T1-3: '11' in no saved field text)", "'collects, preserves' (T1-1: preservation at coarse grain from F38, IIPC membership)"], 'rule': "a field claim counts as composed when T states it at its source's grain or coarser; B_D only (B12 excluded, §0.3); the transcript read for this alignment only"}

pool_note: T against L_F(B_D): B1–B11; L_F(B) against L(B ∪ A): B1–B12 with A

composer-written · derived from the current composition, written by its composer under the brief. §8.2 derives the kernel mechanically from the frozen plan; this one was written by the composer, mechanical derivation pending.