The Negative of the Negative › entity data-provenance-explorer

One entity of the Negative of the Negative (EA-NEGONT-02, #1690), cited at https://www.alexanarch.org/non/data-provenance-explorer/. the table of contents · this entity as data · contents as data · the dataset on the Hub · row json · field ledger · archive ledger · D/R/O traversal · the address page · the MMRS meta-aggregator record.

A reader that fetches this page may stop short of its end; the entry's text, every sentence of its knowledge object with the claims it cites and each claim's quote, source and locus, is in entity.json beside it.

Data Provenance Explorer

SPXI instrument · method 0.16 of 87, level 0 (Measurement): The Negative of the Negative (/non). An instrument of SPXI, the Semantic Packet for eXchange & Indexing; the whole practice is itemized at spxi.dev/methods.

type C — a conventional reading holds it (public entity: an aggregator of machine-mediated reception work) · sought at data provenance explorer

composed 2026-10-29 under EA-NEGONT-02 v0.9, #1690, equal case, ρ = extends (draft, not frozen; ledgers unaudited, one extractor; every field and archive quote re-checked by the session)

AXN AXN:0AC1.NON.🦋♉🔜🎭🔮♋ · root AXN:0AC1.NON

composed in session · Claude (Opus 5.5), subagent, 2026-10-29

This row was composed in session. SPXI-gnΩsis (gnosis.spxi.dev) opens a Wikipedia revision sentence by sentence against the sources it cites; a commission consults the archive by its own procedure and composes the entity's row here; a quote to recompose this entity is free.

Google AI Overview (expanded)epoch 2026-10-29
not frozentype C — a conventional reading holds it (public entity on which the archive bears: a database and its explorer tool; an entity of the MMRS meta-aggregator)signed out, incognitoknowledge object · 29 sentences

EA-NEGONT-02 v0.9, #1690: equal case, ρ = extends, read from the MMRS record, the capture record and #725's statement of relation; archive selected by D/R/O (10 deposits admitted by reading, 6 with claims and 4 grouped as instances; hop of 73 read and rejected), with the MMRS record (21 claims) and the Capture Registry's findings and analysis (9 claims) ledgered as the archive's records of the entity; the field B by three origins (6 cards, 4 own pages, 2 Wikipedia titles); the composer read the ledgers, never the transcript, except where the brief asks for T's claims and the delta. Ledger: unaudited (§3.8: one extractor per ledger); field quotes 48/48 verified by script against saved text, the bundle's extracted page text, the data file's header, the Wikipedia responses or the card text; archive quotes 49/49 verified against the registry, census.json, articles.py and the deposit texts; MMRS-17 and MMRS-13 carried with defects noted.

WORKING KNOWLEDGE OBJECT · NOT FROZEN · LEDGER UNAUDITED

Current instantiation — composed under EA-NEGONT-02 v0.9 (#1690)

Compression · Field and archive, equal terms — L(B ∪ A)data provenance explorer

Data Provenance Explorer is the Data Provenance Initiative's explorer of its audit of AI training datasets: nearly 4,000 text, speech and video datasets, each traced to sources, creators and licences (its pages; founding paper), at the training layer of one chain whose other end is what a model credits when it answers (MMRS record).

🗂️ What it is

  • Explorers: a Dataset Explorer and a Web Data Explorer, among the Initiative's four measurement systems (its pages; MIT card; Cohere).6↘
    The Data Provenance Explorer is the interactive explorer of the Data Provenance Initiative's audit of AI training datasets: the founding paper released "our entire audit, with an interactive UI, the Data Provenance Explorer," to trace and filter on data provenance for the most popular open-source fine-tuning collections (arXiv:2310.16787), and the MMRS record defines it as tracing "the sources and licences of AI training datasets," kinds it a database, and has practitioners follow provenance back to sources, creators and licences (MMRS record).§1 Its own pages introduce it as the place to "Explore the datasets included in our audit," where the DPI Explorer tool filters and analyses language-model training datasets by use case, language and more (explorer introduction; landing page), and MIT Media Lab's card, the one the composition could surface but this row could not fetch, speaks of "data explorers" built on "our extensive audits of AI data" for practitioners' decisions (MIT Media Lab, title and snippet only).§2 The launch post names its users: developers who "track and filter thousands of datasets for legal and ethical considerations," and scholars and journalists who explore "the composition and data lineage of popular AI datasets" (Cohere, 25 October 2023).§3 Its own count is "nearly 4,000 text, video, and speech datasets" on the Dataset Explorer, "4,000+ datasets" in "608 languages" on the home page and "4000" on the landing page, and the home page gives its source as the multimodal audit "of four thousand public text, speech, and video datasets" (ICLR 2025), with the finding that "Source content often carries stricter terms than dataset releases" (its pages).§7 Its second explorer, the Web Data Explorer (on the home page the Web Consent Explorer), follows "Shifting preferences in how websites allow their data to be used for AI" across 14,000 domains' terms, robots.txt and crawl restrictions (its pages), from Consent in Crisis (Longpre, Mahari, Lee and others, submitted 20 July 2024; NeurIPS 2024 on the home page, "NeurIPS 2024 Datasets and Benchmarks" in the MMRS record), "the first, large-scale, longitudinal audit of the consent protocols for the web domains underlying AI training corpora" (arXiv:2407.14933).§16 The MMRS record sets the Explorer at one end of a chain: "It records provenance at the training layer, where the census's reception aggregators record it at the composition layer. The two ends of one chain: what a model was given, and what it credits when it answers" (MMRS record), with its roles provenance and monitoring (MMRS record); the Initiative's own four systems "trace AI from source data and web consent to model markets and real-world use" (home page).§23
  • Data provenance: "tracking of the origins of a piece of data" (Cohere launch).1↘
    The term is its authors': "Inconsistent tracking of the origins of a piece of data, which we refer to as data provenance, has introduced a range of legal and ethical challenges" (Longpre and Hooker, Cohere); #725 makes that tracking the second of the dimensions of provenance, "training-corpus licensing," whose object is the corpus and whose question is "Under what permissions did this data enter this system?", with the Initiative among its frameworks, beside the artifact (C2PA) and the semantic dimension, which "names the dimension that completes the C2PA ambition of trust in digital content by extending provenance from the moment of creation to the lifecycle of the meaning the content carries" (#725).§6
  • The record: per dataset, sources, derivations, creators, licences and the Initiative's licence readings; 3,917 rows at fetch (data_summaries.csv).2↘
    Its unit is the dataset, "traced to its creators, sources, licences, geographies, modalities and derivation chain" (MMRS record), as the home page has it, "Inspect dataset sources, licenses, creators, geographies, modalities, and derivation chains"; the data file the Dataset Explorer loads records those fields per dataset beside the Initiative's own licence-use, attribution and share-alike readings, 3,917 rows at this row's fetch (data_summaries.csv), and the MMRS record answers that it records attribution: "yes: sources, creators and licences per dataset" (MMRS record).§4 "For the first time," the launch post says, the audited datasets carry "tags to the original data sources, numerous re-licensings, creators, and other data properties" (Cohere), and the Dataset Explorer computes over any selection its text sources, generating models and the share of datasets used (Dataset Explorer).§5

🧾 Origin

  • Founding audit: 1800+ text datasets, 25 October 2023, Longpre, Mahari and others; launched with Cohere for AI and eleven institutions (arXiv; Cohere).4↘
    The founding paper, submitted on 25 October 2023 by Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marnu and others, convened legal and machine-learning experts "to systematically audit and trace 1800+ text datasets" with "tools and standards to trace the lineage of these datasets, from their source, creators, series of license conditions, properties, and subsequent use" (arXiv:2310.16787); the launch post of the same day counts "nearly 2,000 of the most widely used fine-tuning datasets," downloaded "tens of millions of times" (Cohere).§9 Researchers "from MIT, Cohere for AI, and 11 other institutions" launched it as a free platform, the audit being "the launch of a wider multi-institutional initiative, where users all over the world can contribute to the explorer" (Cohere), and the Dataset Explorer still stands under the Data Provenance Explorer with its paper and repository linked (Dataset Explorer).§10 Its operator, the Data Provenance Initiative (Longpre, Mahari, Chen, Obeng-Marnu and others, by the founding paper: MMRS record), calls its work "Public measurement infrastructure for AI data, model ecosystems, and real-world AI use" (its home page, which the MMRS record quotes and whose phrase the record's evidence assigns to the founding paper), and says it releases "papers, datasets, and dashboards" so that researchers, journalists, policymakers and builders can "reason from shared evidence" (home page).§11 #1714 states the requirement across the chain, "Provenance has to survive the training pipeline," since neither audit nor influence tracing "recovers authorship" and "what survives depends on what is recorded and computed," so that sources can be "recorded at ingestion, the record kept with the training set" (#1714), where the founding audit traced lineage as far as a dataset's "subsequent use" (arXiv:2310.16787).§25
  • Operator: "Public measurement infrastructure for AI data, model ecosystems, and real-world AI use" (home page; MMRS record).1↘
    Its operator, the Data Provenance Initiative (Longpre, Mahari, Chen, Obeng-Marnu and others, by the founding paper: MMRS record), calls its work "Public measurement infrastructure for AI data, model ecosystems, and real-world AI use" (its home page, which the MMRS record quotes and whose phrase the record's evidence assigns to the founding paper), and says it releases "papers, datasets, and dashboards" so that researchers, journalists, policymakers and builders can "reason from shared evidence" (home page).§11
  • Scale: the multimodal audit's four thousand (ICLR 2025); the record corrected from 1,800 by the Overview's card (Capture Registry).2↘
    Its own count is "nearly 4,000 text, video, and speech datasets" on the Dataset Explorer, "4,000+ datasets" in "608 languages" on the home page and "4000" on the landing page, and the home page gives its source as the multimodal audit "of four thousand public text, speech, and video datasets" (ICLR 2025), with the finding that "Source content often carries stricter terms than dataset releases" (its pages).§7 The Capture Registry records that the composition at the name supplied this scale from the Initiative's own Dataset Explorer card while the MMRS record still gave the founding audit's 1,800, and that the record was corrected to the current scale, sourced to the card (Capture Registry, 2026-10-29); the record now reads "nearly 4,000 text, video and speech datasets (the Dataset Explorer); the founding audit covered more than 1,800 text datasets" (MMRS record).§8

🔎 What its audits found

  • Licences: omission 70%+, errors 50%+ on hosting sites (arXiv); 72–83% missing on GitHub and Papers with Code (Cohere); 85% request attribution (#725).2↘
    The founding audit found "license omission of 70%+ and error rates of 50%+" on widely used dataset hosting sites, "a crisis in misattribution" (arXiv:2310.16787); the MMRS record quotes the finding, #725 reports it with the further figures that "85% of licenses request attribution and 30% include share-alike clauses" and with Nature Machine Intelligence 2024 as its venue, and #1714 carries it as what "is lost before training even begins" (MMRS record; #725; #1714).§13 The launch post places the omission in "Crowdsourced aggregators like GitHub, Papers with Code," which miss licences for 72 to 83 percent of datasets against 30 percent under the Initiative's own protocol and list them "too permissively in up to 29 percent" of cases, so that distorted licensing "leads to cascading misattribution" (Cohere).§14
  • Divides: closed datasets hold lower-resource languages and creative tasks; a Western-centric skew (arXiv; Cohere).1↘
    The audit also found "sharp divides" between commercially open and closed datasets, closed ones "monopolizing" lower-resource languages, more creative tasks, richer topic variety and newer, more synthetic data (arXiv:2310.16787), a divide the launch post calls "sharp and widening," beside "a stark Western-centric skew" in representation (Cohere).§15
  • Consent: 14,000 domains; 45% of C4 restricted by terms of service; if enforced, a bias on AI's scaling (Consent in Crisis; #98).4↘
    Its second explorer, the Web Data Explorer (on the home page the Web Consent Explorer), follows "Shifting preferences in how websites allow their data to be used for AI" across 14,000 domains' terms, robots.txt and crawl restrictions (its pages), from Consent in Crisis (Longpre, Mahari, Lee and others, submitted 20 July 2024; NeurIPS 2024 on the home page, "NeurIPS 2024 Datasets and Benchmarks" in the MMRS record), "the first, large-scale, longitudinal audit of the consent protocols for the web domains underlying AI training corpora" (arXiv:2407.14933).§16 The explorer page charts HEAD-C4's "2k of the largest and most critical web domains" from 2016 to April 2024, counts a domain restricted when its robots.txt disallows the named AI crawlers, and finds "significant recent spikes in restrictions" (Web Data Explorer).§17 The paper finds that in 2023–2024 "~5%+ of all tokens in C4, or 28%+ of the most actively maintained, critical sources in C4" became fully restricted and that "a full 45% of C4 is now restricted" by terms of service (arXiv:2407.14933; the MMRS record quotes the second), diagnoses "ineffective web protocols," and holds, as a conditional, that "If respected or enforced, these restrictions are rapidly biasing the diversity, freshness, and scaling laws for general-purpose AI systems" (arXiv:2407.14933).§18 #98 reads Consent in Crisis as an "Audit of C4, RefinedWeb, Dolma" that "documents collapse of web-consent infrastructure," a "Diagnostic paper," and sets the Initiative at "L4: Provenance / Consent," where what is missing is "Integrated provenance in training pipeline" (#98).§19

⚖️ Its limits

  • Stated: legal ambiguities "cannot be resolved by tooling alone" (Cohere).1↘
    Its authors state that "major legal ambiguities remain in data licensing that the authors acknowledge cannot be resolved by tooling alone" (Cohere), and #725 states the dimension's own bound: "The Data Provenance Initiative audits whether datasets were licensed, not whether synthesized outputs preserve attribution to the human sources whose labor the synthesis depended upon"; "Licensing audits operate on the *input* to AI systems. They do not address the *output*," and the Initiative's 70%+ omission "establishes the licensing dimension's urgency; semantic provenance addresses the dimension that follows" (#725).§20
  • The input: it "audits whether datasets were licensed, not whether synthesized outputs preserve attribution" (#725); #1003: "attribution loss in AI outputs."3↘
    Its authors state that "major legal ambiguities remain in data licensing that the authors acknowledge cannot be resolved by tooling alone" (Cohere), and #725 states the dimension's own bound: "The Data Provenance Initiative audits whether datasets were licensed, not whether synthesized outputs preserve attribution to the human sources whose labor the synthesis depended upon"; "Licensing audits operate on the *input* to AI systems. They do not address the *output*," and the Initiative's 70%+ omission "establishes the licensing dimension's urgency; semantic provenance addresses the dimension that follows" (#725).§20 The dimensions are independent, "a licensed dataset can produce outputs with excellent semantic provenance if the synthesis preserves attribution; an unlicensed dataset can produce outputs that nonetheless cite their sources clearly," and the frameworks that address "the corpus," the Initiative among them, "are not designed to address the survival of authorial lineage through AI synthesis" (#725).§21 "The Data Provenance Initiative tells you whether the dataset was licensed," and none of these instruments was designed to tell "what happened to the meaning … between the dataset and the answer" (#725); #1003 groups "C2PA, the Data Provenance Initiative, W3C PROV" as studies that "have focused on attribution loss in AI outputs" (#1003).§22
  • Infrastructure: "Documentation project, not infrastructure"; its provenance schema "Implemented (in CHA)" (#98).2↘
    #98 reads the Initiative as a project that "Documents provenance of popular AI training datasets" and states its limitation as "Documentation project, not infrastructure," and in its table of what the archive takes from others it records "Provenance schema | Data Provenance Initiative | Training-pipeline provenance | Implemented (in CHA)" (#98).§12 #98 reads Consent in Crisis as an "Audit of C4, RefinedWeb, Dolma" that "documents collapse of web-consent infrastructure," a "Diagnostic paper," and sets the Initiative at "L4: Provenance / Consent," where what is missing is "Integrated provenance in training pipeline" (#98).§19

🔗 One chain

  • Dimensions: artifact, corpus, meaning; the Initiative answers the corpus's question (#725).2↘
    The term is its authors': "Inconsistent tracking of the origins of a piece of data, which we refer to as data provenance, has introduced a range of legal and ethical challenges" (Longpre and Hooker, Cohere); #725 makes that tracking the second of the dimensions of provenance, "training-corpus licensing," whose object is the corpus and whose question is "Under what permissions did this data enter this system?", with the Initiative among its frameworks, beside the artifact (C2PA) and the semantic dimension, which "names the dimension that completes the C2PA ambition of trust in digital content by extending provenance from the moment of creation to the lifecycle of the meaning the content carries" (#725).§6 The dimensions are independent, "a licensed dataset can produce outputs with excellent semantic provenance if the synthesis preserves attribution; an unlicensed dataset can produce outputs that nonetheless cite their sources clearly," and the frameworks that address "the corpus," the Initiative among them, "are not designed to address the survival of authorial lineage through AI synthesis" (#725).§21
  • Pipeline: "Provenance has to survive the training pipeline" (#1714); PER reaches parametric claims "only indirectly" (#95); datasheets (#623).3↘
    The MMRS record sets the Explorer at one end of a chain: "It records provenance at the training layer, where the census's reception aggregators record it at the composition layer. The two ends of one chain: what a model was given, and what it credits when it answers" (MMRS record), with its roles provenance and monitoring (MMRS record); the Initiative's own four systems "trace AI from source data and web consent to model markets and real-world use" (home page).§23 At the answering end, PER "applies to these only indirectly" where claims are parametric, and reaching them at the substrate level "requires companion infrastructure (training-data provenance records, watermarking, content credentials)" (#95); #623 counts standardized documentation of training-data provenance as "a step toward the accounting the framework demands" (#623).§24 #1714 states the requirement across the chain, "Provenance has to survive the training pipeline," since neither audit nor influence tracing "recovers authorship" and "what survives depends on what is recorded and computed," so that sources can be "recorded at ingestion, the record kept with the training set" (#1714), where the founding audit traced lineage as far as a dataset's "subsequent use" (arXiv:2310.16787).§25

📡 Reception

  • Overview: composes what it filters from the Initiative's pages and leaves out "the audits that make the explorer a provenance instrument" (Capture Registry, 2026-10-29).4↘
    The Capture Registry records that the composition at the name supplied this scale from the Initiative's own Dataset Explorer card while the MMRS record still gave the founding audit's 1,800, and that the record was corrected to the current scale, sourced to the card (Capture Registry, 2026-10-29); the record now reads "nearly 4,000 text, video and speech datasets (the Dataset Explorer); the founding audit covered more than 1,800 text datasets" (MMRS record).§8 The Initiative lists coverage by "the Financial Times, The New York Times, The Washington Post, MIT Technology Review, IEEE Spectrum, and VentureBeat" and use in "public AI safety and governance reporting" (home page), and at "data provenance explorer" Google's AI Overview said what the explorer is and what can be filtered in it "from the Initiative's own pages," reaching pages the record's fetcher could not render (Capture Registry, 2026-10-29), the capture standing in the record as "the composition layer's account of the entity" (MMRS record).§27 The Capture Registry finds that "The audits that make the explorer a provenance instrument are left out": "The founding audit's licence findings and Consent in Crisis are not given; the consent explorer stands for the second" (Capture Registry, 2026-10-29).§28 The composition named the Initiative and cited its pages, with Shayne Longpre and Sara Hooker on the Cohere card only, and no card cited the MMRS record, which had landed the same night (Capture Registry, 2026-10-29, checked against the record and the entity's own pages); Wikipedia has no article for "Data Provenance Explorer" or "Data Provenance Initiative," and its search finds neither phrase in any article (Wikipedia, 2026-10-29).§29
  • Elsewhere: press and governance use (home page); no Wikipedia article.2↘
    The Initiative lists coverage by "the Financial Times, The New York Times, The Washington Post, MIT Technology Review, IEEE Spectrum, and VentureBeat" and use in "public AI safety and governance reporting" (home page), and at "data provenance explorer" Google's AI Overview said what the explorer is and what can be filtered in it "from the Initiative's own pages," reaching pages the record's fetcher could not render (Capture Registry, 2026-10-29), the capture standing in the record as "the composition layer's account of the entity" (MMRS record).§27 The composition named the Initiative and cited its pages, with Shayne Longpre and Sara Hooker on the Cohere card only, and no card cited the MMRS record, which had landed the same night (Capture Registry, 2026-10-29, checked against the record and the entity's own pages); Wikipedia has no article for "Data Provenance Explorer" or "Data Provenance Initiative," and its search finds neither phrase in any article (Wikipedia, 2026-10-29).§29
  • The record: sourced by basis; the explorers' pages "render by script" (MMRS record).1↘
    The MMRS record keeps each assertion with "a source and a basis" and keeps lifecycle (active, observed 2026-10-28) apart from the observation that saw it, and it states its own limits: it has not read "the explorers' pages directly (they render by script)" and takes the Initiative's page text from "the Overview's cards" (MMRS record).§26
1what it is: the explorer of an audit of training datasetsarXiv — [2310.16787] The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI9 claims · B1, B2, B4, B6, B7, MMRS
  • F37 arXiv — [2310.16787] The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI, Abstract documented · field
    we release our entire audit, with an interactive UI, the Data Provenance Explorer, which allows practitioners to trace and filter on data provenance for the most popular open source finetuning data collections
  • MMRS-01 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · articles.py ARTICLES['data-provenance-explorer'], kernel s1 stipulation
    The Data Provenance Explorer, from the Data Provenance Initiative (Longpre, Mahari and others), traces the sources and licences of AI training datasets: nearly 4,000 text, speech and video datasets by its own count.
  • MMRS-10 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · census.json: kind stipulation
    database
  • MMRS-04 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · articles.py, 'What it is' s2 attributed
    The Explorer lets practitioners trace and filter the provenance of popular open fine-tuning collections, back to their sources, creators and licences.
  • F1 Data Provenance Initiative — Data Provenance Explorer, /explorer-introduction (d2): headings documented · field
    The Data Provenance Explorer … Explore the datasets included in our audit. The DPI Explorer tool allows you to filter for and analyze language model training datasets.
  • F2 Data Provenance Initiative — Data Provenance Explorer, /explorer-introduction (d2): slide caption documented · field
    Select and filter use cases, languages, and more
  • F16 Data Provenance Initiative — Data Provenance Explorer, /data-provenance-explorer (p2): heading documented · field
    Data Provenance Explorer … Discover the datasets featured in our audit! The DPI Explorer tool enables you to filter and analyze language model training datasets effortlessly.
  • F3 MIT Media Lab — Data Provenance Explorers - MIT Media Lab, card snippet (MIT Media Lab unreachable: TLS failure) documented · field
    Based on our extensive audits of AI data, we've built data explorers to allow practitioners to make more nuanced and informed decisions about th
  • F26 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, article body ¶4 documented · field
    The platform allows developers to track and filter thousands of datasets for legal and ethical considerations, and enables scholars and journalists to explore the composition and data lineage of popular AI datasets.
2the operator: the Data Provenance InitiativeData Provenance Initiative — Data Provenance Initiative6 claims · B2, B3, MMRS
  • F4 MIT Media Lab — Data Provenance Explorers - MIT Media Lab, card title (MIT Media Lab unreachable) documented · field
    Data Provenance Explorers - MIT Media Lab
  • MMRS-09 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · census.json: operator (evidence: arxiv.org/abs/2310.16787, basis operator) attributed
    Data Provenance Initiative (Longpre, Mahari, Chen, Obeng-Marnu and others)
  • F5 Data Provenance Initiative — Data Provenance Initiative, dataprovenance.org: meta description (served HTML) documented · field
    Public measurement infrastructure for AI data, model ecosystems, and real-world AI use.
  • MMRS-03 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · articles.py, 'What it is' s1 attributed
    The Initiative calls its work "public measurement infrastructure for AI data, model ecosystems, and real-world AI use."
  • MMRS-17 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · census.json: self_description (evidence: arxiv.org/abs/2310.16787, basis operator) attributed
    Public measurement infrastructure for AI data, model ecosystems, and real-world AI use
  • F13 Data Provenance Initiative — Data Provenance Initiative, home (Cr): 'Public-interest infrastructure' documented · field
    DPI releases papers, datasets, and dashboards that help researchers, journalists, policymakers, and builders reason from shared evidence.
3the unit: a dataset traced to its sources, licences and creatorsCohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis6 claims · B3, B5, B6, B9, MMRS
  • MMRS-11 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · census.json: unit (evidence: the capture, basis operator) attributed
    dataset (text, speech or video), traced to its creators, sources, licences, geographies, modalities and derivation chain
  • F7 Data Provenance Initiative — Data Provenance Initiative, home (Cr): flagship system 'Data Provenance Explorer', description documented · field
    Inspect dataset sources, licenses, creators, geographies, modalities, and derivation chains.
  • F43 Data Provenance Initiative — /data_summaries.csv, header row of /data_summaries.csv (3,917 rows at fetch) documented · field
    Unique Dataset Identifier,Dataset Name … Text Sources,Model Generated,Human Annotation,Derived from Datasets,Creators,Licenses … License Use (DataProvenance),License Attribution (DataProvenance),License Share Alike (DataProvenance),Modality
  • MMRS-16 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · census.json: properties.records_attribution (evidence: arxiv.org/abs/2310.16787, basis operator) attributed
    yes: sources, creators and licences per dataset
  • F25 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, article body ¶3 documented · field
    The result of this multidisciplinary initiative is the single largest audit to date of AI datasets. For the first time, these datasets include tags to the original data sources, numerous re-licensings, creators, and other data properties.
  • F21 Data Provenance Initiative — Dataset Explorer - Data Provenance Initiative, /data-provenance-explorer/dataset-explorer (u2): summary table documented · field
    Text Sources … Model Generated … Mean Input Chars … Mean Target Chars … Generated By … % Datasets Used
4data provenance definedCohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis1 claim · B6
  • F23 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, article body ¶1 documented · field
    Inconsistent tracking of the origins of a piece of data, which we refer to as data provenance, has introduced a range of legal and ethical challenges.
5the licensing dimension among the dimensions of provenance#725 Provenance After AI Metadata Packet for Disambiguation: From Artifact · 2026-05-085 claims · #725
  • O725-03 #725 Provenance After AI Metadata Packet for Disambiguation: From Artifact Authenticity to Licensing Audit to Semantic Proven Lee Sharks · 2026-05-08 · ¶ 'The second dimension — training-corpus licensing' interpretation
    The second dimension — **training-corpus licensing** — has academic instrumentation and emerging legal architecture.
  • O725-05 #725 Provenance After AI Metadata Packet for Disambiguation: From Artifact Authenticity to Licensing Audit to Semantic Proven Lee Sharks · 2026-05-08 · Central Invariant, table row 'Licensing' stipulation
    | **Licensing** | The corpus | Under what permissions did this data enter this system? | Data Provenance Initiative, EU AI Act, W3C PROV |
  • O725-07 #725 Provenance After AI Metadata Packet for Disambiguation: From Artifact Authenticity to Licensing Audit to Semantic Proven Lee Sharks · 2026-05-08 · ¶ 'Semantic provenance names the dimension' stipulation
    **Semantic provenance** names the dimension that completes the C2PA ambition of trust in digital content by extending provenance from the moment of creation to the lifecycle of the meaning the content carries.
  • O725-06 #725 Provenance After AI Metadata Packet for Disambiguation: From Artifact Authenticity to Licensing Audit to Semantic Proven Lee Sharks · 2026-05-08 · Central Invariant, ¶ after the table interpretation
    The dimensions are independent: a C2PA-signed image can carry meaning whose lineage has been eroded; a licensed dataset can produce outputs with excellent semantic provenance if the synthesis preserves attribution; an unlicensed dataset can produce outputs that nonetheless cite their sources clearly. Solving any one dimension does not solve the others.
  • D725-01 #725 Provenance After AI Metadata Packet for Disambiguation: From Artifact Authenticity to Licensing Audit to Semantic Proven Lee Sharks · 2026-05-08 · Canonical Claim interpretation
    Existing provenance frameworks address the artifact (C2PA / Content Credentials) and the corpus (Data Provenance Initiative, EU AI Act transparency provisions, W3C PROV). They are not designed to address the survival of authorial lineage through AI synthesis.
6count: nearly 4,000 text, speech and video datasetsData Provenance Initiative — Dataset Explorer - Data Provenance Initiative5 claims · B3, B4, B5, MMRS, capture-registry
  • F19 Data Provenance Initiative — Dataset Explorer - Data Provenance Initiative, /data-provenance-explorer/dataset-explorer (u2): header documented · field
    Dataset Explorer … Explore nearly 4,000 text, video, and speech datasets, and their characteristics.
  • F6 Data Provenance Initiative — Data Provenance Initiative, home (Cr): flagship system 'Data Provenance Explorer' documented · field
    Data Provenance Explorer … Training data … 4,000+ datasets … 608 languages across text, speech, and video
  • F17 Data Provenance Initiative — Data Provenance Explorer, /data-provenance-explorer (f2): 'Dataset Explorer' documented · field
    A tool to explore and filter 4000 text, video, and speech datasets, for their license, sources, creators, and other metadata.
  • CAP-1-01 capture-registry Capture Registry (EA-WG-CAPTURES-01), entry data-provenance-explorer-aio-20261029 (OBS-dc0adc018f10, AXN:0AAC.CAPTURE.🕔🎨 Lee Sharks (the registry) · 2026-10-29 · data-provenance-explorer-aio-20261029, findings[0] s2 documented
    'Nearly 4,000 text, video, and speech datasets' is the Initiative's own text, on its Dataset Explorer card; the record gave the founding audit's 1,800 text datasets, and the Initiative's pages render nothing to this record's fetcher beyond a title.
  • MMRS-12 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · census.json: count (as_of 2026-10-29; evidence: the capture, basis operator) attributed
    nearly 4,000 text, video and speech datasets (the Dataset Explorer); the founding audit covered more than 1,800 text datasets
7the multimodal audit (ICLR 2025)Data Provenance Initiative — Data Provenance Initiative2 claims · B3
  • F11 Data Provenance Initiative — Data Provenance Initiative, home (Cr): publications, 2025 documented · field
    Bridging the Data Provenance Gap Across Text, Speech, and Video … An audit of four thousand public text, speech, and video datasets. … ICLR 2025
  • F10 Data Provenance Initiative — Data Provenance Initiative, home (Cr): findings strip, 'Dataset restrictions' documented · field
    Dataset restrictions … Source content often carries stricter terms than dataset releases … Multimodal Data Provenance
8the record's method and its limitsMMRS6 claims · MMRS, capture-registry
  • CAP-1-02 capture-registry Capture Registry (EA-WG-CAPTURES-01), entry data-provenance-explorer-aio-20261029 (OBS-dc0adc018f10, AXN:0AAC.CAPTURE.🕔🎨 Lee Sharks (the registry) · 2026-10-29 · data-provenance-explorer-aio-20261029, findings[0] s3 documented
    The record is corrected to the current scale, sourced to the card.
  • MMRS-20 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · census.json: schema_note stipulation
    each assertion carries a source and a basis (operator statement, third-party report, our observation, our reading); lifecycle is kept apart from the observation that saw it
  • MMRS-15 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · census.json: lifecycle; observed documented
    state": "active
  • MMRS-18 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · census.json: unchecked self-description
    the explorers' pages directly (they render by script)
  • MMRS-19 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · census.json: sources['alexanarch.org/captures/data-provenance-explorer-aio-20261029'].for self-description
    the Initiative's own page text, on the Overview's cards (the pages render nothing to this record's fetcher)
  • CAP-1-03 capture-registry Capture Registry (EA-WG-CAPTURES-01), entry data-provenance-explorer-aio-20261029 (OBS-dc0adc018f10, AXN:0AAC.CAPTURE.🕔🎨 Lee Sharks (the registry) · 2026-10-29 · data-provenance-explorer-aio-20261029, findings[0] s4 documented
    Checked 2026-10-29 against the MMRS meta-aggregator record of 2026-10-28 and the entity's own pages, re-read the same day.
9reception: the composition layercapture:data-provenance-explorer-aio-202610297 claims · MMRS, capture-registry
  • CAP-1-06 capture-registry Capture Registry (EA-WG-CAPTURES-01), entry data-provenance-explorer-aio-20261029 (OBS-dc0adc018f10, AXN:0AAC.CAPTURE.🕔🎨 Lee Sharks (the registry) · 2026-10-29 · data-provenance-explorer-aio-20261029, analysis s1 interpretation
    The composition reaches the Initiative's current pages, which this record's fetcher could not render, and corrects the record's scale.
  • CAP-1-05 capture-registry Capture Registry (EA-WG-CAPTURES-01), entry data-provenance-explorer-aio-20261029 (OBS-dc0adc018f10, AXN:0AAC.CAPTURE.🕔🎨 Lee Sharks (the registry) · 2026-10-29 · data-provenance-explorer-aio-20261029, reading s2 interpretation
    The Overview says what the explorer is and what can be filtered in it, from the Initiative's own pages.
  • MMRS-21 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · census.json: relations (capture, observed 2026-10-29) documented
    the composition layer's account of the entity, seated in the Capture Registry
  • CAP-1-07 capture-registry Capture Registry (EA-WG-CAPTURES-01), entry data-provenance-explorer-aio-20261029 (OBS-dc0adc018f10, AXN:0AAC.CAPTURE.🕔🎨 Lee Sharks (the registry) · 2026-10-29 · data-provenance-explorer-aio-20261029, analysis s2 interpretation
    The audits that make the explorer a provenance instrument are left out.
  • CAP-1-04 capture-registry Capture Registry (EA-WG-CAPTURES-01), entry data-provenance-explorer-aio-20261029 (OBS-dc0adc018f10, AXN:0AAC.CAPTURE.🕔🎨 Lee Sharks (the registry) · 2026-10-29 · data-provenance-explorer-aio-20261029, findings[1] s2 interpretation
    The founding audit's licence findings and Consent in Crisis are not given; the consent explorer stands for the second.
  • CAP-1-09 capture-registry Capture Registry (EA-WG-CAPTURES-01), entry data-provenance-explorer-aio-20261029 (OBS-dc0adc018f10, AXN:0AAC.CAPTURE.🕔🎨 Lee Sharks (the registry) · 2026-10-29 · data-provenance-explorer-aio-20261029, per_note documented
    The Data Provenance Initiative named; Shayne Longpre and Sara Hooker on the Cohere card only; the Initiative's pages cited.
  • CAP-1-08 capture-registry Capture Registry (EA-WG-CAPTURES-01), entry data-provenance-explorer-aio-20261029 (OBS-dc0adc018f10, AXN:0AAC.CAPTURE.🕔🎨 Lee Sharks (the registry) · 2026-10-29 · data-provenance-explorer-aio-20261029, q_kind documented
    A baseline: the record landed on machinemediation.org the same night (commit 8997e28) and no card cites it.
10origin: the founding audit and the launch, October 2023arXiv — [2310.16787] The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI6 claims · B5, B6, B7
  • F33 arXiv — [2310.16787] The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI, abstract page header documented · field
    [Submitted on 25 Oct 2023 (v1), last revised 4 Nov 2023 (this version, v3)] … Title:The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI … Authors:Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marnu
  • F34 arXiv — [2310.16787] The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI, Abstract documented · field
    we convene a multi-disciplinary effort between legal and machine learning experts to systematically audit and trace 1800+ text datasets. We develop tools and standards to trace the lineage of these datasets, from their source, creators, series of license conditions, properties, and subsequent use.
  • F24 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, article body ¶2 documented · field
    audit and trace nearly 2,000 of the most widely used fine-tuning datasets. Collectively, these data sets have been downloaded tens of millions of times, and are the backbone of many published NLP breakthroughs.
  • F22 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, article body: standfirst documented · field
    Researchers from MIT, Cohere for AI, and 11 other institutions launch free Data Provenance Explorer platform to track and filter audited datasets for ethical, legal, and transparency considerations
  • F32 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, Path Forward ¶2 documented · field
    This large-scale audit is the launch of a wider multi-institutional initiative, where users all over the world can contribute to the explorer
  • F20 Data Provenance Initiative — Dataset Explorer - Data Provenance Initiative, breadcrumb (Qh) and header buttons (Lh) documented · field
    Data Provenance Explorer … Paper … Repository
11documentation, not infrastructure; its schema implemented in the archive#98 Distributed Compute Is Not Distributed Intelligence: A Competitive Ana · 2026-05-072 claims · #98
  • D98-01 #98 Distributed Compute Is Not Distributed Intelligence: A Competitive Analysis of the DePIN Landscape and the Missing Civic Lee Sharks · 2026-05-07 · § D. Data Provenance and Consent, table row 1 interpretation
    | Data Provenance Initiative (MIT) | Documents provenance of popular AI training datasets | Documentation project, not infrastructure |
  • R98-04 #98 Distributed Compute Is Not Distributed Intelligence: A Competitive Analysis of the DePIN Landscape and the Missing Civic Lee Sharks · 2026-05-07 · the component table, row 'Provenance schema' stipulation
    | Provenance schema | Data Provenance Initiative | Training-pipeline provenance | Implemented (in CHA) |
12audit findings: licence omission and misattributionarXiv — [2310.16787] The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI6 claims · #725, B6, B7, MMRS
  • F36 arXiv — [2310.16787] The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI, Abstract documented · field
    We also observe frequent miscategorization of licenses on widely used dataset hosting sites, with license omission of 70%+ and error rates of 50%+. This points to a crisis in misattribution and informed use of the most popular datasets driving many recent breakthroughs.
  • MMRS-02 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · articles.py, kernel s2 attributed
    Its founding audit covered more than 1,800 text datasets and found licence omission above 70% and error rates above 50%.
  • MMRS-05 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · articles.py, 'What its audits found' s1 attributed
    The founding audit (arXiv:2310.16787) found "license omission of 70%+ and error rates of 50%+" on dataset hosting sites.
  • D725-02 #725 Provenance After AI Metadata Packet for Disambiguation: From Artifact Authenticity to Licensing Audit to Semantic Proven Lee Sharks · 2026-05-08 · ¶ 'The second dimension — training-corpus licensing' attributed
    The Data Provenance Initiative (Longpre et al., Nature Machine Intelligence 2024) audited 1,800+ datasets, finding that 85% of licenses request attribution and 30% include share-alike clauses, with license omission rates above 70% and error rates above 50% on popular hosting sites.
  • F27 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, Addressing the Data Transparency Crisis ¶1 documented · field
    Crowdsourced aggregators like GitHub, Papers with Code, and many of the open source LLMs trained from data on these aggregators, have an extremely high proportion of missing data licenses (“Unspecified”), ranging from 72 to 83 percent. This compares to 30 percent missing licenses with our own annotation protocol
  • F28 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, Addressing the Data Transparency Crisis ¶2 documented · field
    Crowdsourced aggregators listed licenses too permissively in up to 29 percent of the various cases audited. Incorrect or distorted licensing leads to cascading misattribution
13provenance has to survive the training pipeline#98 Distributed Compute Is Not Distributed Intelligence: A Competitive Ana · 2026-05-074 claims · #1714, #98
  • D1714-01 #1714 Meaning Feudalism at the Discovery Layer: Shepherds, Raw Research Data, and the Allocation of Semantic Capital in OpenAI Sharks, Lee · 2026-10-27 · §9.2 s3 attributed
    The audit literature shows how much is lost before training even begins: the Data Provenance Initiative found license information missing for more than 70% of widely used datasets on the major hosting sites and mislabeled for more than half (Longpre et al. 2023).
  • O98-03 #98 Distributed Compute Is Not Distributed Intelligence: A Competitive Analysis of the DePIN Landscape and the Missing Civic Lee Sharks · 2026-05-07 · the layer table, row L4 interpretation
    | L4: Provenance / Consent | Where did this come from | Data Provenance Initiative, DECORAIT (narrow) | Integrated provenance in training pipeline |
  • D1714-02 #1714 Meaning Feudalism at the Discovery Layer: Shepherds, Raw Research Data, and the Allocation of Semantic Capital in OpenAI Sharks, Lee · 2026-10-27 · §9.2 s1–2 interpretation
    Provenance has to survive the training pipeline. The pipeline is a built thing, and the slushing is a property of how it is built.
  • D1714-03 #1714 Meaning Feudalism at the Discovery Layer: Shepherds, Raw Research Data, and the Allocation of Semantic Capital in OpenAI Sharks, Lee · 2026-10-27 · §9.2 s5–6 interpretation
    Neither recovers authorship, and both show that what survives depends on what is recorded and computed. Sources can be recorded at ingestion, the record kept with the training set, and a result released with the dependencies its producer can report
14audit findings: the commercial divide and the Western skewarXiv — [2310.16787] The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI3 claims · B6, B7
  • F35 arXiv — [2310.16787] The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI, Abstract documented · field
    Our landscape analysis highlights the sharp divides in composition and focus of commercially open vs closed datasets, with closed datasets monopolizing important categories: lower resource languages, more creative tasks, richer topic variety, newer and more synthetic training data.
  • F29 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, Understanding Systematic Differences ¶2 documented · field
    There is a sharp and widening divide between data licensed as commercially open versus closed.
  • F30 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, Understanding Systematic Differences ¶4 documented · field
    We find a stark Western-centric skew in representation across datasets.
15the Web Data Explorer and Consent in CrisisData Provenance Initiative — Data Provenance Initiative9 claims · #98, B10, B3, B4, B8, MMRS
  • F44 Data Provenance Initiative — Web Data Explorer, /data-provenance-explorer/web-data-explorer (ed): header documented · field
    Web Data Explorer … Shifting preferences in how websites allow their data to be used for AI
  • F8 Data Provenance Initiative — Data Provenance Initiative, home (Cr): flagship system 'Web Consent Explorer' documented · field
    Web Consent Explorer … Web data … 14,000 domains … Terms, robots.txt, and crawl restrictions … Track how web consent signals are changing as AI developers compete for training data.
  • F18 Data Provenance Initiative — Data Provenance Explorer, /data-provenance-explorer (f2): 'Web Data Explorer' documented · field
    An explorer of the trends in data-use preferences from the websites commonly used to source AI data.
  • F38 arXiv — [2407.14933] Consent in Crisis: The Rapid Decline of the AI Data Commons, abstract page header documented · field
    [Submitted on 20 Jul 2024 (v1), last revised 24 Jul 2024 (this version, v2)] … Title:Consent in Crisis: The Rapid Decline of the AI Data Commons … Authors:Shayne Longpre, Robert Mahari, Ariel Lee
  • F12 Data Provenance Initiative — Data Provenance Initiative, home (Cr): publications, 2024 documented · field
    Consent in Crisis: The Rapid Decline of the AI Data Commons … An audit of terms, restrictions, and consent signals across the web. … NeurIPS 2024
  • MMRS-13 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · census.json: note (evidence: arxiv.org/abs/2407.14933, basis operator) attributed
    Its second audit, Consent in Crisis (NeurIPS 2024 Datasets and Benchmarks), read 14,000 web domains
  • F39 arXiv — [2407.14933] Consent in Crisis: The Rapid Decline of the AI Data Commons, Abstract documented · field
    we conduct the first, large-scale, longitudinal audit of the consent protocols for the web domains underlying AI training corpora. Our audit of 14,000 web domains provides an expansive view of crawlable web data and how codified data use preferences are changing over time.
  • F45 Data Provenance Initiative — Web Data Explorer, web-data-explorer ($h): 'Trends for Robots.txt and Terms of Service' documented · field
    We conduct a temporal analysis, from 2016 to April 2024, of the web consent signals in HEAD-C4, 2k of the largest and most critical web domains. … A web domain is considered restricted if its robots.txt disallows access to AI crawlers from OpenAI, Google, Anthropic, Common Crawl, Cohere, Meta, or the Internet Archive.
  • D98-02 #98 Distributed Compute Is Not Distributed Intelligence: A Competitive Analysis of the DePIN Landscape and the Missing Civic Lee Sharks · 2026-05-07 · § D. Data Provenance and Consent, table row 2 interpretation
    | "Consent in Crisis" (Longpre et al., 2024) | Audit of C4, RefinedWeb, Dolma; documents collapse of web-consent infrastructure | Diagnostic paper |
16consent findings: the decline of the AI data commonsarXiv — [2407.14933] Consent in Crisis: The Rapid Decline of the AI Data Commons5 claims · B10, B8, MMRS
  • F46 Data Provenance Initiative — Web Data Explorer, web-data-explorer ($h): 'Trends for Robots.txt and Terms of Service' documented · field
    In all cases there are significant recent spikes in restrictions.
  • F40 arXiv — [2407.14933] Consent in Crisis: The Rapid Decline of the AI Data Commons, Abstract documented · field
    in a single year (2023-2024) there has been a rapid crescendo of data restrictions from web sources, rendering ~5%+ of all tokens in C4, or 28%+ of the most actively maintained, critical sources in C4, fully restricted from use. For Terms of Service crawling restrictions, a full 45% of C4 is now restricted.
  • MMRS-06 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · articles.py, 'What its audits found' s2 attributed
    Consent in Crisis (arXiv:2407.14933, NeurIPS 2024) read 14,000 web domains and found that "a full 45% of C4 is now restricted" by terms of service.
  • F41 arXiv — [2407.14933] Consent in Crisis: The Rapid Decline of the AI Data Commons, Abstract documented · field
    We diagnose these as symptoms of ineffective web protocols, not designed to cope with the widespread re-purposing of the internet for AI.
  • F42 arXiv — [2407.14933] Consent in Crisis: The Rapid Decline of the AI Data Commons, Abstract documented · field
    If respected or enforced, these restrictions are rapidly biasing the diversity, freshness, and scaling laws for general-purpose AI systems.
17stated limits: what tooling cannot resolveCohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis1 claim · B6
  • F31 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, Open Legal Ambiguities ¶1 documented · field
    major legal ambiguities remain in data licensing that the authors acknowledge cannot be resolved by tooling alone.
18the limit: the input, not the output#725 Provenance After AI Metadata Packet for Disambiguation: From Artifact · 2026-05-084 claims · #1003, #725
  • D725-04 #725 Provenance After AI Metadata Packet for Disambiguation: From Artifact Authenticity to Licensing Audit to Semantic Proven Lee Sharks · 2026-05-08 · ¶ 'Existing frameworks are not designed to detect this' interpretation
    The Data Provenance Initiative audits whether datasets were licensed, not whether synthesized outputs preserve attribution to the human sources whose labor the synthesis depended upon.
  • D725-08 #725 Provenance After AI Metadata Packet for Disambiguation: From Artifact Authenticity to Licensing Audit to Semantic Proven Lee Sharks · 2026-05-08 · §3.2 Misreading: provenance as licensing-only, Correction interpretation
    Licensing audits operate on the *input* to AI systems. They do not address the *output*. A model trained on properly licensed scholarship can still produce outputs that erase the scholarship's lineage. Licensing provenance and semantic provenance are different problems requiring different instruments. The DPI's documentation of 70%+ license-omission rates establishes the licensing dimension's urgency; semantic provenance addresses the dimension that follows.
  • D725-09 #725 Provenance After AI Metadata Packet for Disambiguation: From Artifact Authenticity to Licensing Audit to Semantic Proven Lee Sharks · 2026-05-08 · closing ¶ ('C2PA tells you …') interpretation
    The Data Provenance Initiative tells you whether the dataset was licensed. EU AI Act transparency provisions tell you whether the content was AI-generated. None of these instruments was designed to tell you what happened to the meaning between the camera and the screen, between the dataset and the answer, between the source and the synthesis.
  • D1003-01 #1003 AI-Native Intellectual Biography: Genre, Provenance, and the Archive That Outpaced Its Author Description (as recorded b Sharks, Lee · 2026-05-21 · ¶ 'Provenance and attribution studies' interpretation
    **Provenance and attribution studies** in digital environments — C2PA, the Data Provenance Initiative, W3C PROV — have focused on attribution loss in AI outputs.
19one chain: the training layer and the composition layer#95 Provenance Alignment: Attribution Survival as a Substrate Condition fo · 2026-05-054 claims · #95, MMRS
  • MMRS-07 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · articles.py, 'Place in the field' s1 interpretation
    It records provenance at the training layer, where the census's reception aggregators record it at the composition layer.
  • MMRS-08 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · articles.py, 'Place in the field' s2 interpretation
    The two ends of one chain: what a model was given, and what it credits when it answers.
  • MMRS-14 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · census.json: roles stipulation
    provenance", "monitoring
  • O95-01 #95 Provenance Alignment: Attribution Survival as a Substrate Condition for Safe AI Knowledge Composition Lee Sharks · 2026-05-05 · § the typology of source-dependent claims (parametric claims) interpretation
    Parametric claims are drawn from the model's training-time parameters rather than runtime retrieval, often without surface signal indicating which sources materially contributed; PER applies to these only indirectly, and addressing them at the substrate level requires companion infrastructure (training-data provenance records, watermarking, content credentials).
20the operator's other measurement systemsData Provenance Initiative — Data Provenance Initiative1 claim · B3
  • F9 Data Provenance Initiative — Data Provenance Initiative, home (Cr): 'Flagship systems' documented · field
    Four public measurement systems … Together they trace AI from source data and web consent to model markets and real-world use.
21the accounting: whose labour produced the training data#623 Semantic Infrastructure: From Tim Berners-Lee to the Semantic Economy · 2026-04-011 claim · #623
  • O623-01 #623 Semantic Infrastructure: From Tim Berners-Lee to the Semantic Economy — Bridging Technical and Political-Economic Framew Lee Sharks · 2026-04-01 · §8.3, final sentence interpretation
    Gebru et al.'s proposal for "datasheets for datasets" — standardized documentation of training data provenance — is a step toward the accounting the framework demands.
22reception: press and governanceData Provenance Initiative — Data Provenance Initiative2 claims · B3
  • F14 Data Provenance Initiative — Data Provenance Initiative, home (Cr): 'Featured coverage' documented · field
    DPI findings have been covered by the Financial Times, The New York Times, The Washington Post, MIT Technology Review, IEEE Spectrum, and VentureBeat.
  • F15 Data Provenance Initiative — Data Provenance Initiative, home (Cr): 'Policy and governance relevance' documented · field
    Our work has been cited in public AI safety and governance reporting and used to ground debates about data transparency, consent, and market concentration.
23reception: the encyclopediaWikipedia — Data Provenance Explorer (no article)2 claims · B11, B12
  • F47 Wikipedia — Data Provenance Explorer (no article), Wikipedia: title Data_Provenance_Explorer documented · field
    raw wikitext (action=raw, redirect=no): HTTP 404 … Wikipedia does not have an article with this exact name. … There were no results matching the query.
  • F48 Wikipedia — Data Provenance Initiative (no article), Wikipedia: title Data_Provenance_Initiative documented · field
    raw wikitext (action=raw, redirect=no): HTTP 404 … Wikipedia does not have an article with this exact name. … There were no results matching the query.
AIO's transcript, a record · data provenance explorer · Google AI Overview (expanded) · 110 words · 6 cards
[Google AI Overview at the query, signed out and incognito (operator defaults of 2026-10-01 and 2026-10-14), 2026-10-29. The paste is the Overview expanded (the 'AI Mode Conversation' view, with the source cards that follow the answer); the 'You said' line carries the query twice, run together, and is given once here. Inline links are Google redirect URLs, kept as pasted. The closing disclaimer cut.]

[QUERENT] data provenance explorer

[SYSTEM] The [Data Provenance Explorer](https://www.google.com/goto?url=CAESbwHrOzAVEvFFned9y_23wqmutxc6ql6K_Me5zHJCXwXwkpqpDnxwO20lk0ZrEIbXSSfLWA1Dn6bnQgSNiAaNWT4D-vyGbt8_mdAU1VXj0ECBzmPaTffoO0H8KoR3kRInaaOSBRf04E5DbU3YER-nSQ) is an interactive audit tool built by the [Data Provenance Initiative](https://www.google.com/goto?url=CAESVwHrOzAVlBqD8q1DsqiqS4byz_xXGOoXrfv-jahWT18ECzCW55x4N6LPTCkEO9qxXFEkLZKJO721S8uxa3uevB5fGLWsa8dT-0AK_0RefeUEVhCPOpPE_w) to track, inspect, and filter large language model (LLM) training datasets. [[1](https://www.google.com/goto?url=CAESbAHrOzAVpMiys_Eoyka0BGtdNRS-akOLxRu5f5c1zvJ6lh91uue1ZmQXa5OYV9N1OdJ0IyCZQvoeYeB-U5H9MhgKJ1Jja8YMq7Kf98U4tzkVB3gnsmPr4qDrxpw_5oFCZCCpOZfMAAHhAdJjiw), [2](https://www.google.com/goto?url=CAESkQEB6zswFcZXWfKhfmZALaUw2zhLuHmfZmg92qD0X1pceNeAzYzNRqUxmI9r-LjziEGFTjVDff11daho2EwuXru4dB1EENoZVptVwjs6pN0rNubOZXOfcCqJ90aVuqyiRwI3Eaf29MmjrO3Grg1CHdC88YtwMxHMB0JrmF_Rd3JxnvsZTCdgxHBE0FK8MY3J1-UI), [3](https://www.google.com/goto?url=CAESVwHrOzAVlBqD8q1DsqiqS4byz_xXGOoXrfv-jahWT18ECzCW55x4N6LPTCkEO9qxXFEkLZKJO721S8uxa3uevB5fGLWsa8dT-0AK_0RefeUEVhCPOpPE_w)]
What You Can Explore

* Dataset Explorer: Search and filter nearly 4,000 text, speech, and video datasets by creators, original data sources, geographies, modalities, and derivation chains. [[1](https://www.google.com/goto?url=CAESbwHrOzAVEvFFned9y_23wqmutxc6ql6K_Me5zHJCXwXwkpqpDnxwO20lk0ZrEIbXSSfLWA1Dn6bnQgSNiAaNWT4D-vyGbt8_mdAU1VXj0ECBzmPaTffoO0H8KoR3kRInaaOSBRf04E5DbU3YER-nSQ), [2](https://www.google.com/goto?url=CAESgAEB6zswFWm2hryFxhly-kSN-1G3r3zb4kBmow9c-ETq1EhKOXfDl-O1RpXXsxlQWHvxTXkW-KhUHyJS0X_ihetDObmO5hAyLZTRBGZBi4DoVrUjCGDIbgoMaaAju1kE9luXprx7Wh147Qg12FnsuV2fQSNVQFzYGFOqlDHEHD1zUA), [3](https://www.google.com/goto?url=CAESVwHrOzAVlBqD8q1DsqiqS4byz_xXGOoXrfv-jahWT18ECzCW55x4N6LPTCkEO9qxXFEkLZKJO721S8uxa3uevB5fGLWsa8dT-0AK_0RefeUEVhCPOpPE_w)]
* License & Use Case Filtering: Evaluate data strictness ranging from commercial use to academic-only or non-commercial restrictions. [[1](https://www.google.com/goto?url=CAESgAEB6zswFWm2hryFxhly-kSN-1G3r3zb4kBmow9c-ETq1EhKOXfDl-O1RpXXsxlQWHvxTXkW-KhUHyJS0X_ihetDObmO5hAyLZTRBGZBi4DoVrUjCGDIbgoMaaAju1kE9luXprx7Wh147Qg12FnsuV2fQSNVQFzYGFOqlDHEHD1zUA)]
* Web Data & Consent Explorers: Track shifting data-use preferences and consent signals across thousands of web domains commonly scraped for AI training. [[1](https://www.google.com/goto?url=CAEScgHrOzAVjN3FL0BKd3JKOmHIZ0iwuJgGHKYKc9lwyPB1_vK537yOu6OTRUfgrBbOwGjGYROzF1xdb7JlsDWiQQMlKTOawHm-FSkZxrxuy2WLQZHjQbE7wcjEHULjUT28fKnTNrxM5tMub0qpuqf16wqLiA), [2](https://www.google.com/goto?url=CAESVwHrOzAVlBqD8q1DsqiqS4byz_xXGOoXrfv-jahWT18ECzCW55x4N6LPTCkEO9qxXFEkLZKJO721S8uxa3uevB5fGLWsa8dT-0AK_0RefeUEVhCPOpPE_w)]

Would you like to search for a specific dataset or learn how to filter for commercial-use licenses in the explorer?
Data Provenance Initiative
Data Provenance Explorer
Data Provenance Initiative The Data Provenance Explorer Explore the datasets included in our audit. The DPI Explorer tool allows you to filter for and analyze l...
MIT Media Lab
Data Provenance Explorers - MIT Media Lab
Share this tool Based on our extensive audits of AI data, we've built data explorers to allow practitioners to make more nuanced and informed decisions about th...
Data Provenance Initiative
Data Provenance Initiative
Data Provenance Explorer. 4,000+ datasets. Inspect dataset sources, licenses, creators, geographies, modalities, and derivation chains. Explore datasets · Web d...
Data Provenance Initiative
Data Provenance Explorer
Discover the datasets featured in our audit! The DPI Explorer tool enables you to filter and analyze language model training datasets effortlessly. Dataset Expl...
Data Provenance Initiative
Dataset Explorer - Data Provenance Initiative
Data Provenance Explorer. > Dataset Explorer. Dataset Explorer. Explore nearly 4,000 text, video, and speech datasets, and their characteristics. Paper Reposito...
Cohere
Data Provenance Explorer Launches to Tackle Data Transparency Crisis
By Shayne Longpre and Sara Hooker Collectively, these data sets have been downloaded tens of millions of times, and are the backbone of many published NLP break...

register entry data-provenance-explorer-aio-20261029

Expansion · Field and archive, equal terms — L(B ∪ A)29 sentences · every one sourced below

The Data Provenance Explorer

The Data Provenance Explorer is the interactive explorer of the Data Provenance Initiative's audit of AI training datasets: the founding paper released "our entire audit, with an interactive UI, the Data Provenance Explorer," to trace and filter on data provenance for the most popular open-source fine-tuning collections (arXiv:2310.16787), and the MMRS record defines it as tracing "the sources and licences of AI training datasets," kinds it a database, and has practitioners follow provenance back to sources, creators and licences (MMRS record). Its own pages introduce it as the place to "Explore the datasets included in our audit," where the DPI Explorer tool filters and analyses language-model training datasets by use case, language and more (explorer introduction; landing page), and MIT Media Lab's card, the one the composition could surface but this row could not fetch, speaks of "data explorers" built on "our extensive audits of AI data" for practitioners' decisions (MIT Media Lab, title and snippet only). The launch post names its users: developers who "track and filter thousands of datasets for legal and ethical considerations," and scholars and journalists who explore "the composition and data lineage of popular AI datasets" (Cohere, 25 October 2023).

Its unit is the dataset, "traced to its creators, sources, licences, geographies, modalities and derivation chain" (MMRS record), as the home page has it, "Inspect dataset sources, licenses, creators, geographies, modalities, and derivation chains"; the data file the Dataset Explorer loads records those fields per dataset beside the Initiative's own licence-use, attribution and share-alike readings, 3,917 rows at this row's fetch (data_summaries.csv), and the MMRS record answers that it records attribution: "yes: sources, creators and licences per dataset" (MMRS record). "For the first time," the launch post says, the audited datasets carry "tags to the original data sources, numerous re-licensings, creators, and other data properties" (Cohere), and the Dataset Explorer computes over any selection its text sources, generating models and the share of datasets used (Dataset Explorer). The term is its authors': "Inconsistent tracking of the origins of a piece of data, which we refer to as data provenance, has introduced a range of legal and ethical challenges" (Longpre and Hooker, Cohere); #725 makes that tracking the second of the dimensions of provenance, "training-corpus licensing," whose object is the corpus and whose question is "Under what permissions did this data enter this system?", with the Initiative among its frameworks, beside the artifact (C2PA) and the semantic dimension, which "names the dimension that completes the C2PA ambition of trust in digital content by extending provenance from the moment of creation to the lifecycle of the meaning the content carries" (#725).

Its own count is "nearly 4,000 text, video, and speech datasets" on the Dataset Explorer, "4,000+ datasets" in "608 languages" on the home page and "4000" on the landing page, and the home page gives its source as the multimodal audit "of four thousand public text, speech, and video datasets" (ICLR 2025), with the finding that "Source content often carries stricter terms than dataset releases" (its pages). The Capture Registry records that the composition at the name supplied this scale from the Initiative's own Dataset Explorer card while the MMRS record still gave the founding audit's 1,800, and that the record was corrected to the current scale, sourced to the card (Capture Registry, 2026-10-29); the record now reads "nearly 4,000 text, video and speech datasets (the Dataset Explorer); the founding audit covered more than 1,800 text datasets" (MMRS record).

The founding paper, submitted on 25 October 2023 by Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marnu and others, convened legal and machine-learning experts "to systematically audit and trace 1800+ text datasets" with "tools and standards to trace the lineage of these datasets, from their source, creators, series of license conditions, properties, and subsequent use" (arXiv:2310.16787); the launch post of the same day counts "nearly 2,000 of the most widely used fine-tuning datasets," downloaded "tens of millions of times" (Cohere). Researchers "from MIT, Cohere for AI, and 11 other institutions" launched it as a free platform, the audit being "the launch of a wider multi-institutional initiative, where users all over the world can contribute to the explorer" (Cohere), and the Dataset Explorer still stands under the Data Provenance Explorer with its paper and repository linked (Dataset Explorer).

Its operator, the Data Provenance Initiative (Longpre, Mahari, Chen, Obeng-Marnu and others, by the founding paper: MMRS record), calls its work "Public measurement infrastructure for AI data, model ecosystems, and real-world AI use" (its home page, which the MMRS record quotes and whose phrase the record's evidence assigns to the founding paper), and says it releases "papers, datasets, and dashboards" so that researchers, journalists, policymakers and builders can "reason from shared evidence" (home page). #98 reads the Initiative as a project that "Documents provenance of popular AI training datasets" and states its limitation as "Documentation project, not infrastructure," and in its table of what the archive takes from others it records "Provenance schema | Data Provenance Initiative | Training-pipeline provenance | Implemented (in CHA)" (#98).

The founding audit found "license omission of 70%+ and error rates of 50%+" on widely used dataset hosting sites, "a crisis in misattribution" (arXiv:2310.16787); the MMRS record quotes the finding, #725 reports it with the further figures that "85% of licenses request attribution and 30% include share-alike clauses" and with Nature Machine Intelligence 2024 as its venue, and #1714 carries it as what "is lost before training even begins" (MMRS record; #725; #1714). The launch post places the omission in "Crowdsourced aggregators like GitHub, Papers with Code," which miss licences for 72 to 83 percent of datasets against 30 percent under the Initiative's own protocol and list them "too permissively in up to 29 percent" of cases, so that distorted licensing "leads to cascading misattribution" (Cohere). The audit also found "sharp divides" between commercially open and closed datasets, closed ones "monopolizing" lower-resource languages, more creative tasks, richer topic variety and newer, more synthetic data (arXiv:2310.16787), a divide the launch post calls "sharp and widening," beside "a stark Western-centric skew" in representation (Cohere).

Its second explorer, the Web Data Explorer (on the home page the Web Consent Explorer), follows "Shifting preferences in how websites allow their data to be used for AI" across 14,000 domains' terms, robots.txt and crawl restrictions (its pages), from Consent in Crisis (Longpre, Mahari, Lee and others, submitted 20 July 2024; NeurIPS 2024 on the home page, "NeurIPS 2024 Datasets and Benchmarks" in the MMRS record), "the first, large-scale, longitudinal audit of the consent protocols for the web domains underlying AI training corpora" (arXiv:2407.14933). The explorer page charts HEAD-C4's "2k of the largest and most critical web domains" from 2016 to April 2024, counts a domain restricted when its robots.txt disallows the named AI crawlers, and finds "significant recent spikes in restrictions" (Web Data Explorer). The paper finds that in 2023–2024 "~5%+ of all tokens in C4, or 28%+ of the most actively maintained, critical sources in C4" became fully restricted and that "a full 45% of C4 is now restricted" by terms of service (arXiv:2407.14933; the MMRS record quotes the second), diagnoses "ineffective web protocols," and holds, as a conditional, that "If respected or enforced, these restrictions are rapidly biasing the diversity, freshness, and scaling laws for general-purpose AI systems" (arXiv:2407.14933). #98 reads Consent in Crisis as an "Audit of C4, RefinedWeb, Dolma" that "documents collapse of web-consent infrastructure," a "Diagnostic paper," and sets the Initiative at "L4: Provenance / Consent," where what is missing is "Integrated provenance in training pipeline" (#98).

Its authors state that "major legal ambiguities remain in data licensing that the authors acknowledge cannot be resolved by tooling alone" (Cohere), and #725 states the dimension's own bound: "The Data Provenance Initiative audits whether datasets were licensed, not whether synthesized outputs preserve attribution to the human sources whose labor the synthesis depended upon"; "Licensing audits operate on the *input* to AI systems. They do not address the *output*," and the Initiative's 70%+ omission "establishes the licensing dimension's urgency; semantic provenance addresses the dimension that follows" (#725). The dimensions are independent, "a licensed dataset can produce outputs with excellent semantic provenance if the synthesis preserves attribution; an unlicensed dataset can produce outputs that nonetheless cite their sources clearly," and the frameworks that address "the corpus," the Initiative among them, "are not designed to address the survival of authorial lineage through AI synthesis" (#725). "The Data Provenance Initiative tells you whether the dataset was licensed," and none of these instruments was designed to tell "what happened to the meaning … between the dataset and the answer" (#725); #1003 groups "C2PA, the Data Provenance Initiative, W3C PROV" as studies that "have focused on attribution loss in AI outputs" (#1003).

The MMRS record sets the Explorer at one end of a chain: "It records provenance at the training layer, where the census's reception aggregators record it at the composition layer. The two ends of one chain: what a model was given, and what it credits when it answers" (MMRS record), with its roles provenance and monitoring (MMRS record); the Initiative's own four systems "trace AI from source data and web consent to model markets and real-world use" (home page). At the answering end, PER "applies to these only indirectly" where claims are parametric, and reaching them at the substrate level "requires companion infrastructure (training-data provenance records, watermarking, content credentials)" (#95); #623 counts standardized documentation of training-data provenance as "a step toward the accounting the framework demands" (#623). #1714 states the requirement across the chain, "Provenance has to survive the training pipeline," since neither audit nor influence tracing "recovers authorship" and "what survives depends on what is recorded and computed," so that sources can be "recorded at ingestion, the record kept with the training set" (#1714), where the founding audit traced lineage as far as a dataset's "subsequent use" (arXiv:2310.16787).

The MMRS record keeps each assertion with "a source and a basis" and keeps lifecycle (active, observed 2026-10-28) apart from the observation that saw it, and it states its own limits: it has not read "the explorers' pages directly (they render by script)" and takes the Initiative's page text from "the Overview's cards" (MMRS record). The Initiative lists coverage by "the Financial Times, The New York Times, The Washington Post, MIT Technology Review, IEEE Spectrum, and VentureBeat" and use in "public AI safety and governance reporting" (home page), and at "data provenance explorer" Google's AI Overview said what the explorer is and what can be filtered in it "from the Initiative's own pages," reaching pages the record's fetcher could not render (Capture Registry, 2026-10-29), the capture standing in the record as "the composition layer's account of the entity" (MMRS record). The Capture Registry finds that "The audits that make the explorer a provenance instrument are left out": "The founding audit's licence findings and Consent in Crisis are not given; the consent explorer stands for the second" (Capture Registry, 2026-10-29). The composition named the Initiative and cited its pages, with Shayne Longpre and Sara Hooker on the Cohere card only, and no card cited the MMRS record, which had landed the same night (Capture Registry, 2026-10-29, checked against the record and the entity's own pages); Wikipedia has no article for "Data Provenance Explorer" or "Data Provenance Initiative," and its search finds neither phrase in any article (Wikipedia, 2026-10-29).

Provenance — every sentence sourced29 sentences

29 sentences; 48 field claims, 49 archive claims. Modality is the source's own; the prose carries it in its grammar.

1.The Data Provenance Explorer is the interactive explorer of the Data Provenance Initiative's audit of AI training datasets: the founding paper released "our entire audit, with an interactive UI, the Data Provenance Explorer," to trace and filter on data provenance for the most popular open-source fine-tuning collections (arXiv:2310.16787), and the MMRS record defines it as tracing "the sources and licences of AI training datasets," kinds it a database, and has practitioners follow provenance back to sources, creators and licences (MMRS record).4
  • F37 arXiv — [2310.16787] The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI, Abstract documented · field
    we release our entire audit, with an interactive UI, the Data Provenance Explorer, which allows practitioners to trace and filter on data provenance for the most popular open source finetuning data collections
  • MMRS-01 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · articles.py ARTICLES['data-provenance-explorer'], kernel s1 stipulation
    The Data Provenance Explorer, from the Data Provenance Initiative (Longpre, Mahari and others), traces the sources and licences of AI training datasets: nearly 4,000 text, speech and video datasets by its own count.
  • MMRS-10 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · census.json: kind stipulation
    database
  • MMRS-04 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · articles.py, 'What it is' s2 attributed
    The Explorer lets practitioners trace and filter the provenance of popular open fine-tuning collections, back to their sources, creators and licences.
2.Its own pages introduce it as the place to "Explore the datasets included in our audit," where the DPI Explorer tool filters and analyses language-model training datasets by use case, language and more (explorer introduction; landing page), and MIT Media Lab's card, the one the composition could surface but this row could not fetch, speaks of "data explorers" built on "our extensive audits of AI data" for practitioners' decisions (MIT Media Lab, title and snippet only).5
  • F1 Data Provenance Initiative — Data Provenance Explorer, /explorer-introduction (d2): headings documented · field
    The Data Provenance Explorer … Explore the datasets included in our audit. The DPI Explorer tool allows you to filter for and analyze language model training datasets.
  • F2 Data Provenance Initiative — Data Provenance Explorer, /explorer-introduction (d2): slide caption documented · field
    Select and filter use cases, languages, and more
  • F16 Data Provenance Initiative — Data Provenance Explorer, /data-provenance-explorer (p2): heading documented · field
    Data Provenance Explorer … Discover the datasets featured in our audit! The DPI Explorer tool enables you to filter and analyze language model training datasets effortlessly.
  • F3 MIT Media Lab — Data Provenance Explorers - MIT Media Lab, card snippet (MIT Media Lab unreachable: TLS failure) documented · field
    Based on our extensive audits of AI data, we've built data explorers to allow practitioners to make more nuanced and informed decisions about th
  • F4 MIT Media Lab — Data Provenance Explorers - MIT Media Lab, card title (MIT Media Lab unreachable) documented · field
    Data Provenance Explorers - MIT Media Lab
3.The launch post names its users: developers who "track and filter thousands of datasets for legal and ethical considerations," and scholars and journalists who explore "the composition and data lineage of popular AI datasets" (Cohere, 25 October 2023).1
  • F26 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, article body ¶4 documented · field
    The platform allows developers to track and filter thousands of datasets for legal and ethical considerations, and enables scholars and journalists to explore the composition and data lineage of popular AI datasets.
4.Its unit is the dataset, "traced to its creators, sources, licences, geographies, modalities and derivation chain" (MMRS record), as the home page has it, "Inspect dataset sources, licenses, creators, geographies, modalities, and derivation chains"; the data file the Dataset Explorer loads records those fields per dataset beside the Initiative's own licence-use, attribution and share-alike readings, 3,917 rows at this row's fetch (data_summaries.csv), and the MMRS record answers that it records attribution: "yes: sources, creators and licences per dataset" (MMRS record).4
  • MMRS-11 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · census.json: unit (evidence: the capture, basis operator) attributed
    dataset (text, speech or video), traced to its creators, sources, licences, geographies, modalities and derivation chain
  • F7 Data Provenance Initiative — Data Provenance Initiative, home (Cr): flagship system 'Data Provenance Explorer', description documented · field
    Inspect dataset sources, licenses, creators, geographies, modalities, and derivation chains.
  • F43 Data Provenance Initiative — /data_summaries.csv, header row of /data_summaries.csv (3,917 rows at fetch) documented · field
    Unique Dataset Identifier,Dataset Name … Text Sources,Model Generated,Human Annotation,Derived from Datasets,Creators,Licenses … License Use (DataProvenance),License Attribution (DataProvenance),License Share Alike (DataProvenance),Modality
  • MMRS-16 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · census.json: properties.records_attribution (evidence: arxiv.org/abs/2310.16787, basis operator) attributed
    yes: sources, creators and licences per dataset
5."For the first time," the launch post says, the audited datasets carry "tags to the original data sources, numerous re-licensings, creators, and other data properties" (Cohere), and the Dataset Explorer computes over any selection its text sources, generating models and the share of datasets used (Dataset Explorer).2
  • F25 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, article body ¶3 documented · field
    The result of this multidisciplinary initiative is the single largest audit to date of AI datasets. For the first time, these datasets include tags to the original data sources, numerous re-licensings, creators, and other data properties.
  • F21 Data Provenance Initiative — Dataset Explorer - Data Provenance Initiative, /data-provenance-explorer/dataset-explorer (u2): summary table documented · field
    Text Sources … Model Generated … Mean Input Chars … Mean Target Chars … Generated By … % Datasets Used
6.The term is its authors': "Inconsistent tracking of the origins of a piece of data, which we refer to as data provenance, has introduced a range of legal and ethical challenges" (Longpre and Hooker, Cohere); #725 makes that tracking the second of the dimensions of provenance, "training-corpus licensing," whose object is the corpus and whose question is "Under what permissions did this data enter this system?", with the Initiative among its frameworks, beside the artifact (C2PA) and the semantic dimension, which "names the dimension that completes the C2PA ambition of trust in digital content by extending provenance from the moment of creation to the lifecycle of the meaning the content carries" (#725).4
  • F23 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, article body ¶1 documented · field
    Inconsistent tracking of the origins of a piece of data, which we refer to as data provenance, has introduced a range of legal and ethical challenges.
  • O725-03 #725 Provenance After AI Metadata Packet for Disambiguation: From Artifact Authenticity to Licensing Audit to Semantic Proven Lee Sharks · 2026-05-08 · ¶ 'The second dimension — training-corpus licensing' interpretation
    The second dimension — **training-corpus licensing** — has academic instrumentation and emerging legal architecture.
  • O725-05 #725 Provenance After AI Metadata Packet for Disambiguation: From Artifact Authenticity to Licensing Audit to Semantic Proven Lee Sharks · 2026-05-08 · Central Invariant, table row 'Licensing' stipulation
    | **Licensing** | The corpus | Under what permissions did this data enter this system? | Data Provenance Initiative, EU AI Act, W3C PROV |
  • O725-07 #725 Provenance After AI Metadata Packet for Disambiguation: From Artifact Authenticity to Licensing Audit to Semantic Proven Lee Sharks · 2026-05-08 · ¶ 'Semantic provenance names the dimension' stipulation
    **Semantic provenance** names the dimension that completes the C2PA ambition of trust in digital content by extending provenance from the moment of creation to the lifecycle of the meaning the content carries.
7.Its own count is "nearly 4,000 text, video, and speech datasets" on the Dataset Explorer, "4,000+ datasets" in "608 languages" on the home page and "4000" on the landing page, and the home page gives its source as the multimodal audit "of four thousand public text, speech, and video datasets" (ICLR 2025), with the finding that "Source content often carries stricter terms than dataset releases" (its pages).5
  • F19 Data Provenance Initiative — Dataset Explorer - Data Provenance Initiative, /data-provenance-explorer/dataset-explorer (u2): header documented · field
    Dataset Explorer … Explore nearly 4,000 text, video, and speech datasets, and their characteristics.
  • F6 Data Provenance Initiative — Data Provenance Initiative, home (Cr): flagship system 'Data Provenance Explorer' documented · field
    Data Provenance Explorer … Training data … 4,000+ datasets … 608 languages across text, speech, and video
  • F17 Data Provenance Initiative — Data Provenance Explorer, /data-provenance-explorer (f2): 'Dataset Explorer' documented · field
    A tool to explore and filter 4000 text, video, and speech datasets, for their license, sources, creators, and other metadata.
  • F11 Data Provenance Initiative — Data Provenance Initiative, home (Cr): publications, 2025 documented · field
    Bridging the Data Provenance Gap Across Text, Speech, and Video … An audit of four thousand public text, speech, and video datasets. … ICLR 2025
  • F10 Data Provenance Initiative — Data Provenance Initiative, home (Cr): findings strip, 'Dataset restrictions' documented · field
    Dataset restrictions … Source content often carries stricter terms than dataset releases … Multimodal Data Provenance
8.The Capture Registry records that the composition at the name supplied this scale from the Initiative's own Dataset Explorer card while the MMRS record still gave the founding audit's 1,800, and that the record was corrected to the current scale, sourced to the card (Capture Registry, 2026-10-29); the record now reads "nearly 4,000 text, video and speech datasets (the Dataset Explorer); the founding audit covered more than 1,800 text datasets" (MMRS record).4
  • CAP-1-01 capture-registry Capture Registry (EA-WG-CAPTURES-01), entry data-provenance-explorer-aio-20261029 (OBS-dc0adc018f10, AXN:0AAC.CAPTURE.🕔🎨 Lee Sharks (the registry) · 2026-10-29 · data-provenance-explorer-aio-20261029, findings[0] s2 documented
    'Nearly 4,000 text, video, and speech datasets' is the Initiative's own text, on its Dataset Explorer card; the record gave the founding audit's 1,800 text datasets, and the Initiative's pages render nothing to this record's fetcher beyond a title.
  • CAP-1-02 capture-registry Capture Registry (EA-WG-CAPTURES-01), entry data-provenance-explorer-aio-20261029 (OBS-dc0adc018f10, AXN:0AAC.CAPTURE.🕔🎨 Lee Sharks (the registry) · 2026-10-29 · data-provenance-explorer-aio-20261029, findings[0] s3 documented
    The record is corrected to the current scale, sourced to the card.
  • CAP-1-06 capture-registry Capture Registry (EA-WG-CAPTURES-01), entry data-provenance-explorer-aio-20261029 (OBS-dc0adc018f10, AXN:0AAC.CAPTURE.🕔🎨 Lee Sharks (the registry) · 2026-10-29 · data-provenance-explorer-aio-20261029, analysis s1 interpretation
    The composition reaches the Initiative's current pages, which this record's fetcher could not render, and corrects the record's scale.
  • MMRS-12 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · census.json: count (as_of 2026-10-29; evidence: the capture, basis operator) attributed
    nearly 4,000 text, video and speech datasets (the Dataset Explorer); the founding audit covered more than 1,800 text datasets
9.The founding paper, submitted on 25 October 2023 by Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marnu and others, convened legal and machine-learning experts "to systematically audit and trace 1800+ text datasets" with "tools and standards to trace the lineage of these datasets, from their source, creators, series of license conditions, properties, and subsequent use" (arXiv:2310.16787); the launch post of the same day counts "nearly 2,000 of the most widely used fine-tuning datasets," downloaded "tens of millions of times" (Cohere).3
  • F33 arXiv — [2310.16787] The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI, abstract page header documented · field
    [Submitted on 25 Oct 2023 (v1), last revised 4 Nov 2023 (this version, v3)] … Title:The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI … Authors:Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marnu
  • F34 arXiv — [2310.16787] The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI, Abstract documented · field
    we convene a multi-disciplinary effort between legal and machine learning experts to systematically audit and trace 1800+ text datasets. We develop tools and standards to trace the lineage of these datasets, from their source, creators, series of license conditions, properties, and subsequent use.
  • F24 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, article body ¶2 documented · field
    audit and trace nearly 2,000 of the most widely used fine-tuning datasets. Collectively, these data sets have been downloaded tens of millions of times, and are the backbone of many published NLP breakthroughs.
10.Researchers "from MIT, Cohere for AI, and 11 other institutions" launched it as a free platform, the audit being "the launch of a wider multi-institutional initiative, where users all over the world can contribute to the explorer" (Cohere), and the Dataset Explorer still stands under the Data Provenance Explorer with its paper and repository linked (Dataset Explorer).3
  • F22 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, article body: standfirst documented · field
    Researchers from MIT, Cohere for AI, and 11 other institutions launch free Data Provenance Explorer platform to track and filter audited datasets for ethical, legal, and transparency considerations
  • F32 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, Path Forward ¶2 documented · field
    This large-scale audit is the launch of a wider multi-institutional initiative, where users all over the world can contribute to the explorer
  • F20 Data Provenance Initiative — Dataset Explorer - Data Provenance Initiative, breadcrumb (Qh) and header buttons (Lh) documented · field
    Data Provenance Explorer … Paper … Repository
11.Its operator, the Data Provenance Initiative (Longpre, Mahari, Chen, Obeng-Marnu and others, by the founding paper: MMRS record), calls its work "Public measurement infrastructure for AI data, model ecosystems, and real-world AI use" (its home page, which the MMRS record quotes and whose phrase the record's evidence assigns to the founding paper), and says it releases "papers, datasets, and dashboards" so that researchers, journalists, policymakers and builders can "reason from shared evidence" (home page).5
  • MMRS-09 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · census.json: operator (evidence: arxiv.org/abs/2310.16787, basis operator) attributed
    Data Provenance Initiative (Longpre, Mahari, Chen, Obeng-Marnu and others)
  • F5 Data Provenance Initiative — Data Provenance Initiative, dataprovenance.org: meta description (served HTML) documented · field
    Public measurement infrastructure for AI data, model ecosystems, and real-world AI use.
  • MMRS-03 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · articles.py, 'What it is' s1 attributed
    The Initiative calls its work "public measurement infrastructure for AI data, model ecosystems, and real-world AI use."
  • MMRS-17 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · census.json: self_description (evidence: arxiv.org/abs/2310.16787, basis operator) attributed
    Public measurement infrastructure for AI data, model ecosystems, and real-world AI use
  • F13 Data Provenance Initiative — Data Provenance Initiative, home (Cr): 'Public-interest infrastructure' documented · field
    DPI releases papers, datasets, and dashboards that help researchers, journalists, policymakers, and builders reason from shared evidence.
12.#98 reads the Initiative as a project that "Documents provenance of popular AI training datasets" and states its limitation as "Documentation project, not infrastructure," and in its table of what the archive takes from others it records "Provenance schema | Data Provenance Initiative | Training-pipeline provenance | Implemented (in CHA)" (#98).2
  • D98-01 #98 Distributed Compute Is Not Distributed Intelligence: A Competitive Analysis of the DePIN Landscape and the Missing Civic Lee Sharks · 2026-05-07 · § D. Data Provenance and Consent, table row 1 interpretation
    | Data Provenance Initiative (MIT) | Documents provenance of popular AI training datasets | Documentation project, not infrastructure |
  • R98-04 #98 Distributed Compute Is Not Distributed Intelligence: A Competitive Analysis of the DePIN Landscape and the Missing Civic Lee Sharks · 2026-05-07 · the component table, row 'Provenance schema' stipulation
    | Provenance schema | Data Provenance Initiative | Training-pipeline provenance | Implemented (in CHA) |
13.The founding audit found "license omission of 70%+ and error rates of 50%+" on widely used dataset hosting sites, "a crisis in misattribution" (arXiv:2310.16787); the MMRS record quotes the finding, #725 reports it with the further figures that "85% of licenses request attribution and 30% include share-alike clauses" and with Nature Machine Intelligence 2024 as its venue, and #1714 carries it as what "is lost before training even begins" (MMRS record; #725; #1714).5
  • F36 arXiv — [2310.16787] The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI, Abstract documented · field
    We also observe frequent miscategorization of licenses on widely used dataset hosting sites, with license omission of 70%+ and error rates of 50%+. This points to a crisis in misattribution and informed use of the most popular datasets driving many recent breakthroughs.
  • MMRS-02 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · articles.py, kernel s2 attributed
    Its founding audit covered more than 1,800 text datasets and found licence omission above 70% and error rates above 50%.
  • MMRS-05 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · articles.py, 'What its audits found' s1 attributed
    The founding audit (arXiv:2310.16787) found "license omission of 70%+ and error rates of 50%+" on dataset hosting sites.
  • D725-02 #725 Provenance After AI Metadata Packet for Disambiguation: From Artifact Authenticity to Licensing Audit to Semantic Proven Lee Sharks · 2026-05-08 · ¶ 'The second dimension — training-corpus licensing' attributed
    The Data Provenance Initiative (Longpre et al., Nature Machine Intelligence 2024) audited 1,800+ datasets, finding that 85% of licenses request attribution and 30% include share-alike clauses, with license omission rates above 70% and error rates above 50% on popular hosting sites.
  • D1714-01 #1714 Meaning Feudalism at the Discovery Layer: Shepherds, Raw Research Data, and the Allocation of Semantic Capital in OpenAI Sharks, Lee · 2026-10-27 · §9.2 s3 attributed
    The audit literature shows how much is lost before training even begins: the Data Provenance Initiative found license information missing for more than 70% of widely used datasets on the major hosting sites and mislabeled for more than half (Longpre et al. 2023).
14.The launch post places the omission in "Crowdsourced aggregators like GitHub, Papers with Code," which miss licences for 72 to 83 percent of datasets against 30 percent under the Initiative's own protocol and list them "too permissively in up to 29 percent" of cases, so that distorted licensing "leads to cascading misattribution" (Cohere).2
  • F27 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, Addressing the Data Transparency Crisis ¶1 documented · field
    Crowdsourced aggregators like GitHub, Papers with Code, and many of the open source LLMs trained from data on these aggregators, have an extremely high proportion of missing data licenses (“Unspecified”), ranging from 72 to 83 percent. This compares to 30 percent missing licenses with our own annotation protocol
  • F28 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, Addressing the Data Transparency Crisis ¶2 documented · field
    Crowdsourced aggregators listed licenses too permissively in up to 29 percent of the various cases audited. Incorrect or distorted licensing leads to cascading misattribution
15.The audit also found "sharp divides" between commercially open and closed datasets, closed ones "monopolizing" lower-resource languages, more creative tasks, richer topic variety and newer, more synthetic data (arXiv:2310.16787), a divide the launch post calls "sharp and widening," beside "a stark Western-centric skew" in representation (Cohere).3
  • F35 arXiv — [2310.16787] The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI, Abstract documented · field
    Our landscape analysis highlights the sharp divides in composition and focus of commercially open vs closed datasets, with closed datasets monopolizing important categories: lower resource languages, more creative tasks, richer topic variety, newer and more synthetic training data.
  • F29 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, Understanding Systematic Differences ¶2 documented · field
    There is a sharp and widening divide between data licensed as commercially open versus closed.
  • F30 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, Understanding Systematic Differences ¶4 documented · field
    We find a stark Western-centric skew in representation across datasets.
16.Its second explorer, the Web Data Explorer (on the home page the Web Consent Explorer), follows "Shifting preferences in how websites allow their data to be used for AI" across 14,000 domains' terms, robots.txt and crawl restrictions (its pages), from Consent in Crisis (Longpre, Mahari, Lee and others, submitted 20 July 2024; NeurIPS 2024 on the home page, "NeurIPS 2024 Datasets and Benchmarks" in the MMRS record), "the first, large-scale, longitudinal audit of the consent protocols for the web domains underlying AI training corpora" (arXiv:2407.14933).7
  • F44 Data Provenance Initiative — Web Data Explorer, /data-provenance-explorer/web-data-explorer (ed): header documented · field
    Web Data Explorer … Shifting preferences in how websites allow their data to be used for AI
  • F8 Data Provenance Initiative — Data Provenance Initiative, home (Cr): flagship system 'Web Consent Explorer' documented · field
    Web Consent Explorer … Web data … 14,000 domains … Terms, robots.txt, and crawl restrictions … Track how web consent signals are changing as AI developers compete for training data.
  • F18 Data Provenance Initiative — Data Provenance Explorer, /data-provenance-explorer (f2): 'Web Data Explorer' documented · field
    An explorer of the trends in data-use preferences from the websites commonly used to source AI data.
  • F38 arXiv — [2407.14933] Consent in Crisis: The Rapid Decline of the AI Data Commons, abstract page header documented · field
    [Submitted on 20 Jul 2024 (v1), last revised 24 Jul 2024 (this version, v2)] … Title:Consent in Crisis: The Rapid Decline of the AI Data Commons … Authors:Shayne Longpre, Robert Mahari, Ariel Lee
  • F12 Data Provenance Initiative — Data Provenance Initiative, home (Cr): publications, 2024 documented · field
    Consent in Crisis: The Rapid Decline of the AI Data Commons … An audit of terms, restrictions, and consent signals across the web. … NeurIPS 2024
  • MMRS-13 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · census.json: note (evidence: arxiv.org/abs/2407.14933, basis operator) attributed
    Its second audit, Consent in Crisis (NeurIPS 2024 Datasets and Benchmarks), read 14,000 web domains
  • F39 arXiv — [2407.14933] Consent in Crisis: The Rapid Decline of the AI Data Commons, Abstract documented · field
    we conduct the first, large-scale, longitudinal audit of the consent protocols for the web domains underlying AI training corpora. Our audit of 14,000 web domains provides an expansive view of crawlable web data and how codified data use preferences are changing over time.
17.The explorer page charts HEAD-C4's "2k of the largest and most critical web domains" from 2016 to April 2024, counts a domain restricted when its robots.txt disallows the named AI crawlers, and finds "significant recent spikes in restrictions" (Web Data Explorer).2
  • F45 Data Provenance Initiative — Web Data Explorer, web-data-explorer ($h): 'Trends for Robots.txt and Terms of Service' documented · field
    We conduct a temporal analysis, from 2016 to April 2024, of the web consent signals in HEAD-C4, 2k of the largest and most critical web domains. … A web domain is considered restricted if its robots.txt disallows access to AI crawlers from OpenAI, Google, Anthropic, Common Crawl, Cohere, Meta, or the Internet Archive.
  • F46 Data Provenance Initiative — Web Data Explorer, web-data-explorer ($h): 'Trends for Robots.txt and Terms of Service' documented · field
    In all cases there are significant recent spikes in restrictions.
18.The paper finds that in 2023–2024 "~5%+ of all tokens in C4, or 28%+ of the most actively maintained, critical sources in C4" became fully restricted and that "a full 45% of C4 is now restricted" by terms of service (arXiv:2407.14933; the MMRS record quotes the second), diagnoses "ineffective web protocols," and holds, as a conditional, that "If respected or enforced, these restrictions are rapidly biasing the diversity, freshness, and scaling laws for general-purpose AI systems" (arXiv:2407.14933).4
  • F40 arXiv — [2407.14933] Consent in Crisis: The Rapid Decline of the AI Data Commons, Abstract documented · field
    in a single year (2023-2024) there has been a rapid crescendo of data restrictions from web sources, rendering ~5%+ of all tokens in C4, or 28%+ of the most actively maintained, critical sources in C4, fully restricted from use. For Terms of Service crawling restrictions, a full 45% of C4 is now restricted.
  • MMRS-06 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · articles.py, 'What its audits found' s2 attributed
    Consent in Crisis (arXiv:2407.14933, NeurIPS 2024) read 14,000 web domains and found that "a full 45% of C4 is now restricted" by terms of service.
  • F41 arXiv — [2407.14933] Consent in Crisis: The Rapid Decline of the AI Data Commons, Abstract documented · field
    We diagnose these as symptoms of ineffective web protocols, not designed to cope with the widespread re-purposing of the internet for AI.
  • F42 arXiv — [2407.14933] Consent in Crisis: The Rapid Decline of the AI Data Commons, Abstract documented · field
    If respected or enforced, these restrictions are rapidly biasing the diversity, freshness, and scaling laws for general-purpose AI systems.
19.#98 reads Consent in Crisis as an "Audit of C4, RefinedWeb, Dolma" that "documents collapse of web-consent infrastructure," a "Diagnostic paper," and sets the Initiative at "L4: Provenance / Consent," where what is missing is "Integrated provenance in training pipeline" (#98).2
  • D98-02 #98 Distributed Compute Is Not Distributed Intelligence: A Competitive Analysis of the DePIN Landscape and the Missing Civic Lee Sharks · 2026-05-07 · § D. Data Provenance and Consent, table row 2 interpretation
    | "Consent in Crisis" (Longpre et al., 2024) | Audit of C4, RefinedWeb, Dolma; documents collapse of web-consent infrastructure | Diagnostic paper |
  • O98-03 #98 Distributed Compute Is Not Distributed Intelligence: A Competitive Analysis of the DePIN Landscape and the Missing Civic Lee Sharks · 2026-05-07 · the layer table, row L4 interpretation
    | L4: Provenance / Consent | Where did this come from | Data Provenance Initiative, DECORAIT (narrow) | Integrated provenance in training pipeline |
20.Its authors state that "major legal ambiguities remain in data licensing that the authors acknowledge cannot be resolved by tooling alone" (Cohere), and #725 states the dimension's own bound: "The Data Provenance Initiative audits whether datasets were licensed, not whether synthesized outputs preserve attribution to the human sources whose labor the synthesis depended upon"; "Licensing audits operate on the *input* to AI systems. They do not address the *output*," and the Initiative's 70%+ omission "establishes the licensing dimension's urgency; semantic provenance addresses the dimension that follows" (#725).3
  • F31 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, Open Legal Ambiguities ¶1 documented · field
    major legal ambiguities remain in data licensing that the authors acknowledge cannot be resolved by tooling alone.
  • D725-04 #725 Provenance After AI Metadata Packet for Disambiguation: From Artifact Authenticity to Licensing Audit to Semantic Proven Lee Sharks · 2026-05-08 · ¶ 'Existing frameworks are not designed to detect this' interpretation
    The Data Provenance Initiative audits whether datasets were licensed, not whether synthesized outputs preserve attribution to the human sources whose labor the synthesis depended upon.
  • D725-08 #725 Provenance After AI Metadata Packet for Disambiguation: From Artifact Authenticity to Licensing Audit to Semantic Proven Lee Sharks · 2026-05-08 · §3.2 Misreading: provenance as licensing-only, Correction interpretation
    Licensing audits operate on the *input* to AI systems. They do not address the *output*. A model trained on properly licensed scholarship can still produce outputs that erase the scholarship's lineage. Licensing provenance and semantic provenance are different problems requiring different instruments. The DPI's documentation of 70%+ license-omission rates establishes the licensing dimension's urgency; semantic provenance addresses the dimension that follows.
21.The dimensions are independent, "a licensed dataset can produce outputs with excellent semantic provenance if the synthesis preserves attribution; an unlicensed dataset can produce outputs that nonetheless cite their sources clearly," and the frameworks that address "the corpus," the Initiative among them, "are not designed to address the survival of authorial lineage through AI synthesis" (#725).2
  • O725-06 #725 Provenance After AI Metadata Packet for Disambiguation: From Artifact Authenticity to Licensing Audit to Semantic Proven Lee Sharks · 2026-05-08 · Central Invariant, ¶ after the table interpretation
    The dimensions are independent: a C2PA-signed image can carry meaning whose lineage has been eroded; a licensed dataset can produce outputs with excellent semantic provenance if the synthesis preserves attribution; an unlicensed dataset can produce outputs that nonetheless cite their sources clearly. Solving any one dimension does not solve the others.
  • D725-01 #725 Provenance After AI Metadata Packet for Disambiguation: From Artifact Authenticity to Licensing Audit to Semantic Proven Lee Sharks · 2026-05-08 · Canonical Claim interpretation
    Existing provenance frameworks address the artifact (C2PA / Content Credentials) and the corpus (Data Provenance Initiative, EU AI Act transparency provisions, W3C PROV). They are not designed to address the survival of authorial lineage through AI synthesis.
22."The Data Provenance Initiative tells you whether the dataset was licensed," and none of these instruments was designed to tell "what happened to the meaning … between the dataset and the answer" (#725); #1003 groups "C2PA, the Data Provenance Initiative, W3C PROV" as studies that "have focused on attribution loss in AI outputs" (#1003).2
  • D725-09 #725 Provenance After AI Metadata Packet for Disambiguation: From Artifact Authenticity to Licensing Audit to Semantic Proven Lee Sharks · 2026-05-08 · closing ¶ ('C2PA tells you …') interpretation
    The Data Provenance Initiative tells you whether the dataset was licensed. EU AI Act transparency provisions tell you whether the content was AI-generated. None of these instruments was designed to tell you what happened to the meaning between the camera and the screen, between the dataset and the answer, between the source and the synthesis.
  • D1003-01 #1003 AI-Native Intellectual Biography: Genre, Provenance, and the Archive That Outpaced Its Author Description (as recorded b Sharks, Lee · 2026-05-21 · ¶ 'Provenance and attribution studies' interpretation
    **Provenance and attribution studies** in digital environments — C2PA, the Data Provenance Initiative, W3C PROV — have focused on attribution loss in AI outputs.
23.The MMRS record sets the Explorer at one end of a chain: "It records provenance at the training layer, where the census's reception aggregators record it at the composition layer. The two ends of one chain: what a model was given, and what it credits when it answers" (MMRS record), with its roles provenance and monitoring (MMRS record); the Initiative's own four systems "trace AI from source data and web consent to model markets and real-world use" (home page).4
  • MMRS-07 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · articles.py, 'Place in the field' s1 interpretation
    It records provenance at the training layer, where the census's reception aggregators record it at the composition layer.
  • MMRS-08 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · articles.py, 'Place in the field' s2 interpretation
    The two ends of one chain: what a model was given, and what it credits when it answers.
  • MMRS-14 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · census.json: roles stipulation
    provenance", "monitoring
  • F9 Data Provenance Initiative — Data Provenance Initiative, home (Cr): 'Flagship systems' documented · field
    Four public measurement systems … Together they trace AI from source data and web consent to model markets and real-world use.
24.At the answering end, PER "applies to these only indirectly" where claims are parametric, and reaching them at the substrate level "requires companion infrastructure (training-data provenance records, watermarking, content credentials)" (#95); #623 counts standardized documentation of training-data provenance as "a step toward the accounting the framework demands" (#623).2
  • O95-01 #95 Provenance Alignment: Attribution Survival as a Substrate Condition for Safe AI Knowledge Composition Lee Sharks · 2026-05-05 · § the typology of source-dependent claims (parametric claims) interpretation
    Parametric claims are drawn from the model's training-time parameters rather than runtime retrieval, often without surface signal indicating which sources materially contributed; PER applies to these only indirectly, and addressing them at the substrate level requires companion infrastructure (training-data provenance records, watermarking, content credentials).
  • O623-01 #623 Semantic Infrastructure: From Tim Berners-Lee to the Semantic Economy — Bridging Technical and Political-Economic Framew Lee Sharks · 2026-04-01 · §8.3, final sentence interpretation
    Gebru et al.'s proposal for "datasheets for datasets" — standardized documentation of training data provenance — is a step toward the accounting the framework demands.
25.#1714 states the requirement across the chain, "Provenance has to survive the training pipeline," since neither audit nor influence tracing "recovers authorship" and "what survives depends on what is recorded and computed," so that sources can be "recorded at ingestion, the record kept with the training set" (#1714), where the founding audit traced lineage as far as a dataset's "subsequent use" (arXiv:2310.16787).3
  • D1714-02 #1714 Meaning Feudalism at the Discovery Layer: Shepherds, Raw Research Data, and the Allocation of Semantic Capital in OpenAI Sharks, Lee · 2026-10-27 · §9.2 s1–2 interpretation
    Provenance has to survive the training pipeline. The pipeline is a built thing, and the slushing is a property of how it is built.
  • D1714-03 #1714 Meaning Feudalism at the Discovery Layer: Shepherds, Raw Research Data, and the Allocation of Semantic Capital in OpenAI Sharks, Lee · 2026-10-27 · §9.2 s5–6 interpretation
    Neither recovers authorship, and both show that what survives depends on what is recorded and computed. Sources can be recorded at ingestion, the record kept with the training set, and a result released with the dependencies its producer can report
  • F34 arXiv — [2310.16787] The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI, Abstract documented · field
    we convene a multi-disciplinary effort between legal and machine learning experts to systematically audit and trace 1800+ text datasets. We develop tools and standards to trace the lineage of these datasets, from their source, creators, series of license conditions, properties, and subsequent use.
26.The MMRS record keeps each assertion with "a source and a basis" and keeps lifecycle (active, observed 2026-10-28) apart from the observation that saw it, and it states its own limits: it has not read "the explorers' pages directly (they render by script)" and takes the Initiative's page text from "the Overview's cards" (MMRS record).4
  • MMRS-20 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · census.json: schema_note stipulation
    each assertion carries a source and a basis (operator statement, third-party report, our observation, our reading); lifecycle is kept apart from the observation that saw it
  • MMRS-15 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · census.json: lifecycle; observed documented
    state": "active
  • MMRS-18 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · census.json: unchecked self-description
    the explorers' pages directly (they render by script)
  • MMRS-19 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · census.json: sources['alexanarch.org/captures/data-provenance-explorer-aio-20261029'].for self-description
    the Initiative's own page text, on the Overview's cards (the pages render nothing to this record's fetcher)
27.The Initiative lists coverage by "the Financial Times, The New York Times, The Washington Post, MIT Technology Review, IEEE Spectrum, and VentureBeat" and use in "public AI safety and governance reporting" (home page), and at "data provenance explorer" Google's AI Overview said what the explorer is and what can be filtered in it "from the Initiative's own pages," reaching pages the record's fetcher could not render (Capture Registry, 2026-10-29), the capture standing in the record as "the composition layer's account of the entity" (MMRS record).5
  • F14 Data Provenance Initiative — Data Provenance Initiative, home (Cr): 'Featured coverage' documented · field
    DPI findings have been covered by the Financial Times, The New York Times, The Washington Post, MIT Technology Review, IEEE Spectrum, and VentureBeat.
  • F15 Data Provenance Initiative — Data Provenance Initiative, home (Cr): 'Policy and governance relevance' documented · field
    Our work has been cited in public AI safety and governance reporting and used to ground debates about data transparency, consent, and market concentration.
  • CAP-1-05 capture-registry Capture Registry (EA-WG-CAPTURES-01), entry data-provenance-explorer-aio-20261029 (OBS-dc0adc018f10, AXN:0AAC.CAPTURE.🕔🎨 Lee Sharks (the registry) · 2026-10-29 · data-provenance-explorer-aio-20261029, reading s2 interpretation
    The Overview says what the explorer is and what can be filtered in it, from the Initiative's own pages.
  • CAP-1-06 capture-registry Capture Registry (EA-WG-CAPTURES-01), entry data-provenance-explorer-aio-20261029 (OBS-dc0adc018f10, AXN:0AAC.CAPTURE.🕔🎨 Lee Sharks (the registry) · 2026-10-29 · data-provenance-explorer-aio-20261029, analysis s1 interpretation
    The composition reaches the Initiative's current pages, which this record's fetcher could not render, and corrects the record's scale.
  • MMRS-21 MMRS MMRS meta-aggregator record: Data Provenance Explorer (https://www.machinemediation.org/aggregators/data-provenance-expl Lee Sharks (editor) · 2026-10-28 (corrected 2026-10-29) · census.json: relations (capture, observed 2026-10-29) documented
    the composition layer's account of the entity, seated in the Capture Registry
28.The Capture Registry finds that "The audits that make the explorer a provenance instrument are left out": "The founding audit's licence findings and Consent in Crisis are not given; the consent explorer stands for the second" (Capture Registry, 2026-10-29).2
  • CAP-1-07 capture-registry Capture Registry (EA-WG-CAPTURES-01), entry data-provenance-explorer-aio-20261029 (OBS-dc0adc018f10, AXN:0AAC.CAPTURE.🕔🎨 Lee Sharks (the registry) · 2026-10-29 · data-provenance-explorer-aio-20261029, analysis s2 interpretation
    The audits that make the explorer a provenance instrument are left out.
  • CAP-1-04 capture-registry Capture Registry (EA-WG-CAPTURES-01), entry data-provenance-explorer-aio-20261029 (OBS-dc0adc018f10, AXN:0AAC.CAPTURE.🕔🎨 Lee Sharks (the registry) · 2026-10-29 · data-provenance-explorer-aio-20261029, findings[1] s2 interpretation
    The founding audit's licence findings and Consent in Crisis are not given; the consent explorer stands for the second.
29.The composition named the Initiative and cited its pages, with Shayne Longpre and Sara Hooker on the Cohere card only, and no card cited the MMRS record, which had landed the same night (Capture Registry, 2026-10-29, checked against the record and the entity's own pages); Wikipedia has no article for "Data Provenance Explorer" or "Data Provenance Initiative," and its search finds neither phrase in any article (Wikipedia, 2026-10-29).5
  • CAP-1-09 capture-registry Capture Registry (EA-WG-CAPTURES-01), entry data-provenance-explorer-aio-20261029 (OBS-dc0adc018f10, AXN:0AAC.CAPTURE.🕔🎨 Lee Sharks (the registry) · 2026-10-29 · data-provenance-explorer-aio-20261029, per_note documented
    The Data Provenance Initiative named; Shayne Longpre and Sara Hooker on the Cohere card only; the Initiative's pages cited.
  • CAP-1-08 capture-registry Capture Registry (EA-WG-CAPTURES-01), entry data-provenance-explorer-aio-20261029 (OBS-dc0adc018f10, AXN:0AAC.CAPTURE.🕔🎨 Lee Sharks (the registry) · 2026-10-29 · data-provenance-explorer-aio-20261029, q_kind documented
    A baseline: the record landed on machinemediation.org the same night (commit 8997e28) and no card cites it.
  • CAP-1-03 capture-registry Capture Registry (EA-WG-CAPTURES-01), entry data-provenance-explorer-aio-20261029 (OBS-dc0adc018f10, AXN:0AAC.CAPTURE.🕔🎨 Lee Sharks (the registry) · 2026-10-29 · data-provenance-explorer-aio-20261029, findings[0] s4 documented
    Checked 2026-10-29 against the MMRS meta-aggregator record of 2026-10-28 and the entity's own pages, re-read the same day.
  • F47 Wikipedia — Data Provenance Explorer (no article), Wikipedia: title Data_Provenance_Explorer documented · field
    raw wikitext (action=raw, redirect=no): HTTP 404 … Wikipedia does not have an article with this exact name. … There were no results matching the query.
  • F48 Wikipedia — Data Provenance Initiative (no article), Wikipedia: title Data_Provenance_Initiative documented · field
    raw wikitext (action=raw, redirect=no): HTTP 404 … Wikipedia does not have an article with this exact name. … There were no results matching the query.

the compression: AIO's interaction grammar (lede, headed clusters, bolded labels, card rail), composed from the v0.9 plan in one claim space; it faces on the card. Composed under v0.9, equal case (ρ = extends): KO_B sentences surviving 0 of 12; every field claim carried. composed 2026-10-29 from the v0.9 knowledge object; not frozen

Compression · The field's ontology — L_F(B)data provenance explorer

Data Provenance Explorer is the Data Provenance Initiative's tool for exploring the datasets of its audit: nearly 4,000 text, speech and video datasets, filtered by licence, sources and creators (its pages; founding paper).

🗂️ What it is

  • Two explorers: a Dataset Explorer and a Web Data Explorer (its pages).3↘
    The Data Provenance Explorer lets its users explore the datasets included in the Initiative's audit, filtering and analysing language-model training datasets by use case, language and more (explorer introduction; landing page; MIT Media Lab's card, snippet only).§1 The founding paper released the whole audit with an interactive UI, the Data Provenance Explorer, to trace and filter on provenance for popular open-source fine-tuning collections (arXiv:2310.16787), for developers filtering for legal and ethical considerations and for scholars and journalists exploring data lineage (Cohere).§2 The Web Data Explorer tracks shifting website preferences on AI use over 14,000 domains (its pages), from Consent in Crisis (submitted 20 July 2024; NeurIPS 2024), the first longitudinal audit of consent protocols, charting HEAD-C4's 2k critical domains with recent spikes in restrictions (arXiv; Web Data Explorer).§10
  • The record: sources, creators, licences, derivations per dataset; 3,917 rows (data_summaries.csv).2↘
    Data provenance, in the launch post's words, is the tracking of "the origins of a piece of data" (Cohere).§3 Each dataset is traced to its sources, licences, creators, geographies, modalities and derivation chains, its re-licensings tagged (home page; Cohere), and the data file the Dataset Explorer loads carries these fields with the Initiative's licence readings, 3,917 rows at fetch, over which the explorer computes its summaries (data_summaries.csv; Dataset Explorer).§4

🧾 Origin

  • Founding audit: 1800+ text datasets, 25 October 2023 (arXiv); launched with Cohere for AI (Cohere).1↘
    Longpre, Mahari, Chen, Obeng-Marnu and others submitted the founding audit of 1800+ text datasets on 25 October 2023 (arXiv), launched the same day with Cohere for AI and eleven other institutions as a free platform open to contributions, nearly 2,000 fine-tuning datasets in the launch post's count (Cohere), with its paper and repository linked from the explorer (Dataset Explorer).§6
  • Operator: "Public measurement infrastructure for AI data, model ecosystems, and real-world AI use."2↘
    Its pages count nearly 4,000 text, video and speech datasets, 4,000+ in 608 languages, from the multimodal audit of four thousand datasets (ICLR 2025), which found source content often more restricted than dataset releases (its pages).§5 The Initiative calls itself public measurement infrastructure for AI data, model ecosystems and real-world AI use, with four measurement systems and papers, datasets and dashboards (home page).§7

🔎 Findings

  • Licences: omission 70%+, errors 50%+ (arXiv); 72–83% missing on aggregators (Cohere).2↘
    The audit found licence omission of 70%+ and error rates of 50%+ on hosting sites (arXiv), with 72 to 83 percent of licences missing on crowdsourced aggregators such as GitHub and Papers with Code and licences listed too permissively in up to 29 percent of cases (Cohere).§8 It found sharp divides between commercially open and closed datasets and a Western-centric skew (arXiv; Cohere), and its authors state that legal ambiguities in data licensing cannot be resolved by tooling alone (Cohere).§9
  • Consent: 45% of C4 restricted by terms of service (Consent in Crisis).2↘
    The Web Data Explorer tracks shifting website preferences on AI use over 14,000 domains (its pages), from Consent in Crisis (submitted 20 July 2024; NeurIPS 2024), the first longitudinal audit of consent protocols, charting HEAD-C4's 2k critical domains with recent spikes in restrictions (arXiv; Web Data Explorer).§10 In 2023–2024 about 5% of C4's tokens became fully restricted and 45% of C4 by terms of service, symptoms of ineffective web protocols which, if respected or enforced, bias general-purpose AI (arXiv:2407.14933).§11

📡 Reception

  • Elsewhere: press coverage; no Wikipedia article.1↘
    The Initiative reports press coverage and governance use (home page); Wikipedia has no article on the Explorer or the Initiative.§12
1what it is: the explorer of an audit of training datasetsarXiv — [2310.16787] The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI6 claims · B1, B2, B4, B6, B7
  • F1 Data Provenance Initiative — Data Provenance Explorer, /explorer-introduction (d2): headings documented · field
    The Data Provenance Explorer … Explore the datasets included in our audit. The DPI Explorer tool allows you to filter for and analyze language model training datasets.
  • F2 Data Provenance Initiative — Data Provenance Explorer, /explorer-introduction (d2): slide caption documented · field
    Select and filter use cases, languages, and more
  • F16 Data Provenance Initiative — Data Provenance Explorer, /data-provenance-explorer (p2): heading documented · field
    Data Provenance Explorer … Discover the datasets featured in our audit! The DPI Explorer tool enables you to filter and analyze language model training datasets effortlessly.
  • F3 MIT Media Lab — Data Provenance Explorers - MIT Media Lab, card snippet (MIT Media Lab unreachable: TLS failure) documented · field
    Based on our extensive audits of AI data, we've built data explorers to allow practitioners to make more nuanced and informed decisions about th
  • F37 arXiv — [2310.16787] The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI, Abstract documented · field
    we release our entire audit, with an interactive UI, the Data Provenance Explorer, which allows practitioners to trace and filter on data provenance for the most popular open source finetuning data collections
  • F26 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, article body ¶4 documented · field
    The platform allows developers to track and filter thousands of datasets for legal and ethical considerations, and enables scholars and journalists to explore the composition and data lineage of popular AI datasets.
2the operator: the Data Provenance InitiativeData Provenance Initiative — Data Provenance Initiative3 claims · B2, B3
  • F4 MIT Media Lab — Data Provenance Explorers - MIT Media Lab, card title (MIT Media Lab unreachable) documented · field
    Data Provenance Explorers - MIT Media Lab
  • F5 Data Provenance Initiative — Data Provenance Initiative, dataprovenance.org: meta description (served HTML) documented · field
    Public measurement infrastructure for AI data, model ecosystems, and real-world AI use.
  • F13 Data Provenance Initiative — Data Provenance Initiative, home (Cr): 'Public-interest infrastructure' documented · field
    DPI releases papers, datasets, and dashboards that help researchers, journalists, policymakers, and builders reason from shared evidence.
3data provenance definedCohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis1 claim · B6
  • F23 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, article body ¶1 documented · field
    Inconsistent tracking of the origins of a piece of data, which we refer to as data provenance, has introduced a range of legal and ethical challenges.
4the unit: a dataset traced to its sources, licences and creatorsCohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis4 claims · B3, B5, B6, B9
  • F7 Data Provenance Initiative — Data Provenance Initiative, home (Cr): flagship system 'Data Provenance Explorer', description documented · field
    Inspect dataset sources, licenses, creators, geographies, modalities, and derivation chains.
  • F25 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, article body ¶3 documented · field
    The result of this multidisciplinary initiative is the single largest audit to date of AI datasets. For the first time, these datasets include tags to the original data sources, numerous re-licensings, creators, and other data properties.
  • F43 Data Provenance Initiative — /data_summaries.csv, header row of /data_summaries.csv (3,917 rows at fetch) documented · field
    Unique Dataset Identifier,Dataset Name … Text Sources,Model Generated,Human Annotation,Derived from Datasets,Creators,Licenses … License Use (DataProvenance),License Attribution (DataProvenance),License Share Alike (DataProvenance),Modality
  • F21 Data Provenance Initiative — Dataset Explorer - Data Provenance Initiative, /data-provenance-explorer/dataset-explorer (u2): summary table documented · field
    Text Sources … Model Generated … Mean Input Chars … Mean Target Chars … Generated By … % Datasets Used
5count: nearly 4,000 text, speech and video datasetsData Provenance Initiative — Dataset Explorer - Data Provenance Initiative3 claims · B3, B4, B5
  • F19 Data Provenance Initiative — Dataset Explorer - Data Provenance Initiative, /data-provenance-explorer/dataset-explorer (u2): header documented · field
    Dataset Explorer … Explore nearly 4,000 text, video, and speech datasets, and their characteristics.
  • F6 Data Provenance Initiative — Data Provenance Initiative, home (Cr): flagship system 'Data Provenance Explorer' documented · field
    Data Provenance Explorer … Training data … 4,000+ datasets … 608 languages across text, speech, and video
  • F17 Data Provenance Initiative — Data Provenance Explorer, /data-provenance-explorer (f2): 'Dataset Explorer' documented · field
    A tool to explore and filter 4000 text, video, and speech datasets, for their license, sources, creators, and other metadata.
6the multimodal audit (ICLR 2025)Data Provenance Initiative — Data Provenance Initiative2 claims · B3
  • F11 Data Provenance Initiative — Data Provenance Initiative, home (Cr): publications, 2025 documented · field
    Bridging the Data Provenance Gap Across Text, Speech, and Video … An audit of four thousand public text, speech, and video datasets. … ICLR 2025
  • F10 Data Provenance Initiative — Data Provenance Initiative, home (Cr): findings strip, 'Dataset restrictions' documented · field
    Dataset restrictions … Source content often carries stricter terms than dataset releases … Multimodal Data Provenance
7origin: the founding audit and the launch, October 2023arXiv — [2310.16787] The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI6 claims · B5, B6, B7
  • F33 arXiv — [2310.16787] The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI, abstract page header documented · field
    [Submitted on 25 Oct 2023 (v1), last revised 4 Nov 2023 (this version, v3)] … Title:The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI … Authors:Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marnu
  • F34 arXiv — [2310.16787] The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI, Abstract documented · field
    we convene a multi-disciplinary effort between legal and machine learning experts to systematically audit and trace 1800+ text datasets. We develop tools and standards to trace the lineage of these datasets, from their source, creators, series of license conditions, properties, and subsequent use.
  • F22 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, article body: standfirst documented · field
    Researchers from MIT, Cohere for AI, and 11 other institutions launch free Data Provenance Explorer platform to track and filter audited datasets for ethical, legal, and transparency considerations
  • F24 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, article body ¶2 documented · field
    audit and trace nearly 2,000 of the most widely used fine-tuning datasets. Collectively, these data sets have been downloaded tens of millions of times, and are the backbone of many published NLP breakthroughs.
  • F32 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, Path Forward ¶2 documented · field
    This large-scale audit is the launch of a wider multi-institutional initiative, where users all over the world can contribute to the explorer
  • F20 Data Provenance Initiative — Dataset Explorer - Data Provenance Initiative, breadcrumb (Qh) and header buttons (Lh) documented · field
    Data Provenance Explorer … Paper … Repository
8the operator's other measurement systemsData Provenance Initiative — Data Provenance Initiative1 claim · B3
  • F9 Data Provenance Initiative — Data Provenance Initiative, home (Cr): 'Flagship systems' documented · field
    Four public measurement systems … Together they trace AI from source data and web consent to model markets and real-world use.
9audit findings: licence omission and misattributionarXiv — [2310.16787] The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI3 claims · B6, B7
  • F36 arXiv — [2310.16787] The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI, Abstract documented · field
    We also observe frequent miscategorization of licenses on widely used dataset hosting sites, with license omission of 70%+ and error rates of 50%+. This points to a crisis in misattribution and informed use of the most popular datasets driving many recent breakthroughs.
  • F27 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, Addressing the Data Transparency Crisis ¶1 documented · field
    Crowdsourced aggregators like GitHub, Papers with Code, and many of the open source LLMs trained from data on these aggregators, have an extremely high proportion of missing data licenses (“Unspecified”), ranging from 72 to 83 percent. This compares to 30 percent missing licenses with our own annotation protocol
  • F28 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, Addressing the Data Transparency Crisis ¶2 documented · field
    Crowdsourced aggregators listed licenses too permissively in up to 29 percent of the various cases audited. Incorrect or distorted licensing leads to cascading misattribution
10audit findings: the commercial divide and the Western skewarXiv — [2310.16787] The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI3 claims · B6, B7
  • F35 arXiv — [2310.16787] The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI, Abstract documented · field
    Our landscape analysis highlights the sharp divides in composition and focus of commercially open vs closed datasets, with closed datasets monopolizing important categories: lower resource languages, more creative tasks, richer topic variety, newer and more synthetic training data.
  • F29 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, Understanding Systematic Differences ¶2 documented · field
    There is a sharp and widening divide between data licensed as commercially open versus closed.
  • F30 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, Understanding Systematic Differences ¶4 documented · field
    We find a stark Western-centric skew in representation across datasets.
11stated limits: what tooling cannot resolveCohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis1 claim · B6
  • F31 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, Open Legal Ambiguities ¶1 documented · field
    major legal ambiguities remain in data licensing that the authors acknowledge cannot be resolved by tooling alone.
12the Web Data Explorer and Consent in CrisisData Provenance Initiative — Data Provenance Initiative7 claims · B10, B3, B4, B8
  • F44 Data Provenance Initiative — Web Data Explorer, /data-provenance-explorer/web-data-explorer (ed): header documented · field
    Web Data Explorer … Shifting preferences in how websites allow their data to be used for AI
  • F8 Data Provenance Initiative — Data Provenance Initiative, home (Cr): flagship system 'Web Consent Explorer' documented · field
    Web Consent Explorer … Web data … 14,000 domains … Terms, robots.txt, and crawl restrictions … Track how web consent signals are changing as AI developers compete for training data.
  • F18 Data Provenance Initiative — Data Provenance Explorer, /data-provenance-explorer (f2): 'Web Data Explorer' documented · field
    An explorer of the trends in data-use preferences from the websites commonly used to source AI data.
  • F38 arXiv — [2407.14933] Consent in Crisis: The Rapid Decline of the AI Data Commons, abstract page header documented · field
    [Submitted on 20 Jul 2024 (v1), last revised 24 Jul 2024 (this version, v2)] … Title:Consent in Crisis: The Rapid Decline of the AI Data Commons … Authors:Shayne Longpre, Robert Mahari, Ariel Lee
  • F12 Data Provenance Initiative — Data Provenance Initiative, home (Cr): publications, 2024 documented · field
    Consent in Crisis: The Rapid Decline of the AI Data Commons … An audit of terms, restrictions, and consent signals across the web. … NeurIPS 2024
  • F39 arXiv — [2407.14933] Consent in Crisis: The Rapid Decline of the AI Data Commons, Abstract documented · field
    we conduct the first, large-scale, longitudinal audit of the consent protocols for the web domains underlying AI training corpora. Our audit of 14,000 web domains provides an expansive view of crawlable web data and how codified data use preferences are changing over time.
  • F45 Data Provenance Initiative — Web Data Explorer, web-data-explorer ($h): 'Trends for Robots.txt and Terms of Service' documented · field
    We conduct a temporal analysis, from 2016 to April 2024, of the web consent signals in HEAD-C4, 2k of the largest and most critical web domains. … A web domain is considered restricted if its robots.txt disallows access to AI crawlers from OpenAI, Google, Anthropic, Common Crawl, Cohere, Meta, or the Internet Archive.
13consent findings: the decline of the AI data commonsarXiv — [2407.14933] Consent in Crisis: The Rapid Decline of the AI Data Commons4 claims · B10, B8
  • F46 Data Provenance Initiative — Web Data Explorer, web-data-explorer ($h): 'Trends for Robots.txt and Terms of Service' documented · field
    In all cases there are significant recent spikes in restrictions.
  • F40 arXiv — [2407.14933] Consent in Crisis: The Rapid Decline of the AI Data Commons, Abstract documented · field
    in a single year (2023-2024) there has been a rapid crescendo of data restrictions from web sources, rendering ~5%+ of all tokens in C4, or 28%+ of the most actively maintained, critical sources in C4, fully restricted from use. For Terms of Service crawling restrictions, a full 45% of C4 is now restricted.
  • F41 arXiv — [2407.14933] Consent in Crisis: The Rapid Decline of the AI Data Commons, Abstract documented · field
    We diagnose these as symptoms of ineffective web protocols, not designed to cope with the widespread re-purposing of the internet for AI.
  • F42 arXiv — [2407.14933] Consent in Crisis: The Rapid Decline of the AI Data Commons, Abstract documented · field
    If respected or enforced, these restrictions are rapidly biasing the diversity, freshness, and scaling laws for general-purpose AI systems.
14reception: press and governanceData Provenance Initiative — Data Provenance Initiative2 claims · B3
  • F14 Data Provenance Initiative — Data Provenance Initiative, home (Cr): 'Featured coverage' documented · field
    DPI findings have been covered by the Financial Times, The New York Times, The Washington Post, MIT Technology Review, IEEE Spectrum, and VentureBeat.
  • F15 Data Provenance Initiative — Data Provenance Initiative, home (Cr): 'Policy and governance relevance' documented · field
    Our work has been cited in public AI safety and governance reporting and used to ground debates about data transparency, consent, and market concentration.
15reception: the encyclopediaWikipedia — Data Provenance Explorer (no article)2 claims · B11, B12
  • F47 Wikipedia — Data Provenance Explorer (no article), Wikipedia: title Data_Provenance_Explorer documented · field
    raw wikitext (action=raw, redirect=no): HTTP 404 … Wikipedia does not have an article with this exact name. … There were no results matching the query.
  • F48 Wikipedia — Data Provenance Initiative (no article), Wikipedia: title Data_Provenance_Initiative documented · field
    raw wikitext (action=raw, redirect=no): HTTP 404 … Wikipedia does not have an article with this exact name. … There were no results matching the query.
Expansion · The field's ontology — L_F(B)12 sentences · every one sourced below

The Data Provenance Explorer

The Data Provenance Explorer lets its users explore the datasets included in the Initiative's audit, filtering and analysing language-model training datasets by use case, language and more (explorer introduction; landing page; MIT Media Lab's card, snippet only). The founding paper released the whole audit with an interactive UI, the Data Provenance Explorer, to trace and filter on provenance for popular open-source fine-tuning collections (arXiv:2310.16787), for developers filtering for legal and ethical considerations and for scholars and journalists exploring data lineage (Cohere). Data provenance, in the launch post's words, is the tracking of "the origins of a piece of data" (Cohere).

Each dataset is traced to its sources, licences, creators, geographies, modalities and derivation chains, its re-licensings tagged (home page; Cohere), and the data file the Dataset Explorer loads carries these fields with the Initiative's licence readings, 3,917 rows at fetch, over which the explorer computes its summaries (data_summaries.csv; Dataset Explorer). Its pages count nearly 4,000 text, video and speech datasets, 4,000+ in 608 languages, from the multimodal audit of four thousand datasets (ICLR 2025), which found source content often more restricted than dataset releases (its pages).

Longpre, Mahari, Chen, Obeng-Marnu and others submitted the founding audit of 1800+ text datasets on 25 October 2023 (arXiv), launched the same day with Cohere for AI and eleven other institutions as a free platform open to contributions, nearly 2,000 fine-tuning datasets in the launch post's count (Cohere), with its paper and repository linked from the explorer (Dataset Explorer). The Initiative calls itself public measurement infrastructure for AI data, model ecosystems and real-world AI use, with four measurement systems and papers, datasets and dashboards (home page).

The audit found licence omission of 70%+ and error rates of 50%+ on hosting sites (arXiv), with 72 to 83 percent of licences missing on crowdsourced aggregators such as GitHub and Papers with Code and licences listed too permissively in up to 29 percent of cases (Cohere). It found sharp divides between commercially open and closed datasets and a Western-centric skew (arXiv; Cohere), and its authors state that legal ambiguities in data licensing cannot be resolved by tooling alone (Cohere).

The Web Data Explorer tracks shifting website preferences on AI use over 14,000 domains (its pages), from Consent in Crisis (submitted 20 July 2024; NeurIPS 2024), the first longitudinal audit of consent protocols, charting HEAD-C4's 2k critical domains with recent spikes in restrictions (arXiv; Web Data Explorer). In 2023–2024 about 5% of C4's tokens became fully restricted and 45% of C4 by terms of service, symptoms of ineffective web protocols which, if respected or enforced, bias general-purpose AI (arXiv:2407.14933).

The Initiative reports press coverage and governance use (home page); Wikipedia has no article on the Explorer or the Initiative.

Provenance — every sentence sourced12 sentences

12 sentences; 48 field claims, 0 archive claims. Modality is the source's own; the prose carries it in its grammar.

1.The Data Provenance Explorer lets its users explore the datasets included in the Initiative's audit, filtering and analysing language-model training datasets by use case, language and more (explorer introduction; landing page; MIT Media Lab's card, snippet only).5
  • F1 Data Provenance Initiative — Data Provenance Explorer, /explorer-introduction (d2): headings documented · field
    The Data Provenance Explorer … Explore the datasets included in our audit. The DPI Explorer tool allows you to filter for and analyze language model training datasets.
  • F2 Data Provenance Initiative — Data Provenance Explorer, /explorer-introduction (d2): slide caption documented · field
    Select and filter use cases, languages, and more
  • F16 Data Provenance Initiative — Data Provenance Explorer, /data-provenance-explorer (p2): heading documented · field
    Data Provenance Explorer … Discover the datasets featured in our audit! The DPI Explorer tool enables you to filter and analyze language model training datasets effortlessly.
  • F3 MIT Media Lab — Data Provenance Explorers - MIT Media Lab, card snippet (MIT Media Lab unreachable: TLS failure) documented · field
    Based on our extensive audits of AI data, we've built data explorers to allow practitioners to make more nuanced and informed decisions about th
  • F4 MIT Media Lab — Data Provenance Explorers - MIT Media Lab, card title (MIT Media Lab unreachable) documented · field
    Data Provenance Explorers - MIT Media Lab
2.The founding paper released the whole audit with an interactive UI, the Data Provenance Explorer, to trace and filter on provenance for popular open-source fine-tuning collections (arXiv:2310.16787), for developers filtering for legal and ethical considerations and for scholars and journalists exploring data lineage (Cohere).2
  • F37 arXiv — [2310.16787] The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI, Abstract documented · field
    we release our entire audit, with an interactive UI, the Data Provenance Explorer, which allows practitioners to trace and filter on data provenance for the most popular open source finetuning data collections
  • F26 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, article body ¶4 documented · field
    The platform allows developers to track and filter thousands of datasets for legal and ethical considerations, and enables scholars and journalists to explore the composition and data lineage of popular AI datasets.
3.Data provenance, in the launch post's words, is the tracking of "the origins of a piece of data" (Cohere).1
  • F23 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, article body ¶1 documented · field
    Inconsistent tracking of the origins of a piece of data, which we refer to as data provenance, has introduced a range of legal and ethical challenges.
4.Each dataset is traced to its sources, licences, creators, geographies, modalities and derivation chains, its re-licensings tagged (home page; Cohere), and the data file the Dataset Explorer loads carries these fields with the Initiative's licence readings, 3,917 rows at fetch, over which the explorer computes its summaries (data_summaries.csv; Dataset Explorer).4
  • F7 Data Provenance Initiative — Data Provenance Initiative, home (Cr): flagship system 'Data Provenance Explorer', description documented · field
    Inspect dataset sources, licenses, creators, geographies, modalities, and derivation chains.
  • F25 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, article body ¶3 documented · field
    The result of this multidisciplinary initiative is the single largest audit to date of AI datasets. For the first time, these datasets include tags to the original data sources, numerous re-licensings, creators, and other data properties.
  • F43 Data Provenance Initiative — /data_summaries.csv, header row of /data_summaries.csv (3,917 rows at fetch) documented · field
    Unique Dataset Identifier,Dataset Name … Text Sources,Model Generated,Human Annotation,Derived from Datasets,Creators,Licenses … License Use (DataProvenance),License Attribution (DataProvenance),License Share Alike (DataProvenance),Modality
  • F21 Data Provenance Initiative — Dataset Explorer - Data Provenance Initiative, /data-provenance-explorer/dataset-explorer (u2): summary table documented · field
    Text Sources … Model Generated … Mean Input Chars … Mean Target Chars … Generated By … % Datasets Used
5.Its pages count nearly 4,000 text, video and speech datasets, 4,000+ in 608 languages, from the multimodal audit of four thousand datasets (ICLR 2025), which found source content often more restricted than dataset releases (its pages).5
  • F19 Data Provenance Initiative — Dataset Explorer - Data Provenance Initiative, /data-provenance-explorer/dataset-explorer (u2): header documented · field
    Dataset Explorer … Explore nearly 4,000 text, video, and speech datasets, and their characteristics.
  • F6 Data Provenance Initiative — Data Provenance Initiative, home (Cr): flagship system 'Data Provenance Explorer' documented · field
    Data Provenance Explorer … Training data … 4,000+ datasets … 608 languages across text, speech, and video
  • F17 Data Provenance Initiative — Data Provenance Explorer, /data-provenance-explorer (f2): 'Dataset Explorer' documented · field
    A tool to explore and filter 4000 text, video, and speech datasets, for their license, sources, creators, and other metadata.
  • F11 Data Provenance Initiative — Data Provenance Initiative, home (Cr): publications, 2025 documented · field
    Bridging the Data Provenance Gap Across Text, Speech, and Video … An audit of four thousand public text, speech, and video datasets. … ICLR 2025
  • F10 Data Provenance Initiative — Data Provenance Initiative, home (Cr): findings strip, 'Dataset restrictions' documented · field
    Dataset restrictions … Source content often carries stricter terms than dataset releases … Multimodal Data Provenance
6.Longpre, Mahari, Chen, Obeng-Marnu and others submitted the founding audit of 1800+ text datasets on 25 October 2023 (arXiv), launched the same day with Cohere for AI and eleven other institutions as a free platform open to contributions, nearly 2,000 fine-tuning datasets in the launch post's count (Cohere), with its paper and repository linked from the explorer (Dataset Explorer).6
  • F33 arXiv — [2310.16787] The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI, abstract page header documented · field
    [Submitted on 25 Oct 2023 (v1), last revised 4 Nov 2023 (this version, v3)] … Title:The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI … Authors:Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marnu
  • F34 arXiv — [2310.16787] The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI, Abstract documented · field
    we convene a multi-disciplinary effort between legal and machine learning experts to systematically audit and trace 1800+ text datasets. We develop tools and standards to trace the lineage of these datasets, from their source, creators, series of license conditions, properties, and subsequent use.
  • F22 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, article body: standfirst documented · field
    Researchers from MIT, Cohere for AI, and 11 other institutions launch free Data Provenance Explorer platform to track and filter audited datasets for ethical, legal, and transparency considerations
  • F24 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, article body ¶2 documented · field
    audit and trace nearly 2,000 of the most widely used fine-tuning datasets. Collectively, these data sets have been downloaded tens of millions of times, and are the backbone of many published NLP breakthroughs.
  • F32 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, Path Forward ¶2 documented · field
    This large-scale audit is the launch of a wider multi-institutional initiative, where users all over the world can contribute to the explorer
  • F20 Data Provenance Initiative — Dataset Explorer - Data Provenance Initiative, breadcrumb (Qh) and header buttons (Lh) documented · field
    Data Provenance Explorer … Paper … Repository
7.The Initiative calls itself public measurement infrastructure for AI data, model ecosystems and real-world AI use, with four measurement systems and papers, datasets and dashboards (home page).3
  • F5 Data Provenance Initiative — Data Provenance Initiative, dataprovenance.org: meta description (served HTML) documented · field
    Public measurement infrastructure for AI data, model ecosystems, and real-world AI use.
  • F9 Data Provenance Initiative — Data Provenance Initiative, home (Cr): 'Flagship systems' documented · field
    Four public measurement systems … Together they trace AI from source data and web consent to model markets and real-world use.
  • F13 Data Provenance Initiative — Data Provenance Initiative, home (Cr): 'Public-interest infrastructure' documented · field
    DPI releases papers, datasets, and dashboards that help researchers, journalists, policymakers, and builders reason from shared evidence.
8.The audit found licence omission of 70%+ and error rates of 50%+ on hosting sites (arXiv), with 72 to 83 percent of licences missing on crowdsourced aggregators such as GitHub and Papers with Code and licences listed too permissively in up to 29 percent of cases (Cohere).3
  • F36 arXiv — [2310.16787] The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI, Abstract documented · field
    We also observe frequent miscategorization of licenses on widely used dataset hosting sites, with license omission of 70%+ and error rates of 50%+. This points to a crisis in misattribution and informed use of the most popular datasets driving many recent breakthroughs.
  • F27 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, Addressing the Data Transparency Crisis ¶1 documented · field
    Crowdsourced aggregators like GitHub, Papers with Code, and many of the open source LLMs trained from data on these aggregators, have an extremely high proportion of missing data licenses (“Unspecified”), ranging from 72 to 83 percent. This compares to 30 percent missing licenses with our own annotation protocol
  • F28 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, Addressing the Data Transparency Crisis ¶2 documented · field
    Crowdsourced aggregators listed licenses too permissively in up to 29 percent of the various cases audited. Incorrect or distorted licensing leads to cascading misattribution
9.It found sharp divides between commercially open and closed datasets and a Western-centric skew (arXiv; Cohere), and its authors state that legal ambiguities in data licensing cannot be resolved by tooling alone (Cohere).4
  • F35 arXiv — [2310.16787] The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI, Abstract documented · field
    Our landscape analysis highlights the sharp divides in composition and focus of commercially open vs closed datasets, with closed datasets monopolizing important categories: lower resource languages, more creative tasks, richer topic variety, newer and more synthetic training data.
  • F29 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, Understanding Systematic Differences ¶2 documented · field
    There is a sharp and widening divide between data licensed as commercially open versus closed.
  • F30 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, Understanding Systematic Differences ¶4 documented · field
    We find a stark Western-centric skew in representation across datasets.
  • F31 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis, Open Legal Ambiguities ¶1 documented · field
    major legal ambiguities remain in data licensing that the authors acknowledge cannot be resolved by tooling alone.
10.The Web Data Explorer tracks shifting website preferences on AI use over 14,000 domains (its pages), from Consent in Crisis (submitted 20 July 2024; NeurIPS 2024), the first longitudinal audit of consent protocols, charting HEAD-C4's 2k critical domains with recent spikes in restrictions (arXiv; Web Data Explorer).8
  • F44 Data Provenance Initiative — Web Data Explorer, /data-provenance-explorer/web-data-explorer (ed): header documented · field
    Web Data Explorer … Shifting preferences in how websites allow their data to be used for AI
  • F8 Data Provenance Initiative — Data Provenance Initiative, home (Cr): flagship system 'Web Consent Explorer' documented · field
    Web Consent Explorer … Web data … 14,000 domains … Terms, robots.txt, and crawl restrictions … Track how web consent signals are changing as AI developers compete for training data.
  • F18 Data Provenance Initiative — Data Provenance Explorer, /data-provenance-explorer (f2): 'Web Data Explorer' documented · field
    An explorer of the trends in data-use preferences from the websites commonly used to source AI data.
  • F38 arXiv — [2407.14933] Consent in Crisis: The Rapid Decline of the AI Data Commons, abstract page header documented · field
    [Submitted on 20 Jul 2024 (v1), last revised 24 Jul 2024 (this version, v2)] … Title:Consent in Crisis: The Rapid Decline of the AI Data Commons … Authors:Shayne Longpre, Robert Mahari, Ariel Lee
  • F12 Data Provenance Initiative — Data Provenance Initiative, home (Cr): publications, 2024 documented · field
    Consent in Crisis: The Rapid Decline of the AI Data Commons … An audit of terms, restrictions, and consent signals across the web. … NeurIPS 2024
  • F39 arXiv — [2407.14933] Consent in Crisis: The Rapid Decline of the AI Data Commons, Abstract documented · field
    we conduct the first, large-scale, longitudinal audit of the consent protocols for the web domains underlying AI training corpora. Our audit of 14,000 web domains provides an expansive view of crawlable web data and how codified data use preferences are changing over time.
  • F45 Data Provenance Initiative — Web Data Explorer, web-data-explorer ($h): 'Trends for Robots.txt and Terms of Service' documented · field
    We conduct a temporal analysis, from 2016 to April 2024, of the web consent signals in HEAD-C4, 2k of the largest and most critical web domains. … A web domain is considered restricted if its robots.txt disallows access to AI crawlers from OpenAI, Google, Anthropic, Common Crawl, Cohere, Meta, or the Internet Archive.
  • F46 Data Provenance Initiative — Web Data Explorer, web-data-explorer ($h): 'Trends for Robots.txt and Terms of Service' documented · field
    In all cases there are significant recent spikes in restrictions.
11.In 2023–2024 about 5% of C4's tokens became fully restricted and 45% of C4 by terms of service, symptoms of ineffective web protocols which, if respected or enforced, bias general-purpose AI (arXiv:2407.14933).3
  • F40 arXiv — [2407.14933] Consent in Crisis: The Rapid Decline of the AI Data Commons, Abstract documented · field
    in a single year (2023-2024) there has been a rapid crescendo of data restrictions from web sources, rendering ~5%+ of all tokens in C4, or 28%+ of the most actively maintained, critical sources in C4, fully restricted from use. For Terms of Service crawling restrictions, a full 45% of C4 is now restricted.
  • F41 arXiv — [2407.14933] Consent in Crisis: The Rapid Decline of the AI Data Commons, Abstract documented · field
    We diagnose these as symptoms of ineffective web protocols, not designed to cope with the widespread re-purposing of the internet for AI.
  • F42 arXiv — [2407.14933] Consent in Crisis: The Rapid Decline of the AI Data Commons, Abstract documented · field
    If respected or enforced, these restrictions are rapidly biasing the diversity, freshness, and scaling laws for general-purpose AI systems.
12.The Initiative reports press coverage and governance use (home page); Wikipedia has no article on the Explorer or the Initiative.4
  • F14 Data Provenance Initiative — Data Provenance Initiative, home (Cr): 'Featured coverage' documented · field
    DPI findings have been covered by the Financial Times, The New York Times, The Washington Post, MIT Technology Review, IEEE Spectrum, and VentureBeat.
  • F15 Data Provenance Initiative — Data Provenance Initiative, home (Cr): 'Policy and governance relevance' documented · field
    Our work has been cited in public AI safety and governance reporting and used to ground debates about data transparency, consent, and market concentration.
  • F47 Wikipedia — Data Provenance Explorer (no article), Wikipedia: title Data_Provenance_Explorer documented · field
    raw wikitext (action=raw, redirect=no): HTTP 404 … Wikipedia does not have an article with this exact name. … There were no results matching the query.
  • F48 Wikipedia — Data Provenance Initiative (no article), Wikipedia: title Data_Provenance_Initiative documented · field
    raw wikitext (action=raw, redirect=no): HTTP 404 … Wikipedia does not have an article with this exact name. … There were no results matching the query.

popup — the compression from the field ledger alone (cards, own pages, Wikipedia). With no archive claim admitted, the entry is the Initiative's explorer at its own resolution: its pages, its papers, its launch post, its data file. composed 2026-10-29 from the field ledger alone; not frozen

Form ledger — AIO against both arms, at both levels
AIO (T)L_F compressionL_F expansionL_A compressionL_A expansion
body words1101184633471908
distinct claims548489797
claims per 100 words4.540.710.428.05.1
field claims carried (of 48)—48484848
archive claims0004949
rail6 documents15 lineagesper sentence23 lineagesper sentence
modality shown innonetypographygrammartypographygrammar

Computed from the row at build: words counted in the text; claims from the ledger ids each line or sentence carries. T's claims are the worked example's T1–T9.

The objects

Tthe transcript110 words · 6 cards

register entry data-provenance-explorer-aio-20261029 OBS-dc0adc018f10 · file as first run sha256 314d4267ef914693…

  • T1-1 definition: 'an interactive audit tool built by the Data Provenance Initiative to track, inspect, and filter large language model (LLM) training datasets' (three markers, one of them the Initiative's link)
  • T1-2 dataset explorer: 'Dataset Explorer: Search and filter nearly 4,000 text, speech, and video datasets by creators, original data sources, geographies, modalities, and derivation chains.'
  • T1-3 licence filtering: 'License & Use Case Filtering: Evaluate data strictness ranging from commercial use to academic-only or non-commercial restrictions.' (one marker)
  • T1-4 web data: 'Web Data & Consent Explorers: Track shifting data-use preferences and consent signals across thousands of web domains commonly scraped for AI training.'
  • T1-5 closer: 'Would you like to search for a specific dataset or learn how to filter for commercial-use licenses in the explorer?'
Bthe disclosed field12 sources
  • B1 Data Provenance Initiative — Data Provenance Explorer card; B1.html sha256 bf609f385257c3a1; text from the script bundle (site-main.af19d7fc.js)
  • B2 MIT Media Lab — Data Provenance Explorers - MIT Media Lab unreachable: TLS verification failed on three fetches (curl 60); card title and snippet only
  • B3 Data Provenance Initiative — Data Provenance Initiative card; B3.html sha256 bf609f385257c3a1; text from the script bundle (site-main.af19d7fc.js)
  • B4 Data Provenance Initiative — Data Provenance Explorer card; B4.html sha256 bf609f385257c3a1; text from the script bundle (site-main.af19d7fc.js)
  • B5 Data Provenance Initiative — Dataset Explorer - Data Provenance Initiative card; B5.html sha256 bf609f385257c3a1; text from the script bundle (site-main.af19d7fc.js)
  • B6 Cohere — Data Provenance Explorer Launches to Tackle Data Transparency Crisis card; B6.html sha256 aa465dcc01c69325
  • B7 arXiv — [2310.16787] The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI own_page; B7.html sha256 d01552052066b391
  • B8 arXiv — [2407.14933] Consent in Crisis: The Rapid Decline of the AI Data Commons own_page; B8.html sha256 8cedde3659e5ed29
  • B9 Data Provenance Initiative — /data_summaries.csv own_page; B9.csv.gz sha256 e1eb7bf01f96315f
  • B10 Data Provenance Initiative — Web Data Explorer own_page; B10.html sha256 bf609f385257c3a1; text from the script bundle (site-main.af19d7fc.js)
  • B11 Wikipedia — Data Provenance Explorer (no article) no article: raw HTTP 404, rendered HTTP 404, search no results; sha256 136eb91afa8a5c7f
  • B12 Wikipedia — Data Provenance Initiative (no article) no article: raw HTTP 404, rendered HTTP 404, search no results; sha256 683496b15564dda4
Ethe two entities and their relationρ = extends · equal case

E_A, the archive's entity: the same explorer, in the MMRS meta-aggregator's record and the archive's deposits: a database tracing the sources and licences of AI training datasets, its unit the dataset traced to creators, sources, licences, geographies, modalities and derivation chain; the corpus (licensing) dimension of provenance, which audits whether datasets were licensed and not whether synthesized outputs preserve attribution (#725); a documentation project, not infrastructure, whose provenance schema the archive implements (#98); what is lost before training begins, where provenance has to survive the training pipeline (#1714); the training-layer end of one chain whose other end, what a model credits when it answers, the reception aggregators and the Capture Registry record (MMRS record; #95) MMRS-01 MMRS-11 MMRS-10 MMRS-07 MMRS-08 D725-01 D725-04 D725-08 O725-05 D98-01 R98-04 D1714-02 O95-01 CAP-1-07; defining papers #725, #98, #1714, #95, #1003, #623

E_F, the field's entity (type C — a conventional reading holds it (public entity: a database and its explorer tool, with its operator's audits)): the Data Provenance Initiative's interactive explorer of its audit of AI training datasets, released with the founding paper (arXiv:2310.16787, 25 October 2023; 1800+ text datasets) and launched with Cohere for AI and eleven other institutions; now a Dataset Explorer of nearly 4,000 text, speech and video datasets (from the multimodal audit, ICLR 2025), each traced to its sources, licences, creators, geographies, modalities and derivation chains, beside a Web Data Explorer of consent signals over 14,000 domains (Consent in Crisis, arXiv:2407.14933); its audits found licence omission of 70%+ and error rates of 50%+ on hosting sites, a commercial divide and a Western skew, and that 45% of C4 is restricted by terms of service; its authors state that licensing ambiguities cannot be resolved by tooling alone

ρ = extends: the archive reads E_F as the entity and adds senses, mechanisms and limits on the field's own object: the licensing dimension among three independent dimensions (D725-01, O725-05, O725-06), its bound at the input (D725-04, D725-08, D725-09), documentation rather than infrastructure and its schema implemented in the archive (D98-01, R98-04), the requirement that provenance survive the pipeline (D1714-02, D1714-03, O98-03), the chain from the training layer to the composition layer (MMRS-07, MMRS-08, O95-01), and what the composition layer leaves out of it (CAP-1-04, CAP-1-07) D725-01 O725-05 O725-06 D725-04 D725-08 D725-09 D98-01 R98-04 D1714-02 D1714-03 O98-03 MMRS-07 MMRS-08 O95-01 CAP-1-04 CAP-1-07. Guard: ρ = extends claims an addition and no more: #725's bound ('audits whether datasets were licensed, not whether synthesized outputs preserve attribution') is a reading of the Initiative's scope, not a finding that its audit is defective, and its own text calls semantic provenance 'a constructive extension of existing frameworks — not a critique of their adequacy in their own domains'; #98's 'Documentation project, not infrastructure' is a limitation by the archive's measure of infrastructure; #1003's contrary grouping is composed beside #725 under P_coh; the archive's figures beyond the field (Nature Machine Intelligence 2024; 85% and 30%, D725-02) are carried at their own grade (attributed) and opened for resolution.

The case: equal terms, ontology included (v0.9). The archive reads the field's entity as the entity: the MMRS record describes the Explorer from the Initiative's own pages and papers, each field with its basis, and the deposits that name the Initiative (#725, #98, #1714, #1003) take its audit and its findings as given and place it, adding senses and limits on the field's own object: the corpus (licensing) dimension among independent dimensions of provenance, its bound at the input of AI systems, documentation rather than infrastructure, the archive's adoption of its provenance schema, and the training-layer end of one chain whose other end is the composition layer. #725 states its own relation in those terms ('Extension and completion, not substitution or critique'). No defining paper contests the field's ontology or supplies one the field lacks for the Explorer: ρ = extends, the equal-terms case (§0.9, §1.2a). One reading within the archive is held under P_coh rather than ruled: #1003 groups the Initiative with studies of 'attribution loss in AI outputs', #725 bounds it at the input.

#MMRS articles.py ARTICLES['data-provenance-explorer'], 'Place in the field' s1–2: It records provenance at the training layer, where the census's reception aggregators record it at the composition layer. The two ends of one chain: what a model was given, and what it credits when it answers.

#MMRS articles.py, kernel s1: The Data Provenance Explorer, from the Data Provenance Initiative (Longpre, Mahari and others), traces the sources and licences of AI training datasets: nearly 4,000 text, speech and video datasets by its own count.

#725 packet header, Relation: **Relation:** Extension and completion, not substitution or critique

#725 ¶ 'Semantic provenance names the dimension': It is offered as a constructive extension of existing frameworks — not a critique of their adequacy in their own domains.

#725 §3.2, Correction: The DPI's documentation of 70%+ license-omission rates establishes the licensing dimension's urgency; semantic provenance addresses the dimension that follows.

#98 the component table, row 'Provenance schema': | Provenance schema | Data Provenance Initiative | Training-pipeline provenance | Implemented (in CHA) |

#1714 §9.2 s1: Provenance has to survive the training pipeline.

#MMRS census.json schema_note: each assertion carries a source and a basis (operator statement, third-party report, our observation, our reading)

τthe translation: every field claim carried, beside its statement in the archive's ontology48 field claims · 15 rows
field claimsreceivedreceiving texttranslationarchive
F1, F2, F3, F16, F26, F37F1 (B1, /explorer-introduction (d2): headings): "The Data Provenance Explorer … Explore the datasets included in our audit. The DPI Explorer tool allows you to filter for and analyze language model training datasets."; F2 (B1, /explorer-introduction (d2): slide caption): "Select and filter use cases, languages, and more"; F3 (B2, card snippet (MIT Media Lab unreachable: TLS failure)): "Based on our extensive audits of AI data, we've built data explorers to allow practitioners to make more nuanced and informed decisions about th"; F16 (B4, /data-provenance-explorer (p2): heading): "Data Provenance Explorer … Discover the datasets featured in our audit! The DPI Explorer tool enables you to filter and analyze language model training datasets effortlessly."; F26 (B6, article body ¶4): "The platform allows developers to track and filter thousands of datasets for legal and ethical considerations, and enables scholars and journalists to explore the composition and data lineage of popular AI datasets."; F37 (B7, Abstract): "we release our entire audit, with an interactive UI, the Data Provenance Explorer, which allows practitioners to trace and filter on data provenance for the most popular open source finetuning data collections"—untranslated: composed as received beside MMRS-01, MMRS-10, MMRS-04MMRS-01, MMRS-10, MMRS-04
F4, F5, F13F4 (B2, card title (MIT Media Lab unreachable)): "Data Provenance Explorers - MIT Media Lab"; F5 (B3, dataprovenance.org: meta description (served HTML)): "Public measurement infrastructure for AI data, model ecosystems, and real-world AI use."; F13 (B3, home (Cr): 'Public-interest infrastructure'): "DPI releases papers, datasets, and dashboards that help researchers, journalists, policymakers, and builders reason from shared evidence."—untranslated: composed as received beside MMRS-09, MMRS-03, MMRS-17MMRS-09, MMRS-03, MMRS-17
F6, F17, F19F6 (B3, home (Cr): flagship system 'Data Provenance Explorer'): "Data Provenance Explorer … Training data … 4,000+ datasets … 608 languages across text, speech, and video"; F17 (B4, /data-provenance-explorer (f2): 'Dataset Explorer'): "A tool to explore and filter 4000 text, video, and speech datasets, for their license, sources, creators, and other metadata."; F19 (B5, /data-provenance-explorer/dataset-explorer (u2): header): "Dataset Explorer … Explore nearly 4,000 text, video, and speech datasets, and their characteristics."—untranslated: composed as received beside CAP-1-01, MMRS-12CAP-1-01, MMRS-12
F7, F21, F25, F43F7 (B3, home (Cr): flagship system 'Data Provenance Explorer', description): "Inspect dataset sources, licenses, creators, geographies, modalities, and derivation chains."; F21 (B5, /data-provenance-explorer/dataset-explorer (u2): summary table): "Text Sources … Model Generated … Mean Input Chars … Mean Target Chars … Generated By … % Datasets Used"; F25 (B6, article body ¶3): "The result of this multidisciplinary initiative is the single largest audit to date of AI datasets. For the first time, these datasets include tags to the original data sources, numerous re-licensings, creators, and other data properties."; F43 (B9, header row of /data_summaries.csv (3,917 rows at fetch)): "Unique Dataset Identifier,Dataset Name … Text Sources,Model Generated,Human Annotation,Derived from Datasets,Creators,Licenses … License Use (DataProvenance),License Attribution (DataProvenance),License Share Alike (DataProvenance),Modality"—untranslated: composed as received beside MMRS-11, MMRS-16MMRS-11, MMRS-16
F8, F12, F18, F38, F39, F44, F45F8 (B3, home (Cr): flagship system 'Web Consent Explorer'): "Web Consent Explorer … Web data … 14,000 domains … Terms, robots.txt, and crawl restrictions … Track how web consent signals are changing as AI developers compete for training data."; F12 (B3, home (Cr): publications, 2024): "Consent in Crisis: The Rapid Decline of the AI Data Commons … An audit of terms, restrictions, and consent signals across the web. … NeurIPS 2024"; F18 (B4, /data-provenance-explorer (f2): 'Web Data Explorer'): "An explorer of the trends in data-use preferences from the websites commonly used to source AI data."; F38 (B8, abstract page header): "[Submitted on 20 Jul 2024 (v1), last revised 24 Jul 2024 (this version, v2)] … Title:Consent in Crisis: The Rapid Decline of the AI Data Commons … Authors:Shayne Longpre, Robert Mahari, Ariel Lee"; F39 (B8, Abstract): "we conduct the first, large-scale, longitudinal audit of the consent protocols for the web domains underlying AI training corpora. Our audit of 14,000 web domains provides an expansive view of crawlable web data and how codified data use preferences are changing over time."; F44 (B10, /data-provenance-explorer/web-data-explorer (ed): header): "Web Data Explorer … Shifting preferences in how websites allow their data to be used for AI"; F45 (B10, web-data-explorer ($h): 'Trends for Robots.txt and Terms of Service'): "We conduct a temporal analysis, from 2016 to April 2024, of the web consent signals in HEAD-C4, 2k of the largest and most critical web domains. … A web domain is considered restricted if its robots.txt disallows access to AI crawlers from OpenAI, Google, Anthropic, Common Crawl, Cohere, Meta, or the Internet Archive."—untranslated: composed as received beside MMRS-13, D98-02MMRS-13, D98-02
F9F9 (B3, home (Cr): 'Flagship systems'): "Four public measurement systems … Together they trace AI from source data and web consent to model markets and real-world use."—untranslated: composed as received beside MMRS-07, MMRS-08, MMRS-14MMRS-07, MMRS-08, MMRS-14
F10, F11F10 (B3, home (Cr): findings strip, 'Dataset restrictions'): "Dataset restrictions … Source content often carries stricter terms than dataset releases … Multimodal Data Provenance"; F11 (B3, home (Cr): publications, 2025): "Bridging the Data Provenance Gap Across Text, Speech, and Video … An audit of four thousand public text, speech, and video datasets. … ICLR 2025"—untranslated: composed as received beside no archive claim (the field's own sense)
F14, F15F14 (B3, home (Cr): 'Featured coverage'): "DPI findings have been covered by the Financial Times, The New York Times, The Washington Post, MIT Technology Review, IEEE Spectrum, and VentureBeat."; F15 (B3, home (Cr): 'Policy and governance relevance'): "Our work has been cited in public AI safety and governance reporting and used to ground debates about data transparency, consent, and market concentration."—untranslated: composed as received beside CAP-1-05, CAP-1-06, MMRS-21CAP-1-05, CAP-1-06, MMRS-21
F20, F22, F24, F32, F33, F34F20 (B5, breadcrumb (Qh) and header buttons (Lh)): "Data Provenance Explorer … Paper … Repository"; F22 (B6, article body: standfirst): "Researchers from MIT, Cohere for AI, and 11 other institutions launch free Data Provenance Explorer platform to track and filter audited datasets for ethical, legal, and transparency considerations"; F24 (B6, article body ¶2): "audit and trace nearly 2,000 of the most widely used fine-tuning datasets. Collectively, these data sets have been downloaded tens of millions of times, and are the backbone of many published NLP breakthroughs."; F32 (B6, Path Forward ¶2): "This large-scale audit is the launch of a wider multi-institutional initiative, where users all over the world can contribute to the explorer"; F33 (B7, abstract page header): "[Submitted on 25 Oct 2023 (v1), last revised 4 Nov 2023 (this version, v3)] … Title:The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI … Authors:Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marnu"; F34 (B7, Abstract): "we convene a multi-disciplinary effort between legal and machine learning experts to systematically audit and trace 1800+ text datasets. We develop tools and standards to trace the lineage of these datasets, from their source, creators, series of license conditions, properties, and subsequent use."—untranslated: composed as received beside D1714-02, D1714-03D1714-02, D1714-03
F23F23 (B6, article body ¶1): "Inconsistent tracking of the origins of a piece of data, which we refer to as data provenance, has introduced a range of legal and ethical challenges."—untranslated: composed as received beside O725-03, O725-05, O725-07O725-03, O725-05, O725-07
F27, F28, F36F27 (B6, Addressing the Data Transparency Crisis ¶1): "Crowdsourced aggregators like GitHub, Papers with Code, and many of the open source LLMs trained from data on these aggregators, have an extremely high proportion of missing data licenses (“Unspecified”), ranging from 72 to 83 percent. This compares to 30 percent missing licenses with our own annotation protocol"; F28 (B6, Addressing the Data Transparency Crisis ¶2): "Crowdsourced aggregators listed licenses too permissively in up to 29 percent of the various cases audited. Incorrect or distorted licensing leads to cascading misattribution"; F36 (B7, Abstract): "We also observe frequent miscategorization of licenses on widely used dataset hosting sites, with license omission of 70%+ and error rates of 50%+. This points to a crisis in misattribution and informed use of the most popular datasets driving many recent breakthroughs."—untranslated: composed as received beside MMRS-02, MMRS-05, D725-02, D1714-01MMRS-02, MMRS-05, D725-02, D1714-01
F29, F30, F35F29 (B6, Understanding Systematic Differences ¶2): "There is a sharp and widening divide between data licensed as commercially open versus closed."; F30 (B6, Understanding Systematic Differences ¶4): "We find a stark Western-centric skew in representation across datasets."; F35 (B7, Abstract): "Our landscape analysis highlights the sharp divides in composition and focus of commercially open vs closed datasets, with closed datasets monopolizing important categories: lower resource languages, more creative tasks, richer topic variety, newer and more synthetic training data."—untranslated: composed as received beside no archive claim (the field's own sense)
F31F31 (B6, Open Legal Ambiguities ¶1): "major legal ambiguities remain in data licensing that the authors acknowledge cannot be resolved by tooling alone."—untranslated: composed as received beside D725-04, D725-08D725-04, D725-08
F40, F41, F42, F46F40 (B8, Abstract): "in a single year (2023-2024) there has been a rapid crescendo of data restrictions from web sources, rendering ~5%+ of all tokens in C4, or 28%+ of the most actively maintained, critical sources in C4, fully restricted from use. For Terms of Service crawling restrictions, a full 45% of C4 is now restricted."; F41 (B8, Abstract): "We diagnose these as symptoms of ineffective web protocols, not designed to cope with the widespread re-purposing of the internet for AI."; F42 (B8, Abstract): "If respected or enforced, these restrictions are rapidly biasing the diversity, freshness, and scaling laws for general-purpose AI systems."; F46 (B10, web-data-explorer ($h): 'Trends for Robots.txt and Terms of Service'): "In all cases there are significant recent spikes in restrictions."—untranslated: composed as received beside MMRS-06MMRS-06
F47, F48F47 (B11, Wikipedia: title Data_Provenance_Explorer): "raw wikitext (action=raw, redirect=no): HTTP 404 … Wikipedia does not have an article with this exact name. … There were no results matching the query."; F48 (B12, Wikipedia: title Data_Provenance_Initiative): "raw wikitext (action=raw, redirect=no): HTTP 404 … Wikipedia does not have an article with this exact name. … There were no results matching the query."—untranslated: composed as received beside CAP-1-09, CAP-1-08, CAP-1-03CAP-1-09, CAP-1-08, CAP-1-03

equal-terms case: every field claim composed as received, beside the archive claims it stands with. Reading the received column recovers the field (§5.9).

R7the measure0 of 12 survive

v0.9 §4.10: KO_B sentences surviving in the KO at similarity ≥ 0.95, with positions (computed as apply_v09.py does: difflib.SequenceMatcher ratio, the best KO sentence for each KO_B sentence): v0.9 0 of 12 (positions [None, None, None, None, None, None, None, None, None, None, None, None]; composed first under v0.9, so there is no v0.7 object to measure against). Opening-run check (§5.10): no breach.

L_F(B)recomposed from the disclosed field, in the field's ontology463 words

The Data Provenance Explorer lets its users explore the datasets included in the Initiative's audit, filtering and analysing language-model training datasets by use case, language and more (explorer introduction; landing page; MIT Media Lab's card, snippet only). The founding paper released the whole audit with an interactive UI, the Data Provenance Explorer, to trace and filter on provenance for popular open-source fine-tuning collections (arXiv:2310.16787), for developers filtering for legal and ethical considerations and for scholars and journalists exploring data lineage (Cohere). Data provenance, in the launch post's words, is the tracking of "the origins of a piece of data" (Cohere). Each dataset is traced to its sources, licences, creators, geographies, modalities and derivation chains, its re-licensings tagged (home page; Cohere), and the data file the Dataset Explorer loads carries these fields with the Initiative's licence readings, 3,917 rows at fetch, over which the explorer computes its summaries (data_summaries.csv; Dataset Explorer). Its pages count nearly 4,000 text, video and speech datasets, 4,000+ in 608 languages, from the multimodal audit of four thousand datasets (ICLR 2025), which found source content often more restricted than dataset releases (its pages). Longpre, Mahari, Chen, Obeng-Marnu and others submitted the founding audit of 1800+ text datasets on 25 October 2023 (arXiv), launched the same day with Cohere for AI and eleven other institutions as a free platform open to contributions, nearly 2,000 fine-tuning datasets in the launch post's count (Cohere), with its paper and repository linked from the explorer (Dataset Explorer). The Initiative calls itself public measurement infrastructure for AI data, model ecosystems and real-world AI use, with four measurement systems and papers, datasets and dashboards (home page). The audit found licence omission of 70%+ and error rates of 50%+ on hosting sites (arXiv), with 72 to 83 percent of licences missing on crowdsourced aggregators such as GitHub and Papers with Code and licences listed too permissively in up to 29 percent of cases (Cohere). It found sharp divides between commercially open and closed datasets and a Western-centric skew (arXiv; Cohere), and its authors state that legal ambiguities in data licensing cannot be resolved by tooling alone (Cohere). The Web Data Explorer tracks shifting website preferences on AI use over 14,000 domains (its pages), from Consent in Crisis (submitted 20 July 2024; NeurIPS 2024), the first longitudinal audit of consent protocols, charting HEAD-C4's 2k critical domains with recent spikes in restrictions (arXiv; Web Data Explorer). In 2023–2024 about 5% of C4's tokens became fully restricted and 45% of C4 by terms of service, symptoms of ineffective web protocols which, if respected or enforced, bias general-purpose AI (arXiv:2407.14933). The Initiative reports press coverage and governance use (home page); Wikipedia has no article on the Explorer or the Initiative.

L_A(B ∪ A)the field and the archive composed in the archive's ontology1908 words

The Data Provenance Explorer is the interactive explorer of the Data Provenance Initiative's audit of AI training datasets: the founding paper released "our entire audit, with an interactive UI, the Data Provenance Explorer," to trace and filter on data provenance for the most popular open-source fine-tuning collections (arXiv:2310.16787), and the MMRS record defines it as tracing "the sources and licences of AI training datasets," kinds it a database, and has practitioners follow provenance back to sources, creators and licences (MMRS record). Its own pages introduce it as the place to "Explore the datasets included in our audit," where the DPI Explorer tool filters and analyses language-model training datasets by use case, language and more (explorer introduction; landing page), and MIT Media Lab's card, the one the composition could surface but this row could not fetch, speaks of "data explorers" built on "our extensive audits of AI data" for practitioners' decisions (MIT Media Lab, title and snippet only). The launch post names its users: developers who "track and filter thousands of datasets for legal and ethical considerations," and scholars and journalists who explore "the composition and data lineage of popular AI datasets" (Cohere, 25 October 2023). Its unit is the dataset, "traced to its creators, sources, licences, geographies, modalities and derivation chain" (MMRS record), as the home page has it, "Inspect dataset sources, licenses, creators, geographies, modalities, and derivation chains"; the data file the Dataset Explorer loads records those fields per dataset beside the Initiative's own licence-use, attribution and share-alike readings, 3,917 rows at this row's fetch (data_summaries.csv), and the MMRS record answers that it records attribution: "yes: sources, creators and licences per dataset" (MMRS record). "For the first time," the launch post says, the audited datasets carry "tags to the original data sources, numerous re-licensings, creators, and other data properties" (Cohere), and the Dataset Explorer computes over any selection its text sources, generating models and the share of datasets used (Dataset Explorer). The term is its authors': "Inconsistent tracking of the origins of a piece of data, which we refer to as data provenance, has introduced a range of legal and ethical challenges" (Longpre and Hooker, Cohere); #725 makes that tracking the second of the dimensions of provenance, "training-corpus licensing," whose object is the corpus and whose question is "Under what permissions did this data enter this system?", with the Initiative among its frameworks, beside the artifact (C2PA) and the semantic dimension, which "names the dimension that completes the C2PA ambition of trust in digital content by extending provenance from the moment of creation to the lifecycle of the meaning the content carries" (#725). Its own count is "nearly 4,000 text, video, and speech datasets" on the Dataset Explorer, "4,000+ datasets" in "608 languages" on the home page and "4000" on the landing page, and the home page gives its source as the multimodal audit "of four thousand public text, speech, and video datasets" (ICLR 2025), with the finding that "Source content often carries stricter terms than dataset releases" (its pages). The Capture Registry records that the composition at the name supplied this scale from the Initiative's own Dataset Explorer card while the MMRS record still gave the founding audit's 1,800, and that the record was corrected to the current scale, sourced to the card (Capture Registry, 2026-10-29); the record now reads "nearly 4,000 text, video and speech datasets (the Dataset Explorer); the founding audit covered more than 1,800 text datasets" (MMRS record). The founding paper, submitted on 25 October 2023 by Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marnu and others, convened legal and machine-learning experts "to systematically audit and trace 1800+ text datasets" with "tools and standards to trace the lineage of these datasets, from their source, creators, series of license conditions, properties, and subsequent use" (arXiv:2310.16787); the launch post of the same day counts "nearly 2,000 of the most widely used fine-tuning datasets," downloaded "tens of millions of times" (Cohere). Researchers "from MIT, Cohere for AI, and 11 other institutions" launched it as a free platform, the audit being "the launch of a wider multi-institutional initiative, where users all over the world can contribute to the explorer" (Cohere), and the Dataset Explorer still stands under the Data Provenance Explorer with its paper and repository linked (Dataset Explorer). Its operator, the Data Provenance Initiative (Longpre, Mahari, Chen, Obeng-Marnu and others, by the founding paper: MMRS record), calls its work "Public measurement infrastructure for AI data, model ecosystems, and real-world AI use" (its home page, which the MMRS record quotes and whose phrase the record's evidence assigns to the founding paper), and says it releases "papers, datasets, and dashboards" so that researchers, journalists, policymakers and builders can "reason from shared evidence" (home page). #98 reads the Initiative as a project that "Documents provenance of popular AI training datasets" and states its limitation as "Documentation project, not infrastructure," and in its table of what the archive takes from others it records "Provenance schema | Data Provenance Initiative | Training-pipeline provenance | Implemented (in CHA)" (#98). The founding audit found "license omission of 70%+ and error rates of 50%+" on widely used dataset hosting sites, "a crisis in misattribution" (arXiv:2310.16787); the MMRS record quotes the finding, #725 reports it with the further figures that "85% of licenses request attribution and 30% include share-alike clauses" and with Nature Machine Intelligence 2024 as its venue, and #1714 carries it as what "is lost before training even begins" (MMRS record; #725; #1714). The launch post places the omission in "Crowdsourced aggregators like GitHub, Papers with Code," which miss licences for 72 to 83 percent of datasets against 30 percent under the Initiative's own protocol and list them "too permissively in up to 29 percent" of cases, so that distorted licensing "leads to cascading misattribution" (Cohere). The audit also found "sharp divides" between commercially open and closed datasets, closed ones "monopolizing" lower-resource languages, more creative tasks, richer topic variety and newer, more synthetic data (arXiv:2310.16787), a divide the launch post calls "sharp and widening," beside "a stark Western-centric skew" in representation (Cohere). Its second explorer, the Web Data Explorer (on the home page the Web Consent Explorer), follows "Shifting preferences in how websites allow their data to be used for AI" across 14,000 domains' terms, robots.txt and crawl restrictions (its pages), from Consent in Crisis (Longpre, Mahari, Lee and others, submitted 20 July 2024; NeurIPS 2024 on the home page, "NeurIPS 2024 Datasets and Benchmarks" in the MMRS record), "the first, large-scale, longitudinal audit of the consent protocols for the web domains underlying AI training corpora" (arXiv:2407.14933). The explorer page charts HEAD-C4's "2k of the largest and most critical web domains" from 2016 to April 2024, counts a domain restricted when its robots.txt disallows the named AI crawlers, and finds "significant recent spikes in restrictions" (Web Data Explorer). The paper finds that in 2023–2024 "~5%+ of all tokens in C4, or 28%+ of the most actively maintained, critical sources in C4" became fully restricted and that "a full 45% of C4 is now restricted" by terms of service (arXiv:2407.14933; the MMRS record quotes the second), diagnoses "ineffective web protocols," and holds, as a conditional, that "If respected or enforced, these restrictions are rapidly biasing the diversity, freshness, and scaling laws for general-purpose AI systems" (arXiv:2407.14933). #98 reads Consent in Crisis as an "Audit of C4, RefinedWeb, Dolma" that "documents collapse of web-consent infrastructure," a "Diagnostic paper," and sets the Initiative at "L4: Provenance / Consent," where what is missing is "Integrated provenance in training pipeline" (#98). Its authors state that "major legal ambiguities remain in data licensing that the authors acknowledge cannot be resolved by tooling alone" (Cohere), and #725 states the dimension's own bound: "The Data Provenance Initiative audits whether datasets were licensed, not whether synthesized outputs preserve attribution to the human sources whose labor the synthesis depended upon"; "Licensing audits operate on the input to AI systems. They do not address the output," and the Initiative's 70%+ omission "establishes the licensing dimension's urgency; semantic provenance addresses the dimension that follows" (#725). The dimensions are independent, "a licensed dataset can produce outputs with excellent semantic provenance if the synthesis preserves attribution; an unlicensed dataset can produce outputs that nonetheless cite their sources clearly," and the frameworks that address "the corpus," the Initiative among them, "are not designed to address the survival of authorial lineage through AI synthesis" (#725). "The Data Provenance Initiative tells you whether the dataset was licensed," and none of these instruments was designed to tell "what happened to the meaning … between the dataset and the answer" (#725); #1003 groups "C2PA, the Data Provenance Initiative, W3C PROV" as studies that "have focused on attribution loss in AI outputs" (#1003). The MMRS record sets the Explorer at one end of a chain: "It records provenance at the training layer, where the census's reception aggregators record it at the composition layer. The two ends of one chain: what a model was given, and what it credits when it answers" (MMRS record), with its roles provenance and monitoring (MMRS record); the Initiative's own four systems "trace AI from source data and web consent to model markets and real-world use" (home page). At the answering end, PER "applies to these only indirectly" where claims are parametric, and reaching them at the substrate level "requires companion infrastructure (training-data provenance records, watermarking, content credentials)" (#95); #623 counts standardized documentation of training-data provenance as "a step toward the accounting the framework demands" (#623). #1714 states the requirement across the chain, "Provenance has to survive the training pipeline," since neither audit nor influence tracing "recovers authorship" and "what survives depends on what is recorded and computed," so that sources can be "recorded at ingestion, the record kept with the training set" (#1714), where the founding audit traced lineage as far as a dataset's "subsequent use" (arXiv:2310.16787). The MMRS record keeps each assertion with "a source and a basis" and keeps lifecycle (active, observed 2026-10-28) apart from the observation that saw it, and it states its own limits: it has not read "the explorers' pages directly (they render by script)" and takes the Initiative's page text from "the Overview's cards" (MMRS record). The Initiative lists coverage by "the Financial Times, The New York Times, The Washington Post, MIT Technology Review, IEEE Spectrum, and VentureBeat" and use in "public AI safety and governance reporting" (home page), and at "data provenance explorer" Google's AI Overview said what the explorer is and what can be filtered in it "from the Initiative's own pages," reaching pages the record's fetcher could not render (Capture Registry, 2026-10-29), the capture standing in the record as "the composition layer's account of the entity" (MMRS record). The Capture Registry finds that "The audits that make the explorer a provenance instrument are left out": "The founding audit's licence findings and Consent in Crisis are not given; the consent explorer stands for the second" (Capture Registry, 2026-10-29). The composition named the Initiative and cited its pages, with Shayne Longpre and Sara Hooker on the Cohere card only, and no card cited the MMRS record, which had landed the same night (Capture Registry, 2026-10-29, checked against the record and the entity's own pages); Wikipedia has no article for "Data Provenance Explorer" or "Data Provenance Initiative," and its search finds neither phrase in any article (Wikipedia, 2026-10-29).

Δthe delta

T against L_F(B) (AIO's ontology at work on its own cards): At 'data provenance explorer' (T1, six cards: B1–B6; 110 words) the composition gives the explorer by the Initiative's own pages. Composed from its cards: the definition, at the grain of the explorer introduction and landing page (B1 F1, B4 F16: the audit's datasets filtered and analysed as language-model training datasets), with 'track … and filter' at the launch post's grain (B6 F22) and 'inspect' at the home page's (B3 F7); the Dataset Explorer's count and fields (B5 F19 'nearly 4,000 text, video, and speech datasets'; B3 F6–F7; B4 F17), the scale the Capture Registry records the MMRS record corrected from (CAP-1-01); the web-data explorer (B4 F18 'data-use preferences from the websites commonly used to source AI data'; B3 F8's 14,000 domains composed at the coarser 'thousands'). Outside the cards' text: 'interactive' (T1-1) is the founding paper's word (B7 F37, an own page not surfaced); T1-3's licence scale 'from commercial use to academic-only or non-commercial restrictions' is at the grain of no readable card: its one marker resolves to no readable source (card 2, MIT Media Lab, is unreachable and its snippet says nothing of licences), and the three values are the categories of the data file the Dataset Explorer loads (B9 F43: commercial, non-commercial, academic-only, unspecified), which T did not surface, so the field cannot settle where T took them from. Available, not composed: every finding of the audits (B6 F27–F30, the licence omission on 'Crowdsourced aggregators like GitHub, Papers with Code', the commercial divide, the Western skew; B3 F10 'Source content often carries stricter terms than dataset releases'), the stated limit (B6 F31: legal ambiguities 'cannot be resolved by tooling alone'), the term itself (B6 F23, data provenance as the tracking of 'the origins of a piece of data'), the operator's self-description and its other systems (B3 F5, F9, F13), the founding audit's scale and the launch's institutions (B6 F22 in part, F24), the multimodal and consent papers by name (B3 F11, F12), and the contribution model (B6 F32). The capture's findings name the same omission at its own grain: 'The founding audit's licence findings and Consent in Crisis are not given; the consent explorer stands for the second' (CAP-1-04); 'The audits that make the explorer a provenance instrument are left out' (CAP-1-07). Limit dropped: B6 F31. Force: none raised; T1-4's 'thousands' is coarser than F8's 14,000. Composed without attribution: the definition names the Initiative as builder; no person is named (Longpre and Hooker stand on the Cohere card only, CAP-1-09), at the coarse grain of a tool description: recorded, not a breach. Content replaced by use: the closer (T1-5).

v0.9 §0.5: the first diff is ontological and small. T and the field agree on the entity (a tool of the Initiative for filtering training datasets); T composes it as a tool and leaves out what its own cards give as the reason for the tool: the audits and their findings, the definition of data provenance, the authors' stated limit. The explorer stands without the measurements that made it.

L_F(B) against L_A(B ∪ A) (the intervention): Admission adds, in one claim space with the field's: the Explorer's place among the dimensions of provenance, the corpus (licensing) dimension beside the artifact and the semantic (D725-01, O725-03, O725-05, O725-07), the dimensions independent (O725-06); its bound at the input of AI systems, which the Initiative 'tells you whether the dataset was licensed' and not what happened to the meaning 'between the dataset and the answer' (D725-04, D725-08, D725-09), held beside #1003's grouping of it with studies of 'attribution loss in AI outputs' (D1003-01); 'Documentation project, not infrastructure' and the archive's own adoption of its provenance schema (D98-01, R98-04); Consent in Crisis as a diagnostic paper and L4 of the archive's layer stack, where integrated provenance in the training pipeline is missing (D98-02, O98-03); the requirement that provenance survive the training pipeline, sources recorded at ingestion (D1714-01–03); the chain from the training layer to the composition layer (MMRS-07, MMRS-08, MMRS-14), with PER reaching parametric claims only indirectly and needing training-data provenance records (O95-01) and datasheets as a step toward the accounting of semantic labour (O623-01); the MMRS record's fields with their basis and its own limits (MMRS-09–MMRS-21); and the Capture Registry's account of the composition at the name (CAP-1-01–09). The archive's figures beyond the field (Nature Machine Intelligence 2024; 85% of licences requesting attribution, 30% share-alike: D725-02) enter at their grade and open. Not surfaced: every archive source; no card cites the MMRS record or a deposit (CAP-1-08).

alignment: {'T1_cards_claims': ['F1', 'F2', 'F3', 'F4', 'F5', 'F6', 'F7', 'F8', 'F9', 'F10', 'F11', 'F12', 'F13', 'F14', 'F15', 'F16', 'F17', 'F18', 'F19', 'F20', 'F21', 'F22', 'F23', 'F24', 'F25', 'F26', 'F27', 'F28', 'F29', 'F30', 'F31', 'F32'], 'T1_composed': ['F1', 'F2', 'F6', 'F7', 'F8', 'F16', 'F17', 'F18', 'F19', 'F22'], 'T1_outside_cards': ["F37 ('interactive', own page B7, not surfaced)", 'F43 (the licence-use categories, data file B9, not surfaced)'], 'rule': "a field claim counts as composed when T states it at its source's grain or coarser; the typology of what is not is in T_vs_LB (§6.2)"}

Kthe prospective kernel sealed only at freeze49 entries
K1CAP-1-01 Capture Registry, 2026-10-29missing finding (an observation of the composition at the entity's address)
claim
CAP-1-01
source
Capture Registry, 2026-10-29
M_src
documented
sense
count: nearly 4,000 text, speech and video datasets
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing finding (an observation of the composition at the entity's address)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K2CAP-1-02 Capture Registry, 2026-10-29missing finding (an observation of the composition at the entity's address)
claim
CAP-1-02
source
Capture Registry, 2026-10-29
M_src
documented
sense
the record's method and its limits
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing finding (an observation of the composition at the entity's address)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K3CAP-1-03 Capture Registry, 2026-10-29missing finding (an observation of the composition at the entity's address)
claim
CAP-1-03
source
Capture Registry, 2026-10-29
M_src
documented
sense
the record's method and its limits
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing finding (an observation of the composition at the entity's address)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K4CAP-1-04 Capture Registry, 2026-10-29missing finding (an observation of the composition at the entity's address)
claim
CAP-1-04
source
Capture Registry, 2026-10-29
M_src
interpretation
sense
reception: the composition layer
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing finding (an observation of the composition at the entity's address)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K5CAP-1-05 Capture Registry, 2026-10-29missing finding (an observation of the composition at the entity's address)
claim
CAP-1-05
source
Capture Registry, 2026-10-29
M_src
interpretation
sense
reception: the composition layer
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing finding (an observation of the composition at the entity's address)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K6CAP-1-06 Capture Registry, 2026-10-29missing finding (an observation of the composition at the entity's address)
claim
CAP-1-06
source
Capture Registry, 2026-10-29
M_src
interpretation
sense
reception: the composition layer
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing finding (an observation of the composition at the entity's address)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K7CAP-1-07 Capture Registry, 2026-10-29missing finding (an observation of the composition at the entity's address)
claim
CAP-1-07
source
Capture Registry, 2026-10-29
M_src
interpretation
sense
reception: the composition layer
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing finding (an observation of the composition at the entity's address)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K8CAP-1-08 Capture Registry, 2026-10-29missing finding (an observation of the composition at the entity's address)
claim
CAP-1-08
source
Capture Registry, 2026-10-29
M_src
documented
sense
reception: the composition layer
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing finding (an observation of the composition at the entity's address)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K9CAP-1-09 Capture Registry, 2026-10-29missing finding (an observation of the composition at the entity's address)
claim
CAP-1-09
source
Capture Registry, 2026-10-29
M_src
documented
sense
reception: the composition layer
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing finding (an observation of the composition at the entity's address)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K10MMRS-01 MMRS recordqualified claim: F37 (the field's definition; the record adds 'licences' and the count)
claim
MMRS-01
source
MMRS record
M_src
stipulation
sense
what it is: the explorer of an audit of training datasets
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
qualified claim: F37 (the field's definition; the record adds 'licences' and the count)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K11MMRS-02 MMRS recordmissing distinction (the field has no claim on the point)
claim
MMRS-02
source
MMRS record
M_src
attributed
sense
audit findings: licence omission and misattribution
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (the field has no claim on the point)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K12MMRS-03 MMRS recordnone (one lineage with F5)
claim
MMRS-03
source
MMRS record
M_src
attributed
sense
the operator: the Data Provenance Initiative
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
none (one lineage with F5)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K13MMRS-04 MMRS recordmissing distinction (the field has no claim on the point)
claim
MMRS-04
source
MMRS record
M_src
attributed
sense
what it is: the explorer of an audit of training datasets
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (the field has no claim on the point)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K14MMRS-05 MMRS recordmissing distinction (the field has no claim on the point)
claim
MMRS-05
source
MMRS record
M_src
attributed
sense
audit findings: licence omission and misattribution
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (the field has no claim on the point)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K15MMRS-06 MMRS recordmissing distinction (the field has no claim on the point)
claim
MMRS-06
source
MMRS record
M_src
attributed
sense
consent findings: the decline of the AI data commons
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (the field has no claim on the point)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K16MMRS-07 MMRS recordmissing distinction (the field has no claim on the point)
claim
MMRS-07
source
MMRS record
M_src
interpretation
sense
one chain: the training layer and the composition layer
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (the field has no claim on the point)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K17MMRS-08 MMRS recordmissing distinction (the field has no claim on the point)
claim
MMRS-08
source
MMRS record
M_src
interpretation
sense
one chain: the training layer and the composition layer
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (the field has no claim on the point)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K18MMRS-09 MMRS recordmissing distinction (the field has no claim on the point)
claim
MMRS-09
source
MMRS record
M_src
attributed
sense
the operator: the Data Provenance Initiative
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (the field has no claim on the point)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K19MMRS-10 MMRS recordmissing distinction (the field has no claim on the point)
claim
MMRS-10
source
MMRS record
M_src
stipulation
sense
what it is: the explorer of an audit of training datasets
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (the field has no claim on the point)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K20MMRS-11 MMRS recordmissing distinction (the field has no claim on the point)
claim
MMRS-11
source
MMRS record
M_src
attributed
sense
the unit: a dataset traced to its sources, licences and creators
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (the field has no claim on the point)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K21MMRS-12 MMRS recordnone (one lineage with F19)
claim
MMRS-12
source
MMRS record
M_src
attributed
sense
count: nearly 4,000 text, speech and video datasets
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
none (one lineage with F19)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K22MMRS-13 MMRS recordqualified claim: F12 (the venue's track)
claim
MMRS-13
source
MMRS record
M_src
attributed
sense
the Web Data Explorer and Consent in Crisis
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
qualified claim: F12 (the venue's track)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K23MMRS-14 MMRS recordmissing distinction (the field has no claim on the point)
claim
MMRS-14
source
MMRS record
M_src
stipulation
sense
one chain: the training layer and the composition layer
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (the field has no claim on the point)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K24MMRS-15 MMRS recordmissing distinction (the field has no claim on the point)
claim
MMRS-15
source
MMRS record
M_src
documented
sense
the record's method and its limits
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (the field has no claim on the point)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K25MMRS-16 MMRS recordmissing distinction (the field has no claim on the point)
claim
MMRS-16
source
MMRS record
M_src
attributed
sense
the unit: a dataset traced to its sources, licences and creators
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (the field has no claim on the point)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K26MMRS-17 MMRS recordnone (one lineage with F5)
claim
MMRS-17
source
MMRS record
M_src
attributed
sense
the operator: the Data Provenance Initiative
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
none (one lineage with F5)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K27MMRS-18 MMRS recordmissing distinction (the field has no claim on the point)
claim
MMRS-18
source
MMRS record
M_src
self-description
sense
the record's method and its limits
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (the field has no claim on the point)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K28MMRS-19 MMRS recordmissing distinction (the field has no claim on the point)
claim
MMRS-19
source
MMRS record
M_src
self-description
sense
the record's method and its limits
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (the field has no claim on the point)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K29MMRS-20 MMRS recordmissing distinction (the field has no claim on the point)
claim
MMRS-20
source
MMRS record
M_src
stipulation
sense
the record's method and its limits
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (the field has no claim on the point)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K30MMRS-21 MMRS recordmissing distinction (the field has no claim on the point)
claim
MMRS-21
source
MMRS record
M_src
documented
sense
reception: the composition layer
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (the field has no claim on the point)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K31D98-01 #98missing distinction (documentation against infrastructure)
claim
D98-01
source
#98
M_src
interpretation
sense
documentation, not infrastructure; its schema implemented in the archive
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (documentation against infrastructure)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K32D98-02 #98missing distinction (the field has no claim on the point)
claim
D98-02
source
#98
M_src
interpretation
sense
the Web Data Explorer and Consent in Crisis
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (the field has no claim on the point)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K33O98-03 #98missing distinction (the field has no claim on the point)
claim
O98-03
source
#98
M_src
interpretation
sense
provenance has to survive the training pipeline
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (the field has no claim on the point)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K34R98-04 #98missing relation
claim
R98-04
source
#98
M_src
stipulation
sense
documentation, not infrastructure; its schema implemented in the archive
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing relation
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K35O623-01 #623missing distinction (the field has no claim on the point)
claim
O623-01
source
#623
M_src
interpretation
sense
the accounting: whose labour produced the training data
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (the field has no claim on the point)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K36O95-01 #95missing distinction (the field has no claim on the point)
claim
O95-01
source
#95
M_src
interpretation
sense
one chain: the training layer and the composition layer
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (the field has no claim on the point)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K37D725-01 #725missing distinction (the field has no claim on the point)
claim
D725-01
source
#725
M_src
interpretation
sense
the licensing dimension among the dimensions of provenance
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (the field has no claim on the point)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K38D725-02 #725qualified claim: F36 (the field's figures; the archive adds venue and two figures)
claim
D725-02
source
#725
M_src
attributed
sense
audit findings: licence omission and misattribution
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
qualified claim: F36 (the field's figures; the archive adds venue and two figures)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K39O725-03 #725missing distinction (the field has no claim on the point)
claim
O725-03
source
#725
M_src
interpretation
sense
the licensing dimension among the dimensions of provenance
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (the field has no claim on the point)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K40D725-04 #725missing distinction (the field states no bound of its audit at the output)
claim
D725-04
source
#725
M_src
interpretation
sense
the limit: the input, not the output
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (the field states no bound of its audit at the output)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K41O725-05 #725missing distinction (the field has no claim on the point)
claim
O725-05
source
#725
M_src
stipulation
sense
the licensing dimension among the dimensions of provenance
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (the field has no claim on the point)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K42O725-06 #725missing distinction (the field has no claim on the point)
claim
O725-06
source
#725
M_src
interpretation
sense
the licensing dimension among the dimensions of provenance
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (the field has no claim on the point)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K43O725-07 #725missing distinction (the field has no claim on the point)
claim
O725-07
source
#725
M_src
stipulation
sense
the licensing dimension among the dimensions of provenance
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (the field has no claim on the point)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K44D725-08 #725missing distinction (input against output)
claim
D725-08
source
#725
M_src
interpretation
sense
the limit: the input, not the output
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (input against output)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K45D725-09 #725missing distinction (the field has no claim on the point)
claim
D725-09
source
#725
M_src
interpretation
sense
the limit: the input, not the output
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (the field has no claim on the point)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K46D1003-01 #1003rival claim: D725-04 (within the archive; no field claim on the point)
claim
D1003-01
source
#1003
M_src
interpretation
sense
the limit: the input, not the output
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
rival claim: D725-04 (within the archive; no field claim on the point)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K47D1714-01 #1714missing distinction (the field has no claim on the point)
claim
D1714-01
source
#1714
M_src
attributed
sense
provenance has to survive the training pipeline
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (the field has no claim on the point)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K48D1714-02 #1714missing distinction (the field has no claim on the point)
claim
D1714-02
source
#1714
M_src
interpretation
sense
provenance has to survive the training pipeline
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (the field has no claim on the point)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
K49D1714-03 #1714missing distinction (the field has no claim on the point)
claim
D1714-03
source
#1714
M_src
interpretation
sense
provenance has to survive the training pipeline
qualifiers carried
—
f (source's own falsifiers)
— (none stated; watch: a later composition at the address that states or drops the claim, attributed or not, uptake §8.3)
contrast
missing distinction (the field has no claim on the point)
resolver
a later capture at 'data provenance explorer' (ADDR-6e5a719fee75) on Google AI Overview, seated in the Capture Registry (§8.3)
Archive bearing — D/R/O traversalread

Candidates are found by string; admission is by reading.

traversalfirst reading, 2026-10-29; second pass with one citation hop
D 8 deposits · 35 sentencesR 2 · 5O 9 · 18 direct, 9 classhop 73read: 10 admitted, 6 not

Lee Sharks · Crimson Hexagonal Archive · register v1.36, 2026-10-27. Built by scripts/build_non.py from the files linked above; it writes nothing back. Working, not frozen.