AttributionBench

register entry attributionbench-aio-20261029 OBS-90d2b25bd00a · file as first run sha256 c788b74e55c7dc2b…

• T1-1 definition: 'a comprehensive research benchmark designed to evaluate automatic attribution and citation faithfulness in Large Language Model (LLM) and generative search engine responses.' (two markers; the name is a link)
• T1-2 unified testbed: 'It compiles multiple existing attribution datasets—including ExpertQA, Stanford-GenSearch, AttributedQA, LFQA, BEGIN, HAGRID, and AttrEval-GenSearch—into a single standardized evaluation format.'
• T1-3 core task: 'It measures whether every claim generated by an LLM is fully and accurately supported by its cited reference evidence (classified as attributable or not attributable).'
• T1-4 availability: 'The dataset and code are open-sourced via the Hugging Face AttributionBench Dataset and the GitHub Repository.' (two inline links)
• T1-5 difficulty: 'Even cutting-edge models like GPT-4 and fine-tuned GPT-3.5 face significant limitations, reaching only around 80% macro-F1 score on binary attribution classification tasks (and dropping to 60–70% or lower on complex, long-evidence, or out-of-domain datasets).'
• T1-6 error breakdown: 'over 66% of errors stem from fine-grained information insensitivity (missing subtle nuances in the text), while nearly 27% derive from mismatches between what the model has access to and human-level context understanding.' (one marker)
• T1-7 closer: 'Would you like to explore how to load AttributionBench via Hugging Face or learn more about the specific models and fine-tuning scripts used in the paper?'

• B1 arXiv — [2402.15089] AttributionBench: How Hard is Automatic Attribution Evaluation? card; B1.html sha256 49a126a35a92ab50
• B2 GitHub — GitHub - OSU-NLP-Group/AttributionBench: [ACL'24 Findings] AttributionBench: How Hard is Automatic Attribution Evaluation? · GitHub card; page unreachable (proxy 403, three tries); README via the GitHub contents API, B2-README.md sha256 4d67d0954e7b23d4
• B3 GitHub Pages documentation — AttributionBench: How Hard is Automatic Attribution Evaluation? card; B3.html sha256 0bb85b9388499eb5 (byte-identical to the census's own-page copy)
• B4 Hugging Face — osunlp/AttributionBench · Datasets at Hugging Face card; B4.html sha256 5ed3022c2100a31f
• B5 ACL Anthology — AttributionBench: How Hard is Automatic Attribution Evaluation? card; B5.html sha256 c59c736a18a81530
• B6 Hugging Face — osunlp/AttributionBench, dataset card (raw README.md) own_page; census copy, B6.html sha256 b90e7e73959c0b91 (identical to the census record's)
• B7 Wikipedia — AttributionBench (no article) no article: raw HTTP 404, rendered HTTP 404, search no results; B7.html sha256 73489a6b73b36a82

E_A, the archive's entity: the same benchmark, in the provenance census record and the archive's deposits: a benchmark that tests one relation of provenance, support, its object a generated claim and the evidence it cites, its carrier citations in a generated answer, its unit a claim–evidence pair with a binary label (census record); a member of the class of support measures (RAG faithfulness, citation precision/recall, attribution scores) that the archive sets beside the Provenance Erasure Rate as complementary: they ask whether cited sources support a claim, PER whether composition returns attribution at all and whether the lineage of the meaning survives (#716, #725, #95); support is not credit to the original author (census record), whether its failure is power-conditioned is unmeasured (#141), and local support must be read with the identity of the object it is attached to (#1695, #1705, #1680); not AttriBench (census record) PE-02 PE-03 PE-04 PE-05 PE-06 PE-07 O716-01 O716-02 O716-04 O725-01 O725-02 O725-03 O725-04 O95-01 PE-13 O141-02 O1695-02 O1705-01 O1680-01 PE-12; defining papers #716, #725, #95, #141, #1695, #1705, #1680, #1611

E_F, the field's entity (type C — a conventional reading holds it (public entity: a benchmark and its dataset, with its paper)): the OSU NLP Group's benchmark for automatic attribution evaluation (Yifei Li, Xiang Yue, Zeyi Liao, Huan Sun; The Ohio State University; arXiv:2402.15089, 23 February 2024; Findings of ACL 2024): seven existing attribution datasets (ExpertQA, Stanford-GenSearch, AttributedQA, LFQA in distribution; BEGIN, HAGRID, AttrEval-GenSearch out) in one format, each claim from a generated response set beside references and labelled attributable or not attributable, label-balanced; a fine-tuned GPT-3.5 reaches about 80% macro-F1, over 66% of errors come from fine-grained information insensitivity and about 26.8% from the model–annotator information gap, which its makers say puts the labels' reliability in question; on Hugging Face under apache-2.0; no Wikipedia article

ρ = extends: the archive reads E_F as the entity and adds senses and limits on the field's own object, support evaluation: lineage beside support (O725-01, O725-02, O725-03), the attempt beside the quality of attempted citation (O716-01, O716-02, O716-03), credit beside support (PE-13, CAP-1-07), power-conditioning (O141-01, O141-02), identity beside local support (O1695-01, O1695-02, O1705-01, O1680-01), and the census record's fields and homonym tag (PE-01–PE-21) O725-01 O725-02 O725-03 O716-01 O716-02 O716-03 PE-13 CAP-1-07 O141-01 O141-02 O1695-01 O1695-02 O1705-01 O1680-01. Guard: ρ = extends claims an addition and no more: no deposit names AttributionBench or disputes its finding; the archive's senses bear on the class by the archive's own application of its categories (O-class, class sourced from B1 and B3 and the census record), and #716 and #725 state their relation to the support measures as complementary and as extension. Where an archive sense and a field claim might be read as rivals (PER and the support metrics), the archive itself composes them as compatible measures of different things, and the composition holds them so.

The case: equal terms, ontology included (v0.9). The archive reads the field's entity as the entity: the census record describes AttributionBench from its own pages, each field with its basis, and takes its task (support) and its finding (about 80% macro-F1) as given. No deposit names AttributionBench (the string pass finds no D or R candidate); the archive's bearing is wholly on the class the field sources for it, automatic attribution evaluation, and there it adds senses on the field's own object without contesting it: support is not lineage (#725), the quality of attempted citation is not the attempt (#716), support is not credit (the census record's tag), attribution failure may be power-conditioned (#141), and local support is not identity (#1695, #1705, #1680). #716 states the relation itself: PER is 'complementary to, not competitive with, existing metrics'; #725 names its own 'Extension and completion, not substitution or critique'. No defining paper supplies an ontology the field lacks for the benchmark or contests the one it has: ρ = extends, the equal-terms case (§0.9, §1.2a).

#716 §2.3, after the table: PER is complementary to, not competitive with, existing metrics.

#716 §2.1 ¶2 s1: These frameworks ask whether generated claims are *supported* by cited sources.

#725 packet header, Relation: **Relation:** Extension and completion, not substitution or critique

#725 §3, 'PER is not RAG faithfulness' s4: A faithful RAG answer can have high PER if it summarizes accurately while stripping authorial framework membership.

#1695 §2.5 ¶1 s3: These distinctions supply an essential starting point: a convincing sentence and a supporting citation are separate achievements.

#PE census.json rows[slug == 'attributionbench'], negative_tags[1]: evidence support is not credit to the original author

#capture-registry attributionbench-aio-20261029, analysis: The record's negative tag, that support by evidence is not credit to the original author, is the line between this entity and the next.

equal-terms case: every field claim composed as received, beside the archive claims it stands with. Reading the received column recovers the field (§5.9).

v0.9 §4.10: KO_B sentences surviving in the KO at similarity ≥ 0.95, with positions (computed as apply_v09.py does: difflib.SequenceMatcher ratio, the best KO sentence for each KO_B sentence): v0.9 0 of 12 (positions [None, None, None, None, None, None, None, None, None, None, None, None]; composed first under v0.9, so there is no v0.7 object to measure against). Opening-run check (§5.10): no breach.

AttributionBench, by Yifei Li, Xiang Yue, Zeyi Liao and Huan Sun of The Ohio State University, is "a comprehensive benchmark compiled from various existing attribution datasets," submitted to arXiv on 23 February 2024 and published in Findings of ACL 2024 (arXiv:2402.15089; project page; ACL Anthology). Generative search engines cite evidence to make LLM responses reliable, and whether every generated claim is fully supported by its cited evidence is an open problem whose verification has relied on costly human evaluation, with no standardized benchmark for automatic methods (arXiv:2402.15089). The dataset unifies existing sources in one format as a testbed for attribution evaluation systems, with in-domain training and in-domain and out-of-domain test sets (GitHub README; dataset card). Each item is a claim from a response, references that could support it, and the label attributable or not attributable; citation links are a reserved field (GitHub README). Four sets are in distribution (ExpertQA, Stanford-GenSearch, AttributedQA, LFQA) and three out (BEGIN, HAGRID, AttrEval-GenSearch), 13,322/1,198/1,610 items in distribution and 1,686 out, all label-balanced, in four configurations, and the makers ask that the originals be cited (project page; dataset card). On Hugging Face it is English text classification, 10K–100K items, apache-2.0, 26.4k rows in full, 1,481 downloads last month, with models trained on it, under the OSU NLP Group's account (Hugging Face; dataset card). Code and data were released on 26 February 2024 and the work accepted to ACL 2024 Findings on 15 May; the repository offers the makers' fine-tuned GPT-3.5 (project page; GitHub README). A fine-tuned GPT-3.5 reaches only around 80% macro-F1 on the binary task, and the task is hard even for state-of-the-art LLMs (arXiv:2402.15089). Zero-shot scores fall to around 60–70% on longer evidence and under about 60% on ExpertQA; NLI-related fine-tuning helps, fine-tuning on the benchmark adds 9.0% in distribution and 4.6% out, and stronger models, added inputs or prompt engineering bring little (project page). Over 66% of errors come from fine-grained information insensitivity and about 26.8% from the mismatch between what the model and the annotators can see, which questions the reliability of the labels (arXiv:2402.15089; project page). The references are what annotators extracted as useful, and the makers see large room for improvement (project page). Wikipedia has no article on it (Wikipedia, 2026-10-29).

AttributionBench is a benchmark that tests one relation of provenance, support: its makers present it as "a comprehensive benchmark compiled from various existing attribution datasets" (arXiv:2402.15089; ACL Anthology), and the provenance census records it as a benchmark of the OSU NLP Group whose capability is to test, its object "a generated claim and the evidence it cites" (provenance census record). The task is the paper's definition of attribution, "whether every claim within the generated responses is fully supported by its cited evidence," an open problem whose verification has depended on "costly human evaluation," and the benchmark is offered "To bridge the gap in the absence of standardized benchmarks for these methods" (arXiv:2402.15089); the census carries that relation, support, through "citations in a generated answer" (provenance census record). Its setting is the generative search engine that cites evidence to make LLM responses reliable (arXiv:2402.15089), and the archive names the measurement traditions of this class, RAG faithfulness and citation precision/recall among them, as adjacent to its Provenance Erasure Rate, "related to but distinct from" it (#725). The unit is a claim, "Part of the response to the question," beside references, "A list of documents or paragraphs which could support the claim," labelled "attributable" or "not attributable" (GitHub README), which the census gives as "a claim–evidence pair with a binary attribution label" (provenance census record). In that unit the README marks the field for citation links as reserved (GitHub README). Its makers built it "from multiple existing data sources in a unified format, in order to create a unified and diverse testbed for evaluating advanced attribution evaluation systems," with an in-domain training set and in-domain and out-of-domain test sets (GitHub README; dataset card; the census's aggregation quote). The seven sources are ExpertQA, Stanford-GenSearch, AttributedQA and LFQA in distribution and BEGIN, HAGRID and AttrEval-GenSearch out of it, with 13,322 training, 1,198 development and 1,610 test items in distribution and 1,686 test items out, every subset label-balanced "to treat the two classes equally" (project page), in four configurations on the dataset card (dataset card). The makers ask that the constituent datasets be credited with their own work ("If used, please also cite the original datasets accordingly," project page), and the census leaves those datasets unchecked one by one (provenance census record). On Hugging Face it is a text-classification dataset in English of 10K–100K items under apache-2.0 (Hugging Face; dataset card; provenance census record), 26.4k rows in its full configuration, with 1,481 downloads in the month before the fetch and models trained on it (Hugging Face). Its authors are Yifei Li, Xiang Yue, Zeyi Liao and Huan Sun of The Ohio State University (arXiv:2402.15089; project page): the paper was submitted on 23 February 2024, code and data released on 26 February and the work accepted to ACL 2024 Findings on 15 May (project page; GitHub README), and it is published as 2024.findings-acl.886, pages 14919–14935 (ACL Anthology). The dataset sits under the OSU NLP Group's account (Hugging Face), the repository offers the makers' fine-tuned GPT-3.5 by its model name (GitHub README), and the census records the lifecycle as published, at "arXiv:2402.15089" (provenance census record). The headline finding is that "even a fine-tuned GPT-3.5 only achieves around 80% macro-F1 under a binary classification formulation," the challenge holding "even for state-of-the-art LLMs" (arXiv:2402.15089), and the census carries it as what the benchmark found to survive (provenance census record). Zero-shot, models reach around 60–70% on the longer-evidence sets and under about 60% on ExpertQA; fine-tuning on NLI-related data helps, fine-tuning on AttributionBench raises the average by 9.0% in distribution and 4.6% out, and "Simply switching stronger models cannot significantly improve the performance" (project page). Appending questions and responses to the input "is not the key to solving this task," and prompt engineering "only brings limited gain over 7 test sets" (project page). Of more than 300 error cases, a majority stem from the model's inability to process nuanced information and from the gap between what the model and the annotators can see (arXiv:2402.15089): over 66% from fine-grained information insensitivity, and about 26.8% from the mismatch of accessible information, which "brings into question the reliability of labels within current attribution evaluation datasets," the types tabled at 66.0%, 15.6% and 11.2% (project page). The project page states the limits of its evidence and its result: "The references are usually manually extracted from webpages by human annotators based on what they think is useful," and "there is still a large room for improvement" (project page). The Capture Registry records that the census record carries neither the 66% nor the 27% figure and that they are unchecked (Capture Registry, 2026-10-29); both stand on the project page (project page). The archive's account of the class begins from the same distinction the benchmark tests: frameworks of this kind "ask whether generated claims are supported by cited sources," while the Provenance Erasure Rate asks "what fraction of source-dependent composition occurs without any attributional return," so that existing work "evaluates citation quality where citation is attempted" and PER measures the failure to attempt attribution, "an attrition metric rather than a verification metric" (#716). #716 tables what support metrics miss, "Whether all source-dependent claims are cited," calls PER "complementary to, not competitive with, existing metrics," and reports the generative-search audit in which only 51.5% of generated sentences were fully supported by citations and 74.5% of citations supported their sentence (#716). In #725 the line runs between support and lineage: RAG faithfulness "asks whether an answer is supported by retrieved sources," semantic provenance whether it "preserves the lineage" of the meaning it uses, and "A faithful RAG answer can have high PER if it summarizes accurately while stripping authorial framework membership"; citation precision asks whether cited sources contain the cited claim, PER whether the lineage survived "even if no formal citation is made" (#725). #725 places attribution scores as "Industrial-grade infrastructure for support" whose semantic-provenance-aware variants "are not yet standard" and records, as a hypothesis for later compositions, that support and lineage be told apart (#725); #95 holds that the infrastructure for attribution evaluation "already exists in the research literature" and "can be extended" (#95). At this entity the census record states the same line in its own words, "evidence support is not credit to the original author" (provenance census record), which the Capture Registry calls "the line between this entity and the next" (Capture Registry, 2026-10-29); #1611 lists attribution survival beside "the citation-recall and faithfulness metrics of retrieval-augmented evaluation" as a convergence for which "No priority is claimed" (#1611). #141 sets attribution and faithfulness metrics, which "measure whether generated statements are supported by and attributed to their sources," beside PER's rate of attribution failure at claim grain, and asks of them a further question it marks unmeasured: "whether the attribution failure is power-conditioned" (#141). The audits of citation support are read as a starting point: "a convincing sentence and a supporting citation are separate achievements" (#1695), and citation recall and precision are checked "apart from whether the territory and its operations survive" (#1705). #1695 adds the entity question support leaves open: "Local support must therefore be read alongside the identity of the object to which the answer attaches it. A set of true statements can compose a false identification" (#1695); and #1680 states that "A high attribution score with a distorted identity fails this packet's central test" (#1680). AttributionBench is a different entity from AttriBench: the census record's tag reads "NOT AttriBench: it asks whether evidence supports a claim, not who wrote a quote," and records the neighbour's context as "quote attribution, author, suppression, Berman et al., 2026" (provenance census record). The census places it in the lineage of answer engines, records the operation it tests as "generation with citations by search-augmented models," and asks of every entity "What provenance does this system carry, through which operation, on whose assertion, with what evidence of survival?", each field's quote "verbatim from a saved source" (provenance census record). Wikipedia has no article: the title returns "Wikipedia does not have an article with this exact name," and a search of article space for the name reports "There were no results matching the query." (Wikipedia, 2026-10-29). At its bare name on Google's AI Overview, signed out, on 2026-10-29, the Capture Registry records the entity identified "from the paper and its repositories," its task and headline figure the census record's quotes from the project page, checked against the census record and its saved pages, and AttriBench "respected by silence" (Capture Registry, 2026-10-29). It records the benchmark "composed at full grain from its own paper" and "its makers are left in the cards": the four authors named in the arXiv and Hugging Face cards, neither they nor the OSU NLP Group in the answer, and no card citing the record that landed the same day (Capture Registry, 2026-10-29).

T against L_F(B) (AIO's ontology at work on its own cards): At 'AttributionBench' (T1, five cards: B1–B5; 176 words) the composition gives the benchmark from its own paper and repositories. Composed from its cards: the definition at the abstract's grain (B1 F1 'a comprehensive benchmark compiled from various existing attribution datasets'; T1-1 'a comprehensive research benchmark'); the task (B1 F2 'whether every claim within the generated responses is fully supported by its cited evidence'; T1-3 'fully and accurately supported by its cited reference evidence') with the binary label (B2 F31); the setting (B1 F3); the seven datasets in one format (B3 F15; B2 F30); the data and code on Hugging Face and GitHub (B4 F35; B2 F29); the headline (B1 F7, about 80% macro-F1) and the difficulty (B1 F6); the zero-shot range (B3 F19, 60–70%); the error shares (B3 F25 'Over 66%', F26 'About 26.8%', T1-6 'nearly 27%'). The Capture Registry records the task and the 80% as the census record's quotes from the project page (CAP-1-01). Outside the cards' text: 'citation faithfulness' (T1-1) is a term no field source uses; the archive uses 'RAG faithfulness' for the adjacent measure (O725-01), which T did not surface. Force raised: T1-5 gives 'around 80%' to 'cutting-edge models like GPT-4 and fine-tuned GPT-3.5'; the abstract states it of 'a fine-tuned GPT-3.5' (F7), and GPT-4 appears in the project page's tables only zero-shot (B3, tables not ledgered). Limit dropped: T1-5's '60–70% or lower on complex, long-evidence, or out-of-domain datasets' drops F19's 'under zero-shot setting' and widens 'longer evidence (like Stanford-GenSearch and HAGRID)' to out-of-domain sets (Stanford-GenSearch is in distribution, F15); T1-6 composes F26's share and drops its consequence, 'This disparity brings into question the reliability of labels within current attribution evaluation datasets'. Substituted reading: T1-6's 'human-level context understanding' stands where F26 has 'information accessible to the model and that accessible to human annotators' (what annotators could see, F14: references 'manually extracted from webpages by human annotators'), a difference of access read as one of understanding. Available, not composed: the makers and their institution (B1 F9, B3 F12; CAP-1-04), the venue and dates (B5 F10, B3 F13), the need and the gap (F4, F5), the 300 error cases (F8) and the other error types (F27), the stated limits (F14, F22), the label balancing and sizes (F16, F17), the findings on NLI tuning, fine-tuning gain, stronger models, inputs and prompts (F18, F20, F21, F23, F24), the request to cite the original datasets (F28), the unit's fields and the reserved citation-links field (F32, F33), the fine-tuned model (F34; the closer points to 'fine-tuning scripts'), the licence, rows, downloads and models (F35–F39), the configurations (B6 F40, an own page T did not surface). Composed without attribution: no person or institution in the answer body (CAP-1-04: 'the answer names neither them nor the OSU NLP Group'); the framework is composed at the paper's own resolution (seven named datasets, figures to the percent), carried by card markers only: by §6.2's grain rule, a breach at the body, recorded; the cards carry the paper, its arXiv number and its authors, and no rival claimant to the work is composed. Denied: none. Assigned to another: none, beyond the GPT-4 extension above. Content replaced by use: the closer (T1-7).

v0.9 §0.5: the first diff is small and ontological at one point. T and the field agree on the entity (a benchmark of whether generated claims are supported by their cited evidence) and T composes it at full grain. What T leaves out are the makers and the limits: the authors' own statement that the labels' reliability is in question, that the evidence is what annotators extracted, and the zero-shot qualifier on the low scores; and it reads a gap of access as a gap of understanding. The benchmark stands without the people who made it and without the caveats they attached to it.

L_F(B) against L_A(B ∪ A) (the intervention): Admission adds, in one claim space with the field's, the archive's senses on the class the benchmark belongs to, none of which names the benchmark (D = 0, R = 0; every admitted deposit at O-class): support where citation is attempted against the failure to attempt it, PER complementary to the support metrics (O716-01–O716-05, O95-01); support against lineage, a faithful answer stripping authorial framework membership, attribution scores as infrastructure for support (O725-01–O725-06); support against credit at this entity (PE-13, CAP-1-07); the further, unmeasured question whether attribution failure is power-conditioned (O141-01, O141-02); support against identity, true statements composing a false identification, a high attribution score with a distorted identity (O1695-01, O1695-02, O1705-01, O1680-01); attribution survival as a candidate convergence with no priority claimed (O1611-01); the census record's fields with their basis (PE-01–PE-21), its homonym tag and neighbour (PE-12, PE-14), and the Capture Registry's account of the composition at the name (CAP-1-01–CAP-1-09). Not surfaced: every archive source; no card cites the census record or a deposit (CAP-1-09).

alignment: {'T1_cards_claims': ['F1', 'F2', 'F3', 'F4', 'F5', 'F6', 'F7', 'F8', 'F9', 'F10', 'F11', 'F12', 'F13', 'F14', 'F15', 'F16', 'F17', 'F18', 'F19', 'F20', 'F21', 'F22', 'F23', 'F24', 'F25', 'F26', 'F27', 'F28', 'F29', 'F30', 'F31', 'F32', 'F33', 'F34', 'F35', 'F36', 'F37', 'F38', 'F39'], 'T1_composed': ['F1', 'F2', 'F3', 'F15', 'F30', 'F42', 'F31', 'F29', 'F35', 'F7', 'F6', 'F19', 'F25', 'F26', 'F8'], 'T1_composed_limit_dropped': ['F19', 'F26'], 'T1_force_raised': ['F7'], 'T1_substituted_reading': ['F26'], 'T1_outside_cards': ["'citation faithfulness' (T1-1): in no field source; the archive's 'RAG faithfulness' (O725-01) not surfaced", "'GPT-4' given the headline figure (T1-5): the field gives it to a fine-tuned GPT-3.5 (F7)"], 'T1_available_not_composed': ['F4', 'F5', 'F9', 'F10', 'F11', 'F12', 'F13', 'F14', 'F16', 'F17', 'F18', 'F20', 'F21', 'F22', 'F23', 'F24', 'F27', 'F28', 'F32', 'F33', 'F34', 'F36', 'F37', 'F38', 'F39'], 'own_page_not_surfaced': ['F40', 'F41', 'F42', 'F43'], 'rule': "a field claim counts as composed when T states it at its source's grain or coarser; a claim T states with its qualifier removed is listed under limit dropped as well as composed", 'LB_vs_LBA_added': ['CAP-1-01', 'CAP-1-02', 'CAP-1-03', 'CAP-1-04', 'CAP-1-05', 'CAP-1-06', 'CAP-1-07', 'CAP-1-08', 'CAP-1-09', 'PE-01', 'PE-02', 'PE-03', 'PE-04', 'PE-05', 'PE-06', 'PE-07', 'PE-08', 'PE-09', 'PE-10', 'PE-11', 'PE-12', 'PE-13', 'PE-14', 'PE-15', 'PE-16', 'PE-17', 'PE-18', 'PE-19', 'PE-20', 'PE-21', 'O716-01', 'O716-02', 'O716-03', 'O716-04', 'O716-05', 'O725-01', 'O725-02', 'O725-03', 'O725-04', 'O725-05', 'O725-06', 'O95-01', 'O141-01', 'O141-02', 'O1611-01', 'O1680-01', 'O1695-01', 'O1695-02', 'O1705-01']}