CiteMe Citation Verification Benchmark v1
A frozen, versioned corpus for evaluating whether systems can distinguish real academic references from corrupted and fabricated ones. The 340 labels are known by construction—not inferred from user submissions.
- labeled references
- 340
- ground-truth classes
- 3
- languages
- 4
- citation styles
- 5
What is in corpus-v1
Correct
EXISTS_CORRECTReal works whose DOI, title, authors, year, and venue were confirmed.
Corrupted
EXISTS_CORRUPTEDReal works with exactly one deterministic metadata corruption.
Fabricated
FABRICATEDConstructed citations whose invented titles had no close OpenAlex or Crossref match at build time.
Construction and provenance
Correct references begin with public OpenAlex metadata and are retained only after a DOI and title confirmation against Crossref. Corrupted references receive exactly one controlled mutation to the title, year, author, journal, DOI, or pages/volume field.
Fabricated references are synthetic negatives. Their generated titles were checked for close matches in both OpenAlex and Crossref when the corpus was built. The deterministic seed is 20260723.
The corpus covers English, Portuguese, Spanish, and German references in APA, ABNT, Vancouver, Harvard, and MLA styles, including deliberately untidy hand-typed variants and multi-reference bundles.
Scope and limitations
- • This is an evaluation corpus, not a prevalence estimate for academic writing.
- • Non-existence checks describe the OpenAlex/Crossref snapshot at build time.
- • The seed makes transformations deterministic; the frozen files and checksums—not mutable upstream APIs—are the reproducibility anchor.
- • The separately curated 29 legitimate but unindexed references used in the 369-reference product evaluation are not included in corpus-v1.
- • Future corrections or additions will be released as a new version; v1 files will not be edited in place.
Use and evaluate
from datasets import load_dataset
data = load_dataset(
"json",
data_files="https://citeme.app/datasets/citation-verification-benchmark-v1/corpus-v1.jsonl",
split="train",
)