---
license: cc-by-4.0
language:
- en
- pt
- es
- de
pretty_name: CiteMe Citation Verification Benchmark v1
size_categories:
- n<1K
tags:
- citation-verification
- citation-hallucination
- fabricated-references
- academic-integrity
- multilingual
- text
configs:
- config_name: default
  data_files:
  - split: test
    path: corpus-v1.jsonl
---

# CiteMe Citation Verification Benchmark v1

A frozen multilingual benchmark for evaluating citation- and reference-verification
systems. It contains 340 labeled references with ground truth known by construction:

| Ground-truth class | Count | Construction |
|---|---:|---|
| `EXISTS_CORRECT` | 160 | Real works confirmed by DOI and title in OpenAlex and Crossref |
| `EXISTS_CORRUPTED` | 100 | Real works with exactly one controlled metadata corruption |
| `FABRICATED` | 80 | Constructed references whose invented titles had no close OpenAlex or Crossref match at build time |

The corpus contains no user submissions, bibliographies, documents, product analytics,
or other user data.

Canonical landing page:
https://citeme.app/datasets/citation-verification-benchmark

## Files

- `corpus-v1.jsonl` — 116 evaluation cases containing 340 labeled references.
- `corpus-v1.meta.json` — build timestamp, seed, counts, and discarded candidates.
- `SHA256SUMS.txt` — integrity hashes for the frozen corpus and metadata.
- `CITATION.cff` — machine-readable citation metadata.
- `LICENSE.md` — dataset license and source-attribution notes.

## Data structure

Each JSONL row is one evaluation case:

```json
{
  "case_id": "case-en-001",
  "shape": "single",
  "style": "harvard",
  "sloppy": false,
  "language": "en",
  "refs": [
    {
      "ref_id": "case-en-001-r1",
      "text": "Reference string presented to the system",
      "class": "EXISTS_CORRUPTED",
      "sub_label": "title",
      "seed_doi": "10.xxxx/example"
    }
  ]
}
```

`sub_label` is present only for corrupted references and identifies the one changed
field: `author`, `doi`, `journal`, `pages_volume`, `title`, or `year`.

## Coverage

- Languages: English (170 references), Portuguese (102), Spanish (51), German (17).
- Citation styles: APA (146), Vancouver (90), Harvard (66), MLA (24), ABNT (14).
- Input shapes: 102 single-reference cases and 14 bibliography bundles.
- Twenty-three cases include deliberately untidy hand-typed formatting.

## Construction and provenance

Correct references start from public OpenAlex metadata. Candidate DOI/title pairs are
confirmed against Crossref before inclusion. Corrupted references receive exactly one
deterministic mutation. Fabricated references are constructed from synthetic titles and
public bibliographic strings, then rejected if OpenAlex or Crossref returns a close title
match at build time.

The build seed is `20260723`. The seed makes sampling order and synthetic transformations
deterministic for a fixed upstream snapshot. Because OpenAlex and Crossref evolve, the
frozen files and their SHA-256 checksums—not a future API rerun—are the reproducibility
anchor for v1.

Source acknowledgements:

- OpenAlex: https://openalex.org — source data is made available under CC0.
- Crossref: https://www.crossref.org — public scholarly metadata used for DOI and title confirmation.

## Synthetic-data notice

Fabricated rows may combine public contributor and venue strings with invented titles.
They are synthetic negative test cases. They must never be interpreted as claims that a
named person authored, or a venue published, the constructed work.

## Limitations

- This is an evaluation corpus, not a prevalence sample of academic writing.
- Non-existence checks describe the OpenAlex/Crossref snapshot at build time.
- The 29 separately curated legitimate but unindexed works used in CiteMe's
  369-reference product evaluation are not part of `corpus-v1`.
- Some constructed references intentionally resemble realistic hand-typed errors.
- Future corrections or additions will receive a new version. Do not edit v1 in place.

## Load with Hugging Face Datasets

On the Hub this corpus is exposed as the `test` split: it is an evaluation set, so
there is deliberately no training split.

```python
from datasets import load_dataset

# From the Hub (uses the split declared in this card's metadata)
data = load_dataset("danielnichiata/citation-verification-benchmark", split="test")
```

Loading the frozen file directly works too. `data_files` splits are named by the
caller, so the name here is arbitrary and does not have to match the Hub split:

```python
data = load_dataset(
    "json",
    data_files="https://citeme.app/datasets/citation-verification-benchmark-v1/corpus-v1.jsonl",
    split="train",
)
```

## License

The CiteMe compilation, labels, and synthetic transformations are licensed under
Creative Commons Attribution 4.0 International. See `LICENSE.md`.

## Citation

```text
CiteMe. (2026). CiteMe Citation Verification Benchmark v1
(Version 1.0.0) [Data set]. Zenodo.
https://doi.org/10.5281/zenodo.21610297
```

The DOI above is the version DOI for v1 and resolves once the Zenodo record is
published. Canonical page:
https://citeme.app/datasets/citation-verification-benchmark
