Bayanat Labs Arabic Citation Check

Arabic citation benchmark reference checking DOI verification fabricated citations OpenAlex

Key numbers

Counted by script from the released files. Each key number is read from results/stats.json or results/baselines.json of the Hugging Face dataset at revision 72eee605981bf7dc33e9f09eb56b5f209d4c1da2 and pinned to its SHA-256; percentages are computed from those files and rounded half up to one decimal place, as the dataset card writes them. Code at tag v0.1.0, commit cecebb15ab5feffe889c736850d5e62f8e8fd2db. Space revision d333428c5469b5112c07cef9a317b811533976c5. MANIFEST.json lists every file with its size and SHA-256.

Downloads and identifiers

Get started

Load a config with the datasets library. The checker installs from GitHub at the v0.1.0 tag; the package is not on PyPI.

from datasets import load_dataset

judgments = load_dataset("Bayanat-Labs/arabic-citation-check", "judgments")
index = load_dataset("Bayanat-Labs/arabic-citation-check", "work_index")

pip install "git+https://github.com/Bayanat-Labs/arabic-citation-check@v0.1.0"
arabic-citation-check check --index works.jsonl "<citation>"

What is released

Attribution: Arabic Citation Check by Bayanat Labs, CC BY 4.0; work index CC0 1.0, from OpenAlex. https://huggingface.co/datasets/Bayanat-Labs/arabic-citation-check

Intended uses

• Test a citation checker on Arabic references, with the citation field alone and the work_index table as input.
• Compare a checker with the exact-DOI lookup and title match, class by class.
• Check references yourself with the command-line checker and a work index.

Out of scope

• Testing whether a paper supports a claim: the benchmark tests existence and attribution only.
• Estimating how often a model invents references: every citation string is rendered from templates.
• Reading the full field check's score as real-world performance: it shows that labels and checker agree on these formats.

Rights at a glance

Training, display, redistribution and synthetic-voice use are separate permissions.

Limitations

• Judgments and works are counted separately: 5,045 judgments over 2,240 works, and one work can have many judgments. Works and venues are counted as released: 851 venue ids carry 846 distinct names, because some venues share a name.
• The full field check was written together with the citation templates, so its score shows that labels and checker agree on these formats. It is not an estimate of how the checker does on references found in the wild.
• The lookup shares (67.7% and 72.4% on test) depend on the mix of classes in this benchmark, which is set by fixed quotas. Another mix gives other shares; the per-class rows on the dataset card do not depend on it.
• What can still pass: a citation still passes when it leaves a field out (shown as missing), gives initials or family names only, cuts the author list after an et-al marker, or has added text after the venue. The benchmark has no class for an added author, a different given name or extra title words; the full field check flags these, and those rules are covered by the package's unit tests, not by the scores.
• All citation strings are rendered from templates in four simplified styles, so the scores say nothing about how often any model invents references. Real reference lists are messier.
• OpenAlex coverage of Arabic journals is partial, and the two samples are not a census. Venue country and language labels are the source's and carry its errors. A venue country is known only for part of the venues, so 517 works have none.
• The verdict verified means the citation agrees with the OpenAlex record on the day it was read, so source errors pass through. Registration was checked; registry metadata was not compared. A DOI unregistered on the check date can be registered later.
• Fabricated records follow a small set of templates, and their family names recur across splits, so a system can recognise them from the text alone. Score this class with lookup-based systems.
• The splits dev, test and test_ood are all public, so a system can be tuned on them. To score your own checker, give it the citation field alone and the work_index table; the other fields hold the answer.
• A citation of a real work shows only that work's authors, as OpenAlex lists them. Fabricated records use given names joined to 24 invented family names that had no match among OpenAlex author and venue names when screened, and each is labelled in its fields. To correct or remove a record about you or your work, write to hello@bayanatlabs.com with the work_id or record_id.

Sources and licences

Two samples with a fixed seed from the OpenAlex API: works labelled Arabic with a DOI, and works with a DOI in journals in each Arab League country. The share of Arabic titles is set by design. Every DOI check is logged by host, status and date in results/doi_checks.jsonl. OpenAlex, Crossref and DataCite do not endorse this benchmark. Source details

How to cite

Cite as:

Bayanat Labs (2026). Arabic Citation Check, version 0.1.0. Bayanat Labs. https://huggingface.co/datasets/Bayanat-Labs/arabic-citation-check

• CITATION.cff

No DOI. Cite the URL in the citation above and the date you read it.

@misc{bayanatlabs2026arabiccitationcheck,
  author       = {{Bayanat Labs}},
  title        = {Arabic Citation Check, version 0.1.0},
  year         = {2026},
  month        = nov,
  howpublished = {\url{https://huggingface.co/datasets/Bayanat-Labs/arabic-citation-check}}
}

How we handle citations

Changelog

Corrections and removal requests

If something in this release should not be there, or is wrong, email hello@bayanatlabs.com. We handle removal requests within 30 days, and platform requests within 24 hours. How removal requests work

Related product: Bayanat Bench