Bayanat Labs Arabic Retrieval Starter Kit

free Arabic RAG evaluation dataset Arabic retrieval evaluation Arabic qrels hard negatives cross-lingual Arabic English retrieval Gulf Arabic questions

Key numbers

Counted by script from the released files. No MANIFEST.json exists for this release, which predates the open-data programme. Each key number is computed from a released file and pinned to its SHA-256 (also listed in SHA256SUMS.txt).

Downloads and identifiers

Get started

Get the files and score a retrieval run

git clone https://github.com/Bayanat-Labs/arabic-retrieval-starter-kit
# Recall@5, counting grades 2 and 3 as relevant (see the README)
trec_eval -l 2 -m recall.5 arabic-retrieval-starter-kit/v0.1/qrels.txt your-run.txt

What is released

Attribution: Bayanat Labs, Arabic Retrieval Starter Kit v0.1, https://www.bayanatlabs.com/research/arabic-retrieval-starter-kit

Intended uses

• Check that a retrieval or RAG pipeline, its scoring and its answer checks behave as expected before a full evaluation.
• Test dialect questions over formal sources, Arabic questions over English evidence, superseded versions and unanswerable questions.
• Learn or teach TREC qrels scoring with trec_eval, pytrec_eval or ir_measures.

Out of scope

• Ranking systems: the kit is built to test a pipeline, not to rank systems.
• Real-world evidence: the company, policies and numbers are fictional, and there is no customer or personal data.
• A sample of a Bayanat Labs commercial dataset.

Rights at a glance

Training, display, redistribution and synthetic-voice use are separate permissions.

Limitations

• Single-author judgments: the grades were written with the fixture, so no reviewer agreement is reported.
• Synthetic text for a fictional telecom company; it does not represent real enterprise documents.
• Small by design: results on the kit say nothing about performance at scale.

Sources and licences

No third-party content. Bayanat Labs wrote every passage, question and judgment for a fictional telecom company. Source details

How to cite

Cite as:

Bayanat Labs (2026). Arabic Retrieval Starter Kit (v0.1) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.23355781

• Concept DOI (cite all versions): 10.5281/zenodo.23355780
• Version DOI (this version): 10.5281/zenodo.23355781
• CITATION.cff

@dataset{bayanat2026retrievalkit,
  author    = {{Bayanat Labs}},
  title     = {Arabic Retrieval Starter Kit},
  version   = {v0.1},
  year      = {2026},
  month     = oct,
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.23355781},
  url       = {https://doi.org/10.5281/zenodo.23355781}
}

How we handle citations

Changelog

Corrections and removal requests

If something in this release should not be there, or is wrong, email hello@bayanatlabs.com. We handle removal requests within 30 days, and platform requests within 24 hours. How removal requests work

Full write-up: Arabic Retrieval Starter Kit on bayanatlabs.com

Related product: Bayanat Search