Bayanat Labs LabelBench
LabelBench is a reproducible audit of whether AI models can label Arabic data, by taxonomy, task and dialect region. This release publishes aggregate results, charts and a technical report under CC BY 4.0 and the Python code under Apache-2.0; the source datasets are not redistributed, so the code downloads each one from its original location.
| Licence | CC BY 4.0 (results, charts, report); Apache-2.0 (code) |
|---|---|
| Version | v0.1 · Benchmark and results v0.1; code release v0.1.1 (no changes to the benchmark, metrics or results). |
| Published | |
| Updated | |
| Who it is for | Data and ML teams deciding whether a model can pre-label Arabic data for their taxonomy and dialect mix. |
| Level | Intermediate |
Key numbers
Counted by script from the released files. No MANIFEST.json exists for this release, which predates the open-data programme. Each key number is computed from a released file and pinned to that file's SHA-256.
Downloads and identifiers
| Where | What |
|---|---|
| bayanatlabs.com | Results (CSV) |
| bayanatlabs.com | Results (JSON) |
| bayanatlabs.com | Checksums (SHA-256) |
| bayanatlabs.com | Technical report (PDF) |
| GitHub | Code on GitHub |
| PyPI | PyPI package |
| Zenodo | Zenodo archive (all versions) 10.5281/zenodo.23355784 |
Get started
Install the command-line tool
# Python 3.10 or later; installs the labelbench command
pip install bayanat-labelbenchWhat is released
| Part | Licence | In this release |
|---|---|---|
| Aggregate results (CSV and JSON) and checksums | CC BY 4.0 | Yes |
| Charts and the v0.1 technical report (PDF) | CC BY 4.0 | Yes |
| Code: the labelbench Python package | Apache-2.0 | Yes |
| Source datasets (MASSIVE 1.1 ar-SA, AraTox, IADD DART subset) | Not redistributed. Each keeps its own licence; see Sources. | No |
Attribution: Bayanat Labs, LabelBench. https://www.bayanatlabs.com/research/labelbench
Intended uses
- Test whether a model can pre-label your Arabic data on your own taxonomy and dialect mix before you trust it.
- Reproduce the v0.1 results: the code fetches each source at a pinned revision and checks its SHA-256.
- Cite or reuse the findings, charts and results files with attribution.
Out of scope
- A ranking of current frontier models: v0.1 tests small local models on fixed manifests.
- A rating of Bayanat Labs catalogue products.
- A source of Arabic training text: the release contains no source text.
Rights at a glance
Training, display, redistribution and synthetic-voice use are separate permissions.
| Use | Status | Note |
|---|---|---|
| Training | Not applicable | No source text or labels are released. |
| Display | Allowed | Results, charts and report under CC BY 4.0, with attribution. |
| Redistribution | Allowed | Results, charts and report under CC BY 4.0; code under Apache-2.0. Source datasets are not redistributed. |
| Synthetic voice | Not applicable | No audio. |
Limitations
- Small local models on fixed held-out manifests, not current frontier models.
- MASSIVE ar-SA is localized from English, so it cannot support a claim about native Saudi Arabic.
- IADD states no licence, so it is used for evaluation only and no IADD rows are published.
- Safe Automation Coverage for the generative runs was measured with tied-confidence structured output; calibrated confidence is future work.
Sources and licences
| Source | Licence | Redistributed here |
|---|---|---|
| MASSIVE 1.1, ar-SA locale (Amazon) | CC BY 4.0 | No |
| AraTox | CC BY 4.0 (dataset card) | No |
| IADD, DART subset | None stated; evaluation only | No |
The code downloads each source from its original location at a pinned revision and checks its SHA-256. Public LabelBench files contain aggregate scores and hashes, not source text. Details: DATA.md in the GitHub repository. Source details
Findings we state
On the fixed held-out manifest, a local language model (Qwen2.5 3B) reached 78% binary toxicity accuracy, 25.3% exact match on the seven-label taxonomy and 22.4% across five dialect regions (chance 20%). (as of 5 October 2026; see the claims register)
How to cite
Cite as:
Bayanat Labs Research (2026). Arabic Annotation Is Not Binary: LabelBench, a Reproducible Audit of Automated Data Labeling. Bayanat Labs, Technical report v0.1. https://www.bayanatlabs.com/research/labelbench
Bayanat Labs Research (2026). LabelBench (v0.1.1) [Software]. Zenodo. https://doi.org/10.5281/zenodo.23355785
- Concept DOI (cite all versions): 10.5281/zenodo.23355784
- Version DOI (this version): 10.5281/zenodo.23355785
- CITATION.cff
@techreport{bayanat2026labelbench,
title = {Arabic Annotation Is Not Binary: LabelBench, a Reproducible Audit of Automated Data Labeling},
author = {{Bayanat Labs Research}},
institution = {Bayanat Labs},
year = {2026},
month = jul,
number = {v0.1},
url = {https://www.bayanatlabs.com/research/labelbench}
}
@software{bayanat2026labelbenchcode,
author = {{Bayanat Labs Research}},
title = {LabelBench},
version = {v0.1.1},
year = {2026},
month = oct,
publisher = {Zenodo},
doi = {10.5281/zenodo.23355785},
url = {https://doi.org/10.5281/zenodo.23355785}
}
Changelog
| Date | Version | Change |
|---|---|---|
| v0.1.1 | Code archived on Zenodo with a DOI. No changes to the benchmark code, metrics or v0.1 results. | |
| v0.1.0 | Code released on GitHub and PyPI under Apache-2.0. | |
| v0.1 | Technical report, aggregate results, charts and checksums published on bayanatlabs.com. |
Corrections and removal requests
If something in this release should not be there, or is wrong, email hello@bayanatlabs.com. We handle removal requests within 30 days, and platform requests within 24 hours. How removal requests work
Full write-up: LabelBench on bayanatlabs.com
Related product: Bayanat Bench