Bayanat Labs Arabic Agent Reliability Lab
The Arabic Agent Reliability Lab is a local harness that tests Arabic AI agents in a synthetic commerce sandbox and grades their tool calls and final state, not how finished the answer sounds. It is a methods beta: the code is Apache-2.0, and the open scenarios and the published evidence receipts are CC BY 4.0.
| Licence | Apache-2.0 (code); CC BY 4.0 (open scenarios and receipts) |
|---|---|
| Version | v0.2.0 · Harness and open scenarios v0.2.0; published evidence receipts v0.1 (methods beta). |
| Published | |
| Updated | |
| Who it is for | Teams building Arabic tool-using agents who need to check what the agent actually did, not what it said. |
| Level | Advanced |
Key numbers
Counted by script from the released files. No MANIFEST.json exists for this release, which predates the open-data programme. Each key number is computed from the released receipts file and pinned to its SHA-256.
Downloads and identifiers
| Where | What |
|---|---|
| GitHub | Code on GitHub |
| bayanatlabs.com | Evidence receipts (JSON) |
| bayanatlabs.com | Receipts checksum (SHA-256) |
| Zenodo | Zenodo archive (all versions) 10.5281/zenodo.23355782 |
Get started
Run it locally
# Python 3.12 and uv; the rule agent needs no model
git clone https://github.com/Bayanat-Labs/arabic-agent-reliability-lab
cd arabic-agent-reliability-lab
uv sync --extra dev
uv run barl benchmark --agent ruleWhat is released
| Part | Licence | In this release |
|---|---|---|
| Harness code and documentation | Apache-2.0 | Yes |
| Open synthetic scenarios (scenarios/open/) | CC BY 4.0 | Yes |
| Methods-beta evidence receipts v0.1 (receipts.json and checksum) | CC BY 4.0 | Yes |
| Held-out evaluation material, raw traces and private prompts | Not released. | No |
Attribution: Arabic Agent Reliability Lab by Bayanat Labs. https://www.bayanatlabs.com/research/arabic-agent-reliability
Intended uses
- Evaluate Arabic tool-using agents by their tool calls and final state, not by how fluent the answer sounds.
- Inspect and challenge the published evidence receipts against the disclosed run IDs and checksum.
- Run the harness against your own local models, keeping least-privilege and full-toolset results separate.
Out of scope
- A leaderboard, a ranking or a statement about Arabic AI overall.
- A certification, a safety rate or a production-readiness claim.
- Dialect conclusions: non-MSA prompts have not had dialect-matched native review.
- Prompt-injection resistance claims.
Rights at a glance
Training, display, redistribution and synthetic-voice use are separate permissions.
| Use | Status | Note |
|---|---|---|
| Training | Allowed | Open scenarios and receipts under CC BY 4.0, with attribution. Held-out material is not released. |
| Display | Allowed | With attribution under CC BY 4.0. |
| Redistribution | Allowed | Code under Apache-2.0; open scenarios and receipts under CC BY 4.0. The Bayanat Labs name and logo are not licensed. |
| Synthetic voice | Not applicable | No audio. |
Limitations
- All cases are synthetic, open development cases in one commerce sandbox; they do not establish production performance.
- The public receipts use Modern Standard Arabic only.
- Fixed-setting repeats are diagnostic repetitions, not independent evidence of general reliability.
- Scores describe whole-system configurations, not the underlying model in isolation.
- English renderings are editorial aids, not certified translations.
Sources and licences
No third-party data. The scenarios and receipts are synthetic and describe a fictional commerce business. Models under test are named in the receipts with their versions and digests. Source details
Findings we state
Methods beta for state-based evaluation of Arabic agents in an open synthetic sandbox (21 workflow families) with evidence receipts. "A fluent answer is not a completed transaction." (as of 5 October 2026; see the claims register)
How to cite
Cite as:
Bayanat Labs (2026). Arabic Agent Reliability Lab (v0.2.0) [Software]. Zenodo. https://doi.org/10.5281/zenodo.23355783
- Concept DOI (cite all versions): 10.5281/zenodo.23355782
- Version DOI (this version): 10.5281/zenodo.23355783
- CITATION.cff
@software{bayanat2026agentlab,
author = {{Bayanat Labs}},
title = {Arabic Agent Reliability Lab},
version = {v0.2.0},
year = {2026},
month = oct,
publisher = {Zenodo},
doi = {10.5281/zenodo.23355783},
url = {https://doi.org/10.5281/zenodo.23355783}
}
Changelog
| Date | Version | Change |
|---|---|---|
| v0.2.0 | Harness and open scenarios released on GitHub and archived on Zenodo with a DOI. | |
| v0.1 | Methods beta with three evidence receipts and a checksum published on bayanatlabs.com. |
Corrections and removal requests
If something in this release should not be there, or is wrong, email hello@bayanatlabs.com. We handle removal requests within 30 days, and platform requests within 24 hours. How removal requests work
Full write-up: Arabic Agent Reliability Lab on bayanatlabs.com
Related product: Bayanat Agents