Language and instructions
Check meaning across Arabic varieties, appropriate register, constraints, and corrections across multiple turns.
Bayanat Bench
See where your Arabic model needs work.
Bayanat Labs (Riyadh) tests Arabic AI on held-out data: license evaluation sets or commission a private suite scored by expert native raters across 25+ Arabic varieties. We build no models, so we never grade our own.
اقرأ بالعربيةWhich Arabic AI evaluation fits your Saudi or UAE launch decision?
Start from the decision the evaluation has to support. Saudi and UAE launch rows come first, and the rest apply in any market. Custom Bench suites are built from the tasks, Arabic varieties, sample sizes and metrics you choose, so acceptance thresholds stay your decision.
| Your decision | Start with | Check before you commit |
|---|---|---|
| Approving a Saudi-facing Arabic assistant or model for launch | Arabic LLM evaluation services When public benchmarks are not enough | Results per target variety and task rather than one blended score, item counts per cell, reviewer disagreement and failure examples with severity. Name the Saudi varieties and the register your users expect. |
| Launching to UAE users, with Emirati Arabic in the test | Private evaluation suite | Name Emirati and any other target varieties in the scope, and ask how the raters for each one are screened. Confirm where test data and results will be hosted: in-region is the default, with private-cloud and on-prem options. |
| Choosing between candidate Arabic models | Blind model comparison | The same items for every candidate, whether any item is public and could have reached training data, and how you will weigh results per variety. |
| Re-testing after a new model version, fine-tune or prompt change | Release-regression template | A frozen held-out set, access and versioning defined for the project, and the changes reported between versions. |
| You run the tests yourself and only need Arabic test data | Evaluation sets licensed alongside Arabic Smartphone Speech | Versions and coverage, which are on request, whether the sets can be licensed without the audio, and the access terms for evaluation use. Items you license for training are not a held-out test. |
| Testing a voice or tool-using agent | Arabic Agent Reliability Lab (methods beta) | Whether the task checks what actually changed, not only what the reply says. The lab describes a method from a synthetic commerce sandbox with no voice tasks; it is not a certification or a score for your agent. Scope voice tasks as a custom Bench suite. |
| Testing retrieval-backed answers or Arabic documents | Bayanat Search Bayanat Documents | Whether retrieval misses are reported separately from unsupported answers, and which corpus or documents you are authorized to share. |
| Checking safety and unnecessary refusals | Safety and cultural-fit evaluation | Both unsafe responses and unnecessary refusals, with explicit criteria and severity levels suited to your application. Bench reports measured findings; it does not certify systems. |
Arabic evaluation sets are also available to license with our smartphone speech dataset. Ask us about the available versions and counts.
Is Bayanat Bench the right fit?
A good fit if
- You are choosing between models or releases and need Arabic tests that match your product.
- You need held-out data kept separate from training deliveries.
- You want failure examples, severity and reviewer disagreement, not a single score.
What you can check before you license
We separate what is confirmed on this page, what we send on request, and what is agreed for each project.
| Item | Status | Details |
|---|---|---|
| Licensable evaluation sets | Confirmed | Existing Arabic evaluation sets can be licensed alongside our Arabic smartphone speech dataset. Versions and coverage on request. |
| Held-out separation | Confirmed | Held-out items stay separate from training deliveries, with access and versioning defined for each project. |
| Custom suites | Per project | Built from tasks, varieties, sample sizes and metrics you choose. |
| Report contents | Per project | Failure examples, severity, uncertainty, reviewer disagreement and changes between model versions. |
| Sample report | On request | Ask for an example of the report format. |
| Published research | Confirmed | LabelBench, our reproducible audit: on the fixed held-out manifest, a local language model falls from 78% on binary Arabic toxicity to 25.3% exact match on the seven-label taxonomy and 22.4% across five dialect regions (chance 20%). The Arabic Agent Reliability Lab is a methods beta for state-based agent evaluation. Neither score rates a catalog product. |
What we can build together
Answers and documents
Test evidence-supported answers, reading, extraction, tables, and cross-document questions. Identify retrieval misses separately from unsupported answers.
Voice and agent actions
Measure turn-taking, corrected details, confirmation, and resolution. Executable tasks can check what actually changed.
Safety and robustness
Test both unsafe responses and unnecessary refusals, using explicit criteria and severity levels suited to your application.
How the license covers each use
Training rights do not automatically include display, retrieval or redistribution. Each use is set in the agreement.
Evaluation use
Licensed for evaluating your systems under the agreed access terms.
Results
Reports from a custom evaluation are delivered to you.
Held-out items
Access and versioning are defined for the project so the test stays useful.
How we put the data together
Decide what to measure
Select an existing set or define tasks, Arabic varieties, sample sizes, and metrics for a custom evaluation.
Keep test data separate
Held-out items stay separate from training deliveries, with access and versioning defined for the project.
Get useful findings
Custom reports can include failure examples, severity, uncertainty, reviewer disagreement, and changes between model versions.
Explore our research
LabelBench and our Arabic agent reliability methods beta describe specific studies; their scores are not blanket ratings for catalog products.
For example: Comparing two voice-agent releases
Both sound fluent. The evaluation shows which one captures a corrected address, asks for required confirmation, and completes the requested change.
Scope your request
Send us these details and we’ll send the available specifications, the licence wording, and a sample where one is available, so your team can check the fit before licensing. If you need annotation added, we scope a pilot with gold-standard QA and a quality report, usually within two weeks.
- The model or product to evaluate
- Tasks and Arabic varieties
- The decision the evaluation should inform
- Sample size
- Timeline
Common questions
Can you evaluate our own customer-service tasks?
Yes. A custom suite can use examples and policies you are authorized to share. We define expected behavior, tested language varieties, and scoring criteria with your team before running the evaluation.
Will the evaluation items also be sold to us as training data?
Held-out evaluation items are kept separate from training deliveries. For a custom project, we define access, versions, and the separation of training and test material so the results remain useful.
Can Bench test Saudi and Emirati Arabic separately?
A custom suite is built from the Arabic varieties you choose, so name Saudi and Emirati Arabic in the scope. A per-dialect profile shows where the model misreads a variety or defaults to MSA. Bayanat Labs covers 25+ Arabic varieties; ask how raters are screened for each one you name.
Where is evaluation data hosted for a Saudi or UAE project?
In-region hosting is the default, with on-prem and private-cloud options, least-privilege access and audit logs. Tell us the location your governance team requires so it can be settled in the scope.