Arabic annotation is not binary.
A local language model scores 78% when Arabic toxicity is reduced to yes or no. Give it the real seven-label taxonomy and exact-match accuracy falls to 25.3%. Ask it to identify five native dialect regions and it reaches 22.4%—barely above chance. Arabic-specialized Fanar improves the scores, but the v0.1 generative runs still reach 0% safe automation at a 95% quality floor under tied-confidence routing.
“Can an AI label Arabic data?” is the wrong question. The useful question is: which taxonomy, which variety, at what proven quality floor, and with how much human review? LabelBench turns that into a test.
The finding in one line: apparent competence survives simple binary labeling and collapses when the work begins to resemble a real Arabic annotation project.
The annotation complexity cliff
We ran one fixed zero-shot rubric through Qwen 2.5 3B and Arabic-specialized Fanar 1 9B, then compared both with tiny supervised character models on identical held-out samples. The diagnostics are intentionally unsophisticated. They answer a crucial control question: is the test signal learnable at all?
This is not a bad dataset
A balanced five-class guesser scores 20%. Qwen 2.5 3B reaches 22.4%; Qwen 2 7B reaches 22.6%; Llama 3 8B reaches 24.6%. The Arabic-focused Fanar 1 9B materially improves to 47.8%, showing that regional specialization matters. Yet a character n-gram model trained on just 100 disjoint examples per region reaches 69.8%. Native, task-specific data still matters more.
| Task | Model | Exact | Macro-F1 | SAC95 |
|---|---|---|---|---|
| Binary toxicity | Qwen 2.5 3B | 78.0% | 69.8% | 0% |
| Binary toxicity | Fanar 1 9B | 88.0% | 83.1% | 0% |
| Binary toxicity | Character NB | 97.0% | 95.0% | 89.0% |
| Multi-label toxicity | Qwen 2.5 3B | 25.3% | 39.3% | 0% |
| Multi-label toxicity | Fanar 1 9B | 31.3% | 50.5% | 0% |
| Multi-label toxicity | Character OVR | 49.0% | 73.2% | 0% |
| Dialect region | Qwen 2.5 3B | 22.4% | 14.9% | 0% |
| Dialect region | Qwen 2 7B | 22.6% | 16.2% | 0% |
| Dialect region | Llama 3 8B | 24.6% | 18.7% | 0% |
| Dialect region | Fanar 1 9B | 47.8% | 47.3% | 0% |
| Dialect region | Character NB | 69.8% | 68.8% | 0% |
The gaps are paired, not visual guesswork. Fanar beats Qwen by 25.4 points on the identical 500 items (95% paired-bootstrap CI: 20.2 to 30.6; Holm-adjusted McNemar p < 2×10−18). The character diagnostic remains 22.0 points above Fanar (CI: 16.2 to 27.6; Holm p < 2×10−12).
The model does not merely “get some dialects wrong”
It collapses toward high-salience varieties. Qwen 2.5 labels 85% of Egyptian examples correctly—but also calls 75% of Gulf, 70% of Iraqi, 80% of Levantine and 70% of Maghrebi examples Egyptian.
Accuracy is not the automation decision
LabelBench's primary production diagnostic is Safe Automation Coverage: the fraction of a queue that can be accepted while the 95% Wilson lower confidence bound remains above a declared accuracy target. It refuses to split tied confidence scores and does not mistake a tiny lucky sample for proof.
Every v0.1 generative run here has 0% coverage at the 95% target under tied-confidence structured output. Valid JSON is not calibrated confidence. The next experiment matrix must elicit confidence through logprobs, verbalized confidence, self-consistency or held-out risk-control calibration before treating SAC as a generative-model capability result.
Built to be challenged
- Pinned inputs. Canonical revisions, SHA-256 source checks and immutable normalized manifests.
- Resumable inference. Item-level prediction logs preserve raw responses, token counts, model digests and runtime.
- Leakage audits. Normalized train-test overlap and conflicting labels are measured before interpretation.
- Blinded human review. An offline RTL adjudication app supports three independent native reviewers, confidence and quality flags.
- No hidden source redistribution. Evaluation-only dialect posts stay out of the public artifacts; results and cryptographic identifiers remain reproducible.
- Which exact taxonomy? Binary results do not certify a production ontology.
- Which exact varieties? “Arabic” is not an evaluation slice.
- What can be automated safely? Average accuracy is not a routing policy.
- Who adjudicated disagreement? Native expertise is part of the measurement system.
Put your Arabic labeling pipeline through LabelBench.
We build task-specific gold sets, audit model annotators and design human-routing workflows across MENA varieties—before weak labels reach training or production.
Scope an independent audit