Human-in-the-loop Arabic annotation: should you auto-label Arabic data?

Should you auto-label Arabic data? Only where you have proven it is safe: automate by task, by Arabic variety and against a quality floor you declared in advance and measured on your own gold set. Everything else stays in a human-in-the-loop Arabic annotation workflow, where model pre-labels are routed to native reviewers by confidence, disagreement and dialect triggers. For most real Arabic taxonomies today, models help triage and flag, and humans still decide.

Our LabelBench audit asks whether AI can label Arabic data. This page turns those measurements into a decision table, a routing policy, a measurement plan and a cost model.

Human-in-the-loop Arabic annotation: automate by task, variety and quality floor

"Can the model label our data?" is too coarse a question. Only automate a slice when three answers come back yes:

1. Task. Is the label set small, closed and objective (toxic or not, a closed list of intents)? Multi-label, graded or interpretive labels (a seven-label harm taxonomy, sarcasm, stance) are much harder to automate.
2. Variety. Has the model been measured on the variety in front of it? A result on MSA or Egyptian Arabic says little about Najdi, Iraqi or Moroccan text. Our guide to Arabic dialect identification explains why models lean toward the most visible varieties.
3. Quality floor. Have you declared a target, such as 95% accuracy on accepted labels, and shown with a confidence bound rather than a point estimate that the accepted slice clears it?

A slice that fails any test stays human-labeled.

What did LabelBench find about LLM pre-labeling in Arabic?

LabelBench v0.1 (run July 2026) ran one fixed zero-shot rubric through local language models and compared them with small supervised character models on identical samples. On the fixed held-out manifest:

Three findings shape a routing policy. First, the complexity cliff: a local language model at 78% on binary toxicity falls to 25.3% on the real seven-label taxonomy. Second, dialect collapse: Qwen 2.5 3B predicts Egyptian for 70% to 80% of items from every other region, so its Gulf recall is 4%. The Arabic-specialized Fanar 1 9B does far better at 47.8%, but sits 22.0 points below a small supervised character baseline. Third, every v0.1 generative run reached 0% Safe Automation Coverage at a 95% floor under tied-confidence routing.

These results cover the evaluated local models, not frontier APIs and not all LLMs. The zero SAC diagnoses the v0.1 confidence protocol; it does not prove that generative models cannot support selective automation. They do show that a binary score does not certify a production taxonomy.

Why accuracy is not the automation decision

Look at the binary row again. Fanar 1 9B reached 88.0% accuracy and 0% coverage; the character baseline reached 97.0% and 89.0% coverage. The baseline outputs a graded probability, so its most confident predictions can be accepted and the rest routed. The generative runs emitted categorical confidence, so large blocks of predictions tied at the same score, and SAC accepts a tied block whole or not at all.

First lesson: a routing policy needs a routing signal. Valid JSON is not calibrated confidence. Xiong et al. (ICLR 2024) found that LLMs verbalizing their confidence tend to be overconfident. Consistency across multiple sampled answers helped, but no method won consistently, and all struggled on tasks needing professional knowledge. Token log-probabilities, self-consistency and agreement between independent models are candidate signals; trust none until it is checked against gold labels.

Second lesson: small samples lie. SAC uses the lower end of a Wilson score interval, which Brown, Cai and DasGupta recommend over the textbook Wald interval when samples are small. The bound punishes lucky streaks:

• 99 correct out of 100 accepted labels gives a lower bound of about 94.5%. That fails a 95% floor.
• 196 out of 200 (98% observed) gives about 94.97%. Still a fail.
• 73 out of 73 is the smallest perfect record that clears 95%.

SAC is a practical diagnostic, not a guarantee. For a formal bound on the error of an automated slice, look at conformal risk control (Angelopoulos et al.). To justify replacing human annotators outright, use the Alternative Annotator Test (Calderon, Reichart and Dror, 2025), which was designed for that claim.

How to design an LLM annotation routing policy

A routing policy decides, item by item, whether a pre-label is accepted or sent to a person. Apply these gates in order; an item goes to a human the moment any gate fires.

1. Hard rules. Gold-set items, safety and red-team labels, medical, legal and financial judgments, and any label your guideline marks high-risk are always human. No confidence score overrides this gate.
2. Variety gate. Route anything outside the varieties you have measured. Cheap lexical cues catch many cases. "I want" is أبغى (abgha) in Gulf Arabic, عايز (ʿāyiz) in Egyptian, بدي (baddi) in Levantine and بغيت (bghit) in Moroccan. A model proven on one is not proven on the others.
3. Script gate. Arabizi and code-switched text go to reviewers who read it fluently: "3ayez" writes ع as the digit 3, and "7abibi" writes ح as 7.
4. Confidence gate. Accept only above a threshold chosen on a calibration split and confirmed on a separate evaluation split. A threshold chosen and reported on the same items is optimistic.
5. Disagreement gate. Run a second, independent labeler (another model family or a supervised classifier) and route every disagreement. Wang et al. (CHI 2024) describe a related design in which a verifier scores each LLM label and humans re-annotate the lower-scoring subset.
6. Interpretation gate. Irony and cultural reference go to humans. شكراً على التأخير ("thanks for the delay") reads as gratitude to a naive classifier and as a complaint to anyone who has waited for a delivery.
7. Random audit. Send a blind random sample of accepted labels to humans, pre-label hidden. It is the only unbiased estimate of what you automated.

Hide pre-labels wherever you can. In 7,000 annotations from 350 annotators on subjective tasks, Schroeder, Roy and Kabbara (Findings of ACL 2025) found that LLM suggestions did not make annotators faster but pulled their labels toward the model's. Evaluated against those assisted labels, the model's reported performance rose. An anchored reviewer is not an independent check.

Which Arabic annotation tasks are safer to pre-label?

Treat this table as a starting position and let your gold-set results move each row. "Candidate" means worth measuring, not safe by default.

How to measure before you automate

Pangakis, Wolken and Fasching (2023) found LLM annotation performance "highly contingent on both the dataset and the type of annotation task" across 27 tasks, and concluded that any automated process must be validated against human labels. For Arabic, validate per variety too.

1. Build the gold set blind. Label without pre-labels and adjudicate with native reviewers; our annotation guidelines template covers adjudication rules.
2. Stratify. Sample every variety, script (Arabic, Arabizi, mixed) and label you intend to automate, including rare ones.
3. Split calibration from evaluation, and report realized coverage on the held-out partition.
4. Size each slice for the bound. The table shows roughly how many accepted items a slice needs, computed from the Wilson formula at 95% confidence.
5. Re-measure on drift. A new source, channel, region or model version resets the evidence.

With five dialect regions and a 95% floor, even a near-perfect model needs hundreds of adjudicated items per region before any region leaves the human queue.

Does AI-assisted Arabic annotation actually save money?

Inference is cheap: Gilardi, Alizadeh and Kubli (PNAS, 2023) put ChatGPT's per-annotation cost below $0.003 on a set of English tweets, about twenty times cheaper than MTurk at the time. The expensive parts are review, the gold set and errors that slip through. The illustrative model below assumes 100,000 items, 1.0 unit per human label from scratch, 3,500 units for the gold set and calibration, and negligible inference cost.

Pre-labeling without routing saves money only if review is genuinely faster than labeling, and the Schroeder study found no speed gain on subjective tasks. Measurement costs money even when the answer is "don't automate", but that is the cheap outcome: it keeps weak labels out of training data. Our breakdown of Arabic data annotation cost lists the line items behind the unit price.

Bayanat Labs supplies the human side of this framework: vetted native speakers and linguists who pass dialect and domain screening, work calibrated against gold standards with adjudication and rework loops, custom evaluations of model output per dialect and label, and an audit trail on every task. Our Arabic data annotation services describe the workflow; a fixed-scope pilot is the place to measure how much of your own queue can safely leave human hands.