Arabic data labeling: the complete guide for AI teams (2026)

Arabic data labeling is the work of attaching machine-readable labels to Arabic text, speech, images, video and model outputs so they can train or evaluate AI systems. Typical labels are intents, entities, sentiment, dialect, transcripts, bounding boxes and preference rankings. It is harder than English labeling: one written word can pack a conjunction, a verb and its object, spelling is inconsistent, and the same request looks different in Gulf, Egyptian and Maghrebi Arabic. A usable dataset depends on five decisions made before anyone labels an item: which varieties, which taxonomy, which conventions, which quality bar, and who does the work.

Below: each decision with worked Arabic examples, two decision tables, LabelBench data on model pre-labeling, and a checklist for your team or vendor.

What is Arabic data labeling (or labelling)?

Labeling, labelling and annotation describe the same job. "Labelling" is the British spelling, widely used outside the US. "Annotation" is the usual term in NLP research and often implies richer structure such as spans, relations or timestamps. Someone searching for Arabic data labelling services and someone searching for Arabic annotation want the same result: a dataset in which every label means the same thing on every row.

A complete Arabic labeling project has four outputs, not one:

1. The labeled data, with a variety and register tag on every item.
2. The guideline, which defines every label with Arabic examples and counterexamples.
3. A gold set of expert-adjudicated items, used to measure quality.
4. A quality report with agreement and error rates by label and by variety.

If you only receive the first output, you have no way to audit it.

Why is Arabic harder to label than English?

The CAMeL Tools paper from NYU Abu Dhabi lists four core difficulties: orthographic ambiguity, morphological richness, dialectal variation and orthographic noise. Each one turns into a labeling decision.

• One word, several tokens. وسيكتبونها (wa-sa-yaktubūnahā, "and they will write it") is a conjunction, a future marker, a verb and an object pronoun written as one word. The same paper notes that MSA verbs have upwards of 5,400 forms. Your guideline must say whether labels attach to whole words or to segments. Entity spans get awkward when prepositions attach. In للرياض (li-l-riyāḍ, "for Riyadh"), the alef of الرياض is not written, so the entity is not a clean substring of the text.
• Missing short vowels. Written without diacritics, كتب can be read as kataba ("he wrote"), kutiba ("it was written") or kutub ("books"). Native readers resolve this from context. Labels that depend on the reading, such as lemmas, part of speech or text-to-speech data, need an explicit rule.
• Spelling variation. Users often type احمد for أحمد (Aḥmad), leaving the hamza off the alef. They also type مدرسه for مدرسة (madrasa, "school") and swap final ى and ي. Normalizing these variants helps search, but it can destroy meaning: على (ʿalā, "on") and علي (ʿAlī, a name) collapse into one string. This is why CAMeL Tools ships separate normalizers for alef, alef maksura and teh marbuta: each one is a policy choice. The CAMeL Tools paper also cites a study in which 32% of words in online MSA comments contained spelling errors.
• No dialect spelling standard. Arabic dialects have no official orthography. The CODA* guidelines document 27 attested online spellings of a single Egyptian word, مبيقولهاش (mabiʾulhāsh, "he doesn't say it"). Transcription and dialect datasets need a declared spelling convention. Without one, every annotator invents their own.
• MSA tools misread dialect. The CAMeL Tools paper cites results in which an MSA morphological tagger scored about 96% part-of-speech accuracy on MSA but about 72% on Gulf Arabic. Pre-processing built for MSA quietly degrades dialect data before any human sees it.
• Arabizi and code-switching. Arabic is often written in Latin letters, with digits standing in for sounds English lacks (Darwish, 2014). Take "wallahi el service kan 7elw bas el delivery et2a5ar", which is والله السيرفس كان حلو بس الدليفري اتأخر in Arabic script ("Honestly, the service was nice but the delivery was late"). Here 7 stands for ح, 2 for ء and 5 for خ. That one line contains Egyptian Arabic, Latin script, English loanwords and two opposite sentiments.

Types of Arabic data labeling: text, speech, image, video and LLM outputs

Most projects combine several task types. The table lists the labels each task produces and the Arabic-specific failure to design against.

For model outputs, labels are judgments rather than categories, so rater variety and rubric design matter more; see evaluating Arabic LLMs.

Which dialects and registers should you label?

Settle the variety mix before you write a single label definition. It determines who can annotate, how the guideline reads and how quality gets reported. Here is one intent in three varieties:

• MSA: أريد إلغاء اشتراكي الآن (urīdu ilghāʾa ishtirākī al-āna)
• Gulf: أبي ألغي اشتراكي الحين (abi alghi ishtirāki al-ḥīn)
• Egyptian: عايز ألغي اشتراكي دلوقتي (ʿāyiz alghi ishtirāki dilwaʾti)

All three mean "I want to cancel my subscription now" and share one intent label. But Gulf أبي ("I want") is spelled exactly like MSA أبي (abī, "my father"), and the word for "now" differs in each variety. A labeler who doesn't know the variety can get it wrong, and a model trained on one variety may never see the others. Four decisions follow:

1. Match real traffic. Mirror the countries and channels your model will serve, whether app chat, voice, social media or documents.
2. Choose a granularity. Regions (Gulf, Levantine, Egyptian, Maghrebi, Iraqi) are enough for most routing. Speech and localization often need country or city. Research corpora go further: MADAR covers 25 city dialects and CODA* covers 28.
3. Tag every item. Record variety and register (MSA, dialect, mixed, Arabizi) even when variety is not the target label. Without the tag, you cannot report coverage or quality per variety.
4. Match annotators to varieties. A Cairo native labeling Emirati speech is not a native annotator for that item.

The modeling case is in MSA vs dialectal Arabic; for rarer varieties, see Arabic dialect data.

How to design an Arabic label taxonomy

Many labeling problems that surface in production start in the taxonomy. Six rules prevent the common ones:

• Start from the downstream decision. Every label should change what the product or model does. If two labels trigger the same action, merge them.
• Keep dimensions separate. Intent, sentiment, variety and register each get their own field. "Egyptian complaint" is two labels, not one class.
• Allow multiple labels where the data does. One Arabic insult can hit several harm categories at once. Forcing a single choice throws away signal and leaves annotators arguing over which harm wins.
• Add escape labels. Unclear, not Arabic, mixed variety and needs expert. A forced choice hides ambiguity, and ambiguity is what you most need to see.
• Write Arabic boundary rules. Cover quoted religious text, sarcasm, reported speech, clitic-attached entities and names that are also common words.
• Version everything. The taxonomy will change after the pilot. Versioned labels can be remapped; unversioned ones have to be relabeled.

Guidelines, gold sets and inter-annotator agreement

Guidelines. Every label needs a definition, positive Arabic examples from each target variety, near-miss counterexamples, and a precedence rule for when two labels both seem to apply. Add three conventions most teams forget. The first is a tokenization scheme. The second is a normalization policy that says what is normalized for the model and what is preserved for annotators. The third is a spelling convention for dialect transcription, for which CODA* is a well-documented starting point.

Gold sets. A gold set is a set of items labeled and adjudicated by senior linguists or domain experts, stratified by label and variety. It serves as the onboarding test, the hidden checks in the live queue, the drift detector, and the fair comparison of annotators, vendors or models. Keep it out of training data.

Agreement. Pick the metric that fits the task:

• Cohen's kappa for two annotators.
• Fleiss' kappa for more than two.
• Krippendorff's alpha when ratings are missing or the scale is ordinal.
• Span-level F1 for entities.

In their survey of inter-coder agreement in computational linguistics, Artstein and Poesio report that researchers commonly aim for κ or α of at least 0.8 and treat 0.67–0.8 as supporting only tentative conclusions. They caution that no single threshold suits every task, and show that skewed label distributions push these coefficients down even when raw agreement is high. Common practice is therefore to report agreement per label and per variety, not as one global number. A strong overall kappa can hide a Maghrebi slice on which annotators routinely disagree.

Expect errors even after QA. Northcutt, Athalye and Mueller found label errors averaging at least 3.3% across ten widely used test sets, including at least 6% of the ImageNet validation set. Budget for adjudication and rework instead of treating the first pass as final. The step-by-step operator loop is in how to annotate Arabic text.

Can AI label Arabic data? When pre-labels are safe

Sometimes, on narrow tasks, if you measure first. LAraBench (EACL 2024) evaluated GPT-3.5, GPT-4, BLOOMZ, Jais and speech models on 33 Arabic tasks and 61 datasets. It found that task-specific state-of-the-art models generally beat zero-shot LLMs, although few-shot prompting of larger models narrowed the gap.

Our own audit, LabelBench, measures what happens as the labeling task becomes realistic. The v0.1 results (run July 2026) cover the evaluated local models on this fixed held-out manifest:

• Qwen 2.5 3B scores 78.0% on binary Arabic toxicity. Exact-set accuracy falls to 25.3% on the seven-label taxonomy, and five-region dialect ID reaches 22.4%, where a uniform guesser gets 20%. It predicts Egyptian for 70% to 80% of items from every non-Egyptian region, so its Gulf recall is only 4%.
• Fanar 1 9B, which is Arabic-specialized, reaches 88.0%, 31.3% and 47.8% on the same three tasks.
• Safe Automation Coverage. Every v0.1 generative run reached 0% SAC at a 95% quality floor under tied-confidence routing. SAC is the share of a queue that can be accepted while the lower confidence bound stays above the target; it is a diagnostic, not a substitute for a human-replacement test.

That zero diagnoses the confidence protocol; it does not prove generative models cannot support selective automation. Frontier APIs were not part of v0.1.

Pre-labels are reasonable when all four conditions hold:

1. The task is narrow and well defined, and the model clears your quality floor on your own gold set, measured per variety.
2. Confidence is elicited and calibrated, so low-confidence items route to humans.
3. Humans review a random sample of accepted pre-labels, not only the items the model flagged.
4. Some items are relabeled blind, to check that reviewers are not rubber-stamping.

Keep humans in charge of multi-label taxonomies, dialect ID, sarcasm, domain judgments, safety labels and every gold set.

How much does Arabic data labeling cost?

Arabic labeling is priced per item, per label, per audio hour, per preference pair or per expert hour. Published Arabic rate cards are rare, so compare quotes on the things that drive them:

• Task complexity. Labels per item, spans versus whole-item labels, and multi-label versus single-label.
• Variety rarity. Widely spoken varieties are easier to staff than rare ones, especially for speech.
• Domain expertise. Physicians, lawyers and bankers cost more per hour, but good routing means they only see the items that need them.
• QA depth. Double-labeling overlap, gold-item frequency and adjudication.
• Residency and security. In-region processing, controlled workspaces and audit logs.
• Guideline churn. Rework when the taxonomy changes mid-project.

An illustrative example of how QA affects effort: if 20% of items are double-labeled and a tenth of those go to an adjudicator, human effort is roughly 1.2 times a single pass, plus adjudication time. Cutting overlap to zero saves that margin, but you lose the ability to measure quality at all. Our Arabic data annotation cost guide covers pricing units and budgeting in detail.

In-house, crowd or managed vendor?

Many teams go hybrid: taxonomy and gold set in-house, volume with a vendor that can prove per-variety quality. Our vendor checklist and scorecard lists the questions to ask.

How Bayanat approaches this. Bayanat Labs is a Riyadh-based Arabic data engine. We do data only, so we never compete with your model. Our Arabic data annotation service runs Vet → Train → Produce → QA → Deliver. Contributors pass dialect and domain screening before they touch client data. Their work is calibrated against gold standards with adjudication and rework loops, and every task has an audit trail. Hosting is in-region by default. Engagements start with a fixed-scope pilot with gold-standard QA and a quality report, usually within two weeks.

Arabic data labeling checklist

1. Traffic map. Countries, channels and scripts (Arabic, Arabizi, mixed) your model will face.
2. Variety plan. Target varieties, granularity, and a variety and register tag on every item.
3. Taxonomy v1. Separate dimensions, multi-label where needed, escape labels and version numbers.
4. Conventions. Tokenization scheme, normalization policy, dialect spelling convention and diacritics policy.
5. Guideline. Arabic examples and counterexamples per label and per variety, plus precedence rules.
6. Gold set. Expert-adjudicated and stratified, kept out of training data and seeded into the queue.
7. Annotator routing. Screened on the varieties and domains they will label.
8. Agreement plan. Metric chosen per task, reported per label and per variety, with an adjudication path.
9. Automation gate. Pre-labels only after measured accuracy and calibrated routing on your gold set.
10. Data controls. Minimization, residency, NDAs, audit logs and a deletion date.