Arabic dialect identification: datasets, models and why LLMs struggle

Arabic dialect identification (ADI) is the task of deciding which variety of Arabic a text or recording is in: Modern Standard Arabic, a regional family such as Gulf or Maghrebi, a country, or a city. Supervised classifiers trained on native, in-domain data do this reasonably well for regions and poorly for fine-grained labels on short texts. General-purpose LLMs used zero-shot are much weaker than their fluency suggests. On LabelBench's balanced five-region test, a local language model (Qwen 2.5 3B) scored 22.4% on the fixed held-out manifest, against 20% for random guessing.

Below: the datasets most cited for dialect detection, the LLM numbers, how to pick a label granularity, and the labeling rules that make a dialect ID gold set worth having. The focus is written text.

What is Arabic dialect identification, and why does it matter?

Arabic is diglossic: people write formal MSA but speak, and increasingly type, their dialects (our explainer on MSA vs dialectal Arabic covers why that gap sets a model's ceiling). The same request looks very different across the region:

Common everyday forms; each has several accepted spellings, and local variation exists inside every row.

Dialect ID is rarely the product; it is the router in front of one. Teams use it to pick the right intent model or ASR engine, to measure how dialectal a corpus really is, to balance fine-tuning data by variety, and to report evaluation per dialect. If the router is wrong, everything downstream inherits the error.

How well do LLMs identify Arabic dialects?

Less well than their fluency suggests. LabelBench, our reproducible audit, tested this on a balanced 500-item held-out manifest of native tweets from the DART dataset, 100 per region, drawn from the IADD integration. A five-class guesser scores 20%. Scores are exact-match accuracy on this fixed held-out manifest:

Source: LabelBench v0.1 (July 2026). The evaluated local models were run with structured JSON output; frontier API models were not part of v0.1.

Three findings matter for practitioners:

• The failure is collapse, not noise. Qwen 2.5 3B labeled 85% of Egyptian examples correctly but also called 75% of Gulf, 70% of Iraqi, 80% of Levantine and 70% of Maghrebi examples Egyptian. On a test set that is mostly Egyptian, that model would look fine.
• Arabic specialization helps a lot, but not enough. Fanar 1 9B beat Qwen 2.5 3B by 25.4 points on the identical items, with per-region recall ranging from 13% (Iraqi) to 75% (Gulf).
• The task is learnable. A small character model trained on 500 disjoint examples still beat Fanar by 22 points. Dialect cues sit in spelling and morphology, which character n-grams capture cheaply.

No evaluated generative system achieved non-zero Safe Automation Coverage at a 95% quality floor under the tied-confidence v0.1 protocol, meaning none of their labels could be safely routed past human review. That is the evaluated local models on one manifest, not a verdict on every LLM, but independent work points the same way. Kanjirangat et al. (2025) report a zero-shot F-score of only 8% for Phi-3.5-mini on NADI-2023, with a strong bias toward Egyptian, while the Arabic-specific SILMA leaned toward Saudi; LoRA fine-tuning performed best, at an 85% F-score. Test any model on your own labeled data first; our guide to Arabic LLM evaluation covers how, and our human-in-the-loop framework covers when machine labels can leave the review queue.

Arabic dialect identification datasets: NADI, MADAR, QADI and others

Most dialect detection papers use a handful of public corpora. What matters most is how each label was produced:

The NADI shared task has run as a recurring shared task since 2020. Its 2024 edition changed the question: instead of assigning each tweet one country, annotators from each country were asked whether the tweet could have been written by a speaker of one of their country's dialects, answering yes, maybe or no. A tweet counted as valid for a country when one annotator said yes and another said yes or maybe. The change matches the argument of Keleg and Magdy (2023), who had seven native speakers review a single-label classifier's errors and found about 66% of the validated errors were not true errors: the "wrong" dialect was also a plausible one.

For off-the-shelf tooling, CAMeL Tools ships a pretrained classifier based on Salameh et al. (2018), covering the 25 MADAR cities plus MSA and reporting results at city, country or region level. A solid baseline, though the system it follows was built on MADAR's translated travel sentences.

Region, country or city labels?

Granularity is a product decision, not a research default. Finer labels need rarer annotators and are less accurate on short text. Salameh et al. report 67.9% accuracy across 25 cities plus MSA for sentences averaging 7 words, and more than 90% when considering 16 words. A four-word support chat gives a classifier very little.

A useful pattern is a hierarchical label: store the finest label the annotator can defend, and roll up to region for training and reporting. Saudi Arabia shows why. Najdi, Hijazi, southern and eastern speech differ enough that a single "Saudi" or "Gulf" label can hide failures (see our breakdown of Saudi Arabic dialects for AI).

How to label mixed-dialect and MSA-heavy text

Real text rarely fits one class: comments mix MSA with dialect, messages switch into English, and short phrases are shared across regions. Take الخدمة الجديدة حلوة بس الأسعار غالية ("the new service is nice but the prices are high"). حلوة (ḥilwa, "nice") and بس (bass, "but") mark it as dialectal, yet both are used across Egyptian, Levantine and Gulf speech. Forcing one label here creates exactly the false errors Keleg and Magdy measured. Rules that work:

1. Separate "is it dialect?" from "which dialect?". Record a dialectness level first. NADI 2024's ALDi subtask used levels 0–3, from MSA to strongly dialectal.
2. Allow multiple valid labels. Ask each native annotator whether the text is plausible in their variety, then aggregate, rather than asking one person to pick a single winner.
3. Add explicit escape classes. MSA, mixed/unclear and too short to tell keep ambiguous items out of your training signal instead of polluting it.
4. Label the evidence. Annotators highlight the deciding tokens, such as عايز or دابا, which speeds adjudication and exposes guessing.
5. Route by dialectness. Keleg, Magdy and Goldwater (2024) found across 15 public datasets that highly dialectal samples are harder to label, especially for annotators who do not speak the dialect, and recommend routing them to native speakers of that dialect.
6. Handle Arabizi and code-switching separately. Latin-script Arabic and English or French insertions need their own rules; tag them rather than forcing a dialect label.

Building an Arabic dialect identification gold set

A leaderboard score tells you little about your traffic; a gold set from your own channel does. The minimum:

1. Fix the taxonomy and granularity first, including MSA and escape classes, and write them into a guideline with examples per class. Our annotation guidelines template is a starting structure.
2. Sample from production, stratified. Cover each channel (chat, voice transcripts, social), each length bucket and each market. Keep a natural-distribution slice alongside any balanced one.
3. Use native annotators per variety. Screen them on the dialect itself, not general Arabic fluency, and give each item to more than one.
4. Aggregate with a written rule (for example NADI's yes-plus-yes-or-maybe), then adjudicate disagreements with a senior native reviewer and record the reason.
5. Prevent leakage. Keep all texts from one user or conversation on the same side of the train/test split, so the model learns the dialect rather than the author.
6. Report per-variety recall, macro-F1 and a confusion matrix, not headline accuracy. LabelBench's Qwen result shows how a collapse toward one dialect stays invisible in an average.
7. Always run a cheap baseline. A character n-gram model on a few hundred labeled examples is a floor any LLM or vendor system should beat. On LabelBench's manifest, none of the evaluated zero-shot LLMs did.

This is the work Bayanat Labs does on Arabic dialect data collection and annotation. We cover 25+ Arabic varieties with vetted native speakers who pass dialect screening before touching client data, and work is calibrated against gold standards with adjudication and rework loops. A fixed-scope pilot, scoped to the label granularity you need, includes gold-standard QA and a benchmark/quality report, usually within two weeks.