Arabizi NLP: Detecting and Annotating Arabizi and Code-Switched Arabic

Arabizi is Arabic written in Latin letters, with digits standing in for sounds English lacks: 3 for ع, 7 for ح, 2 for the glottal stop. In Arabizi NLP the hard part is rarely the digits. It is that Arabizi arrives mixed with English or French, has no standard spelling, and shares word forms with English. Handle it with three habits: detect it at the word level, label the original text as written, and turn it into Arabic script only as a separate, reviewed step.

What is Arabizi? The Arabizi numbers chart

Arabizi, also called Arabish or Franco-Arabic, writes MSA or dialect in Latin letters, with digits for letters that have no Latin equivalent (Darwish, 2014). The digit usually looks like the letter, and an apostrophe adds the dot of a sister letter. Regional variants below follow the compiled table in Wikipedia's Arabizi article, which draws on studies of Egyptian and Lebanese usage.

Two lessons for annotation. One digit can mean different letters: in the Egyptian sentence ana kont mt25er fe t2demm l projects ("I was late in submitting the projects"), from the LDC Arabizi corpus, the first 2 is a hamza and the second is a qāf. And letters vary too: where French is the main second language, ش is often written ch and ج dj (especially in Algeria); elsewhere sh and j are more usual.

How common are Arabizi and code-switching in real traffic?

There is no single prevalence figure. The studies that measured it looked at different channels, years and dialects:

The pattern holds even where the numbers differ: script choice depends on channel and audience. The only figure that should drive your data plan is the one you measure. Have native speakers label the script of a random sample from each channel (Arabic, Arabizi, Latin-script foreign, mixed); a few hundred messages will show whether Arabizi is a rounding error or a slice that needs its own training and test data.

Arabizi NLP step one: how do you detect Arabizi automatically?

Detection is word-level, in context. Darwish (2014) trained a sequence labeler on tweets labeled word by word as Arabizi, English or other, and reached 98.5% word-level accuracy on his test set. Tobaili (2016) classified whole tweets; most of the Arabizi tweets his classifier missed were Lebanese tweets mixing English and Arabizi, which the Langdetect library labeled mostly as English. Failure cases from both papers:

• Words shared with English. In Ana 3awez aroo7 men America leh Canada ("I want to go from America to Canada"), men is Arabic من "from".
• Digits that are English shorthand. In 4 those going 2 tahrir, 4 and 2 mean "for" and "to", not Arabic letters.
• Arabizi with no digits. Darwish lists ta7rir, t7rir, tahrir, ta7reer, tahreer as spellings of one word, تحرير "liberation". Digit filters miss the last three.
• Vowel-dropping. Tobaili saw Egyptian users shorten ana to na and hane3mal to hn3ml.
• Trilingual messages. bonsoir 7ewalit a3melik add 3ala fb bass i didnt find you, a Lebanese tweet in Tobaili's data, mixes French, Arabizi and English: "Good evening, I tried to add you on FB but I didn't find you".

A workable pipeline: profile each message's script, flag Latin tokens mixing letters with 2, 3, 5, 6, 7, 8 or 9 (a candidate, not a verdict), run a word-level classifier in context, and send uncertain messages to native reviewers, whose labels double as the detector's gold set.

Arabic–English and Arabic–French code-switching

Code-switching happens between sentences, within a sentence, or inside a word, the three levels the Mixat authors distinguish, and in Arabic it also crosses scripts. The COLING 2025 survey of code-switched Arabic NLP (Hamed et al.) finds research concentrated on dialect–foreign-language and MSA–dialect switching, with Egyptian–English and Algerian–French the most studied bilingual pairs. Patterns to expect:

• Gulf and Saudi Arabic with English. English product and service words appear in Latin script, as in ممكن تسوي لي refund؟ (mumkin tsawwi li refund?, "can you process a refund for me?"). They also appear in Arabic script with Arabic morphology, as in أبغى أكنسل الطلب (abgha akansil al-ṭalab, "I want to cancel the order"). Both examples are illustrative. For speech, Mixat covers Emirati–English; its name is itself a code-mixed word for "mixes".
• Egyptian Arabic with English. English words take Arabic determiners (el anniversary) or Egyptian pronunciation (lozar, "loser"), per Bies et al.
• Maghrebi Arabic with French. The Algerian sentence salem 3alikoum inchalah le pondium et les midailes d'or ("peace be upon you, God willing the podium and the gold medals"), from Seddah et al. (2020), mixes Arabizi with French spelled non-standardly: pondium, midailes.
• MSA with dialect. No script signal at all; shared tasks reviewed in the survey found it the hardest pair to tag word by word.

Arabizi annotation rules that keep labels consistent

Most disagreement on this data comes from undecided policy, not misreading. The LDC team's numbers show where the difficulty lies: annotators agreed over 98% of the time when flagging tokens as foreign, name, sound or punctuation, but only 86.4% on the full transliteration, and 60% of those disagreements came from CODA spelling decisions that did not follow the guidelines (Bies et al., 2014). Flagging is easy to make consistent; spelling is not. Decide these cases before the pilot:

Four more rules:

1. Label the original text. Labels go on the message as typed. Nobody transliterates or corrects inside a labeling task.
2. Keep dialect, language and script separate. An Arabizi message still has a dialect, and the same digit 9 can mean ص or ق depending on the writer (see Arabic dialect identification).
3. Own the borrowing question. The survey notes many projects leave borrowing versus switching to annotator judgment; some add a label for borrowed words. Pick one and keep a shared borrowing list (fb, whatsapp, ok).
4. Route by dialect and seed gold items. Egyptian, Gulf and Maghrebi Arabizi need different readers; put adjudicated Arabizi and hybrid-word items in the gold set. Our guidelines template has a script section to copy; the Arabic text annotation playbook covers token-level language tags.

Transliterate, normalize or keep Arabizi?

Choose by what the data is for. Converting everything loses the user's actual input; converting nothing breaks tools that expect Arabic script, the reason Bies et al. give for transliterating at all.

When you transliterate, machine-draft and have humans correct: Darwish reported 88.7% conversion accuracy on his test set, and the Atar model reported 79% accuracy on its own Jordanian dataset. Bies et al. estimated that pre-filling annotators' screens with automatic transliteration cut typing time by two-thirds. Standardize the output with a conventional orthography such as CODA, never the source.

Arabizi and Arabic code-switching datasets

Public resources are thin. The survey counts 22 corpora for word-level language identification and 7 for transliteration, with most other tasks poorly supported. Ones to know:

Two gaps stand out. None of the text resources above covers Gulf or Saudi Arabizi. And the largest recent set is machine-generated: per its dataset card, NileChat's Arabizi was transliterated by a model, so it carries the model's spelling habits, not users'. Fine for pre-training; don't evaluate on it. Our note on the limits of synthetic Arabic data explains why.

At Bayanat Labs, annotation work goes to vetted native speakers who pass dialect screening before touching client data, and it is calibrated against gold standards with adjudication and rework loops; for mixed-script projects, the rules above belong in the guideline and Arabizi items belong in the gold set. Our Arabic data annotation and dialect data services start with a fixed-scope pilot on your own messages, with gold-standard QA and a quality report, usually within two weeks. Be wary of automating the routing: in LabelBench, a local language model scored 22.4% exact-match accuracy on five dialect regions (chance 20%) on the fixed held-out manifest. LabelBench did not test Arabizi.