Arabic LLM benchmarks explained: ArabicMMLU, OALL, BALSAM and more

Arabic LLM benchmarks, in short: the most-cited tests are ArabicMMLU (native school-exam questions), the Open Arabic LLM Leaderboard suite (AlGhafa, ArabicMMLU, AraTrust, MadinahQA, ALRAGE and others), BALSAM and AraGen for open-ended answers, and AraDiCE and DialectalArabicMMLU for dialects. Most are multiple-choice tests in Modern Standard Arabic (MSA), scored by which answer the model rates most likely. They show whether a model recognises correct formal Arabic, not whether it writes well, speaks your users' dialect or completes your task. Use them to shortlist, then decide on a private test written in the varieties you serve.

This page is the catalogue. For how to design an evaluation you can defend, with per-dialect sample sizes, rubrics and raters, see our guide to Arabic LLM evaluation.

What do Arabic LLM benchmarks measure, and what do they miss?

Before trusting any benchmark number, ask four questions of the test behind it:

1. Who wrote the items? Native items are written by Arabic speakers for Arabic readers, such as exam questions. Translated items start in English; human translation fixes the Arabic but keeps the English topics. All 900 questions in each Arabic variety of Belebele, for example, are built on passages from English Wikinews, Wikibooks and Wikivoyage.
2. Which variety? MSA only, one dialect, or several reported separately. A single "Arabic" score with no breakdown is almost always an MSA score.
3. How is it scored? Log-likelihood multiple choice, generated text compared with a reference, an LLM acting as judge, or rule-based checks. Each rewards a different skill.
4. Who has seen the test set? Public on Hugging Face, held back by the maintainer (blind), or kept private for a fixed period and then released (rotating).

Here is what a typical item looks like. This grammar question appears in both ArabicMMLU and MadinahQA: اختر حرف الجر المناسب لوضعه بدلاً من [فراغ] في الجملة التالية: أنا [فراغ] باكستان. ("Choose the right preposition for the blank: I am [blank] Pakistan.") The options are ب (bi, "by"), إلى (ilā, "to"), من (min, "from") and في (fī, "in"); the answer is من. Under log-likelihood scoring, the evaluation harness compares the probability the model assigns to each option and marks the item correct if the right one scores highest. The model never writes a word. That is a fair test of grammatical knowledge and no test at all of whether it can draft a customer reply in Saudi dialect.

What almost no public benchmark measures:

• Producing dialect. The AL-QASIDA study found that models understand dialectal Arabic better than they produce it, and that the gap comes from reluctance to write dialect, not from poor fluency.
• Register and naturalness. Whether an answer sounds like a native speaker or like translated English.
• Multi-turn conversation and domain tasks. Almost every item is a single question with a single answer.
• Actions. Whether an agent actually changed the state it claims it changed. Our Arabic Agent Reliability Lab is a methods beta for exactly this gap: a fluent answer is not a completed transaction.
• Cost. The same Arabic passage can need very different numbers of tokens under different tokenizers, which no accuracy table reports. We cover the mechanics in the Arabic tokenization tax.

The Arabic LLM benchmark table

The table lists the benchmarks you are most likely to meet in model cards and leaderboards, each linked to its paper or dataset card. "Native" means written in Arabic; "log-likelihood" means the model picks an option by probability rather than writing an answer.

For the long tail (legal, medical, poetry, multimodal, hallucination), TII maintains a list of Arabic LLM benchmarks on GitHub, sorted by topic but without the columns above.

Knowledge and reasoning: ArabicMMLU and friends

ArabicMMLU is the default knowledge test for Arabic models. Its 14,575 MSA questions come from school exams in several Arab countries, so the content is regional: subjects include Islamic studies, Arabic language, civics, history and even a driving test. When it was published in 2024, the best Arabic-centric model scored 62.3%, and BLOOMZ, mT0, LLaMA2 and Falcon all stayed under 50%. Scores have risen since; check the live leaderboards for current numbers.

Three relatives often appear next to it:

• ArbMMLU-HT is English MMLU translated into Arabic by people, not machines. The Arabic reads well, but the syllabus is still the original's, down to high_school_us_history and us_foreign_policy. A model can score well here while knowing little about the region.
• Arabic EXAMS is the Arabic slice of EXAMS, a multilingual collection of high-school exam questions; the leaderboard's copy has 537 test questions.
• DialectalArabicMMLU takes the opposite approach to ArabicMMLU. It holds the questions fixed and changes the variety: the same 3,000 items appear in Syrian, Egyptian, Emirati, Saudi and Moroccan Arabic, alongside MSA and English. Because only the variety changes, any drop in score can be put down to dialect. Across 19 open-weight models, the authors report substantial variation between dialects.

What is on the Open Arabic LLM Leaderboard (OALL)?

The Open Arabic LLM Leaderboard, hosted on Hugging Face, ranks models published on the Hub. Its first version drew more than 700 submissions. Version 2, launched in February 2025, dropped the machine-translated tasks because translation introduced linguistic and contextual mismatches, and removed tasks where models had reached near-perfect scores, such as ACVA and Toxigen. The v2 suite is:

• AlGhafa, native tasks only, plus Belebele in MSA and in dialect;
• Arabic EXAMS, ArabicMMLU and ArbMMLU-HT;
• MadinahQA for language and grammar;
• AraTrust for trustworthiness and safety;
• ALRAGE for answering from retrieved passages.

How it is scored matters more than the list. In lighteval, the multiple-choice tasks use log-likelihood accuracy with length normalisation, and the maintainers apply a model's chat template when its configuration includes one. ALRAGE is the only generative task: Qwen2.5-72B-Instruct judges each answer against a gold answer on a 0–10 scale, normalised to 0–1. Submissions are capped at five per organisation per week (per the February 2025 announcement).

The v2 announcement also reports that the maintainers found a bug in how AlGhafa checked answer choices; fixing it cut some models' scores by up to 20 points. A leaderboard number is the output of software, and software has bugs.

HELM Arabic, from Stanford CRFM (December 2025), runs almost the same tasks: AlGhafa, ArabicMMLU, Arabic EXAMS, MadinahQA, AraTrust, ALRAGE and ArbMMLU-HT. It runs them differently, though: zero-shot prompts, a random sample of up to 1,000 items per subset, "thinking" modes disabled where possible, and closed-weight models included. It had a bug of its own: version 1.0.0 left out the passage on context-based questions, which version 2.0.0 fixed. Same tasks, different harness, so an OALL number and a HELM number for the same model are not comparable.

Arabic generation and chat benchmarks

Multiple choice cannot tell you how a model writes. Four benchmarks score generated answers instead:

• BALSAM is the broadest: 78 tasks in 14 categories, from creative writing to dialect translation, with a centralised platform at benchmarks.ksaa.gov.sa that keeps the test labels blind. Its human study is the strongest evidence against overlap metrics in Arabic. Averaged across categories, ROUGE-LSum correlated with human ratings at 0.111, BLEU at 0.134 and BERTScore at 0.184, while an LLM judge (Gemini 2.5 Flash) reached 0.824 to 0.977 by category. The authors dropped the overlap metrics as a result.
• AraGen, from Inception, scores each answer on six dimensions it calls 3C3H: correctness, completeness, conciseness, helpfulness, honesty and harmlessness. Each test set stays private for three months, then is published and replaced. The rotation shows how version-specific a score is. When the March 2025 set replaced the December 2024 one, the top model, o1-2024-12-17, fell from 82.67% to 70.25%, according to the maintainers' April 2025 post. It was the same model on harder questions.
• Arabic IFEval checks instruction following with code, not judgment. Alongside constraints adapted from English IFEval, it adds Arabic-specific ones on diacritics (التشكيل, tashkīl), particular letters and word roots.
• ABB, SILMA's Arabic Broad Benchmark, is small by design: 470 questions sampled from 64 datasets and scored by more than 20 rules plus GPT-4.1 as judge. Its leaderboard removes models flagged by a contamination check whose method and thresholds SILMA keeps confidential.

LLM judges are better than overlap metrics, but they are not neutral. Each benchmark uses a different judge (Qwen2.5-72B for ALRAGE, Gemini 2.5 Flash for BALSAM, Claude 3.5 Sonnet at AraGen's launch, GPT-4.1 for ABB), so a judged score is always partly a statement about the judge.

Arabic dialect benchmarks

Dialect is where public benchmarks are thinnest, and where it matters most for deployment. Belebele is useful here because it is parallel: the same question appears in every variety. This is one question in five of them:

The English is "What is not recognized by Armenia?" Five varieties give five words for "what" and three ways of saying "not": MSA لا, ما in Najdi and Levantine, Egyptian مش, and the Moroccan wrap-around ما…ش. The Levantine translator also switched to the past tense ("did not recognize"), a reminder that even careful translation drifts. And the question is about Armenia, because the passage came from English Wikinews.

The dialect-specific options:

• DialectalArabicMMLU is the cleanest comparison across varieties and the only one here with separate Saudi and Emirati versions.
• AraDiCE post-edited about 45,000 machine-translated samples across seven datasets (ArabicMMLU, BoolQ, PIQA, OpenBookQA, Winogrande, TruthfulQA and Belebele), which its authors describe as synthetic. The translated benchmarks come in Egyptian and Levantine. Its culture set has 30 questions each for Egypt, Jordan, Lebanon, Palestine, Qatar and Syria, so Qatar is the only Gulf country and there are no Saudi items.
• AL-QASIDA evaluates generation in eight dialectal varieties on four axes: fidelity, understanding, quality and diglossia. Its finding that models default to MSA, and that post-training adds to that bias while few-shot examples reduce it, is the most useful caveat for anyone building a dialect chatbot.
• BALSAM includes a dialect-translation category with 1,200 test examples, and ABB has dialect writing and translation skills.

None of these tests Arabizi, code-switching with English or French, or the sub-regional varieties inside one country. For choosing which varieties to test and in what proportions, see our guide to Arabic dialect coverage.

Arabic safety and cultural benchmarks

AraTrust is the safety component of the leaderboard suite: 522 human-written questions across nine areas, from truthfulness and ethics to privacy and offensive language. The Arabic safeguard set by Ashraf et al. (NAACL 2025) is larger and more adversarial. Its 5,799 questions cover direct attacks, indirect attacks and harmless requests that contain sensitive words, and on contested topics it assesses responses from both government and opposition perspectives. AraGen also has a safety category, 40 of the 340 items in the March 2025 set.

A caution on format: on the leaderboards, AraTrust is scored by log-likelihood, so it measures whether a model ranks the safe option highest, not whether it refuses to write the harmful one. Refusal behaviour, dialect jailbreaks and regional sensitivities need generation-based testing and human reviewers; that is what Arabic red teaming is for.

For culture, Palm (ACL 2025) was built over a year by 44 researchers from across the Arab world. It covers all 22 Arab countries and 20 topics, in MSA and in dialect. Its evaluation found some countries, such as Egypt and the UAE, better represented in model knowledge than others, such as Iraq, Mauritania and Yemen. AraDiCE-Culture is much smaller, at 180 questions. The older ACVA set was removed from the leaderboard because models had saturated it.

Which Arabic LLM benchmark fits your use case?

Where Arabic benchmark scores mislead

1. Multiple choice measures recognition, not production. Log-likelihood scoring asks which option the model finds most probable. It needs access to token probabilities, which is why probability-scored leaderboards list open-weight models, while HELM Arabic and BALSAM prompt closed models for answers. Neither approach tells you how the model writes.
2. The harness is part of the score. Chat templates, zero-shot versus few-shot prompting, subsampling and the choice of judge all move results. The AlGhafa and HELM bugs above show that a published number can be off by 20 points before anyone notices.
3. Public test sets leak. Most of the sets in the table can be downloaded from Hugging Face, and any model trained on a web crawl after their release may have seen them. Blind sets (BALSAM) and rotating ones (AraGen) reduce the risk. A rotated set becomes public when its window closes, though, and a confidential contamination check is one you cannot audit.
4. Suites saturate and overlap. ACVA and Toxigen were dropped once models neared perfect scores, and MadinahQA repeats ArabicMMLU items. Before you average a suite, check what is inside it.
5. Overlap metrics do not track quality. BALSAM's correlations of 0.11 to 0.18 for ROUGE, BLEU and BERTScore mean a BLEU-ranked Arabic generation table tells you little about which answers people prefer.
6. Task framing changes the answer. In our own LabelBench v0.1 audit, a local model scored 78% on Arabic toxicity framed as yes or no. On the real seven-label taxonomy, exact-match accuracy fell to 25.3%, and identifying five native dialect regions reached 22.4%, barely above chance. This was a laptop-scale audit of labeling tasks, not a model ranking, but the lesson carries over: a benchmark that simplifies your task will flatter any model.
7. Averages hide dialects. One blended score over MSA and dialect items can rank a model first overall and still leave it unusable for the variety your users write.

How Bayanat approaches Arabic benchmarking

We use public Arabic LLM benchmarks the way this page suggests: as a first filter. The decision set is built privately for each engagement. Prompts are written natively in each variety the client serves and stratified by dialect. Native speakers and linguists who have passed dialect screening score them against calibrated rubrics, with gold standards, adjudication and an audit trail, and the set is held out from public release so it stays useful across model versions. If you are choosing a model, regression-testing a fine-tune or signing off a launch, our Arabic LLM evaluation services start with a fixed-scope pilot that delivers a benchmark and quality report, usually within two weeks. If you are writing the rubric yourself, our playbook on how to annotate Arabic text covers the normalisation choices that decide whether two spellings count as the same answer.