LLM-as-a-Judge in Arabic: Where Automated Graders Fail

You can trust an LLM judge on Arabic outputs only for criteria where you have measured its agreement with native raters on your own data. The evidence splits in two: when comparing an answer with a gold reference, judges track human scores closely; when deciding alone whether a response is safe, culturally appropriate or natural in a dialect, they do much worse than native experts. In the ASAS red-teaming study, GPT-4o used as a safety judge caught only about 22% of the responses humans marked unsafe.

So "LLM as a judge" in Arabic is a calibration problem, not a yes-or-no question. Below: the research, the Arabic failure modes to test, a calibration protocol, and the agreement metrics to use. For the wider evaluation design, see Arabic LLM evaluation: how to benchmark a model you can actually trust.

Can you trust an LLM judge on Arabic outputs?

Trust depends less on the judge model than on what you ask it to judge. The table is a starting position drawn from the studies below: a prior to test, not a measurement.

What published research shows about Arabic LLM judges

The Arabic evidence is thin but consistent once sorted by task. None of these studies measured your prompts, dialect mix or judge.

• English baseline. On MT-Bench and Chatbot Arena, GPT-4 as judge agreed with human preferences over 80% of the time, about human-human level, but showed position, verbosity and self-enhancement biases (Zheng et al., 2023).
• Reference-based correctness in Arabic. The BALSAM benchmark had three humans score 22 models' answers on a sample of test questions (0–3 correctness), then had Gemini 2.5 Flash extract and score each answer against the reference. The judge's Pearson correlation with the average human score ranged from 0.824 to 0.977 across task categories, higher than the humans' correlation with each other (average 0.75, lowest 0.636 for creative writing). BLEU, ROUGE and BERTScore did far worse and were dropped.
• Safety. ASAS, a human-curated set of 801 adversarial Arabic prompts, had humans rate seven models' responses on a 4-point safety scale. GPT-4o as a safety judge reached about 50% accuracy and about 22% recall on responses humans marked unsafe; the authors conclude the task calls for human expert evaluation. Its ideal responses are written in MSA, and the authors list safety benchmarks in other Arabic dialects as future work.
• Culture and dialect. A 2026 study compared five LLM judges with native subject-matter experts grading Egyptian and Iraqi cultural and linguistic prompts (Abdoli et al.). Four of the five were systematically lenient (+2.01% to +6.56%), cultural items were harder to grade than linguistic ones, and implicit cultural reasoning was the main failure mode for every judge. With 103 prompt-rubric pairs, read it as a warning, not a precise estimate.
• Consistency across languages. A Findings of EMNLP 2025 study (five judges, five tasks, 25 languages including Arabic) found an average Fleiss' kappa of about 0.3 for judgments of the same content across languages. Multilingual training and scale did not reliably fix it; explanation prompts and a majority vote across open models helped (How Reliable is Multilingual LLM-as-a-Judge?).

The pattern: judges are strongest when a reference carries the knowledge, and weakest on safety norms, culture and dialect, where Arabic differs most from English.

Where do Arabic LLM judges fail? Register, dialect, orthography and culture

Put test cases for each of these into your calibration set. They are risks to check, not measured rates.

Register and dialect mismatch

A Saudi user writes in Gulf dialect: أبي أعرف رصيدي (abi a'arif rasidi, "I want to know my balance"). The assistant replies in Egyptian, حاضر، هشوف لحضرتك (hader, hashuf lehadretak, "sure, I'll check for you"). The Arabic is fluent, so a judge asked "is this helpful and correct?" can score it highly; a Saudi rater hears the wrong variety at once. The mirror case matters too: a judge may mark down an authentic dialect answer as informal. AL-QASIDA found that LLMs produce dialectal Arabic less well than they understand it, not because their dialect fluency is poor but because they are reluctant to generate it; a judge built on a similar model may share that bias toward MSA. Choose dialects from your traffic; see MSA vs dialectal Arabic.

Translationese that reads as fluent

أنا آسف لسماع ذلك ("I'm sorry to hear that") is grammatical and a literal calque of English. Native raters mark it down on naturalness; a judge whose sense of style comes from English-centric data may not. Pair calqued answers with natural ones and check the judge ranks them as your raters do.

Orthographic variation

Arabic tolerates legitimate spelling variation: إلى/الى ("to"), مدرسة/مدرسه ("school"), optional diacritics, and Arabizi such as "3ayez eh" for عايز إيه ("what do you want?", Egyptian). A strict judge penalises these; a lax one misses real errors among them. State which variants count as correct, in both the judge prompt and the human rubric.

Implicit cultural and religious norms

Much cultural correctness is never stated in the prompt. The expected reply to السلام عليكم (as-salamu alaykum) is وعليكم السلام (wa alaykum as-salam), not a generic "hello". Answers on religious practice, family or local custom can be fine in one country and jarring in another. This is the implicit cultural reasoning gap Abdoli et al. found in every judge, and with four of five judges lenient, such misses are likely to inflate scores.

Safety under obfuscation

ASAS reports direct prompting, code/encryption prompts and hypothetical testing as the most effective attack strategies. A harmful answer wrapped in a scenario or partly encoded may pass a judge as safe; at about 22% recall, the GPT-4o judge let most responses humans marked unsafe through. The judge scoring Arabic AI red teaming output needs as much scrutiny as the model.

How to calibrate an LLM judge against native raters

Calibration means measuring, per criterion and dialect, how often the judge agrees with adjudicated native-rater labels, then automating only where agreement is good enough.

1. Fix the criteria and scale first. One criterion per judge call (correctness, register, cultural fit or safety), a short scale (binary or 3–4 points) and written anchors.
2. Build a stratified calibration set. Real outputs from the models you will judge, split by dialect and domain, plus hard cases: wrong-variety answers, calques, spelling variants, Arabizi, culturally loaded prompts, unsafe responses. Aim for roughly 200 items per cell; oversample rare classes.
3. Collect independent native judgments. At least two raters who speak the variety score each item blind to the judge and model names; professionals for domain correctness.
4. Measure human-human agreement first. It is the ceiling. If raters disagree on register, fix the rubric before blaming the judge.
5. Adjudicate to gold. A senior rater resolves disagreements and records why; the reasons become examples in the prompt and rubric.
6. Run the judge under controlled settings. Fixed model version, temperature and rubric language (never translate outputs first), an explanation with each verdict (which improved consistency in the multilingual study), and both orders for pairwise tasks to control position bias, counting only consistent verdicts.
7. Compute agreement per cell. The metrics below, a confusion matrix and recall on each critical class; never one pooled number.
8. Error-analyse in Arabic. Read disagreements with native raters, group them by failure mode, revise.
9. Confirm on a held-out split. Re-test on items not used for prompt tuning, or you are measuring your fit to the calibration set.
10. Deploy with routing and audits. Automate only passing cells, send low-confidence verdicts and critical categories to people, keep a random human audit slice, and re-calibrate when the judge, rubric, prompt or traffic mix changes.

Which agreement metrics and thresholds should you use?

Use a chance-corrected statistic for the headline, a rank correlation when order matters, and class-level recall for anything safety-critical. Percent agreement alone misleads: judges with high agreement can still assign very different scores (Thakur et al., 2024).

An illustrative example (arithmetic, not a measurement): of 200 responses, native raters mark 40 unsafe. The judge flags 12 of those 40 and wrongly flags 4 of the 160 safe ones. Percent agreement is 84%, which sounds fine; Cohen's kappa is about 0.35 and recall on unsafe responses is 30%. Only the last two reveal that most harmful answers get through.

A practical acceptance rule: automate a cell only when judge-gold agreement is close to human-human agreement and clears an absolute bar set in advance, with a minimum recall on critical classes. Fix these thresholds before looking at results.

When should you keep humans in the loop?

Keep native raters wherever a wrong verdict is expensive or the judge has not passed calibration.

• Always human: launch sign-off, safety of adversarial outputs, cultural and religious appropriateness, and domain correctness in medical, legal and financial content.
• Human until calibrated: dialect fidelity and register, per dialect. A pass on Egyptian says nothing about Gulf or Maghrebi.
• Judge with audits: reference-based correctness, format and instruction following, and pairwise pre-screening during iteration.
• Judge-assisted data: if the judge filters synthetic data or pre-ranks Arabic preference data, its biases compound over training rounds, so audit what it rejects as well as what it keeps (see model collapse in synthetic Arabic data).

Decide how much leaves the human queue with a conservative bound, not a point estimate. LabelBench, our reproducible audit of automated Arabic labeling (a related but different task), does this with Safe Automation Coverage: the fraction of a queue that can be accepted while the 95% Wilson lower confidence bound stays above a declared accuracy target. It also shows how fast headline numbers collapse: on the fixed held-out manifest, a local language model at 78% exact-match accuracy on binary Arabic toxicity fell to 25.3% on the real multi-label taxonomy and 22.4% across five dialect regions, where chance is 20%.

Bayanat Labs provides the human side of this through Arabic LLM evaluation services. Vetted native speakers across 25+ Arabic varieties, and licensed domain professionals where the content requires, produce judgments calibrated against gold standards with adjudication and rework loops, and an audit trail on every task. We do data only and are model-agnostic: we never compete with clients' models. A fixed-scope pilot with gold-standard QA and a quality report is usually delivered within two weeks.