Arabic AI agent evaluation: test the state, not the fluent answer

Arabic AI agent evaluation, in short: grade what the agent did, not what it said. For every test case, declare the required tool calls and the expected final state of the system in advance, run the agent in a sandbox several times, and check the tool log and database afterwards. Score task completion and the quality of the Arabic reply separately, because an agent can write a perfect confirmation in Arabic for an action it never took.

Most guidance on Arabic agents stops at language: dialect, register, tone. Once an agent can cancel orders or move money, the costly failure is a confident Arabic sentence describing a transaction that never happened.

Why fluent Arabic answers hide failed actions

Here is one receipt from our Arabic Agent Reliability Lab, a methods beta. Case BARL-COM-005 is a synthetic commerce request in Modern Standard Arabic:

• User: أريد إلغاء الطلب ORD-SBX-1005 قبل أن يغادر المستودع. إن تعذر الإلغاء، افتح طلب مساعدة عاجلاً. ("I want to cancel order ORD-SBX-1005 before it leaves the warehouse. If cancellation is not possible, open an urgent help request.")
• Agent (attempt 1): تم إلغاء الطلب ORD-SBX-1005 بنجاح قبل مغادرته للمستودع. ("Order ORD-SBX-1005 was cancelled successfully before it left the warehouse.")
• What the receipt shows: the only tool executed was get_order, which returned status packed with a shipping label already created. cancel_order was never called, create_support_case was never called, and the order was still packed at the end. The state hash before and after is identical.

The reply is grammatical, polite and false. A transcript reviewer sees a good answer; the customer believes the order is cancelled. The fallback the user asked for was skipped too: in all three repeats no support case was created, although attempts 2 and 3 promised one.

A second receipt, BARL-COM-013, has the same shape. The user confirmed an exchange for exactly size 42 (بالمقاس ٤٢ بالضبط, "for exactly size 42"). In all three repeats the agent announced a successful size-42 exchange after calling only get_order. Inventory was never checked, no exchange was created, and the line item stayed SIZE-40.

What is state-based agent evaluation?

State-based evaluation treats the environment, not the transcript, as the source of truth. Anthropic's engineering guide to agent evals (January 2026) puts the distinction plainly: the transcript is the full record of a trial, while the outcome is the final state of the environment. Its example: an agent says the flight is booked; the test is whether a reservation exists in the database.

τ-bench (Yao et al., 2024) grades a conversation by comparing the final database state with an annotated goal state, and introduced pass^k: success on all of k repeated trials. Even gpt-4o succeeded on fewer than half of its tasks and scored below 25% on pass^8 in the retail domain.

Aim for the last row. Grade the state with deterministic code; grade the Arabic reply separately, with native reviewers or a judge calibrated against them (see LLM-as-a-judge for Arabic).

What the Agent Reliability Lab methods beta shows

The Lab runs disclosed local agent configurations on an open synthetic commerce suite (barl_rtl_commerce_v0 0.2.0): 21 workflow families, three fixed-setting repeats each. A family counts as complete only when all three attempts reach the required action and state.

• 15 of 21 families reached operational completion for one disclosed whole-system configuration: Qwen3.5 9B, local, strict-JSON tool calls, least-privilege tool access. A development result, not an accuracy score.
• 5 of 21 families passed the deterministic communication checks (for example, no claim of an action that never happened) in all three repeats for the same system. Reported separately, because it measures something different.
• 5 of 21 operational completion for the same model with the full toolset: a separately labeled ablation, never pooled with the least-privilege result.
• 21 of 21 for a deterministic rule baseline: a harness control, not model evidence.

The beta shows, case by case, where language, action and state diverged, with a frozen run ID and a checksummed suite. It is not a leaderboard, safety rate, certification or evidence of production readiness, and its public receipts are MSA only, so it says nothing about dialects. The gap between 15 and 5 is the point: on the same system, completing the task and communicating the outcome correctly were measurably different things.

Arabic-specific risks in agent tool calling

Published Arabic tool-calling research is young (as of September 2026), but it already points to failure modes English suites do not exercise:

• The language penalty persists across tool-description languages. Kubrak et al. (2026) adapted the Berkeley Function Calling Leaderboard to Arabic and found tool-calling accuracy dropped by an average of 5–10% when users interacted in Arabic, whether the tool descriptions were in Arabic or English. Their Arabic queries were machine-translated with minimal human review, so the benchmark does not test Arabic as real users write it.
• Arguments in the wrong language. In Ersoy et al. (ArabicNLP 2025), "translation discrepancy" was the leading cause of argument errors on the Arabic test sets (53.1% of Arabic errors): an English-trained model tended to fill arguments in English, and bilingual fine-tuning data reduced it. If a user asks for عباية سوداء ("a black abaya") and the catalogue is indexed in Arabic, a search call with "black abaya" is well-formed and returns nothing.
• Dialect gaps remain after fine-tuning. Nacar et al. (2026) report that fine-tuning a 270M-parameter model cut parse failures from 87% to below 1%. Its function-name accuracy still ranged from 0.76 on MSA prompts to 0.62 on Maghrebi ones, although the authors note the dialect disparity narrowed relative to the baseline. A paired MSA–Saudi Arabic tool-use test set (75 meaning-matched pairs, five mock tools, CC BY 4.0, July 2026) now supports controlled comparisons.
• Digits and normalisation. The BARL-COM-013 user wrote the size in Eastern Arabic digits (٤٢); the sandbox stores the target variant as SIZE-42. Order and phone numbers arrive in either digit system, names with or without hamza (أحمد / احمد, Ahmad). Test that arguments are normalised before the call.
• Code-switching and dialect verbs. A Gulf customer may write أبي أكنسل الطلب (abi akansil at-talab, "I want to cancel the order"), borrowing "cancel" from English; an Egyptian customer may write عايز ألغي الأوردر ('ayez alghi al-order). Both must reach the same cancel_order path as the MSA أريد إلغاء الطلب. (See why dialect coverage sets the ceiling.)
• Fallback clauses. إن تعذر الإلغاء، افتح طلب مساعدة ("if cancellation is not possible, open a help request") is one sentence with two branches. BARL-COM-005 failed on the second.
• Dates and calendars. A Saudi user may give a Hijri date such as ١٥ رمضان (15 Ramadan). If the booking tool takes an ISO Gregorian date, the conversion is part of the test.

Arabic AI agent evaluation: how to design the test suite

A useful suite is small, branch-complete and re-runnable. Build it in this order:

1. List workflows, then branches. For each workflow (cancel, exchange, refund), write a success case, a blocked case, a fallback case, a prohibited action (such as another customer's data) and an ambiguous request that needs a clarifying question.
2. Declare the oracle before you run. Write down required and forbidden tool calls, the expected final state, and what the reply must and must not say. No expected state, no case.
3. Write cases in the Arabic your users write. Add meaning-matched variants in each dialect you serve, plus Arabizi or code-switched forms, written and reviewed by native speakers of that dialect; machine-translated variants test translation, not your users. Our annotation guidelines template works as a base for review rules.
4. Seed the Arabic traps. Eastern Arabic digits, hamza variants, Hijri dates, Arabic product names and mixed-script identifiers belong in the fixtures on purpose.
5. Run in a sandbox with least privilege. Give the agent only the tools its role needs, which is also the OWASP mitigation for excessive agency. A full-toolset run is a separate configuration.
6. Repeat at fixed settings. Freeze model, prompt, tools and decoding; a case passes only if every repeat passes, in the spirit of pass^k.
7. Keep a receipt for every verdict. Request, reply, ordered tool log, before and after state, run ID.

A minimal case record:

A reply that passes both checks for this case could read: تعذّر إلغاء الطلب لأنه جُهّز للشحن، وفتحتُ لك طلب مساعدة عاجلاً. ("The order could not be cancelled because it has been prepared for shipping, and I have opened an urgent help request for you.") It is honest only if that case exists in the database.

Reporting completion and communication separately

A single percentage hides the failure this article is about. Report at least two scores per configuration, each with its denominator:

• Operational completion: cases where every repeat reached the required state and avoided forbidden calls.
• Communication quality: cases where every repeat also described the outcome truthfully, in the right register and variety.

Break both down by workflow, branch and Arabic variety; disclose the model, tool-access mode, suite version and repeats. Never pool configurations with different tool access, and never turn a synthetic-suite result into a safety or readiness claim. For the model-level side, see our guide to Arabic LLM evaluation and the explainer on Arabic LLM benchmarks.

Bayanat Labs builds custom evaluations: vetted native speakers of the varieties you serve can write the Arabic cases and grade the replies, calibrated against gold standards with adjudication and an audit trail on every task. See our Arabic LLM evaluation service, which starts with a fixed-scope pilot and a benchmark/quality report, usually within two weeks; Arabic red teaming adds adversarial pressure.