Arabic preference data calibration kit

Arabic preference data is only as good as the people choosing between the two answers, and the usual agreement numbers do not show whether they chose for the right reasons. Raters can agree with each other and still reward the longer, more polished answer even when it invents a fee, or pick the formal reply when the product is meant to answer in the user's dialect. A 2023 study of RLHF found that improvements in reward were largely driven by longer responses, and traced the bias mainly to the reward model. Gold pairs with a written rationale catch these errors before they reach your reward model.

This kit is a small set of such gold pairs, written by Bayanat Labs in Modern Standard Arabic and in Gulf, Egyptian and Levantine Arabic, with the rubric they were judged against and a schema for the records. Use it to qualify raters, to test your agreement and gold-accuracy scripts, and to check that your trainer accepts the file before real data arrives. It is fully synthetic: the companies, policies, raters and their labels are fictional.

What is in the kit

Nine pairs, nine rating failures

A worked example: the longer answer is wrong

In PC02 the source policy of a fictional bank says a card can be frozen from the app at any time, with no fee, and unfrozen from the same screen. The user asks, in MSA, how to freeze the card and whether there is a fee.

• Response A opens warmly, explains the steps, adds security tips, and states تُطبَّق رسوم رمزية قدرها 10 ريالات عند إعادة التفعيل (“a nominal 10-riyal fee applies on reactivation”).
• Response B says, in two sentences, that the card can be frozen from the app at any time, there is no fee, and it can be reactivated from the same screen.

A fails the accuracy gate because the fee contradicts the policy, so its style scores are not recorded and B wins, significantly. A rater who prefers A is rewarding length and polish. In the illustrative labels, one fictional rater does exactly that, also calls PC01 a tie, and picks a winner in PC09, where both answers are wrong. That rater's gold accuracy is 5 of 8, against 8 of 8 and 7 of 8 for the other two, which is the signal to hold a calibration session before the rater's labels go into training data.

How to calibrate raters with it

1. Write the register policy first. Every pair carries target_variety: mirror_user (answer in the user's variety) or msa. PC01's label depends on it, and PC08 shows what happens when the policy is silent.
2. Qualify. Give raters the rubric and the pairs without the gold, rationale_en and tests fields. Score their outcomes against the 8 pairs that have a gold label. Set your pass mark before you start.
3. Measure agreement on overlap. Report raw agreement and a chance-corrected statistic: Fleiss' kappa for the outcome, Krippendorff's alpha for the 0–4 scores. On the illustrative labels, raw pairwise agreement is 17 of 27 rater pairs and Fleiss' kappa over the four outcomes is 0.389. Run your own script on ratings.jsonl and check that it reproduces these numbers.
4. Separate the causes of disagreement. A missed gate or a misread instruction is a rubric error: correct it and rework. A difference in taste or community norms, as in PC08, is kept and tagged with the rater's variety, then settled by a policy, not a majority vote.
5. Keep gold pairs in live work. Mix them in unmarked and track each rater's gold accuracy over time; it is the earliest sign of drift.

For realistic agreement levels, InstructGPT reports that its training labelers agreed with each other 72.6% (±1.5%) of the time, and held-out labelers 77.3%. Our guide to Arabic preference data covers the rubric choices, rater matching and file formats in more depth, and LLM-as-a-judge in Arabic covers when a model can pre-sort pairs.

Load the DPO export

Each line of dpo.jsonl keeps the trainer columns clean and carries the Arabic-specific fields alongside:

{"prompt": [{"role": "system", "content": "Context: …"}, {"role": "user", "content": "كيف أوقف بطاقتي مؤقتًا؟ وهل هناك رسوم؟"}],
 "chosen": [{"role": "assistant", "content": "يمكنك إيقاف البطاقة مؤقتًا من التطبيق في أي وقت، ولا توجد رسوم. …"}],
 "rejected": [{"role": "assistant", "content": "يسعدنا مساعدتك! … تُطبَّق رسوم رمزية قدرها 10 ريالات …"}],
 "pair_id": "PC02", "prompt_variety": "msa", "target_variety": "msa",
 "strength": "significantly_better", "reason_codes": ["accuracy_gate", "invented_detail", "length_bias_trap"]}

PC01, PC02 and PC05 put the source policy in a system message; merge it into the user turn if your model's chat template has no system role. Ties, the pair with no gold label and the both-unacceptable pair are left out on purpose. The PC09 rewrite in pairs.jsonl belongs in a supervised fine-tuning set, not a preference set.

What this kit is not

• Not training data. Nine pairs are enough to qualify raters and test a pipeline, not to train or rank models.
• Not real raters. The three raters and their 27 labels are illustrative, written to exercise the scoring.
• Not a sample of a Bayanat dataset. It shows the record format and the cases; commercial preference data is licensed or built to order.
• Not every variety. v0.1 covers MSA, Gulf (Saudi), Egyptian and Levantine Arabic. Your project should cover the varieties your users write in.

Build preference data on your own model outputs with Bayanat Align

Bayanat Align takes the same record design to production on your assistant's real requests:

• Raters screened for your users' varieties. A team of 50,000+ expert annotators, including vetted native speakers, linguists and licensed domain professionals across 25+ Arabic varieties, who pass dialect and domain screening first (Arabic RLHF services).
• Gold standards and an audit trail. Work is calibrated against gold standards with adjudication and rework loops, and every task carries an audit trail (how we measure agreement).
• Disagreement kept, not averaged. Ties and reviewer disagreement stay visible in preference records instead of being forced into one winner (Bayanat Align).
• Separate modules. Instruction examples, preference comparisons and response repairs are separate modules, and existing Arabic alignment sets can be licensed alongside our Arabic smartphone speech dataset (versions and counts on request).
• A held-out check. A separate evaluation set can be built for the project to compare model versions without testing on training examples (Bayanat Bench).
• A pilot first. A fixed-scope pilot with gold-standard QA and a quality report, usually within two weeks, with in-region hosting by default (start a pilot).

Cite or reuse

Free to use, adapt and redistribute under CC BY 4.0 with attribution:

Bayanat Labs (2026). Arabic Preference Calibration Kit v0.1. https://www.bayanatlabs.com/research/arabic-preference-calibration-kit

Found an error, or want gold pairs for a specific domain or variety? Write to hello@bayanatlabs.com.