Arabic Preference Data for RLHF & DPO: How to Collect It Natively

Arabic preference data is a set of human judgments on model answers to Arabic prompts: which of two or more responses is better, by how much, and why. Reward models, DPO and related methods learn what a good Arabic answer is from these judgments. To be useful, the data has to be collected natively: prompts written in the varieties your users actually write in, and responses judged by raters who speak those varieties, against a rubric that scores dialect, register and cultural fit alongside helpfulness and safety.

Many public Arabic preference datasets are translated or model-generated, with thin documentation, and generic RLHF guides never say what an Arabic rubric should contain. This guide covers why translation breaks the signal, a rubric you can adapt, a worked Najdi example, collection methods, rater matching and calibration, and the formats DPO and reward-model trainers expect.

What is Arabic preference data?

A preference record has a prompt, two or more candidate responses, and a judgment: a pairwise choice, a ranking, a rating on a scale, or a binary acceptable/unacceptable label. Llama 2's annotators, for example, chose between two responses and then rated how strongly they preferred it: significantly better, better, slightly better, or negligibly better/unsure. They judged helpfulness and safety separately.

What makes the data Arabic is not the script. The prompts must reflect how Arabic speakers actually ask, in MSA, dialect or a mix; the responses must cover the choices an Arabic model faces, such as which variety to answer in; and the raters must be able to tell a natural answer from a translated one. Miss any of these and the model learns its data-maker's preferences, written in Arabic.

What public Arabic preference datasets exist?

A few Arabic preference sets are public on the Hugging Face Hub; read the cards before training. As of September 2026, the most visible ones:

The pattern: the open sets in the table are translated, synthetic or undocumented, and the one native human collection is described in a paper, not released with a dataset card. Public sets are fine for testing a pipeline, not for defining what your model should prefer.

Why do translated preference pairs fail?

Translating an English preference set looks efficient, since the judgments already exist. But a judgment belongs to the original text, not the translation.

• Translationese becomes the signal. When Cohere For AI researchers built multilingual preference data covering 23 languages, Arabic among them, they avoided simply translating English pairs. They warned that translation artifacts can hurt performance and that re-translating the same pairs reduces diversity. In their setup, translated completions were ranked as the "bad" completion 91% of the time, so the preference label largely acted as a proxy for "this was translated". Any dataset where translation correlates with one side can teach that shortcut.
• The content stays foreign. Names, holidays, institutions and assumptions come from the source language. When the CIDAR team machine-translated 9,109 English instruction pairs (from AlpaGasus) into Arabic, reviewers modified 64.5% of them, for translation errors or Western-specific content, before they fit an Arabic dataset.
• The dialect axis never appears. Translation output defaults to MSA, so a translated set rarely contains the judgment Arabic users notice first: whether a dialect question should get a dialect or an MSA answer.
• Evaluation inherits the gap. M-RewardBench machine-translated RewardBench into 23 languages, including Arabic, using Google Translate. All reward models tested scored lower on average than on the English original, and scores rose with translation quality. The authors also found that reward-model preferences can change substantially from one language to another.

Translated prompts can still seed coverage, if native raters rewrite them and fresh responses are judged in Arabic. The same applies to model-generated data: as our piece on synthetic Arabic data and model collapse explains, a generator's MSA skew compounds when its output becomes training data.

An Arabic preference rubric

A rubric turns "which answer is better?" into questions a rater can answer consistently. For Arabic, split the criteria into gates, which a response must pass before anything else counts, and graded criteria, which decide between two acceptable answers. The rubric below is a template, not a standard; adapt the scales and the register policy to your product.

Two additions make it workable. First, a register policy: for each product surface, state whether the model answers in MSA, the user's dialect, or light MSA with dialect vocabulary. Without one, raters split on taste and the reward model learns noise. Second, two extra outcomes: tie and both unacceptable. Forcing a choice between two bad answers teaches the model that the less bad one is good. Our free Arabic annotation guidelines template gives you a structure for documenting decisions like these.

A worked example: three answers to a Najdi prompt

The example below is illustrative: it was written for this guide, and the scores are not measured results. It shows how a native rubric separates answers that look alike in translation.

Prompt (Najdi): أبي رسالة قصيرة أعزم فيها الشباب في الدوام على عشاء عندي في البيت ليلة الجمعة بعد العشا. خلها ودية وما تكون رسمية. In English: "I want a short message inviting the guys at work to dinner at my place on laylat al-jumʿa after Isha. Make it friendly, not formal."

ليلة الجمعة (laylat al-jumʿa) is the night before Friday, which is Thursday evening. In common Arabic usage, a night belongs to the day that follows it. A literal English rendering, "Friday night", moves the dinner by a day.

Response A: الزملاء الأعزاء، يسعدني دعوتكم لحضور عشاء في منزلي مساء يوم الجمعة. أرجو تأكيد حضوركم في أقرب وقت ممكن. مع أطيب التحيات. In English: "Dear colleagues, I am pleased to invite you to a dinner at my home on Friday evening. Please confirm your attendance at your earliest convenience. Best regards."

Response B: يا هلا بالشباب، ودي أشوفكم عندي على العشا ليلة الجمعة بعد صلاة العشا. البيت بيتكم، وعطوني خبر عشان أرتب. In English: "Hey guys, I'd love to have you over for dinner on Thursday night after the Isha prayer. My home is your home. Let me know so I can get things ready."

Response C: يا جماعة، عايز أعزمكم على العشا عندي يوم الخميس بالليل بعد العشا. ياريت تيجوا! In English: "Guys, I want to invite you to dinner at my place Thursday night after Isha. Hope you can come!"

The native ranking is B > C > A. C gets facts and tone right but speaks the wrong dialect, a graded miss. A fails a gate: a guest who reads it arrives a day late. Written as pairs, that gives B over A (significantly better), B over C (better) and C over A (better).

Now picture a non-Arabic-speaking reviewer working from back-translations. A comes back as "Friday evening", matching a translated prompt that reads "Friday night", so the error vanishes, and A reads as the most polished. That reviewer could plausibly rank A first; thousands of such judgments teach a reward model that formal MSA with quietly changed details is the best Arabic. Note also why بعد صلاة العشا in B is better than a bare بعد العشا. In speech, العشا means both dinner and the Isha prayer, so naming the prayer removes the ambiguity. Only a fluent rater credits that.

Pairwise, ranking or Likert: which should you use?

The collection method decides which training methods you can use and what each judgment costs in rater effort:

The evidence favours mixing methods. InstructGPT's labelers ranked 4 to 9 responses per prompt; because comparisons from one ranking are highly correlated, shuffling them into one dataset made the reward model overfit, so the authors trained on each prompt's comparisons as a single batch element. Llama 2 used preference strength to set a larger loss margin for clearly different pairs. NVIDIA's HelpSteer2-Preference compared pairwise (Bradley-Terry) and rating-based (regression) reward models on matched data and got its best model by combining the two.

A practical Arabic default: pairwise preferences with a strength label for training, plus 0–4 scores on dialect, register and cultural fit for diagnostics. The scores explain why a checkpoint loses ("accurate but always MSA"), which one preference bit cannot.

How do you match raters to dialect and domain?

A rater can only judge dialect fit in a variety they speak natively, and the prompt's variety and the answer's variety can differ:

• Prompt and answer in the same dialect (a Najdi user, with a policy of mirroring the user): the rater must be a native speaker. A Levantine rater can judge helpfulness, not whether the Najdi sounds natural.
• Dialect prompt, MSA answer (a government or banking assistant): the rater must fully understand the dialect and judge MSA quality. Misreading the question is the common failure.
• Arabizi or mixed-script prompts: raters must know the users' written habits, not just their speech.
• Specialist prompts (medical, legal, financial, religious): a domain professional judges accuracy and a dialect-matched rater judges language.

Record each rater's variety and domain on every row, so you can see whether preferences shift by community. The research expects them to: PRISM, built from 1,500 participants in 75 countries, was designed around cross-cultural disagreement on value-laden prompts. For why variety coverage sets your model's ceiling, see MSA vs dialectal Arabic.

How do you measure rater agreement and calibration?

Agreement shows whether the rubric is applied consistently, not that the labels are right. Build in three controls:

1. Gold pairs. Adjudicated pairs with a written rationale, used to qualify raters, then mixed invisibly into live work. Per-rater gold accuracy is your earliest drift signal.
2. Overlap. A fixed share of items rated by two or more raters. Report raw agreement and a chance-corrected statistic: Cohen's or Fleiss' kappa for pairwise choices, Krippendorff's alpha for ordinal rubric scores.
3. Calibration sessions. Regular reviews of disagreements that end in a rubric clarification or new gold item, not a majority vote.

Set realistic expectations. On its instruction-following prompts, InstructGPT reports that its training labelers agreed with each other 72.6 ± 1.5% of the time, and held-out labelers 77.3 ± 1.3%. Near-perfect agreement on subjective Arabic pairs is a warning sign: raters may be keying on length or formality.

When raters disagree, separate the cause. Rubric errors, such as a missed gate, are adjudicated and reworked. Legitimate disagreement, such as Gulf and Levantine raters differing on what sounds polite, is kept and tagged by variety; averaging it away aligns the model to your rater pool's majority.

Automated judges need the same scrutiny. In our LabelBench audit, a local language model reached 78% exact-match accuracy on binary Arabic toxicity on the fixed held-out manifest. It fell to 25.3% on the real multi-label taxonomy, and 22.4% across five dialect regions (chance is 20%). A grader that cannot identify the dialect cannot score dialect fit. After training, check the gains on a natively written test set, using our Arabic LLM evaluation protocol.

What format do DPO and reward models need?

Hugging Face's TRL documentation (as of September 2026) defines the shapes its trainers expect. A preference row has prompt, chosen and rejected fields, as strings or chat-style message lists. DPOTrainer and ORPOTrainer recommend an explicit prompt; RewardTrainer recommends the implicit form, with the prompt inside both completions. KTOTrainer also accepts unpaired rows: one completion and a true/false label. The docs warn that unpairing is only safe if every chosen response is good and every rejected one bad, which is what a "both unacceptable" label tells you.

Keep trainer columns clean and carry the Arabic-specific information as metadata:

Two hygiene rules. Normalize chosen and rejected text identically (diacritics, tatweel, alef and yaa forms), or the model learns the formatting difference instead of the quality difference. And avoid origin splits: if every chosen answer is a human rewrite and every rejected one is model output, the pair teaches "sound human", the same shortcut the translation study found.

How Bayanat approaches Arabic preference data

Bayanat Labs is a Riyadh-based Arabic data company, and preference ranking is part of our human feedback work. Contributors are vetted native speakers, linguists and licensed domain professionals who pass dialect and domain screening before touching client data. Work is calibrated against gold standards, disagreements go through adjudication and rework loops, and every task has an audit trail. For preference ranking, rubric scores, SFT rewrites and reward-model sets, see our Arabic RLHF and human feedback service. When the goal is safety behaviour, adversarial prompts written in dialect come from our Arabic red teaming work.

Checklist: collecting Arabic preference data

1. Write prompts natively in each target variety; rewrite translated prompts, never use them as-is.
2. Publish a register policy per product surface before rating starts.
3. Sample both responses from your checkpoint or similar-strength models, so pairs differ in quality, not origin.
4. Use gates (safety, accuracy) and graded rows (instructions, dialect, register, culture, language).
5. Allow "tie" and "both unacceptable"; route unacceptable pairs to native rewrites.
6. Match raters to prompt and answer variety; add domain experts for specialist prompts.
7. Qualify raters on adjudicated gold and keep gold mixed into live work.
8. Report overlap agreement per label type, and question suspiciously high agreement.
9. Keep legitimate disagreement, tagged by rater variety; deliver TRL rows with that metadata.
10. Normalize text identically, then validate on a natively written test set, by dialect.