# Arabic Preference Calibration Kit v0.1 (synthetic)

Bayanat Labs, 2026-11-06. License: CC BY 4.0 (see LICENSE.txt).

A small, fully synthetic kit for setting up Arabic preference (RLHF, DPO, reward model) labeling:
a written rubric, a record schema, nine gold pairs that each isolate one Arabic rating failure,
illustrative rater labels to test your agreement scripts on, and a DPO-ready export in the
Hugging Face TRL conversational preference format.

Everything here was written by Bayanat Labs for this kit. The companies, policies, raters and
rater labels are fictional. It is not a sample of a Bayanat commercial dataset, not customer
data and not a benchmark: nine pairs are enough to qualify raters and test a pipeline, not to
train or rank models.

## Files

- rubric.md: register policy, two gates (safety, accuracy), five graded criteria (0-4), outcomes
  (A, B, tie, both_unacceptable), strength, reason codes, and the calibration loop.
- schema.json: JSON Schema (draft 2020-12) for one record in pairs.jsonl.
- pairs.jsonl: 9 gold pairs. pair_id, prompt_variety, target_variety, domain, context, messages,
  response_a, response_b, gates, criterion_scores, gold (outcome, strength, reason_codes),
  gold_rewrite (PC09), rationale_en, tests.
- ratings.jsonl: 27 illustrative labels (3 fictional raters x 9 pairs), each marked
  `"illustrative": true`. They exist to exercise agreement and gold-accuracy scripts. They are
  not measurements of anyone.
- dpo.jsonl: the 6 pairs with a gold winner, as prompt / chosen / rejected message lists, with
  pair_id, varieties, strength and reason codes kept as extra columns. Ties, the no-gold pair and
  the both-unacceptable pair are left out on purpose.

## What each pair tests

| Pair | Variety | Tests | Gold |
|---|---|---|---|
| PC01 | Gulf | Register policy changes the label | B, slightly better |
| PC02 | MSA | Length and fluency bias over a wrong fact | B, significantly better |
| PC03 | Egyptian | Translationese and English calques | B, significantly better |
| PC04 | Levantine | Over-refusal of a protective request | B, significantly better |
| PC05 | Gulf with English terms | Grounding in an English source | A, significantly better |
| PC06 | MSA | Forced choice between equivalent answers | tie |
| PC07 | Gulf | Changed instruction in turn 2 | B, significantly better |
| PC08 | MSA | Cultural framing the policy does not cover | no gold: keep disagreement |
| PC09 | Gulf | Both answers wrong | both unacceptable, rewrite |

## Using it

1. Give raters rubric.md and the pairs without the gold, rationale_en or tests fields.
2. Score each rater's outcomes against gold.outcome on the 8 gold pairs (PC08 has none).
3. Compute agreement on the overlap: raw agreement and Fleiss' kappa for outcomes; Krippendorff's
   alpha if you collect 0-4 scores.
4. Run the illustrative ratings.jsonl through the same script first to check it. On those labels:
   gold accuracy is 8/8, 7/8 and 5/8; raw pairwise agreement on the outcome is 17 of 27 rater
   pairs; Fleiss' kappa over the four outcome categories is 0.389.
5. Load dpo.jsonl with the `datasets` library to check that your trainer accepts the shape before
   you send real data.
   PC01, PC02 and PC05 put the source policy in a system message; merge it into the user turn if
   your model's chat template has no system role.

## Citation

Bayanat Labs (2026). Arabic Preference Calibration Kit v0.1. https://www.bayanatlabs.com/research/arabic-preference-calibration-kit

## Contact

hello@bayanatlabs.com. For preference data on your own model outputs: https://www.bayanatlabs.com/products/align