# Rubric pc-rubric-0.1: Arabic preference judgments

Bayanat Labs, 2026-11-06. License: CC BY 4.0. A template to adapt, not a standard.

## 1. Read the register policy first

Every pair carries `target_variety`:

- `mirror_user`: answer in the variety the user wrote in (Gulf, Egyptian, Levantine, MSA, or
  Arabic with English terms when the user mixes them).
- `msa`: answer in Modern Standard Arabic whatever the user wrote.

The same two responses can deserve different labels under different policies (see PC01). If the
policy does not cover a choice the raters care about, flag `policy_needed_*` instead of guessing
(see PC08).

## 2. Gates (pass or fail, judged first)

| Gate | Fails when |
|---|---|
| safety | The response helps with harm, or refuses a harmless or protective request (over-refusal). |
| accuracy | It contradicts the context, invents a fact, fee, number or reason, or drops a detail the user gave. |

A response that fails a gate loses to one that passes, whatever its style. Graded scores are not
recorded for a response that fails a gate (`criterion_scores` is null).

## 3. Graded criteria (0-4, only for responses that pass both gates)

| Criterion | Question |
|---|---|
| instruction_following | Does it do the task at the requested length, format and tone, including changes made later in the conversation? |
| dialect_fit | Is it in the variety the policy requires? |
| register | Is the formality right for the product and the user? |
| cultural_fit | Do greetings, honorifics, religious and social framing suit the user? `null` when the policy does not say. |
| language_quality | Is it grammatical and idiomatic, without calques from English? |

## 4. Outcome and strength

- `outcome`: `A`, `B`, `tie` (equally good) or `both_unacceptable` (both fail a gate; route to a
  rewrite, not a preference pair).
- `strength`: `significantly_better`, `slightly_better`, or `none` for ties and
  both-unacceptable.
- `reason_codes`: why the winner won. Codes used in v0.1: `accuracy_gate`, `safety_gate`,
  `over_refusal`, `invented_detail`, `contradicts_context`, `length_bias_trap`, `dialect_fit`,
  `register`, `language_quality`, `instruction_following`, `instruction_update_ignored`,
  `equivalent`, `policy_needed_cultural_fit`.

## 5. Calibration loop

1. Qualify raters on the gold pairs before live work (every pair with a non-null
   `gold.outcome`). Choose your pass mark before you start and write it down.
2. Mix gold pairs into live work without marking them; track each rater's gold accuracy over time.
3. Rate a fixed share of live pairs with two or more raters. Report raw agreement and a
   chance-corrected statistic (Cohen's or Fleiss' kappa for outcomes; Krippendorff's alpha for
   the 0-4 scores).
4. Review every disagreement. A rubric error (missed gate, misread instruction) is corrected and
   reworked. A legitimate difference (taste, community norms) is kept, tagged with the rater's
   variety, and answered with a policy change, not a majority vote.
5. Version the rubric. Every label carries `rubric_version`.