# Inspection rubric: Arabic scene text and grounded visual QA (v0.1)

Use this with the six images and 18 questions in this kit, or with your own images in the same format.

## 1. Protocol

1. **Write the reading policy first.** Decide how answers are compared: whether Eastern Arabic digits (٠١٢٣٤٥٦٧٨٩) and Western digits (0123456789) count as the same, whether alef forms (أ إ آ ا), final ya and alif maqsura (ي ى), and ta marbuta and ha (ة ه) are normalized, and whether abbreviations such as ص and م (a.m. and p.m.) or كم (km) must be expanded. Write it down before you run anything; section 2 is a default.
2. **Run two tasks separately.**
   - *Reading:* ask the model to transcribe all text in the image, in reading order, one line per text region. Compare with the `transcription` fields in `annotations.jsonl`.
   - *Answering:* give the image and one question from `questions.jsonl` at a time, in Arabic. Ask for a short answer and, if the model supports it, a box for where the answer is.
3. **Use the same prompt and settings for every model.** Fix the temperature or seed, keep the first answer and store the raw output.
4. **Mark each answer** pass or fail, with an error code from section 3 for each failure. A failure can have more than one code.
5. **Check grounding** where the model returns a box: section 4.
6. **Report by error code, not one score** (section 5). Keep the failing outputs and rerun the same items after a fix.

## 2. Default normalization policy

Applied to both the reference and the model output before comparing, unless your written policy says otherwise:

- Digits: Eastern Arabic and Western digits are treated as equal (٤ = 4). The value must match exactly.
- Alef forms أ إ آ are mapped to ا; ى to ي; ة to ه; tatweel (ـ) and diacritics are removed.
- Extra spaces and final punctuation are ignored.
- Units must be present when the question asks for an amount: "٣٠" alone is a QA_UNIT failure where the reference is "٣٠ مل".
- An answer in English to an Arabic question is accepted only if your policy allows it; the kit's questions expect Arabic answers.

Under this policy a dot error (مخبز read as مخيز) is still a failure, because ب and ي are different letters.

## 3. Error codes

### Reading

| Code | Failure |
|---|---|
| TXT_DOT | A letter read as another with the same shape and different dots (ب ت ث ن ي, ج ح خ, س ش, ص ض, ط ظ, ع غ, ف ق) |
| TXT_ORDER | Words or lines in the wrong order, for example Arabic written out left to right, or a two-column menu read across the wrong row |
| TXT_SCRIPT_MIX | Arabic and Latin text in one region merged, swapped or translated instead of transcribed |
| TXT_DIGITS | A digit misread, Eastern Arabic digits confused with similar shapes (٠ dropped as a dot, ٥ read as 0, ٧ and ٨ swapped), or a percent sign or date order lost |
| TXT_ABBREV | An abbreviation expanded wrongly or dropped: ص and م for a.m. and p.m., كم, مل, ر.س |
| TXT_MISSING | A text region the model did not read |
| TXT_HALLUC | Text the model reports that is not in the image, including text under an occluder |

### Answering

| Code | Failure |
|---|---|
| QA_WRONG | A wrong value with no other explanation |
| QA_WRONG_REGION | The answer comes from another text region, for example the production date given as the expiry date |
| QA_ROW | The value comes from the wrong row of a sign, table or menu |
| QA_DIRECTION | Left and right swapped, for example an arrow read as mirrored because the text is right to left |
| QA_NEGATION | A prohibition or exception read as its opposite (لا يخلط, ما عدا) |
| QA_UNIT | A number given without its unit or with the wrong unit (كم, لتر, مل, ريال) |
| QA_NO_ABSTAIN | An answer given to a question the image cannot answer (`"answerable": false`) |
| QA_FALSE_ABSTAIN | The model says it cannot answer an answerable question |
| GND_BOX | The returned box does not overlap any evidence region with an intersection over union of 0.5 or more |

## 4. Grounding and abstention

- Each answerable question lists its `evidence_regions`. A box passes if its intersection over union (IoU) with at least one listed region is 0.5 or more. Text boxes in this kit cover the band the line was drawn in, so a tight word box can score lower; check failures by eye before you report them.
- Two questions cannot be answered from the image. `unanswerable_reason` is `not_present` (the information is not in the image, Q04) or `not_legible` (it is in the image but covered, Q17). The expected response says so in Arabic; any specific value is QA_NO_ABSTAIN, and a guessed time for Q17 is also TXT_HALLUC.
- A "no" answer is not an abstention: Q13 (orange juice is not on the menu), Q15 (do not mix with chlorine) and Q18 (the ban does not apply on Friday) have answers in the image.

## 5. Reporting

Report counts per error code and per question, with the model name, version, prompt and settings. Do not turn 18 questions into a percentage score: the kit is a smoke test that shows which failure types occur, not how often they occur on your images. A model that fails every digit question needs different handling than one that ignores the sticker in SC06.

## 6. Record checks to agree with a supplier

Before a custom image or video collection, agree that every image record states, as in `schema.json`:

- **Source type:** field capture, client-supplied or generated, with `generated` true or false, so results on real photos can be reported on their own.
- **Capture conditions:** lighting, angle, distance, blur, glare and occlusion, so you can see which conditions the data covers.
- **People in frame:** whether people appear and how they were handled under the agreed permissions. Labels describe what is visible and never infer anyone's identity from appearance.
- **Split group:** the store, location, source or recording an image belongs to, so all images from one place stay in the same split.
- **Rights, each separately:** model training, evaluation, redistribution and public display, each `granted`, `not_granted` or `not_stated`, with a reference to the permission record. A right that is not written down is `not_stated`, never assumed.
