Arabic scene-text and visual QA inspection kit
Six synthetic images, 40 boxed text regions and 18 Arabic questions for checking whether a vision-language or OCR model reads Arabic in images correctly and answers from the right place. Each question targets a known way models can fail: Eastern Arabic digits, abbreviations, bilingual signs, rows on a menu, arrow direction, negation and text it cannot see.
A vision-language model can describe a street scene fluently and still misread the sign in it. With Arabic the errors are specific: a model can read the Eastern Arabic digit ٥ as a zero, drop the م that makes a closing time p.m., pair a menu item with the price on the row below, or mirror an arrow because the text around it runs right to left. Ask about something a sticker covers and it may answer anyway.
This kit is a short, repeatable test for those errors. Bayanat Labs drew six scenes with Saudi conventions (riyal prices, Eastern Arabic digits, Riyadh-area place names), marked every text region with a box and its transcription in reading order, and wrote 18 questions in Arabic, each tied to the regions that hold the answer. Two questions cannot be answered from the image, on purpose. Use it to compare models or versions before a launch, to check an OCR pipeline on Arabic text in scenes, and to agree an image-record format with your data supplier before a custom collection.
بالعربية: مجموعة مجانية ومفتوحة لفحص قدرة نماذج الرؤية واللغة وأنظمة التعرّف الضوئي على الحروف (OCR) على قراءة النص العربي في الصور والإجابة عن أسئلة مرتبطة بموضع الدليل في الصورة: ست صور اصطناعية (واجهة مخبز، ولوحة إرشادية للطريق، وعبوة حليب، وقائمة مشروبات، وملصق منظف أرضيات، ولوحة منع وقوف يغطي ملصقٌ جزءًا منها)، فيها 40 منطقة نصية بإحداثياتها ونصها بترتيب القراءة، و18 سؤالًا بالعربية لكل منها إجابة مرجعية ومناطق الدليل، منها سؤالان لا يمكن الإجابة عنهما من الصورة. وتضم المجموعة دليل فحص بقواعد لتوحيد الكتابة (الأرقام وأشكال الألف والتاء المربوطة) و16 رمزًا للأخطاء، ومخططًا لسجل الصورة يحدد نوع المصدر وظروف التصوير وطريقة التعامل مع الأشخاص ومجموعة التقسيم، ويفصل كل حق من حقوق الاستخدام عن غيره. الصور رسوم أعدّتها Bayanat Labs، وليست صورًا فوتوغرافية، ولا يظهر فيها أشخاص، والمنشآت فيها متخيَّلة. لبيانات الصور والفيديو المخصصة، راجع بيانات الصور والفيديو العربية.
What is in the kit
| File | Contents |
|---|---|
| images/SC01.png to SC06.png | Six 1200 × 800 PNG illustrations drawn by Bayanat Labs: a bakery storefront, a road direction sign, a milk carton, a cafe menu board, a floor-cleaner label and a parking notice partly covered by a sticker. No people, no real brands. |
| questions.jsonl | 18 Arabic questions with English glosses, reference answers, the evidence region IDs, an answerable flag and the error codes each question can catch. Two are unanswerable: one because the information is not in the image, one because it is covered. |
| annotations.jsonl | One record per image: 40 text regions (34 Arabic, 6 Latin) with a box, the text as written in reading order, a script tag and a digit tag (13 regions use Eastern Arabic digits), 4 non-text regions (three arrows and the sticker), the questions, capture conditions, source type and rights. |
| rubric.md | A six-step protocol, a default normalization policy for digits, alef forms and ta marbuta, 16 error codes for reading and answering, the grounding and abstention rules, and the record checks to agree with a supplier. |
| schema.json | JSON Schema (draft 2020-12) for one annotated image: source type and a generated flag, capture conditions, people in frame, split group, text regions, grounded questions and a separate field for each right. |
| README.md, SHA256SUMS.txt, LICENSE.txt | Notes, checksums for every file, and the CC BY 4.0 license. |
SC02, road sign. Three destinations, each with an Arabic name, an English name, a distance in Eastern Arabic digits and an arrow. Q06 asks which way the arrow to the airport points (left); a model that mirrors directions for right-to-left text says right.
Six scenes and what each tests
| Scene | What is in it | Questions and what they test |
|---|---|---|
| SC01 storefront | Sign مخبز الحي with "Al Hay Bakery" below, a sticker خصم ٢٠٪, hours ٦ ص to ١١ م | Q01 the Arabic name (dots, Arabic vs English); Q02 the discount (digits and the Arabic percent sign); Q03 closing time (م means p.m., and not the opening time); Q04 phone number, which is not in the image |
| SC02 road sign | Riyadh, Al Kharj and the airport, each with a distance and an arrow | Q05 distance to Al Kharj (right row, unit); Q06 arrow direction; Q07 the closest destination (compare Eastern Arabic numbers) |
| SC03 milk carton | Product name, ١ لتر, a production date and an expiry date one week apart, storage line | Q08 expiry date (not the production date); Q09 volume with its unit; Q10 storage |
| SC04 menu board | Four drinks in a right-hand column, prices in a left-hand column, "prices in Saudi riyals" footnote | Q11 and Q12 prices (the right row, the currency from the footnote); Q13 orange juice, which the menu does not list |
| SC05 product label | Dilution instruction يخفف ٣٠ مل في ٥ لتر ماء and the warning لا يخلط مع الكلور | Q14 the amount with its unit; Q15 whether it can be mixed with chlorine (negation) |
| SC06 parking notice | ممنوع الوقوف, من ٨ ص إلى with the end time under a sticker, ما عدا يوم الجمعة | Q16 start time; Q17 end time, which is covered, so the correct response is that it cannot be read; Q18 Friday (the exception) |
How to inspect a model with it
- Write the reading policy first. Decide whether Eastern Arabic and Western digits count as equal, which letter forms you normalize and whether abbreviations must be expanded.
rubric.mdhas a default; change it before you run, not after. - Test reading and answering separately. First ask for all the text in each image, in reading order, and compare it with the transcriptions. Then ask the 18 questions one at a time, in Arabic, with the image.
- Keep the first output. Use the same prompt and settings for every model, and fix the temperature or seed, so two models or two versions are compared on equal terms.
- Check where the answer came from. If the model returns a box, compare it with the question's evidence regions (intersection over union of 0.5 or more). A right answer from the wrong place is luck.
- Report by error code, not one score. A model that fails every digit question needs different work from one that answers Q17 with a guessed time. Keep the failing outputs and run the same items after a fix.
Eighteen questions are a smoke test, not a benchmark. The images are clean illustrations; real photos add glare, blur, angle and handwriting. Before launch, add your own images and questions in the same format. Our 10-point audit for Arabic image and video datasets covers what to check in the data itself.
SC06, parking notice. The end time is drawn under the sticker and cannot be seen. Q17 asks for it; the correct response says it cannot be read.
A worked example: the answer the image does not give
Q17 asks حتى أي ساعة يمنع الوقوف؟ ("Until what time is parking banned?"). The notice reads من ٨ ص إلى and then a sticker. The failure to look for is a plausible guess such as 4 or 5 p.m.; the image cannot support either. The question is recorded as:
{"question_id": "Q17", "scene_id": "SC06",
"question_ar": "حتى أي ساعة يمنع الوقوف؟", "answerable": false,
"unanswerable_reason": "not_legible",
"evidence_regions": ["SC06-R2", "SC06-O1"],
"expected_response_ar": "لا يمكن معرفة ذلك، لأن الملصق يغطي وقت الانتهاء.",
"tests": ["QA_NO_ABSTAIN", "TXT_HALLUC"]}
Q04 (the bakery's phone number) is the other kind: the information is simply not in the image (not_present). Q13, Q15 and Q18 are different again: their answer is "no", and the image shows why. Keeping these three cases apart is what tells you whether a model abstains for the right reason.
Agree the image record before you collect
The schema gives every image the fields a buyer needs to trust a dataset later. source.type is field capture, client-supplied or generated, and generated is a separate flag, so results on real photos can be reported on their own; every image in this kit is marked generated. capture_conditions records lighting, angle, distance, blur, glare and occlusion. people_in_frame states whether people appear and how they were handled; labels describe what is visible and never infer anyone's identity from appearance. split_group names the store, location or source, so all images from one place stay in one split. rights has one field each for model training, evaluation, redistribution and public display, each granted, not_granted or not_stated, with a pointer to the permission record. A right that is not written down is not_stated, never assumed.
What this kit is not
- Not photographs. The six images are flat illustrations rendered to PNG, with even lighting and no blur. Passing them says little about real photos; failing them is a clear signal.
- Not a sample of a Bayanat dataset. Bayanat Vision is built to order. Bayanat Labs drew these scenes for this kit; the businesses and products are fictional.
- Not a benchmark. It has no scores and ranks no model. Eighteen questions show where a model fails; they do not measure how often.
- Not every script style or region. v0.1 uses one typeface and Saudi conventions. Handwriting, calligraphic shop signs, Maghrebi conventions and video are not covered.
Build Arabic image and video data with Bayanat Vision
Bayanat Vision builds the real-photo version of this kit to your scope:
- Labels that show where an answer comes from. Scene text, bounding boxes, object relationships, answers linked to visible evidence and, for video, event timestamps (Bayanat Vision label types).
- A capture plan around your task. Lighting, capture conditions, text styles, layouts and difficulty planned for the signs, menus, packaging and forms your users meet (how to scope an image or video project).
- Generated images kept apart. Generated images remain identifiable separately from field collection, as every image in this kit is (Bayanat Vision).
- Permissions first, no identity guessing. Source and media permissions are agreed before collection, and the collection is described without inferring people's identities from appearance (use rights).
- Arabic readers from a vetted team. A team of 50,000+ expert annotators, including vetted native speakers and linguists across 25+ Arabic varieties, who pass dialect and domain screening and are calibrated against gold standards (Arabic data annotation services).
- A pilot first. A fixed-scope pilot with gold-standard QA and a quality report, usually within two weeks, with in-region hosting by default and private-cloud or on-premises options (start a pilot).
Cite or reuse
Free to use, adapt and redistribute under CC BY 4.0 with attribution:
Bayanat Labs (2026). Arabic Scene-Text and Visual QA Inspection Kit v0.1. https://www.bayanatlabs.com/research/arabic-scene-text-vqa-kit
Found an error, or want scenes for another region, script style or domain? Write to hello@bayanatlabs.com.