Queries in your users’ words
Formal and informal Arabic, regional vocabulary, mixed-language terminology, and ambiguous requests against Arabic or English sources.
Bayanat Search
Arabic queries paired with the evidence that answers them.
Bayanat Labs (Riyadh) builds Arabic retrieval and RAG evaluation data from the documents your product uses: queries, relevance judgments and grounded answers from a team of 50,000+ expert annotators.
اقرأ بالعربيةWhich Arabic retrieval or RAG evaluation data fits your project?
Start from what your system has to do. Bayanat Search is built to order on your authorized documents. The free starter kit shows the format; it is not a sample of custom delivery.
| Your project | Start with | Check before you commit |
|---|---|---|
| Evaluating Arabic RAG over your own documents | Custom Arabic RAG evaluation set How to score retrieval and answers separately | Test queries kept separate from training, graded judgments over the document versions you authorize, and whether you can score retrieval (did it find the right passage) apart from the answer (did it stay faithful to that passage). |
| Trying the format or testing a pipeline first | Arabic Retrieval Starter Kit (free, synthetic, CC BY 4.0) | It has 12 passages, 12 queries and 30 graded judgments about a fictional company, and the judgments have a single author. It tests a pipeline. It cannot show accuracy on your documents, and it reports no reviewer agreement. |
| Training or tuning a reranker | Custom graded relevance and hard negatives | Whether hard negatives include your outdated policies and rules for other customer groups, and whether grades separate direct evidence, useful background and passages that only mention the topic. |
| Arabic questions over English documents | Custom cross-language queries | Whether the set pairs Arabic or mixed Arabic-English queries with your English sources, and whether you can score those queries apart from same-language ones. |
| Testing what the system does when the answer is missing | Custom unanswerable questions | Whether answerable questions link to exact evidence, whether questions the sources cannot resolve are included, and what response is expected: asking for a detail, or saying the sources do not establish an answer. |
| A held-out test to re-run across model or index releases | Custom Bayanat Bench suite | Held-out items stay separate from training deliveries, with access and versioning defined for the project. Acceptance thresholds stay your decision. |
| Evidence inside scanned forms and contracts | Bayanat Documents | How OCR text, reading order and evidence locations are labeled, and which documents you are authorized to share. |
| Question-answer pairs for training, with no corpus to search | Bayanat Knowledge Arabic Regional Q&A (180,000 pairs) | These are pairs, not relevance judgments over your documents, so they do not measure your retriever. Licensed pairs used in training cannot also serve as a held-out test. |
| Deploying in Saudi Arabia or the UAE | Eight Saudi and UAE procurement checks Custom scope, including hosting location The same checks in Arabic | Which rights cover retrieval and display of your corpus, and which personal-data rules apply; your counsel decides. Confirm the hosting location in the scope: in-region is the default, with private-cloud and on-premises options. |
Saudi and UAE procurement checks for Arabic retrieval and RAG data
Relevance judgments can be accurate and still unusable: the corpus license may not allow passages to be shown in answers, test queries copied from logs may carry customer names and phone numbers, or the system may answer staff from the English version of a policy that was only judged in Arabic. Get a written answer on each check before you sign, from a vendor or your own team, and check it again at delivery.
| Check | What to get in writing | What Bayanat Search provides |
|---|---|---|
| Corpus rights | Who owns the documents, and explicit permission to use them for retrieval and to show passages inside answers, separate from training and redistribution permission. Warning sign: a general "commercial use" clause that mentions neither retrieval nor display. | Built only on the documents, versions and effective dates you authorize. Use and delivery rights for the judgments are set in the agreement. |
| Personal data and hosting | Personal identifiers removed from queries before reviewers see them; where the data is hosted and who can access it; and which law applies, depending on the data subjects and the controller: the Saudi Personal Data Protection Law, UAE Federal Decree-Law No. 45 of 2021, or a financial free zone's own data protection rules, such as those of the DIFC or ADGM. Warning sign: raw logs sent to reviewers or to a model hosted abroad with no assessment of that transfer. | In-region hosting by default, with private-cloud and on-premises options; least-privilege access, audit logs and contributors under NDA. Which law applies is for your counsel to decide. |
| Query and passage language | The language of the query and of the passage in every judgment, with both versions judged when a policy exists in Arabic and English. Warning sign: judgments on the Arabic text only, while the system answers staff from the English version. | Arabic or mixed Arabic-English queries can be paired with your English documents, and you can score those queries apart from same-language ones. |
| Variety mix | Strata with shares taken from your own logs: MSA, Gulf dialects, other dialects some of your users write in, such as Egyptian or Levantine, mixed Arabic-English queries and Arabizi. Warning sign: a dialect mix based on the organization's country rather than on what its users actually type. | Judged by a team of 50,000+ expert annotators, including vetted native speakers and linguists who pass dialect and domain screening, across 25+ Arabic varieties. The variety mix is set in the project scope. |
| Spelling and digits | Test queries in their original spelling: alef and hamza forms (أ إ آ ا), taa marbuta and haa (ة ه), yaa and alef maqsura (ي ى), and Arabic-Indic and Western digits (٣٠ and 30); retrieval results with and without normalization. Warning sign: queries normalized before evaluation, which removes the spellings your users will type. | Agreed in the project scope: which spelling and digit variants the test queries keep. Queries are written in your users' words, including formal and informal Arabic, regional vocabulary and mixed-language terms. |
| Versions and effective dates | The version of each passage and its effective date in the calendar the document uses (Hijri or Gregorian), with the Gregorian equivalent in one format. Warning sign: passages with no effective date, so a repealed policy cannot be told apart from the current one. | Hard negatives include outdated policies and rules for a different customer group when they resemble the right answer. The free starter kit shows the format: every passage carries a version, an effective date and a status. |
| Delivery format | A qrels file that trec_eval can read (query ID, iteration column, passage ID, grade, no spaces inside IDs), a query file, the corpus version that was judged, and reviewer reasoning; plus a sample you review yourself before full production. Warning sign: grades in a spreadsheet with no stable passage IDs, so the evaluation cannot be re-run after the corpus changes. | The free starter kit ships graded relevance judgments (qrels) in TREC format, with the queries and the corpus. For custom work, sample judgments and reviewer reasoning, including disagreements, are shared before production. |
| Test isolation | Test queries excluded from every delivery meant for training, and a defined list of who can access them. Warning sign: the same queries delivered in both training and test data. | Held-out items stay separate from training deliveries, with access and versioning defined for the project. A re-runnable held-out suite can be built as a custom Bayanat Bench suite. Bayanat Bench |
These checks are practical procurement points, not legal advice.
Is Bayanat Search the right fit?
A good fit if
- You are building Arabic or bilingual enterprise search, reranking or RAG.
- You need graded relevance, hard negatives and unanswerable questions built from your own documents.
- Your users ask in Arabic, or mix Arabic and English, while some sources are in English.
What you can check before you commission
We separate what is confirmed on this page, what we send on request, and what is agreed for each project.
| Item | Status | Details |
|---|---|---|
| Volume | Per project | Built to order around your corpus. Query count and relevance depth are agreed with you. |
| Relevance scale | Confirmed | Judgments distinguish direct evidence, useful background and passages that mention a topic without answering the question. |
| Hard negatives | Confirmed | Outdated policies and rules for a different customer group, when they resemble the right answer. |
| Unanswerable questions | Confirmed | Included so you can train and test what the system does when evidence is missing. |
| Arabic queries over English sources | Confirmed | A custom dataset can pair Arabic or mixed Arabic-English queries with English documents. |
| Open worked example | Confirmed | Our free, synthetic Arabic Retrieval Starter Kit shows the format: Gulf and MSA questions, graded qrels, outdated-policy hard negatives and unanswerable questions. See the starter kit |
| Sample judgments and reviewer reasoning | On request | Review them, including disagreements, before production. |
| Who judges | Confirmed | A team of 50,000+ expert annotators, including vetted native speakers, linguists and licensed domain professionals who pass dialect and domain screening, across 25+ Arabic varieties. |
| Quality control | Confirmed | Judgments are calibrated against gold standards with adjudication and rework loops; every task carries an audit trail. |
What we can build together
Graded relevance
Judgments distinguish direct evidence, useful background, and passages that mention a topic without answering the question.
Difficult alternatives
Outdated policies and rules for a different customer group make useful hard negatives when they resemble the right answer.
Grounded answers
Answers link to exact evidence. Include questions the sources cannot resolve, so missing information is part of the training.
How the license covers each use
Training rights do not automatically include display, retrieval or redistribution. Each use is set in the agreement.
Your corpus
Authorized documents, versions and effective dates you choose.
Delivered judgments
Use and delivery rights are set in the agreement.
How we put the data together
Choose the sources
Start with authorized documents, their versions, and effective dates in one domain.
Review the judgments
Check sample queries, relevance grades, and reviewer reasoning, including cases where reviewers disagree.
Evaluate both steps
Test whether retrieval finds the right passage and whether the answer stays faithful to it. Keep test queries separate from training.
For example: Finding the current returns policy
A user asks informally whether an order can be returned. The right passage comes from the current policy for that customer, even if an old version has almost identical wording.
Scope your request
Tell us these details. We come back with a fixed-scope pilot plan on your own data: gold-standard QA with adjudication and rework loops and a quality report, usually within two weeks, hosted in-region by default.
- Your corpus and its languages
- The questions users ask
- Relevance scale and number of queries
- Grounded answers needed?
- Timeline
Common questions
Can Arabic queries retrieve English documents?
Yes. A custom dataset can pair Arabic or mixed Arabic-English queries with English sources. This is useful when employees ask questions in Arabic but company documentation is written in English.
Why include questions with no answer in the source documents?
They help you train and test what the system does when evidence is missing. The expected response may be to ask for a detail or explain that the available sources do not establish an answer.
What should an Arabic RAG evaluation dataset include?
Queries in your users’ words, graded relevance judgments over your authorized documents, hard negatives such as outdated policies, answers linked to exact evidence, and questions the sources cannot answer. Keep test queries separate from training, and score separately whether retrieval found the right passage and whether the answer stayed faithful to it.
Can the free starter kit replace a custom Arabic RAG evaluation set?
No. The Arabic Retrieval Starter Kit has 12 passages, 12 queries and 30 graded judgments about a fictional company, and the judgments have a single author. It shows the format and tests a pipeline. It cannot show how your system performs on your own documents and users, and it is not a sample of a Bayanat dataset.