Bayanat Labs Arabic NER Offset Validator

Arabic NER NER annotation validator character offsets BIO tagging Arabic NLP

Key numbers

Counted by script from the released files. Each key number is read from docs/qa/numbers.json, released at tag v0.1.0 and recomputed by the repository's own script, and pinned to its SHA-256. The test-case dataset's MANIFEST.json gives the cases per split and category with checksums, and the web app's MANIFEST.json lists every file it serves with its SHA-256.

Downloads and identifiers

Get started

Install the library and command-line tool from GitHub at the v0.1.0 tag; the package is not on PyPI yet. The web app needs no install.

pip install "git+https://github.com/Bayanat-Labs/arabic-ner-validator@v0.1.0"

arabic-ner-validator check annotations.jsonl --json report.json
arabic-ner-validator check export.jsonl --format doccano --offset-unit utf16 --clitic-policy exclude -v
arabic-ner-validator check train.conll --format conll --scheme BIOES
arabic-ner-validator convert export.jsonl --format spacy --offset-unit codepoint --to BILUO
arabic-ner-validator codes

What is released

Attribution: Arabic NER Offset Validator (arabic-ner-validator) by Bayanat Labs. https://github.com/Bayanat-Labs/arabic-ner-validator

Intended uses

• Check an Arabic NER annotation file before training, delivery or acceptance, with a code for each finding.
• Find offsets counted in the wrong unit, or text that changed after annotation, when data moves between doccano, Label Studio, spaCy, CoNLL and Hugging Face token rows.
• Convert between character offsets and BIOES or BILUO tags, and check tag sequences, in scripts and continuous integration.

Out of scope

• Judging whether an entity label is correct: the validator checks offsets, text, boundaries and tag structure, not the meaning of a label.
• Repairing data without review: suggested offsets are listed as an audit diff, and a repaired copy changes offsets only.

Rights at a glance

Training, display, redistribution and synthetic-voice use are separate permissions.

Limitations

• The offset unit must be declared; the validator never guesses it.
• Some changes leave no trace in a record: when the stored text and the span texts were both normalised after annotation, the validator reports the mismatch but cannot name the cause. The quality report explains each acceptance case that does not pass.
• 96 of the 100 acceptance cases pass; on the build frozen before the acceptance cases were read, 86 passed. The changes made after reading them are listed in the changelog, and no test case was edited.
• A span that ends with Arabic punctuation gets a warning, which a project's own guidelines may allow.
• The test cases are synthetic sentences written for these sets, with places, organisations, public or historical figures or invented names. The Arabic text has not yet been reviewed by a native Arabic writer; the English governs.
• Install from GitHub for now; the package is not on PyPI yet.

Sources and licences

The test cases were written for the validator: synthetic sentences with places, organisations, public or historical figures or invented names, none describing a real private person. The offset-convention matrix was measured with Python, JavaScript in Node and Chromium, spaCy and Hugging Face fast tokenizers on 20 Arabic probe texts; the Label Studio and doccano rows are read from their source code, not measured. Source details

How to cite

Cite as:

Bayanat Labs (2026). Arabic NER Offset Validator (arabic-ner-validator), version 0.1.0. Bayanat Labs. https://github.com/Bayanat-Labs/arabic-ner-validator

• CITATION.cff

No DOI. Cite the URL in the citation above and the date you read it.

@misc{bayanatlabs2026arabicnervalidator,
  author       = {{Bayanat Labs}},
  title        = {Arabic NER Offset Validator (arabic-ner-validator), version 0.1.0},
  year         = {2026},
  month        = oct,
  howpublished = {\url{https://github.com/Bayanat-Labs/arabic-ner-validator}}
}

How we handle citations

Changelog

Corrections and removal requests

If something in this release should not be there, or is wrong, email hello@bayanatlabs.com. We handle removal requests within 30 days, and platform requests within 24 hours. How removal requests work

Related product: Bayanat Documents