Bayanat Labs ArabicWER: Arabic WER and CER Scorer
ArabicWER (arabicwer) computes the word error rate (WER) and character error rate (CER) of Arabic speech recognition under six named, versioned normalisation policies side by side, from raw text to lenient orthography, so every score can name the policy it used, for example WER (arabicwer-1/lenient). It comes as a Python package and command line with a jiwer-style API, a specification with exact rules and a deterministic alignment, conformance fixtures with expected values under every policy, a free in-browser calculator, and a case study that scores two open Whisper models on FLEURS Egyptian Arabic under every policy.
| Licence | Apache-2.0 (code); CC BY 4.0 (policies, specification, fixtures and case-study outputs) |
|---|---|
| Version | v0.1.0 · Python package, specification, policies and fixtures v0.1.0, tagged on GitHub. Not on PyPI yet. |
| Published | |
| Updated | |
| Who it is for | Teams that evaluate Arabic speech recognition and need to report exactly how the text was normalised before scoring. |
| Level | Intermediate |
Key numbers
Counted by script from the released files. Each key number is computed from a file in the Hugging Face fixtures dataset (the two fixture files and the case-study summary) and pinned to that file's SHA-256.
Downloads and identifiers
| Where | What |
|---|---|
| bayanatlabs.com | Open the calculator |
| GitHub | Code and specification on GitHub |
| Hugging Face | Fixtures and case study on Hugging Face |
| Hugging Face | In-browser calculator on Hugging Face Spaces |
Get started
Install from GitHub at the v0.1.0 tag (the package is not on PyPI yet). The TSV file holds one id, reference and hypothesis per line.
pip install "git+https://github.com/Bayanat-Labs/arabicwer@v0.1.0"
# Every policy side by side, with the swing
arabicwer score --tsv pairs.tsv
# In Python
>>> import arabicwer
>>> arabicwer.wer("ذهب الولد إلى المدرسة", "ذهب الولد الى المدرسه") # lenient (default)
0.0
>>> arabicwer.wer("ذهب الولد إلى المدرسة", "ذهب الولد الى المدرسه", policy="raw") # no normalisation
0.5What is released
| Part | Licence | In this release |
|---|---|---|
| Python package and command line | Apache-2.0 | Yes |
| Policies, specification and conformance fixtures | CC BY 4.0 | Yes |
| Case study: model outputs, scores and report | CC BY 4.0 | Yes |
| In-browser calculator on Hugging Face Spaces | Apache-2.0 | Yes |
Attribution: ArabicWER (arabicwer) by Bayanat Labs. https://github.com/Bayanat-Labs/arabicwer
Intended uses
- Score Arabic speech recognition under several normalisation policies at once and report the policy with every number.
- See which normalisation rule moves a score, with a word-by-word alignment of substitutions, deletions and insertions.
- Test another WER implementation against the conformance fixtures.
Out of scope
- Ranking speech-recognition models: the case study measures how much a score depends on the policy, not model quality.
- Changing a released policy: any change in behaviour ships under a new policy id.
Rights at a glance
Training, display, redistribution and synthetic-voice use are separate permissions.
| Use | Status | Note |
|---|---|---|
| Training | Allowed | Code under Apache-2.0; policies, fixtures and case-study outputs under CC BY 4.0 with attribution. |
| Display | Allowed | CC BY 4.0, with attribution; the FLEURS reference text keeps its CC BY 4.0 notice (Google). |
| Redistribution | Allowed | Code under Apache-2.0; Bayanat Labs data under CC BY 4.0 with attribution; FLEURS reference text under CC BY 4.0 (Google); Unicode data under the Unicode License V3. |
| Synthetic voice | Not applicable | Text only; no audio is included. |
Limitations
- Install from GitHub for now; the package is not on PyPI yet.
- The case study covers one test set (FLEURS Egyptian Arabic) and two open Whisper models.
- The lenient policy can hide real errors, such as على written as علي, so report a stricter score next to it when the difference matters.
Sources and licences
| Source | Licence | Redistributed here |
|---|---|---|
| FLEURS (google/fleurs), Egyptian Arabic test split | CC BY 4.0 (Google) | Yes |
| Unicode Character Database 18.0.0 | Unicode License V3 | Yes |
The fixtures are synthetic Arabic written by Bayanat Labs. The case study includes the FLEURS reference transcriptions, without audio, under their CC BY 4.0 licence; the model outputs were produced by Bayanat Labs with two open Whisper models. The in-browser calculator bundles data derived from the Unicode Character Database (see NOTICE in the repository). Source details
How to cite
Cite as:
Bayanat Labs (2026). arabicwer: Arabic WER and CER under named, versioned normalisation policies, version 0.1.0. Bayanat Labs. https://github.com/Bayanat-Labs/arabicwer
No DOI. Cite the URL in the citation above and the date you read it.
@misc{bayanatlabs2026arabicwer,
author = {{Bayanat Labs}},
title = {arabicwer: Arabic WER and CER under named, versioned normalisation policies, version 0.1.0},
year = {2026},
month = oct,
howpublished = {\url{https://github.com/Bayanat-Labs/arabicwer}}
}
Changelog
| Date | Version | Change |
|---|---|---|
| v0.1.0 | Python package, command line, specification and fixtures on GitHub; fixtures and the FLEURS case study on Hugging Face; the in-browser calculator on Hugging Face Spaces. |
Corrections and removal requests
If something in this release should not be there, or is wrong, email hello@bayanatlabs.com. We handle removal requests within 30 days, and platform requests within 24 hours. How removal requests work
Related product: Bayanat Voice