Arabic WER and CER Calculator

ArabicWER by Bayanat Labs computes the word error rate (WER) and character error rate (CER) of Arabic speech recognition under six named normalisation policies side by side. Paste reference and hypothesis transcripts, one utterance per line, or upload a TSV file: the calculator shows the score under every policy, the swing between them, and a word-by-word alignment that labels every substitution (S), deletion (D) and insertion (I). Report the policy with every number, for example WER (arabicwer-1/lenient).

Open the calculator · Code and specification on GitHub · Fixtures and case study on Hugging Face

Scoring runs in your browser. Your text is not uploaded, stored or logged.

Worked example

The calculator opens with 7 short pairs, each with one kind of difference between the reference and the hypothesis:

Corpus scores under each policy:

The same transcripts score 56.00% WER without normalisation and 8.00% under lenient. Lenient folding also hides real errors (على “on” against علي, a name), so the calculator always shows the stricter scores next to it.

Why the policy matters

On 100 FLEURS Egyptian Arabic test utterances (18.8 minutes of audio), transcribed with two open Whisper models, the corpus WER moves by 15.7 points for whisper-large-v3-turbo and 13.4 points for whisper-small depending on the policy alone:

Most of the swing comes from punctuation and harakat: the hypotheses and the references punctuate and vocalise differently. These numbers measure how sensitive the score is to normalisation on a public test set, not model quality.

Should I remove diacritics before computing WER?

Usually yes, if the reference and the system do not vocalise in the same way: on the 100 FLEURS utterances, keeping harakat added 4.55 points of WER for whisper-large-v3-turbo under lenient. But say so: report the policy with the number, for example WER (arabicwer-1/dediac) or WER (arabicwer-1/lenient).

How it is computed

1. Each policy normalises both texts with arabic-normalize, with Unicode 18.0.0 data bundled, so the result does not depend on the browser or the Python version.
2. Words are the runs of characters between white space; CER counts the code points of the words joined by single spaces.
3. Words are aligned with the Levenshtein distance and a fixed tie-break rule written in the specification.
4. WER = (S + D + I) / N, summed over all pairs before dividing.
5. A released policy never changes: a change in behaviour ships under a new policy id, and every policy is identified by the SHA-256 of its file.

The calculator gives the same numbers as the Python package on all 160 conformance fixtures under every policy.

Policies

Use it in code

pip install "git+https://github.com/Bayanat-Labs/arabicwer@v0.1.0"

# Every policy side by side, with the swing; --format html writes an RTL-safe word diff
arabicwer score --tsv pairs.tsv

The PyPI package has not been released yet. Until it is, install from GitHub at the release tag, as above; pip install arabicwer will work once the package is published. The TSV file holds one id, reference and hypothesis per line.

>>> import arabicwer
>>> ref, hyp = "ذهب الولد إلى المدرسة", "ذهب الولد الى المدرسه"
>>> arabicwer.wer(ref, hyp)                    # default policy: lenient
0.0
>>> arabicwer.wer(ref, hyp, policy="raw")      # no normalisation
0.5

Cite as

Scored with arabicwer-1/lenient (sha256 b097f6eac978…), arabicwer 0.1.0, Bayanat Labs.

@software{bayanatlabs_arabicwer_2026,
  author  = {{Bayanat Labs}},
  title   = {arabicwer: Arabic WER and CER under named, versioned normalisation policies},
  version = {0.1.0},
  year    = {2026},
  url     = {https://github.com/Bayanat-Labs/arabicwer},
  license = {Apache-2.0}
}

Name the policy id with every score. The release record, with licences and changelog, is ArabicWER on Bayanat Labs Open.

Licence and contact

Code: Apache-2.0. Policies, specification, fixtures, case-study outputs and scores: CC BY 4.0. FLEURS reference text: Google, CC BY 4.0. Questions and corrections: hello@bayanatlabs.com.

Need production-grade, rights-cleared Arabic speech data?

ArabicWER is free. For licensed Arabic speech data and evaluation sets, see Bayanat Voice.