Arabic Text Normalizer

arabic-normalize by Bayanat Labs applies a named normalisation policy, such as bayanat-norm-v1, to Arabic text and shows exactly what it changed: the normalised text, a change log with the position of every change, the lossy changes (for example ى → ي), the diacritization ratio and the SHA-256 of the policy. A paper, a WER score or a dataset card can then say which normalisation it used, and anyone can reproduce the text exactly.

Open the normalizer · Code on GitHub · Conformance fixtures on Hugging Face

The normalizer runs in your browser. Your text is not uploaded, stored or logged.

Worked example

Input: إِلَى المَدْرَسَةِ الجَدِيدَةِ ١٢٣ ("to the new school 123", written with harakat and Arabic-Indic digits)

Under bayanat-norm-v1 the harakat are removed (they count towards the diacritization ratio, 0.647 here), the hamza alef of إلى becomes a bare alef and its final ى becomes ي (both logged as lossy), the Arabic-Indic digits become 123, and the taa marbuta ة is kept and flagged. Only asr-eval-v1, the policy for speech-recognition scoring, folds ة to ه. raw-safe-v1 changes no letter, mark or digit. The two lossy changes in the change log:

How it is computed

1. A policy names a Unicode form (NFC, NFKC or none) and sets one mode for each of 18 stages, from invalid characters, invisible controls and presentation forms to tatweel, harakat, alef variants, alef maqsura, taa marbuta, digits, punctuation and whitespace. The stages always run in the same order, with a final NFC pass.
2. Character data comes from the Unicode 18.0.0 database bundled with the library, not from the Unicode version of your programming language or browser, so the output does not change when they do.
3. Every change is logged with its position in the original text, and an offset map links each output character back to the input.
4. Changes that can make two different words identical, such as إنّ and أنّ, are flagged as lossy.
5. The policy hash is the SHA-256 of the policy's canonical JSON. A released policy never changes: a change in behaviour ships as a new policy id, such as bayanat-norm-v2.

Policies

Use it in code

# Python 3.9 or later
pip install "git+https://github.com/Bayanat-Labs/arabic-normalize@3ad22326a66596e8dc413321017251eed6e60439"

# JavaScript (Node 20 or later, browsers): the js/ folder of a clone at the same commit
git clone https://github.com/Bayanat-Labs/arabic-normalize.git
git -C arabic-normalize checkout 3ad22326a66596e8dc413321017251eed6e60439
npm install ./arabic-normalize/js

The PyPI and npm packages have not been released yet. Until they are, install from GitHub, pinned to a commit, as above; npm links the package to the cloned folder, so keep it. Installs from PyPI and npm will work once the packages are published.

from arabic_normalize import normalize
normalize("إِلَى المَدْرَسَةِ الجَدِيدَةِ ١٢٣")   # 'الي المدرسة الجديدة 123'

import { normalize } from '@bayanat-labs/arabic-normalize';
normalize('إِلَى المَدْرَسَةِ الجَدِيدَةِ ١٢٣');  // 'الي المدرسة الجديدة 123'

The Python and JavaScript libraries give the same output: 80 of 80 conformance fixtures pass in JavaScript and 80 of 80 in Python, and 560 of 560 reference cases are identical between the two, offset map and change log included. In a stress run on 113,752 Arabic Wikipedia articles (raw wikitext), every policy ran with 0 crashes and 0 offset-map or change-log errors, and bayanat-norm-v1 flagged at least one lossy change in 112,965 of them (99.3%).

Cite as

Normalised with bayanat-norm-v1 (sha256 ebdb82000d07…), arabic-normalize 0.1.0, Bayanat Labs.

@software{bayanatlabs_arabic_normalize_2026,
  title   = {arabic-normalize: Arabic text normalisation with named, versioned policies},
  author  = {{Bayanat Labs}},
  year    = {2026},
  version = {0.1.0},
  url     = {https://github.com/Bayanat-Labs/arabic-normalize},
  license = {Apache-2.0}
}

Name the policy id and its hash in every paper, score or dataset card. A DOI will be added with the first archived release.

Licence and contact

Code: Apache-2.0. Policy files and conformance fixtures: CC BY 4.0. Questions and corrections: hello@bayanatlabs.com.

Need production-grade, rights-cleared Arabic data?

arabic-normalize is free. For licensed Arabic text and knowledge datasets with documented normalisation, see Bayanat Knowledge.