Arabic Token Counter for LLM Tokenisers

The Arabic token counter is the command line and Python package behind the Arabic Token-Tax Index by Bayanat Labs. It counts the tokens of any text with the 12 LLM tokenisers in the index, prints the words it counts, and re-runs the whole measurement from the public sources. The leaderboard shows the results in your browser.

Open the leaderboard · Code on GitHub · Dataset on Hugging Face

How many more tokens Arabic takes

On the United Nations 2015 test set (3,990 Arabic–English sentence pairs from 731 UN documents), the median Arabic/English token ratio across the 12 tokenisers is 1.42: the same content takes 1.42 times as many tokens in Arabic as in English. The ratio runs from 0.93 (ALLaM) to 3.35 (Falcon3), and 11 of the 12 tokenisers need more tokens for Arabic. A ratio below 1 means Arabic took fewer tokens than English.

The ratio is total Arabic tokens divided by total English tokens over the same sentence pairs; the 95% interval comes from 10,000 bootstrap resamples of UN documents. Words are runs of letters, marks and digits, with Arabic clitics left attached, so compare the ratio, not the per-word figures. Fewer tokens means lower cost and more text per context window; it is not a measure of model quality.

What a page costs

A page is the content of 1,000 English words; its Arabic translation on the UN test set has about 892 words. Input price in US dollars as listed on the vendors' official price pages on 16 October 2026 (UTC); the page lists gpt-5.6-sol at a promotional price:

Output prices, the Tatoeba sets for Standard, Egyptian and Algerian Arabic, and the tokenisers without a published price are on the leaderboard and in the dataset.

Install and run

pip install "git+https://github.com/Bayanat-Labs/arabic-token-tax-index@v0.1.0"

arabic-token-tax count "تقرير الأمين العام عن أعمال المنظمة"     # tokens per tokeniser
arabic-token-tax count --tokeniser o200k_base,qwen3.5 "نص عربي"   # selected tokenisers
arabic-token-tax words "تقرير الأمين العام"                       # wordcount-v1 words

Python 3.10 or later. The PyPI package has not been released yet; pip install arabic-token-tax-index will work once it is published. Tokenizer files are fetched from their publishers on first use, at pinned revisions: tokenizer files only, never model weights. To reproduce the release, clone the repository at the v0.1.0 tag and follow its README.

from datasets import load_dataset
results = load_dataset("Bayanat-Labs/arabic-token-tax-index", "results", split="train")
segments = load_dataset("Bayanat-Labs/arabic-token-tax-index", "segments_un", split="test2015")

What it counts

• OpenAI encodings through OpenAI's open-source BPE library (encode_ordinary), Hugging Face tokenizer.json files through the tokenizers library (add_special_tokens=False) and tokenizer.model files through sentencepiece (no BOS or EOS).
• Each segment is encoded on its own, with no special tokens.
• Tokenisers whose files are not public, or sit behind an access request, are not measured in this version; the dataset lists each one with the reason.
• Add a tokeniser by adding an entry with a pinned revision to config/tokenisers.yaml.

Cite as

Bayanat Labs (2026). Arabic Token-Tax Index, version 0.1.0 (run 2026-10-16). https://huggingface.co/datasets/Bayanat-Labs/arabic-token-tax-index. CC BY 4.0.

Give the version and run date with every number. The release record, with licences and changelog, is the Arabic Token-Tax Index on Bayanat Labs Open.

Licence and contact

Harness code and leaderboard: Apache-2.0. Index data and documentation: CC BY 4.0. UN text: distributed under the United Nations Parallel Corpus disclaimer; credit the United Nations as the source. Questions, corrections and requests for new tokenisers: hello@bayanatlabs.com.

Need Arabic evaluation data for your models?

The token counter is free. For licensed Arabic evaluation sets and private test suites, see Bayanat Bench.