Bayanat Labs Arabic Token-Tax Calculator
LLM APIs bill and limit by tokens, and the same content usually takes more tokens in Arabic than in English: on the UN test set of the Arabic Token-Tax Index, the median Arabic/English token ratio across its tokenizers is 1.42. The calculator shows the difference for your own text. Paste Arabic text, and its English translation if you have one, and see for each tokenizer the token count, tokens per word, the Arabic/English token ratio, characters per token, whether the text fits a context budget, and the cost of sending or generating it at list prices read from the vendors' price pages on 30 October 2026, which you can edit. The calculator page runs 9 tokenizers in your browser; your text stays on the page, which may connect only to its own origin and stores nothing. The command-line tool token-tax runs 12 tokenizers at pinned revisions and gives the same token ids.
| Licence | Apache-2.0 (code and calculator page); CC BY 4.0 (fixtures, price table and documentation); tokenizer files keep their own licences (MIT or Apache-2.0) |
|---|---|
| Version | v0.1.0 · Code, fixtures and calculator page v0.1.0, tagged on GitHub. Not on PyPI yet. |
| Published | |
| Updated | |
| Who it is for | Developers and teams who send Arabic text to LLM APIs and want to know, before they do, how many tokens it takes with each tokenizer, whether it fits a context window and what it costs. |
| Level | Beginner |
Key numbers
Counted by script from the released files. Each key number is counted from a file released at tag v0.1.0 (the tokenizer registry, the price table, the fixtures and the browser parity report docs/parity-v0.1.0.json) and pinned to its SHA-256; the median ratio comes from the Arabic Token-Tax Index. The calculator page's MANIFEST.json lists every file it serves with its SHA-256.
Downloads and identifiers
| Where | What |
|---|---|
| Hugging Face | Open the calculator |
| bayanatlabs.com | Command line and tokenizers |
| GitHub | Code, fixtures and method on GitHub |
| GitHub | Method |
| bayanatlabs.com | Arabic/English token ratios on UN text (Arabic Token-Tax Index) |
Get started
Install the command-line tool from GitHub at the v0.1.0 tag; the package is not on PyPI yet. The calculator page needs no install.
pip install "git+https://github.com/Bayanat-Labs/arabic-token-tax-calculator@v0.1.0"
token-tax count "قالت البلدية إن أعمال صيانة الطريق الساحلي ستستمر ثلاثة أسابيع" \
--en "The municipality said that maintenance work on the coastal road will continue for three weeks"
token-tax count --file ar.txt --en-file en.txt --tokenizers o200k_base,qwen3.5 --budget 8192
token-tax count --file ar.txt --json
token-tax tokenizersWhat is released
| Part | Licence | In this release |
|---|---|---|
| Calculator page on Hugging Face Spaces: 9 tokenizers that run in your browser, interface in English and Arabic | Apache-2.0 | Yes |
| Python library and command-line tool (token-tax), for Python 3.10 or later | Apache-2.0 | Yes |
| Fixture texts, price table, method and documentation | CC BY 4.0 | Yes |
| Tokenizer files served by the calculator page: OpenAI o200k_base and cl100k_base, Qwen3, Qwen3.5 / 3.6 / 3.8, DeepSeek V3 / V4, Mistral Tekken, Gemma 4 / Fanar-2, ALLaM, Fanar-1 | Each file keeps its own licence, MIT or Apache-2.0, listed with the release | Yes |
| IBM Plex Sans Arabic font in the calculator page, unmodified (SIL Open Font License 1.1) | SIL Open Font License, copyright IBM Corp. | Yes |
Attribution: Arabic Token-Tax Calculator (token-tax) by Bayanat Labs. https://github.com/Bayanat-Labs/arabic-token-tax-calculator
Intended uses
- Estimate the tokens, context use and cost of an Arabic text with a given tokenizer before you send it.
- Compare tokenizers on your own Arabic text and its English parallel.
- Check token counts in scripts and continuous integration with the command-line tool's JSON output.
Out of scope
- Judging model quality: fewer tokens means lower cost and more text per context window, not a better model.
- Billing: costs are estimates at list prices; a service's bill also counts chat formatting, system prompts and discounted rates.
Rights at a glance
Training, display, redistribution and synthetic-voice use are separate permissions.
| Use | Status | Note |
|---|---|---|
| Training | Not applicable | A calculator, fixtures and a price table, not training data. The fixtures are CC BY 4.0 if you reuse them. |
| Display | Allowed | Fixtures, price table and documentation under CC BY 4.0, with attribution. |
| Redistribution | Allowed | Code under Apache-2.0; fixtures, price table and documentation under CC BY 4.0; tokenizer files under their own licences (MIT or Apache-2.0); the bundled font under the SIL Open Font License. |
| Synthetic voice | Not applicable | No audio. |
Limitations
- Tokens per word overstate the gap between the languages, because Arabic clitics stay attached to their words; quote the Arabic/English ratio on the same content.
- Counts exclude special tokens and the chat formatting a service adds.
- List prices were read from the vendors' price pages on 30 October 2026 and change; one of them is a promotional price and is marked as such. Edit them or add your own.
- Falcon3, Falcon-H1 and Falcon-H1R run in the command-line tool only.
- The fixtures were written for this tool with fictional names and amounts; the dialect rows are illustrative.
- Install the command-line tool from GitHub for now; the package is not on PyPI yet.
Sources and licences
| Source | Licence | Redistributed here |
|---|---|---|
| Arabic Token-Tax Index 0.1.0: tokenizer set, pinned revisions and the priced models | CC BY 4.0 (index data); Apache-2.0 (harness) | Yes |
| Official price pages of OpenAI, DeepSeek and Alibaba Cloud, read on 30 October 2026 | Prices recorded as facts with the page URL, the time read and the SHA-256 of the page; the pages themselves are not included | Yes |
| Tokenizer files from the model developers' repositories, at pinned revisions: OpenAI o200k_base and cl100k_base, Qwen3, Qwen3.5 / 3.6 / 3.8, DeepSeek V3 / V4, Mistral Tekken, Gemma 4 / Fanar-2, ALLaM, Fanar-1, Falcon3, Falcon-H1 and Falcon-H1R | Each file under its own licence; the calculator page includes only files under MIT or Apache-2.0, and the command-line tool downloads the others on first use | Yes |
Tokenizers, revisions and the priced models come from the Arabic Token-Tax Index 0.1.0; the prices were read again from the vendors' official price pages on 30 October 2026, and a promotional price is marked as such. Each tokenizer file is checked by SHA-256 before use. The fixtures were written for this tool, with fictional names and amounts. Source details
How to cite
Cite as:
Bayanat Labs (2026). Arabic Token-Tax Calculator, version 0.1.0. Bayanat Labs. https://github.com/Bayanat-Labs/arabic-token-tax-calculator
No DOI. Cite the URL in the citation above and the date you read it.
@misc{bayanatlabs2026tokentaxcalculator,
author = {{Bayanat Labs}},
title = {Arabic Token-Tax Calculator, version 0.1.0},
year = {2026},
month = oct,
howpublished = {\url{https://github.com/Bayanat-Labs/arabic-token-tax-calculator}}
}
Changelog
| Date | Version | Change |
|---|---|---|
| v0.1.0 | Calculator page in English and Arabic on Hugging Face Spaces; Python library, command-line tool, fixtures and method on GitHub. |
Corrections and removal requests
If something in this release should not be there, or is wrong, email hello@bayanatlabs.com. We handle removal requests within 30 days, and platform requests within 24 hours. How removal requests work
Related product: Bayanat Bench