Bayanat Labs Arabic Token-Tax Index

Arabic tokenizer Arabic token count LLM tokenizer comparison Arabic LLM cost token tax

Key numbers

Counted by script from the released files. Each key number is read from a file in the Hugging Face dataset (meta/summary.json, meta/tokenisers.json, meta/slices.csv and meta/costs.csv) and pinned to its SHA-256; MANIFEST.json lists every file with its SHA-256.

Downloads and identifiers

Get started

Load the results with the datasets library, or install the harness from GitHub at the v0.1.0 tag to count the tokens of your own text; pip install arabic-token-tax-index will work once it is published on PyPI.

# Python
from datasets import load_dataset
results = load_dataset("Bayanat-Labs/arabic-token-tax-index", "results", split="train")

# Harness
pip install "git+https://github.com/Bayanat-Labs/arabic-token-tax-index@v0.1.0"
arabic-token-tax count "تقرير الأمين العام عن أعمال المنظمة"

What is released

Attribution: Arabic Token-Tax Index by Bayanat Labs, CC BY 4.0. UN text: source, the United Nations. https://huggingface.co/datasets/Bayanat-Labs/arabic-token-tax-index

Intended uses

• Estimate how many more tokens, and how much more money, Arabic content takes than the same content in English with a given tokeniser.
• Compare tokenisers on the same parallel text, with a confidence interval for every ratio.
• Count the tokens of your own Arabic text with every tokeniser in the index, or re-run the measurement from the public sources.

Out of scope

• Judging model quality: fewer tokens means lower cost and more text per context window, not a better model.
• Exact invoices: costs use the list prices shown on each vendor's page on the retrieval date (one of them is marked there as a promotional price), without batch, cache or other discounts.

Rights at a glance

Training, display, redistribution and synthetic-voice use are separate permissions.

Limitations

• Tokenisers whose files are not public, or sit behind an access request, are not measured in this version; the dataset lists each one with the reason.
• An API service may count tokens differently from the published tokenizer file, for example by adding special tokens or using another revision.
• Prices change; every price row records when and where it was read.
• The UN text is formal Modern Standard Arabic and the Tatoeba slices are short everyday sentences, so compare ratios within a slice.
• Install the harness from GitHub for now; it is not on PyPI yet.

Sources and licences

Token counts, ratios, intervals, costs and the diacritics probe are by Bayanat Labs. The cleaned UN sentence pairs are distributed with the full United Nations Parallel Corpus disclaimer (NOTICE in the dataset). Tatoeba sentences are identified by id and hash. Tokenizer files are fetched from their publishers at run time and are not part of the release. Prices are facts read from the vendors' official price pages at the time stated in each row. Source details

How to cite

Cite as:

Bayanat Labs (2026). Arabic Token-Tax Index, version 0.1.0 (run 2026-10-16). Bayanat Labs. https://huggingface.co/datasets/Bayanat-Labs/arabic-token-tax-index

• CITATION.cff

No DOI. Cite the URL in the citation above and the date you read it.

@misc{bayanatlabs2026arabictokentaxindex,
  author       = {{Bayanat Labs}},
  title        = {Arabic Token-Tax Index, version 0.1.0 (run 2026-10-16)},
  year         = {2026},
  month        = oct,
  howpublished = {\url{https://huggingface.co/datasets/Bayanat-Labs/arabic-token-tax-index}}
}

How we handle citations

Changelog

Corrections and removal requests

If something in this release should not be there, or is wrong, email hello@bayanatlabs.com. We handle removal requests within 30 days, and platform requests within 24 hours. How removal requests work

Related product: Bayanat Bench