Bayanat Labs Arabic Data Index

Arabic datasets Arabic dataset statistics dataset licences Hugging Face datasets Masader

Key numbers

Counted by script from the released files. Each key number is read from the aggregates CSV or a qa/ file of the Hugging Face dataset and pinned to its SHA-256. The speech hours are read from latest.json of the Open Arabic Speech Hours Counter, and the possible copies not folded are counted from the entities table. MANIFEST.json lists every file with its size and SHA-256.

Downloads and identifiers

Get started

Load a table with the datasets library. To recompute the aggregates from the frozen entities table and compare them with the released CSV, install from a clone at the v0.1.0 tag; the package is not on PyPI.

from datasets import load_dataset

agg = load_dataset("Bayanat-Labs/arabic-data-index", "aggregates", split="train")
ent = load_dataset("Bayanat-Labs/arabic-data-index", "entities", split="train")

git clone --branch v0.1.0 https://github.com/Bayanat-Labs/arabic-data-index && cd arabic-data-index && pip install .
adi rerun --snapshot-dir snapshots/2026-11 --check

What is released

Attribution: Bayanat Labs Arabic Data Index, snapshot 2026-11, CC BY 4.0. https://huggingface.co/datasets/Bayanat-Labs/arabic-data-index

Intended uses

• Cite a dated count of Arabic AI datasets, by snapshot and metric, with its denominator and coverage where they apply.
• Compare declared licence classes, modalities, varieties and tasks when scoping work on Arabic data.
• Start a search for Arabic datasets from the entities table, which has one row per dataset with its declared licence and links.

Out of scope

• Treating a licence class as permission to use a dataset: the classes describe what each dataset declares, and each dataset keeps its own licence and terms.
• Reading the headline as every Arabic dataset in existence: it covers Hugging Face datasets with an Arabic language tag and Masader entries.
• Using row totals as a measure of how much data exists: rows mean different things in text, speech and image datasets.

Rights at a glance

Training, display, redistribution and synthetic-voice use are separate permissions.

Limitations

• Licence classes describe the licence each dataset declares (the Hub licence tag, or Masader's licence when the Hub has none). They were not checked against the data and are not legal advice.
• The snapshot does not read Kaggle or Zenodo. A dataset hosted on either, or anywhere else, counts only if it is on Hugging Face with an Arabic language tag or listed in Masader.
• Copies are folded only from complete viewer sizes. 20 headline datasets, in 18 groups, have the same repository name and viewer counts as another but are not folded, because the viewer size is partial (14) or has no original-file size (6), so the headline can count them more than once; in 8 of the groups the viewer reports zero rows, so only the name matches. Copies whose viewer request failed can also be counted more than once.
• Failed dataset-viewer requests (609) were requested once more about two and a half hours later (median 2.48 hours, from 1.1 to 3.4): 1 then succeeded, and the 608 that failed again keep the status of their original reply.
• Modality, dialect and task come from what each repository or Masader entry declares; they are not inferred where nothing is declared. Masader entries marked multilingual stay in the headline, because Masader gives no number of languages.
• The creation-month series counts datasets still on the Hub, so earlier months are undercounted.
• Row totals cover the 71.73% of headline Hugging Face datasets with a complete viewer size; they are counts, not a measure of data.
• The speech hours are quoted from the Open Arabic Speech Hours Counter as published, with its as-of date; they are not recomputed here.

Sources and licences

The Hugging Face datasets API was read for every Arabic language tag, and the dataset viewer for each repository's size and validity, with the Bayanat Labs open-data user agent and at least one second between requests to a host; no data file was downloaded. Copies count once: copy groups of the Arabic Dataset Rights Index and same-size copies on Hugging Face, 224 repositories in all. Name matches, shared links and same-size copies are only candidates: Bayanat Labs checked 406 pairs, including all 296 candidates, and merged 201 of those. Each figure is computed in SQL and again in pandas, and the two agree on all 249 rows of the 32 metrics. Source details

How to cite

Cite as:

Bayanat Labs (2026). Arabic Data Index, snapshot 2026-11, as of 2026-11-01. Bayanat Labs. https://huggingface.co/datasets/Bayanat-Labs/arabic-data-index

• CITATION.cff

No DOI. Cite the URL in the citation above and the date you read it.

@misc{bayanatlabs2026arabicdataindex,
  author       = {{Bayanat Labs}},
  title        = {Arabic Data Index, snapshot 2026-11, as of 2026-11-01},
  year         = {2026},
  month        = nov,
  howpublished = {\url{https://huggingface.co/datasets/Bayanat-Labs/arabic-data-index}}
}

How we handle citations

Changelog

Corrections and removal requests

If something in this release should not be there, or is wrong, email hello@bayanatlabs.com. We handle removal requests within 30 days, and platform requests within 24 hours. How removal requests work

Related product: Bayanat Knowledge