Bayanat Labs Arabic Web Rights Census

Arabic websites robots.txt AI crawlers AI training opt-out TDMRep Content-Signal

Key numbers

Counted by script from the released files. Each key number is read from stats.json in the Hugging Face dataset, which holds every number on the dataset card with the table, row and column it comes from, and pinned to its SHA-256; MANIFEST.json lists every file with its size and SHA-256.

Downloads and identifiers

Get started

Load a table with the datasets library, or query it with DuckDB straight from the Hub. The parser and fetcher install from GitHub at the v0.1.0 tag; the package is not on PyPI yet.

from datasets import load_dataset

hosts = load_dataset("Bayanat-Labs/arabic-web-rights-census", "hosts", split="train")
agg = load_dataset("Bayanat-Labs/arabic-web-rights-census", "aggregates", split="train")

pip install "arabic-web-rights-census @ git+https://github.com/Bayanat-Labs/arabic-web-rights-census@v0.1.0"
awrc robots robots.txt --bot GPTBot

What is released

Attribution: Arabic Web Rights Census by Bayanat Labs. Data CC0 1.0; card text and charts CC BY 4.0. https://huggingface.co/datasets/Bayanat-Labs/arabic-web-rights-census

Intended uses

• Measure how many of the listed sites signal an AI-training opt-out, by crawler, sector and country label, weighted by hosts or by Arabic pages.
• Look up the robots.txt rule, Content-Signal and TDMRep values that a host published for each crawler token on the census dates.
• Use the exclusion list as one input when a corpus honours machine-readable reservations of text and data mining rights.

Out of scope

• Treating a robots.txt allow or a missing signal as permission to reuse a site's content: terms of use and licences still apply and were not read.
• Using the exclusion list as a complete register of reservations or as legal advice: it covers only the hosts and signals in this census, at the time they were read.

Rights at a glance

Training, display, redistribution and synthetic-voice use are separate permissions.

Limitations

• All requests came from one network connection in the United States; some hosts, including some government hosts, may refuse or drop connections from outside their region and appear as unreachable or timed out.
• Unreachable (1,170), timed-out (650) and challenged (385) hosts are reported and never counted as allowing anything; they are left out of every rate.
• Page weights come from one Common Crawl crawl, which does not collect pages from hosts that block CCBot, so blocking hosts are under-represented in the list; the Tranco hosts reduce this bias but do not remove it, and their language was not checked.
• Sector labels are rule-based and are not checked site by site; country labels come from the top-level domain, and hosts under generic top-level domains are labelled gTLD.
• Hosts whose robots.txt request was refused as unauthorised or forbidden (78) are counted as having no robots.txt, as the robots.txt standard provides for client-error replies, and were not contacted again; such a reply may be a block on unknown crawlers rather than a missing file.
• Terms of use, licences and robots.txt comments are not read, and robots.txt bodies are not released: each host row keeps the SHA-256 of the body as served.

Sources and licences

Each host's robots.txt, /.well-known/tdmrep.json and homepage were requested by the census fetcher with the BayanatLabs-OpenData user agent; the homepage was requested only when robots.txt allowed it, its headers and head were read and its body was dropped. Each host received at most three requests (7 hosts may have received a fourth: 11 of the 1,313 requests made during local network outages ended in a timeout, connection or TLS error rather than at the DNS look-up and may have reached the host), with at least 5 seconds between requests to the same host, measured on the machine's monotonic clock (the ledger records wall-clock times, and 74 gaps read between 4.924 and 5 seconds where the system clock was corrected backwards, at three instants). A host was not contacted again after an access-denied or rate-limit reply, a challenge page or a login redirect, and no challenge was solved. Hosts whose first robots.txt request timed out or failed to connect were requested once more, at least an hour later. Every request is in the provenance ledger with its time, status and SHA-256. The census used its own fetcher (awrc.fetch), not the polite-fetch library described on the crawler page. Where they differ: it requested /.well-known/tdmrep.json without checking robots.txt; it kept cookies a host set in memory for the run and sent them back to that host; it retried a 5xx robots.txt once after 30 seconds; it treated more than five robots.txt redirects as no robots.txt (14 hosts); it followed homepage redirects without reading the target host's robots.txt; and when a certificate check failed it repeated the request once without verification (1,708 requests). Platform subdomains (blog and site-builder hosts) and IP addresses are left out of the list, because their robots.txt is set by the platform and their names can name private people. Related from Bayanat Labs: the crawler page, which states the user agent and how to opt out. Source details

How to cite

Cite as:

Bayanat Labs (2026). Arabic Web Rights Census, census 2026-10 (version 0.1.0), as of 2026-10-31. Bayanat Labs. https://huggingface.co/datasets/Bayanat-Labs/arabic-web-rights-census

• CITATION.cff

No DOI. Cite the URL in the citation above and the date you read it.

@misc{bayanatlabs2026arabicwebrightscensus,
  author       = {{Bayanat Labs}},
  title        = {Arabic Web Rights Census, census 2026-10 (version 0.1.0), as of 2026-10-31},
  year         = {2026},
  month        = oct,
  howpublished = {\url{https://huggingface.co/datasets/Bayanat-Labs/arabic-web-rights-census}}
}

How we handle citations

Changelog

Corrections and removal requests

If something in this release should not be there, or is wrong, email hello@bayanatlabs.com. We handle removal requests within 30 days, and platform requests within 24 hours. How removal requests work

Related product: Bayanat Knowledge