Bayanat Labs polite-fetch Crawler Library

robots.txt AI crawler opt-out TDMRep Content-Signal polite crawler

Key numbers

Counted by script from the released files. Each key number is read from docs/conformance/summary.json or policy/crawler.json in the repository at tag v0.1.0 and pinned to its SHA-256. The same crawler.json is served at bayanatlabs.com/open/crawler.json.

Downloads and identifiers

Get started

Install from GitHub at the v0.1.0 tag; the package is not on PyPI yet. check gives the decision for a URL from robots.txt and tdmrep.json without requesting the page.

pip install "git+https://github.com/Bayanat-Labs/polite-fetch@v0.1.0"

polite-fetch check https://example.org/page --purpose training_release
polite-fetch fetch https://example.org/page --headers-only --ledger signals.jsonl

from bayanatlabs_polite_fetch import Fetcher

with Fetcher(worker_id="worker-1", ledger="signals.jsonl") as f:
    r = f.fetch("https://example.org/page", purpose="training_release")
    print(r.decision, r.decision_reason)

What is released

Attribution: polite-fetch by Bayanat Labs. https://github.com/Bayanat-Labs/polite-fetch

Intended uses

• Fetch pages and files for a dataset under a fixed, published crawler policy, with the decision and its rule recorded for every URL.
• See what the crawler would decide for a URL (robots.txt group, AI crawler blocks, TDMRep, Content-Signal, Content-Usage) before the page is requested.
• Reuse the fixture server and conformance tests to check how another crawler handles robots.txt and opt-out signals.

Out of scope

• Deciding whether content may legally be reused: the library reads machine-readable signals, not terms of use or licences.
• Getting past access controls: it stops at logins, bot challenges and CAPTCHA pages and has no option to bypass them.

Rights at a glance

Training, display, redistribution and synthetic-voice use are separate permissions.

Limitations

• Rate limits and stop rules hold per worker; several workers on the same host are not coordinated by the library, so run one worker per host.
• The IETF AI preferences vocabulary (Content-Usage) is still a draft; values are recorded as found, and any n value makes a record index only.
• Challenge and CAPTCHA detection uses response headers and page markers; when only headers are read, meta tags and page bodies are not checked.
• The documented-download exception relies on the documentation address the caller gives; the library records it but does not read that page.
• The conformance tests run against a local fixture server, never against real sites, so real servers can answer in ways the fixtures do not cover.
• Install from GitHub for now; the package is not on PyPI yet.

Sources and licences

polite-fetch implements published specifications and does not reproduce their text. The AI crawler names come from robots.json of the ai.robots.txt project, pinned to one commit, and are bundled under its MIT licence with the notice. The crawler page on bayanatlabs.com describes crawler behaviour only where the library enforces it; the traceability table in the repository maps each statement about crawler behaviour to the tests that check it, and the build fails when such a statement has no test. Source details

How to cite

Cite as:

Bayanat Labs (2026). polite-fetch: the library behind the BayanatLabs-OpenData crawler, version 0.1.0, crawler policy 1.0.0. Bayanat Labs. https://github.com/Bayanat-Labs/polite-fetch

• CITATION.cff

No DOI. Cite the URL in the citation above and the date you read it.

@misc{bayanatlabs2026politefetch,
  author       = {{Bayanat Labs}},
  title        = {polite-fetch: the library behind the BayanatLabs-OpenData crawler, version 0.1.0, crawler policy 1.0.0},
  year         = {2026},
  month        = oct,
  howpublished = {\url{https://github.com/Bayanat-Labs/polite-fetch}}
}

How we handle citations

Changelog

Corrections and removal requests

If something in this release should not be there, or is wrong, email hello@bayanatlabs.com. We handle removal requests within 30 days, and platform requests within 24 hours. How removal requests work

Related product: Bayanat Knowledge