Bayanat Labs Arabic Wikisource Public-Domain Works
Whole works from Arabic Wikisource, rebuilt from the official Wikimedia dump of 1 October 2026: books, collections and single poems, each joined from its pages, its subpages and the scan pages it transcludes, split into chapters and converted to plain text, with verse lines and footnotes kept apart. Every work carries the people named as its author, translator or editor, their death dates from Wikidata or the Author pages, and the public-domain tier and rule that follow. Translators are taken only from Wikidata items for an Arabic edition or translation, from the page headers and Index pages, and from a translator named in the title or the opening lines of the text. works-pd-core holds 40,782 works whose authors, translators and editors all died in or before 1955. works-life50 holds 663 works whose latest death is between 1956 and 1975: they are in the public domain where copyright lasts 50 years after death, but not where it lasts 70. Most released works are poetry (40,732 works, most of them single poems). pd_decisions lists all 44,638 works of the dump with the tier, the rule and the reason, including 86 excluded and 2,723 unverified works, which are not released as text; neither are 339 decisions in works-pd-core and 45 in works-life50 that have no text. A separate check of 100 sampled decisions gave the same tier for all of them. The transcribed text is by Arabic Wikisource contributors, under CC BY-SA 4.0.
| Licence | CC BY-SA 4.0 (transcribed text, from Arabic Wikisource contributors); CC BY 4.0 (tables, statistics and checks); Apache-2.0 (builder code) |
|---|---|
| Version | v0.1.0 · Version v0.1.0, built from the Arabic Wikisource dump of 1 October 2026 with Wikidata as read from 30 to 31 October 2026; the builder is tagged on GitHub and not on PyPI yet. |
| Published | |
| Updated | |
| Who it is for | Teams that pretrain or evaluate Arabic language models on classical and early modern text, and researchers of Arabic literature who want whole works with their people, dates and public-domain status. |
| Level | Intermediate |
Key numbers
Counted by script from the released files. Each key number is read from stats/stats.json, MANIFEST.json, qa/pd_audit.json or qa/text_spotcheck.json in the Hugging Face dataset and pinned to its SHA-256; MANIFEST.json lists every file with its size, SHA-256 and row count.
Downloads and identifiers
| Where | What |
|---|---|
| Hugging Face | Dataset on Hugging Face |
| GitHub | Builder on GitHub |
| huggingface.co | works-pd-core as JSON Lines (gzip) |
| huggingface.co | Counts by tier, rule and genre (stats.json) |
| huggingface.co | Separate check of the public-domain decisions (pd_audit.json) |
| bayanatlabs.com | Page-level transcriptions: Arabic Document Atlas |
Get started
Load a config with the datasets library. To rebuild from a dump, install the builder from GitHub at the v0.1.0 tag; the package is not on PyPI yet.
from datasets import load_dataset
core = load_dataset("Bayanat-Labs/arabic-wikisource-pd", "works-pd-core", split="train")
print(core[0]["title"], core[0]["author"], len(core[0]["chapters"]))
pip install "git+https://github.com/Bayanat-Labs/arabic-wikisource-pd@v0.1.0"
arabic-wikisource-pd fetch --date 20261001
arabic-wikisource-pd wikidata --date 20261001
arabic-wikisource-pd build --date 20261001What is released
| Part | Licence | In this release |
|---|---|---|
| works-pd-core: one row per work whose authors, translators and editors all died in or before 1955, with text, chapters, verse lines, footnotes, people, dates and source pages | CC BY-SA 4.0 | Yes |
| works-life50: works whose latest death is between 1956 and 1975, public domain where copyright lasts 50 years after death but not where it lasts 70 | CC BY-SA 4.0 | Yes |
| JSON Lines export of works-pd-core, one work per line | CC BY-SA 4.0 | Yes |
| authors and pd_decisions tables (rules such as all-died-by-1955, latest-death-1956-1975, death-after-1975, anonymous-not-pre-1900 and translation-no-translator), statistics and checks | CC BY 4.0 | Yes |
| Builder code on GitHub | Apache-2.0 | Yes |
Attribution: Text: Arabic Wikisource contributors, CC BY-SA 4.0 (each record links its pages and their history). Dataset: Arabic Wikisource Public-Domain Works by Bayanat Labs. https://huggingface.co/datasets/Bayanat-Labs/arabic-wikisource-pd
Intended uses
- Pretrain or evaluate Arabic language models on whole works with their chapter structure, keeping the CC BY-SA 4.0 credit to Arabic Wikisource contributors.
- Choose works by public-domain tier for your jurisdiction: works-pd-core, or works-life50 only where copyright lasts 50 years after death.
- Study Arabic texts by author, century of death, genre or proofreading status, with links to every source page and revision.
Out of scope
- Treating the tiers as legal advice: they are rules applied to the dates recorded in Wikidata and on Wikisource.
- Using works-life50 where copyright lasts 70 years after death, such as the member states of the European Union.
- Page-level OCR or image work: page transcriptions with their scans are in the Arabic Document Atlas.
Rights at a glance
Training, display, redistribution and synthetic-voice use are separate permissions.
| Use | Status | Note |
|---|---|---|
| Training | Allowed | The transcribed text is CC BY-SA 4.0 from Arabic Wikisource; keep the credit and the share-alike terms. The underlying works are in the public domain in the tier described. |
| Display | Allowed | With credit to Arabic Wikisource contributors and links to the pages (CC BY-SA 4.0); tables under CC BY 4.0 with attribution. |
| Redistribution | Allowed | Text under CC BY-SA 4.0 (share alike); authors, pd_decisions, statistics and checks under CC BY 4.0; builder code under Apache-2.0. |
| Synthetic voice | Not applicable | No audio. |
Limitations
- Many texts were copied from modern printed editions whose editors are not named; the tier rests on the people who are named, so an unnamed modern editor's notes or corrections may be in the text.
- Modern editors' introductions are removed where they are headed as such; an introduction under another heading stays in the text.
- A translation counts as one only when a category, Wikidata or the author's recorded languages show it, and its translator only when Wikidata (on an item for the Arabic version), the page header, the Index page, the title or the opening lines of the text name one; a modern translation of an old work with none of these signals is decided by the original author's dates.
- A periodical or collection is decided as one work, so one late contributor keeps all of it out.
- 36 released works are marked incomplete on Wikisource, and 29 contain wikitext markup that was never closed.
- Text that pages draw from Lua modules or templates, such as the Quran pages, is not in the dump's wikitext and is not included.
- Most released works are poetry, and modern prose is a small part; genres come from rules, not from reading the texts.
- Counts, texts and dates are those of the dump of 1 October 2026, with Wikidata as read from 30 to 31 October 2026; the checks were run by Bayanat Labs, and no person reviewed the texts or the decisions.
Sources and licences
| Source | Licence | Redistributed here |
|---|---|---|
| Arabic Wikisource dump of 1 October 2026 (arwikisource-20261001: pages-articles, page_props, categorylinks and linktarget) | CC BY-SA 4.0; transcribed text redistributed with credit and page links | Yes |
| Wikidata (authors, translators, editors and death dates), read through its API from 30 to 31 October 2026 | CC0 1.0 | Yes |
| arabic-normalize 0.1.0, policy bayanat-norm-v1, for text_norm and the word counts | Apache-2.0; used to build the release, not included | No |
The text comes only from the Wikimedia dump, with the checksums of the dump files recorded in PROVENANCE.jsonl. Wikidata was read through its API with the Bayanat Labs open-data user agent, one request per second. No contributor account names are included. Wikisource and Wikidata are trademarks of the Wikimedia Foundation; this dataset is not affiliated with or endorsed by it. Related from Bayanat Labs: the Arabic Document Atlas, with page-level transcriptions and their proofreading status. Source details
How to cite
Cite as:
Bayanat Labs (2026). Arabic Wikisource Public-Domain Works, version 0.1.0 (Arabic Wikisource dump of 2026-10-01; text by Arabic Wikisource contributors, CC BY-SA 4.0). Bayanat Labs. https://huggingface.co/datasets/Bayanat-Labs/arabic-wikisource-pd
No DOI. Cite the URL in the citation above and the date you read it.
@misc{bayanatlabs2026arabicwikisourcepd,
author = {{Bayanat Labs}},
title = {Arabic Wikisource Public-Domain Works, version 0.1.0 (Arabic Wikisource dump of 2026-10-01; text by Arabic Wikisource contributors, CC BY-SA 4.0)},
year = {2026},
month = oct,
howpublished = {\url{https://huggingface.co/datasets/Bayanat-Labs/arabic-wikisource-pd}}
}
Changelog
| Date | Version | Change |
|---|---|---|
| v0.1.0 | The two public-domain tiers, the authors and decisions tables, the JSON Lines export, statistics and checks on Hugging Face; builder on GitHub. |
Corrections and removal requests
If something in this release should not be there, or is wrong, email hello@bayanatlabs.com. We handle removal requests within 30 days, and platform requests within 24 hours. How removal requests work
Related product: Bayanat Knowledge