Midad Arabic Sources

by Joseph Hill
5
4
3
2
1
Score: 52/100

Description

Integrating Arabic sources into your Obsidian vault to facilitate research and study. Tools for OCR transcription, translation, and importing texts from vast online libraries of already digitized texts.

Reviews

No reviews yet.

Stats

1
stars
101
downloads
1
forks
21
days
3
days
3
days
0
total PRs
0
open PRs
0
closed PRs
0
merged PRs
0
total issues
0
open issues
0
closed issues
269
commits

Latest Version

4 days ago

Changelog

Install/update via BRAT.

0.2.12

One Qur'ān engine for every text, a margin that numbers the whole chapter, and citations that are corrected from the words they quote.

One Qur'ān engine

  • The Qur'ān work is now two modules instead of one mixed file: quran.py holds the general work (verse access, quote discovery and matching, cross-page quotes, span matching, citation hygiene, canonical replacement), and tafsir.py holds the tafsīr mode only (focal verse, marginal markers, linear progression, cross-reference rules).
  • transcription-quran.py is gone. A scanned transcription, a repository tafsīr and a Turath book now call the same functions, so a rule cannot exist for one source and not for another.
  • The riwāya is detected from the text — Warsh from its own mark set — or declared per work; matching, replacement and every generated link follow it, and an other reading is matched without replacing the author's text.

The margin numbers the whole chapter

  • Marginal verse markers are decided for the whole note, not page by page: a chapter written one page at a time is collected first, because whether a verse is discussed in order is a fact about the chapter.
  • A verse the commentary takes up in order is numbered where the discussion reaches it; a verse it only mentions in passing is numbered at that mention, and that marker does not move the cursor. Every cited verse therefore gets exactly one marker — Aḍwāʾ al-bayān's Sūrat al-Naḥl went from 9 markers to 78, and al-Qurṭubī's from a handful to 116 (16:1 … 16:128).
  • A jump — forward or backward — is a genuine skip unless the text soon returns to near where it was: a return is a digression, a cross-reference, and it does not move the margin. A new passage may start at a lower verse after a long detour.
  • A verse is numbered once in a note; the commentary's constant returns no longer repeat it in the margin. Verses introduced together on one line still become one range marker, but only when they are within a few verses of each other.

A citation is evidence of a quotation

  • wrap_cited_quotes now runs for every Arabic text with citations, not only tafsīrs: a link never appears without its Qur'ān formatting. The quotation is the longest suffix of the words before the citation that occurs contiguously in the cited verse, so a phrase from the middle of a verse is found, prose before it stays outside, and a lead-in is only a boundary hint (and the sole signal for an uncited quotation).
  • A one-word quotation is formatted when it verifies (شَيْئًا → 16:73), a word quoted inside guillemets is wrapped inside them (وَ«مُسَخَّرَاتٌ» → 16:12, the conjunction left outside), and a closing quote mark, an ellipsis or الآية between the quotation and its citation stays outside the boundary.
  • Two citations can no longer claim the same words. An overlapping span is skipped rather than nested, which is what used to split a word in two (~﴿~وَهُ~﴿~…يًّا).

Citation repair

  • A citation the muṣḥaf does not support is corrected from the words that follow it, before anything is wrapped, numbered or displayed. The number is the clue, so the nearest reading of it wins: an adjacent verse in the same sūra (24:31 → 24:30), the same āya number in another sūra (16:136 → 6:136, 95:7 → 59:7), another verse of the same sūra (7:124 → 7:142), then a single match anywhere (16:78 → 16:87).
  • At least three words must agree, because two words occur in dozens of verses; an impossible verse number with no quotation is corrected only when one transposition of its digits exists (16:221 → 16:122, the duplicate absorbed); anything ambiguous is left exactly as the source wrote it.
  • A text whose quotations disagree wholesale is the wrong edition or riwāya, not a text of typos: once two thirds of a page's quotations fail to verify, that page is left untouched and the reason is logged. On a clean text the corrections are occasional — about 2% of Aḍwāʾ al-bayān's citations, with the offsets of a transposed or dropped digit while 97% verify as printed.

Turath and templates

  • A Turath tafsīr now carries the tafsir class, quotes inside invisible boundaries with the canonical text, and the same marginal markers as a transcribed tafsīr. A stray toc anchor inside a citation bracket is unwrapped instead of becoming a heading.
  • Same-line source and recognition ranges stack in the margin (data-marker-index), one above the other, instead of overlapping.
  • A tafsīr's # N emblem is read from a numeric or range heading, so the verse the block covers is the verse it shows.

Fixes

  • Overlapping quotation edits can no longer nest the ornate boundaries or split a word; both wrapper passes reject an edit that would overlap one already planned.
  • A citation link is no longer left unformatted because the wrapping pass ran before the link was added: it is re-run at the end of the tafsīr pass.
  • Updating the plugin now removes scripts and templates it no longer ships, so a retired module cannot linger in the installed copy.

Documentation

  • The template and transcription guides describe the chapter-wide margin, the one-off marker, the citation-is-evidence rule and the citation-repair ladder.

0.2.11

One verse marker for every source, and Arabic typography that no longer depends on the view.

One verse marker

  • Every source now writes its correspondence to the canonical Qurʾān as **[[002 Al-Baqarah#3|2:3]]**: tafsīr, every riwāya (replacing the old Ḥafṣ N heading), and translations.
  • The marker sits in the left gutter of an RTL text and in the right gutter of an LTR one, from the same class and the same code path.
  • The # N emblem is unchanged: it is the page's own verse number, so it keeps the side where the verse begins (right for Arabic, left for a translation). A riwāya page therefore reserves both bands, exactly as before.
  • The marker's emphasis is structural, not decorative: a bare wikilink becomes an editor widget that CSS cannot position. It is never drawn bold.
  • A marker now ends with an invisible U+FEFF, so the closing ** is not followed directly by an Arabic letter — an emphasis that cannot close is shown as literal stars by the editor.

Typography

  • Reading view no longer substitutes the general Arabic face for the Qur'ān one. Three rules were out-ranking the class-scoped verse rules (the generic RTL rule, a marker paragraph hook, and a quote-span hook); the face is now declared with the view class for both views, and --quran-warsh-font (Harmattan) is honoured throughout a Warsh page.
  • Qur'ān passages inside other notes — a tafsīr, a ḥadīth commentary — keep the Qur'ān face in Reading view as well as in Live Preview.
  • The Warsh page and the Ḥafṣ page are laid out identically; only the face differs.

Transcription and post-processing

  • strip_raw_quran_citations no longer mistakes a wikilink's inner brackets for a source citation, which had deleted every marker link and left **\[] debris across a re-processed note.
  • normalize_quran_link_display leaves an S:V display in Latin digits; only citations are converted to Arabic-Indic numerals.
  • link_open_fragments no longer reads an HTML attribute (class="mas-managed">) as an opening quotation mark and wraps the managed-region marker.
  • insert_verse_headings removes the **\[] debris an older run left, so re-processing repairs a note instead of carrying it forward.
  • Page separators are imported from the shared post-processor and always applied; a run of --- rules separated by blank lines now collapses to one.

Fixes

  • quranFile/quranWorks accept canonical and prefixed filenames; the callout and reference commands list every installed riwāya and translation.
  • Scoped re-renders refresh the note baselines.
  • The tafsir class is no longer added to notes that are not tafsīr.
  • midad-up topic repair is add-only, so it no longer removes template-supplied topics.

Documentation

  • Transcription and template guides describe the single marker form, the gutter sides, and why the emphasis and the U+FEFF separator are required.
  • Qur'ān, tafsīr and translation notes are documented as reading-view first; the Live Preview marker layout remains parked.

README file from

Github

Midad Arabic Sources

Midad Arabic Sources brings classical Arabic texts into your Obsidian vault and turns them into a linkable, searchable, annotated part of your notes. It removes the manual labour between there is an Arabic text I want to study and it is integrated into my vault.

Some of its many features:

  • locates and imports already digitized texts using an extensive curated list of online repositories
  • transcribes (OCRs) your scanned PDFs, either directly or through fetching them as Zotero attachments
  • translates those transcriptions into English
  • transliterates Arabic passages for integration into English writing
  • provides an annotation interface similar to Zotero’s PDF annotation features, preserving your annotations even if the source document is updated or removed from Obsidian
  • archives sources as ZIP files so you can “shelf” sources you’re not actively using, keeping your annotations handy and your Obsidian vault lean.
  • auto-updates and re-renders generated source notes when templates and upstream sources change, without overwriting any corrections you may have made.
  • manages saved versions per note when you correct OCR errors, with virtual Version 0 for the installed model, per-work version archives, and safe restoration after you remove or archive a source
  • preserves user-written content when sources are removed, while allowing explicit full deletion or model-only cleanup
  • exports sources as auto-updated Zotero items

The plugin was built for research on Islam and the Arabic language — Qurʾān commentaries, ḥadīth, the Qurʾān in its riwāyāt, and the classical dictionaries — but the importer is catalog-driven, so more repositories on any topic can be wired in over time, and users can add works by URL or PDF import that are not listed in the plugin.

Midad is optimized for integration with Zotero using the ScholarWeft Obsidian plugin, although these two are not required.

[!note] This plugin is desktop only. It runs local Python scripts and reads files outside the vault.

Documentation

The full guides are in docs/; this README is a summary. Every page is also bundled into the plugin and readable in-app with the command Open documentation (or from Settings → Midad Arabic Sources → Documentation).

  • Setup — install, first run, the folder layout, and where state lives.
  • Dependencies — Python, CAMeL Tools, a Gemini key, and the resident transliteration server.
  • Transliteration — commands, the plugin’s local romanization server and its backend options (Python engines or LLM), romanization options, the user lexicon.
  • Transcription — the Gemini OCR pipeline, post-processing, status, and reprocessing.
  • Translation — translating a finished transcription.
  • Document import — collections, the Manage text sources dialogue, naming, and updates.
  • Versions, removal, and cleanup — Version 0, saved versions, re-rendering, removal, and model deletion.
  • Managed regions, identity, and adoption — managed tags, re-render boundaries, adoption, Unadopt, and identity cleanup.
  • Templates — the ETA templates and the render-model variable reference.
  • Zotero — Local / Web / CSL-JSON export, update-in-place, and auto-sync.
  • Midad Arabic Transliteration Zotero plugin — the optional Zotero 7 companion (downloaded and installed separately from the Obsidian plugin) that romanizes an item's Arabic fields using Midad’s local transliteration server.
  • Commands — every command, grouped by function, including command-line functions that work outside Obsidian.
  • Troubleshooting — the common errors and how to fix them.

What it does

Read imported sources in Reading view. The generated layout — Qur'ān verse emblems and markers, the gilded quotes, and a tafsīr's margin references — is drawn for Reading view, and that is where annotation works; keep Live Preview for the note you are actively editing. See Templates.

Transliteration

The rule-based/LLM romanizer talks to a resident Python service (CAMeL), so Obsidian and Zotero do not reload the model on every call.

  • Transliterate selection to clipboard — romanize the selection (or current line) to the clipboard.
  • Transliterate note to new note — romanize a whole note or selection into a sibling note.
  • Check transliteration server — report loaded engines and available LLM providers.
  • Start transliteration server / Stop transliteration server — launch or stop the service without a terminal.

Settings cover the server URL and script, the engine (bert / mle / none), sun-letter assimilation, case endings, name mode, final dotless yāʾ, and the LLM configuration assigned to transliteration. See Transliteration.

Transcription and translation (scanned PDFs)

A Gemini pipeline turns a literature item’s PDFs into page-separated Arabic markdown (and translations), with local post-processing for Qurʾān quotes, poetry callouts, footnotes, tashkīl, and (in a tafsīr) a ##### verse heading wherever a new āya is introduced. The Zotero route assumes a @citekey literature note with the item’s zotero-key and attachments, normally created by ScholarWeft or ZotLit; Midad does not create those notes. Without one, use Transcribe a PDF….

  • Transcribe this item’s PDF file(s) — burst to page PDFs, OCR via Gemini batch mode, post-process.
  • Transcribe a PDF… — pick any PDF on disk; Gemini guesses the title and author from the title page, and a verification dialog saves them into the new transcription’s bibliography.
  • Translate this item’s transcription(s) — translate an existing transcription.
  • Prompt dropdown in the transcription and translation dialogs — choose a named prompt, or edit, copy, and delete prompts for texts the built-in one does not suit.
  • Reprocess from archive… — re-run post-processing from an archived batch without resubmitting.
  • Check transcription pipeline, Show pipeline status, Open last pipeline log — status and diagnostics (non-modal corner panel).

See Transcription and Translation.

Text sources (born-digital corpora)

A catalog-driven importer pulls classical texts from public repositories into your vault, one note per sūra, chapter, or dictionary root, with full frontmatter and links back to the Qurʾān text.

  • Manage text sources… — a two-pane dialogue (install/uninstall, or send references to Zotero) with a genre sidebar, a repository list, and one clickable row per work.
  • Copy import templates to vault — copy the editable ETA templates into your vault.
  • Re-render imported sources — regenerate every note from its cached model.
  • Check imported sources — find works whose notes or cached model are missing.
  • Check for source updates — ask the repositories whether a text changed upstream.
  • Archive this source… — zip the work's model and rendered notes, remove the live notes, and preserve its annotations; restore it from Manage text sources.
  • Manage versions… — save, select, compare, and restore note versions. Version 0 is the installed model/archive; saved user versions live in per-work archives and are reapplied automatically when a source is restored.
  • Remove a source safely — user-written text outside the managed region is kept by default; the removal dialog also offers full deletion and model-only deletion.

Collections currently wired in: Qurʾān commentaries (tafsīr) from spa5k/tafsir_api and the Qurʾānic Universal Library; Qurʾān translations from fawazahmed0/quran-api; ḥadīth collections from AhmedBaset/hadith-json; the Qurʾān in its riwāyāt from risan/quran-json; two dictionaries, Hans Wehr’s Dictionary of Modern Written Arabic (read from its FODT edition) and Lane’s An Arabic-English Lexicon (read from its SQLite edition), one note per root; and classical books from Turath.io (one note per TOC chapter, a volume index per volume, two category MOCs — Turath and the work type). Folders and hub files use Author - Title (repository) (for example Aḥmad b. Ḥanbal - Musnad Aḥmad (hadith-json)); the dictionaries and Qur'ān editions keep their established titles. Imports are cached, and re-rendering never overwrites your edits: generated content sits in a managed region and only template-declared frontmatter properties are reconciled. See Document import and Templates.

Annotations

Select text inside an imported source to annotate it without changing the generated note. Annotations are stored separately, painted in Live Preview and Reading view, shown through margin icons and a comment tooltip, and listed in the unified Midad sidebar. The sidebar has a Navigation tab for every note — midad-up parents, siblings, sections, and child notes — and, on annotatable notes, an Annotations tab with click-to-scroll cards. It stays available as an ordinary background right-sidebar tab: on a non-Midad note it does not steal focus and shows the collection index, while a Midad note brings its navigation forward and leaving returns the right sidebar to the previous tab. In Live Preview, the active annotation shows oval resize handles and links are treated as atomic selections. Each work gets a generated annotation note and the shared Source Annotations MOC; the note uses a light blue annotation callout; comments come first on a gradient inset, and each quote follows as highlighted text on a white inset page. Re-renders relocate quotes by exact text and context; unresolved quotes stay in the note's Unanchored section and can be recovered from a new selection. Copy source with annotations copies the source plus a colour-callout annotation section. Archiving a source keeps its annotations. Known transcriptions and translations are adopted automatically; Adopt note for annotations covers other Markdown notes, and multi-volume series share one annotation note.

Zotero export

Installed works can be sent to Zotero as book items, each tagged Midad Arabic Sources and filed in a collection of that name. Settings choose the method — Local API (the Zotero running on :23119), Web API (key + library id), or CSL-JSON (writes a file for a manual File → Import). Already-exported items are updated in place rather than duplicated. See Zotero.

Requirements

  • Obsidian 1.13 or later.
  • Python 3 with PyMuPDF (fitz), requests, and google-genai. The plugin finds a usable interpreter automatically and lets you pin one in settings.
  • CAMeL Tools for the transliteration engine, with its model data.
  • A Gemini API key in the LLM configuration assigned to transcription and translation, plus optional LLM configurations for romanization.
  • Zotero (optional) for sending imported works to your library.

See Dependencies for the exact install lines.

Fonts

The plugin downloads no fonts and embeds none. It requests the faces it was designed with, then falls back through fonts that ship with macOS, Windows, and Linux, so Arabic, transliteration, and Qur'ānic text always render. For the intended typography, install these (all free):

Face Used for Licence
Noto Serif Latin text and transliteration SIL OFL
Scheherazade New Arabic text SIL OFL
Noto Naskh Arabic Arabic fallback SIL OFL
DigitalKhatt New Madina Qur'ānic text (preferred) Free from the DigitalKhatt project
Amiri Quran Qur'ānic text fallback SIL OFL
KFGQPC Uthmanic Script HAFS ۝ verse emblem Free from the King Fahd Complex

The stacks live in styles.css as --latin-font, --arabic-font, and --quran-font; override those variables in a CSS snippet to use different faces. The fallback chains are documented in Setup.

Install

  1. Open Settings → Community plugins and turn off Restricted Mode.
  2. Choose Browse, search for Midad Arabic Sources, then Install and Enable.

If the listing is not visible yet — the plugin is submitted for review — use BRAT below.

BRAT (alternative, latest beta)

  1. Disable Restricted Mode, then install and enable BRAT from the Community Plugins list.
  2. In BRAT’s settings, add nebedaay/Midad-Arabic-Sources to the Beta plugin list.
  3. Enable Midad Arabic Sources in Community Plugins. BRAT keeps it updated.

Manual

Download main.js, manifest.json, and styles.css from the latest release and copy them into your plugin folder (<config folder>/plugins/midad-arabic-sources/; see Settings → Files and links → Override config folder) and then enable the plugin in Settings → Community plugins. The plugin extracts its Python scripts and import templates into that folder on load.

Network use and file access

Core importing and rendering use only the repositories you import from. The following features contact services; nothing is sent anywhere else, and the plugin collects no telemetry.

Feature Contacts Why
Text source import The public repositories listed above (GitHub, qul.tarteel.ai, turath.io) Downloads the source texts you choose to import.
Transliteration transliterate-server.py on 127.0.0.1 Romanizes Arabic. Optionally calls Gemini, DeepSeek, or an OpenAI-compatible endpoint when LLM mode is on.
Transcription / translation The Gemini API OCRs and translates scanned PDFs you submit.
Zotero export Your local Zotero (127.0.0.1:23119) or api.zotero.org Reads and writes the bibliographic items you export.
Midad Arabic Transliteration (optional Zotero companion) The local transliteration server (127.0.0.1:8756) Romanizes the Arabic fields of a Zotero item.

Fonts are not fetched: the plugin uses locally installed faces with system fallbacks (see Fonts).

Files outside your vault: the transcription pipeline runs external programs and reads and writes the source PDFs and output files you choose, including absolute paths. This is required to OCR and translate documents.

How it works

The importer is acquisition only: it downloads a work’s source, normalises it into a language-neutral render model, and renders that model into notes with ETA templates — the same template language used by ScholarWeft. Regenerating a note never overwrites your edits. A content-hash timeline decides whether a template change affects notes, and a two-stage check asks each repository whether a text changed upstream. The plugin ships its Python scripts, ETA templates, and CSS inside main.js, so a release is self-contained.

Building from source

Documentation is edited in the vault's Midad Docs/ folder, which is the source of truth. Run npm run sync-docs to copy it into the development repo's docs/ and README.md; npm run build and npm run dev do this automatically. In CI, where the vault folder is absent, the committed copies are used.

npm install
npm run sync-docs     # Midad Docs/ → docs/ + README.md
npm run check-types   # tsc --noemit
npm run lint          # eslint (Obsidian community-plugin rules)
npm run build         # esbuild → main.js

Copy main.js, manifest.json, and styles.css into your plugin folder (<config folder>/plugins/midad-arabic-sources/), or run npm run deploy from inside a vault. Restart Obsidian after every build — the plugin loads main.js only at startup.

Credits

Midad Arabic Sources’ ETA templating, managed-region merge, and note conventions follow ScholarWeft, which pioneered them. The importer draws on the public corpora of spa5k/tafsir_api, the Qurʾānic Universal Library, fawazahmed0/quran-api, AhmedBaset/hadith-json, risan/quran-json, Turath.io, GibreelAbdullah’s Lane and Hans Wehr dictionary editions, and muhammad-abdurrahman/hans-wehr-fodt-parser for Wehr’s FODT text. Transliteration uses CAMeL Tools.

License

GPL-3.0-or-later.