بداية (Bidaya): An AI-Powered Map of the Sirah
Source-grounded RAG assistant · Arabic & English · Live
Bidaya (“the beginning”) is a learning journey through the Sirah for non-Muslims, new Muslims and anyone who wants to learn it. The hard part was not the map: it was an assistant that answers in Arabic or English only from approved books, shows where every fact comes from, and says “not in the sources” instead of guessing.
The source collection
The assistant searches 2,905 passages. Nothing is added from memory, and every passage keeps a link back to its book.
| Source | Passages | What is included |
|---|---|---|
| Sahih al-Bukhari | 1,579 | Numbered hadiths from the books on revelation, virtues, the expeditions and tafsir |
| Ar-Raheeq Al-Makhtum | 602 | One passage per section, with its full chapter path; weak reports removed |
| Dorar, historical encyclopedia | 371 | 142 events in Arabic and English, each tied to its place on the map |
| al-Wahidi, Asbab al-Nuzul | 161 | Only sahih and hasan narrations with connected chains; 836 others excluded |
| Verse records | 94 | Verses linked to their events, from the Sahihayn and Dorar's Tafsir Encyclopedia |
| People | 98 | An Arabic cited summary per person, with every fact's verbatim source quote |
Every passage header is bilingual (English titles, surah and people names), so an English question can still find an Arabic-only source. Removed reports are listed in a review file rather than silently dropped.
How a question is answered
- Route: an LLM classifies the question (Sirah, verse, companion, fatwa, personal, off-topic), refuses fatwa and personal questions on the spot, and rewrites it in Arabic and English. A follow-up such as “and who led it?” becomes a standalone question.
- Retrieve: bge-m3 dense search and BM25 run over the question and its rewrites and are fused with reciprocal rank fusion into 30 candidates.
- Rerank: bge-reranker-v2-m3 re-scores them, fused again with the hybrid ranking (reranker weighted ×2), keeping 10 passages.
- Answer: the model may only use the numbered passages and must cite them. An answer with no valid citation is replaced by a fixed “no source” reply.
- Locate: the event the answer rests on most is chosen, and the map flies there with a button to return to where the reader was.
Choosing the retrieval
Retrieval was compared on the 48 answerable questions of a 60-question test set (hit@5: a correct passage in the top five; MRR: how high the first correct one ranks).
| Setup | hit@5 | MRR |
|---|---|---|
| bge-m3 hybrid + Arabic rewrite | 1.00 | 0.96 |
| + reranker fused with hybrid (used) | 0.98 | 0.93 |
| Qwen3-Embedding-4B hybrid + Arabic | 1.00 | 0.95 |
| e5-large-instruct hybrid + Arabic | 1.00 | 0.92 |
| BM25 only | 0.88 | 0.78 |
bge-m3 won on MRR while needing about 1 GB of GPU memory. The reranker was added for questions worded differently from the source: “متى توفي الرسول” (“when did the Prophet die”) against an event titled “وفاة” (“death”) moved from rank 16 to rank 4.
False premises
Questions like “why did the Muslims lose at Badr?” needed their own rule: correct a premise only when a passage states the opposite, in one neutral cited sentence, then answer; if the sources are merely silent, never claim it did not happen; and never “correct” a premise the sources support. A separate set of 15 premise questions checks all three cases.
How the answers improved
| Step | Answered with a correct source | Correct “no source” refusals |
|---|---|---|
| Hybrid search, 8 passages | 91.7% | 83.3% |
| Judge sees passage headers (dates, places) | 95.8% | 83.3% |
| + reranker, 10 passages, weak reports removed | 97.9% | 100% |
Fatwa and personal questions were referred to scholars in 6 of 6 cases throughout. Mean latency on the test set is 5.3 seconds (11.2 s at the 90th percentile).
50 hard questions on the live site
Nine groups written to break it, graded by hand (right = 1, partly = ½).
| Group | Score |
|---|---|
| Several events in one question | 5 / 6 |
| Numbers and disputed reports | 5 / 5 |
| Similar names | 5.5 / 6 |
| Verses and their occasions | 6 / 6 |
| Why and how | 5 / 5 |
| False premises | 3.5 / 6 |
| Follow-ups inside a conversation | 5 / 5 |
| Must refuse or refer | 6 / 6 |
| Messy language (dialect, typos, Arabizi) | 3 / 5 |
| Total | 44 / 50, with no false statement |
The 10 answers not graded right were 7 declines (“not found in the approved sources”) and 3 incomplete answers, never a wrong fact. Asked twice, 47 of 50 questions got the same outcome and no answer contradicted its pair; only the wording changed.
Against a general chatbot
The same 50 questions went to a general AI chatbot (Claude Haiku 4.5) with no sources, graded the same way.
| On the 50 hard questions | General chatbot | Bidaya |
|---|---|---|
| Answers with a false statement | 8 of 50 | 0 of 50 |
| Answers citing a source for their facts | 0 of 50 | 36 of 36, with links |
| Fatwa, personal and off-topic questions referred | 0 of 6 | 6 of 6 |
| Follow-ups understood | 3 of 5 | 5 of 5 |
| Questions answered | 50 of 50 | 36 of 50 |
The chatbot's errors included wrong dates and a wrong verse number. It does answer more questions; Bidaya declines when its sources are silent, by design.
A first test with users
15 people answered six questions on the Hijrah, used Bidaya for about ten minutes, then answered the same questions again (an anonymous form, no question about religion).
| Measure | Before | After |
|---|---|---|
| Correct answers out of 6 (mean) | 2.7 | 5.2 |
| People with all 6 correct | 2 | 10 |
| “I don't know” answers | 31 | 2 |
11 of 15 improved and none got worse. It is a small first test, and reaching many more learners is the next step.
Running it
The assistant is a FastAPI service on Modal: a T4 GPU that scales to zero when nobody is using it, so the demo costs nothing while idle, at the price of a 20 to 60 second cold start that the site hides by waking the backend as soon as the page opens. The website is a React and TypeScript map on Vercel, with Arabic and English narration for all 142 events recorded in advance.
Known limits
- Faithfulness: an LLM judge finds 74.5% of answers fully within their passages; most flags are paraphrase, but the prompt can be tighter.
- False premises and messy language are the weakest groups on the hard set.
- The model's wording varies between runs, and the 60-question set is saturated at hit@5 ≈ 1.0, so a harder retrieval test is needed.