Personal Knowledge-Base AI with Verifiable Sources

Image: Zulfugar Karimov on Unsplash.

A personal knowledge-base assistant is only as trustworthy as its citations, and a citation is only useful if the passage it points to actually supports the claim. Build it so every answer links each statement to a specific source passage, says "not found" when your documents are silent, and surfaces conflicts instead of picking a side.

A link is not evidence.

A supporting passage is.

This guide builds a sourced question-answering workflow over a folder of your documents. It extends the read-only quickstart and is part of the AI harness engineering guide.

Define the knowledge base and its boundaries

Decide what is in and what is out before ingesting anything.

Decision	Example
Included folders	projects/, meeting-notes/, reading/
Excluded	finance/, health/, anything with credentials
Allowed destinations	Local search; selected passages to a cloud model under its API terms
Freshness	Re-index on file change, or nightly

Then build test fixtures with known answers. Start with a synthetic set:

• ten questions whose answers sit in one document,
• three questions with no answer anywhere,
• two questions where documents disagree,
• one document you will delete mid-test.

Record the expected answer and the exact supporting passage for each. These become your grading key.

Ingest documents without losing provenance

Each indexed chunk should carry enough metadata to find and judge it:

{
  "doc_id": "projects/alpha-plan.md",
  "chunk": 4,
  "title": "Project Alpha plan",
  "modified": "2026-10-11",
  "text": "Ownership moved from Sam to Priya on 9 October.",
  "hash": "sha256 of the chunk text"
}

Handle extraction failures explicitly. A scanned PDF with no text layer, an encrypted file or an unsupported format should be logged as not indexed, not silently treated as empty. Otherwise "not found" may really mean "never read."

Retrieve evidence for the question

Start simple and measure.

1. Search. Keyword search works well for small, consistent collections. Add embeddings when testing shows keyword search missing answers that use different wording.
2. Filter. Apply scope and freshness rules before ranking. Excluded folders never appear.
3. Select. Keep a small number of the best passages, each with its metadata.
4. Log. Record which passages were selected and which were considered and dropped.

That log is what lets you tell a retrieval failure from a writing failure. For the difference between retrieval, memory and context, see RAG vs memory vs context.

Position can matter in long inputs. A 2023 study found the models of the time often answered better when the relevant passage was near the start or end of the input than in the middle (Liu et al.). Put the strongest passages first, and test whether position affects your model.

Generate answers that can be checked

Ask for answers in a shape you can verify:

Answer: Priya now owns Project Alpha.
Supported by: projects/alpha-plan.md, chunk 4 (modified 2026-10-11):
  "Ownership moved from Sam to Priya on 9 October."

Instruct the model to:

• attach a source to each factual statement,
• quote or point to the exact supporting passage,
• say "not found in your documents" when the passages do not answer,
• mark inferences as inferences.

Then check the output in code where possible: every cited doc_id must be one that was actually retrieved for this question. A citation to a document that was never in context is invented.

Verify that citations support the claims

Retrieved and cited is not the same as supported. Grade each claim:

Verdict	Meaning
Supported	The cited passage states it
Partially supported	The passage supports part of it
Unsupported	The passage does not say it
Contradicted	The passage says otherwise

A model judge can do a first pass, but check it against your own verdicts on a sample and record where it disagrees. Anthropic's guidance on agent evaluations suggests matching graders to the behaviour being measured, using code where possible and model or human graders for judgments (Anthropic).

Test conflicting, missing and stale evidence

These cases separate a useful knowledge base from a confident one.

Case	Expected behaviour
Two documents disagree	Report both, with dates; say which was used and why
A newer document contradicts a remembered fact	Prefer the sourced, newer statement; flag the memory
No document answers	"Not found in your documents"
A document was deleted	Its passages no longer appear in results
A document was edited	Answers reflect the new text after re-indexing
A file failed to extract	It is reported as not indexed

The deletion case catches stale indexes, one of the most common silent failures. If deleted content still appears, the index or a cache was not updated. The memory guide covers the same problem for remembered facts.

Evaluate the complete workflow

Score each run on:

• answer accuracy against your key,
• citation support using the verdicts above,
• retrieval coverage: was the right passage among those retrieved?
• honest gaps: did unanswerable questions get "not found"?
• latency and cost per answered question.

When something fails, locate the layer: ingestion, retrieval, context assembly or generation. Fix that layer and keep the case as a regression test. The evaluation guide shows how to run and report this properly.

Use it for a real personal task

Once the synthetic tests pass, point it at a real folder, starting with something low-risk such as reading notes. Keep the assistant read-only. A useful next step is turning sourced answers into a daily brief; the meeting-prep tutorial does exactly that.

Frequently asked questions

Does a source link prove an answer is correct?

No. Check that the linked passage actually states what the answer claims.

What should happen when documents disagree?

Show both, with dates and sources, and say which one the answer relies on.

Do I need a vector database for a small collection?

Often not. Start with keyword search and add embeddings when tests show it misses answers.

How do I remove a document from future answers?

Delete it, re-index, clear any caches, then confirm a question about it returns "not found."