RAG vs Memory vs Context: Three Different Jobs in Your AI Assistant

Image: Jean Vella on Unsplash.

Context is what the model sees for one call. Retrieval, often called RAG, finds the external evidence relevant to a question. Memory stores what should persist across tasks. They work together, but they are different jobs, and confusing them causes most "my assistant forgot" and "my assistant made that up" bugs.

The short version:

context is the desk,
retrieval is the trip to the library,
memory is the notebook you carry between trips.

This guide is part of the AI harness engineering guide.

Definitions, side by side

	Context	Retrieval (RAG)	Memory
Holds	Everything sent with one model call	Passages found in external documents	Facts and preferences meant to persist
Source of truth	Assembled each time	The original documents	Its own records
Lifetime	One call	One task	Until corrected or deleted
Typical failure	Overloaded, or missing the key fact	Wrong passage, stale index	Outdated fact still treated as true
Main control	What you include and where	What you search and how	What you save and how you correct it

Anthropic describes context engineering as curating what enters a model's limited context window over the course of a task (Anthropic). Retrieval and memory are two of the main sources that feed it.

Follow one question through all three

The question: "Can I move Thursday's Project Alpha review to the afternoon?"

1. Memory supplies a preference: you prefer meetings after 13:00. It carries a source and a date.
2. Retrieval finds evidence in your documents: the project plan says the review needs Sam, and Sam's notes say Thursday afternoons are blocked.
3. Context assembles the call: your question, the preference, the two passages with their sources, and an instruction to answer only from them.

The model answers: "Probably not Thursday afternoon: Sam's notes say Thursday afternoons are blocked. Friday afternoon would match your preference."

Each layer did one job. Remove memory and it forgets your preference. Remove retrieval and it guesses about Sam. Stuff every document into context and the key passage may get lost.

Choose what belongs in each layer

Information	Belongs in	Why
"I prefer afternoon meetings"	Memory	Stable, personal, reused across tasks
The project plan	Retrieval, from the original file	It changes; copies go stale
"Step 2 of 4 done"	Task state, not memory	Only matters until the task ends
Today's question and instructions	Context	Only needed for this call
A fact stated in a document	Retrieval, cited	The document is the authority, not a copy

The rule of thumb: if a fact has an authoritative source document, retrieve it from there each time. Save something to memory only when no better source exists, such as a preference you stated.

Preserve provenance and resolve conflicts

Conflicts are normal. A good assistant shows them rather than resolving them silently.

Memory vs a newer document. Memory says Sam owns Project Alpha. A document from last week says ownership moved to Priya. Prefer the newer, sourced statement, and flag the stale memory for correction.

Two documents disagree. Report both, with dates and sources, and say which you used and why.

No evidence. Say "not found." An honest gap beats a fluent guess.

All three depend on provenance: every passage and memory record must carry where it came from and when. Without that, the model cannot tell new from old, and neither can you.

Manage context without relying on capacity alone

Bigger context windows help, but they do not remove the need to choose.

A 2023 study of the language models of the time found that accuracy on multi-document question answering was often highest when the relevant passage sat at the beginning or end of a long input, and degraded when it sat in the middle (Liu et al.). Current models differ, so do not treat this as a law. Treat it as a reason to test.

A simple evidence-placement test:

1. Pick ten questions whose answers sit in one known passage.
2. Build contexts where that passage appears at the start, middle and end of the same set of distractors.
3. Run each version several times and record accuracy by position.
4. Repeat after changing models or context size.

If accuracy drops with position, select fewer, better passages, or put the most relevant first. Anthropic also describes compaction, structured note-taking and sub-agent architectures for keeping long tasks inside a useful context (Anthropic).

Choose the simplest useful implementation

You do not need every technique on day one.

Situation	Start with
A few dozen notes	Plain files and keyword search
Hundreds of documents, varied wording	Add a search index; consider embeddings
Preferences and personal facts	A small memory store with sources and status
Long multi-step tasks	Checkpointed task state and summaries

Add complexity when a test shows you need it, such as retrieval missing answers that keyword search should find. Each addition is something else to keep in sync when documents change or are deleted.

For the memory lifecycle, see the personal AI memory guide. For a complete retrieval workflow with citation checks, see the personal knowledge base guide. For the research view, see AI agent memory explained.

Frequently asked questions

Is RAG the same as long-term memory?

No. RAG retrieves passages from documents that remain the source of truth. Memory stores its own records meant to persist.

Does a longer context window remove the need for retrieval?

No. You still need to choose relevant evidence, and placement within long inputs can affect accuracy. Test it on your setup.

Do I always need a vector database?

No. For small collections, plain search often works. Add embeddings when testing shows you need them.

Where should a user's corrections live?

In memory, as a new record that supersedes the old one, with your correction as its source.