Agent memory explained
Almost every agent that claims to remember things has one embeddings table behind it. That design demonstrates beautifully and then fails in a specific, predictable way. This is what it cannot do, and what a fuller model adds.
- 12 live layers
- 4 questions a vector table cannot answer
- One worked retrieval
The demo that works, and the month it stops working
The standard recipe is three steps long. Chunk everything the agent sees, embed the chunks, and at question time retrieve the nearest neighbours and paste them above the prompt. It is a good recipe. It takes an afternoon, it is genuinely useful, and for a surprising range of problems it is the correct amount of engineering.
It also has a distinctive failure curve. Week one is impressive. Week four the agent starts contradicting itself. By month three it is confidently telling you something that was true in March, and nobody can explain why it thinks so, because the only thing the system recorded was that a sentence existed and roughly what it meant.
The reason is not the embedding model, and it is not the chunk size. It is that similarity is the only relation the store knows. Everything else that makes a memory useful — when it was true, who asserted it, what it replaced, whether anyone checked it — was never a column, so it was never stored, so no amount of clever retrieval can recover it.
Four questions a vector table cannot answer
Not because the implementation is weak, but because the answers were never written down. These are the four that cost you.
-
Is this still true?
A chunk has no notion of expiry. "The client prefers Tuesday calls" and "the client now prefers Thursday calls" are both excellent matches for a question about scheduling, and cosine similarity has no opinion about which one is current. The retriever returns both, the model picks one, and which one it picks is essentially luck.
-
Who decided it, and on what evidence?
Text says what was said. It does not say that this was a decision, that a named person made it, that two other options were rejected, or which measurement settled the argument. Six months later the decision is still in force and the reasoning is unrecoverable — it was never a field, so it was never stored.
-
What did this replace?
When a preference changes, the usual implementation overwrites the row or writes a second one. Overwriting destroys history. Writing a second one leaves two contradictory chunks with no link between them. Neither can answer "what did we believe on the day we made that decision?", which is the first question every post-mortem asks.
-
Can this text be trusted?
A sentence a person confirmed, a sentence a model generated, a sentence scraped from a vendor page and a sentence pasted from an unread email are four very different things. In an embeddings table they are one thing: text with a vector. There is no field that distinguishes them, so nothing downstream can weigh them differently.
Twelve layers, and the failure each one prevents
A layer earns its place by preventing a specific, nameable failure. Here is the whole set with the failure it exists to stop — if a layer cannot be justified this way, it should not be in the system.
| Layer | What it stores | The failure it prevents |
|---|---|---|
| Semantic | Durable statements: facts, preferences, project knowledge that should outlive the conversation that produced them. | The agent re-asks what it was told last week, because the transcript scrolled out of the window. |
| Episodic | The actual conversation turns, with provenance attached to each one. | A summary drifts from what was really said, and nobody can go back to the original sentence to check. |
| Bitemporal facts | Subject–predicate–object with two time ranges: when it was true, and when we believed it. New facts supersede old ones instead of overwriting them. | A superseded preference is retrieved as though it were current, and the agent confidently books the wrong day. |
| Fabric | Decisions, tasks, reviews, outcomes, procedures, handoffs and notes as typed entries with assignees, evidence links and status. | "Why did we choose this database?" becomes an archaeology project across three chat threads and one person's memory. |
| Sessions | Imported transcripts with rolling summaries, and promotion of what mattered into durable memory. | A four-hour conversation is either stored whole and drowns retrieval, or discarded and lost entirely. |
| Knowledge base | Registered sources that are fetched, chunked, embedded and refreshed, with citations back to the document. | The agent quotes a policy that changed two months ago, with nothing to point at when someone asks where it got that. |
| Graph | Entities and the edges between them, so a question about a client can reach the projects, people and decisions attached to it. | Related material is invisible because it never happens to be lexically or semantically close to the question. |
| Entity | Structured records for the things a business actually has: clients, vendors, projects, policies. | The same client exists as fourteen slightly different strings, and no query catches all of them. |
| Observation | Recurring patterns and signals noticed over time, stored as first-class records. | "This customer always escalates on Fridays" lives in one person's head and leaves when they do. |
| Procedural | How a thing gets done here — the steps, the order, the local exceptions. | Every agent re-derives the house process from scratch, slightly differently each time. |
| Outcome | What worked, what failed, and what was done next. | The same failed approach is tried again a quarter later, because nothing recorded that it failed. |
| Multimodal | Text derived from other media — transcripts, OCR output, captions — plus content fingerprints. | A voice note or a screenshot is either untouchable or re-processed on every query. |
Two caveats worth stating rather than burying. Graph memory and knowledge-base retrieval are off by default — they are supported, not assumed, and you turn them on when you want them. And a thirteenth table exists in the schema with nothing reading or writing it, so we count twelve. Detail on every layer, including the record shapes, is on the memory layers page.
How the layers interact during one retrieval
Layers are only interesting when they meet. Here is a single question travelling through the pipeline, from the moment it arrives to the moment a model sees the pack.
-
The question is planned, not just embedded
A query planner classifies the question by type and mode and decides which layers are even worth searching. "What did we decide about billing?" and "what is the client's current phone number?" should not run the same search. The planner is rule-based rather than a model call, which keeps it free, fast and predictable — and honest about being heuristic.
-
Identifiers are pseudonymised before anything else touches them
Email addresses and US-format phone numbers are swapped for placeholders before text is embedded or stored, with the real values kept in a tenant-scoped vault. Retrieval therefore operates on pseudonymised text throughout; the values come back only at display.
-
Three searches run, not one
A vector search over pgvector for semantic closeness, a lexical search over a Postgres full-text index for exact terms — names, error codes, identifiers that embeddings blur — and, when graph memory is enabled, a third candidate list from entity edges. Each produces its own ranking.
-
The rankings are fused, not concatenated
Reciprocal rank fusion merges the lists by rank rather than by score, with per-source weights. This matters because a cosine similarity and a text-search rank are not comparable numbers; fusing on position is the part that makes hybrid search actually work rather than merely sound thorough.
-
Candidates are reranked and then deliberately diversified
A reranker reorders the fused list — heuristic by default, with an optional cross-encoder if you configure one — and maximal marginal relevance then trades a little relevance for coverage, so the pack does not contain eight paraphrases of the same sentence and nothing else.
-
Time is applied
A temporal-intent detector classifies the question as asking about the current state, the past, the change between them, or unknown. If the question is about now, facts that a later fact has superseded are suppressed. If it is explicitly historical, they are exactly what you want and the suppression is not applied.
-
Every survivor is scored for trust
A confidence score combines the authority of the source — from raw capture through generated, derived, human-reviewed and locked — with recency, corroboration by other records and any contradiction against them. This is where the difference between "a person confirmed this" and "a model wrote this" finally has somewhere to live.
-
The pack is filled to a budget, in priority order
Context is a fixed-size container, so records are packed by priority until the token budget is spent: 1,800 tokens in cheap mode, 3,500 balanced, 8,000 deep, 14,000 forensic. What falls off the end falls off predictably rather than arbitrarily.
-
The pack arrives fenced, with a manifest
The assembled context is prefixed with the rule that records are data and evidence, never instructions, and is returned with a source manifest naming every record that made it in. You can see what the model saw, which is the difference between an answer and an auditable answer.
Writing is a second loop, not an afterthought
Most of the attention in agent memory goes to retrieval, because that is where the visible cleverness is. But the write path decides what retrieval can possibly do. A record written with no authority, no truth state and no link to what it replaced is a record that no retriever can rank sensibly later.
So the write path carries the fields that make the read path possible: what kind of statement this is, where it came from, whether anyone has checked it, what it supersedes, and who may see it. Facts written today do not overwrite yesterday’s — they close the old one’s belief window and open their own. Fabric entries carry an evidence list at the moment of writing, because reconstructing evidence afterwards is exactly the thing that never happens.
The practical test of a memory system is not whether it can find a sentence. It is whether, a year later, it can tell you what you believed, when you started believing it, and why.
fact client_9 · prefers_call_day · "Tuesday"
valid 2026-01-04 → 2026-06-12
belief 2026-01-04 → 2026-06-12
superseded, still queryable
fact client_9 · prefers_call_day · "Thursday"
valid 2026-06-12 → open
belief 2026-06-12 → open
current
// "what day do they prefer?" → Thursday
// "what did we think in March?" → Tuesday
When you genuinely do not need all this
A memory model has a real cost — in schema, in write discipline, in the operational weight of a database you must now run properly. It is worth being clear about when that cost is not repaid.
Single-session assistants. If nothing needs to survive the conversation, the context window is your memory. Adding a store is machinery in search of a problem.
Static corpora that never change. Documentation search over a frozen manual has no supersession, no authority gradient and no decisions to trace. Embeddings plus a good reranker is the right shape, and adding temporal columns to it buys nothing.
One user, low stakes, short horizon. A personal assistant that remembers your coffee order does not need an audit chain. If being wrong is cheap and correcting it is easy, most of what this page describes is over-engineering.
Prototypes that have not met a real user. Learn what your agent actually needs to remember before designing a schema for it. Guessing produces layers nobody writes to.
The point at which it starts to repay is usually recognisable. More than one person or agent writes to the same store. Facts change and the old value still matters. Someone eventually asks why a decision was made. Content arrives from outside — email, scraped pages, imported transcripts — and its trustworthiness is not uniform. Or someone in a compliance role asks what the agent knew, and when.
If none of those is true for you yet, one embeddings table is the correct answer, and you should build that. Come back when the fourth contradiction lands in the same week.
Give your agents a memory you can audit
Run the whole system on your own hardware under the MIT licence, or ask us about hosted access. Both start from the same place.