Pcnaid Memory OS
Get started

Documents that go stale on purpose

A knowledge base that never re-reads its sources is a snapshot with ambitions. This one gives every source its own freshness clock, tells you when one has gone off, asks before it fetches anything, does the fetching in a window you chose, and keeps the old text when the new text arrives.

  • 9 endpoints
  • Per-source refresh TTL
  • Nightly window in your timezone
  • Versioned, never replaced

The refresh cycle

Five steps, and a human decision sits in the middle of them on purpose. Automatic fetching of arbitrary URLs is how a knowledge base becomes an attack surface.

  1. A source goes past its TTL

    Each source carries its own refresh_ttl_seconds, defaulting to seven days and settable from five minutes to a year. Nothing runs on a global cron that ignores what the document actually is — a price list and a company handbook do not need the same cadence.

  2. The system suggests, it does not act

    A stale source produces a row in kb_suggestions with a reason. Listing suggestions is what generates them, so the queue is always current when you look at it. No refresh happens because a timer fired; a refresh happens because somebody approved one.

  3. You approve, and it is scheduled — not run

    Approval computes the next occurrence of your configured night start in your configured timezone and enqueues a KB_REFRESH_SOURCE event with that availability time. Nothing before that moment will claim it. If you need it sooner, queue it with when=now — and either way, KB mutations are rate-limited to ten per tenant per hour by default.

  4. The worker fetches it, carefully

    A worker claims events with FOR UPDATE SKIP LOCKED, so several workers can drain the same queue safely. The fetch sends the stored ETag and Last-Modified back as conditional headers: a 304 costs almost nothing and simply moves the last-checked timestamp. A failure records the status on the source and retries with a ten-minute backoff rather than losing the job.

  5. New text supersedes old text without deleting it

    HTML is reduced to text with script, style and noscript content dropped. The text is chunked at 1,200 characters with 150 characters of overlap, each chunk is embedded locally, and the write closes the previous version’s system_time range before inserting the new rows with an open one. Yesterday’s wording is still readable; only “current” moved.

Staleness is detected automatically, refreshes are approved deliberately, and the work happens in a window you set — in your timezone, under your nightly job cap.

Nine endpoints

Reads take an API token. Every mutation — adding a source, changing the window, approving a refresh — additionally requires an admin token, because each one can cause the server to make an outbound request on your behalf.

EndpointWhat it does
GET /v1/kb/status Active sources, open suggestions, approved suggestions. Reports enabled:false rather than erroring when KB is off.
GET /v1/kb/sources Every source with its URI, type, TTL, last check, last status and stored validators.
POST /v1/kb/sources Add or update a source. Quota-checked on create; refresh TTL from five minutes to a year.
POST /v1/kb/sources/{id}/toggle Deactivate a source without deleting it. Inactive sources are skipped by the worker and marked as such.
POST /v1/kb/sources/{id}/queue Queue a refresh now, or at the next night window in the tenant’s timezone.
GET /v1/kb/settings The tenant’s refresh window, timezone, nightly job cap and enabled flag.
POST /v1/kb/settings Change them. Night start and end are wall-clock times in the timezone you set.
GET /v1/kb/suggestions Open suggestions — and generating them is a side effect of asking, so stale sources surface on read.
POST /v1/kb/suggestions/{id}/approve · snooze · dismiss Approve schedules the refresh for the next window. Snooze defers by one to 365 days. Dismiss closes it.

Fetching a URL somebody else chose is the dangerous part

The moment a product will fetch a URL on request, it becomes a proxy into whatever network it is running in — cloud metadata endpoints, internal admin panels, a database on a private subnet. The usual defence validates the hostname and then hands the URL to an HTTP client that resolves it again, which is a race an attacker can win.

Scheme and credentials
Only http and https. A URL carrying a username or password is rejected outright rather than sanitised.
Domain allowlist
Suffix-matched against KB_ALLOWLIST_DOMAINS. Note that this setting is empty by default, which means no domain restriction — set it before you let anyone add sources.
Resolve once, then pin
DNS is resolved a single time and every returned address is checked. The request then connects to that exact IP.
Private space blocked
Private, loopback, link-local, multicast, reserved and unspecified addresses fail the check. On by default.
Rebinding closed
Because the connection goes to the pinned address while the original Host header and TLS hostname verification are preserved, a name that resolves publicly on check and privately on fetch has nothing to exploit.
Redirects revalidated
Up to five redirects, each target fully revalidated and freshly pinned. Conditional headers are stripped across the chain rather than leaked to a new host.
Byte and header caps
Bodies are capped at two megabytes, enforced on both the content-length and chunked paths. A single header line over 8 KB, or headers over 64 KB in total, terminate the read. Twenty-second timeout.

All of it lives in one readable file, backend/app/kb/ssrf.py, rather than being spread across middleware. Security you cannot read in one sitting is security you cannot audit.

What gets stored, and how it is found again

A source is a row in kb_sources holding its URI, type, TTL, last check time, last status and the ETag and Last-Modified values used for conditional requests. Its content lives in kb_documents, one row per chunk, each with a content_hash, a chunk index, a vector embedding and a search_vector generated by Postgres.

Documents carry a system_time range exactly like facts do. Current chunks are the ones whose range is still open, and a partial index makes that the cheap query. Refreshing a source closes the old range and opens a new one — so “what did this page say last quarter” remains a question with an answer.

Retrieval over KB chunks is vector search restricted to current chunks of active sources, and each extract that reaches a context pack carries its source name into the manifest. Citations are a property of the storage, not a prompt instruction.

Stored and refreshed is not the same as injected

ENABLE_KB_RETRIEVAL defaults to false. With the default configuration, knowledge-base sources are fetched, chunked, embedded, versioned and searchable through the API — and they do not enter a context pack. Nothing is quietly pushed into your agent's prompt because you added a URL.

That is the right default for a feature whose input is text somebody else controls. Turn it on when you have set your allowlist and you are satisfied with what the sources contain. When you do, KB extracts are fenced as evidence like everything else in the pack, and they sit at 0.66 on the trust table — above a raw conversation turn, below a verified policy.

Source quotas are per plan and enforced as a hard block: exceeding the count returns 402 with the metric and the limit rather than billing you for the overage.

One more thing we would rather you heard from us. The nightly job cap is stored on the settings row and returned by the API, but the current worker drains whatever is due in the window and does not yet enforce that ceiling. The bound that does bite today is the request rate limit on KB mutations, and that limiter is in-process — it does not coordinate across replicas.

How untrusted text is fenced

Keep reading

Give your agents a memory you can audit

Run the whole system on your own hardware under the MIT licence, or ask us about hosted access. Both start from the same place.

Get started Read the API docs