Everything needed to run it, in the box
A status page, an incident feed, an alert receiver, a ticket desk, a transactional outbox with a dead-letter console and a maintenance cycle — all in the same repository and the same database as the product.
- 32 admin operations
- Probes every 60 seconds
- Atom incident feed
- No status, helpdesk or paging vendor
The admin surface
Thirty-two operations under /v1/admin. Each one is
reachable from the dashboard and from your own scripts, because they
are the same API.
Observability
What the service is doing right now, per tenant rather than in aggregate.
- GET /status Component health, database reachability and configuration state. Redacted unless an admin token is presented.
- GET /metrics Per-tenant counts: memories by type, current versus historical facts, knowledge-base documents, chat requests in 24 hours, errors in 24 hours, and outbox depth.
- GET /logs A tail of the server log through the API, so a first look does not require shell access to the container.
Compliance
Reading the tamper-evident record, and getting it somewhere you can query it.
- GET /audit Filtered audit events for a tenant — operation, memory ID, actor, trigger and the request ID that caused it.
- GET /audit/export The same range streamed as JSONL for a date window, ready for a warehouse or a verifier run.
Privacy
Working an erasure request without a database console.
- GET /pii-vault List the placeholder mappings held for a tenant and user.
- DELETE /pii-vault Remove mappings. The placeholders left behind in stored records become permanently unresolvable.
- GET /pii-vault/export Export the mappings for a subject-access response, gated behind the owner-only reveal permission.
Reliability
A full dead-letter console for the transactional outbox — the five operations that stop a poison event becoming an incident.
- GET /outbox Pending, in-flight and completed events with their attempt counts.
- GET /outbox/dead Events that exhausted OUTBOX_MAX_ATTEMPTS, with the last error recorded against each.
- POST /outbox/retry-dead Requeue dead events in bulk — a specific list of IDs, or up to a limit you pass.
- POST /outbox/{id}/retry Requeue one event after you have fixed the cause.
- POST /outbox/{id}/skip Abandon one event with a written reason, when replaying it would be worse than losing it.
Configuration
Tenant-level settings, with an explicit allowlist so an admin cannot set arbitrary keys.
- GET /settings The stored settings for this tenant, and which backend holds them.
- POST /settings Update them. Keys outside the allowlist are dropped, not rejected loudly.
- GET /settings/schema The allowed keys grouped into General, Retrieval, Quotas and Privacy — enough for a settings screen to build itself.
Onboarding
Provisioning state that survives a page refresh and a change of operator.
- GET /onboarding Which steps a tenant has completed, and when.
- POST /onboarding/{step} Mark a step complete with a payload. A settings object inside it is applied through the same allowlist.
Tenant lifecycle
The GDPR operations, each behind a typed confirmation.
- POST /tenants/{id}/export Queue a full tenant export. Requires the literal string DELETE-<tenant_id> in the body.
- GET /tenants/{id}/export/{job} Poll until the archive is ready, then collect it.
- POST /tenants/{id}/delete Soft-delete with a grace period, after which a scheduler hard-purges the rows.
Incidents, support and control
Mounted under the same /v1/admin prefix by the status, support and alert routers.
- Status components & incidents Create and update components; open an incident, post updates against it, and resolve it.
- Support queue The cross-tenant staff queue, assignment, status changes and staff replies.
- Alert history Everything the alert receiver has accepted, filterable by severity.
- POST /control system_start, system_restart, system_shutdown and services_status, shelling out to docker compose. Returns 403 unless SYSTEM_CONTROL_ENABLED=true.
The graph cannot quietly drift from the database
Writing a memory has consequences beyond one row: the entity graph needs updating, facts may need extracting, a knowledge source may need refreshing, an email may need sending. Doing that work inline makes the write slow and fragile. Doing it afterwards, in a separate queue, means the two can disagree the first time something fails between them.
So the side effect is enqueued in the same transaction as the write. Either both land or neither does. There is no window in which the memory exists and the follow-up work has been forgotten.
Workers claim events with FOR UPDATE SKIP LOCKED, so
you scale out by starting more of them rather than by partitioning
anything. Failures retry with bounded backoff up to
OUTBOX_MAX_ATTEMPTS, which defaults to five, after
which the event moves to dead — a poison event parks
itself instead of blocking the queue behind it. From there the five
outbox operations above are your console.
INSERT INTO memories …
INSERT INTO outbox_events
(event_type, payload, status)
VALUES ('GRAPH_UPSERT', …, 'pending')
COMMIT
// any number of workers, no coordination
SELECT … FOR UPDATE SKIP LOCKED
// after 5 attempts
status → dead // parked, not blocking
Telling people it is broken
Two mechanisms: one for customers, one for whoever is carrying the phone.
The status page
GET /status/ returns components and incidents —
active, and resolved in the last 30 days.
GET /status/probe runs the checks on demand. A
scheduler runs the same probes every 60 seconds and keeps a ring
buffer of results, so the page reflects the last few minutes
rather than the moment you loaded it.
Operators create components, open an incident with a severity of
investigating, identified,
monitoring or resolved, post updates
against it, and resolve it. Resolving writes a final update as
well as closing the incident, so the timeline reads as a story
rather than a state change.
GET /status/atom.xml publishes the same incidents as
a subscribable feed. Customers who want to be told do not have to
remember to check.
The live feed.
The alert receiver
POST /_/alerts/{p1|p2|p3} accepts an alert
from whatever is watching — a rule name, an optional runbook ID,
recent request IDs and affected tenant IDs. The
X-Alert-Token is compared in constant time and the
endpoint is limited to 10 requests a minute per IP, because an
alert receiver that can be flooded is a denial-of-service target.
p1 opens a P1 ticket, assigns it to
ON_CALL_USER and emails
SUPPORT_P1_EMAIL_TO using the
alert_p1 template.
p2 opens a P2 ticket without the email.
p3 is logged only.
Identical alerts inside a 60-second window are deduplicated against a hash of the payload and return the existing ticket ID, so a flapping check produces one ticket rather than forty.
The support desk is part of the product
Tickets live in the same Postgres, under the same row-level security, with the same audit trail as everything else. There is no third-party helpdesk, and customer content is never handed to one.
A floating widget in the dashboard captures the page the customer is
on and the latest X-Request-ID the API client saw, and
attaches both to the ticket. POST /v1/support/tickets
creates the ticket and its first message inside the tenant's RLS
context; the customer can reply on the thread.
Staff holding admin:ops work a cross-tenant queue
ordered by priority and then by age. They can assign — which moves
the ticket to in_progress — reply, mark it
waiting_on_customer, resolve or close. Every create,
reply, assignment and status change writes an audit event carrying
the request ID.
Email is the Python standard library over SMTP, with seven templates
in the repository and a noop backend for development
that records what would have been sent. A send is attempted three
times with exponential backoff; a permanent failure — a refused
recipient, a 5xx response — is not retried at all, and a transient
one that exhausts its attempts is enqueued as an
EMAIL_SEND outbox event so the worker can pick it up.
Ticket lifecycle
Email templates
A CI script enforces the boundary rather than trusting it: it scans the backend, the docs, the dashboard source and the workflows for the names of hosted status, paging and email-delivery vendors, and fails the build on a hit. Auth0 and Stripe are the two explicitly permitted exceptions.
The sleep cycle
Memory that only grows becomes a cost centre and then a retrieval problem. A scheduled maintenance pass does the tidying that a running request has no time for.
RETENTION_THRESHOLD. Pinned
records are exempt. Work is paged —
MAINTENANCE_PAGE_SIZE rows at a time, up to
MAINTENANCE_MAX_PAGES_PER_RUN — so a large tenant does
not produce one enormous transaction.
CONSOLIDATE_MIN_AGE_HOURS and
CONSOLIDATE_MAX_AGE_HOURS old, above a cosine threshold
of 0.78, are clustered and consolidated into a semantic summary. The
write is atomic: the summary and the archival of its sources land
together, or neither does.
FACT_EXTRACT events are drained away from the request
path and write bitemporal facts, gated on
FACTS_MIN_CONFIDENCE. A chat turn is never slowed down
by extraction, and a failed extraction never fails a chat turn.
GRAPH_UPSERT and GRAPH_DELETE events are
applied to the entity graph, keeping it consistent with the rows that
caused them.
scripts/backup.py takes a timestamped
pg_dump and prunes to the last N archives;
restore.py is the other half and is meant to be
rehearsed. The production compose file runs a dedicated backup
service so this is not something an operator has to remember to wire
up.
Targets, and which of them is measured
These are engineering targets. Only the first is enforced by anything, and we would rather label the difference than let a table imply otherwise.
| Target | Value | Status |
|---|---|---|
| Chat p95 latency | < 1.5 s | Enforced. k6 runs on every push to main and a threshold miss fails the build. |
| Memory CRUD p95 latency | < 200 ms | Documented goal. Not gated in CI. |
| Health endpoint p95 latency | < 100 ms | Documented goal. Not gated in CI. |
| Outbox processing lag p95 | < 60 s | Documented goal. Not gated in CI. |
| Monthly availability | 99.9% | Documented goal. Not a contractual commitment. |
Support response times
| Plan | Target response |
|---|---|
| Free | Best effort. |
| Pro | Reply within two business days. |
| Team | Reply within one business day. |
| Enterprise | P1 routing, and on-call email through SUPPORT_P1_EMAIL_TO. |
Two documents in the repository quote different support commitments. Where they disagree we publish the conservative one, which is the set above.
Single region, with a written playbook
Multi-region is documented, not implemented. There is an active/passive playbook covering a primary region, a standby with point-in-time recovery, and the order in which Postgres, the graph snapshots and the audit artefacts must be restored before DNS moves.
Read-replica routing is in the same state.
POSTGRES_READ_DSN exists as a configuration field and
has no consumers in the code — nothing reads from a replica today.
Treat the deployment as single-region with a rehearsed disaster
recovery procedure, and plan accordingly.
Worker scale-out, by contrast, is real: multiple outbox workers are
safe today because of SKIP LOCKED claiming.
Give your agents a memory you can audit
Run the whole system on your own hardware under the MIT licence, or ask us about hosted access. Both start from the same place.