From 782700d6c6182bd2357a8c3c5190540cc7954d50 Mon Sep 17 00:00:00 2001 From: alexeybe1kin <210597588+alexeybe1kin@users.noreply.github.com> Date: Fri, 25 Sep 2026 21:06:30 +0300 Subject: [PATCH] Rewrite the README as a front page and correct the retrieval claim The README still said embeddings were not shipped, but MemoryGate now calls the Embeddings service and degrades to word search, visibly, when it is missing. It also linked to the old personal repository. The front page now covers what MemoryGate is, how memory is built, retrieval, a quick start and the security model; the dashboard gallery, operations, AI runtime and development sections moved verbatim into docs/. Co-Authored-By: Claude Opus 5.5 --- README.md | 384 ++++++++++---------------------------------- docs/ai-runtime.md | 23 +++ docs/dashboard.md | 75 +++++++++ docs/development.md | 53 ++++++ docs/operations.md | 48 ++++++ docs/security.md | 34 ++++ 6 files changed, 320 insertions(+), 297 deletions(-) create mode 100644 docs/ai-runtime.md create mode 100644 docs/dashboard.md create mode 100644 docs/development.md create mode 100644 docs/operations.md create mode 100644 docs/security.md diff --git a/README.md b/README.md index dc66460..825a2b5 100644 --- a/README.md +++ b/README.md @@ -1,340 +1,130 @@ -# MemoryGate - -Part of **[Conker](https://github.com/alexeybe1kin/conker)**, independently usable and deployable. [Project map](https://github.com/alexeybe1kin/conker/blob/feat/dashboard/docs/conker-project.md) · [Connected local setup](https://github.com/alexeybe1kin/conker/blob/feat/dashboard/docs/local-windows-startup.md). - -Bootstrap read-key configuration is initial setup, not key rotation. Once its -label or credential exists, startup preserves the owner's revocation, agent -assignment and stored hash. Use the owner key-management API to issue replacement -credentials; changing bootstrap environment variables never restores authority. -Retain revoked key rows: they record the decision that restart must respect. - -[Pi conversation memory: admission, retries, forgetting and bilingual drills](docs/conversation-memory.md). - - -MemoryGate is a local-first memory service for one personal AI agent. It receives evidence, preserves lineage, turns durable signals into structured memory, and returns a bounded context package that an agent can use without direct database access. - -It is deliberately not a general chatbot, autonomous executor, or replacement for your main agent. MemoryGate stores, retrieves, and explains knowledge. Your agent remains responsible for reasoning and action. - -## What It Solves - -Personal agents need reliable context without carrying an entire history in every prompt. MemoryGate separates that problem into durable layers: - -| Layer | Purpose | -| --- | --- | -| Evidence | Immutable raw inputs from listeners, sessions, APIs, or manual capture. | -| Analysis | A recorded interpretation of one or more evidence objects. | -| Memory | Durable facts, phases, context, and watch items suitable for retrieval. | -| Entity | Structured people, projects, places, concepts, habits, and objects. | -| Episode | A time-bounded event that groups related evidence. | - -Every object can be inspected in the dashboard alongside its history, links, and supporting material. - -## Architecture - -```text -Agent / Listener - | - v -Evidence ingress --> Evidence object --> Processing job --> Analysis --> Memory / Entity / Episode - | | - +------------------------------ lineage -----------------+ - -Agent read key --> bounded context retrieval --> Memory Lab or agent response +

+

MemoryGate

+

Memory for a personal AI agent, with every fact traceable to its source.
+Evidence in, bounded context out. Nothing is silently overwritten.

+

+ CI + Python 3.12 + MIT license + Part of Conker +

+ +MemoryGate is a self-hosted memory service for one personal agent. It receives evidence, keeps its +lineage, turns lasting signals into structured memory, and returns a small context package the +agent can use without touching a database. + +It stores, retrieves and explains. It is not a chatbot and it never acts: the agent stays +responsible for reasoning and action. Part of [Conker](https://github.com/Conker-AI/conker), and +usable on its own. + +## Where it fits + +```mermaid +flowchart LR + Agent[Agent
e.g. Conker's Pi] -->|evidence| MG[MemoryGate] + Agent -->|read key: what's relevant?| MG + MG --> PG[(PostgreSQL
source of truth)] + MG --> QD[(Qdrant
vector index)] + MG -->|text to vectors| EM[Embeddings] + classDef focus fill:#e36b2c,color:#fff,stroke:#b4521f + class MG focus ``` -### Storage and Search - -- **PostgreSQL** is the source of truth for all memory, evidence, history, audit, and configuration records. -- **Qdrant** is the semantic vector index used to retrieve meaningfully related memories, entities, and observations. -- **Lexical matching** supplements vector results so exact names and project terms are not hidden by similarity ranking. -- **Embeddings are not shipped yet.** No embedding provider is installed in the API image, so semantic - retrieval is unavailable and MemoryGate says so instead of guessing. - -### Semantic retrieval is currently degraded - -This is a known, reported state, not a silent one: - -- `GET /health` reports `degraded` and names `embeddings`. -- `POST /runtime/context` and `POST /memory/search` return a `retrieval` block giving the mode - (`hybrid` or `lexical`) and why semantic search is unavailable, and every result carries the - `retrieval_path` that produced it. -- `/runtime/context` also states the degradation inside `usage.instruction`, which is the text the - reading model actually sees. -- Writes still succeed - Postgres is the source of truth - and report `indexing: degraded` when the - row could not be added to the vector index, plus `novelty_check: degraded` when the near-duplicate - check could not run. - -An earlier `EMBED_MODEL=hash` mode has been **removed**. It derived each vector component from -`sha256(index:text)`, so near-identical sentences produced uncorrelated vectors and cosine similarity -over them was noise. It made retrieval look like it worked while returning confident nonsense. +## How memory is built -Changing the configured LLM does **not** change the vector database or embeddings. The LLM is used only for bounded evidence analysis and read-only answers. - -## Security Model +```mermaid +flowchart LR + E[Evidence
immutable input] --> J[Processing job] --> A[Analysis] --> M[Memory · Entity · Episode] + M -. lineage .-> E +``` -MemoryGate assumes the dashboard is an administrative surface and keeps agents on a separate read-only interface. +| Layer | What it holds | +|---|---| +| **Evidence** | Raw inputs from conversations, listeners, APIs or manual capture. Never edited. | +| **Analysis** | A recorded interpretation of one or more pieces of evidence. | +| **Memory** | Durable facts, phases, context and watch items, ready for retrieval. | +| **Entity** | People, projects, places, concepts, habits and objects. | +| **Episode** | A time-bounded event grouping related evidence. | -- **MemoryGate refuses to start with no admin key configured.** There is no open fallback tier; the - startup error names the exact fix. A key supplied through `MEMORYGATE_ADMIN_KEY` must be at least - 16 characters. -- **CORS defaults to the bundled dashboard's own origins** (`http://localhost:8021`, - `http://127.0.0.1:8021`). `MEMORYGATE_CORS_ORIGINS=*` is a development override only - a wildcard - puts every route in reach of any page the owner has open, and it is logged as a warning at startup. -- Destructive actions need a second, deliberate confirmation on top of admin auth: `POST - /system/memory-reset` requires the exact phrase `RESET MEMORY`. A valid admin key alone is not - enough. -- Admin keys are stored as PBKDF2-SHA256 hashes, never plaintext. -- Failed key verification is limited to five attempts with a five-minute lockout per client scope. -- Agent read keys are separate, scoped credentials. They can retrieve context but cannot ingest, edit, reset, or administer MemoryGate. -- Listener ingestion uses a source-specific secret, not the admin key. -- LLMs receive no write, delete, shell, or tool capability through MemoryGate. -- OpenAI API keys, when configured, are encrypted at rest in the MemoryGate server volume and never returned to the dashboard after saving. -- Backups exclude admin/read-key hashes and listener secrets. +Every object can be opened in the dashboard with its history, links and supporting evidence. -Local deployment protects against remote misuse, not a fully compromised host. Running the agent and MemoryGate services on separate machines is the recommended next isolation step. +## Retrieval, and saying when it's degraded -## Quick Start +PostgreSQL is the source of truth. Qdrant indexes meaning, using vectors from the +[Embeddings](https://github.com/Conker-AI/embeddings) service; word matching runs alongside so exact +names are never hidden by similarity ranking. -### Prerequisites +If Embeddings is unavailable, search falls back to word matching and says so. `/health` reports +`degraded` and names `embeddings`, and every result carries the `retrieval_path` that produced it. +Writes still succeed, and report when they could not be indexed. -- Docker Desktop with Compose -- Node.js 22+ only when running the dashboard outside Docker +![MemoryGate command center](docs/screenshots/overview.png) -### Start the services +## Quick start -`docker-compose.yml` joins an external Docker network and reads an optional `.env`. Both are -prerequisites of a clean checkout: +Requires Docker with Compose. ```bash -docker network create conker_net # once; compose declares it external +docker network create conker_net # once; shared with the other Conker services cp .env.example .env echo "MEMORYGATE_ADMIN_KEY=$(openssl rand -base64 24)" >> .env docker compose up -d --build ``` -Without an admin key the API exits at startup with an error naming this exact fix. That is -deliberate: a service with no key configured must not fall back to open. - -Start the optional local Ollama runtime only when you explicitly want it: - -```bash -docker compose --profile local-ai up -d ollama -``` - -The default services are: - -| Service | Address | -| --- | --- | +| | | +|---|---| | Dashboard | `http://localhost:8021` | | API | `http://localhost:8020` | -| PostgreSQL | `localhost:5434` | -| Qdrant | `localhost:6335` | -| Ollama | `localhost:11434` when the `local-ai` profile is enabled | - -The dashboard can also be started with `npm run build` from `dashboard/`. Development runtime addresses may differ from the Compose defaults. - -### Configure access - -1. Open **Settings** in the dashboard. -2. Change the initial admin key to a long unique value. -3. Create one read key for your agent integration. -4. Add evidence sources and their listener secrets only through the dashboard. - -## AI Runtime - -MemoryGate supports two bounded model providers from **Settings -> AI Runtime**: - -- **Ollama** is the optional local provider. Start the `local-ai` profile first, then select any installed local model, such as `qwen3:4b`. -- **OpenAI API** accepts a model identifier and an OpenAI API key. The key is sent only from MemoryGate's API server to `api.openai.com`; it is never stored in browser storage or exposed to an agent. - -The selected model can: - -- Propose observations and memory candidates from evidence. -- Answer a read-only Memory Lab question from retrieved context. - -The selected model cannot: - -- Write, delete, reset, or call tools through MemoryGate. -- Receive a hidden Memory Lab conversation history. -- Replace semantic retrieval or directly access PostgreSQL/Qdrant. - -OpenAI uses the server-side Responses API with a Bearer API key. Keep the key private and treat provider usage as paid external processing. See the [OpenAI API quickstart](https://platform.openai.com/docs/quickstart/make-your-first-api-request) and [model catalog](https://developers.openai.com/api/docs/models). - -## Dashboard - -The dashboard is a single-workspace operating console for one agent: - -- **Command Center**: system counts, signal health, recent activity, and promotion metrics. -- **Live Pipeline**: a timestamped trace of incoming data through evidence, analysis, knowledge, and write decisions. -- **Memories**: inspect, create, edit, and connect durable fact, phase, context, and watch records. -- **Entities**: browse people, projects, concepts, and linked evidence with graph navigation. -- **Memory Lab**: saved browser-session investigations. Each question is independent and read-only; inspect the exact objects supplied to the model. -- **Database**: search and inspect every object type in one table. -- **Sources & Evidence**: manage listener credentials, inspect immutable evidence, and verify ingest endpoints. -- **Episodes / Sessions / Observations / Derived Patterns**: focused views for time-bounded events, transcript archives, extracted signals, and promoted patterns. -- **Architecture**: developer-facing object, lineage, truth, and search model. -- **Settings**: keys, backups, AI runtime, and destructive operations. - -## Screenshots - -### Command Center - -![MemoryGate command center](docs/screenshots/overview.png) - -### Memories -![MemoryGate memories](docs/screenshots/memories.png) +Without an admin key the API refuses to start and names the fix. For meaning-based search, run +[Embeddings](https://github.com/Conker-AI/embeddings) on the same network and set `EMBEDDINGS_KEY`. +Then, in **Settings**, change the admin key and create one read key for your agent. -### Database Inspection +## Using it from an agent -Search every durable object from one table, then open an object to inspect its -metadata, history, and connected records. +Give the agent a **read key**, never the admin key. -![MemoryGate database inspection](docs/screenshots/database.png) - -### Entities Graph - -![MemoryGate entities graph](docs/screenshots/entities-graph.png) - -### Observations - -![MemoryGate observations](docs/screenshots/observations.png) - -### Derived Patterns - -![MemoryGate derived patterns](docs/screenshots/patterns.png) - -### Briefing - -![MemoryGate briefing](docs/screenshots/briefing.png) - -### Beliefs - -![MemoryGate beliefs](docs/screenshots/beliefs.png) - -### Memory Lab - -Memory Lab answers independent, read-only questions. It keeps an investigation -list only in the current browser session and exposes the exact retrieved -objects used for each answer. - -![MemoryGate Memory Lab](docs/screenshots/memory-lab.png) - -### Operations and Safety - -Settings keeps access controls, backups, model configuration, and destructive -reset controls together. Reset actions require the current admin key and an -explicit confirmation phrase. - -![MemoryGate settings and danger zone](docs/screenshots/settings.png) - -### Transcript Detail - -![MemoryGate transcript detail](docs/screenshots/transcript-detail.png) - -## Agent Integration - -Give external agents a **read key**, not the admin key. They should retrieve context before answering or acting, then send raw events through a listener or approved ingest path. - -```powershell +```bash python services/cli/memorygate.py context "What should I remember about this project?" ``` -An MCP configuration and a read-only agent skill are included under `integrations/`. These integrations are intentionally read-focused: external agents should not be able to rewrite the memory architecture by prompt injection. +A read-only MCP configuration and agent skill live in `integrations/`. Raw events go in through a +listener with its own secret: `POST /runtime/listeners/{source_key}`. -## Evidence Ingestion +## Security model, briefly -Create a source in **Sources & Evidence**, then send an event to its dedicated listener endpoint: +- No admin key, no start. There is no open fallback. +- Read keys can retrieve context and nothing else. Listener secrets can only ingest. +- Keys are stored as PBKDF2 hashes; failed attempts are rate-limited. +- Destructive resets need the admin key *and* the phrase `RESET MEMORY`, and take a backup first. +- Models used by MemoryGate get no write, delete, shell or tool ability. -```text -POST /runtime/listeners/{source_key} -X-MemoryGate-Listener-Key: -``` - -An event becomes an immutable evidence object. With automatic processing enabled, it receives a processing job that may produce analysis, observations, and durable memory candidates. All resulting objects retain lineage rather than silently replacing their source. - -## Backups and Reset - -Settings provides logical JSON backups and a **Danger Zone**. - -- **Create backup** exports memory data, lineage, and processing state to the persistent backup volume. -- **Reset all memory** removes all stored memory, evidence, entities, transcripts, analysis, episodes, processing records, and matching vector points. -- **Reset data from a date** removes records created on or after the selected date. - -Every reset requires the current admin key and the exact phrase `RESET MEMORY`. A backup is created before any destructive change. Admin access, agent read keys, listener configuration, backups, and AI configuration are preserved. +Keep the dashboard and API on a private network. Full model: [security](docs/security.md). ## Development -### API - -```powershell -cd services/api -docker build -t memorygate-api:local . -``` - -### Dashboard - -```powershell -cd dashboard -npm ci -npm run build -``` - -### Tests - ```bash cd services/api -python -m venv .venv && . .venv/bin/activate pip install -r requirements.txt -r requirements-dev.txt python -m pytest tests ``` -On Windows, drop `uvloop` from the install: it has no Windows build. The suite needs no running -services - it uses a real SQLite database and points its dependency probes at a closed port so the -degraded paths are exercised for real rather than mocked. - -### Verification - -```powershell -docker ps -Invoke-WebRequest -UseBasicParsing http://127.0.0.1:8020/health -``` - -`GET /health` is unauthenticated and runs real probes against PostgreSQL, Qdrant, and the embedding -provider. It reports `ok` only when all three answer, and otherwise `degraded` with each failing -dependency named. Probe detail is deliberately coarse, since the route has no auth. Results are -cached for five seconds and carry their `age_seconds`. - -## Project Layout - -```text -dashboard/ React administrative console -services/api/ FastAPI service, models, retrieval, workers, and security -services/cli/ Terminal client for agent integrations -services/mcp/ MCP server bridge -integrations/ Read-only agent skill and MCP configuration -``` - -## Operational Notes - -- Keep all secrets out of Git. Use the Settings UI or server environment configuration. -- Do not expose the dashboard/API directly to the public internet. Put them behind a private network, VPN, or authenticated reverse proxy when leaving localhost. -- Backups are logical exports, not an encrypted disaster-recovery system. Protect the Docker volume and copy important backups to secure storage. -- MemoryGate can preserve evidence and history, but no automated system can guarantee a fact is true. Confidence, provenance, and review remain part of the design. +The suite needs no running services: it uses a real SQLite database and aims its health probes at a +closed port, so the degraded paths run for real. More in [development](docs/development.md). -Direct OpenAI generation is refused until it has a durable shared-budget adapter. -Refusals appear in the owner audit as `hosted_generation_refused`, without prompt text. -Optional `MEMORYGATE_HOSTED_COST_QUOTE` JSON records an owner-supplied estimate: -`model`, `valid_until` (Unix seconds), HTTPS `source`, `input_token_ceiling`, -`input_per_million_microusd`, `output_per_million_microusd`. Without a current quote, -cost is explicitly unknown. An estimate never enables spending; choose local Ollama. +## Documentation -Qdrant health is degraded when any existing collection cannot be inspected or has an unknown vector dimension, even if collection listing succeeded. +| | | +|---|---| +| [Security model](docs/security.md) | Keys, CORS, destructive actions, bootstrap | +| [Operations](docs/operations.md) | Ingestion, backups, resets, limits | +| [Dashboard](docs/dashboard.md) | Every screen, with screenshots | +| [AI runtime](docs/ai-runtime.md) | Which models MemoryGate may use, and for what | +| [Agent integration](docs/AGENT_INTEGRATION.md) | Connecting an agent | +| [Conversation memory](docs/conversation-memory.md) | Admission, retries and forgetting for Pi | +| [Development](docs/development.md) | Building, testing, layout | +| [API (OpenAPI)](docs/openapi.json) | Full route reference | -Conversation ingestion above 16,000 characters returns HTTP 413 with `detail.code=CONTENT_TOO_LARGE`, `retryable=false`, and `max_content_characters=16000`. Preserve the original transcript; retries of the same oversized payload cannot succeed. +## License -`cryptography` is pinned to 50.0.0: 48.0.1 fixes the bundled OpenSSL advisory, -but [the PKCS#7 advisory](https://github.com/pyca/cryptography/security/advisories/GHSA-g6cj-pr64-35w5) -requires 50.0.0. MemoryGate uses Fernet, so this is maintenance of a security dependency, -not a claim of a demonstrated vault exploit. Tests include the longstanding -[Fernet verification vector](https://github.com/fernet/spec/blob/master/verify.json). +[MIT](LICENSE) diff --git a/docs/ai-runtime.md b/docs/ai-runtime.md new file mode 100644 index 0000000..00e67ab --- /dev/null +++ b/docs/ai-runtime.md @@ -0,0 +1,23 @@ +# AI runtime + +The bounded model MemoryGate may use to analyse evidence and answer read-only questions. + +## AI Runtime + +MemoryGate supports two bounded model providers from **Settings -> AI Runtime**: + +- **Ollama** is the optional local provider. Start the `local-ai` profile first, then select any installed local model, such as `qwen3:4b`. +- **OpenAI API** accepts a model identifier and an OpenAI API key. The key is sent only from MemoryGate's API server to `api.openai.com`; it is never stored in browser storage or exposed to an agent. + +The selected model can: + +- Propose observations and memory candidates from evidence. +- Answer a read-only Memory Lab question from retrieved context. + +The selected model cannot: + +- Write, delete, reset, or call tools through MemoryGate. +- Receive a hidden Memory Lab conversation history. +- Replace semantic retrieval or directly access PostgreSQL/Qdrant. + +OpenAI uses the server-side Responses API with a Bearer API key. Keep the key private and treat provider usage as paid external processing. See the [OpenAI API quickstart](https://platform.openai.com/docs/quickstart/make-your-first-api-request) and [model catalog](https://developers.openai.com/api/docs/models). diff --git a/docs/dashboard.md b/docs/dashboard.md new file mode 100644 index 0000000..931a1da --- /dev/null +++ b/docs/dashboard.md @@ -0,0 +1,75 @@ +# The MemoryGate dashboard + +A single-workspace console for inspecting everything MemoryGate holds and why. + +## Dashboard + +The dashboard is a single-workspace operating console for one agent: + +- **Command Center**: system counts, signal health, recent activity, and promotion metrics. +- **Live Pipeline**: a timestamped trace of incoming data through evidence, analysis, knowledge, and write decisions. +- **Memories**: inspect, create, edit, and connect durable fact, phase, context, and watch records. +- **Entities**: browse people, projects, concepts, and linked evidence with graph navigation. +- **Memory Lab**: saved browser-session investigations. Each question is independent and read-only; inspect the exact objects supplied to the model. +- **Database**: search and inspect every object type in one table. +- **Sources & Evidence**: manage listener credentials, inspect immutable evidence, and verify ingest endpoints. +- **Episodes / Sessions / Observations / Derived Patterns**: focused views for time-bounded events, transcript archives, extracted signals, and promoted patterns. +- **Architecture**: developer-facing object, lineage, truth, and search model. +- **Settings**: keys, backups, AI runtime, and destructive operations. + +## Screenshots + +### Command Center + +![MemoryGate command center](screenshots/overview.png) + +### Memories + +![MemoryGate memories](screenshots/memories.png) + +### Database Inspection + +Search every durable object from one table, then open an object to inspect its +metadata, history, and connected records. + +![MemoryGate database inspection](screenshots/database.png) + +### Entities Graph + +![MemoryGate entities graph](screenshots/entities-graph.png) + +### Observations + +![MemoryGate observations](screenshots/observations.png) + +### Derived Patterns + +![MemoryGate derived patterns](screenshots/patterns.png) + +### Briefing + +![MemoryGate briefing](screenshots/briefing.png) + +### Beliefs + +![MemoryGate beliefs](screenshots/beliefs.png) + +### Memory Lab + +Memory Lab answers independent, read-only questions. It keeps an investigation +list only in the current browser session and exposes the exact retrieved +objects used for each answer. + +![MemoryGate Memory Lab](screenshots/memory-lab.png) + +### Operations and Safety + +Settings keeps access controls, backups, model configuration, and destructive +reset controls together. Reset actions require the current admin key and an +explicit confirmation phrase. + +![MemoryGate settings and danger zone](screenshots/settings.png) + +### Transcript Detail + +![MemoryGate transcript detail](screenshots/transcript-detail.png) diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 0000000..a24d584 --- /dev/null +++ b/docs/development.md @@ -0,0 +1,53 @@ +# Developing MemoryGate + +## Development + +### API + +```powershell +cd services/api +docker build -t memorygate-api:local . +``` + +### Dashboard + +```powershell +cd dashboard +npm ci +npm run build +``` + +### Tests + +```bash +cd services/api +python -m venv .venv && . .venv/bin/activate +pip install -r requirements.txt -r requirements-dev.txt +python -m pytest tests +``` + +On Windows, drop `uvloop` from the install: it has no Windows build. The suite needs no running +services - it uses a real SQLite database and points its dependency probes at a closed port so the +degraded paths are exercised for real rather than mocked. + +### Verification + +```powershell +docker ps +Invoke-WebRequest -UseBasicParsing http://127.0.0.1:8020/health +``` + +`GET /health` is unauthenticated and runs real probes against PostgreSQL, Qdrant, and the embedding +provider. It reports `ok` only when all three answer, and otherwise `degraded` with each failing +dependency named. Probe detail is deliberately coarse, since the route has no auth. Results are +cached for five seconds and carry their `age_seconds`. + +## Project Layout + +```text +dashboard/ React administrative console +services/api/ FastAPI service, models, retrieval, workers, and security +services/cli/ Terminal client for agent integrations +services/mcp/ MCP server bridge +integrations/ Read-only agent skill and MCP configuration +``` diff --git a/docs/operations.md b/docs/operations.md new file mode 100644 index 0000000..acef96d --- /dev/null +++ b/docs/operations.md @@ -0,0 +1,48 @@ +# Operating MemoryGate + +Ingesting evidence, backups, resets and the operational limits worth knowing. + +## Evidence Ingestion + +Create a source in **Sources & Evidence**, then send an event to its dedicated listener endpoint: + +```text +POST /runtime/listeners/{source_key} +X-MemoryGate-Listener-Key: +``` + +An event becomes an immutable evidence object. With automatic processing enabled, it receives a processing job that may produce analysis, observations, and durable memory candidates. All resulting objects retain lineage rather than silently replacing their source. + +## Backups and Reset + +Settings provides logical JSON backups and a **Danger Zone**. + +- **Create backup** exports memory data, lineage, and processing state to the persistent backup volume. +- **Reset all memory** removes all stored memory, evidence, entities, transcripts, analysis, episodes, processing records, and matching vector points. +- **Reset data from a date** removes records created on or after the selected date. + +Every reset requires the current admin key and the exact phrase `RESET MEMORY`. A backup is created before any destructive change. Admin access, agent read keys, listener configuration, backups, and AI configuration are preserved. + +## Operational Notes + +- Keep all secrets out of Git. Use the Settings UI or server environment configuration. +- Do not expose the dashboard/API directly to the public internet. Put them behind a private network, VPN, or authenticated reverse proxy when leaving localhost. +- Backups are logical exports, not an encrypted disaster-recovery system. Protect the Docker volume and copy important backups to secure storage. +- MemoryGate can preserve evidence and history, but no automated system can guarantee a fact is true. Confidence, provenance, and review remain part of the design. + +Direct OpenAI generation is refused until it has a durable shared-budget adapter. +Refusals appear in the owner audit as `hosted_generation_refused`, without prompt text. +Optional `MEMORYGATE_HOSTED_COST_QUOTE` JSON records an owner-supplied estimate: +`model`, `valid_until` (Unix seconds), HTTPS `source`, `input_token_ceiling`, +`input_per_million_microusd`, `output_per_million_microusd`. Without a current quote, +cost is explicitly unknown. An estimate never enables spending; choose local Ollama. + +Qdrant health is degraded when any existing collection cannot be inspected or has an unknown vector dimension, even if collection listing succeeded. + +Conversation ingestion above 16,000 characters returns HTTP 413 with `detail.code=CONTENT_TOO_LARGE`, `retryable=false`, and `max_content_characters=16000`. Preserve the original transcript; retries of the same oversized payload cannot succeed. + +`cryptography` is pinned to 50.0.0: 48.0.1 fixes the bundled OpenSSL advisory, +but [the PKCS#7 advisory](https://github.com/pyca/cryptography/security/advisories/GHSA-g6cj-pr64-35w5) +requires 50.0.0. MemoryGate uses Fernet, so this is maintenance of a security dependency, +not a claim of a demonstrated vault exploit. Tests include the longstanding +[Fernet verification vector](https://github.com/fernet/spec/blob/master/verify.json). diff --git a/docs/security.md b/docs/security.md new file mode 100644 index 0000000..87f6cb1 --- /dev/null +++ b/docs/security.md @@ -0,0 +1,34 @@ +# MemoryGate security model + +MemoryGate treats its dashboard as an administrative surface and keeps agents on a separate, read-only interface. + +## Security Model + +MemoryGate assumes the dashboard is an administrative surface and keeps agents on a separate read-only interface. + +- **MemoryGate refuses to start with no admin key configured.** There is no open fallback tier; the + startup error names the exact fix. A key supplied through `MEMORYGATE_ADMIN_KEY` must be at least + 16 characters. +- **CORS defaults to the bundled dashboard's own origins** (`http://localhost:8021`, + `http://127.0.0.1:8021`). `MEMORYGATE_CORS_ORIGINS=*` is a development override only - a wildcard + puts every route in reach of any page the owner has open, and it is logged as a warning at startup. +- Destructive actions need a second, deliberate confirmation on top of admin auth: `POST + /system/memory-reset` requires the exact phrase `RESET MEMORY`. A valid admin key alone is not + enough. +- Admin keys are stored as PBKDF2-SHA256 hashes, never plaintext. +- Failed key verification is limited to five attempts with a five-minute lockout per client scope. +- Agent read keys are separate, scoped credentials. They can retrieve context but cannot ingest, edit, reset, or administer MemoryGate. +- Listener ingestion uses a source-specific secret, not the admin key. +- LLMs receive no write, delete, shell, or tool capability through MemoryGate. +- OpenAI API keys, when configured, are encrypted at rest in the MemoryGate server volume and never returned to the dashboard after saving. +- Backups exclude admin/read-key hashes and listener secrets. + +Local deployment protects against remote misuse, not a fully compromised host. Running the agent and MemoryGate services on separate machines is the recommended next isolation step. + +## Bootstrap keys are setup, not rotation + +Bootstrap read-key configuration is initial setup, not key rotation. Once its +label or credential exists, startup preserves the owner's revocation, agent +assignment and stored hash. Use the owner key-management API to issue replacement +credentials; changing bootstrap environment variables never restores authority. +Retain revoked key rows: they record the decision that restart must respect.