From 4d8f80b20a6fa6a9d25550b1e8be7e5c82930470 Mon Sep 17 00:00:00 2001 From: Shudipto Trafder Date: Sat, 31 Oct 2026 13:22:34 +0600 Subject: [PATCH 1/3] feat: Update documentation and code for 1.0.0rc4 release - Updated the versioning information in various documentation files to reflect the new release (1.0.0rc4). - Clarified backend unreachable error messages in the API server documentation. - Enhanced WebSocket documentation to specify cache store behavior. - Added a new document on bringing your own backend, detailing the custom checkpointer, store, and cache store implementations. - Introduced breaking changes regarding the new BaseCacheStore and its implications for existing configurations. - Updated project statistics to reflect the latest release counts and dates. - Fixed various references to Redis and cache store in troubleshooting documentation. - Added a new release note for the 1.0.0rc4 version, highlighting breaking changes and new features. --- .../blog/ai-agents-fall-2026-what-changed.mdx | 2 +- ...hat-production-ready-means-in-10xgraph.mdx | 4 +- ...-user-cannot-read-another-users-thread.mdx | 4 +- .../build/multi-tenant-support-agent.mdx | 11 +- src/content/build/production-react-agent.mdx | 12 +- src/content/build/ship-to-kubernetes.mdx | 10 +- src/content/docs/client/error-handling.md | 2 +- src/content/docs/concepts/extensibility.md | 61 ++--- src/content/docs/concepts/serving-agents.md | 73 +----- src/content/docs/examples/rag-assistant.mdx | 15 +- src/content/docs/examples/serve-with-auth.md | 10 +- src/content/docs/get-started/installation.md | 6 +- .../docs/get-started/project-structure.mdx | 7 +- src/content/docs/get-started/why-10xgraph.mdx | 7 +- src/content/docs/guides/graceful-shutdown.md | 2 +- .../docs/guides/set-up-checkpointing.md | 4 + src/content/docs/production/architecture.md | 6 +- src/content/docs/production/index.md | 4 +- src/content/docs/production/limits.md | 6 + src/content/docs/production/scaling.md | 27 +- src/content/docs/production/security-model.md | 2 +- src/content/docs/project/index.md | 2 +- src/content/docs/project/upgrade-to-1.0.md | 71 +++++- src/content/docs/reference/api-cli/auth.md | 28 +- .../docs/reference/api-cli/commands.mdx | 8 +- .../docs/reference/api-cli/configuration.mdx | 20 +- .../docs/reference/api-cli/environment.md | 14 +- .../docs/reference/api-cli/rate-limiting.md | 97 ++----- src/content/docs/reference/client/errors.md | 4 +- src/content/docs/reference/error-codes.md | 2 +- .../docs/reference/python/checkpointers.md | 8 + .../docs/reference/python/memory-stores.md | 4 +- .../docs/reference/rest-api/conventions.md | 2 +- src/content/docs/reference/rest-api/live.md | 2 +- src/content/docs/reference/rest-api/ping.md | 20 +- src/content/docs/server/ag-ui.md | 10 +- src/content/docs/server/auth.mdx | 4 +- .../docs/server/bring-your-own-backend.md | 240 ++++++++++++++++++ src/content/docs/server/cli.md | 4 +- src/content/docs/server/configure.md | 48 ++-- .../docs/server/cors-and-security-headers.md | 2 +- src/content/docs/server/deploy.mdx | 4 +- src/content/docs/server/index.md | 2 +- src/content/docs/server/kubernetes.md | 4 +- src/content/docs/server/playground.md | 2 +- .../docs/server/production-checklist.md | 35 ++- src/content/docs/server/project-setup.md | 3 +- src/content/docs/server/rate-limiting.md | 119 +++------ src/content/docs/server/run-the-server.md | 4 +- src/content/docs/server/websockets.md | 6 +- .../docs/troubleshooting/api-server.md | 19 +- src/content/docs/troubleshooting/client.md | 4 +- .../docs/troubleshooting/playground.md | 2 +- src/content/releases/api-1.0.0rc4.md | 42 +++ src/content/releases/core-1.0.0rc4.md | 23 ++ src/data/project-stats.json | 12 +- src/lib/site.ts | 2 +- src/pages/index.astro | 8 +- 58 files changed, 692 insertions(+), 464 deletions(-) create mode 100644 src/content/docs/server/bring-your-own-backend.md create mode 100644 src/content/releases/api-1.0.0rc4.md create mode 100644 src/content/releases/core-1.0.0rc4.md diff --git a/src/content/blog/ai-agents-fall-2026-what-changed.mdx b/src/content/blog/ai-agents-fall-2026-what-changed.mdx index 8db4bb9..5fd1170 100644 --- a/src/content/blog/ai-agents-fall-2026-what-changed.mdx +++ b/src/content/blog/ai-agents-fall-2026-what-changed.mdx @@ -651,9 +651,9 @@ If EU users talk to your support agent, the disclosure is a product requirement "env": ".env", "auth": "jwt", "authorization": "ownership", + "redis": "${REDIS_URL}", "rate_limit": { "enabled": true, - "backend": "redis", "requests": 100, "window": 60, "by": "user" diff --git a/src/content/blog/what-production-ready-means-in-10xgraph.mdx b/src/content/blog/what-production-ready-means-in-10xgraph.mdx index 0fe1c3f..ec92756 100644 --- a/src/content/blog/what-production-ready-means-in-10xgraph.mdx +++ b/src/content/blog/what-production-ready-means-in-10xgraph.mdx @@ -64,7 +64,7 @@ The short version: you build the agent, and the server owns the request surface, - A thread runs one request at a time. A second request waits up to 30 seconds, then gets 409 "thread is busy". `on_busy: "cancel"` stops the running one instead. - A model turn runs at most 10 tool calls at once (`runs.max_parallel_tool_calls`). -**Config:** `rate_limit` (by `ip`, `user` or `global`, memory or Redis backend) is off until you add it to `10xgraph.json`. `websocket.max_connections` and `websocket.max_connections_per_user` cap sockets. +**Config:** `rate_limit` (by `ip`, `user` or `global`) is off until you add it to `10xgraph.json`. Its counters live in the API's cache store, in memory or in Redis (`redis` or `REDIS_URL`). `websocket.max_connections` and `websocket.max_connections_per_user` cap sockets. **You still must:** add rate limiting, and set `trusted_proxies` to the real number of proxy hops. Streaming runs have no per-node or per-tool deadlines, only the run timeout. @@ -96,7 +96,7 @@ The short version: you build the agent, and the server owns the request surface, **Default:** `10xgraph build` writes a Dockerfile that runs Gunicorn as a non-root user, and with flags a `docker-compose.yml` and `k8s.yaml`. `GET /ping` is liveness and `GET /health/ready` is readiness. Workers hold no conversation state; it lives in the checkpointer. With Redis configured, the ownership cache, run caps and thread locks are shared across workers. -**Config:** `10xgraph build --docker-compose --k8s`, `redis` or `REDIS_URL`, `rate_limit.backend: "redis"`. +**Config:** `10xgraph build --docker-compose --k8s`, `redis` or `REDIS_URL` (or `cache_store` for your own `BaseCacheStore`), `token_revocation: true`. **You still must:** terminate TLS at your load balancer or ingress, run two or more instances if you need availability, and size your database. diff --git a/src/content/blog/why-a-user-cannot-read-another-users-thread.mdx b/src/content/blog/why-a-user-cannot-read-another-users-thread.mdx index 6c70997..96ac100 100644 --- a/src/content/blog/why-a-user-cannot-read-another-users-thread.mdx +++ b/src/content/blog/why-a-user-cannot-read-another-users-thread.mdx @@ -20,7 +20,7 @@ If you run a multi-tenant product on an agent framework, the question a security ## Layer 1: identity comes from the token, nowhere else -Authentication runs before anything else. With `"auth": "jwt"`, `JwtAuth` requires `exp` on every token and a `user_id` that is a non-empty string or a whole number; anything else is a 401. When `JWT_ISSUER` or `JWT_AUDIENCE` is set, `iss` and `aud` are required and checked as well. Scope claims must have the right type, or the request gets a 401. Revoked tokens are rejected when you configure `token_revocation`, and a backend failure there fails closed with a 503. +Authentication runs before anything else. With `"auth": "jwt"`, `JwtAuth` requires `exp` on every token and a `user_id` that is a non-empty string or a whole number; anything else is a 401. When `JWT_ISSUER` or `JWT_AUDIENCE` is set, `iss` and `aud` are required and checked as well. Scope claims must have the right type, or the request gets a 401. Revoked tokens are rejected when you configure `token_revocation`, and a cache store failure there fails closed with a 503. The verified `user_id` is the only identity the server uses. A `user`, `user_id` or `authz` key in the request config is dropped, and state or message content is never read for identity. Your tools get the verified identity through dependency injection. @@ -57,7 +57,7 @@ If you write your own `AuthorizationBackend`, `isolation_scope()` defaults to `" ## Layer 6: ownership is cached, and deletes are handled -Resolving the owner on every request would cost a database call each time. Ownership never changes, so the server caches it: an in-process LRU (L1, 30 seconds by default) and, when Redis is configured, a shared tier (L2, one hour by default). When a thread is deleted, the cache entry is evicted and a tombstone is left for one L1 TTL, in L1 and in Redis. Without it, another worker that still cached the old owner could let someone else recreate the deleted id and inherit a stale record. The tombstone window is why multi-worker deployments should configure Redis: with only the in-process tier the tombstone exists on the worker that handled the delete. +Resolving the owner on every request would cost a database call each time. Ownership never changes, so the server caches it: an in-process LRU (L1, 30 seconds by default) and, when a shared cache store is configured (Redis by default), a shared tier (L2, one hour by default). When a thread is deleted, the cache entry is evicted and a tombstone is left for one L1 TTL, in L1 and in the cache store. Without it, another worker that still cached the old owner could let someone else recreate the deleted id and inherit a stale record. The tombstone window is why multi-worker deployments should configure a shared cache store: with only the in-process tier the tombstone exists on the worker that handled the delete. ## Check it yourself diff --git a/src/content/build/multi-tenant-support-agent.mdx b/src/content/build/multi-tenant-support-agent.mdx index f6c9ca0..82c08ec 100644 --- a/src/content/build/multi-tenant-support-agent.mdx +++ b/src/content/build/multi-tenant-support-agent.mdx @@ -43,7 +43,7 @@ pip install "10xgraph[google-genai,pg_checkpoint,qdrant]" "10xgraph-api[jwt,redi 10xgraph init --yes --template production --auth jwt --rate-limit redis ``` -The production template writes `10xgraph.json` with JWT auth, `"authorization": "ownership"` and a Redis rate limit. Two changes matter for this guide. Point the agent at an async factory (step 2), add a store (step 3), and rate-limit by user instead of by address, so one noisy customer cannot spend another customer's quota: +The production template writes `10xgraph.json` with JWT auth, `"authorization": "ownership"`, a Redis cache store and a rate limit. Two changes matter for this guide. Point the agent at an async factory (step 2), add a store (step 3), and rate-limit by user instead of by address, so one noisy customer cannot spend another customer's quota: ```json title="10xgraph.json" { @@ -51,13 +51,12 @@ The production template writes `10xgraph.json` with JWT auth, `"authorization": "auth": "jwt", "authorization": "ownership", "store": "graph.memory:store", // [!code ++] + "redis": "${REDIS_URL}", "rate_limit": { "enabled": true, - "backend": "redis", "requests": 100, "window": 60, - "by": "user", // [!code highlight] - "redis": { "url": "${REDIS_URL}" } + "by": "user" // [!code highlight] } } ``` @@ -186,7 +185,7 @@ store = QdrantStore(embedding=HashEmbedding(), path=":memory:") "agent": "agent:app", "auth": "jwt", "store": "agent:store", - "rate_limit": { "enabled": true, "backend": "memory", "requests": 8, "window": 60, "by": "user" } + "rate_limit": { "enabled": true, "requests": 8, "window": 60, "by": "user" } } ``` @@ -282,7 +281,7 @@ x-ratelimit-remaining: 0 "limit":8,"window_seconds":60,"retry_after_seconds":47}} ``` -Alice, in the same minute, still got 200, because the bucket is per user. In your real config use the Redis backend: the memory backend counts per process, so with several workers the real limit is the number of workers times the limit. The server prints a warning at startup to say so. +Alice, in the same minute, still got 200, because the bucket is per user. In your real config set `"redis": "${REDIS_URL}"`: without it the in-memory cache store counts per process, so with several workers the real limit is the number of workers times the limit. The server prints a warning at startup to say so. ## What this guide covers, and what you still own diff --git a/src/content/build/production-react-agent.mdx b/src/content/build/production-react-agent.mdx index 4624884..04b77b1 100644 --- a/src/content/build/production-react-agent.mdx +++ b/src/content/build/production-react-agent.mdx @@ -17,7 +17,7 @@ faq: - q: Can I use OpenAI or Claude instead of Gemini? a: Yes. Change the model and provider arguments of ReactAgent, for example provider="openai" or provider="anthropic", and install the matching extra (openai or anthropic) instead of google-genai. The tools, graph and server stay the same. - q: Do I need Redis running for local development? - a: REDIS_URL must be set, because graph/agent.py reads it, but the server starts without a reachable Redis. The rate limiter fails open by default, which means it logs the error and lets requests through, and without DATABASE_URL the in-memory checkpointer is used. + a: REDIS_URL must be set, because graph/agent.py reads it. The production template also turns on token_revocation, which keeps its entries in the Redis cache store, so with an unreachable Redis the server refuses to boot (set STARTUP_CHECKS=false to skip the check, or remove token_revocation while you develop). Without a reachable Redis and without that setting the server logs one warning and starts. Without DATABASE_URL the in-memory checkpointer is used. - q: Why do the tools raise exceptions instead of returning an error message? a: A raised exception stops the tool and marks the result as failed. The model gets a generic failure with a reference id, and the full message goes to the server log, so internal details never reach the model or the customer. A returned dictionary treats the keys error, status, is_error and success as result metadata and removes them, so an error returned that way never reaches the model either. To give the model a message it can pass on, return it as a plain string. --- @@ -32,7 +32,7 @@ flowchart LR A --> M[MAIN: the model] M -->|tool call| T[TOOL: lookup_order, refund_order] T --> M - A <--> R[(Redis: hot state, rate limits)] + A <--> R[(Redis: hot state, cache store)] A <--> P[(Postgres: every thread)] ``` @@ -47,7 +47,7 @@ pip install "10xgraph[google-genai,pg_checkpoint]" "10xgraph-api[jwt,redis]" 10xgraph init --yes --template production --auth jwt --rate-limit redis ``` -The template writes a working agent with JWT auth, thread ownership checks and a Redis rate limit already set in `10xgraph.json`: +The template writes a working agent with JWT auth, thread ownership checks and a Redis cache store (`"redis": "${REDIS_URL}"`) and a rate limit already set in `10xgraph.json`: @@ -195,7 +195,8 @@ load_dotenv() # 10xgraph.json points "injectq" here. Anything bound in it is shared with the server. container = InjectQ.get_instance() -# One Redis client for the rate limiter and the checkpointer's hot cache. +# One Redis client for the checkpointer's hot cache. The API opens its own +# connection for its cache store from "redis" in 10xgraph.json. redis = Redis.from_url(os.environ["REDIS_URL"]) container.bind_instance("redis", redis) @@ -234,7 +235,7 @@ async def build_app() -> CompiledGraph: `ReactAgent` builds the loop for you: a `MAIN` node that calls the model and a `TOOL` node that runs the tools it asks for, until the model answers without a tool call. The rest is production plumbing: - **`build_app` is async on purpose.** `PgCheckpointer` creates its tables only when `asetup()` is called. The server accepts an async factory as the agent entry point and awaits it at startup, so setup runs on the server's own event loop. -- **One Redis client, shared.** The rate limiter looks for a client bound as `"redis"` before creating its own, and `PgCheckpointer` takes the same client for its cache. +- **Two Redis uses, one server.** `PgCheckpointer` takes the Redis client you pass for its hot cache. The API opens its own connection for its cache store (rate limits, run locks, caps, token revocation) from `"redis": "${REDIS_URL}"` in `10xgraph.json`. - **Replays skip finished tools.** With `PgCheckpointer`, a tool call that completed before a crash is recorded, and the resumed run returns the recorded result instead of calling it again. See [Replay-safe tools](/docs/concepts/replay-safe-tools). - **Context stays bounded.** The context manager keeps the last 20 user messages in the prompt; the full history stays in the checkpointer. @@ -245,7 +246,6 @@ Point the server at the factory and rate-limit per user instead of per IP. In `1 "agent": "graph.agent:build_app", // [!code highlight] "rate_limit": { "enabled": true, - "backend": "redis", "requests": 100, "window": 60, "by": "user" // [!code highlight] diff --git a/src/content/build/ship-to-kubernetes.mdx b/src/content/build/ship-to-kubernetes.mdx index 98456fb..c5705a4 100644 --- a/src/content/build/ship-to-kubernetes.mdx +++ b/src/content/build/ship-to-kubernetes.mdx @@ -44,7 +44,7 @@ pip install "10xgraph[google-genai,pg_checkpoint]" "10xgraph-api[jwt,redis]" 10xgraph init --yes --template production --auth jwt --rate-limit redis --name shop-support ``` -That writes `10xgraph.json` (JWT auth, `ownership` authorization, a Redis rate limit at 100 requests per 60 seconds per IP, `${REDIS_URL}` read from the environment), a `graph/` package with a sample weather agent, tests and evals. Build the Docker image only after two edits. +That writes `10xgraph.json` (JWT auth, `ownership` authorization, a Redis cache store from `${REDIS_URL}`, token revocation on, and a rate limit at 100 requests per 60 seconds per IP, with the counters shared through that Redis), a `graph/` package with a sample weather agent, tests and evals. Build the Docker image only after two edits. First, the generated Dockerfile installs exactly the `dependencies` in `pyproject.toml`, so list the Postgres extra there: @@ -135,7 +135,7 @@ A Deployment and a Service named after `--service-name`. The values that matter: | `replicas` | 2 | Redundancy. The manifest has no autoscaler. | | `terminationGracePeriodSeconds` | 660 | Longer than the 600-second graceful timeout, so Kubernetes does not send SIGKILL while runs drain. | | `preStop` | `sleep 15` | Gives the load balancer time to stop sending new requests before the app starts shutting down. | -| `readinessProbe` | `GET /health/ready`, every 10 s, first check after 5 s | Takes a pod out of rotation when its checkpointer, Redis or store cannot be reached. | +| `readinessProbe` | `GET /health/ready`, every 10 s, first check after 5 s | Takes a pod out of rotation when its checkpointer or store cannot be reached, or when the cache store cannot be reached and token revocation or a fail-closed rate limit needs it. | | `livenessProbe` | `GET /ping`, every 30 s, first check after 30 s, 5 failures | Deliberately slack. A worker busy with a long run must not be mistaken for a hung one. | | Security | non-root uid 10001, `readOnlyRootFilesystem`, `allowPrivilegeEscalation: false`, `capabilities.drop: ALL`, `seccompProfile: RuntimeDefault`, `automountServiceAccountToken: false` | The server never calls the Kubernetes API and writes only to `/tmp` and `/app/uploads`, both `emptyDir`. | | Secrets | `envFrom: secretRef: shop-support-secrets` | Every key of that Secret becomes an environment variable. The pod does not start until it exists. | @@ -176,6 +176,8 @@ healthy The image builds, runs as uid 10001 on a read-only root, and starts with all capabilities dropped. `/ping` stays 200 while `/health/ready` is 503 because Redis cannot be reached. That split is what the manifest relies on. Readiness fails, so the pod is taken out of the Service and receives no new requests. Liveness still passes, so Kubernetes does not restart a pod that is merely waiting for a dependency. The unauthenticated request got 401, because every route except `/ping` and `/health/ready` needs a token. Stop and remove the container with `docker rm -f shop`. +This transcript comes from a 0.9.0 image. With 1.0.0rc4 the server also refuses to boot when the cache store is unreachable and token revocation is on (the production template turns it on), the readiness key is `cache` instead of `redis`, and `STARTUP_CHECKS=false` skips the boot check if you want to see the probes alone. + The full Compose file with Postgres and Redis passed `docker compose config`. I did not start it, and I did not run the app against a real Postgres. ## 4. Secrets and dependencies @@ -212,9 +214,9 @@ kubectl scale deployment shop-support --replicas=5 Three things change when you go past one process: -- **Use the Redis rate limiter.** The memory backend counts per process, so the real limit is the limit times the number of workers, and the server prints a warning at startup to say so. +- **Use a shared cache store.** Set `redis` (or `REDIS_URL`). The in-memory store counts per process, so the real rate limit is the limit times the number of workers, and token revocations reach only one process. The server warns about it at startup. - **Set load balancer idle timeouts above your longest streaming run.** The pod's grace period is 660 seconds, but a proxy that cuts idle connections at 60 seconds will still break a stream. -- **One run per thread.** A second request to a thread that is busy waits up to 30 seconds and then gets 409. With Redis configured, that lock is shared by all replicas. +- **One run per thread.** A second request to a thread that is busy waits up to 30 seconds and then gets 409. With a Redis cache store configured, that lock is shared by all replicas. For autoscaling, CPU is a poor signal for agents, which wait on a model provider. Scale on concurrent requests per pod where your ingress can export it. [Scaling and reliability](/docs/production/scaling) has the details. diff --git a/src/content/docs/client/error-handling.md b/src/content/docs/client/error-handling.md index dd400ee..4dfc971 100644 --- a/src/content/docs/client/error-handling.md +++ b/src/content/docs/client/error-handling.md @@ -167,7 +167,7 @@ const result = await retryWithBackoff(() => The server returns 429 when the `rate_limit` window is used up, or when one user has more graph runs in flight than `runs.max_concurrent_per_user` allows (default 10). The rate-limit response includes a `Retry-After` header in seconds and ends its message with `Retry after s.` The run-cap response has no `Retry-After` header. The client has no class for 429, so it throws a plain `TenxGraphError` with `statusCode` 429, and it does not keep the response headers. -If the server's rate-limit backend (Redis) is down and `fail_open` is `false`, it answers `503` with `RATE_LIMIT_UNAVAILABLE` and `Retry-After: 5` instead. Retry on 503 the same way you retry on 429. Added after 1.0.0rc2. +If the server's cache store (Redis) is down and `fail_open` is `false`, it answers `503` with `RATE_LIMIT_UNAVAILABLE` and `Retry-After: 5` instead. Retry on 503 the same way you retry on 429. Added in 1.0.0rc3. ```ts import { TenxGraphError } from '10xgraph-client'; diff --git a/src/content/docs/concepts/extensibility.md b/src/content/docs/concepts/extensibility.md index 55ef8ae..f5d0948 100644 --- a/src/content/docs/concepts/extensibility.md +++ b/src/content/docs/concepts/extensibility.md @@ -12,7 +12,7 @@ faq: - q: "Can I extend multiple base classes in the same graph?" a: "Yes. You can extend a checkpointer for storage, a publisher for events, and an authorization backend all in the same graph. They compose independently." - q: "Which base classes go in compile() vs 10xgraph.json?" - a: "Checkpointer, store and media_store go in compile(). Context manager, publisher and ID generator go in the StateGraph constructor. Auth, authorization and thread_name_generator are set in 10xgraph.json; a custom rate-limit backend is bound in the InjectQ container." + a: "Checkpointer, store and media_store go in compile(). Context manager, publisher and ID generator go in the StateGraph constructor. Auth, authorization, thread_name_generator and cache_store are set in 10xgraph.json." --- 10xGraph's major components are built on abstract base classes. Extend any of them to swap storage backends, auth mechanisms, LLM providers, ID schemes, rate limiting, or event routing. Your graph logic stays unchanged. This page explains the pattern and shows when and how to extend each. @@ -37,7 +37,7 @@ flowchart TB subgraph "API / Server" BAUTH[BaseAuth] BAUTHZ[AuthorizationBackend] - BRL[BaseRateLimitBackend] + BCS[BaseCacheStore] TNG[ThreadNameGenerator] BID[BaseIDGenerator] end @@ -49,7 +49,7 @@ flowchart TB end YOUR[Your Subclass] -->|extend any| BCP & BS & BE & BMS YOUR -->|extend any| BA & BC & BCM - YOUR -->|extend any| BAUTH & BAUTHZ & BRL & TNG & BID + YOUR -->|extend any| BAUTH & BAUTHZ & BCS & TNG & BID YOUR -->|extend any| BP & BV & BCR & BR ``` @@ -57,7 +57,7 @@ flowchart TB |---|---|---| | `BaseAgent` | `tenxgraph/core/graph/base_agent.py` | `execute()`, `_call_llm()` | | `BaseContextManager` | `tenxgraph/core/state/base_context.py` | `trim_context()`, `atrim_context()` | -| `BaseCheckpointer` | `tenxgraph/storage/checkpointer/base_checkpointer.py` | Async state, cache, message and thread methods | +| `BaseCheckpointer` | `tenxgraph/storage/checkpointer/base_checkpointer.py` | Async state, cache, message and thread methods, and `aping()` | | `BaseStore` | `tenxgraph/storage/store/base_store.py` | `astore()`, `asearch()`, `aget()`, `aget_all()`, `aupdate()`, `adelete()`, `aforget_memory()` | | `BaseEmbedding` | `tenxgraph/storage/store/embedding/base_embedding.py` | `aembed()`, `aembed_batch()`, `dimension` | | `BaseMediaStore` | `tenxgraph/storage/media/storage/base.py` | `store()`, `retrieve()`, `delete()`, `exists()` | @@ -67,7 +67,7 @@ flowchart TB | `BaseIDGenerator` | `tenxgraph/utils/id_generator.py` | `id_type`, `generate()` | | `BaseAuth` | `tenxgraph_api/src/app/core/auth/base_auth.py` | `authenticate(request, response, credential)` | | `AuthorizationBackend` | `tenxgraph_api/src/app/core/auth/authorization.py` | `authorize(user, resource, action, resource_id=None, **context)` | -| `BaseRateLimitBackend` | `tenxgraph_api/src/app/core/middleware/rate_limit/base.py` | `check(key, limit, window)`, `close()` | +| `BaseCacheStore` | `tenxgraph_api/src/app/core/cache_store/base.py` | `aping()`, `now()`, `acquire()`, `renew()`, `release()`, `get()`, `set()`, `delete()` | | `ThreadNameGenerator` | `tenxgraph_api/src/app/utils/thread_name_generator.py` | `generate_name(messages)` | | `BaseCriterion` | `tenxgraph/qa/evaluation/criteria/base.py` | `async evaluate(actual, expected)` | | `BaseReporter` | `tenxgraph/qa/evaluation/reporters/base.py` | `generate(report, output_dir=None)` | @@ -97,7 +97,7 @@ flowchart LR ### BaseCheckpointer: conversational state -A checkpointer persists your graph's state and thread history to a backend. You extend `BaseCheckpointer` when your deployment requires a specific database (DynamoDB for AWS shops, Firestore for Google Cloud, or something custom like a legacy oracle system). All of the async methods below are abstract, so a subclass must implement every one; the synchronous wrappers (`put_state()` and similar) are provided by the base class. +A checkpointer persists your graph's state and thread history to a backend. You extend `BaseCheckpointer` when your deployment requires a specific database (DynamoDB for AWS shops, Firestore for Google Cloud, or something custom like a legacy oracle system). All of the async methods below are abstract (`aping()` since 1.0.0rc4), so a subclass must implement every one; the synchronous wrappers (`put_state()` and similar) are provided by the base class. ```python from tenxgraph.storage.checkpointer.base_checkpointer import BaseCheckpointer @@ -112,6 +112,7 @@ class DynamoCheckpointer(BaseCheckpointer): async def arelease(self) -> None: ... # close connections # Cache, messages and thread metadata + async def aping(self) -> None: ... # raise if the backend is unreachable; read-only and cheap async def aput_state_cache(self, config, state) -> None: ... # hot-path write cache async def aget_state_cache(self, config) -> AgentState | None: ... # hot-path read cache async def aput_messages(self, config, messages, metadata=None) -> None: ... # store messages @@ -145,7 +146,7 @@ class PineconeStore(BaseStore): compiled = graph.compile(store=PineconeStore()) ``` -`aping` is abstract: a store that does not implement it raises `TypeError` when created. Return when the backend is reachable and raise otherwise; an in-process store just returns. `10xgraph-api` calls it for `/health/ready` and at startup. Added after 1.0.0rc2. +`aping` is abstract: a store that does not implement it raises `TypeError` when created. Return when the backend is reachable and raise otherwise; an in-process store just returns. `10xgraph-api` calls it for `/health/ready` and at startup. Added in 1.0.0rc3. ### BaseEmbedding: custom embedding model @@ -310,7 +311,7 @@ class PrefixedIDGenerator(BaseIDGenerator): ## API and server extension -Server-layer ABCs are wired via `10xgraph.json` (or the InjectQ container for rate-limit backends), with no code changes to the server required. You extend these to enforce your deployment's security policies, integrate custom identity systems, or implement rate limiting specific to your infrastructure. +Server-layer ABCs are wired via `10xgraph.json` (), with no code changes to the server required. You extend these to enforce your deployment's security policies, integrate custom identity systems, or keep the API's shared state in your own store. ### BaseAuth @@ -339,22 +340,25 @@ class RBACBackend(AuthorizationBackend): For simple role-to-scope mapping, you do not need a class. Set `authorization` to an RBAC config block (`{"backend": "rbac", "roles": {...}}`). The built-in `"ownership"` selector gives owner-only threads with no code, and `"allow_all"` permits any authenticated user. -### BaseRateLimitBackend +### BaseCacheStore -You extend `BaseRateLimitBackend` when you need rate limiting with custom logic: tiered limits based on user tier, graceful degradation, or integration with an external rate limiter service. +The API keeps everything it shares between workers in one `BaseCacheStore`: the per-thread run lock, run and WebSocket caps, idempotency, the ownership cache, token revocation and rate-limit counters. `RedisCacheStore` is the default when you set `redis` or `REDIS_URL` (Valkey, Dragonfly and KeyDB work through the URL); `InMemoryCacheStore` is the fallback for one process. You extend `BaseCacheStore` when the shared state has to live in something that does not speak the Redis protocol. ```python -from tenxgraph_api.src.app.core.middleware.rate_limit.base import ( - BaseRateLimitBackend, - RateLimitDecision, -) - -class RedisClusterRateLimiter(BaseRateLimitBackend): - async def check(self, key: str, *, limit: int, window: int) -> RateLimitDecision: ... - async def close(self) -> None: ... +from tenxgraph_api import BaseCacheStore, SlotResult + +class MyCacheStore(BaseCacheStore): + async def aping(self) -> None: ... + async def now(self) -> float: ... + async def acquire(self, key: str, limit: int, ttl: float) -> SlotResult: ... + async def renew(self, key: str, slot_id: str, ttl: float) -> bool: ... + async def release(self, key: str, slot_id: str) -> None: ... + async def get(self, key: str) -> bytes | None: ... + async def set(self, key: str, value: bytes, ttl: float | None, mode: str = "always") -> bool: ... + async def delete(self, key: str) -> None: ... ``` -The `check` method returns a `RateLimitDecision(allowed, remaining, reset_after)` (import it from the same `base` module). Bind an instance of your backend in the InjectQ container and set `backend: custom`; there is no import-path key for rate-limit backends. +The contract is strict (atomic `acquire`, TTLs never dropped early, no eviction). The full rules, a skeleton and the contract test suite are on [Bring your own backend](/docs/server/bring-your-own-backend). The `BaseRateLimitBackend` and `TokenRevocationBackend` classes and the `ConcurrencyLimiter` were removed in 1.0.0rc4; `BaseCacheStore` replaces all three. ### ThreadNameGenerator @@ -368,31 +372,18 @@ class DatePrefixNameGenerator(ThreadNameGenerator): return f"{date.today()}: {messages[0][:30]}" ``` -Wire any of these in `10xgraph.json`. Auth, authorization, and thread naming each have a dedicated top-level key. Custom rate-limit backends are bound in the InjectQ container instead: +Wire any of these in `10xgraph.json`. Auth, authorization, thread naming and the cache store each have a dedicated top-level key: ```json { "auth": { "method": "custom", "path": "auth.my_auth:ApiKeyAuth" }, "authorization": "auth.my_auth:RBACBackend", "thread_name_generator": "services.naming:DatePrefixNameGenerator", - "rate_limit": { - "enabled": true, - "backend": "custom", - "requests": 100, - "window": 60 - } + "cache_store": "services.cache:store" } ``` -The string format for class references is `module.path:ClassName` (dot-separated module path, colon, then the class or instance name). For the custom rate-limit backend, bind it in the module your injectq key points to: - -```python -from injectq import InjectQ -from tenxgraph_api.src.app.core.middleware.rate_limit.base import BaseRateLimitBackend - -container = InjectQ.get_instance() -container.bind_instance(BaseRateLimitBackend, RedisClusterRateLimiter()) -``` +The string format for class references is `module.path:ClassName` (dot-separated module path, colon, then the class or instance name). `cache_store` accepts an instance or a zero-argument factory (a class or function) that returns one. --- diff --git a/src/content/docs/concepts/serving-agents.md b/src/content/docs/concepts/serving-agents.md index 8c333ad..1b9f626 100644 --- a/src/content/docs/concepts/serving-agents.md +++ b/src/content/docs/concepts/serving-agents.md @@ -49,7 +49,6 @@ Import paths are **dotted module paths** (`module:attribute`), resolved with `im "agent": "graph.agent:get_compiled_graph", "auth": "jwt", "rate_limit": { - "backend": "memory", "requests": 100, "window": 60 } @@ -66,16 +65,15 @@ Import paths are **dotted module paths** (`module:attribute`), resolved with `im "authorization": "auth.agent_auth:MyAuthorizationBackend", "injectq": "graph.agent:container", "store": "services.store:my_store", - "redis": "redis://localhost:6379", + "redis": "${REDIS_URL}", + "token_revocation": true, "thread_name_generator": "services.naming:MyNameGenerator", "rate_limit": { - "backend": "redis", "requests": 1000, "window": 60, "by": "ip", "trusted_proxy_headers": true, - "exclude_paths": ["/ping", "/docs"], - "redis": {"url": "redis://localhost:6379"} + "exclude_paths": ["/ping", "/docs"] } } ``` @@ -209,12 +207,12 @@ Override by setting `authorization` to a `module:attr` path, a built-in name (`" ## Rate limiting -Rate limiting is pluggable via `BaseRateLimitBackend`. Two backends are built in (`memory` and `redis`); set `backend` to `custom` to supply your own. +Rate limiting keeps its counters in the API's cache store, the same store that holds the run locks and caps. There is no separate rate-limit backend. Set `redis` (or `REDIS_URL`) to share the counters between workers, or `cache_store` to supply your own `BaseCacheStore`. ```json { + "redis": "${REDIS_URL}", "rate_limit": { - "backend": "memory", "requests": 100, "window": 60, "by": "ip", @@ -225,72 +223,19 @@ Rate limiting is pluggable via `BaseRateLimitBackend`. Two backends are built in } ``` -The memory backend counts per process, so use `redis` when you run several workers. +Without a Redis URL the counters live in each process, so use `redis` when you run several workers. | Config key | Default | Purpose | |---|---|---| | `enabled` | true | Set `false` to turn rate limiting off. | -| `backend` | `memory` | `memory` (single-worker dev), `redis` (multi-worker), or `custom` (a `BaseRateLimitBackend` bound in InjectQ). | | `requests` | 100 | Requests allowed per window. | | `window` | 60 | Time window in seconds. | | `by` | `ip` | `ip` (per client IP), `user`, or `global` (all callers share one limit). | | `trusted_proxy_headers` | false | Honor `X-Forwarded-For` when true. | -| `fail_open` | true | Allow requests if the backend is unreachable. | +| `fail_open` | true | Allow requests if the cache store errors. `false` answers 503 `RATE_LIMIT_UNAVAILABLE`. | | `exclude_paths` | `[]` | Paths exempt from rate limiting. | -For the `redis` backend, add a `redis` key with the connection URL: - -```json -{ - "rate_limit": { - "backend": "redis", - "redis": { - "url": "redis://localhost:6379", - "prefix": "ratelimit:" - } - } -} -``` - -**Custom backend:** - -```python -# services/rate_limit.py -from tenxgraph_api.src.app.core.middleware.rate_limit.base import ( - BaseRateLimitBackend, - RateLimitDecision, -) - -class CustomRateLimitBackend(BaseRateLimitBackend): - async def check(self, key: str, *, limit: int, window: int) -> RateLimitDecision: - # Atomically count the request, then return a decision. - # allowed=False produces a 429 response. Return - # RateLimitDecision.backend_unavailable() for a 503 when your store is down. - return RateLimitDecision(allowed=True, remaining=limit - 1, reset_after=window) - - async def close(self) -> None: - ... -``` - -Set `"backend": "custom"` in `rate_limit`, and register the backend through the container that `injectq` points to: - -```json -{ - "injectq": "graph.agent:container", - "rate_limit": {"backend": "custom", "requests": 100, "window": 60} -} -``` - -Then in your graph module: - -```python -from injectq import InjectQ -from services.rate_limit import CustomRateLimitBackend -from tenxgraph_api.src.app.core.middleware.rate_limit.base import BaseRateLimitBackend - -container = InjectQ.get_instance() -container.bind_instance(BaseRateLimitBackend, CustomRateLimitBackend()) -``` +The keys `rate_limit.backend` and `rate_limit.redis` and the `BaseRateLimitBackend` class were removed in 1.0.0rc4. To store counters somewhere other than Redis, implement `BaseCacheStore`; see [Bring your own backend](/docs/server/bring-your-own-backend). ## Publishers @@ -429,7 +374,7 @@ ORIGINS=https://example.com,https://app.example.com | Variable | Default | Purpose | |---|---|---| | `MODE` | `development` | `production` or `prod` enables security checks; `development`, `dev` and `local` are the other values. Anything else stops startup | -| `REDIS_URL` | none | Redis connection used by server components such as the ownership cache | +| `REDIS_URL` | none | Redis connection for the API's cache store (locks, caps, idempotency, ownership cache, token revocation, rate limiting) | | `JWT_SECRET_KEY` | none | Required for JWT auth; use 32+ random chars | | `SENTRY_DSN` | none | Sentry error tracking | | `OTEL_ENABLED` | `false` | Enable OpenTelemetry tracing | diff --git a/src/content/docs/examples/rag-assistant.mdx b/src/content/docs/examples/rag-assistant.mdx index 07bb190..48b94a0 100644 --- a/src/content/docs/examples/rag-assistant.mdx +++ b/src/content/docs/examples/rag-assistant.mdx @@ -40,7 +40,7 @@ The stack runs with one `docker compose up`: the 10xGraph API server, an nginx-s | Retrieval | `QdrantStore`, collection `docs_kb` | Vector similarity search over doc chunks | | API server | 10xGraph API server (`10xgraph-api`), gunicorn with uvicorn workers | Loads `graph.agent:app`, exposes `/v1/graph/stream`, threads and `/ping` | | Thread storage | `PgCheckpointer`: Postgres 16 plus Redis 7 | Durable thread state (Postgres) and a fast cache (Redis) | -| Rate limiting | Redis backend, configured in `10xgraph.json` | 60 requests per 60 seconds per IP | +| Rate limiting | Shared through the Redis cache store (`"redis": "${REDIS_URL}"` in `10xgraph.json`) | 60 requests per 60 seconds per IP | | Frontend | React 19, Vite 6, `10xgraph-client`, served by nginx | Chat, thread list, source display | | Ingest job | `ingest.py` | Chunks `backend/data/*` and writes the chunks to Qdrant | @@ -363,11 +363,6 @@ Every chunk is stored with `source` and `title` metadata, which `search_docs` la ```json "enabled": true, - "backend": "redis", - "redis": { - "url": "${REDIS_URL}", - "prefix": "docs-assistant:rate-limit" - }, "requests": 60, "window": 60, "by": "ip", @@ -735,13 +730,9 @@ if __name__ == "__main__": "ag_ui": { "enabled": false }, + "redis": "${REDIS_URL}", "rate_limit": { "enabled": true, - "backend": "redis", - "redis": { - "url": "${REDIS_URL}", - "prefix": "docs-assistant:rate-limit" - }, "requests": 60, "window": 60, "by": "ip", @@ -1715,7 +1706,7 @@ The demo is set up for one command, not for the open internet. These are the cha - **Auth.** `10xgraph.json` ships with `"auth": null`. For production set `"auth": "jwt"`, set `JWT_SECRET_KEY` and `JWT_ALGORITHM`, and pass `authToken` to `TenxGraphClient` in `frontend/src/client.ts`. Thread ownership is then enforced per user. Use `"auth": {"method": "custom", "path": "module:Class"}` for your own identity provider. - **Mode and CORS.** Compose sets `MODE: development`. The image itself defaults to `MODE=production`, which refuses wildcard `ORIGINS` and `ALLOWED_HOST`. Remove the `MODE` override and set both to your real domain. -- **Redis rate limiting.** Already enabled with the Redis backend, 60 requests per 60 seconds per IP. Switch `"by"` to `"user"` once auth is on, and set `trusted_proxy_headers` when running behind a load balancer. +- **Redis rate limiting.** Already enabled through the Redis cache store, 60 requests per 60 seconds per IP. Switch `"by"` to `"user"` once auth is on, and set `trusted_proxy_headers` when running behind a load balancer. - **Postgres and Redis.** Compose runs both with a demo password (`docs`). Replace the password, point `POSTGRES_DSN` and `REDIS_URL` at managed instances if you have them, and do not publish the Postgres, Redis or Qdrant ports. - **Graceful shutdown.** The server drains in-flight runs on SIGTERM, then calls `aclose()` on the compiled graph. Keep your orchestrator's grace period at least 600 seconds (`stop_grace_period` here, `terminationGracePeriodSeconds` on Kubernetes). - **Workers.** Tune `WEB_CONCURRENCY` (default 2) to your CPU and memory. diff --git a/src/content/docs/examples/serve-with-auth.md b/src/content/docs/examples/serve-with-auth.md index 9efe687..a1ca7ad 100644 --- a/src/content/docs/examples/serve-with-auth.md +++ b/src/content/docs/examples/serve-with-auth.md @@ -24,7 +24,7 @@ You write the agent. `10xgraph-api` turns it into an HTTP service with JWT auth, 10xgraph init --template production --auth jwt --rate-limit redis --provider openai --yes ``` -`--rate-limit` takes `none`, `memory` or `redis`. Use `memory` for a first run without Redis; the Redis backend is the one that is shared between workers. +`--rate-limit` takes `none`, `memory` or `redis`. `redis` turns rate limiting on with a Redis cache store (`"redis": "${REDIS_URL}"`), which is shared between workers. `memory` turns it on without Redis, so the counters live in each process; use it for a first run. The command writes the project into the current folder. The files you will touch: @@ -35,18 +35,18 @@ The command writes the project into the current folder. The files you will touch | `.env.example` | `MODE="production"`, `ORIGINS`, `ALLOWED_HOST`, `JWT_SECRET_KEY`, `JWT_ALGORITHM`, `REDIS_URL`, `OPENAI_API_KEY`, empty `DOCS_PATH` | | `tests/`, `evals/` | A starter test and a starter eval file | -The rate limit section of `10xgraph.json` as generated: +The Redis, token revocation and rate limit sections of `10xgraph.json` as generated: ```json +"redis": "${REDIS_URL}", +"token_revocation": true, "rate_limit": { "enabled": true, - "backend": "redis", "requests": 100, "window": 60, "by": "ip", "trusted_proxy_headers": false, - "exclude_paths": ["/ping", "/docs", "/redocs", "/openapi.json"], - "redis": { "url": "${REDIS_URL}" } + "exclude_paths": ["/ping", "/docs", "/redocs", "/openapi.json"] } ``` diff --git a/src/content/docs/get-started/installation.md b/src/content/docs/get-started/installation.md index dc8b710..44e79a9 100644 --- a/src/content/docs/get-started/installation.md +++ b/src/content/docs/get-started/installation.md @@ -111,7 +111,7 @@ The `all` extra installs every other extra at once and is useful for local devel Version pinning still matters. From 1.0 the public API is stable under a deprecation policy: breaking changes arrive only in a major release, after at least one minor release of deprecation. The three packages share major.minor, and patch versions are independent. While 1.0 is a release candidate, install it with `--pre`: ```bash -pip install --pre "10xgraph==1.0.0rc2" "10xgraph-api==1.0.0rc2" +pip install --pre "10xgraph==1.0.0rc4" "10xgraph-api==1.0.0rc4" ``` Drop `--pre` once 1.0.0 is out. Read the [changelog](/changelog) before upgrading, and see the [upgrade guide](/docs/project/upgrade-to-1.0) if you come from 0.x. @@ -132,9 +132,9 @@ Example output (the exact formatting and versions vary by release): ```text $ 10xgraph version 10xgraph-api - Version: 1.0.0rc2 + Version: 1.0.0rc4 10xgraph (core) - Version: 1.0.0rc2 + Version: 1.0.0rc4 $ python -c "import tenxgraph; print(tenxgraph.__file__)" /path/to/.venv/lib/python3.12/site-packages/tenxgraph/__init__.py diff --git a/src/content/docs/get-started/project-structure.mdx b/src/content/docs/get-started/project-structure.mdx index 55a1d4f..5bd8398 100644 --- a/src/content/docs/get-started/project-structure.mdx +++ b/src/content/docs/get-started/project-structure.mdx @@ -405,9 +405,10 @@ Init builds `10xgraph.json` from your answers and writes it after the template f }, "authorization": "ownership", "injectq": "graph.agent:container", + "redis": "${REDIS_URL}", + "token_revocation": true, "rate_limit": { "enabled": true, - "backend": "redis", "requests": 100, "window": 60, "by": "ip", @@ -426,7 +427,9 @@ Init builds `10xgraph.json` from your answers and writes it after the template f | `thread_name_generator` | Always | `null` for quick-start, `graph.thread_name_generator:MyNameGenerator` for production | | `authorization` | Production with jwt or custom | `"ownership"`: a thread is accessible only to its owner. Other values are `"allow_all"` or a `module:attribute` backend. | | `injectq` | Production | `graph.agent:container` (see the warning below) | -| `rate_limit` | Production with a rate-limit backend | `enabled`, `backend`, `requests`, `window`, `by`, `trusted_proxy_headers`, `exclude_paths` | +| `redis` | Production | `${REDIS_URL}`: the Redis cache store the API shares between workers | +| `token_revocation` | Production | `true` | +| `rate_limit` | Production with rate limiting | `enabled`, `requests`, `window`, `by`, `trusted_proxy_headers`, `exclude_paths` | Defaults come from the prompts or flags: 100 requests per 60 seconds, counted `ip` (interactive mode also offers `global`), with `trusted_proxy_headers` true only if you say you are behind a reverse proxy. Keys such as `test`, `evaluation`, `remote_tools` and `store` are not written for production, though you can add them yourself. There is no `checkpointer` key to set: the server uses the checkpointer you pass to `graph.compile(checkpointer=...)`. The quick-start template adds none of these either. For every key and default, see the [configuration reference](/docs/reference/api-cli/configuration), and for rate-limit options see [Rate limiting](/docs/server/rate-limiting). diff --git a/src/content/docs/get-started/why-10xgraph.mdx b/src/content/docs/get-started/why-10xgraph.mdx index 3c0e7b1..7c2660f 100644 --- a/src/content/docs/get-started/why-10xgraph.mdx +++ b/src/content/docs/get-started/why-10xgraph.mdx @@ -60,7 +60,7 @@ Then run `10xgraph dev` locally, or `10xgraph build --docker-compose --k8s` for | Memory APIs | Save, search, update and forget long-term memories under `/v1/store` | | File uploads | Upload, download and signed URLs under `/v1/files`, with size and type limits | | AG-UI | `POST /v1/ag-ui` for CopilotKit and other AG-UI clients (opt-in) | -| Health | `/ping` for liveness, `/health/ready` for readiness (checks checkpointer, Redis and store) | +| Health | `/ping` for liveness, `/health/ready` for readiness (checks checkpointer, store and cache store) | | Security | Auth, scoped authorization, owner-only threads, token revocation, rate limits, security headers, size limits | | Deployment | Hardened Dockerfile, Docker Compose file, Kubernetes manifest | | Tooling | Playground, config editor, `test`, `eval` and `audit` commands | @@ -117,8 +117,9 @@ Defaults are there to get you to production quickly, not to lock you in. Each pa |---|---|---|---| | Authentication | Off, or built-in JWT | Any identity provider or API keys | Subclass `BaseAuth`; `"auth": {"method": "custom", "path": "auth.my_auth:MyAuth"}` | | Authorization | Owner-only threads | Role-based access, or your own rules | `{"backend": "rbac", ...}` or subclass `AuthorizationBackend` and set `"authorization": "module:Class"` | -| Token revocation | Off | Redis, memory or your own store | `"token_revocation": "redis"` or a `TokenRevocationBackend` | -| Rate limiting | Off | Memory, Redis or a custom backend | `rate_limit` block; `"backend": "custom"` with a `BaseRateLimitBackend` bound in your container | +| Token revocation | Off | Kept in the API's cache store (Redis, memory or your own) | `"token_revocation": true` plus `redis` or `cache_store` | +| Rate limiting | Off | Counters in the API's cache store (memory, Redis or your own) | `rate_limit` block plus `redis` or `"cache_store": "module:attr"` | +| Shared state (locks, caps, idempotency) | One process | Redis, Valkey, Dragonfly, KeyDB or your own `BaseCacheStore` | `redis`, `REDIS_URL` or `cache_store`; see [Bring your own backend](/docs/server/bring-your-own-backend) | | Persistence | In-memory checkpointer | Postgres + Redis, SQLite or your own | Pass it to `compile(checkpointer=...)` | | Long-term memory | None | Qdrant, Mem0 or your own `BaseStore` | `"store": "module:instance"` | | Events and tracing | None | Console, Redis, Kafka, RabbitMQ, OTel, Logfire, LangSmith | Bind a publisher, or use the `observability` block | diff --git a/src/content/docs/guides/graceful-shutdown.md b/src/content/docs/guides/graceful-shutdown.md index 869f8a0..06ee303 100644 --- a/src/content/docs/guides/graceful-shutdown.md +++ b/src/content/docs/guides/graceful-shutdown.md @@ -294,7 +294,7 @@ The image runs Gunicorn with `TenxGraphUvicornWorker`. On SIGTERM it does this, 1. The app is marked as draining. `GET /health/ready` answers `503` with `status: draining`, so the load balancer stops sending new requests. `/ping` stays `200`. See [Health check endpoint](/docs/reference/rest-api/ping). 2. Open responses get up to `TENXGRAPH_GRACEFUL_SHUTDOWN_SECONDS` (540 by default) to finish. Five seconds before that deadline, open streams end with a final chunk saying the run was interrupted. The thread is saved, so the client can send a new request on the same `thread_id`. -3. The lifespan stops the media sweeper, then calls `graph.aclose(grace, stop_timeout, timeout)` once, with `SHUTDOWN_GRACE_SECONDS` (10), `SHUTDOWN_STOP_SECONDS` (10) and `SHUTDOWN_TIMEOUT_SECONDS` (30). Active runs finish, are stopped and saved, or are cancelled and saved; the graph drains its own background work and releases the checkpointer, publisher and store. Then the rate-limit, concurrency and revocation backends are closed. +3. The lifespan stops the media sweeper, then calls `graph.aclose(grace, stop_timeout, timeout)` once, with `SHUTDOWN_GRACE_SECONDS` (10), `SHUTDOWN_STOP_SECONDS` (10) and `SHUTDOWN_TIMEOUT_SECONDS` (30). Active runs finish, are stopped and saved, or are cancelled and saved; the graph drains its own background work and releases the checkpointer, publisher and store. Then the API waits for pending lease releases and closes its cache store. A run that starts on this replica after `aclose()` begins is refused with `503` `SERVICE_UNAVAILABLE` and `Retry-After: 5`, so the client retries on another replica. 4. Open WebSockets are closed with code `1012` (service restart). The run is saved INTERRUPTED and the client reconnects. A run refused while the graph is closing sends a terminal `error` chunk (stream) or frame (WebSocket) with `data.code = "SERVICE_UNAVAILABLE"` and `retry_after_seconds`, and no `done` chunk. On a WebSocket the server then closes with 1012. diff --git a/src/content/docs/guides/set-up-checkpointing.md b/src/content/docs/guides/set-up-checkpointing.md index e90f536..9b2cdb3 100644 --- a/src/content/docs/guides/set-up-checkpointing.md +++ b/src/content/docs/guides/set-up-checkpointing.md @@ -195,6 +195,10 @@ async def lifespan(app: FastAPI): app = FastAPI(lifespan=lifespan) ``` +### Check that the checkpointer is reachable + +Every checkpointer has `await checkpointer.aping()`. It returns when the backends answer and raises when they do not. For `PgCheckpointer` it runs `SELECT 1`, checks that the schema version is not older than the code expects, and pings Redis. It never creates the schema, so call `asetup()` first. The API server calls it at startup and on `/health/ready`. A custom `BaseCheckpointer` must implement it (abstract since 1.0.0rc4); see [Checkpointers](/docs/reference/python/checkpointers#aping). + ### Multi-user isolation By default, `PgCheckpointer` treats `user_id` as an **ownership boundary**. Threads, state, messages, and the [tool ledger](/docs/concepts/replay-safe-tools) are scoped to the authenticated user, so even if an attacker knows another user's `thread_id`, they cannot read, write, or delete it. This is critical in multi-tenant deployments. diff --git a/src/content/docs/production/architecture.md b/src/content/docs/production/architecture.md index 8c90a4e..c5aeece 100644 --- a/src/content/docs/production/architecture.md +++ b/src/content/docs/production/architecture.md @@ -25,12 +25,12 @@ flowchart LR LB --> W2[API worker] LB --> W3[API worker] W1 & W2 & W3 --> PG[(Postgres: durable state)] - W1 & W2 & W3 --> R[(Redis: cache, locks, caps)] + W1 & W2 & W3 --> R[(Cache store: Redis by default; locks, caps, revocation)] W1 & W2 & W3 --> V[(Vector store: long-term memory)] W1 & W2 & W3 --> M[(Media store)] ``` -Workers hold no conversation state between requests. They hold caches (a short-lived thread-ownership cache) and, without Redis, per-process counters, which is why Redis matters once you run more than one worker. See [Scaling and reliability](/docs/production/scaling) for exactly what each Redis feature shares. +Workers hold no conversation state between requests. They hold caches (a short-lived thread-ownership cache) and, with the in-memory cache store, per-process counters, which is why a shared cache store (Redis by default) matters once you run more than one worker. See [Scaling and reliability](/docs/production/scaling) for exactly what it shares, and [Bring your own backend](/docs/server/bring-your-own-backend) for replacing it. ## The request path @@ -42,7 +42,7 @@ This is the order a request travels, outermost layer first. The middleware order | 2 | Request id | Adds `X-Request-ID` and `X-Timestamp` to every response. | | 3 | Security headers | Adds HSTS, frame, content-type and CSP headers when `SECURITY_HEADERS_ENABLED` is on. | | 4 | Trusted host | Rejects Host headers outside `ALLOWED_HOST`. `/ping` and `/health/ready` skip this check so probes on `localhost` or a pod IP still work. | -| 5 | Rate limit | Runs only when `rate_limit` is set in `10xgraph.json`. Backend is memory, Redis or custom. | +| 5 | Rate limit | Runs only when `rate_limit` is set in `10xgraph.json`. Counters live in the API's cache store (in-memory, Redis or your own). | | 6 | Request size | Rejects bodies over `MAX_REQUEST_SIZE`. The upload route gets room for `MEDIA_MAX_SIZE_MB`. | | 7 | Gzip | Compresses responses over 1000 bytes, except `/v1/graph/stream`, which must send data as it is produced. | | 8 | Route guard | `RequirePermission` authenticates the caller (JWT or your `BaseAuth`), checks token revocation if configured, then asks the authorization backend. The default `ownership` backend lets a thread be read or run only by its owner. A foreign thread is a 403 before the graph is touched. | diff --git a/src/content/docs/production/index.md b/src/content/docs/production/index.md index f756c14..6a2b559 100644 --- a/src/content/docs/production/index.md +++ b/src/content/docs/production/index.md @@ -33,8 +33,8 @@ When you serve a compiled graph with 10xgraph-api, the server owns the productio | Durable state | With `PgCheckpointer`, state lives in PostgreSQL, with Redis as a hot cache. Writes are versioned per thread (compare-and-swap), and a stale run cannot move the cache backwards. | `PgCheckpointer` passed to `compile()`, `DATABASE_URL`, `REDIS_URL`. | [Set up checkpointing](/docs/guides/set-up-checkpointing) | | Replay-safe tools | A tool call that finished before a crash is recorded in a ledger and not run again on resume (invoke path, with a ledger-capable checkpointer). | `PgCheckpointer`; send an idempotency key to external services as a second layer. | [Replay-safe tools](/docs/concepts/replay-safe-tools) | | Observability | OpenTelemetry traces and metrics when enabled. Sentry reports only 5xx server faults and does not send default PII. Core log output redacts known secret formats. | `OTEL_ENABLED`, `OTEL_EXPORTER_OTLP_ENDPOINT`, `SENTRY_DSN`, `LOG_LEVEL`. | [Observability](/docs/server/observability) | -| Deployment | `10xgraph build` writes a Dockerfile that runs Gunicorn, with optional `docker-compose.yml` and `k8s.yaml`. `GET /ping` is liveness and `GET /health/ready` is readiness (checkpointer, Redis, store). | `10xgraph build --docker-compose --k8s`. | [Deploy](/docs/server/deploy), [Kubernetes](/docs/server/kubernetes) | -| Scaling | Workers hold no conversation state; it lives in the checkpointer. With Redis configured, the ownership cache, run caps and thread locks are shared across workers. | `redis` or `REDIS_URL`, `rate_limit.backend: "redis"`, two or more instances. | [Scaling and reliability](/docs/production/scaling) | +| Deployment | `10xgraph build` writes a Dockerfile that runs Gunicorn, with optional `docker-compose.yml` and `k8s.yaml`. `GET /ping` is liveness and `GET /health/ready` is readiness (checkpointer, store, cache store). | `10xgraph build --docker-compose --k8s`. | [Deploy](/docs/server/deploy), [Kubernetes](/docs/server/kubernetes) | +| Scaling | Workers hold no conversation state; it lives in the checkpointer. With a shared cache store configured (Redis), the ownership cache, run caps, thread locks and token revocations are shared across workers. | `redis` or `REDIS_URL` (or `cache_store`), two or more instances. | [Scaling and reliability](/docs/production/scaling) | The first three rows decide who can reach what. If you only do one thing beyond the quickstart, configure `auth`. diff --git a/src/content/docs/production/limits.md b/src/content/docs/production/limits.md index 7f8411b..2e3256f 100644 --- a/src/content/docs/production/limits.md +++ b/src/content/docs/production/limits.md @@ -54,6 +54,12 @@ The container serves plain HTTP. The server sets security headers and can emit H What to do: terminate TLS at your load balancer or ingress, set `ALLOWED_HOST` and `ORIGINS`, and if you use client IP rate limiting behind a proxy, set `rate_limit.trusted_proxies`. See [CORS and security headers](/docs/server/cors-and-security-headers) and [Kubernetes](/docs/server/kubernetes). +## The default cache store is per process + +Without `redis`, `REDIS_URL` or `cache_store`, the API keeps its run locks, caps, idempotency state, rate-limit counters and token revocations in the memory of one process (`InMemoryCacheStore`). With more than one worker or replica those do not add up: a revocation reaches only one process, and caps multiply by the worker count. The store also caps its memory, so under heavy churn it evicts the least recently used rate-limit and cache entries (never run leases or revocations). + +What to do: set `redis` or `REDIS_URL` (or your own `cache_store`) for any multi-worker deployment, and run Redis with `maxmemory-policy` set to `noeviction` or a `volatile-*` policy. See [Bring your own backend](/docs/server/bring-your-own-backend). + ## Data retention and backups Threads, messages and uploaded files stay until something deletes them. Deleting a thread or message also releases the files it referenced, but 10xGraph sets no retention period and does not back up your database. diff --git a/src/content/docs/production/scaling.md b/src/content/docs/production/scaling.md index 4201d79..9f37262 100644 --- a/src/content/docs/production/scaling.md +++ b/src/content/docs/production/scaling.md @@ -7,29 +7,30 @@ order: 30 updated: "2026-10-28" faq: - q: "Do I need Redis to run more than one worker?" - a: "You can run without it, but the per-thread run lock, the per-user run cap, WebSocket caps, rate limits and token revocation then count per process. A cap of 10 across 4 workers becomes up to 40. Configure Redis for any multi-worker deployment." + a: "You can run without it, but the per-thread run lock, the per-user run cap, WebSocket caps, idempotency, rate limits and token revocation then live in each process (the in-memory cache store). A cap of 10 across 4 workers becomes up to 40. Configure Redis, or your own cache store, for any multi-worker deployment." - q: "What happens if Redis goes down?" - a: "Rate limiting follows rate_limit.fail_open (default true: requests are allowed). The run lock, run cap and WebSocket caps fail open. Ownership lookups fall back to the database. Token revocation fails closed with a 503. The readiness probe reports Redis as failed." + a: "Rate limiting follows rate_limit.fail_open (default true: requests are allowed; false answers 503 RATE_LIMIT_UNAVAILABLE). The run lock, run cap, WebSocket caps and idempotency fail open. Ownership lookups fall back to the database. Token revocation fails closed with a 503. The readiness probe reports the cache store as failed, and returns 503 only when token revocation or a fail-closed rate limit uses it." - q: "What happens if Postgres goes down?" a: "Runs that need the checkpointer fail, and GET /health/ready returns 503 so an orchestrator takes the pod out of rotation. Liveness (/ping) stays up." - q: "Is there a queue for long runs?" a: "No. A run executes in the worker that received the request, bounded by runs.timeout. Size workers and the graceful timeout for your longest run." --- -You scale a served 10xGraph agent by running more stateless API workers behind a load balancer, with Postgres for durable state and Redis for the things workers must share: locks, caps, rate limits and caches. Without Redis, those features still work but count per process. This page lists exactly what Redis shares, how reliability works under failure, and where the limits are. For the request path and state layers, read [Architecture](/docs/production/architecture) first. +You scale a served 10xGraph agent by running more stateless API workers behind a load balancer, with Postgres for durable state and a shared cache store (Redis by default) for the things workers must share: locks, caps, rate limits and caches. Without one, those features still work but count per process. This page lists exactly what the cache store shares, how reliability works under failure, and where the limits are. For the request path and state layers, read [Architecture](/docs/production/architecture) first. -## What Redis shares across workers +## What the cache store shares across workers -Set the `redis` key in `10xgraph.json` or `REDIS_URL`. The loader then binds a shared limiter for the run and WebSocket caps, and the ownership cache gets its second tier. +The API keeps its shared state in one `BaseCacheStore`. Set the `redis` key in `10xgraph.json` or `REDIS_URL` to use `RedisCacheStore` (Valkey, Dragonfly and KeyDB work through the same URL), or set `cache_store` to your own class. With none of them set the store is `InMemoryCacheStore`, in one process. See [Bring your own backend](/docs/server/bring-your-own-backend). The tables below say "Redis" for the shared case. | Feature | With Redis | Without Redis (per process) | |---|---|---| | Thread ownership cache | L1 in each worker (30 s), L2 shared in Redis (1 h). Only known owners are cached, never "no owner". Deleting a thread evicts both and leaves a tombstone for 30 s so another user cannot claim the id while a worker still holds the old owner. | L1 only; the tombstone exists only on the worker that handled the delete. | -| Rate limit | `rate_limit.backend: "redis"`: one sliding window for all workers. | `memory` backend: each worker counts alone, so the real limit is `requests x workers`. | +| Rate limit | One sliding window for all workers. | Each worker counts alone, so the real limit is `requests x workers`. | | Per-thread run lock | One-slot lease shared by all workers. Renewed while the run is alive, expires if the worker dies. | Lock holds only for runs on the same worker. Two workers can run one thread at once, and the Postgres version check then rejects the later write with a 409. | | Per-user run cap (`runs.max_concurrent_per_user`) | Leases shared by all workers. | Counted per process: up to `limit x workers`. | | WebSocket caps (`websocket.max_connections`, `max_connections_per_user`) | Shared through the same lease limiter. | Counted per process. | -| Token revocation | `"token_revocation": "redis"`: a revoked token is refused on every worker. | `"memory"` revokes on one process only; use it for tests or a single worker. | +| Idempotency (running lease and stored records) | Shared, so a retry on another worker sees the first request. | Seen only by the worker that handled the first request. | +| Token revocation | `"token_revocation": true`: a revoked token is refused on every worker. | A revocation reaches only the process that holds it, and the server warns about it in production or when `WEB_CONCURRENCY` is above 1. `10xgraph revoke` refuses to run against an in-memory store. | The leases expire on their own, so a crashed worker cannot hold a thread or a user's slots forever. @@ -37,7 +38,7 @@ The leases expire on their own, so a crashed worker cannot hold a thread or a us `10xgraph build` generates a Dockerfile that runs Gunicorn with Uvicorn workers. The worker count comes from `WEB_CONCURRENCY`, set to `2` in the image; override it at deploy time, for example `-e WEB_CONCURRENCY=8`. The image runs `TenxGraphUvicornWorker` and sets a graceful timeout of 600 seconds and a worker timeout of 660 seconds, because an agent run can last minutes and the default 30 seconds would kill runs during a rolling deploy. With `--k8s`, the generated manifest sets `terminationGracePeriodSeconds` above that and a `preStop` sleep so the load balancer stops sending new traffic first. -Each in-flight run holds a connection open and waits on model calls, so size and autoscale on concurrent runs, not CPU alone. Choose the worker count and the per-user cap together: without Redis the cap is per worker. +Each in-flight run holds a connection open and waits on model calls, so size and autoscale on concurrent runs, not CPU alone. Choose the worker count and the per-user cap together: without a shared cache store the cap is per worker. ## Snowflake IDs @@ -57,19 +58,19 @@ A publisher sends run events (node start and end, tool calls, errors) to systems | Timeouts | `runs.timeout` bounds a whole run. Set `node_timeout` and `tool_timeout` (no default) so one stuck tool cannot hold a run forever. | | Parallel tool cap | Tools run in parallel up to 10 by default. Change it with `runs.max_parallel_tool_calls` or `TENXGRAPH_MAX_PARALLEL_TOOL_CALLS`. | | Graceful shutdown | On SIGTERM the worker marks the app as draining, so `/health/ready` answers 503 and the load balancer stops sending new requests. Open runs get up to `TENXGRAPH_GRACEFUL_SHUTDOWN_SECONDS` (540 by default) to finish. Open streams then end with a final "interrupted" chunk. The app calls `graph.aclose()` with `SHUTDOWN_GRACE_SECONDS`, `SHUTDOWN_STOP_SECONDS` and `SHUTDOWN_TIMEOUT_SECONDS` (10, 10 and 30 seconds by default), which also drains background work, and closes limiters, revocation and authorization backends. A run that was cut off leaves a thread the next message can use. See [Graceful shutdown](/docs/guides/graceful-shutdown). | -| Readiness probe | `GET /health/ready` checks the checkpointer, Redis and store (each time-boxed to 2 s, cached for 3 s) and returns 200 or 503. It is public and returns only `ok` or `fail` per component. `/ping` is the liveness probe. | +| Readiness probe | `GET /health/ready` calls `aping()` on the checkpointer, the store and the cache store (each time-boxed to 2 s, cached for 3 s) and returns 200 or 503. The response keys are `checkpointer`, `store` and `cache`. The cache store fails readiness only when token revocation or a fail-closed rate limit uses it. It is public and returns only `ok` or `fail` per component. `/ping` is the liveness probe. | ## Failure behaviour | Failure | What happens | |---|---| -| Redis down: rate limit | Follows `rate_limit.fail_open`. Default `true` allows the request; set `false` to deny. | -| Redis down: run lock, run cap, WebSocket caps | Fail open. The request proceeds and the error is logged, because refusing every request would be an outage of its own. The Postgres version check still protects state. | +| Redis down: rate limit | Follows `rate_limit.fail_open`. Default `true` allows the request; set `false` to answer `503` `RATE_LIMIT_UNAVAILABLE` with `Retry-After: 5`. | +| Redis down: run lock, run cap, WebSocket caps, idempotency | Fail open. The request proceeds and the error is logged, because refusing every request would be an outage of its own. The Postgres version check still protects state. | | Redis down: ownership | Falls back to the checkpointer lookup in the database. A request is never failed by the cache. | | Redis clock | Lease expiry for the run cap uses the Redis clock (`TIME`), not each pod's own, so clock skew between pods cannot trim a live lease. | | Redis not answering | Every call fails after `REDIS_SOCKET_TIMEOUT` (5 seconds by default) instead of hanging the request. See [environment variables](/docs/reference/api-cli/environment#redis-variables). | | Redis down: token revocation | Fails closed: a 503, not a possibly revoked token served. | -| Redis down: readiness | `/health/ready` reports `redis: fail` and returns 503. | +| Redis down: readiness | `/health/ready` reports `cache: fail`. It returns 503 only when token revocation or a fail-closed rate limit uses the cache store; otherwise the pod stays in rotation. | | Postgres down | Runs that need the checkpointer fail. `/health/ready` returns 503, so Kubernetes stops routing to the pod. `/ping` stays 200 so the pod is not restarted for a dependency outage. | | Worker dies mid-run | Locks and caps held in Redis expire. Resume by sending the thread another request; finished tool calls are skipped. | @@ -77,7 +78,7 @@ A publisher sends run events (node start and end, tool calls, errors) to systems - Runs execute in the worker that received the request. There is no separate queue or worker tier, so a long run ties up that worker's connection and a deploy waits on the graceful timeout. - A tool can still run twice if a run crashes in the short gap between the tool finishing and its ledger record being written, or if the run was cut off while the tool was running. Use idempotency keys for tools with side effects. -- Without Redis, every cap and lock in the table above is per process. +- With the in-memory cache store, every cap and lock in the table above is per process. - There are no published benchmarks for throughput or latency. Load-test your own graph, model and tools. - Pin versions and read the changelog before upgrading. 1.0 is a release candidate until the final ships; from 1.0 breaking changes come only in a major release. diff --git a/src/content/docs/production/security-model.md b/src/content/docs/production/security-model.md index 8a68030..1c4dfe8 100644 --- a/src/content/docs/production/security-model.md +++ b/src/content/docs/production/security-model.md @@ -41,7 +41,7 @@ Identity comes only from the authenticated user in the run config. The server ne | Threat | What the server does | Where to configure | More | |---|---|---|---| -| Stolen or forged token | Validates the signature and expiry, requires a non-empty `user_id` claim, checks `iss` and `aud` when `JWT_ISSUER` and `JWT_AUDIENCE` are set, refuses an HS* secret under 32 bytes in production, and rejects revoked tokens (HTTP, WebSocket runs, and every 30 s during a live session; a backend failure fails closed with 503) | `auth`, `JWT_SECRET_KEY`, `JWT_ALGORITHM`, `token_revocation` | [Authentication](/docs/server/auth) | +| Stolen or forged token | Validates the signature and expiry, requires a non-empty `user_id` claim, checks `iss` and `aud` when `JWT_ISSUER` and `JWT_AUDIENCE` are set, refuses an HS* secret under 32 bytes in production, and rejects revoked tokens (HTTP, WebSocket runs, and every 30 s during a live session; a cache store failure fails closed with 503) | `auth`, `JWT_SECRET_KEY`, `JWT_ALGORITHM`, `token_revocation` | [Authentication](/docs/server/auth) | | Another user's thread | Ownership is object-level: a foreign `invoke` or `stream` is rejected with 403 before the graph starts, and every checkpointer read, write and delete is checked. With `runs.thread_ids: "server"` (default) a client may only name an existing thread; a new thread starts by omitting the id | `authorization`, `runs.thread_ids` | [Authentication](/docs/server/auth) | | Another user's files and memories | Threads and memories are scoped per user by default (`isolation_scope` returns `owner`), downloads and deletes are owner-only, `MEDIA_REQUIRE_OWNER` is on in production, and no identity starting with `__system__` is accepted, so a caller cannot act as the owner of shared data | custom `AuthorizationBackend.isolation_scope`, `MEDIA_REQUIRE_OWNER` | [Files and multimodal](/docs/server/files-and-multimodal) | | Forged state or messages | Clients send `user` messages only (tool results only to answer a pending remote tool call), `context_summary` and `execution_meta` are never writable, the server replaces client message ids, `current_node` must be a real node, and state values must match the state model | none; always on | [Client can and cannot send](#what-a-client-can-and-cannot-send) | diff --git a/src/content/docs/project/index.md b/src/content/docs/project/index.md index 0c272cb..c53e496 100644 --- a/src/content/docs/project/index.md +++ b/src/content/docs/project/index.md @@ -8,7 +8,7 @@ label: Overview updated: "2026-10-28" --- -This section covers 10xGraph as a project: who maintains it, how releases are announced, how to report a security problem, where to get help and how to contribute. It is MIT licensed, maintained by 10xScale, and at the 1.0 release candidate stage (1.0.0rc2, 2026-10-30) after its October 2026 rename from Agentflow. +This section covers 10xGraph as a project: who maintains it, how releases are announced, how to report a security problem, where to get help and how to contribute. It is MIT licensed, maintained by 10xScale, and at the 1.0 release candidate stage (1.0.0rc4, 2026-10-31) after its October 2026 rename from Agentflow. ## Start here diff --git a/src/content/docs/project/upgrade-to-1.0.md b/src/content/docs/project/upgrade-to-1.0.md index ede12d1..37e22e0 100644 --- a/src/content/docs/project/upgrade-to-1.0.md +++ b/src/content/docs/project/upgrade-to-1.0.md @@ -6,7 +6,7 @@ description: Migration guide from 10xgraph 0.10, 10xgraph-api 0.9 and 10xgraph-c section: Project order: 20 label: Upgrade to 1.0 -updated: "2026-10-29" +updated: "2026-10-31" --- 1.0 is the release where the public API is stable under a deprecation policy. The core @@ -24,7 +24,7 @@ read [Upgrade to 0.9](/docs/project/upgrade-to-0.9) first. client candidate is published under the `next` dist-tag. ```bash -pip install -U --pre "10xgraph==1.0.0rc2" "10xgraph-api==1.0.0rc2" +pip install -U --pre "10xgraph==1.0.0rc4" "10xgraph-api==1.0.0rc4" npm install 10xgraph-client@next ``` @@ -41,6 +41,67 @@ npm install 10xgraph-client@next --- +## From 1.0.0rc3 to 1.0.0rc4 + +Added in 1.0.0rc4. The API's shared state (run locks and caps, idempotency, the ownership cache, +token revocation and rate limiting) now lives behind one class, `BaseCacheStore`. Its default is +`RedisCacheStore` when you set a Redis URL and `InMemoryCacheStore` otherwise. The old +extension points are removed with no aliases; a config that still uses one stops the server at +boot with an error that names `cache_store`. + +| Removed | Use instead | +|---|---| +| `ConcurrencyLimiter` (and its Redis and memory variants) | `BaseCacheStore`; the run lock, run caps, WebSocket caps and idempotency use its timed slots. Nothing to configure. | +| `TokenRevocationBackend` (with its in-memory and Redis variants) and `build_token_revocation` | `BaseCacheStore`. Revocations are kept in the cache store. | +| `"token_revocation": "redis"`, `"memory"` or `"module:attr"` | `"token_revocation": true`, or `{"enabled": true, "token_lifetime": , "user_ttl": }`. `user_ttl` defaults to 30 days; `null` keeps entries forever. Set `redis` (or `REDIS_URL`) to share revocations between workers. | +| `BaseRateLimitBackend`, `MemoryRateLimitBackend`, `RedisRateLimitBackend` | `BaseCacheStore`. Rate limiting takes timed slots from the cache store. | +| `rate_limit.backend` (`memory`, `redis`, `custom`) | Delete it. The counters follow the cache store: `redis` or `REDIS_URL` shares them, `cache_store` supplies your own. | +| `rate_limit.redis` (`url`, `prefix`) | Delete it and set the top-level `"redis": "${REDIS_URL}"` (or `REDIS_URL`). | + +Other changes: + +- **`BaseCheckpointer.aping()` is now abstract (core).** A custom checkpointer must implement + `async def aping(self) -> None`: return when its backends are reachable and usable, raise when + they are not. Keep it read-only and cheap, because the server calls it on `/health/ready` and + at startup. An in-process checkpointer just returns. A checkpointer without it raises + `TypeError` when created. `InMemoryCheckpointer`, `SqliteCheckpointer` and `PgCheckpointer` + implement it. `PgCheckpointer.aping()` never creates the schema. +- **Choosing the cache store.** `"cache_store": "module:attr"` (an instance or a zero-argument + factory) wins over `redis`, which wins over `REDIS_URL`; with none of them the store is + `InMemoryCacheStore`. The server logs one INFO line at boot that names the store and where the + choice came from. Valkey, Dragonfly and KeyDB work through the Redis URL. +- **The startup check and `/health/ready` only call `aping()`.** The cache store stops the boot, + and fails readiness, only when a fail-closed feature uses it: `token_revocation` is on, or + `rate_limit` is on with `fail_open` set to `false`. Otherwise an unreachable cache store logs + one WARNING and the server starts. This fixes a bug in rc3 where a leftover `REDIS_URL` blocked + boot for a setup that needed no Redis. The `/health/ready` keys are `checkpointer`, `store` and + `cache`; the key `redis` is gone. Update probes or monitors that read it. +- **Failure behaviour.** Run locks, caps, idempotency and the ownership cache fail open. Token + revocation fails closed (503). Rate limiting follows `rate_limit.fail_open` (default `true`; + `false` answers 503 `RATE_LIMIT_UNAVAILABLE` with `Retry-After: 5`). +- **Rate-limit counters reset once.** The Redis keys moved to the `10xgraph:ratelimit2` + namespace. Other Redis key layouts are unchanged, so a rolling deploy shares locks between old + and new pods. +- **An in-memory cache store with token revocation warns.** In production, or with + `WEB_CONCURRENCY` above 1, the server warns that revocations only reach this process. +- **`10xgraph revoke` builds the configured cache store** the same way as the server (the + `--redis-url` override stays) and refuses with a clear error when it is `InMemoryCacheStore`. +- **`10xgraph init --rate-limit redis`** now means rate limiting on with a Redis cache store + (`"redis": "${REDIS_URL}"`); `memory` means rate limiting on without Redis. The production + template has `"redis": "${REDIS_URL}"` and `"token_revocation": true`, and no + `rate_limit.backend` or `rate_limit.redis`. + +Migration steps: + +1. Replace `token_revocation` strings with `true` or an object, and delete `rate_limit.backend` + and `rate_limit.redis`. +2. Add `"redis": "${REDIS_URL}"` (or set `REDIS_URL`) if you ran with more than one worker. +3. Implement `aping()` on any custom checkpointer. Port a custom rate-limit or revocation backend + to `BaseCacheStore`; see [Bring your own backend](/docs/server/bring-your-own-backend). +4. Update monitors that read `checks.redis` from `/health/ready` to read `checks.cache`. + +--- + ## Core (`10xgraph`) ### OpenAI @@ -100,7 +161,7 @@ npm install 10xgraph-client@next - **Skill and memory text is no longer run through `str.format`.** Only your own `system_prompt` is interpolated, and a bad placeholder is logged instead of failing the run. Remove any reliance on `{placeholder}` expansion inside skill or memory text. -- **`BaseStore.aping()` is now abstract (added after 1.0.0rc2).** A custom store must implement +- **`BaseStore.aping()` is now abstract (added in 1.0.0rc3).** A custom store must implement `async def aping(self) -> None`: return when its backend is reachable, raise when it is not. Keep it read-only and cheap, because `10xgraph-api` calls it on `/health/ready` and at startup. An in-process store just returns. A store without it raises `TypeError` when @@ -304,7 +365,7 @@ npm install 10xgraph-client@next ### Startup check and rate limiter outages -Added after 1.0.0rc2. +Added in 1.0.0rc3. - **The server refuses to start while a configured backend is unreachable.** At boot it pings the checkpointer's Postgres and Redis, the API's Redis, the rate limiter's own Redis and the @@ -416,7 +477,7 @@ The extras raise these floors. A fresh install resolves them; an existing enviro After upgrading, before sending production traffic: -1. `10xgraph version` reports 1.0.0rc2 for core and the CLI. +1. `10xgraph version` reports 1.0.0rc4 for core and the CLI. 2. The server starts with `MODE=production` and explicit `ALLOWED_HOST` and `ORIGINS`. 3. A new conversation without a `thread_id` succeeds, and a client-invented id for a new thread gets 403 (unless you set `runs.thread_ids` to `"client"`). diff --git a/src/content/docs/reference/api-cli/auth.md b/src/content/docs/reference/api-cli/auth.md index 4a3f625..392ee28 100644 --- a/src/content/docs/reference/api-cli/auth.md +++ b/src/content/docs/reference/api-cli/auth.md @@ -6,7 +6,7 @@ section: Reference group: "CLI and configuration" order: 440 label: Authentication -updated: "2026-10-27" +updated: "2026-10-31" --- By default, the 10xGraph API accepts all requests without authentication (suitable for local development). For production, configure either JWT or a custom `BaseAuth` backend to verify identity, and optionally set authorization rules via `AuthorizationBackend` to control what authenticated users may do. @@ -77,7 +77,7 @@ Other claims (`roles`, `scopes`, `email`, custom fields) are passed through to ` Two claims are also read by the server: - `scope` (a space-separated string, as in OAuth) and `scopes` (a list of strings, or a space-separated string). Both are merged into one set of scope names, and a scope matches only by its exact name. `graph:invoke_admin` does not grant `graph:invoke`. A token with neither claim is not scope-restricted. A token with `scope` set to an empty string is restricted to no scopes at all. A scope claim of the wrong type (a number, an object, or a list that holds non-strings) is rejected with 401. -- `jti`, the token ID. When `token_revocation` is set in `10xgraph.json`, a revoked `jti` is refused. The check runs even when `user_id` is missing or unusual. +- `jti`, the token ID. When `token_revocation` is on in `10xgraph.json`, a revoked `jti` is refused. The check runs even when `user_id` is missing or unusual. See [Token revocation](#token-revocation). ### Rotating the signing key @@ -257,7 +257,7 @@ Decision table: @@ -268,11 +268,27 @@ Ownership is immutable, so it is cached by `ThreadOwnershipResolver` for scalabi | Tier | Details | |---|---| | L1 | In-process LRU: 10,000 entries per worker, 30 s TTL | -| L2 | Optional shared Redis client (owner entries under `af:authz:owner:id:`, kept for 1 hour; a delete removes the entry and leaves a "just deleted" marker under `af:authz:owner:tomb:` for 30 seconds, so the id cannot be claimed while another worker still caches the old owner; Redis errors degrade to database lookup and never fail the request) | +| L2 | Optional shared tier in the API's cache store (owner entries under `af:authz:owner:id:`, kept for 1 hour; a delete removes the entry and leaves a "just deleted" marker under `af:authz:owner:tomb:` for 30 seconds, so the id cannot be claimed while another worker still caches the old owner; cache store errors degrade to database lookup and never fail the request) | -Negative results are not cached (a missing thread could be created by a later request). The L2 tier is wired from `redis` in `10xgraph.json`, falling back to `REDIS_URL`. When neither is set or the `redis` package is not installed, L1 only is used and a warning is logged at startup. +Negative results are not cached (a missing thread could be created by a later request). The L2 tier is the API's cache store: `cache_store` in `10xgraph.json`, else `redis`, else `REDIS_URL`. With the in-memory cache store the L2 tier is local to the process, so it adds nothing over L1. -Cache eviction happens on thread delete via `CheckpointerService.delete_thread`. A custom caching backend should implement `evict(thread_id)` (returns the resolver's `evict` coroutine or `None`) and `aclose()` (closes any client on shutdown). +Cache eviction happens on thread delete via `CheckpointerService.delete_thread`. A custom authorization backend should implement `evict(thread_id)` (returns the resolver's `evict` coroutine or `None`) and `aclose()`. + +### Token revocation + +Revoke a token (by `jti`) or all of a user's tokens with `10xgraph revoke`. The revocations live in the API's cache store, so with Redis they reach every worker at once and the server needs no restart. Turn the check on in `10xgraph.json`: + +```json +{ + "auth": "jwt", + "redis": "${REDIS_URL}", + "token_revocation": true +} +``` + +`token_revocation` is `true`, `false`, or an object: `{"enabled": true, "token_lifetime": , "user_ttl": }`. `user_ttl` is how long a per-user cutoff is kept and defaults to 30 days; `null` keeps entries forever. The old values `"redis"`, `"memory"` and `"module:attr"`, and the `TokenRevocationBackend` class, were removed in 1.0.0rc4 and now stop startup with an error that names `cache_store`. + +Revocation fails closed: if the cache store raises an error, requests that need the check answer `503`. With an in-memory cache store in production, or `WEB_CONCURRENCY` above 1, the server warns that revocations only reach this process; set `redis` or `REDIS_URL`, or a shared `cache_store`. ### Custom authorization backend diff --git a/src/content/docs/reference/api-cli/commands.mdx b/src/content/docs/reference/api-cli/commands.mdx index 78363c0..5ce7ef6 100644 --- a/src/content/docs/reference/api-cli/commands.mdx +++ b/src/content/docs/reference/api-cli/commands.mdx @@ -61,7 +61,7 @@ Scaffold a new agent project. | `--name` | inferred | Agent name, required only when it cannot be inferred | | `--template` | `quick-start` | `quick-start` or `production` | | `--auth` | prompt | Production only: `none`, `jwt` or `custom` | -| `--rate-limit` | prompt | Production only: `none`, `memory` or `redis` | +| `--rate-limit` | prompt | Production only: `none`, `memory` or `redis`. `redis` turns rate limiting on with a Redis cache store (`"redis": "${REDIS_URL}"`); `memory` turns it on without Redis | | `--yes`, `-y` | off | Accept recommended defaults | | `--non-interactive` | off | Fail instead of prompting | | `--dry-run` | off | Preview without writing files | @@ -184,7 +184,7 @@ Read-only environment check: Python version, installed packages, core evaluation ## revoke -Revoke one token or all of a user's tokens. It writes to the same Redis the server reads, so the revocation reaches every worker at once and the server needs no restart. It needs `"token_revocation": "redis"` in `10xgraph.json`. +Revoke one token or all of a user's tokens. It builds the configured cache store the same way the server does (`cache_store`, else `redis`, else `REDIS_URL`) and writes to it, so with a shared store the revocation reaches every worker at once and the server needs no restart. Turn the check on with `"token_revocation": true` in `10xgraph.json`. The command refuses with an error when the cache store is `InMemoryCacheStore`, because a separate process cannot reach another process's memory. ```bash 10xgraph revoke (--user USER_ID | --jti JTI) [--before EPOCH] [--expires-at EPOCH] [--redis-url URL] [--config FILE] @@ -196,10 +196,10 @@ Revoke one token or all of a user's tokens. It writes to the same Redis the serv | `--jti` | unset | Revoke the one token with this `jti` claim | | `--before` | unset | With `--user`: revoke tokens issued at or before this epoch time | | `--expires-at` | unset | With `--jti`: the token's `exp` (epoch seconds), to drop the entry | -| `--redis-url` | `REDIS_URL`, then the `redis` key in `10xgraph.json` | Redis to write to | +| `--redis-url` | unset | Redis URL to write to, in place of the configured cache store | | `--config`, `-c` | `10xgraph.json` | Config file | -Pass exactly one of `--user` and `--jti`. Without a Redis URL from any of the three sources, the command fails. +Pass exactly one of `--user` and `--jti`. With no shared cache store configured (no `cache_store`, `redis` or `REDIS_URL`) and no `--redis-url`, the command fails. ## config diff --git a/src/content/docs/reference/api-cli/configuration.mdx b/src/content/docs/reference/api-cli/configuration.mdx index 134271e..eb184d4 100644 --- a/src/content/docs/reference/api-cli/configuration.mdx +++ b/src/content/docs/reference/api-cli/configuration.mdx @@ -30,7 +30,9 @@ Projects created before 0.7.0 use `agentflow.json`. It still works until 2.0: th | `store` | string or null | none | Import path of a `BaseStore` for long-term memory | | `checkpointer` | string or null | none | Not applied by the API server. If set, the server logs a warning and ignores it. Pass the checkpointer to `compile()` | | `thread_name_generator` | string or null | none | Import path of a thread name generator | -| `redis` | string or null | none | Redis URL for the shared ownership cache. Falls back to `REDIS_URL` | +| `redis` | string or null | none | Redis URL for the API's cache store (run locks and caps, idempotency, ownership cache, token revocation, rate limiting). Falls back to `REDIS_URL`. `$VAR` and `${VAR}` are expanded | +| `cache_store` | string or null | none | `module:attr` of a `BaseCacheStore` instance or zero-argument factory. Wins over `redis` and `REDIS_URL`. See [Bring your own backend](/docs/server/bring-your-own-backend) | +| `token_revocation` | boolean or object | `false` | Refuse revoked tokens. `true`, or `{"enabled": true, "token_lifetime": , "user_ttl": }`. `user_ttl` defaults to 30 days; `null` keeps entries forever. Entries live in the cache store. The old values `"redis"`, `"memory"` and `"module:attr"` were removed in 1.0.0rc4 and stop startup with an error that names `cache_store` | | `remote_tools` | array | `[]` | Schemas of tools that run in the client | | `runs` | object | built-in defaults | Who names threads, which client input reaches the graph, and run limits | | `rate_limit` | object | absent (off) | Request rate limiting | @@ -120,7 +122,7 @@ Set these when a tool or node trusts a value such as `role` or `plan` from state ### max_concurrent_per_user -Caps the HTTP graph runs (invoke, stream and AG-UI) one verified user may have in flight at once. The count is shared by all workers when Redis is configured (`redis` key or `REDIS_URL`) and kept per process otherwise. Over the cap the request gets `429`. `0` or `null` means no cap. Requests without a verified user (auth off) are not capped. The value must be a non-negative integer. +Caps the HTTP graph runs (invoke, stream and AG-UI) one verified user may have in flight at once. The count is shared by all workers when a shared cache store is configured (`redis` key, `REDIS_URL` or `cache_store`) and kept per process otherwise. Over the cap the request gets `429`. `0` or `null` means no cap. Requests without a verified user (auth off) are not capped. The value must be a non-negative integer. ### timeout @@ -137,25 +139,22 @@ Rate limiting applies to REST requests. `/ping` is always excluded. Set `"enable | Key | Type | Default | Purpose | |---|---|---|---| | `enabled` | boolean | `true` | Turn limiting on or off | -| `backend` | string | `memory` | `memory`, `redis` or `custom` (bind a `BaseRateLimitBackend` in InjectQ) | | `requests` | integer | `100` | Requests allowed per window. Must be above 0 | | `window` | integer | `60` | Window length in seconds. Must be above 0 | | `by` | string | `ip` | Count per `ip`, `user` or `global` | | `exclude_paths` | array | `[]` | Extra paths to skip | -| `redis` | object or string | none | `{"url": "...", "prefix": "..."}` or a URL string. `$VAR` and `${VAR}` are expanded | -| `redis.prefix` | string | `10xgraph:rate-limit` | Redis key prefix | -| `fail_open` | boolean | `true` | Allow requests when the backend errors. `false` denies them | +| `fail_open` | boolean | `true` | Allow requests when the cache store errors. `false` denies them with `503` `RATE_LIMIT_UNAVAILABLE` | | `trusted_proxy_headers` | boolean | `false` | Honour `X-Forwarded-For`. Only behind a proxy you control | | `trusted_proxy_hops` | integer | `1` | Proxies you control, counted from the right of `X-Forwarded-For`. At least 1 | | `trusted_proxies` | array | `[]` | IPs or CIDR ranges your proxies connect from. Others cannot spoof the header | -The `memory` backend counts per process, so with N workers the real limit is `requests` times N and it resets on restart. Use `redis` for any multi-worker deployment. +The counters live in the API's cache store. With the in-memory store they count per process, so with N workers the real limit is `requests` times N and it resets on restart. Set `redis` (or `REDIS_URL`) for any multi-worker deployment. The keys `rate_limit.backend` and `rate_limit.redis` were removed in 1.0.0rc4 and stop startup with an error that names `cache_store`. ## websocket, ag_ui and observability | Key | Type | Default | Purpose | |---|---|---|---| -| `websocket.max_connections` | integer or null | `1000` | Concurrent connections. Shared by all workers when Redis is configured, per process otherwise. `0` or `null` means unlimited | +| `websocket.max_connections` | integer or null | `1000` | Concurrent connections. Shared by all workers when a shared cache store is configured, per process otherwise. `0` or `null` means unlimited | | `websocket.max_connections_per_user` | integer or null | `10` | Per verified user. `0` or `null` means unlimited | | `websocket.realtime_models` | array of strings | `[]` | Models a `/v1/graph/live` client may request | | `ag_ui.enabled` | boolean | `false` | Mounts `POST /v1/ag-ui`. Needs the `ag-ui` extra | @@ -214,18 +213,17 @@ The server reads configuration from environment variables, which are listed in f "authorization": "ownership", "injectq": "graph.agent:container", "store": null, - "redis": "redis://localhost:6379/0", + "redis": "${REDIS_URL}", + "token_revocation": true, "thread_name_generator": "graph.thread_name_generator:MyNameGenerator", "rate_limit": { "enabled": true, - "backend": "redis", "requests": 100, "window": 60, "by": "user", "trusted_proxy_headers": true, "trusted_proxy_hops": 1, "exclude_paths": ["/docs"], - "redis": { "url": "${REDIS_URL}", "prefix": "myapp:rate-limit" }, "fail_open": true }, "websocket": { "max_connections": 500, "max_connections_per_user": 5 }, diff --git a/src/content/docs/reference/api-cli/environment.md b/src/content/docs/reference/api-cli/environment.md index 5dacae8..e9e039d 100644 --- a/src/content/docs/reference/api-cli/environment.md +++ b/src/content/docs/reference/api-cli/environment.md @@ -59,14 +59,14 @@ The settings model allows extra variables, so unknown names in the environment a ## Startup check variables -Added after 1.0.0rc2. +Added in 1.0.0rc3. | Variable | Default | Description | | --- | --- | --- | | `STARTUP_CHECKS` | `true` | Before serving, ping every configured backend and refuse to start while one stays unreachable. Applies in development and production. `false` skips the check and logs a warning. | | `STARTUP_CHECK_TIMEOUT_SECONDS` | `30` | Seconds the check keeps retrying an unreachable backend (with a short backoff) before it gives up, so services started together can come up in any order. Must be 0 or more. | -The check covers the checkpointer's Postgres (a query plus the schema version) and its Redis cache, the API's Redis (`redis` in `10xgraph.json`, else `REDIS_URL`), the rate limiter's Redis when it has its own URL, and the memory store through `aping()`. It does not check the model provider. When a backend stays down, the server exits with a message that names each backend and its `host:port`, never credentials. See [Startup check](/docs/server/production-checklist#startup-check) and [Backend unreachable at startup](/docs/troubleshooting/api-server#issue-backend-unreachable-at-startup). +The check calls `aping()` on the checkpointer (Postgres, its schema version and its Redis cache), on the store when one is bound, and on the API's cache store. It does not check the model provider. The checkpointer and the store are fatal when they stay down. The cache store is fatal only when a fail-closed feature uses it (`token_revocation` on, or `rate_limit` on with `fail_open` set to `false`); otherwise the server logs one warning and starts. When the server exits, the message names each backend and its endpoint, never credentials. See [Startup check](/docs/server/production-checklist#startup-check) and [Backend unreachable at startup](/docs/troubleshooting/api-server#issue-backend-unreachable-at-startup). --- @@ -144,17 +144,17 @@ Both must be set when `10xgraph.json` has `"auth": "jwt"`; the config load raise | `REDIS_CONNECT_TIMEOUT` | `5.0` | Seconds to wait when opening a Redis connection. `0` turns it off. | | `REDIS_MAX_CONNECTIONS` | `50` | Pool size of each Redis client the API opens. `0` means no bound. | | `REDIS_POOL_TIMEOUT` | `5.0` | Seconds a request waits for a free pooled connection. `0` means no timeout. | -| `REDIS_KEY_PREFIX` | `""` | Put in front of every key the API writes to Redis (run locks and caps, ownership cache, token revocation), so two deployments sharing one Redis cannot read each other's entries. The core checkpointer has its own `redis_key_prefix` option for its cache keys. To separate a whole deployment, set both to the same value. | +| `REDIS_KEY_PREFIX` | `""` | Put in front of every key the API writes to Redis (run locks and caps, idempotency, ownership cache, token revocation, rate limiting), so two deployments sharing one Redis cannot read each other's entries. The core checkpointer has its own `redis_key_prefix` option for its cache keys. To separate a whole deployment, set both to the same value. | | `REDIS_HEALTH_CHECK_INTERVAL` | `30.0` | Seconds between health checks of an idle connection. `0` turns them off. | -The `REDIS_*` limits apply to every Redis client the API opens: run locks and caps, rate limiting, token revocation, the ownership cache and the readiness probe. +The `REDIS_*` limits apply to the Redis client behind the API's cache store (`RedisCacheStore`). -`REDIS_URL` is **optional everywhere**. Two things use it: +`REDIS_URL` is **optional everywhere**. The API reads it to build its cache store, together with the `redis` and `cache_store` keys in `10xgraph.json`. The first one set wins: `cache_store`, then `redis`, then `REDIS_URL`, then an in-memory store. The server logs one INFO line at boot that names the store and where it came from. -- **The ownership authorization cache (L2).** The `ownership` and `rbac` backends resolve their Redis URL from the `redis` key in `10xgraph.json` first, falling back to `REDIS_URL`. With neither set, or with the `redis` package not installed, the cache runs in-process only (L1) and the server logs a warning at startup. Nothing breaks; each worker just pays its own first lookup per thread. +- **The cache store.** `RedisCacheStore` shares run locks, run and WebSocket caps, idempotency, the ownership cache (L2), token revocation and rate limiting between workers. Without a Redis URL they live in each process (`InMemoryCacheStore`). Valkey, Dragonfly and KeyDB work through the same URL. A leftover `REDIS_URL` that points at nothing no longer blocks boot when nothing you configured needs it to be reachable: the server logs a warning and starts. - **`PgCheckpointer`.** It requires a Redis connection (`redis_url`, `redis_pool` or `redis`) as the hot cache in front of Postgres; constructing it without one raises `ValueError`. It does not read `REDIS_URL` itself, so pass the URL to its constructor. The durable data lives in Postgres. -The rate limiter does **not** read `REDIS_URL`. Configure its connection under `rate_limit.redis.url` in `10xgraph.json`. +The rate limiter has no Redis setting of its own; it uses the cache store. The `rate_limit.redis` key was removed in 1.0.0rc4. --- diff --git a/src/content/docs/reference/api-cli/rate-limiting.md b/src/content/docs/reference/api-cli/rate-limiting.md index c931cf3..425a827 100644 --- a/src/content/docs/reference/api-cli/rate-limiting.md +++ b/src/content/docs/reference/api-cli/rate-limiting.md @@ -1,11 +1,11 @@ --- title: Rate Limiting seoTitle: "rate_limit configuration reference" -description: "Complete reference for the rate_limit block in 10xgraph.json: the backends, limits, and every option for throttling requests to the 10xGraph API server." +description: "Complete reference for the rate_limit block in 10xgraph.json: the cache store, limits, and every option for throttling requests to the 10xGraph API server." section: Reference group: "CLI and configuration" order: 450 -updated: "2026-10-27" +updated: "2026-10-31" --- The `rate_limit` block in `10xgraph.json` turns on a sliding-window limiter for the API server. It counts REST requests, WebSocket handshakes and WebSocket graph runs against one bucket per client. The limiter is off unless the block exists; this page lists every option, header and error. @@ -15,7 +15,6 @@ The `rate_limit` block in `10xgraph.json` turns on a sliding-window limiter for | Field | Type | Default | Description | |---|---|---|---| | `enabled` | boolean | `true` | Enable the rate limiter when the `rate_limit` block exists. Set to `false` to temporarily disable without removing the block. | -| `backend` | string | `"memory"` | Counter storage: `"memory"`, `"redis"`, or `"custom"`. | | `requests` | integer | `100` | Maximum requests allowed per window. Must be positive. | | `window` | integer | `60` | Window size in seconds. Must be positive. | | `by` | string | `"ip"` | Bucket key: `"ip"` (client address), `"user"` (authenticated user ID), or `"global"` (whole service). See [Bucket keys](#bucket-keys). | @@ -23,12 +22,11 @@ The `rate_limit` block in `10xgraph.json` turns on a sliding-window limiter for | `trusted_proxy_headers` | boolean | `false` | Honor the `X-Forwarded-For` header to extract the client IP. Only enable behind a proxy you control. | | `trusted_proxy_hops` | integer | `1` | Number of proxies (counted from the right of `X-Forwarded-For`) that your infrastructure operates. Requires `trusted_proxy_headers: true`. Must be >= 1. | | `trusted_proxies` | array | `[]` | IP addresses or CIDR ranges your proxies connect from. When set, `X-Forwarded-For` is honored only for requests from these networks. | -| `redis` | object or string | `null` | Redis connection for the `"redis"` backend. Either `{"url": "...", "prefix": "..."}` or a bare URL string (uses default prefix). The URL supports `$ENV_VAR` and `${ENV_VAR}` expansion. | -| `redis.url` | string | `null` | Redis connection URL. Required for `"redis"` backend unless a Redis client is bound in InjectQ. | -| `redis.prefix` | string | `"10xgraph:rate-limit"` | Key prefix for all Redis entries. The default changed from `"agentflow:rate-limit"` in version 0.7.0, so counters restart once on upgrade. | -| `fail_open` | boolean | `true` | When `true`, requests are allowed if the Redis backend is unreachable (fail-open). When `false`, requests are denied with `503` `RATE_LIMIT_UNAVAILABLE` (fail-closed). Only applies to `"redis"` backend. | +| `fail_open` | boolean | `true` | When `true`, requests are allowed if the cache store raises an error (fail-open). When `false`, requests are denied with `503` `RATE_LIMIT_UNAVAILABLE` (fail-closed). | -Config load validates all values. Invalid `by`, `backend`, non-positive `requests` or `window`, `trusted_proxy_hops` < 1, invalid networks in `trusted_proxies`, or an unresolved `$ENV_VAR` in the Redis URL raise `ValueError` and prevent the server from starting. A `rate_limit` block with `enabled: false` behaves as if the block were absent. +The counters are kept in the API's cache store, not in this block. Set `redis` in `10xgraph.json` (or `REDIS_URL`) to share them between workers, or `cache_store` for your own store. The keys `backend` and `redis` inside `rate_limit` were removed in 1.0.0rc4: they stop the server with an error that names `cache_store`. The Redis keys moved to the `10xgraph:ratelimit2` namespace, so counters reset once at upgrade. + +Config load validates all values. Invalid `by`, non-positive `requests` or `window`, `trusted_proxy_hops` < 1, invalid networks in `trusted_proxies` raise `ValueError` and prevent the server from starting. A `rate_limit` block with `enabled: false` behaves as if the block were absent. ## Bucket keys @@ -92,7 +90,7 @@ When a client exceeds the limit, the server responds with status `429 Too Many R } ``` -When the Redis backend cannot be reached and `fail_open` is `false`, the server responds with `503 Service Unavailable` and `Retry-After: 5` instead of 429. A real limit hit stays 429. Added after 1.0.0rc2. +When the cache store raises an error and `fail_open` is `false`, the server responds with `503 Service Unavailable` and `Retry-After: 5` instead of 429. A real limit hit stays 429. Added in 1.0.0rc3. ```json { @@ -110,80 +108,33 @@ When the Redis backend cannot be reached and `fail_open` is `false`, the server ## WebSocket handshakes and graph runs -WebSocket handshakes bypass HTTP middleware, so the server re-applies the same check when a client connects to `/v1/graph/ws` or `/v1/graph/live`. A handshake counts like an HTTP request against the same bucket and backend. If the bucket is full, the handshake is rejected with close code `1013` (Try Again Later) and the reason `Rate limit exceeded`. +WebSocket handshakes bypass HTTP middleware, so the server re-applies the same check when a client connects to `/v1/graph/ws` or `/v1/graph/live`. A handshake counts like an HTTP request against the same bucket and cache store. If the bucket is full, the handshake is rejected with close code `1013` (Try Again Later) and the reason `Rate limit exceeded`. Each graph run sent over an open `/v1/graph/ws` connection also counts against the bucket. When the bucket is full, the connection stays open, the run is skipped, and the server sends an `error` stream event whose data holds `reason` (`Rate limit exceeded. Retry after s.`) and `retry_after_seconds`. -The separate `websocket.max_connections` and `websocket.max_connections_per_user` caps (in the `websocket` block of `10xgraph.json`) are enforced by the same handshake guard and also reject with close code `1013`. They are shared by all workers when Redis is configured and per process otherwise. - -## Backend comparison - -| Backend | Use case | -|---|---| -| `memory` | Development, tests, single-process deployments. Counters are in-process memory and reset when the worker restarts. With N workers, the effective limit is `requests x N`. A startup warning is logged. | -| `redis` | Production multi-worker deployments (Gunicorn/Uvicorn), Docker, Kubernetes. Counters are shared across all workers. Respects `fail_open` for backend outages. | -| `custom` | Custom storage backends, external quota services, or non-standard enforcement logic. Implement `BaseRateLimitBackend` and bind an instance in InjectQ. Startup fails with `ValueError` if none is bound. | - -## Custom backend - -Subclass `BaseRateLimitBackend` when neither built-in backend fits, for example to call an external quota service. Both methods are abstract. `check` must atomically record the request and return a `RateLimitDecision` with `allowed`, `remaining` and `reset_after` (seconds). `close` releases resources at shutdown. A backend that cannot reach its store and must not let the request through returns `RateLimitDecision.backend_unavailable()` (optional `retry_after` seconds, default 5); the middleware answers it with the 503 above. Added after 1.0.0rc2. - -```python title="my_backend.py" -import time - -from tenxgraph_api.src.app.core.middleware.rate_limit import ( - BaseRateLimitBackend, - RateLimitDecision, -) +The separate `websocket.max_connections` and `websocket.max_connections_per_user` caps (in the `websocket` block of `10xgraph.json`) are enforced by the same handshake guard and also reject with close code `1013`. They are shared by all workers when a shared cache store is configured and per process otherwise. +## Cache store -class FixedWindowBackend(BaseRateLimitBackend): - """Single-process fixed-window counter, shown as the smallest working example.""" +The limiter takes one slot per request from the API's cache store (`BaseCacheStore.acquire`); the slot is never released and expires after `window` seconds. A denied call adds nothing, so a client over its limit does not extend its own penalty. - def __init__(self) -> None: - self._windows: dict[str, tuple[float, int]] = {} - - async def check(self, key: str, *, limit: int, window: int) -> RateLimitDecision: - now = time.monotonic() - start, count = self._windows.get(key, (now, 0)) - if now - start >= window: - start, count = now, 0 # window expired: start a new one - count += 1 - self._windows[key] = (start, count) - reset_after = max(1, int(start + window - now)) - return RateLimitDecision( - allowed=count <= limit, - remaining=max(0, limit - count), - reset_after=reset_after, - ) - - async def close(self) -> None: - self._windows.clear() -``` - -Set `"backend": "custom"` in `10xgraph.json`, then bind an instance under the `BaseRateLimitBackend` type in the InjectQ container the server uses (see the `injectq` key in the [configuration reference](/docs/reference/api-cli/configuration)). - -```python title="container.py" -from injectq import InjectQ - -from my_backend import FixedWindowBackend -from tenxgraph_api.src.app.core.middleware.rate_limit import BaseRateLimitBackend +| Cache store | Use case | +|---|---| +| `InMemoryCacheStore` | Development, tests, single-process deployments. Counters are in-process memory and reset when the worker restarts. With N workers, the effective limit is `requests x N`. It keeps at most 50,000 slot keys and evicts the least recently used. This is the default when no Redis URL or `cache_store` is set. | +| `RedisCacheStore` | Production multi-worker deployments (Gunicorn/Uvicorn), Docker, Kubernetes. Counters are shared across all workers. Chosen by `redis` in `10xgraph.json` or `REDIS_URL`. Valkey, Dragonfly and KeyDB work through the same URL. | +| Your own | A store that does not speak the Redis protocol. Implement `BaseCacheStore` and set `"cache_store": "module:attr"`. See [Bring your own backend](/docs/server/bring-your-own-backend). | -container = InjectQ.get_instance() -# The server resolves the backend by this type at startup -container.bind_instance(BaseRateLimitBackend, FixedWindowBackend()) -``` +Removed in 1.0.0rc4: `BaseRateLimitBackend`, `MemoryRateLimitBackend`, `RedisRateLimitBackend`, and the `rate_limit.backend` and `rate_limit.redis` keys. A cache store signals an outage by raising; `fail_open` decides what happens next. ## Examples -Minimal configuration with in-memory backend and default limits: +Minimal configuration with the in-memory cache store and default limits: ```json { "agent": "graph.react:app", "rate_limit": { "enabled": true, - "backend": "memory", "requests": 100, "window": 60, "by": "ip", @@ -192,14 +143,14 @@ Minimal configuration with in-memory backend and default limits: } ``` -Production setup with Redis, user-based limiting, trusted proxies: +Production setup with a Redis cache store, user-based limiting, trusted proxies: ```json { "agent": "graph.react:app", + "redis": "${REDIS_URL}", "rate_limit": { "enabled": true, - "backend": "redis", "requests": 1000, "window": 60, "by": "user", @@ -207,10 +158,6 @@ Production setup with Redis, user-based limiting, trusted proxies: "trusted_proxy_hops": 1, "trusted_proxies": ["10.0.0.0/8"], "exclude_paths": ["/ping", "/docs", "/redocs"], - "redis": { - "url": "${RATE_LIMIT_REDIS_URL}", - "prefix": "10xgraph:rate-limit" - }, "fail_open": false } } @@ -219,10 +166,10 @@ Production setup with Redis, user-based limiting, trusted proxies: With the `.env` file: ```bash -RATE_LIMIT_REDIS_URL=redis://localhost:6379/0 +REDIS_URL=redis://localhost:6379/0 ``` -The `${RATE_LIMIT_REDIS_URL}` value is expanded from the environment when the config loads. To use the Redis backend, install the optional dependency: +The `${REDIS_URL}` value is expanded from the environment when the config loads. To use Redis, install the optional dependency: ```bash pip install "10xgraph-api[redis]" diff --git a/src/content/docs/reference/client/errors.md b/src/content/docs/reference/client/errors.md index 050afdb..2a5332c 100644 --- a/src/content/docs/reference/client/errors.md +++ b/src/content/docs/reference/client/errors.md @@ -179,9 +179,9 @@ try { ### Rate limiter backend down (503) -Added after 1.0.0rc2. +Added in 1.0.0rc3. -When the server's rate-limit backend is unreachable and `fail_open` is `false`, it answers 503 with `error.code` `RATE_LIMIT_UNAVAILABLE` and `Retry-After: 5`, not 429. The client throws a `ServerError` with `statusCode` 503. Retry after a short delay, as for 429. +When the server's cache store is unreachable and `fail_open` is `false`, it answers 503 with `error.code` `RATE_LIMIT_UNAVAILABLE` and `Retry-After: 5`, not 429. The client throws a `ServerError` with `statusCode` 503. Retry after a short delay, as for 429. ### ServerError diff --git a/src/content/docs/reference/error-codes.md b/src/content/docs/reference/error-codes.md index 9e4db8e..7e29917 100644 --- a/src/content/docs/reference/error-codes.md +++ b/src/content/docs/reference/error-codes.md @@ -347,7 +347,7 @@ The server maps exceptions to responses in `tenxgraph_api/src/app/core/exception | `StateFormatVersionError` | 500 (message says to upgrade) | | `ResumeValidationError` | 422 | -`RATE_LIMIT_UNAVAILABLE` is a `503` from the rate limiter, not a core exception: the `rate_limit` backend could not be reached and `fail_open` is `false`. It carries `Retry-After: 5`. The code `RATE_LIMIT_EXCEEDED` (429) stays for a real limit hit. Added after 1.0.0rc2. +`RATE_LIMIT_UNAVAILABLE` is a `503` from the rate limiter, not a core exception: the cache store behind the rate limiter raised an error and `fail_open` is `false`. It carries `Retry-After: 5`. The code `RATE_LIMIT_EXCEEDED` (429) stays for a real limit hit. Added in 1.0.0rc3. A `429` is not a core exception either. The server returns it when the `rate_limit` window is used up (`error.code` `RATE_LIMIT_EXCEEDED`, with a `Retry-After` header) or when one user has more than `runs.max_concurrent_per_user` runs in flight (standard body, code `HTTPException`, no `Retry-After`). diff --git a/src/content/docs/reference/python/checkpointers.md b/src/content/docs/reference/python/checkpointers.md index c7e352f..11d4fa2 100644 --- a/src/content/docs/reference/python/checkpointers.md +++ b/src/content/docs/reference/python/checkpointers.md @@ -51,10 +51,15 @@ The methods below are abstract and must be implemented by a subclass: | `aget_thread` | `async (config) -> ThreadInfo \| None` | Get thread metadata. | | `alist_threads` | `async (config, search=None, offset=None, limit=None) -> list[ThreadInfo]` | List threads. | | `aclean_thread` | `async (config) -> Any \| None` | Delete a thread and its state and messages. | +| `aping` | `async () -> None` | Return when the checkpointer's backends are reachable and usable, raise when they are not. Must be read-only (no writes, no table creation) and cheap, because the API server calls it on every readiness probe and at startup. An in-process checkpointer just returns. Abstract since 1.0.0rc4: a subclass without it raises `TypeError` when created. | | `arelease` | `async () -> Any \| None` | Release resources (close pools and connections). | `alist_messages` and `alist_threads` apply `offset` and `limit` in the backend (in SQL on SQLite and Postgres). A negative `offset` or `limit` raises `ValueError`, and a `limit` above `MAX_LIST_LIMIT` (1000, in `tenxgraph.storage.checkpointer.base_checkpointer`) is lowered to it. With no `limit` (`None` or `0`) the whole list is returned, so code that reads a full thread, such as realtime history loading, still works. +### `aping` + +`aping()` is the connectivity check. The API server calls it at startup and on `/health/ready`, so it must be read-only and cheap. `InMemoryCheckpointer.aping()` just returns, `SqliteCheckpointer.aping()` runs `SELECT 1`, and `PgCheckpointer.aping()` runs `SELECT 1`, compares the recorded schema version with the one the code expects (a `RuntimeError` that names both when it is older), and pings its Redis cache directly, so an outage is reported and the circuit breaker stays unchanged. It never creates the schema: the server runs `asetup()` at boot. Added in 1.0.0rc4. + ### Sync wrappers For every `async axxx()` method there is a sync `xxx()` wrapper that runs the async method to completion. Use these only from non-async contexts such as a management script: @@ -286,6 +291,9 @@ class DynamoDBCheckpointer(BaseCheckpointer): async def aclear_state(self, config: dict) -> Any: ... + async def aping(self) -> None: + ... # raise if DynamoDB cannot be reached; read-only and cheap + async def aput_state_cache(self, config: dict, state: AgentState) -> Any: ... # in-memory dict, no external call diff --git a/src/content/docs/reference/python/memory-stores.md b/src/content/docs/reference/python/memory-stores.md index 4e2ee06..965efc8 100644 --- a/src/content/docs/reference/python/memory-stores.md +++ b/src/content/docs/reference/python/memory-stores.md @@ -373,12 +373,12 @@ Abstract base class that every store backend implements. Every method takes a `c | `adelete` | `async (config, memory_id, **kwargs) -> Any` | Delete a memory by ID. | | `aforget_memory` | `async (config, **kwargs) -> Any` | Delete memories for the `user_id` in `config`. `thread_id` is an optional extra filter. `QdrantStore` also narrows by the keyword arguments `memory_type`, `category` and `filters` (combined with AND; a `filters` value that is not a string, integer or boolean raises `ValueError`). `Mem0Store` can delete only by user, so it raises `ValueError` when any of the three is passed. | | `asetup` | `async () -> Any` | Create the collection or other backing resources. Implemented by `QdrantStore`; the base class raises `NotImplementedError`. | -| `aping` | `async () -> None` | Abstract. Return when the store's backend is reachable, raise when it is not. Must be read-only and cheap, because a server may call it on every readiness probe. An in-process store just returns. `10xgraph-api` calls it for `/health/ready` and at startup. A custom store that does not implement it raises `TypeError` when created. Added after 1.0.0rc2. | +| `aping` | `async () -> None` | Abstract. Return when the store's backend is reachable, raise when it is not. Must be read-only and cheap, because a server may call it on every readiness probe. An in-process store just returns. `10xgraph-api` calls it for `/health/ready` and at startup. A custom store that does not implement it raises `TypeError` when created. Added in 1.0.0rc3. | | `arelease` | `async () -> None` | Release connections. Not abstract, but the base implementation raises `NotImplementedError`; `Mem0Store` overrides it. | There is no bulk `abatch_store`; call `astore` in a loop or with `asyncio.gather`. -`QdrantStore.aping()` lists the collections on the server, `Mem0Store.aping()` lists the collections of its vector store, and `InMemoryStore.aping()` returns at once. Added after 1.0.0rc2. +`QdrantStore.aping()` lists the collections on the server, `Mem0Store.aping()` lists the collections of its vector store, and `InMemoryStore.aping()` returns at once. Added in 1.0.0rc3. ### Sync wrappers diff --git a/src/content/docs/reference/rest-api/conventions.md b/src/content/docs/reference/rest-api/conventions.md index 60477b9..194d44f 100644 --- a/src/content/docs/reference/rest-api/conventions.md +++ b/src/content/docs/reference/rest-api/conventions.md @@ -157,4 +157,4 @@ These are the status codes the server returns. Routes document their own specifi | `429` | Rate limit exceeded. Body has `error.code` `RATE_LIMIT_EXCEEDED`; the `Retry-After` and `X-RateLimit-*` headers say when to retry. The same status is returned when a user has more than `runs.max_concurrent_per_user` runs in flight (default 10); that body has code `HTTPException` and `Retry-After: 5`. The model provider rate limiting the server also returns `429`, with code `PROVIDER_RATE_LIMITED` and a `Retry-After` taken from the provider (5 seconds if it sends none) | | `499` | The client closed the request while it waited for a busy thread. The response is never read | | `500` | Unexpected server error. Body has code `INTERNAL_SERVER_ERROR`; in production the message is generic | -| `503` | Checkpointer or storage temporarily unavailable (code `SERVICE_UNAVAILABLE` for a database or cache driver error), with `Retry-After`. Readiness answers 503 while the server is shutting down. A rate limiter whose backend is unreachable with `fail_open: false` also answers 503, with code `RATE_LIMIT_UNAVAILABLE` and `Retry-After: 5` (added after 1.0.0rc2) | +| `503` | Checkpointer or storage temporarily unavailable (code `SERVICE_UNAVAILABLE` for a database or cache driver error), with `Retry-After`. Readiness answers 503 while the server is shutting down. A rate limiter whose backend is unreachable with `fail_open: false` also answers 503, with code `RATE_LIMIT_UNAVAILABLE` and `Retry-After: 5` (added in 1.0.0rc3) | diff --git a/src/content/docs/reference/rest-api/live.md b/src/content/docs/reference/rest-api/live.md index 4feee95..abfd645 100644 --- a/src/content/docs/reference/rest-api/live.md +++ b/src/content/docs/reference/rest-api/live.md @@ -41,7 +41,7 @@ The older `agentflow-bearer` sentinel is still accepted until 2.0. An authentication or authorization failure closes the handshake with code `1008` before `accept()`. -The token is not trusted for the whole session. The server checks it again after the init frame and then every 30 seconds. Each check covers expiry and revocation (`token_revocation`), and it fails closed: a revocation backend that cannot answer counts as revoked. When a check fails, the server sends a fatal `session_expired` error event and closes the socket with code `1008`. The client must reconnect with a fresh token. +The token is not trusted for the whole session. The server checks it again after the init frame and then every 30 seconds. Each check covers expiry and revocation (`token_revocation`), and it fails closed: a cache store that cannot answer the revocation check counts as revoked. When a check fails, the server sends a fatal `session_expired` error event and closes the socket with code `1008`. The client must reconnect with a fresh token. --- diff --git a/src/content/docs/reference/rest-api/ping.md b/src/content/docs/reference/rest-api/ping.md index bd2461a..161af52 100644 --- a/src/content/docs/reference/rest-api/ping.md +++ b/src/content/docs/reference/rest-api/ping.md @@ -44,11 +44,13 @@ The envelope comes from the server's `success_response` helper. `data` holds the ## GET /health/ready -The readiness probe. `/ping` says the process is alive; `/health/ready` says the server can do its job. It checks, concurrently and each with a 2 second limit: +The readiness probe. `/ping` says the process is alive; `/health/ready` says the server can do its job. It calls `aping()` on each component, concurrently and each with a 2 second limit: -- **checkpointer**: the component's own `readiness_check()` if it has one, otherwise a `SELECT 1` on the PgCheckpointer pool plus a ping of its Redis cache, or a read-only owner lookup for other checkpointers. -- **redis**: a ping of the configured Redis (`redis` in `10xgraph.json` or `REDIS_URL`). -- **store**: the store's own `readiness_check()`, or its `aping()` / `ping()` method. +- **checkpointer**: `aping()` of the checkpointer. For `PgCheckpointer` that is a `SELECT 1`, a check that the schema version is not older than the code expects, and a ping of its Redis cache. It never creates the schema. +- **store**: `aping()` of the store, when one is bound. +- **cache**: `aping()` of the API's cache store (Redis, or your own `BaseCacheStore`). It is reported always, but it fails readiness only when a fail-closed feature uses it: `token_revocation` is on, or `rate_limit` is on with `fail_open` set to `false`. For everything else (locks, caps, idempotency, the ownership cache) a cache store outage is reported and the server keeps taking traffic. + +The key `cache` replaced `redis` in 1.0.0rc4. A component that is not configured, or has nothing to check, is left out of the response. The endpoint is public, so it returns only `ok` or `fail` per component and logs the reason on the server. Results are cached for 3 seconds and concurrent callers share one check, so a flood of probes does not become a flood of database round trips. @@ -57,16 +59,16 @@ curl -i http://127.0.0.1:8000/health/ready ``` ```json title="Response (200)" -{ "status": "ready", "checks": { "checkpointer": "ok", "redis": "ok" } } +{ "status": "ready", "checks": { "checkpointer": "ok", "cache": "ok" } } ``` | Status | Meaning | | --- | --- | -| `200` | Every configured component is reachable. `status` is `ready`. | -| `503` | At least one check is `fail`. `status` is `not_ready`. The checkpointer check fails when the database schema version is below what the code expects (an upgrade whose migration has not run). | +| `200` | Every component is reachable (or only the cache store is down and no fail-closed feature uses it). `status` is `ready`. | +| `503` | A required check is `fail`. `status` is `not_ready`. The checkpointer check fails when the database schema version is below what the code expects (an upgrade whose migration has not run). | | `503` | The server is shutting down. `status` is `draining` and `checks` is empty. No component is checked. | -The server runs the same checks once at boot (the [startup check](/docs/server/production-checklist#startup-check)). There an unreachable backend stops the server from starting; here it only turns readiness to 503, so traffic stops until the backend returns. Added after 1.0.0rc2. +The server runs the same checks once at boot (the [startup check](/docs/server/production-checklist#startup-check)). There an unreachable checkpointer or store stops the server from starting (so does the cache store, when a fail-closed feature uses it); here it only turns readiness to 503, so traffic stops until the backend returns. Added in 1.0.0rc3. Once the process gets SIGTERM, the server marks itself as draining. Readiness answers `503` from then on, so the load balancer stops sending new requests while open runs finish. `/ping` stays `200` until the process exits. @@ -81,7 +83,7 @@ The response is a plain JSON object, not the standard `data` and `metadata` enve Three pieces of middleware also skip `/ping` and `/health/ready` on purpose, because orchestrator probes would otherwise restart healthy containers: - **Trusted host check.** Probes call `localhost` or the pod IP, which are not in `ALLOWED_HOST`. The host check is skipped for these two probe paths only. -- **Rate limiting.** `/ping` and `/health/ready` are always excluded, so a rate-limit backend outage (for example Redis) does not fail the probe. The production template also lists it in `rate_limit.exclude_paths`. +- **Rate limiting.** `/ping` and `/health/ready` are always excluded, so a cache store outage (for example Redis) does not fail the probe through the rate limiter. The production template also lists it in `rate_limit.exclude_paths`. - **Authentication.** No token is needed. --- diff --git a/src/content/docs/server/ag-ui.md b/src/content/docs/server/ag-ui.md index cde15a7..3462c69 100644 --- a/src/content/docs/server/ag-ui.md +++ b/src/content/docs/server/ag-ui.md @@ -141,19 +141,15 @@ With `allow_client_tools: false`, only tools declared in `remote_tools` are offe ### Rate limiting -The AG-UI endpoint counts against the global `rate_limit` configuration like any other route. In production with multiple replicas, use the Redis backend: +The AG-UI endpoint counts against the global `rate_limit` configuration like any other route. In production with multiple replicas, give the API a Redis cache store so the counters are shared: ```json { + "redis": "${REDIS_URL}", "rate_limit": { "enabled": true, - "backend": "redis", "requests": 100, - "window": 60, - "redis": { - "url": "redis://localhost:6379/0", - "prefix": "10xgraph:rate-limit" - } + "window": 60 } } ``` diff --git a/src/content/docs/server/auth.mdx b/src/content/docs/server/auth.mdx index bb38a37..66f8c76 100644 --- a/src/content/docs/server/auth.mdx +++ b/src/content/docs/server/auth.mdx @@ -4,7 +4,7 @@ description: "Enable JWT or custom auth, enforce thread ownership, and protect y section: "API server" group: "Security" order: 60 -updated: "2026-10-27" +updated: "2026-10-31" faq: - q: "Which claims must my JWT contain?" a: "Every token needs `exp` (expiration) and `user_id` claims. If you set JWT_ISSUER or JWT_AUDIENCE, the token must also carry matching `iss` or `aud` claims. A token without `user_id` is rejected with status 401, code `INVALID_TOKEN`." @@ -141,7 +141,7 @@ If you leave `authorization` unset, the default depends on `MODE`: ownership whe A custom `AuthorizationBackend` also isolates users by default: its `isolation_scope()` returns `"owner"`, so each user sees only their own threads and memories. A backend that wants shared data returns `"none"`. The built-in `allow_all` shares data, and `ownership` and `rbac` isolate it. -Ownership is looked up through the checkpointer's `aget_thread_owner` method. The server uses the checkpointer you pass to `graph.compile(checkpointer=...)`; the `checkpointer` key in `10xgraph.json` is not read. That checkpointer must be able to name thread owners: `PgCheckpointer`, `InMemoryCheckpointer` and the SQLite checkpointer do, and a custom `BaseCheckpointer` must implement `aget_thread_owner`. With auth on, the server refuses to boot if there is no checkpointer or its `aget_thread_owner` is not implemented, and at request time ownership fails closed (thread requests are denied). Use a durable checkpointer such as `PgCheckpointer` so owners survive restarts. For high-concurrency workloads, set `redis` in `10xgraph.json` (or `REDIS_URL` in your environment) to cache ownership lookups across workers: +Ownership is looked up through the checkpointer's `aget_thread_owner` method. The server uses the checkpointer you pass to `graph.compile(checkpointer=...)`; the `checkpointer` key in `10xgraph.json` is not read. That checkpointer must be able to name thread owners: `PgCheckpointer`, `InMemoryCheckpointer` and the SQLite checkpointer do, and a custom `BaseCheckpointer` must implement `aget_thread_owner` and `aping()` (a read-only, cheap check that the backend answers; the server calls it for the startup check and `/health/ready`). With auth on, the server refuses to boot if there is no checkpointer or its `aget_thread_owner` is not implemented, and at request time ownership fails closed (thread requests are denied). Use a durable checkpointer such as `PgCheckpointer` so owners survive restarts. For high-concurrency workloads, set `redis` in `10xgraph.json` (or `REDIS_URL` in your environment) to cache ownership lookups across workers (the API's cache store holds the shared tier; see [Bring your own backend](/docs/server/bring-your-own-backend)): ```jsonc title="10xgraph.json" { diff --git a/src/content/docs/server/bring-your-own-backend.md b/src/content/docs/server/bring-your-own-backend.md new file mode 100644 index 0000000..a77b4b6 --- /dev/null +++ b/src/content/docs/server/bring-your-own-backend.md @@ -0,0 +1,240 @@ +--- +title: Bring your own backend +seoTitle: "Custom checkpointer, store and cache store" +description: "The three storage classes behind a 10xGraph server, what each holds, how to plug in a custom one, the aping rules and the BaseCacheStore contract." +section: "API server" +group: "Operate" +order: 175 +label: Bring your own backend +updated: "2026-10-31" +faq: + - question: "Do I need a custom cache store to use Valkey, Dragonfly or KeyDB?" + answer: "No. They speak the Redis protocol, so point `redis` in 10xgraph.json (or REDIS_URL) at them and RedisCacheStore works. Write a BaseCacheStore only for a store that does not speak the Redis protocol." + - question: "Which Redis eviction policy is safe?" + answer: "noeviction or a volatile-* policy. Any other policy can drop a lock or a revocation before its TTL, which lets two workers run the same thread." + - question: "Can the server start if the cache store is down?" + answer: "Yes, unless a fail-closed feature uses it: token_revocation is on, or rate_limit is on with fail_open set to false. Otherwise the server logs one warning and starts, with locks and caps per process." +--- + +A 10xGraph server keeps its state in three places, and each one is an abstract class you can replace. This page lists them, shows how to select a custom one, and documents the rules a replacement must keep. The default for each is enough for most deployments; read on when your infrastructure is different, for example a database the built-in classes do not cover. + +Valkey, Dragonfly and KeyDB speak the Redis protocol, so you do not need any of this for them: point the Redis URL at the server and the built-in classes work. + +## The three classes + +| Class | Package | Default | Holds | +|---|---|---|---| +| `BaseCheckpointer` | `10xgraph` | `PgCheckpointer` (Postgres plus its own Redis hot cache) | Threads, messages and state | +| `BaseStore` | `10xgraph` | `QdrantStore` or `Mem0Store` | Long-term memory | +| `BaseCacheStore` | `10xgraph-api` | `RedisCacheStore`; `InMemoryCacheStore` when no Redis is set | Everything the API shares between workers | + +Each of the three has an abstract `aping()`. The server calls it at startup and on `/health/ready`, so a replacement tells the server whether it is usable in the same way as a built-in. + +### What the cache store backs + +| Feature | What it keeps | If the store raises an error | +|---|---|---| +| Per-thread run lock | A timed slot, limit 1 | Fails open: the run proceeds and the error is logged | +| Run cap (`runs.max_concurrent_per_user`) | Timed slots | Fails open | +| WebSocket caps (`websocket.max_connections`, `max_connections_per_user`) | Timed slots | Fails open | +| Idempotency | The running lease and the stored records | Fails open | +| Ownership cache (L2) | Keys with a TTL | Fails open: lookups fall back to the checkpointer | +| Token revocation | Keys with a TTL; a per-user cutoff that only moves forward | Fails closed: the request gets `503` | +| Rate limiting | Timed slots that are never released | Follows `rate_limit.fail_open` (default `true`); `false` answers `503` `RATE_LIMIT_UNAVAILABLE` with `Retry-After: 5` | + +## Select a custom backend + +The checkpointer, store and media store are passed to `compile()` in your graph module. The server does not read the `checkpointer` or `store` keys to build them for you (see [Configure the server](/docs/server/configure)): + +```python title="graph/agent.py" +app = graph.compile(checkpointer=MyCheckpointer(), store=MyStore()) +``` + +The cache store is the one backend chosen in `10xgraph.json`. The first of these that is set wins: + +1. `"cache_store": "module:attr"`: a `BaseCacheStore` instance, or a zero-argument factory (a class or function) that returns one. +2. `"redis": "redis://..."`: `RedisCacheStore` on that URL. The value supports `${VAR}`. +3. `REDIS_URL` in the environment: the same. +4. Otherwise `InMemoryCacheStore`. + +```json title="10xgraph.json" +{ + "agent": "graph.agent:app", + "cache_store": "graph.cache:store" +} +``` + +```python title="graph/cache.py" +from my_company.cache import MyCacheStore + +store = MyCacheStore("https://cache.internal") +``` + +The server logs one INFO line at boot that names the store and where the choice came from, for example `Cache store: RedisCacheStore (from REDIS_URL, localhost:6379)`. A store can add its own endpoint to that line by implementing `describe()`, which returns a string such as `host:port` and never credentials. + +## The `aping()` rules + +All three classes use the same pattern: + +- Return when the backend is reachable and usable. Raise any exception when it is not. +- Be read-only and cheap. The server may call it on every readiness probe, which is every few seconds. Do not write data, create tables or take locks. +- An in-process implementation just returns. + +`PgCheckpointer.aping()` is the reference: it runs `SELECT 1`, raises `RuntimeError` when the recorded schema version is older than the code expects, and pings its own Redis. It never creates the schema; the server runs `asetup()` at boot. + +`aping()` is abstract on `BaseCheckpointer` (since 1.0.0rc4), `BaseStore` (since 1.0.0rc3) and `BaseCacheStore`. A subclass without it raises `TypeError` when created. + +What a failed ping does: + +| Component | Fatal at startup | Fails `/health/ready` | +|---|---|---| +| Checkpointer | Yes | Yes | +| Store (when one is bound) | Yes | Yes | +| Cache store | Only when a fail-closed feature uses it: `token_revocation` is on, or `rate_limit` is on with `fail_open` set to `false` | Under the same rule | + +When the cache store is down and no fail-closed feature uses it, the server logs one WARNING and starts, with locks and caps per process. The readiness response then reports `"cache": "fail"` and still answers `200`. See [Ping and health](/docs/reference/rest-api/ping#get-healthready) and the [startup check](/docs/server/production-checklist#startup-check). + +## The `BaseCacheStore` contract + +Two primitives cover every feature above. **Timed slots** (`acquire`, `renew`, `release`) are a set of expiring slots per key, capped at `limit`. A lock is a slot set with `limit=1`; a rate limit is a slot set whose slots are never released and expire after the window. **Keys with a TTL** (`get`, `set`, `delete`) hold values such as idempotency records, ownership entries and revocations. + +| Method | Meaning | +|---|---| +| `aping() -> None` | Return when the store answers; raise otherwise. | +| `now() -> float` | The store's clock, in seconds since the epoch. | +| `acquire(key, limit, ttl) -> SlotResult` | Take one of `limit` slots for `key`, expiring `ttl` seconds from now. | +| `renew(key, slot_id, ttl) -> bool` | Move a held slot's expiry to `now() + ttl`. Returns `False` when the slot is gone. | +| `release(key, slot_id) -> None` | Remove a slot. No error when it is already gone. | +| `get(key) -> bytes \| None` | The value under `key`, or `None`. | +| `set(key, value, ttl, mode="always") -> bool` | Store a value. `mode` is `"always"`, `"if_absent"` or `"if_greater"`. Returns whether it wrote. | +| `delete(key) -> None` | Remove `key`. No error when it does not exist. | +| `describe() -> str \| None` | Optional. Where the store lives, without credentials, for logs and startup messages. | +| `aclose() -> None` | Optional. Release resources; called once on shutdown. | + +`acquire` returns `SlotResult(granted, count, oldest_expiry, slot_id)`: whether a slot was granted, the live slots after the call (including yours when granted), the expiry of the oldest live slot on the store's clock (or `None`), and the new slot's id when granted (or `None`). + +An implementation must keep these rules. The contract test suite checks them. + +- **`acquire` is atomic.** Drop expired slots, and only if the live count is below `limit`, add a new slot expiring at `now() + ttl`. Two callers can never both take the last slot. `limit < 1` always denies. +- **A denied `acquire` adds nothing.** The rate limiter relies on this: a client over its limit does not extend its own penalty. +- **`renew` never re-creates a slot.** It returns `False` when the slot expired or was released. +- **`release` of a missing slot is not an error.** +- **`set(mode="if_absent")`** writes only when the key does not exist. +- **`set(mode="if_greater")`** treats the value as a decimal number and writes only when it is greater than the stored one, or nothing is stored. It is atomic. Token revocation uses it so an older cutoff never overwrites a newer one. A value that is not a decimal number raises `ValueError`. +- **`ttl` is in seconds and must be positive.** `set(ttl=None)` keeps the key until it is deleted. A write replaces the TTL of the key it writes. +- **TTLs are never dropped early.** A key or slot must not disappear before its TTL. A store that evicts data under memory pressure breaks locks: two workers run the same thread. For Redis this means `maxmemory-policy` must be `noeviction` or a `volatile-*` policy. Any other policy is a deployment mistake. +- **Every method touches exactly one key.** That keeps the store safe on Redis Cluster and on sharded stores. +- **Raise on errors.** When the backend fails, raise any exception. The features decide whether to fail open or closed (see the table above). The store never swallows errors. +- **Use the store's clock.** All times come from `now()`, so pods with skewed clocks agree on when a slot expires. + +Keys arrive as `":"`, for example `concurrency:runs:thread:42`. The store may map the feature segment to its own namespace and add a deployment prefix, but two different keys must never land on the same entry, and any string (with `:`, spaces or unicode) is a valid key. A key holds either slots or a value, never both. + +### Built-in stores + +`RedisCacheStore` uses Lua scripts on sorted sets for the slots, `SET` with `PX` for values, and a small script for `if_greater`. It applies `REDIS_KEY_PREFIX` and keeps the Redis key layouts of earlier releases, except for the rate limiter, which moved to the `10xgraph:ratelimit2` namespace in 1.0.0rc4 (counters reset once on upgrade). Its connection limits come from the `REDIS_*` settings; see the [environment reference](/docs/reference/api-cli/environment). + +`InMemoryCacheStore` keeps everything in one process. It is correct for a single worker and wrong for several: locks, caps and revocations do not cross processes. To bound memory it caps slot keys at 50,000 and plain values at 10,000 and evicts the least recently used entry at the cap, with a warning. That is its one exception to "TTLs are never dropped early". It never evicts lease keys (`concurrency:`) or revocation keys (`revoked:`) while they are live, so a lock or a revoked token cannot disappear. + +## A minimal skeleton + +This skeleton shows the shape. The comments say what each method must do on your backend. + +```python title="my_company/cache.py" +from typing import Literal + +from tenxgraph_api import BaseCacheStore, SlotResult + + +class MyCacheStore(BaseCacheStore): + def __init__(self, url: str) -> None: + self._url = url + self._client = ... # your async client + + async def aping(self) -> None: + # Read-only and cheap: one round trip. Raise if the backend does not answer. + ... + + async def now(self) -> float: + # The backend's own clock, not time.time() on this pod. + ... + + async def acquire(self, key: str, limit: int, ttl: float) -> SlotResult: + # One atomic step: drop expired slots, count live ones, add a slot only if + # count < limit. Never add on denial. Return the count after the call, the + # oldest live expiry on the store's clock, and the new slot id when granted. + ... + + async def renew(self, key: str, slot_id: str, ttl: float) -> bool: + # Move the expiry of a slot that still exists. Return False if it is gone. + ... + + async def release(self, key: str, slot_id: str) -> None: + # Remove the slot; a missing slot is fine. + ... + + async def get(self, key: str) -> bytes | None: + ... + + async def set( + self, + key: str, + value: bytes, + ttl: float | None, + mode: Literal["always", "if_absent", "if_greater"] = "always", + ) -> bool: + # "if_absent": write only if missing. "if_greater": atomic compare of two + # decimal numbers. Return True when you wrote. + ... + + async def delete(self, key: str) -> None: + ... + + def describe(self) -> str | None: + return "cache.internal:443" # no credentials + + async def aclose(self) -> None: + ... +``` + +Select it with `"cache_store": "my_company.cache:store"` as shown above. `store` can be an instance or a function that takes no arguments and returns one. + +## Test it with the contract suite + +`tenxgraph_api.testing.cache_store_contract` ships the checks the built-in stores pass: `aping`, the slot limit, 50 concurrent `acquire` calls granting exactly the limit, `release` and `renew`, slot expiry, `get`, `set` and `delete` with all three modes (including 50 concurrent `if_greater` writes leaving the maximum), the clock, and keys with odd characters. Each check opens its own store from your factory, works under its own keys and closes the store, so it is safe to run against a shared server. + +As a pytest mixin, subclass `CacheStoreContract` and implement `make_store()`: + +```python title="tests/test_my_cache.py" +from tenxgraph_api.testing.cache_store_contract import CacheStoreContract + +from my_company.cache import MyCacheStore + + +class TestMyCacheStore(CacheStoreContract): + def make_store(self): + return MyCacheStore("https://cache.internal") +``` + +Or call it from any async code: + +```python +from tenxgraph_api.testing.cache_store_contract import run_cache_store_contract + +await run_cache_store_contract(lambda: MyCacheStore("https://cache.internal")) +``` + +`make_store()` and the factory can return a store or an awaitable of one. A failed check raises `AssertionError`. The timing checks use TTLs of a few hundred milliseconds, so the store must answer well within that. + +## Custom checkpointers and stores + +A custom `BaseCheckpointer` implements every abstract async method, including `aping()`, and `aget_thread_owner` when auth is on. A custom `BaseStore` implements its methods and `aping()`. Both are passed to `compile()`. The method lists are in the [checkpointer reference](/docs/reference/python/checkpointers#writing-a-custom-checkpointer) and the [store reference](/docs/reference/python/memory-stores), and [Extension points](/docs/concepts/extensibility) shows where every extension class is wired. + +## What changed in 1.0.0rc4 + +`BaseCacheStore` replaced `ConcurrencyLimiter`, `TokenRevocationBackend` and `BaseRateLimitBackend`. Those classes, `"token_revocation": "redis" | "memory" | "module:attr"` and the `rate_limit.backend` and `rate_limit.redis` keys are gone, with no aliases. See [Upgrade to 1.0](/docs/project/upgrade-to-1.0#from-100rc3-to-100rc4). + +## See also + +- [Production checklist](/docs/server/production-checklist) for the startup check and the Redis settings +- [Rate limiting](/docs/server/rate-limiting) for the limiter that uses the cache store +- [Scaling and reliability](/docs/production/scaling) for what the cache store shares across workers diff --git a/src/content/docs/server/cli.md b/src/content/docs/server/cli.md index b444428..12e1a99 100644 --- a/src/content/docs/server/cli.md +++ b/src/content/docs/server/cli.md @@ -163,9 +163,9 @@ Output example: ``` 10xgraph-api - Version: 1.0.0rc2 + Version: 1.0.0rc4 10xgraph (core) - Version: 1.0.0rc2 + Version: 1.0.0rc4 ``` ### Preview animations diff --git a/src/content/docs/server/configure.md b/src/content/docs/server/configure.md index 25d3c20..c19c26e 100644 --- a/src/content/docs/server/configure.md +++ b/src/content/docs/server/configure.md @@ -108,7 +108,7 @@ container.bind_instance(MyDatabase, MyDatabase(dsn=os.environ["DATABASE_URL"])) } ``` -The server loads your container at startup and makes it the global instance. Your services land in the same container as the server's internal bindings, so they integrate seamlessly. This is also where you bind a custom `BaseRateLimitBackend` if you use `"backend": "custom"` for rate limiting. +The server loads your container at startup and makes it the global instance. Your services land in the same container as the server's internal bindings, so they integrate seamlessly. ## Generate readable thread names @@ -150,34 +150,38 @@ A minimally secured production config looks like this: "agent": "graph.react:app", "auth": "jwt", "authorization": "ownership", + "redis": "${REDIS_URL}", + "token_revocation": true, "rate_limit": { "enabled": true, - "backend": "redis", "requests": 100, "window": 60, - "by": "user", - "redis": { "url": "${REDIS_URL}" } + "by": "user" } } ``` - `"auth": "jwt"` requires `JWT_SECRET_KEY` (at least 32 characters; shorter keys are rejected in production) and `JWT_ALGORITHM` (for example `HS256`) in the environment. See [Authentication](/docs/server/auth) for custom auth. - `"authorization": "ownership"` makes threads private to their creator. With the default `runs.thread_ids` of `"server"`, it also refuses thread ids the server did not create (see [`runs.thread_ids`](/docs/reference/api-cli/configuration#thread_ids)). With `allow_all`, any authenticated user who knows a thread ID can read it. -- The rate limit `backend: "redis"` shares the limit across all replicas. The `"memory"` backend counts per process, so three replicas allow three times the configured limit, making it unsuitable for production. +- The top-level `redis` key gives the API a Redis cache store, which shares the rate limit (and the run locks, caps and revocations) across all replicas. Without it the counters live in each process, so three replicas allow three times the configured limit, which is unsuitable for production. + +## Choose the cache store -## Share a Redis connection for the thread cache +The API keeps everything it shares between workers in one **cache store**: the per-thread run lock, run and WebSocket caps, idempotency state, the ownership cache, token revocation and rate-limit counters. Pick it with these settings; the first one set wins: -The top-level `redis` key is a URL string used by the optional shared thread-ownership cache. This is separate from `rate_limit.redis`: +1. `"cache_store": "module:attr"`: your own `BaseCacheStore` (an instance or a zero-argument factory). See [Bring your own backend](/docs/server/bring-your-own-backend). +2. `"redis": "redis://..."`: `RedisCacheStore` on that URL. +3. `REDIS_URL` in the environment: the same. +4. Otherwise `InMemoryCacheStore`, which is correct for one process only. ```json { "agent": "graph.react:app", - "redis": "redis://localhost:6379/0", - "rate_limit": { "backend": "redis", "redis": { "url": "redis://localhost:6379/1" } } + "redis": "${REDIS_URL}" } ``` -If the `redis` key is not set, the server falls back to the `REDIS_URL` environment variable. With neither, the ownership cache is per-process (an L1 cache with a 30 second TTL; with Redis, an L2 cache with a 1 hour TTL). Without a shared cache, each replica maintains its own view of which user owns each thread, leading to inconsistent access control across replicas. +The server logs one INFO line at boot that names the store and where the choice came from. Valkey, Dragonfly and KeyDB work through the Redis URL. With an in-memory store, each replica keeps its own locks, caps and view of which user owns each thread, so use a shared store once you run more than one worker. The rate limiter no longer has a Redis URL of its own; `rate_limit.redis` and `rate_limit.backend` were removed in 1.0.0rc4. ## Set test and evaluation defaults @@ -248,11 +252,11 @@ The `websocket` key manages connection limits for `/v1/graph/ws` (streaming) and - `realtime_client_overrides` lists the session options a `/v1/graph/live` client may set in its init frame: any of `voice`, `modalities`, `vad` and `system_prompt`. The default is `["voice", "modalities", "vad"]`. `system_prompt` replaces the agent's instructions, so list it only when every caller is trusted to rewrite them. - `enabled` set to `false` leaves `/v1/graph/ws` and `/v1/graph/live` out of the app. -When `redis` (or `REDIS_URL`) is configured, the connection caps are shared by all workers and replicas through Redis. Without Redis, each process counts its own connections, so the real cap is the limit times the number of workers. If a worker dies, its slots expire on their own. If Redis is unreachable, the server allows the connection and logs the error. +When a shared cache store is configured (`redis`, `REDIS_URL` or `cache_store`), the connection caps are shared by all workers and replicas. Without one, each process counts its own connections, so the real cap is the limit times the number of workers. If a worker dies, its slots expire on their own. If the cache store is unreachable, the server allows the connection and logs the error. ## Limit concurrent runs per user -`runs.max_concurrent_per_user` caps the graph runs (invoke, stream, AG-UI) one verified user can have in flight (default 10). Over the cap, the request gets `429`. Set it to `0` or `null` for no cap. Requests with no verified user (auth off) are not capped. Like the WebSocket caps, the count is shared across workers and replicas through Redis when `redis` or `REDIS_URL` is set, and per process otherwise. +`runs.max_concurrent_per_user` caps the graph runs (invoke, stream, AG-UI) one verified user can have in flight (default 10). Over the cap, the request gets `429`. Set it to `0` or `null` for no cap. Requests with no verified user (auth off) are not capped. Like the WebSocket caps, the count is shared across workers and replicas through the cache store when a shared one is configured, and per process otherwise. ```json { @@ -283,19 +287,16 @@ A client can send a `config` object with a run, and your nodes and tools can rea ## Environment variable expansion in Redis URLs -Only the `rate_limit.redis` configuration supports environment variable expansion. Both `$VAR` and `${VAR}` forms work: +The `redis` key supports environment variable expansion. Both `$VAR` and `${VAR}` forms work: ```json { "agent": "graph.react:app", - "rate_limit": { - "backend": "redis", - "redis": { "url": "${REDIS_URL}" } - } + "redis": "${REDIS_URL}" } ``` -If a variable is missing, the server fails at startup with an error rather than silently starting without rate limiting. This is deliberate: a misconfigured server is worse than one that refuses to start. +If a variable is missing, the server fails at startup with an error rather than silently starting without a shared store. This is deliberate: a misconfigured server is worse than one that refuses to start. All other configuration keys in `10xgraph.json` are read literally. Keep credentials out of the file: use the environment, `.env` files, or your orchestration platform. @@ -338,14 +339,13 @@ A typical prod config (see production checklist below): "auth": "jwt", "authorization": "ownership", "store": "graph.dependencies:store", - "redis": "redis://redis:6379/0", + "redis": "${REDIS_URL}", + "token_revocation": true, "rate_limit": { "enabled": true, - "backend": "redis", "requests": 100, "window": 60, "by": "user", - "redis": { "url": "${REDIS_URL}", "prefix": "10xgraph:rate-limit" }, "trusted_proxy_headers": true, "trusted_proxy_hops": 1, "fail_open": true @@ -369,9 +369,9 @@ If `authorization` is unset, the server uses `ownership` whenever auth is config `InMemoryCheckpointer` loses all threads on restart and is invisible to other replicas. In production, pass `PgCheckpointer` to `compile()` in your graph module. This ensures thread history survives restarts and is consistent across all replicas. -### 4. Set rate limit backend to Redis with multiple replicas +### 4. Use a shared cache store with multiple replicas -The `"memory"` backend counts per process, so with three workers you allow three times the configured limit. With multiple replicas, use `"backend": "redis"` and ensure all replicas point to the same Redis instance. Also set `trusted_proxy_hops` correctly: it counts from the right of the `X-Forwarded-For` header, so the value must match how many proxies sit in front of your server. Set it too high and a client can forge its own address to bypass limits. +The in-memory cache store counts per process, so with three workers you allow three times the configured limit. With multiple replicas, set `"redis": "${REDIS_URL}"` and ensure all replicas point to the same Redis instance. Also set `trusted_proxy_hops` correctly: it counts from the right of the `X-Forwarded-For` header, so the value must match how many proxies sit in front of your server. Set it too high and a client can forge its own address to bypass limits. ### 5. Wire observability @@ -415,7 +415,7 @@ Finally, verify that persistence survives a restart: start a thread by invoking | `AttributeError: module has no attribute` | `graph.react:app` expects an `app` variable in the `graph/react.py` file. Check the name and that it is a compiled graph. | | Checkpointer connection fails | The Postgres DSN is correct and the database is reachable. Test: `psql `. The Redis URL is reachable: `redis-cli -u ping`. | | `JWT_SECRET_KEY and JWT_ALGORITHM must be set` | Export both variables, or set `"env": ".env"` and add them there. The secret needs at least 32 characters. | -| Rate limiting not working on multi-replica setup | Set `rate_limit.backend` to `"redis"`, not `"memory"`. Ensure all replicas point to the same Redis instance. | +| Rate limiting not working on multi-replica setup | Set `redis` (or `REDIS_URL`) so the cache store is shared. Ensure all replicas point to the same Redis instance. | | Threads visible to users who did not create them | Set `"authorization": "ownership"` or implement a custom `AuthorizationBackend`. Check that all replicas use the same setting. | For a complete list of configuration keys, defaults, and types, see the [configuration reference](/docs/reference/api-cli/configuration). diff --git a/src/content/docs/server/cors-and-security-headers.md b/src/content/docs/server/cors-and-security-headers.md index 0b40b40..6c05af8 100644 --- a/src/content/docs/server/cors-and-security-headers.md +++ b/src/content/docs/server/cors-and-security-headers.md @@ -54,7 +54,7 @@ export CORS_ALLOW_CREDENTIALS="true" ### Private-network preflight for the hosted playground -Added after 1.0.0rc2. +Added in 1.0.0rc3. When a public page such as `https://play.10xgraph.com` calls a server on `localhost`, Chrome first sends a preflight with `Access-Control-Request-Private-Network`. The API answers it with `Access-Control-Allow-Private-Network: true`, but only for an origin listed in `ORIGINS`. For any other origin the preflight is refused as before and the header is not sent. The browser also asks the user for permission; see [Allow the playground to reach your local server](/docs/server/playground#allow-the-playground-to-reach-your-local-server). diff --git a/src/content/docs/server/deploy.mdx b/src/content/docs/server/deploy.mdx index 0577741..83f7d6a 100644 --- a/src/content/docs/server/deploy.mdx +++ b/src/content/docs/server/deploy.mdx @@ -195,7 +195,7 @@ For local development, `10xgraph dev` is sufficient. For shared or production us - **Persistence required:** Set a durable checkpointer (for example `PgCheckpointer`), by passing it to `graph.compile(checkpointer=...)` in your graph code (the `checkpointer` key in `10xgraph.json` is not read). In-memory checkpointing loses all threads on restart. - **Multiple instances:** Use a shared durable checkpointer and a shared store (if using long-term memory). Each instance must access the same database backend. -- **Rate limiting across instances:** Use the `redis` rate-limit backend (the memory backend counts per process and is not shared). +- **Shared state across instances:** Set `redis` (or `REDIS_URL`) so the API's cache store is shared. It holds the run locks, caps, idempotency state, ownership cache, token revocation and rate-limit counters. The in-memory store counts per process and is not shared. ## Production environment settings @@ -207,7 +207,7 @@ The generated files set `MODE=production`. Add the following at runtime: | `ALLOWED_HOST` | Your hostnames | Comma-separated. `k8s.yaml` reads it from the Secret; compose reads it from `.env`. A wildcard `*` stops startup in production unless `STRICT_PRODUCTION_CHECKS=false` | | `JWT_SECRET_KEY` | 32+ bytes, from a secret manager | Required if `auth` is `jwt`. See [Add JWT authentication](/docs/server/auth) | | `JWT_ALGORITHM` | `HS256` or your choice | Required with `jwt` | -| `REDIS_URL` | Your Redis connection URL | Optional. The server falls back to it for the shared tier of the authorization ownership cache when the `redis` key in `10xgraph.json` is unset. For shared rate limiting, point `rate_limit.redis.url` at it as `${REDIS_URL}` | +| `REDIS_URL` | Your Redis connection URL | Optional. The server uses it for its shared cache store when the `redis` and `cache_store` keys in `10xgraph.json` are unset. Without any of them the cache store is in-memory and per process | | `WEB_CONCURRENCY` | Workers per container | Default 2. Tune for your CPU and load | For durable persistence and distributed deployments, also configure: diff --git a/src/content/docs/server/index.md b/src/content/docs/server/index.md index 441b2d8..bad89d5 100644 --- a/src/content/docs/server/index.md +++ b/src/content/docs/server/index.md @@ -20,7 +20,7 @@ Serving your graph with `10xgraph-api` gives you these guarantees without code i | Ownership isolation | In production (and whenever auth is on) a thread is readable and writable only by its creator; a foreign `invoke` is rejected with 403 before the model runs. | [Thread ownership](/docs/server/auth) | | Scoped authorization | Per-route scopes, and a boot-time route guard that refuses to start if any non-public route has no permission check. | [Authorization scopes](/docs/guides/authorization-scopes) | | Input rules | Request size limit, client messages limited to the `user` role, built-in state fields not writable, `current_node` and state values validated. | [Invoke and stream](/docs/server/invoke-and-stream), [CORS and security headers](/docs/server/cors-and-security-headers) | -| Rate limits and run caps | Sliding-window limits (memory or Redis) and a per-run timeout (300 s by default in production). | [Rate limiting](/docs/server/rate-limiting) | +| Rate limits and run caps | Sliding-window limits (counters in the cache store: memory or Redis) and a per-run timeout (300 s by default in production). | [Rate limiting](/docs/server/rate-limiting) | | Production mode | `MODE=production` validates itself at startup, turns off `/docs`, and hides 5xx error text. | [Production checklist](/docs/server/production-checklist) | | Durable state | Threads, state and messages go through your checkpointer; with `PgCheckpointer` they survive restarts and are shared across replicas. | [Backup and restore](/docs/server/backup-and-restore), [Kubernetes](/docs/server/kubernetes) | | Deploy files | `10xgraph build` writes a hardened Dockerfile and optional Compose and Kubernetes files. | [Deploy](/docs/server/deploy) | diff --git a/src/content/docs/server/kubernetes.md b/src/content/docs/server/kubernetes.md index d01dd17..91a95c9 100644 --- a/src/content/docs/server/kubernetes.md +++ b/src/content/docs/server/kubernetes.md @@ -54,7 +54,7 @@ The manifest is a working skeleton. It sets these values, which are easy to get | `readinessProbe` | 10s interval, 5s initial delay | Removes unhealthy pods from the load balancer quickly. Checked via `GET /health/ready`. | | `livenessProbe` | 30s interval, 30s initial delay, 5 failures | Deliberately slack. A worker handling a long run must not be restarted. Checked via `GET /ping`. | -The readiness probe is strict (10-second intervals) because false positives only temporarily remove a pod from rotation. The liveness probe is slack (30-second intervals, 5 consecutive failures required) because a false positive kills the pod mid-run. Readiness hits `/health/ready` (checkpointer, store and Redis reachable) and liveness hits `/ping`; neither requires authentication. +The readiness probe is strict (10-second intervals) because false positives only temporarily remove a pod from rotation. The liveness probe is slack (30-second intervals, 5 consecutive failures required) because a false positive kills the pod mid-run. Readiness hits `/health/ready` (checkpointer, store and cache store reachable) and liveness hits `/ping`; neither requires authentication. Default resource requests and limits are also set (500m CPU and 512 Mi memory requested; 2 CPU and 2 Gi memory as limits), but you should tune these based on your agents and deployment environment. @@ -194,6 +194,8 @@ Before scaling, understand how 10xgraph handles distributed state: 1. **Shared checkpointer is mandatory.** All replicas must read and write to the same Postgres and Redis. If they do not, the same thread will resolve differently depending on which pod answers, causing data loss. Use `PgCheckpointer` in production (see [Set up checkpointing](/docs/guides/set-up-checkpointing)). + Set `redis` (or `REDIS_URL`) as well so the API's cache store is shared: the run locks, caps, idempotency state, token revocations and rate-limit counters all live there. The in-memory store is per pod. + 2. **Threads are not sticky.** State lives in the checkpointer, not in the pod. You do not need (and should not use) session affinity. A client can invoke a thread on any replica. 3. **Streaming connections are long-lived.** If your load balancer (or ingress) has an idle timeout shorter than your longest run, the connection will close mid-stream even though the pod is healthy. Set idle timeouts to at least 15 minutes. For AWS ALB/NLB, increase `deregistration_delay`. For Nginx, increase `proxy_read_timeout`. diff --git a/src/content/docs/server/playground.md b/src/content/docs/server/playground.md index 29a2b6d..ecf36ce 100644 --- a/src/content/docs/server/playground.md +++ b/src/content/docs/server/playground.md @@ -42,7 +42,7 @@ If the browser tab does not open automatically, copy the playground URL from the ## Allow the playground to reach your local server -Added after 1.0.0rc2. +Added in 1.0.0rc3. The playground is a public HTTPS page and your server runs on your own machine. Chrome's Local Network Access feature therefore asks you for permission the first time the page calls `localhost`, and the playground makes one early request to your server so the prompt appears on the Connect page. Click Allow. diff --git a/src/content/docs/server/production-checklist.md b/src/content/docs/server/production-checklist.md index d6117b1..83bcd86 100644 --- a/src/content/docs/server/production-checklist.md +++ b/src/content/docs/server/production-checklist.md @@ -13,7 +13,7 @@ faq: - question: "Can I use in-memory checkpointing in production?" answer: "No. InMemoryCheckpointer loses all state on restart and cannot be shared across instances. Use PgCheckpointer for all production deployments." - question: "Do I have to use Redis?" - answer: "PgCheckpointer requires a Redis connection (redis_url, a redis_pool or a redis instance) in addition to Postgres; Redis is its cache and Postgres is the durable store. Redis is also what the Redis rate-limit backend uses." + answer: "PgCheckpointer requires a Redis connection (redis_url, a redis_pool or a redis instance) in addition to Postgres; Redis is its cache and Postgres is the durable store. The API itself uses Redis as its cache store (run locks and caps, idempotency, ownership cache, token revocation, rate-limit counters) when you set `redis` or `REDIS_URL`; without it those live in each process." --- Deploying 10xGraph to production requires hardening your API server for security, reliability, and scale. This checklist walks through every critical setting, from runtime mode to persistence topology to horizontal scaling, with verification steps to confirm each piece is working before traffic arrives. @@ -30,7 +30,7 @@ With `MODE=production`, the server does these for you. You do not configure them - Scopes graph introspection (`GET /v1/graph`, `/v1/graph/tools`, `/v1/graph:StateSchema`) to callers with the `graph:read` scope, unless you set `graph_introspection` to `"open"`. - Defaults authorization to ownership, so each user reaches only their own threads. - Turns debug mode off. -- Pings every configured backend at boot and refuses to start while one is unreachable (see [Startup check](#startup-check)). Added after 1.0.0rc2. +- Pings every configured backend at boot and refuses to start while one is unreachable (see [Startup check](#startup-check)). Added in 1.0.0rc3. - Ships a hardened image when you use `10xgraph build`: non-root user, Gunicorn with `TenxGraphUvicornWorker` and long graceful timeouts, health check on `/ping`. On SIGTERM `/health/ready` answers 503 while open runs finish. For why these matter, see the [security model](/docs/production/security-model) and [scaling and reliability](/docs/production/scaling). @@ -191,17 +191,19 @@ Expected tables in the `public` schema (or the `schema=` you pass): `threads`, ` ### Startup check -At boot the server pings every backend it is configured to use, in development and production: the checkpointer's Postgres (a query and the schema version) and its Redis cache, the API's Redis (run limits, ownership cache, token revocation), the rate limiter's Redis when it has its own URL, and the memory store through `aping()`. It retries with a short backoff for `STARTUP_CHECK_TIMEOUT_SECONDS` (30 by default), so containers started together can come up in any order. If a backend is still down, the server exits with a message naming each backend and its `host:port` (no credentials). A broken connection string therefore fails the deploy instead of the first request, and the orchestrator restarts the container until the backends are up. +At boot the server calls `aping()` on the checkpointer (Postgres, the schema version and its Redis cache), on the store when one is bound, and on the API's cache store. It retries with a short backoff for `STARTUP_CHECK_TIMEOUT_SECONDS` (30 by default), so containers started together can come up in any order. The checkpointer and the store are fatal: if one is still down, the server exits with a message naming each backend and its endpoint (no credentials). The cache store is fatal only when a fail-closed feature uses it: `token_revocation` is on, or `rate_limit` is on with `fail_open` set to `false`. Otherwise the server logs one WARNING that the cache store is unreachable and that locks and caps fall back to per process, and starts anyway. A broken connection string therefore fails the deploy instead of the first request, and the orchestrator restarts the container until the backends are up. -`STARTUP_CHECKS=false` skips the check. It runs the same probes as `/health/ready`, with a different outcome: at startup an unreachable backend stops the server, while at runtime `/health/ready` answers 503 so the load balancer stops sending traffic until the backend returns. See [Ping and health](/docs/reference/rest-api/ping#get-healthready). +A leftover `REDIS_URL` no longer blocks boot when nothing you configured needs it. Before 1.0.0rc4 it did. -Added after 1.0.0rc2. +`STARTUP_CHECKS=false` skips the check. It runs the same probes as `/health/ready`, with a different outcome: at startup an unreachable checkpointer or store stops the server, while at runtime `/health/ready` answers 503 so the load balancer stops sending traffic until the backend returns. See [Ping and health](/docs/reference/rest-api/ping#get-healthready). + +Added in 1.0.0rc3. ### Redis caching Redis acts as a hot cache in front of Postgres. `PgCheckpointer` requires a Redis connection, but the durable data lives in Postgres. After a Redis connection failure, `PgCheckpointer` skips Redis for a short cool-down (`redis_breaker_cooldown`, 5 seconds by default, `0` disables it) and uses Postgres at once, so a Redis outage does not slow every run step. -If several deployments share one Redis, set `REDIS_KEY_PREFIX` for the API (run locks, caps, ownership cache, token revocation) and `redis_key_prefix` on `PgCheckpointer` to the same value. They are two separate settings, and each prefixes only its own keys. +If several deployments share one Redis, set `REDIS_KEY_PREFIX` for the API (run locks and caps, idempotency, ownership cache, token revocation, rate limiting) and `redis_key_prefix` on `PgCheckpointer` to the same value. They are two separate settings, and each prefixes only its own keys. ```bash REDIS_URL=redis://redis.internal:6379/0 @@ -216,11 +218,11 @@ redis-cli -u redis://redis.internal:6379/0 ping ### One run per thread -The server runs one graph run at a time on a thread. With Redis configured, the lock is shared across all instances, so two replicas cannot run the same thread at once. Without Redis, the lock is per process and does not stop two replicas from running one thread together. A second run waits up to 30 seconds, then gets `409` (`thread is busy`). The lock expires on its own if a worker dies. If Redis is unreachable, runs proceed without the lock and the error is logged. See [Invoke and stream](/docs/server/invoke-and-stream#one-run-per-thread). +The server runs one graph run at a time on a thread. With a shared cache store configured (Redis), the lock is shared across all instances, so two replicas cannot run the same thread at once. Without one, the lock is per process and does not stop two replicas from running one thread together. A second run waits up to 30 seconds, then gets `409` (`thread is busy`). The lock expires on its own if a worker dies. If the cache store is unreachable, runs proceed without the lock and the error is logged. See [Invoke and stream](/docs/server/invoke-and-stream#one-run-per-thread). ### Warnings you may see -The server logs a WARNING at startup in production, or when `WEB_CONCURRENCY` is above 1, if the checkpointer is `InMemoryCheckpointer` or `SqliteCheckpointer` (threads live in one process) or if no Redis is configured (run locks, caps and in-memory rate limits count per process). `compile()` logs a WARNING when you give it no checkpointer and it falls back to in-memory, and a graph logs one WARNING per graph on its first run with no `user_id`. These are your call; use `PgCheckpointer` with Redis for more than one process. +The server logs a WARNING at startup in production, or when `WEB_CONCURRENCY` is above 1, if the checkpointer is `InMemoryCheckpointer` or `SqliteCheckpointer` (threads live in one process) or if the cache store is in-memory (run locks, caps and rate limits count per process, and token revocations only reach the process that received them). `compile()` logs a WARNING when you give it no checkpointer and it falls back to in-memory, and a graph logs one WARNING per graph on its first run with no `user_id`. These are your call; use `PgCheckpointer` with Redis for more than one process. ### Topology diagram @@ -304,16 +306,13 @@ Enable rate limiting to protect against abuse and unintended traffic spikes: "enabled": true, "requests": 100, "window": 60, - "by": "ip", - "backend": "redis", - "redis": { - "url": "redis://redis.internal:6379/1" - } - } + "by": "ip" + }, + "redis": "${REDIS_URL}" } ``` -This allows 100 requests per IP address per 60-second window. Use a separate Redis database (`/1`) to isolate rate limit data from the checkpointer cache. +This allows 100 requests per IP address per 60-second window. The counters live in the API's cache store, so set `redis` (or `REDIS_URL`) to share them across replicas. They use their own key namespace, separate from the checkpointer cache. `/ping` is always excluded from rate limiting. Add other paths with `exclude_paths`: @@ -339,7 +338,7 @@ done # After threshold, requests should return 429 Too Many Requests ``` -With `fail_open` set to `false` and the Redis backend unreachable, the API answers `503` with `RATE_LIMIT_UNAVAILABLE` and `Retry-After: 5` instead of 429. See [Rate limiting](/docs/server/rate-limiting). Added after 1.0.0rc2. +With `fail_open` set to `false` and the cache store unreachable, the API answers `503` with `RATE_LIMIT_UNAVAILABLE` and `Retry-After: 5` instead of 429. See [Rate limiting](/docs/server/rate-limiting). Added in 1.0.0rc3. ## 8. Proxy buffering and request handling @@ -511,8 +510,8 @@ curl http://127.0.0.1:8000/ping | One worker sees state, another does not | Workers are not sharing the same backend | Point all instances at the same Postgres and Redis | | CORS requests are blocked | `ORIGINS` does not include the client domain | Add the domain to `ORIGINS`. A public API can set `ORIGINS=*` with `CORS_ALLOW_CREDENTIALS=false` and `STRICT_PRODUCTION_CHECKS=false` | | `409 thread is busy` on a thread | Another run is still active, and the 30 s wait ran out | Wait for the run to end, use `on_busy: "cancel"`, or use one thread per conversation | -| Server exits at boot with `Refusing to start: ... backend(s) unreachable` | A configured Postgres, Redis or store did not answer within `STARTUP_CHECK_TIMEOUT_SECONDS` | Fix the URL or start the backend. `STARTUP_CHECKS=false` skips the check | -| Rate limiting not working | No `rate_limit` block, or Redis is down (with `fail_open` true, requests pass) | Verify `rate_limit.redis.url` and Redis connectivity | +| Server exits at boot with `Refusing to start: ... backend(s) unreachable` | A configured Postgres, store, or (with `token_revocation` or a fail-closed `rate_limit`) cache store did not answer within `STARTUP_CHECK_TIMEOUT_SECONDS` | Fix the URL or start the backend. `STARTUP_CHECKS=false` skips the check | +| Rate limiting not working | No `rate_limit` block, or Redis is down (with `fail_open` true, requests pass) | Verify the `redis` key or `REDIS_URL` and Redis connectivity | ## Related pages diff --git a/src/content/docs/server/project-setup.md b/src/content/docs/server/project-setup.md index 353c6fd..4e440ee 100644 --- a/src/content/docs/server/project-setup.md +++ b/src/content/docs/server/project-setup.md @@ -189,9 +189,10 @@ Init builds your config from your answers, not a static template. Examples: "authorization": "ownership", "thread_name_generator": "graph.thread_name_generator:MyNameGenerator", "injectq": "graph.agent:container", + "redis": "${REDIS_URL}", + "token_revocation": true, "rate_limit": { "enabled": true, - "backend": "redis", "requests": 100, "window": 60, "by": "ip", diff --git a/src/content/docs/server/rate-limiting.md b/src/content/docs/server/rate-limiting.md index 5fc7f76..4df8ce9 100644 --- a/src/content/docs/server/rate-limiting.md +++ b/src/content/docs/server/rate-limiting.md @@ -4,19 +4,19 @@ description: How to enable and configure the built-in sliding-window rate limite section: "API server" group: "Security" order: 70 -updated: "2026-10-27" +updated: "2026-10-31" faq: - question: Do I need rate limiting in development? answer: "In development, you can run without limits. In any production environment, rate limiting protects your infrastructure from both accidental overuse and abuse." - question: How much does Redis add to the setup? - answer: "The limiter reuses a Redis client already bound in the InjectQ container (under `redis`, `redis_client` or `Redis`). Otherwise it needs `rate_limit.redis.url` and opens its own connection." - - question: What happens if Redis is unreachable? - answer: "With `fail_open: true` (the default), requests pass through; with `fail_open: false`, requests are denied. Choose based on whether uptime matters more than protection." + answer: "Set `redis` in `10xgraph.json` or `REDIS_URL` in the environment. The rate limiter then shares the API's cache store with the run locks and caps. There is no separate rate-limit Redis setting." + - question: What happens if the cache store is unreachable? + answer: "With `fail_open: true` (the default), requests pass through; with `fail_open: false`, requests are denied with 503 `RATE_LIMIT_UNAVAILABLE`. Choose based on whether uptime matters more than protection." --- Rate limiting protects your API from overuse and abuse by enforcing a ceiling on how many requests a client can make within a time window. Without it, a single user or malicious actor can consume all your infrastructure resources and degrade service for everyone. The 10xGraph API server includes a built-in sliding-window rate limiter that you enable with a few lines of configuration. -By default, the 10xGraph API accepts unlimited requests. This guide shows how to enable the rate limiter, choose a backend suitable for your deployment, and handle edge cases like proxy headers and WebSocket connections. +By default, the 10xGraph API accepts unlimited requests. This guide shows how to enable the rate limiter, choose where the counters live, and handle edge cases like proxy headers and WebSocket connections. Rate limits and run caps are part of how the server scales safely; see [scaling and reliability](/docs/production/scaling) and the [security model](/docs/production/security-model). @@ -26,11 +26,11 @@ The rate limiter is HTTP middleware that executes before your graph logic runs o The limiter uses a **sliding-window algorithm**, not fixed time buckets. This means the window rolls with each request: a client with a limit of 100 requests per 60 seconds can send 100 requests in the first 30 seconds, then nothing until 60 seconds have passed, then 100 more. This prevents bursts that would succeed with a simple bucket-based counter. -Three backends are available: **memory** for development, **Redis** for production deployments, and **custom** if you need to store counters somewhere else. Your choice depends on how many workers you run and whether you prioritize simplicity or distributed enforcement. +The counters live in the API's **cache store**, the same one that holds the run locks and caps. There is no separate rate-limit backend. With a Redis URL set (`redis` in `10xgraph.json` or `REDIS_URL`) every worker shares the counters; without one they live in each process. To keep counters somewhere else, supply your own cache store (see [Bring your own backend](/docs/server/bring-your-own-backend)). -## Method 1: In-memory backend (development / single process) +## Single process (development) -The simplest setup stores counters in the process memory. It works with a single Uvicorn worker and requires no extra dependencies. +With no Redis URL the counters live in process memory. It works with a single Uvicorn worker and needs no extra dependencies. **Update `10xgraph.json`:** @@ -39,7 +39,6 @@ The simplest setup stores counters in the process memory. It works with a single "agent": "graph.react:app", "rate_limit": { "enabled": true, - "backend": "memory", "requests": 100, "window": 60, "by": "ip", @@ -73,19 +72,15 @@ curl -i http://localhost:8000/v1/graph/invoke \ # Returns: 429 Too Many Requests ``` -