Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 5 additions & 6 deletions src/content/blog/ai-agents-fall-2026-what-changed.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ faq:
- q: Does 10xGraph retry or fall back when a model provider fails?
a: Yes. Agent retries 429, 500, 502, 503 and 529 responses by default with exponential backoff, configurable with RetryConfig. Pass fallback_models to try other models or providers, each with its own retry budget, after the primary fails.
- q: How does 10xGraph stop a resumed run from repeating a payment?
a: With a checkpointer that supports the tool ledger (PgCheckpointer durably, InMemoryCheckpointer in process), each completed tool call is recorded, and a resumed invoke or ainvoke skips calls that already finished. Streaming runs do not use the ledger yet, so add idempotency keys downstream too.
a: With a checkpointer that supports the tool ledger (PgCheckpointer durably, InMemoryCheckpointer in process), each completed tool call is recorded, and a resumed run skips calls that already finished, on invoke and stream alike. Add idempotency keys downstream too.
- q: Are OpenTelemetry GenAI semantic conventions stable?
a: Not as of October 2026. The gen_ai attributes are still marked Development and moved to a separate semantic-conventions-genai repository in June 2026, so pin versions and expect renames.
- q: Did the EU AI Act deadlines move?
Expand Down Expand Up @@ -558,10 +558,10 @@ Ledger support depends on the checkpointer:

The same config sets deadlines:

- `tool_timeout` caps a single tool call. The default is 300 seconds.
- `node_timeout` caps a whole node. The default is 900 seconds.
- `tool_timeout` caps a single tool call. There is no default.
- `node_timeout` caps a whole node. There is no default.

A value of `None` or 0 disables a timeout. Without deadlines, one hung payment API holds a worker indefinitely.
A value you set applies to both invoke and stream. `None` or 0 means no limit. Without deadlines, one hung payment API holds a worker indefinitely, so set them.

### Keep idempotency keys downstream

Expand All @@ -576,7 +576,7 @@ def charge_card(order_id: str, amount: float, config: dict | None = None) -> str

<Callout type="note" title="Streaming runs">

As of 10xgraph 0.10, the tool ledger and the `node_timeout` and `tool_timeout` deadlines apply to `invoke` and `ainvoke`. `stream` and `astream` run tools at least once and have no deadlines. Run side-effect-heavy turns through `ainvoke`, or rely on downstream idempotency keys.
The tool ledger and the `node_timeout` and `tool_timeout` deadlines apply to `invoke`, `ainvoke`, `stream` and `astream`. Tools still run at least once, so rely on downstream idempotency keys too.

</Callout>

Expand Down Expand Up @@ -819,7 +819,6 @@ Point `"agent"` in `10xgraph.json` at `graph.agent:app`, run `10xgraph dev`, and
Here are the gaps that matter for this post:

- **No built-in model comparison command.** `10xgraph eval` runs one agent. Comparing models means the loop shown in section 2.
- **Streaming has no ledger and no deadlines.** Tool replay protection and the `node_timeout` and `tool_timeout` deadlines cover `invoke` and `ainvoke` only.
- **No automatic per-tool permissions.** Scope checks are code you write inside the tool with `has_scope`.
- **New MCP spec support follows the libraries.** It arrives as `mcp` and `fastmcp` ship it.
- **A smaller ecosystem.** There are fewer integrations, tutorials and answered questions than for LangGraph or CrewAI, and the project is pre-1.0, so pin versions and read the [changelog](/changelog).
Expand Down
11 changes: 5 additions & 6 deletions src/content/blog/bounded-runs.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ faq:
- q: What happens when two requests use the same thread at once?
a: The second waits up to 30 seconds for the first to finish, then gets 409. With on_busy set to cancel, it asks the active run to stop first.
- q: Do these limits apply to streaming runs?
a: runs.timeout, the per-user run cap, the thread lock and the request size limits do. Node and tool timeouts and the tool ledger are part of the invoke path and do not apply to streaming runs.
a: Yes. runs.timeout, the per-user run cap, the thread lock, the request size limits, and any node_timeout or tool_timeout you set apply to streaming runs too.
---

Agents are loops that call a model and a set of tools, and nobody wrote down in advance how long the loop will take or how many tools one turn will start. A hung tool, a model that emits hundreds of calls in one message, two tabs sending to the same thread and a client that posts a huge body are all ordinary events once real users arrive. If nothing bounds them, the first one that happens takes the worker down.
Expand All @@ -32,12 +32,12 @@ At most `max_parallel_tool_calls` run at once, and the default is 10. Set it in
| Bound | Default | Where it is set |
|---|---|---|
| Whole run | 300 s with `MODE=production`, no limit in development | `runs.timeout` in `10xgraph.json` |
| One node | 900 s | `node_timeout` in the run config |
| One tool call | 300 s | `tool_timeout` in the run config |
| One node | No limit | `node_timeout` in the run config |
| One tool call | No limit | `tool_timeout` in the run config |

A run that passes `runs.timeout` is cancelled: invoke answers 504, and a stream, WebSocket run or AG-UI run ends with an error. Realtime sessions are not limited. An explicit number overrides the default and an explicit `null` removes the limit, in either mode.

The node and tool defaults are generous on purpose. The node limit sits above the 600 s default timeout of the model SDKs, so a slow model call fails with its own error instead of being masked by the node deadline. Both can be disabled with `None` or `0`.
There is no default node or tool limit: how long they may run is your call. A value you set applies to both invoke and stream. Set the node limit above the 600 s default timeout of the model SDKs, so a slow model call fails with its own error instead of being masked by the node deadline. `None` or `0` means no limit.

## One run per thread

Expand Down Expand Up @@ -72,7 +72,6 @@ The in-memory rate limiter keeps at most 50,000 keys. It used to sweep every key

Be clear about the gaps.

- Streaming runs do not have a tool ledger. The ledger lookup is on the invoke path, so a streamed run that is resumed after a crash can repeat a tool that already finished. Use idempotency keys for tools with side effects on streamed runs.
- Node and tool timeouts apply to invoke. For a streaming run, `runs.timeout` is the deadline that holds.
- Node and tool timeouts are off until you set them. Without them, `runs.timeout` is the only deadline on a run.
- Counts such as the per-user cap and the thread lock are per process until you add Redis.
- These bounds protect the server. They do not cap what a model costs you. Set provider-side spend limits too.
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ The short version: you build the agent, and the server owns the request surface,

## 6. What the server reveals: production mode

**Default with `MODE=production`:** an unknown `MODE` value refuses to start. A wildcard `ORIGINS` or `ALLOWED_HOST` refuses to start (`STRICT_PRODUCTION_CHECKS` is on by default in production). API docs are off, graph introspection is scoped, `IS_DEBUG` is forced off, and 5xx bodies and error chunks carry generic text. A node exception reaches the client as 500 `NODE_001` "An error occurred in a graph node.", with no stack trace. A tool that raises reaches the model only as "failed with an internal error (reference ...)"; the raw text stays in the server log under that reference.
**Default with `MODE=production`:** an unknown `MODE` value refuses to start. A wildcard `ORIGINS` or `ALLOWED_HOST` refuses to start (`STRICT_PRODUCTION_CHECKS` is on by default in production). API docs are off, graph introspection is scoped, debug output is off, and 5xx bodies and error chunks carry generic text. A node exception reaches the client as 500 `NODE_001` "An error occurred in a graph node.", with no stack trace. A tool that raises reaches the model only as "failed with an internal error (reference ...)"; the raw text stays in the server log under that reference.

**Config:** `MODE`, `ORIGINS`, `ALLOWED_HOST`, `STRICT_PRODUCTION_CHECKS`.

Expand Down
15 changes: 7 additions & 8 deletions src/content/blog/your-agent-charged-the-card-twice.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ faq:
- q: Which checkpointers support the tool ledger?
a: PgCheckpointer stores it durably in a tool_executions table. InMemoryCheckpointer keeps it in process memory. SqliteCheckpointer does not implement it, so tools there run at least once, as before.
- q: Does the ledger work for streaming runs?
a: Not yet. As of 0.10.3 the ledger, per-step durable checkpoints and node and tool timeouts apply to invoke and ainvoke. Streaming runs persist state at terminal points and run tools without a ledger lookup.
a: Yes. The ledger, per-step durable checkpoints and node and tool timeouts apply to invoke, ainvoke, stream and astream.
- q: What happens if recording a tool result fails?
a: The tool result is still returned to the model and the run continues. The failure is logged at error level with an explicit warning that a replay of that node may run the tool again.
---
Expand Down Expand Up @@ -99,7 +99,7 @@ if checkpointer and ledger_key:
if recorded is not None:
return recorded # already ran: replay the result, do not call the tool

timeout = resolve_timeout(config, "tool_timeout", DEFAULT_TOOL_TIMEOUT_SECONDS)
timeout = resolve_timeout(config, "tool_timeout", DEFAULT_TOOL_TIMEOUT_SECONDS) # default None
tool_result = await self._invoke_tool_guarded(
function_name, function_args, tool_call_id, state, config, timeout, tool_attrs
)
Expand Down Expand Up @@ -237,14 +237,14 @@ The Redis cache has its own version guard and is cleared when a write conflicts.

Crashes are not the only way a node fails to finish. A tool blocked on a half-open socket, or an MCP server that accepts a connection and never answers, will hold a node open forever. The model SDK's request timeout does not help, because it does not cover tool execution. The step counter never advances, so the recursion limit never trips. Eventually an operator kills the worker, and you are back in the crash-and-replay case.

10xGraph puts a deadline on both levels:
10xGraph can put a deadline on both levels. Neither has a default, so you set them:

| Config key | Default | Bounds |
|---|---|---|
| `node_timeout` | 900 seconds | A whole node, including every tool it runs |
| `tool_timeout` | 300 seconds | A single tool call |
| `node_timeout` | None (no limit) | A whole node, including every tool it runs |
| `tool_timeout` | None (no limit) | A single tool call |

Both can be overridden per run in the config. `None` or any non-positive value, such as `0`, disables the deadline. An invalid value logs a warning and falls back to the default. The node default deliberately sits above the default model request timeout of 600 seconds, so a slow model call fails with its own error instead of a generic node timeout.
Both can be overridden per run in the config. `None` or any non-positive value, such as `0`, disables the deadline. An invalid value logs a warning and falls back to the default, which is no limit. A value you set applies to both invoke and stream. Keep the node limit above the default model request timeout of 600 seconds, so a slow model call fails with its own error instead of a generic node timeout.

```python title="run_agent.py"
config = {
Expand Down Expand Up @@ -274,7 +274,6 @@ You can turn this off with `"durable_checkpoint_every_step": False` in the run c

These are the limits we know about as of 0.10.3.

- **Streaming runs are not covered yet.** The ledger lookup, per-step durable checkpoints and node and tool timeouts are implemented in the `invoke` / `ainvoke` path. The streaming path (`stream` / `astream`) runs tools without a ledger lookup, writes durably only at terminal points, and has no deadlines. If an agent has non-idempotent tools and runs over streaming, the provider idempotency key is your protection until streaming gets the same treatment.
- **At least once, not exactly once.** The window between a side effect and its ledger record is small but real. See the section above.
- **The ledger records outcomes, not intent.** A tool that starts a long external job and crashes before returning has no record, so it is started again on replay. For long-running work, have the tool return a job handle quickly and track the job's progress separately.
- **SQLite has no ledger.** Use `PgCheckpointer` anywhere double execution matters.
Expand All @@ -285,7 +284,7 @@ These are the limits we know about as of 0.10.3.
- [ ] Use `invoke` / `ainvoke` for runs whose tools have side effects you cannot repeat.
- [ ] Give every money-moving or message-sending tool an idempotency key derived from your domain.
- [ ] Put side effects after `interrupt()` calls, never before them.
- [ ] Set `tool_timeout` to what your slowest healthy tool needs plus some margin, not to the 300 second default.
- [ ] Set `tool_timeout` to what your slowest healthy tool needs plus some margin, and do not leave it unset.
- [ ] Alert on the "COMPLETED but could not be recorded" log line and on `tenxgraph.tool.timeouts`.
- [ ] Always pass an explicit `thread_id`. Without one the runtime generates a random ID, and the run can be neither resumed nor stopped.
- [ ] Keep `durable_checkpoint_every_step` at its default of `true`.
Expand Down
1 change: 0 additions & 1 deletion src/content/build/production-react-agent.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -437,7 +437,6 @@ services:
- PYTHONUNBUFFERED=1
- PYTHONDONTWRITEBYTECODE=1
- MODE=production
- IS_DEBUG=false
- DATABASE_URL=postgresql://postgres:dev@db:5432/postgres # [!code ++]
- REDIS_URL=redis://redis:6379/0 # [!code ++]
depends_on: [db, redis] # [!code ++]
Expand Down
2 changes: 1 addition & 1 deletion src/content/build/ship-to-kubernetes.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ The folder now has four new files. Here is what is in them.
- **Two stages.** The builder installs your dependencies into a virtualenv. It then removes the CLI-only parts of the API package (project templates, the config editor, typer, rich and questionary where nothing else needs them) and uninstalls `pip`. The runtime stage copies the virtualenv and your code onto `python:3.13-slim`.
- **Gunicorn with Uvicorn workers**, started as `CMD ["gunicorn", "-k", "uvicorn_worker.UvicornWorker", "-b", "0.0.0.0:8000", "--graceful-timeout", "600", "--timeout", "660", "tenxgraph_api.src.app.main:app"]`. The image sets `WEB_CONCURRENCY=2` workers; override it with an environment variable.
- **A graceful timeout of 600 seconds**, because an agent run can last minutes and Gunicorn's default of 30 seconds would cut runs off at every deploy.
- **A non-root user**, `USER 10001:10001`, and `MODE=production`, `IS_DEBUG=false` as defaults.
- **A non-root user**, `USER 10001:10001`, and `MODE=production` as the default (which turns debug off).
- **A health check** that calls `GET /ping` from Python's standard library, so the image needs no curl.
- **`.dockerignore`** excludes `.env`, `.env.*`, keys, certificates and cloud credential files, so `COPY . .` cannot ship a secret.

Expand Down
2 changes: 1 addition & 1 deletion src/content/docs/client/graph-utilities.md
Original file line number Diff line number Diff line change
Expand Up @@ -260,7 +260,7 @@ await client.invoke([Message.text_message('Hello')], { initial_state: initialSta

## observability()

Fetches the reconstructed execution trace for a thread from `GET /v1/observability/{thread_id}`. The server returns the most recent run by default; pass a `runId` to fetch a specific one. Use it to show a timeline, attribute latency to a node, or report token usage per run.
Fetches the reconstructed execution trace for a thread from `GET /v1/observability/{thread_id}`. The server returns the most recent run by default; pass a `runId` to fetch a specific one. Use it to show a timeline, attribute latency to a node, or report token usage per run. It is a development aid: with `MODE=production` on the server it returns an empty result with `enabled: false`.

```ts
const result = await client.observability('thread-abc123');
Expand Down
2 changes: 1 addition & 1 deletion src/content/docs/client/realtime-audio.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ const client = new TenxGraphClient({
let player = createPcmPlayer(); // built inside the click handler

const session = client.realtime(
{ model: 'gemini-2.5-flash-live', modalities: 'AUDIO' },
{ model: 'gemini-3.8-live', modalities: 'AUDIO' },
{ reconnect: { enabled: false } },
);

Expand Down
2 changes: 1 addition & 1 deletion src/content/docs/client/stream-responses.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ for await (const chunk of stream) {
}
```

When a message is streaming, the server sends many chunks with `message.delta === true`. After the message completes, it sends one final chunk with `delta === false`.
When a message is streaming, the server sends many chunks with `message.delta === true`. After the message completes, it sends one final chunk with `delta === false`. Reasoning arrives as `{ type: 'reasoning' }` blocks in `content`, ordered before the text. A delta carries only the new piece of reasoning; the final message carries the full reasoning block.

---

Expand Down
7 changes: 6 additions & 1 deletion src/content/docs/client/use-memory-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -248,11 +248,16 @@ List all stored memories:

```ts
const response = await client.listMemories({ limit: 100 });
console.log(`Total memories: ${response.data.memories.length}`);
console.log(`Memories on this page: ${response.data.memories.length}`);

for (const mem of response.data.memories) {
console.log(`[${mem.memory_type}] ${mem.content.slice(0, 60)}`);
}

// More pages: response.data.has_more is true, and next_offset is the next offset
if (response.data.has_more) {
const next = await client.listMemories({ limit: 100, offset: response.data.next_offset ?? 0 });
}
```

Delete a single memory by ID:
Expand Down
Loading
Loading