diff --git a/docs-site/assets/kagent-docs/snippets/examples/a2a-port-warning.md b/docs-site/assets/kagent-docs/snippets/examples/a2a-port-warning.md new file mode 100644 index 00000000..558c97c5 --- /dev/null +++ b/docs-site/assets/kagent-docs/snippets/examples/a2a-port-warning.md @@ -0,0 +1,2 @@ +> [!WARNING] +> A default installation sets `config.server.auth.mode` to `insecure`. In this mode any caller that can reach the port can invoke any Agent. Do not expose port `8083` outside the cluster. For the wider identity model, see [Identity]({{< link path="about/substrate-runtime/identity" >}}). \ No newline at end of file diff --git a/docs-site/assets/kagent-docs/snippets/examples/mcp-endpoint-warning.md b/docs-site/assets/kagent-docs/snippets/examples/mcp-endpoint-warning.md new file mode 100644 index 00000000..87c4dfd9 --- /dev/null +++ b/docs-site/assets/kagent-docs/snippets/examples/mcp-endpoint-warning.md @@ -0,0 +1,2 @@ +> [!WARNING] +> A default installation sets `config.server.auth.mode` to `insecure`. In this mode the endpoint accepts every request and reads the caller's identity from an `X-User-Id` header that the caller sets itself, defaulting to `admin@kagent.dev`. The endpoint can invoke agents, create checkpoints, and create Sessions. Do not expose port `8083` outside the cluster. For the wider identity model, see [Identity]({{< link path="about/substrate-runtime/identity" >}}). \ No newline at end of file diff --git a/docs-site/assets/kagent-docs/snippets/observability/actor-state-records.md b/docs-site/assets/kagent-docs/snippets/observability/actor-state-records.md new file mode 100644 index 00000000..1d1d6ec1 --- /dev/null +++ b/docs-site/assets/kagent-docs/snippets/observability/actor-state-records.md @@ -0,0 +1 @@ +In your logging backend, select the records with the body `Actor state changed`, and filter them by the `ate.actor.name` attribute. \ No newline at end of file diff --git a/docs-site/assets/kagent-docs/snippets/observability/claude-logs-backend.md b/docs-site/assets/kagent-docs/snippets/observability/claude-logs-backend.md new file mode 100644 index 00000000..61f5d38d --- /dev/null +++ b/docs-site/assets/kagent-docs/snippets/observability/claude-logs-backend.md @@ -0,0 +1 @@ +Loki in the [OTel stack]({{< link path="observability/otel-stack" >}}) is one such backend. \ No newline at end of file diff --git a/docs-site/assets/kagent-docs/snippets/observability/grafana-password.md b/docs-site/assets/kagent-docs/snippets/observability/grafana-password.md new file mode 100644 index 00000000..016b785d --- /dev/null +++ b/docs-site/assets/kagent-docs/snippets/observability/grafana-password.md @@ -0,0 +1 @@ +For the password, see [Explore the telemetry in Grafana]({{< link path="observability/otel-stack#explore-the-telemetry-in-grafana" >}}). \ No newline at end of file diff --git a/docs-site/assets/kagent-docs/snippets/observability/substrate-otel-setup.md b/docs-site/assets/kagent-docs/snippets/observability/substrate-otel-setup.md new file mode 100644 index 00000000..761f997b --- /dev/null +++ b/docs-site/assets/kagent-docs/snippets/observability/substrate-otel-setup.md @@ -0,0 +1 @@ +The [OTel stack]({{< link path="observability/otel-stack#send-agent-substrate-telemetry-to-the-collector" >}}) and [Lightweight OTel stack]({{< link path="observability/lightweight-otel-stack" >}}) guides set both. \ No newline at end of file diff --git a/docs-site/assets/kagent-docs/snippets/observability/substrate-span-filter.md b/docs-site/assets/kagent-docs/snippets/observability/substrate-span-filter.md new file mode 100644 index 00000000..e69de29b diff --git a/docs-site/assets/kagent-docs/snippets/observability/tracing-backend.md b/docs-site/assets/kagent-docs/snippets/observability/tracing-backend.md new file mode 100644 index 00000000..d0bf7c9e --- /dev/null +++ b/docs-site/assets/kagent-docs/snippets/observability/tracing-backend.md @@ -0,0 +1 @@ +The [OTel stack]({{< link path="observability/otel-stack" >}}) sends traces to Tempo, and the [Lightweight OTel stack]({{< link path="observability/lightweight-otel-stack" >}}) sends traces to Jaeger. Both guides turn on tracing. \ No newline at end of file diff --git a/docs-site/content/kagent/1.x/about/substrate-runtime/networking-and-egress.md b/docs-site/content/kagent/1.x/about/substrate-runtime/networking-and-egress.md index cb4c51c6..53601bdd 100644 --- a/docs-site/content/kagent/1.x/about/substrate-runtime/networking-and-egress.md +++ b/docs-site/content/kagent/1.x/about/substrate-runtime/networking-and-egress.md @@ -114,7 +114,6 @@ kagent sets this up from two resources. Every ModelConfig that holds an `apiKeyS A credential is declared on a rule as a `replace_headers` entry. The entry names the header to replace, an optional prefix such as `Bearer `, and either a reference that a registered credential provider resolves or a JWT that Agent Substrate issues to assert the Actor's identity. The gateway replaces the header only when the request already sends it. A request that omits the placeholder header is forwarded unchanged, without the credential, rather than being denied. -> [!NOTE] -> Replacement happens only on a connection that the gateway terminates, and only when a credential provider is configured for the reference. If either is missing, the request reaches the destination without the credential. The request fails closed when replacement is attempted and fails. `403` is returned when the provider does not hold the credential or refuses to release it, and `503` is returned when the provider is unreachable. +Replacement happens only on a connection that the gateway terminates, and only when a credential provider is configured for the reference. If either is missing, the request reaches the destination without the credential. The request fails closed when replacement is attempted and fails. `403` is returned when the provider does not hold the credential or refuses to release it, and `503` is returned when the provider is unreachable. Replacement also works on a cleartext `http` rule, where the gateway does not originate TLS to the destination. The credential is then readable by anything on the path between the gateway and the destination unless something outside the gateway encrypts that segment. Prefer an `https` destination for any host that takes a credential. diff --git a/docs-site/content/kagent/1.x/examples/a2a-agents.md b/docs-site/content/kagent/1.x/examples/a2a-agents.md index e4d14b48..e9f0c31e 100644 --- a/docs-site/content/kagent/1.x/examples/a2a-agents.md +++ b/docs-site/content/kagent/1.x/examples/a2a-agents.md @@ -28,8 +28,7 @@ Both forms replace the `/api/a2a///` URL paths that kagen A caller addresses the Agent rather than one conversation. The message's `contextId` selects the conversation. Omitting both a `contextId` and a task ID starts a new {{< gloss "Session" >}}Session{{< /gloss >}}, and repeating a `contextId` continues that Session. -> [!WARNING] -> A default installation sets `config.server.auth.mode` to `insecure`. In this mode any caller that can reach the port can invoke any Agent. Do not expose port `8083` outside the cluster. For the wider identity model, see [Identity]({{< link path="about/substrate-runtime/identity" >}}). +{{< reuse "kagent-docs/snippets/examples/a2a-port-warning.md" >}} ### A2A methods diff --git a/docs-site/content/kagent/1.x/examples/agents-via-mcp.md b/docs-site/content/kagent/1.x/examples/agents-via-mcp.md index 07345288..48e77cd8 100644 --- a/docs-site/content/kagent/1.x/examples/agents-via-mcp.md +++ b/docs-site/content/kagent/1.x/examples/agents-via-mcp.md @@ -19,8 +19,7 @@ The MCP server is part of the controller's HTTP port rather than a separate depl - **Sessions**: The handler is stateless, so each request stands alone and a client does not need to establish a session first. - **Extensions**: The server advertises the `io.modelcontextprotocol/tasks` extension, which changes how invocations behave. For more information, see [Invoke without waiting](#invoke-without-waiting). -> [!WARNING] -> A default installation sets `config.server.auth.mode` to `insecure`. In this mode the endpoint accepts every request and reads the caller's identity from an `X-User-Id` header that the caller sets itself, defaulting to `admin@kagent.dev`. The endpoint can invoke agents, create checkpoints, and create Sessions. Do not expose port `8083` outside the cluster. For the wider identity model, see [Identity]({{< link path="about/substrate-runtime/identity" >}}). +{{< reuse "kagent-docs/snippets/examples/mcp-endpoint-warning.md" >}} ## Before you begin diff --git a/docs-site/content/kagent/1.x/get-started/your-first-mcp-tool.md b/docs-site/content/kagent/1.x/get-started/your-first-mcp-tool.md index 29bb1050..431331f3 100644 --- a/docs-site/content/kagent/1.x/get-started/your-first-mcp-tool.md +++ b/docs-site/content/kagent/1.x/get-started/your-first-mcp-tool.md @@ -85,8 +85,7 @@ kagent records what it discovered on the server's status, so the tool names come When the two values differ, kagent is still compiling, or compilation failed. A binding that names a RemoteMCPServer that does not exist in the namespace fails at the `ResolvedRefs` condition with the reason `ReferenceResolutionFailed`. - > [!WARNING] - > kagent resolves the server, but it does not check the tool names against the tools that the server actually serves. A misspelled tool name compiles into a ready revision, and the only symptom is an agent that never calls the tool that you expected. If a bound tool appears to be missing, check the spelling against the server's catalog. + kagent resolves the server, but it does not check the tool names against the tools that the server actually serves. A misspelled tool name compiles into a ready revision, and the only symptom is an agent that never calls the tool that you expected. If a bound tool appears to be missing, check the spelling against the server's catalog. ## Create a Session that has the tool diff --git a/docs-site/content/kagent/1.x/observability/audit-prompts.md b/docs-site/content/kagent/1.x/observability/audit-prompts.md index 87c97425..939184ca 100644 --- a/docs-site/content/kagent/1.x/observability/audit-prompts.md +++ b/docs-site/content/kagent/1.x/observability/audit-prompts.md @@ -58,7 +58,7 @@ Agent Substrate suspends an Actor as soon as a response completes. The controlle 1. [Install kagent]({{< link path="setup/installation" >}}). 2. [Create your first agent]({{< link path="get-started/your-first-agent" >}}), so that you have an {{< gloss "Agent" >}}Agent{{< /gloss >}} to send requests to. That guide also installs the kagent CLI. The steps on this page need the {{< reuse "kagent-docs/versions/kagent.md" >}} CLI, because the CLIs of other releases, newer ones included, do not have the Session commands that these steps use. To check your version, run `kagent version`. 3. Install [`jq`](https://jqlang.org/download/), to read the Session ID and revision out of the CLI's JSON output. -4. Set up a tracing backend, and turn on tracing. The [OTel stack]({{< link path="observability/otel-stack" >}}) sends traces to Tempo, and the [Lightweight OTel stack]({{< link path="observability/lightweight-otel-stack" >}}) sends traces to Jaeger. Both guides turn on tracing for you. +4. Set up a tracing backend, and turn on tracing. {{< reuse "kagent-docs/snippets/observability/tracing-backend.md" >}} ## Turn on content capture @@ -86,7 +86,7 @@ Turn on content capture in the kagent Helm release, then create a Session that p --set config.runtimeDefaults.telemetry.captureContent=true ``` - For an agent on the `claude` runtime, also set `config.runtimeDefaults.telemetry.logs.endpoint` to the address of your collector, and send the logs to a backend that stores them, such as Loki in the [OTel stack]({{< link path="observability/otel-stack" >}}). Without log export, the replies of a `claude` agent are not recorded. + For an agent on the `claude` runtime, also set `config.runtimeDefaults.telemetry.logs.endpoint` to the address of your collector, and send the logs to a backend that stores them. {{< reuse "kagent-docs/snippets/observability/claude-logs-backend.md" >}} Without log export, the replies of a `claude` agent are not recorded. 3. Wait for the controller to roll out. ```bash @@ -138,7 +138,7 @@ Send a request that contains a distinctive phrase, then find that phrase in the ```bash kubectl port-forward -n telemetry svc/kube-prometheus-stack-grafana 3000:80 ``` - 2. In your browser, open Grafana at [http://localhost:3000](http://localhost:3000), and log in. For the password, see [Explore the telemetry in Grafana]({{< link path="observability/otel-stack#explore-the-telemetry-in-grafana" >}}). + 2. In your browser, open Grafana at [http://localhost:3000](http://localhost:3000), and log in. {{< reuse "kagent-docs/snippets/observability/grafana-password.md" >}} 3. Open **Explore**, select the **Tempo** data source, and select the **TraceQL** query type. 4. Run the following query, which returns the model calls whose request contains the phrase. ```text diff --git a/docs-site/content/kagent/1.x/observability/lightweight-otel-stack.md b/docs-site/content/kagent/1.x/observability/lightweight-otel-stack.md index a4abbf83..9e9be1f9 100644 --- a/docs-site/content/kagent/1.x/observability/lightweight-otel-stack.md +++ b/docs-site/content/kagent/1.x/observability/lightweight-otel-stack.md @@ -338,7 +338,7 @@ Check each signal in turn: traces in Jaeger, metrics in Prometheus, and logs in kubectl port-forward -n telemetry svc/jaeger 16686:16686 ``` 2. In your browser, open Jaeger at [http://localhost:16686](http://localhost:16686). - 3. From the **Service** list, select `my-first-agent`, and click **Find Traces**. To see Agent Substrate's own work, select one of its services instead: `ateapi`, `atenet-router`, `atelet`, `atecontroller`, or `ateom-gvisor`. For what each service reports, see [Agent Substrate traces]({{< link path="observability/tracing#agent-substrate-traces" >}}). + 3. From the **Service** list, select `my-first-agent`, and click **Find Traces**. To see Agent Substrate's own work, select one of its services instead: `ateapi`, `atelet`, `atecontroller`, or `ateom-gvisor`. For what each service reports, see [Agent Substrate traces]({{< link path="observability/tracing#agent-substrate-traces" >}}). 2. Review the metrics in Prometheus. 1. Forward the Prometheus port, and leave the command running. diff --git a/docs-site/content/kagent/1.x/observability/otel-stack.md b/docs-site/content/kagent/1.x/observability/otel-stack.md index d5beca80..96578d25 100644 --- a/docs-site/content/kagent/1.x/observability/otel-stack.md +++ b/docs-site/content/kagent/1.x/observability/otel-stack.md @@ -504,7 +504,7 @@ Log in to Grafana, and query each backend from the **Explore** view. {{% tab name="Traces" %}} Select the **Tempo** data source, then select the **Search** query type. From the **Service Name** list, select `my-first-agent`, the service that the Agent reports as, and run the query. Open a trace to see the controller, proxy, and agent runtime spans of one request. - To see Agent Substrate's own work, select one of its services from the **Service Name** list instead: `ateapi`, `atenet-router`, `atelet`, `atecontroller`, or `ateom-gvisor`. For what each service reports, see [Agent Substrate traces]({{< link path="observability/tracing#agent-substrate-traces" >}}). + To see Agent Substrate's own work, select one of its services from the **Service Name** list instead: `ateapi`, `atelet`, `atecontroller`, or `ateom-gvisor`. For what each service reports, see [Agent Substrate traces]({{< link path="observability/tracing#agent-substrate-traces" >}}). {{% /tab %}} {{% tab name="Logs" %}} Select the **Loki** data source, and run the following query to show the state changes of your agent's Actor. Agent Substrate names the Actor `session-` followed by the Session ID, so replace `` with the value of `$SESSION_ID` from the previous section. diff --git a/docs-site/content/kagent/1.x/observability/substrate-telemetry.md b/docs-site/content/kagent/1.x/observability/substrate-telemetry.md index de457448..75d678e6 100644 --- a/docs-site/content/kagent/1.x/observability/substrate-telemetry.md +++ b/docs-site/content/kagent/1.x/observability/substrate-telemetry.md @@ -14,13 +14,16 @@ Agent Substrate reports telemetry from its own components, separately from the t | Signal | Where it goes | Details | | ------ | ------------- | ------- | | Actor logs | The stdout of the Worker pod that runs the Actor | The Actor's own output, and records for each suspend and resume. See [Read an Actor's logs](#read-an-actors-logs). | -| Component logs | The stdout of each component, and over OTLP when an OTLP endpoint is set | Logs from the Agent Substrate components, including a record of every Actor state change and an access log entry for every request that the router receives. See [Actor state changes](#actor-state-changes). | -| Traces | Over OTLP, when an OTLP endpoint is set | Scheduling, resume, and suspend work. These traces are separate from the agent request trace. See [Agent Substrate traces]({{< link path="observability/tracing#agent-substrate-traces" >}}). | +| Actor events | Over OTLP when an OTLP endpoint is set. Otherwise, the stdout of the component that records them. | An event for every Actor state change, and samples of each Actor's CPU and memory use. See [Actor state changes](#actor-state-changes) and [Actor usage samples](#actor-usage-samples). | +| Component logs | The stdout of each component | Logs from the Agent Substrate components, including an access log entry for every request that the router receives. The router also exports its access log over OTLP when an OTLP endpoint is set. | +| Traces | Over OTLP, when an OTLP endpoint is set | Scheduling, resume, and suspend work, in traces that are separate from the agent request trace. The router's `agentgateway` spans are the exception, and join the agent request trace. See [Agent Substrate traces]({{< link path="observability/tracing#agent-substrate-traces" >}}). | | Metrics | Over OTLP, when an OTLP endpoint is set | Worker capacity, scheduling, and snapshot timing. See [Agent Substrate metrics]({{< link path="observability/metrics#agent-substrate-metrics" >}}). | -The Agent Substrate Helm chart sets one OTLP endpoint for traces, metrics, and logs through `otel.endpoint`, and samples traces at a ratio of `0.01` by default through `otel.traces.samplingRatio`. The [OTel stack]({{< link path="observability/otel-stack#send-agent-substrate-telemetry-to-the-collector" >}}) and [Lightweight OTel stack]({{< link path="observability/lightweight-otel-stack" >}}) guides set both. +The Agent Substrate Helm chart sets one OTLP endpoint for traces, metrics, and logs through `otel.endpoint`, and samples traces at a ratio of `0.01` by default through `otel.traces.samplingRatio`. {{< reuse "kagent-docs/snippets/observability/substrate-otel-setup.md" >}} -The router access log and the Actor state change records are the two sources of Agent Substrate logs over OTLP, and the `otel.logs` settings control both together. To send them to a different backend than traces and metrics, set `otel.logs.endpoint`. To stop exporting them, set `otel.logs.enabled` to `false`. The chart has no setting for the access log alone. +The router access log and the Actor events are the Agent Substrate logs over OTLP, and the `otel.logs` settings control them together. To send them to a different backend than traces and metrics, set `otel.logs.endpoint`. To stop exporting them, set `otel.logs.enabled` to `false`. The chart has no setting for the access log alone. + +When the Agent Substrate release sets an OTLP endpoint, the component that records an Actor event exports the event and does not write it to stdout. Read the events from your logging backend. ## Actor identity @@ -62,7 +65,7 @@ An Actor writes to the stdout of whichever Worker pod runs it, and a suspended A Example output: ```console - ATESPACE NAME TEMPLATE STATE WORKER POD WORKER IP VERSION AGE + ATESPACE NAME TEMPLATE STATE WORKER POD WORKER IPS VERSION AGE kagent session-01a0d409-d249-728e-865c-ee58b55dff04 kagent/my-first-agent-a995d20d30a7 ACTOR_STATE_SUSPENDED 5 32m ``` @@ -95,12 +98,12 @@ An Actor writes to the stdout of whichever Worker pod runs it, and a suspended A The filter returns the Actor's lines from every Worker that currently runs in the pool, including the lines from before the Actor last suspended. -`kubectl logs` reads only the Worker pods that exist now. When a Worker pod is replaced, the output that it held is gone. To keep an Actor's history across Worker replacements, collect the stdout of the Worker pods with a log agent that stores structured JSON, and query it by the same labels. Neither the OTel stack guide nor the Lightweight OTel stack guide collects pod stdout, because both configure the Collector with the OTLP receiver only. +`kubectl logs` reads only the Worker pods that exist now. When a Worker pod is replaced, the output that it held is gone. To keep an Actor's history across Worker replacements, collect the stdout of the Worker pods with a log agent that stores structured JSON, and query it by the same labels. The OTel stack and Lightweight OTel stack guides configure their collectors with only the OTLP receiver. Neither collector reads pod stdout. Agent Substrate can also stream a single Actor's output directly, without the pool-wide filter. ```bash -kubectl ate logs actors session-$SESSION_ID --atespace kagent +kubectl ate logs actor session-$SESSION_ID --atespace kagent ``` The command reads the Worker that the Actor runs on, so it works only while the Actor is running. A kagent agent runs only during a turn, and the command returns an error the rest of the time. @@ -142,11 +145,11 @@ Each turn produces one cycle of the four records. The time between `Actor restor The Actor that builds a pair's first snapshot, in the `ate-golden` atespace, writes `Actor starting` and `Actor started` instead of the restore records, because it has no snapshot to restore from. -Each record also carries `trace_id`, `span_id`, and `trace_flags`, which join the record to the Agent Substrate trace for the same operation. A `trace_flags` value of `01` means that the trace was sampled and is in your tracing backend. A value of `00` means that the trace was not sampled, which is the case for most requests at the default ratio of `0.01`. +Each record also carries `trace_id`, `span_id`, and `trace_flags`, which join the record to the Agent Substrate trace for the same operation. A `trace_flags` value of `01` means that Agent Substrate sampled the trace and exported it. A value of `00` means that the trace was not sampled, which is the case for most requests at the default ratio of `0.01`. A sampled trace reaches your tracing backend only if your collector keeps the Agent Substrate spans. {{< reuse "kagent-docs/snippets/observability/substrate-span-filter.md" >}} ## Actor state changes -The Agent Substrate API server, `ateapi`, writes an `Actor state changed` record each time an Actor moves to a new state. The record carries the Actor identity labels, the operation that caused the change in `ate.actor.operation.name`, and the new state in `ate.actor.state`. The last record for an Actor tells you its state, and when the Actor entered it. +The Agent Substrate API server, `ateapi`, writes an `Actor state changed` event each time an Actor moves to a new state. The event includes the Actor identity labels, the operation that caused the change in `ate.actor.operation.name`, and the new state in `ate.actor.state`. The last event for an Actor tells you its state, and when the Actor entered it. | `ate.actor.state` | Meaning | | ----------------- | ------- | @@ -154,9 +157,13 @@ The Agent Substrate API server, `ateapi`, writes an `Actor state changed` record | `running` | The Actor is on a Worker and can serve the turn. | | `suspending` | The turn ended, and the Actor is writing its snapshot. | | `suspended` | The snapshot is written, and the Actor does not hold a Worker. | -| `deleting`, `deleted` | The Session was deleted. `deleted` is the last record that an Actor gets. | +| `deleting`, `deleted` | The Session was deleted. `deleted` is the last event that an Actor gets. | + +Where the events go depends on whether the Agent Substrate release sets an OTLP endpoint. When the release sets an endpoint, `ateapi` exports each event as an OTLP log with the same attributes. A logging backend then receives the state changes of every Actor without reading pod output. Do not sample this stream. A dropped event leaves the last known state wrong, with nothing to show that an event is missing. {{< reuse "kagent-docs/snippets/observability/actor-state-records.md" >}} -`ateapi` runs more than one replica, and each replica writes only the changes that it handles. Read the logs of every replica. +### Read state changes from stdout + +When the Agent Substrate release does not set an OTLP endpoint, `ateapi` writes each event to stdout. `ateapi` runs more than one replica, and each replica writes only the changes that it handles. Read the logs of every replica. ```bash for pod in $(kubectl get pods -n ate-system -o name | grep ate-api-server); do kubectl logs -n ate-system "$pod" @@ -169,11 +176,13 @@ Example output: {"time":"2026-09-24T17:00:43.194765137Z","level":"INFO","msg":"Actor state changed","ate.atespace":"kagent","ate.actor.name":"session-01a0d45b-c729-7d67-85c6-daa053202bff","ate.actor.uid":"831c62c5-b5ac-411f-b15e-7addfef97d73","ate.template.atespace":"kagent","ate.template.name":"my-first-agent-699a5ed2f709","ate.actor.operation.name":"resume","ate.actor.state":"running","trace_id":"402701838ffd50932dc0aec8e43a1604","span_id":"f92aecd47d89ad68","trace_flags":"00"} ``` -When the Agent Substrate release sets an OTLP endpoint, `ateapi` also exports each record as an OTLP log with the same attributes, so a logging backend receives the state changes of every Actor without reading pod output. Do not sample this stream. A dropped record leaves the last known state wrong, with nothing to show that a record is missing. +## Actor usage samples + +The Worker records an `Actor usage sampled` event each time an Actor starts running and again when it stops. Each event includes the Actor identity labels, the WorkerPool, the Actor's CPU time, and its memory use, peak memory, and working set. The `ate.stats.kind` attribute is `initial` for the first sample of a run and `final` for the last. ## Learn more -The Agent Substrate project documents its telemetry in full, including the complete metric registry, in [Actor observability](https://github.com/agent-substrate/substrate/blob/main/docs/observability.md). That page describes the newest Agent Substrate, which can be ahead of version {{< reuse "kagent-docs/versions/agent-substrate.md" >}} that this guide uses. Records that it describes might not exist in your version, such as per-Actor usage samples and the restore timing breakdown. +The Agent Substrate project documents its telemetry in full, including the complete metric registry, in [Actor observability](https://github.com/agent-substrate/substrate/blob/main/docs/observability.md). That page describes the newest Agent Substrate, which can be ahead of version {{< reuse "kagent-docs/versions/agent-substrate.md" >}} that this guide uses. Records that it describes might not exist in your version. You can also check out the following resources in this documentation set. diff --git a/docs-site/content/kagent/1.x/observability/tracing.md b/docs-site/content/kagent/1.x/observability/tracing.md index d23f8e09..3f5f6ef1 100644 --- a/docs-site/content/kagent/1.x/observability/tracing.md +++ b/docs-site/content/kagent/1.x/observability/tracing.md @@ -9,7 +9,7 @@ A trace records one agent request as a tree of timed spans, so you can see where ## About trace coverage -A trace follows a W3C Trace Context header that the controller passes along with each request. The following diagram traces one request from the caller to the agent runtime. +A trace follows a W3C Trace Context header that the controller passes along with each request. The following diagram shows the path of one request, from the caller to the agent runtime.

```mermaid @@ -39,15 +39,15 @@ flowchart LR A caller reaches the gRPC API on the kagent controller, which starts the trace. The controller hands the request to its A2A gateway, which opens an {{< gloss "A2A" >}}A2A{{< /gloss >}} (Agent-to-Agent) connection to the Session's Actor and injects a `traceparent` header into that call. The Agent Substrate router forwards the call to the Worker that runs the Actor, and adds its own spans to the trace. The agent runtime inside the Actor reads the header and continues the same trace, so the model and tool spans it produces hang off the controller's spans rather than starting a trace of their own. -Each hop reports itself as a separate OpenTelemetry (OTel) service. A tracing backend uses these service names to group the spans. +Each hop sends its spans under a separate OpenTelemetry (OTel) service name. A tracing backend groups the spans by those names. - **The controller** reports as `kagent-controller` in the `kagent` service namespace. Its spans also carry the pod, node, and namespace that the controller runs on. -- **The Agent Substrate router** reports as two services, because its pod runs two containers. The router's own spans, such as its lookup of the Actor for a request, report as `atenet-router`. The spans of the agentgateway proxy that forwards the request to the Worker report as `agentgateway`. Only the `agentgateway` spans join the agent request trace. The `atenet-router` spans form separate [Agent Substrate traces](#agent-substrate-traces). -- **Each agent runtime** reports as its own service, named for the {{< gloss "Agent" >}}Agent{{< /gloss >}} it was compiled from. The `my-first-agent` Agent reports as `my-first-agent`. This is a change from kagent 0.x, where every agent reported under one `kagent` service. A backend that you filter by service now shows one entry for each Agent, and adding an Agent adds a service. +- **The Agent Substrate router** runs the agentgateway proxy, and reports as `agentgateway`. +- **Each agent runtime** reports as its own service, named for the {{< gloss "Agent" >}}Agent{{< /gloss >}} it was compiled from. The `my-first-agent` Agent reports as `my-first-agent`. In kagent 0.x, every agent reported under a single `kagent` service. A backend that you filter by service now lists one entry for each Agent, so the list grows as you add Agents. ### Harness coverage -The controller passes its tracing configuration to every harness and adds the collector to the Actor's egress allowlist, but the spans that each one sends depend on its own instrumentation. For the available harnesses, see [Choose a harness]({{< link path="agents/agent-harness#choose-a-harness" >}}). For how the allowlist is built, see [Egress control]({{< link path="about/substrate-runtime/networking-and-egress#policy-generation" >}}). +The controller passes its tracing configuration to every harness, and adds the collector to the Actor's egress allowlist. Which spans a harness then sends depends on its own instrumentation. For the available harnesses, see [Choose a harness]({{< link path="agents/agent-harness#choose-a-harness" >}}). For how the allowlist is built, see [Egress control]({{< link path="about/substrate-runtime/networking-and-egress#policy-generation" >}}). | Harness | What the trace shows | | ------- | -------------------- | @@ -57,7 +57,7 @@ The controller passes its tracing configuration to every harness and adds the co ### Spans -The `kagent` runtime creates the same spans for every agent, and most span names describe the operation rather than the agent. The `invoke_agent` span is the exception, because its name carries the name of the agent that ran. To narrow a search to one agent, filter by service name rather than by span name. The following spans appear in nesting order, from the span that accepts the request down to the model and tool calls that serve it. +The `kagent` runtime creates the same spans for every agent, and most span names describe the operation rather than the agent. The `invoke_agent` span is the exception, because its name carries the name of the agent that ran. To narrow a search to one agent, filter by service name rather than by span name. The following table lists the spans in nesting order, starting with the span that accepts the request. | Span | When it is created | | ---- | ------------------ | @@ -81,7 +81,7 @@ Use attributes, not span names, to identify a request in a trace. The runtime co | `gen_ai.main_agent.id` | Resource | The main Agent that the runtime process was compiled for, as `/`. This resource attribute replaces the resource-level `gen_ai.agent.id` name. | | `kagent.runtime` | Resource | The runtime that served the request, such as `adk-go`. | -The runtime also adds each scalar value in the A2A message's metadata as an `a2a.message.metadata.` attribute, so a client can tag a request and search for it later. Unlike the correlation attributes, these tags stay on the `a2a.request` span alone, so a search on one returns that span instead of the whole subtree. +The runtime also adds each scalar value in the A2A message's metadata as an `a2a.message.metadata.` attribute, so a client can tag a request and search for it later. These tags stay on the `a2a.request` span alone, unlike the correlation attributes. A search on one therefore returns that span rather than the whole subtree. > [!WARNING] > When the `config.runtimeDefaults.telemetry.captureContent` Helm setting is `true`, prompts and replies reach your tracing backend. On the `kagent` runtime, the `generate_content` span of each model call then carries the conversation as the `gen_ai.input.messages` and `gen_ai.output.messages` attributes, and the Agent's system prompt as `gen_ai.system_instructions`. The setting defaults to `false`, which omits all three. @@ -92,11 +92,13 @@ The runtime also adds each scalar value in the A2A message's metadata as an `a2a 1. [Install kagent]({{< link path="setup/installation" >}}). 2. [Create your first agent]({{< link path="get-started/your-first-agent" >}}), so that you have an {{< gloss "Agent" >}}Agent{{< /gloss >}} to send a request to. That guide also installs the kagent CLI. The steps on this page need the {{< reuse "kagent-docs/versions/kagent.md" >}} CLI, because the CLIs of other releases, newer ones included, do not have the Session commands that these steps use. To check your version, run `kagent version`. -3. Set up a tracing backend. The [OTel stack]({{< link path="observability/otel-stack" >}}) sends traces to Tempo, and the [Lightweight OTel stack]({{< link path="observability/lightweight-otel-stack" >}}) sends traces to Jaeger. Both guides turn on tracing for you, so you can skip to [Review a trace](#review-a-trace). +3. Set up a tracing backend. + + The [OTel stack]({{< link path="observability/otel-stack" >}}) sends traces to Tempo, and the [Lightweight OTel stack]({{< link path="observability/lightweight-otel-stack" >}}) sends traces to Jaeger. Both guides turn on tracing, so you can skip to [Review a trace](#review-a-trace). ## Enable tracing -Tracing is off by default. Turning it on is a Helm change, because the controller reads its tracing configuration from the environment and passes that configuration to the agent runtimes that the controller starts. The following steps send traces to the collector that both stack guides install. To send traces to another OTLP backend, change the endpoint. +Tracing is off by default. Turning it on is a Helm change, because the controller reads its tracing configuration from the environment and passes that configuration to the agent runtimes that the controller starts. The following steps send traces over OTLP to the collector that both stack guides install. To send traces to another OTLP backend, change the endpoint. 1. Save the current revision of your Agent. A later step uses it to tell when kagent recompiles the Agent with the new settings. The command first waits for any recompile that is still in progress, such as one from an earlier Helm upgrade, so that it saves a finished revision. ```bash @@ -231,17 +233,18 @@ Send a request to a new Session, then find its trace in the backend that you set ## Agent Substrate traces -Agent Substrate records traces for its own work, such as scheduling an Actor onto a Worker and restoring it from a snapshot. These traces are separate from the agent request trace. Agent Substrate traces do not share the request trace's ID, so a request trace does not show how long the Actor took to resume. To investigate a slow start, look up the Agent Substrate traces from the same time window, or read the Actor's [suspend and resume records]({{< link path="observability/substrate-telemetry#suspend-and-resume-records" >}}), which carry the trace ID of each operation. +Agent Substrate records traces for its own work, such as scheduling an Actor onto a Worker and restoring it from a snapshot. These traces are separate from the agent request trace, and do not share its ID. The router's `agentgateway` spans are the exception, and join the request trace. The `POST /*` span covers the whole request at the router, including any wait for the Actor to resume, but does not show which step of the resume took the time. To investigate a slow start, look up the Agent Substrate traces from the same time window, or read the Actor's [suspend and resume records]({{< link path="observability/substrate-telemetry#suspend-and-resume-records" >}}), which carry the trace ID of each operation. | Service | Reports | | ------- | ------- | -| `atenet-router` | Requests that the router receives, and its calls to `ateapi` to find or resume the Actor for each request. | | `ateapi` | Actor lifecycle operations, such as create, resume, and suspend, and the scheduling of Actors onto Workers. | | `atelet` | Work on a Worker's node, such as restoring an Actor from a snapshot. | | `ateom-gvisor` | Work inside the sandbox that runs the Actor. | | `atecontroller` | Reconciliation of Agent Substrate resources, such as WorkerPools. | -Agent Substrate exports traces only when its Helm release sets `otel.endpoint`, and it keeps 1% of its traces by default. To keep more, raise `otel.traces.samplingRatio`, as the stack guides do. For the steps, see [Send Agent Substrate telemetry to the collector]({{< link path="observability/otel-stack#send-agent-substrate-telemetry-to-the-collector" >}}). +Agent Substrate exports traces only when its Helm release sets `otel.endpoint`, and it keeps 1% of its traces by default. To keep more, raise `otel.traces.samplingRatio`. + +For the steps, see [Send Agent Substrate telemetry to the collector]({{< link path="observability/otel-stack#send-agent-substrate-telemetry-to-the-collector" >}}). ## Traces from a suspended Actor