[davidzhao]

Summary

At each turn, the LLM spans record all of the conversation and all of the system instructions. Thus, the size of the spans increases at each turn of a call.

This change adds an option. When you set the option, the spans record only the input that changed. The model always receives all of the input. The option changes only the telemetry.

await session.start(agent, record={"input_delta": True})

The option is off by default. When the option is off, the spans do not change.

Design

The module telemetry/input_delta.py contains the full description. This is a summary.

Terms

• Record: the list of entries that a span records. On an llm_node span, the record is lk.pii.chat_ctx, with one entry for each chat item. On an llm_request span, the record is gen_ai.input.messages, with one entry for each GenAI message.
• Instructions: the instructions message of the agent. On an llm_request span, the instructions are in gen_ai.system_instructions, not in the record. All other system messages are entries of the record, at their position. This change to gen_ai.system_instructions also applies when the option is off.
• Commit: a generation is committed when its speech is scheduled. A discarded preemptive generation is not committed.
• Parent: the span of the last committed generation that has the same span name.

Procedure for each span

1. The tracker calculates a key for each item of the record: the item ID and a fingerprint of the content.
2. It compares the keys with the keys of the parent, in order. It finds the longest shared prefix.
3. The span records only the entries after this prefix.
4. lk.input.dropped_from_base gives the number of parent entries after this prefix.
5. On an llm_request span, if the instructions are the same as in the parent, the span does not record gen_ai.system_instructions.
6. If the span shares no entries and no instructions with the parent, the span records all of the input.
7. When the generation is committed, the keys of the span replace the keys of the parent.

Rebuild

1. Make the full record of the parent (lk.input.base_span_id). Use this procedure again if the parent also has a parent.
2. Remove the last lk.input.dropped_from_base entries.
3. Add the entries of this span.

If an llm_request span with a parent has no gen_ai.system_instructions, use the instructions of the parent.

Rules

• The comparison does not use the role of an item. A system message that stays at its position is shared. The framework adds the expressive guide again at the end of each reply. Thus, the old guide is dropped, and the new guide is recorded.
• An edited, removed or moved item stops the shared prefix at that item. The span records the entries from that item. It does not record all of the input.
• In gen_ai.input.messages, consecutive tool calls of an assistant turn are one message, also when a skipped item (for example, a config update) is between them. The prefix never stops inside such a message.
• On an llm_node span, the instructions are the first entry of lk.pii.chat_ctx. Thus, a change of the instructions causes a full record of lk.pii.chat_ctx.
• The fingerprint does not include media data. It uses the image ID and the audio transcript.
• Each AgentActivity has its own tracker. After a handoff, the first span records all of the input.
• The span of the FallbackAdapter wrapper does not record the input. Only the span of the provider records the input.

Example (expressive mode)

Generation	Input sent to the model	Record of llm_node	lk.input.dropped_from_base
Turn 1	[I, u1, G]	[I, u1, G] (full)	none
Turn 2	[I, u1, a1, u2, G]	[a1, u2, G]	1
Turn 3, tool call	[I, …, u2, a2, u3, G]	[a2, u3, G]	1
Turn 3, tool reply	[I, …, u3, G, fc, fo]	[fc, fo]	0

Attributes on a delta span: lk.input.delta, lk.input.base_span_id (and a span link to the parent), lk.input.dropped_from_base.

Relation to #7539

This change replaces the delta mode of #7539. It keeps these items from #7539: content fingerprints, no input on delegating spans, and the test scenarios.

This change is different from #7539 in these items:

• Each delta span identifies its parent span and the number of dropped entries. Thus, you can make the full input again.
• Only committed generations change the baseline. Preemptive generations and other LLM calls do not change the baseline.
• The span does not record the system instructions again when they do not change.
• The option applies to one session. It does not apply to all of the process.
• lk.pii.chat_ctx stays on the span as a delta. feat: add opt-in telemetry content export controls #7539 removes it.

Refactor

Each ChatItem type now has the base class _ChatItemBase. This base class has no fields. Each type declares its essential fields one time in _essential_fields(). ChatContext.is_equivalent compares these fields, and _fingerprint() hashes them. The operation of is_equivalent does not change. New tests in tests/test_chat_ctx.py examine it.

Tests

tests/test_input_delta.py contains unit tests for the tracker. It also contains session tests: two turns with a tool step, the option off, and a fallback.

[chenghao-mou]

nit: some provider format formats the non-first system message as an user message, so the otel data do not reflect the actual request. Example: a generate_reply(instructions=...) greeting, [I, greet]:

telemetry gen_ai.system_instructions: ['be brief', 'greet the user']
anthropic system:                     ['be brief']
anthropic messages:                   [user: '<instructions>greet the user</instructions>']

[chenghao-mou]

lgtm, two nits. Tested in DD: