chore: update telemetry, add inputTokens and internal deployType

[mckornfield]

Summary

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update
• Refactoring

Testing

• make test passes locally
• make check passes locally (format + lint + typecheck + lock-check)
• Added/updated tests for changes

Documentation

• If docs changed: make docs-build passes locally

Related Issues

[greptile-apps]

Greptile Summary

This PR adds cumulative input-token tracking to NddAdapter and surfaces the count in the telemetry event, adds a new NVIDIA_INTERNAL deployment type, and bumps the schema version from 1.7 to 1.9.

• adapter.py: Introduces _cumulative_input_tokens, a total_input_tokens property, and _add_input_tokens; changes _DataDesignerUsageProbe to always be enabled=True so token data is captured even when no collector is present; caches model-usage into local variables before passing to record_ndd_workflow.
• telemetry.py: Adds DeploymentTypeEnum.NVIDIA_INTERNAL = \"nvidia-internal\", a new input_tokens: int field (default 0), and bumps _schema_version to \"1.9\".
• tests/test_telemetry.py: Updates the schema-version assertion from \"1.7\" to \"1.9\"; no new tests cover inputTokens or the nvidia-internal deployment type.

Confidence Score: 3/5

The success path of run_workflow raises a NameError on every call due to _error_model_usage being referenced before assignment; this was flagged in an earlier review thread but remains unresolved in the current head commit.

Every successful run_workflow invocation hits line 379 (self._add_input_tokens(_error_model_usage)) before _error_model_usage is ever assigned. The resulting NameError is caught by the outer except block and re-raised as AnonymizerWorkflowError, so the entire happy path is currently broken. This regression was identified in a prior review thread but has not been fixed in this commit.

src/anonymizer/engine/ndd/adapter.py — the _add_input_tokens call on line 379 references _error_model_usage before it is defined.

Important Files Changed

Filename	Overview
src/anonymizer/engine/ndd/adapter.py	Adds cumulative input-token tracking; contains a NameError on every successful run path (already flagged in prior review thread) and a potential race condition on _cumulative_input_tokens (also previously flagged).
src/anonymizer/telemetry.py	Adds NVIDIA_INTERNAL enum value, bumps schema version 1.7→1.9, and adds input_tokens field with default 0 (inconsistent with the -1 sentinel used by all other count fields).
src/anonymizer/interface/anonymizer.py	Passes adapter.total_input_tokens into the telemetry event constructor; one-line change, straightforward.
tests/test_telemetry.py	Updates schema-version assertion to 1.9; no new tests for inputTokens or the nvidia-internal deployment type round-trip.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant A as Anonymizer
    participant N as NddAdapter
    participant P as _DataDesignerUsageProbe
    participant T as Telemetry

    A->>N: run_workflow(df, ...)
    N->>P: "__enter__() [always enabled=True]"
    Note over P: patches _create_resource_provider
    N->>N: DataDesigner.create() / .preview()
    P-->>P: captures resource_providers
    N->>P: __exit__()
    Note over P: restores _create_resource_provider

    alt Success path
        N->>P: model_usage() → _success_model_usage
        N->>N: _add_input_tokens(_success_model_usage)
        N->>T: "record_ndd_workflow(model_usage=_success_model_usage)"
        N-->>A: WorkflowRunResult
        A->>A: "build AnonymizerEvent(input_tokens=adapter.total_input_tokens)"
        A->>T: enqueue(AnonymizerEvent)
    else Error path
        N->>P: model_usage() → _error_model_usage
        N->>T: "record_ndd_workflow(model_usage=_error_model_usage)"
        N-->>A: AnonymizerWorkflowError
    end

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant A as Anonymizer
    participant N as NddAdapter
    participant P as _DataDesignerUsageProbe
    participant T as Telemetry

    A->>N: run_workflow(df, ...)
    N->>P: "__enter__() [always enabled=True]"
    Note over P: patches _create_resource_provider
    N->>N: DataDesigner.create() / .preview()
    P-->>P: captures resource_providers
    N->>P: __exit__()
    Note over P: restores _create_resource_provider

    alt Success path
        N->>P: model_usage() → _success_model_usage
        N->>N: _add_input_tokens(_success_model_usage)
        N->>T: "record_ndd_workflow(model_usage=_success_model_usage)"
        N-->>A: WorkflowRunResult
        A->>A: "build AnonymizerEvent(input_tokens=adapter.total_input_tokens)"
        A->>T: enqueue(AnonymizerEvent)
    else Error path
        N->>P: model_usage() → _error_model_usage
        N->>T: "record_ndd_workflow(model_usage=_error_model_usage)"
        N-->>A: AnonymizerWorkflowError
    end

Loading

_{Reviews (3): Last reviewed commit: "chore: move token count outside of lock ..." | Re-trigger Greptile}