feat(cli): redact known secrets before trace and eval uploads - #888
feat(cli): redact known secrets before trace and eval uploads#888xeophon wants to merge 27 commits into
Conversation
`prime traces upload`, `prime eval push`, and the auto-push after `prime eval run` replace every known secret value with `[REDACTED]` inside JSON strings before the bytes leave the machine. Known values are credential-named environment variables, the Prime API key, and `--secret` arguments (a literal or a file with one per line). Matching is exact over the serialized JSON, in every JSON spelling, inside strings only; a line with no hit is uploaded byte-identical. Local files are never modified.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 42b9dcbfcc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 42b9dcb. Configure here.
The redactor decodes each JSON string before matching, so every valid escape spelling of the outer string is covered and one more nesting level of quoted JSON documents is reached; strings without a hit keep their exact bytes. `prime traces upload` also redacts `--context` values, and the auto-push after `prime eval run` redacts metrics and task_type with the rest of the payload. Drops a test artifact that was swept into the first commit.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ad008f3212
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The redactor no longer enumerates escape spellings: each JSON string is decoded and then searched again for quoted JSON inside it, so any escape (including `\/` and uppercase hex) at any nesting depth is matched, and the code shrinks. `prime eval push --name` is redacted with the loaded eval data.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bef6ef918b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
`prime eval push` and the auto-push after `prime eval run` pass all of their inputs — identifiers included — through the redactor once before anything is sent, so the whole outbound request is covered with no per-field carve-outs. A secret inside an identifier makes the request fail rather than leak it.
`env_credentials` is the one rule for what counts as a credential in the process environment: values under credential-like names, and the password (or bare user token) inside a `scheme://user:password@host` value whatever its name — DATABASE_URL, an authenticated HTTP_PROXY. Mirrors verifiers#2508.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0d1dcb5be6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
A percent-encoded URL password is registered as written and decoded, since a client echoes the decoded form. `--secret` probes the filesystem with `os.path.isfile`, which is False for anything that cannot be a path, so a literal longer than a filename no longer raises.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7d3b1e443c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f940c459a3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 74e4d149e5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bb91a03ddf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 80b39fe3fe
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7abb41f154
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…the marker Credential-named query values register their unquote_plus spelling too. A secret that begins with the marker's tail or ends with its head could be formed by a replacement, so overlaps_marker extends the inside-the-marker check: such a discovered value is skipped, an explicit --secret is refused.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 834dda9c90
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 37f934f37f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Only a value inside the marker is a placeholder to skip (or refuse when explicit). A credential that contains or borders the marker was being dropped from the known set, which uploads it for certain; registering it can miss only an instance a replacement forms around another registered secret.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 05cdc53b19
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 564c7b9f27
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 16ae2a08d1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 09d6404f18
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Compound names are head-final: TOKEN_URL names a URL, COOKIE_DOMAIN a domain, KEY_FILE a file. The credential word therefore has to be the last segment (optionally numbered, API_KEY_2), which generalises the AUTH rule to every word and drops metadata values without an exclusion list.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bf4475fcd8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| console.print(f"\n[blue]Uploading evaluation results, using upstream: {env_identifier}[/blue]") | ||
|
|
||
| api_client = APIClient() | ||
| redactor = Redactor(known_secrets(api_client.api_key)) |
There was a problem hiding this comment.
When prime eval run is invoked with --api-key-var CRED (or any variable whose name does not match SECRET_NAME), _add_default_inference_and_key_args recognizes that variable as the inference credential, but this redactor is seeded only with the Prime platform key and heuristic environment discovery. If a saved result or log contains the inference key, the automatic upload therefore sends it unchanged; pass the selected variable's value through to push_eval_results_to_hub and register it explicitly.
Useful? React with 👍 / 👎.
1 participant
Overview
Replaces #882 with a from-scratch, minimal implementation of the same goal: secrets do not leave the machine on upload. One ~75-line util in the CLI instead of a 973-line preflight module plus SDK changes.
What it does
prime traces upload,prime eval push, and the auto-push afterprime eval runreplace every known secret value with[REDACTED]before the bytes go out — in the JSONL lines and--contextvalues, in the eval data and--name, and in the auto-push's metadata, metrics, and task type. Known values are:KEY|TOKEN|SECRET|PASSWORD|CREDENTIAL|COOKIE|AUTHORIZATION|AUTH|SIGNATURE|SIG|PATas the name's last word, optionally numbered — compound names are head-final, soHF_TOKEN,X-Api-Key,PGPASSWORD,API_KEY_2are credentials whileTOKEN_URL,COOKIE_DOMAIN,KEY_FILE,SSH_AUTH_SOCK,KEYCLOAK_REALMare not), plus, for any value that is a URL (scheme://host…, never prose), the password (or bare user token) of its userinfo and its credential-named query values (DATABASE_URL, an authenticatedHTTP_PROXY,wss://…?token=…); a value that is a JSON object is a mapping too and gets the same rules inside it (DOCKER_AUTH_CONFIG, a service-account blob); 8+ chars--secret <value>or--secret <file with one per line>, repeatable, ontraces uploadandeval pushMatching is exact, over the serialized JSON text: each JSON string is decoded and then searched again for quoted JSON inside it, recursively, so any escape spelling (
\/, uppercase hex, …) at any nesting depth is matched. Only JSON strings are touched, so numbers and structure cannot be corrupted, ordinary text is never rewritten, and a line with no hit is uploaded byte-identical, so content-addressed receipts still replay on rerun. Local files are never modified. The command reports how many occurrences were replaced (.redactedin JSON output).Deliberately not carried over from #882
token_usage,oauth,BearerAuth, …). That is a scanner's job: run trufflehog over the file before uploading if you want it. The CLI redacts what it knows, which is also pi-share-hf's position.push_evaluation, exact batching and async upload rewrites, the fingerprint API in prime-evals. The SDKs stay transports; redaction is a CLI concern, so no prime-evals release is needed and verifiers keeps no dependency on it.traces upload:TracesClient.upload_linesalready exists, so lines are redacted in a generator on the way to the batcher.Companion
PrimeIntellect-ai/verifiers#2508 handles what only verifiers knows (client headers, harness env, resolved API keys, per-rollout tokens) with the same redactor. The two copies are intentionally independent so neither repo waits on the other's release.
Tests
test_redact.pycovers escaping at several depths (including\/and uppercase hex), the number/structure guarantee, and the secret sources.test_traces_command.pyandtest_eval_push.pycheck the commands end to end through Typer with stubbed clients: bytes untouched when nothing matches, secrets from env / API key /--secretliteral /--secretfile /--context/--namegone, local files unchanged.Note
Medium Risk
Changes what leaves the machine on upload (security-sensitive); redaction is best-effort on known values only, so missed secrets or edge-case false negatives remain possible despite extensive tests.
Overview
Adds a small
redacthelper and wires it intoprime traces upload,prime eval push, and the post-run eval auto-push so outbound payloads are scrubbed before they hit the platform.Secret sources are credential-like env vars (plus URL/JSON-nested credentials), the Prime API key, and repeatable
--secret(literal or one-per-line file). Matches are exact inside JSON strings only (including nested quoted JSON), replaced with[REDACTED]; local files stay unchanged. Commands print a yellow notice and JSON output includesredactedcounts where applicable.Traces now stream through
upload_lineswith per-line redaction instead of uploading the raw file bytes unchanged. Eval push redacts loaded eval data and CLI fields such asrun_idandnameso secrets in identifiers cannot leak.Reviewed by Cursor Bugbot for commit bf4475f. Bugbot is set up for automated code reviews on this repo. Configure here.