Conversation
The `--push` upload drops the config fields that hold credentials (client headers, harness env) from the episode projection and replaces every credential value the run knows — client API keys, credential-named header / harness / host environment values, and the per-rollout interception tokens — wherever it appears in the serialized samples. Redaction is exact-match only; nothing is guessed from the shape of the text. Rollouts record their interception tokens on `Trace.upload_secrets`, which rides the eval wire but is excluded from disk and stdout like the raw tensors.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 11eb4d611d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
ApprovabilityVerdict: Not approved Macroscope's review found this PR not approvable — This PR introduces a broad credential-protection path spanning rollouts, retries, serialization, runtimes, and platform uploads. Since it changes how sensitive credentials are collected and removed from outbound data, security-focused human review is required. You can add or adjust custom eligibility rules. Learn more. |
Task.runtime_env() values are live-only and never traced, so credential-named ones are recorded on Trace.upload_secrets at rollout time like the interception tokens. The redactor also matches the `\/` escape other JSON encoders emit for `/`, and moves to utils/redact.py so rollout.py can share the name filter without importing the platform client.
Mirrors prime#888: each JSON string is decoded before matching, so every valid escape spelling of the outer string is covered and one more nesting level of quoted JSON documents is reached; strings without a hit keep their exact bytes.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9b195c7454
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ime credentials The redactor no longer enumerates escape spellings: each JSON string is decoded and then searched again for quoted JSON inside it, so any escape at any nesting depth is matched and the code shrinks. Every outbound request body goes through it, not only the samples. Three more known-value sources: `Rollout.open()` records the client key and credential-named header, harness, and task runtime variables as they are while the harness runs (a key rotated before `--push` is still redacted); `known_secrets` scans environment-like task-data fields (Harbor's `env`, `verifier_env`); and `Runtime.secrets` lets a runtime expose credentials it minted — the Docker egress proxy token — which the rollout records before the harness session opens.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a599d34932
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
`env_credentials` is now the one rule for what counts as a credential in an environment-like mapping: values under credential-like names, and the password (or bare user token) inside a `scheme://user:password@host` value whatever its name — DATABASE_URL, an authenticated HTTP_PROXY. The rollout capture and the push-time scan both use it, so the URL keeps its shape and only the secret goes.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7a3efefd1a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 409e33375b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
`env_mappings` walks the task-data dump so a nested `verifier.env` is scanned like the top-level `env`; `url_credentials` is split out of `env_credentials` so each client's base_url goes through the same userinfo rule.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0fa49ff4ad
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…s behind Agent and episode retries prepend the failed attempts' errors onto the final trace or episode; those errors can quote that attempt's minted tokens, so the attempts' upload_secrets now ride along. Episode.upload_secrets holds them at the episode level, on the wire but off disk like the trace field.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 096bfa2. Configure here.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bc4ed8ad7b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
known_secrets now dumps config.env alongside the client, agent configs, and task data, and treats `headers` mappings like `env` mappings, so a task config's headers (NeMo Gym) or a seat's client headers are known values even though the trace itself never carries them. The explicit client-headers term is subsumed.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e732474f25
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
url_credentials applies the SECRET_NAME rule to a URL's query string, so `?token=…` authentication is a known value like userinfo is. Only values that are URLs (scheme://host) are parsed, so prose containing `?token=` is left alone.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f47a72ad29
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… marker (mirrors prime#888)
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7d416e12cd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 18cac2cbdb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9635b1530b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…e PAT names known_secrets returns only the run-wide set; build_samples adds each episode's upload_secrets to a per-sample redactor, so the pattern every string is searched with no longer grows with the number of rollouts. PAT joins the credential words (GITHUB_PAT), with the segment boundary keeping PATH out.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8e85d7666a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 40cd0cb5f8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
1 participant
Overview
Replaces #2475 with a from-scratch, minimal implementation of the same goal: credentials never reach the Platform. No
prime-evalsdependency, no release ordering, no sidecar files.What
--pushdoes nowStructural. The uploaded episode projection (
UPLOAD_EXCLUDE) is the disk record minus the two config fields that hold credentials:client.headersandharness.env.harness.forward_envonly names variables and stays.By value. Every credential the run knows is replaced with
[REDACTED]in every outbound request body:env,*_env,headers) anywhere in the run's env config, the traced agent configs, and the task data (a client's headers, a harnessenv, a task config'sheaders, Harbor'sverifier_envand nestedverifier.env) — values under credential-like names (KEY|TOKEN|SECRET|PASSWORD|CREDENTIAL|COOKIE|AUTHORIZATION|AUTH|SIGNATURE|SIG|PATas the name's last word, optionally numbered — compound names are head-final, soHF_TOKEN,X-Api-Key,PGPASSWORD,API_KEY_2are credentials whileTOKEN_URL,COOKIE_DOMAIN,KEY_FILE,SSH_AUTH_SOCK,KEYCLOAK_REALMare not); plus URL credentials anywhere in the traced agent configs and task data (a clientbase_url, a harnesscdp_url, a task's connection string), plus, for any value that is a URL (scheme://host…, never prose), the password (or bare user token) of its userinfo and its credential-named query values (DATABASE_URL, an authenticatedHTTP_PROXY,wss://…?token=…); a value that is a JSON object is a mapping too and gets the same rules inside it (DOCKER_AUTH_CONFIG, a service-account blob); 8+ charsTrace.upload_secrets(rides the eval wire, excluded from disk and stdout like the raw tensors): its interception tokens, the shared-tool URL signatures and any external tool URL's credentials (tool endpoints are not traced), credentials the runtime minted (Runtime.secrets, e.g. the Docker egress proxy token), and the client key plus credential-named header, harness, and task runtime variables as they were while the harness ran, so a key rotated before--pushis still redacted. A retried attempt's secrets ride with the errors it leaves behind (Agent.run,run_episode_with_retryviaEpisode.upload_secrets)Redaction is exact-match only, over the serialized JSON text: each JSON string is decoded and then searched again for quoted JSON inside it, recursively, so any escape spelling at any nesting depth is matched. Only JSON strings are touched, so numbers and structure cannot be corrupted, and a string without a hit keeps its exact bytes. Nothing is guessed from the shape of the text, so ordinary content is never rewritten.
Each sample is redacted with the small run-wide set plus its own rollout's
upload_secrets, so the pattern every string is searched with does not grow with the number of rollouts. Saved traces are unchanged, so the run stays reproducible from its output dir. The log reports how many occurrences were redacted.Deliberately not carried over from #2475
apiKeys, escaped quotes, …) and another false-positive guard (token_usage,oauth, schema context). That is a scanner's job; we redact what we know. pi-share-hf takes the same position and leaves generic detection to trufflehog.--pushredacts everything except the values only the earlier attempt's rollouts knew: its interception and runtime tokens (dead with that run) and credentials rotated since; the module docstring says so.resolve_headers/ team-id relocation, theprime-evalsclient,run_shieldedcall-site moves: unrelated to keeping secrets out.Split with the prime CLI
Verifiers redacts what only it knows (its config, its tokens). User-supplied values live in the prime CLI: the companion PR PrimeIntellect-ai/prime#888 adds
--secrettoprime traces uploadandprime eval push, plus the same environment scan, using the same ~40-line redactor. The two copies are intentionally independent so neither repo waits on the other's release.Test
test_push_traces_uploads_redacted_projectiondrivespush_tracesagainst a mock transport: headers/env dropped, every known secret gone from the body (echoed plainly, inside a quoted tool result with\/and uppercase-hex escapes, and two documents deep), a credential inside the task'sverifier_envreplaced while its key and non-credential neighbours stay, ordinary and short values kept, disk record untouched.Note
High Risk
Changes the security boundary for eval uploads; missed or incorrect redaction could still expose secrets in platform payloads, and secrets shorter than 8 characters are intentionally not redacted.
Overview
Platform
--pushno longer ships raw credentials. Uploads use a stricter episode projection (dropsclient.headersandharness.env) and run every outbound JSON body through exact-match redaction to[REDACTED], including secrets nested inside quoted tool-result strings.Rollouts now record what to scrub.
Trace.upload_secrets(andEpisode.upload_secretson retries) captures API keys, env/header/task runtime credentials, interception and tool URL tokens, and runtime-minted secrets (e.g. Docker egress proxy). Those lists ride the eval wire but are stripped from disk,traces.jsonl, and CLI JSON viaEXCLUDE_FIELDS/EPISODE_EXCLUDE_FIELDS.New
verifiers/v1/utils/redact.pydiscovers credential values from env-like maps and URLs, thenRedactorrewrites only JSON string leaves. A large integration test asserts the mock upload body is clean while localto_record()stays full-fidelity.Reviewed by Cursor Bugbot for commit 41205cf. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Redact credentials from platform upload bodies and exclude secrets from saved records
upload_secretslists retained in memoryRedactorin redact.py that replaces known secret occurrences inside JSON string tokens at any nesting depth, preserving structure and non-matching contentRedactorbefore uploadEXCLUDE_FIELDS,EPISODE_EXCLUDE_FIELDS,UPLOAD_EXCLUDE) soTrace.to_record,Episode.to_record, episode-file serialization, and CLI trace output omitupload_secrets, client headers, harness environments, and raw node fields from saved/disk recordsupload_secretsfrom discarded retry attempts into the final episode via retries.py and agent.pyRedactor.jsonin redact.py rejects non-finite numbers; any code path that previously uploaded JSON containingNaN/Infinitywill now raise. Oversized serialized samples are rejected against the configured request limit inpush_traces.Macroscope summarized 41205cf.