Skip to content

Propose fixes for agent errors with Claude Managed Agents - #1756

Closed
kevmo314 wants to merge 2 commits into
mainfrom
autofix-managed-agents
Closed

kevmo314 wants to merge 2 commits into
mainfrom
autofix-managed-agents

Conversation

@kevmo314

@kevmo314 kevmo314 commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Adds autofix/, which turns open Error Reporting groups for rtchat-agent into draft PRs. Claude writes each fix in a Managed Agents sandbox. Error text can carry attacker-controlled content, so the design assumes any session may be hijacked by prompt injection and limits what a hijacked session can do.

Flow. autofix-run.yml runs every 15 minutes on main:

  1. Collects finished sessions: downloads patch.diff from the session outputs, validates it, applies it with git apply on the commit the session started from, pushes to the bot's fork, and opens a draft PR.
  2. Starts a session for one new open error group, with at most 2 sessions in flight. A Firestore autofix/{key} document records each attempt, one per group.

Claude never runs in GitHub Actions. The workflow only runs our code, which treats session output as data.

Safety.

  • Normalized input. The agent only sees validated stack frames inside agent/, plus a message with quoted strings, URLs, long tokens and unusual characters stripped, capped at 500 characters.
  • Nothing to steal in the sandbox. No credentials, since the public repo is mounted without a token. Network is limited to proxy.golang.org and sum.golang.org, web tools are off, and each session has a hard $10 budget.
  • The patch is treated as data. Only edits or additions to .go files under agent/ (not agent/monitoring/) pass. No renames, deletions, mode changes, symlinks or binary content, at most 10 files and 300 changed lines. git apply --numstat must agree with the validator. Git runs with hooks and global config disabled, and nothing from the patch is executed.
  • Locked-down workflow.
    • permissions: {} apart from id-token: write.
    • Secrets live in an autofix environment that only main can use.
    • A dedicated GCP service account is bound to repo:muxable/rtchat:environment:autofix.
    • No Actions cache is written.
    • Logs contain only IDs and outcomes, since they're public. Failure reasons go to Firestore.
  • Fork PRs, drafts only. They get no secrets, and firebase-hosting-pull-request.yml already skips forks. The title comes from validated fields. The agent's summary is shown in a fence so its links, images and mentions don't render.
  • No upstream write access. The bot can't push to this repo.

Not done in this PR. These one-time steps are in autofix/README.md:

  • the bot account and fork
  • ant apply for the agent and environment, which writes claude-lock.json (commit it, since the workflow reads the IDs from it)
  • the GCP service account
  • the autofix GitHub environment

Testing.

  • Unit tests cover the report normalizer (including a </error_report> breakout attempt), the patch validator (20 rejection cases), numstat parsing and PR body fencing.
  • The git apply path was checked by hand against real git apply --numstat -z output.
  • Not exercised against live Managed Agents, Error Reporting or GitHub APIs.

Open questions.

  • I haven't confirmed that the sandbox's Go toolchain can fetch Go 1.26 through proxy.golang.org alone.
  • Finished sessions are left idle, not archived.

@kevmo314

kevmo314 commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

Closing: too much setup for what it buys.

@kevmo314 kevmo314 closed this Nov 4, 2026

This branch was previously deployed

2 inactive deployments
ios-testflight — cdd23d43 Deployed Nov 4, 2026 by kevmo314 via ios / ipa #1879
android-internal — cdd23d43 Deployed Nov 4, 2026 by kevmo314 via android / apkaab #1879
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant