Skip to content

feat(builtins): add markkit and fetchkit builtins with the upstream CLI - #2803

Open
chaliy wants to merge 5 commits into
mainfrom
claude/project-thread-solgcd
Open

chaliy wants to merge 5 commits into
mainfrom
claude/project-thread-solgcd

Conversation

@chaliy

@chaliy chaliy commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

What changed

Two new opt-in Cargo features, markkit and fetchkit (both off by default), make markkit and fetchkit available as commands inside a bashkit session.

  • Same CLI as the standalone tools. Each builtin calls the upstream run(args, host) library entry point (markkit-cli 0.3, fetchkit-cli 0.7), the same function the real binaries call. Flags, --help, --version, error text, output and exit codes are identical; bashkit parses nothing.
  • Sandbox boundary. Files come from the VFS (resolved against cwd), stdin from the pipeline. Every HTTP hop, including redirects, fetchkit API calls and discovery probes, goes through the per-Bash HttpClient: allowlist, SSRF precheck, credential injection, bot-auth signing, hooks and host HttpTransport all apply. No second HTTP/TLS stack is linked (no aws-lc).
  • Fail closed on host-only features. --ocr, --render-rakers, --allow-env-proxy, --bot-auth-key/-agent, --allow-remote-conversion and mcp still parse and show in help, then exit 1 with Error: <flag> is not available in bashkit.
  • Limits only go down. markkit --max-*/--timeout can lower the default limits, never raise them, so one tenant can't buy a bigger budget in a shared process.
  • Not on wasm32. Docs: rustdoc guide markkit_fetchkit_guide (crates/bashkit/docs/markkit-fetchkit.md, also the site page); runnable example examples/markkit_fetchkit.rs (in the CI Examples job); knowledge: knowledge/runtimes/markkit-fetchkit-builtins.md, TM-NET-030, TM-DOS-139, L-DOC-001.

Why

Agents want to turn documents and web pages into LLM-ready Markdown from inside the sandbox, using the exact CLI they already know.

Before / After

Before (any build):

$ markkit extract --file /doc.html
bash: markkit: command not found

After (--features markkit,fetchkit):

$ echo '<h1>Hi</h1><p>x</p>' | markkit extract -
# Hi

x
$ fetchkit fetch https://x.test --render-rakers
Error: --render-rakers is not available in bashkit   (exit 1)
$ fetchkit fetch https://example.com/   # no network configured
Error: network access not configured (see BashBuilder::network)

Local results: 24 integration tests (kit_builtin) and 13 lib tests (limit clamping + TM-INF-022 no_leak_*) pass; clippy -D warnings clean for markkit, markkit,fetchkit,http_client; cargo deny check ok; cargo tree -i aws-lc-sys empty; check-doc-links and check-okf ok. A new CI step runs the builtin tests.

Live smoke (LIVE_URL=... cargo run --example markkit_fetchkit --features markkit,fetchkit), all exit 0:

  • https://example.com: fetchkit frontmatter + body; markkit Markdown
  • https://docs.rs/tokio/latest/tokio/: full page
  • W3C dummy.pdf: fetchkit extraction_method: "pdf_inspector", markkit **Dummy PDF file**
  • https://arxiv.org/pdf/1706.03762 with only arxiv.org allowed: refused, URL not in allowlist: https://export.arxiv.org/api/query?... (fetchkit's arXiv API subrequest goes through the allowlist too)

Risk

  • Low: both features are off by default; default builds and dependency graphs are unchanged.
  • With the features on: new dependency trees (markkit, fetchkit). Turning them on in bashkit-cli/Python/Node is a separate follow-up.

Checklist

  • Tests added or updated
  • Backward compatibility considered

&lt;source media="(prefers-color-scheme: dark)" srcset="https://pr-comments-assets.blacksmith.sh/codesmith/view-with-codesmith-dark-v3.svg"&gt;&lt;source media="(prefers-color-scheme: light)" srcset="https://pr-comments-assets.blacksmith.sh/codesmith/view-with-codesmith-light-v3.svg"&gt;&lt;img alt="View with [code]smith" src="https://pr-comments-assets.blacksmith.sh/codesmith/view-with-codesmith-dark-v3-3x.png" width="162" height="28"&gt; <a href="''''https://backend.blacksmith.sh/track/enable-autofix?expires=1796095440&installation_model_id=447267&pr_number=2803&ref=codesmith_pr_footer&repository=everruns%2Fbashkit&return_to=https%3A%2F%2Fgithub.com%2Feverruns%2Fbashkit%2Fpull%2F2803&signature=0d67cfb63b786a8948d93a170a9cfdb00f21e0ab10d7da94f4814c8de71f870a"&gt;&lt;picture&gt;&lt;source media="(prefers-color-scheme: dark)" srcset="https://pr-comments-assets.blacksmith.sh/codesmith/autofix-with-codesmith-dark.svg"&gt;&lt;source media="(prefers-color-scheme: light)" srcset="https://pr-comments-assets.blacksmith.sh/codesmith/autofix-with-codesmith-light.svg"&gt;&lt;img alt="Autofix with [code]smith" src="https://pr-comments-assets.blacksmith.sh/codesmith/autofix-with-codesmith-dark-3x.png" width="190" height="28">````
Need help on this PR? Tag @codesmith-bot with what you need.


Generated by Claude Code

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
bashkit 541653b Commit Preview URL

Branch Preview URL
Nov 01 2026, 05:07 AM

@chaliy chaliy self-assigned this Nov 1, 2026

chaliy commented Nov 1, 2026

Copy link
Copy Markdown
Contributor Author

Fuzz Compile Check is failing, and it isn't this PR's. cargo fuzz build uses an unpinned nightly. On rustc 1.101.0-nightly (f9ceb4a4d 2026-10-31) the bashkit lib fails with error[E0391]: cycle detected when computing when parser::ast::CommandList needs async drop, which is a new nightly regression. I reproduced it on unmodified origin/main with cargo +nightly check -p bashkit --lib. The same command passes on nightly-2026-10-30. This PR doesn't touch parser/ast.rs or the default feature set.

No fix exists yet. The proposed workaround is to pin the fuzz job to nightly-2026-10-30 until upstream fixes it, as a separate PR so this one stays focused. The job already failed on two heads (8d91f87, f07e0cc), so I'm not re-running it.


Generated by Claude Code

@chaliy chaliy mentioned this pull request Nov 1, 2026
2 tasks done
chaliy and others added 4 commits November 1, 2026 04:14
Opt-in `markkit` and `fetchkit` features (off by default) register both
tools as builtins. Each calls the upstream `run(args, host)` library
entry point, so flags, help, errors, output and exit codes match the
standalone binaries. Files come from the VFS; every HTTP hop goes
through the sandbox HttpClient (allowlist, SSRF precheck, credential
injection, signing). Host-only flags (`--ocr`, `--render-rakers`,
proxies, bot-auth keys, remote conversion) and `mcp` parse but fail
closed. markkit limits can only be lowered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wy7MByPXsW4jHxfLeTfGEc
…cargo-vet

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wy7MByPXsW4jHxfLeTfGEc
@chaliy
chaliy force-pushed the claude/project-thread-solgcd branch from 17efc2d to faf18d4 Compare November 1, 2026 04:15
Self-checking example (VFS file, stdin + jq, URL with redirect, fetchkit
Markdown and JSON, allowlist denial, refused flag) over an in-process
HttpTransport, run in the CI Examples job; LIVE_URL adds a real fetch.
The user guide moves to crates/bashkit/docs and is embedded as
markkit_fetchkit_guide with a compiled doctest.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant