Conversation
fix(docker): tolerate unavailable IPv6 loopback
The self-hosting guide predated the 0.9.0 secure-by-default release and its quickstarts produced an unreachable server. Verified against a fresh 0.9.2 deployment. self-hosting.md: - Require CRAWL4AI_API_TOKEN in every quickstart (docker run, compose, manual build); explain the loopback-only bind and the resulting "connection reset" on published ports, plus startup-delay troubleshooting - Compose: .llm.env is required; the token must be set inside it (host `export` is not forwarded to the container) - Replace the removed inline-Python hooks API (~600 lines) with the declarative hooks reference: CRAWL4AI_HOOKS_ENABLED flag, action table, /hooks/info, and 0.8.x migration notes - Rewrite /screenshot and /pdf for the artifact flow; note output_path is removed and currently silently ignored - Fix dashboard URL (/dashboard, not /monitor); document the token bar in the playground and dashboard UIs - Fix stale versions (0.8.0 -> 0.9.2); drop the SDK function-hooks example MIGRATION.md: - Token setup instructions for compose (.llm.env) and docker run (explicit -e form); clarify the loopback failure symptom - Document CRAWL4AI_HOOKS_ENABLED and the silent failure modes of legacy hooks.code and output_path .llm.env.example: - Add CRAWL4AI_API_TOKEN with guidance; add CRAWL4AI_HOOKS_ENABLED
fix(docker): preserve failed crawl results
fix(docker): preserve single-url crawl failure details
Signed-off-by: nightcityblade <nightcityblade@gmail.com>
PDFCrawlerStrategy returns stub html (real content comes from PDFContentScrapingStrategy), which the anti-bot checks misread as a block. Add AsyncCrawlResponse.placeholder_html so strategies can declare stub html and both anti-bot call sites skip it.
…al scripts In-page timers never fire on CSP-sandboxed pages (GitHub/HuggingFace raw), hanging crawls 30s-to-forever; waits now run Python-side or only after a consent action actually fired.
Dial via the corporate proxy by CONNECT-to-the-pinned-IP when proxy env vars are set, restoring crawls on proxy-only hosts (discussion #2041) without weakening SSRF/rebinding guarantees.
Remove unconditional setTimeout waits from overlay/consent removal scripts
Don't veto PDFCrawlerStrategy placeholder responses
PDFCrawlerStrategy returns a placeholder response and relies on the scraping strategy for the actual download and extraction. Since #2138 the placeholder is flagged with placeholder_html=True so the anti-bot heuristics skip it, which means an unpaired crawl now succeeds quietly with the placeholder text as content instead of failing loudly. State the pairing as required, document the placeholder_html flag, and note that custom strategies deferring extraction should set it too.
fix(config): make body visibility timeout configurable
The wait result is discarded when ignore_body_visibility is True (the default), so a page whose body never becomes visible — ng-cloak/v-cloak left behind by an app that failed to bootstrap — costs a flat body_visibility_timeout ms on every crawl while still reporting success=True with an empty error_message. Nothing was logged, so the delay could only be found by instrumenting the pipeline, and body_visibility_timeout (added in #2131) was undiscoverable by the users who most needed it. Warn once when the wait times out and its result is ignored, naming the option and the timeout that applied. The strict path (ignore_body_visibility=False) already raises with visibility details, so it stays quiet. Fixes #2144
Review follow-up on three points: force_verbose — AsyncLogger drops a plain warning when verbose is off (async_logger.py:238), and CrawlerRunConfig.verbose overrides the logger at arun() time. Servers and batch jobs run with verbose off, so the warning was suppressed in exactly the deployments that hit this and can least afford a silent 30s. Verified end to end: with verbose=False the warning was invisible before, and prints now. False positives — csp_compliant_wait also returns False when the page evaluation itself fails (page closed, context destroyed by a redirect), which costs no time. Time the wait and only warn when it actually burned its budget, and report the elapsed time rather than the configured timeout so the number matches the delay being explained. Test — select the warning call by message instead of reading the last call, which only passed because text_mode=True happened to skip the wait_for_images warning. Added coverage for force_verbose and for the early-failure path.
fix(crawler): warn when the body-visibility wait times out
fix(docker): add pypdf and honor stream scraping strategy
fix(docker): cap mcp below 2 to keep the v1 low-level API used by mcp_bridge
Update self-hosting & migration guides for 0.9.x
…trategy (#2127) Headless Chromium can't render PDFs inline, so pair the client's PDF scraping strategy with a per-request PDFCrawlerStrategy crawler instead of a pooled browser, as the library documents.
Follow redirects manually in PDFContentScrapingStrategy so the Docker server's validate_url_destination vets the download URL and every hop before fetch, blocking redirects to internal addresses.
…rs (#2127) Vet PDF strategies sent via crawler_configs, close PDF crawlers on stream error paths, and fix the stream test to spy on AsyncWebCrawler instead of the pool.
…y schemes Keep the first proxy env candidate that parses, warn on unparseable/unsupported values, and match NO_PROXY host:port.
…und error handling - docker-compose.yml: move the PID cap to deploy.resources.limits.pids (Compose v5 rejects pids_limit alongside a limits block; same behavior on v2.x). - entrypoint.sh: explain the loopback-only bind when no CRAWL4AI_API_TOKEN is set and how to fix it. - server.py: warn when the removed output_path (screenshot/pdf) or legacy hooks.code is sent - fields are ignored, never executed; hooks status reads "ignored" and /crawl/stream sends X-Hooks-Warning. Make "/" public so the /playground redirect works; data routes stay gated. - schemas.py: capture legacy hooks.code so it can be reported (never run). - playground: check response.ok on the streaming branch, surface server error details, hint at the token bar on 401. - tests: add deploy/docker/tests/test_legacy_compat.py (13 tests).
Per review: enabling CRAWL4AI_HOOKS_ENABLED cannot run inline hook code (removed in 0.9.0), so code-carrying payloads now get a removal message instead of the misleading generic hint.
…ompat tests - output_path back as a deprecated schema field (visible in OpenAPI); hooks.code widened to Optional[Any] - pids test parses compose YAML instead of grepping raw text - security pins replaced with a behavioral no-file-written test - playground: readable 422 errors, non-JSON error bodies handled - entrypoint: JWT flag tied to security.jwt_enabled in config.yml
…v optional Pass the token from the shell (or project .env) via environment, make .llm.env optional with required: false, drop the obsolete version key, and align entrypoint hint and docs
…ining Docker: Chain egress proxy through upstream HTTP(S)_PROXY
Add compose v5 compatibility, legacy-field warnings, UI error fixes
The optional LLM-key lines sat at 5-space indent once uncommented, while the live entry uses 4. Removing the '#' produced 'did not find expected - indicator' and broke every compose command until re-indented by hand. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fix(docker): align commented env var lines with the environment list
…t 7) Pre-0.9 docs pointed at /monitor for the dashboard, which now lives at /dashboard. /monitor is the monitoring API prefix with no page of its own, so a browser landed on a bare 401 with no pointer to the real UI. Redirect the exact path /monitor -> /dashboard and allow it in the auth gate's exact-match public_paths. The /monitor/* API routes - including /monitor/ws and the destructive admin actions - stay gated; making the prefix public would expose them unauthenticated. Tests pin both halves and were mutation-checked: widening the gate to the /monitor prefix fails the gated-route and websocket tests, and dropping /monitor from public_paths fails the redirect test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fix(docker): redirect /monitor to the dashboard UI (#2091, point 7)
Enable PDF scraping through the Docker API
handle_crawl_request builds the per-URL config list on a separate branch from the top-level config, so the url_validator wiring has to be repeated there. #2150 fixed that, but nothing exercised it — the existing pairing tests only go through the top-level path, leaving the guard free to be dropped by a future refactor without a red test. Verified by mutation: removing the guard from api.py fails this test and nothing else. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#2150 switched the PDF download to allow_redirects=False and followed hops by hand so url_validator can vet each one before it is fetched. Each hop used a bare requests.get(), which starts with an empty cookie jar, so a host that sets a cookie and then redirects never gets its own cookie back and answers 403. That is the normal shape for gated and CDN-signed PDFs, and it worked before #2150 because allow_redirects=True carried cookies implicitly. Use one requests.Session for the chain. The SSRF guarantee is unchanged: hops are still validated before the fetch and still not auto-followed. Verified by mutation: reverting session.get to requests.get fails the new test and nothing else. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fix(pdf): carry cookies across manual redirect hops
test(docker): cover the per-URL crawler_configs PDF SSRF guard
Removed unnecessary wait for animations before scrolling.
fix: remove_overlay_elements can remove body if it has a global popup…
…__aenter__ Roll back partial startup in BrowserManager.start() so a failed browser launch stops the already-started driver process before re-raising. Fixes #2155
Clean up leaked Playwright driver when browser launch fails in `__aenter__`
PDFContentScrapingStrategy is an UNTRUSTED_ALLOWED_TYPE but had no field allowlist, so _filter_untrusted_fields fell open and kept its filesystem-write knobs (image_save_dir / save_images_locally). An untrusted request body could steer the PDF image writer to an attacker-chosen directory and write attacker-controlled image-stream bytes there. - Forbid image_save_dir / save_images_locally on PDFContentScrapingStrategy. - Add UNTRUSTED_GLOBAL_FORBIDDEN_FIELDS: a fail-closed backstop of write- and code-bearing arg names checked on every untrusted type, so a future strategy cannot reopen the same class of hole. - extract_images stays allowed (base64-inline, no disk write). - Trusted (SDK/in-process) construction is unchanged. Add regression tests exercising the real from_serializable_dict / CrawlerRunConfig.load(provenance=UNTRUSTED) path. Reported by sec-reex. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> (cherry picked from commit 215bb4c7d30daaf7e044bdc5ebd810754d2d0e82)
…fetch Addresses two advisories in the PDFContentScrapingStrategy download path, which fetches with `requests` outside the browser and so bypassed the server's egress and resource controls. GHSA-q5rj-45vw-vp2g (SSRF, high): - Stop following redirects blindly. Redirects are now resolved by hand with a per-hop destination check (max 5 hops), and the peer IP of the response we read back is validated to close DNS rebinding. - The library exposes injectable validators (set_url_validator / set_peer_ip_validator), defaulting to no-op so plain library use is unchanged. The Docker server wires in egress_broker at boot, giving the PDF path the same non-global-IP policy the browser path already has. GHSA-v2rm-hvrj-2x9q (DoS, medium): - Cap the streamed download at max_pdf_bytes (100 MiB default), enforced on the running total rather than the attacker-controlled content-length. - Cap parsed pages at max_pdf_pages (2000 default) in both process paths. - Ship a non-zero wall_clock_s (300s) default in the Docker config. - Clamp these caps and force extract_images off for untrusted request bodies so a client cannot raise its own limits back to unbounded. Adds tests/unit/test_pdf_download_limits.py (22 tests) covering per-hop validation, rebinding, redirect bounds, byte/page caps, and untrusted clamping. Reported by Nguyen Tran Thanh Lam (https://github.com/c240030). (cherry picked from commit 0cd4fae0d717f9b78694682fcb0abd05e2fa996b)
GHSA-7g3g-vhm6-79f3 (medium). clean_pdf_text_to_html() escaped every sink except the paragraph body, where html.escape() had been commented out. PDF paragraph text is attacker-controlled and flows verbatim into cleaned_html and the crawl JSON, so injected markup such as <img onerror=...> survived and executed when the result was rendered. Re-enable html.escape() on paragraph text. This is the "reliable injection vector" half of the advisory; the primary DOM sink (the Playground innerHTML round-trip) is fixed separately under GHSA-m446-hp3q-qfxp. Adds tests/unit/test_pdf_html_escaping.py asserting < > " ' appear only as entities in PDF cleaned_html. Reported by Nguyen Tran Thanh Lam (https://github.com/c240030). (cherry picked from commit 9b312db8533bf4cc3e0a48ea7d48713e5ca30d36)
forceHighlightElement() reset the response code block with `element.innerHTML = element.textContent`, which re-parsed attacker-controlled crawled content (e.g. a reflected page title in metadata.title) as live HTML and executed it in the operator's authenticated Playground session. Removing the innerHTML round-trip closes the sink; hljs.highlightElement() re-highlights safely from textContent and emits escaped markup. Fixes GHSA-m446-hp3q-qfxp (cherry picked from commit 82950b4a1af26190edfed11b2d779f51750780b8)
Addresses five coordinated-disclosure advisories in the PDF processing and Docker playground paths: - GHSA-xpp7-j28w-2gvx arbitrary file write via PDF image-write fields - GHSA-q5rj-45vw-vp2g SSRF: PDF fetch followed redirects unvalidated - GHSA-v2rm-hvrj-2x9q DoS: unbounded PDF size and page count - GHSA-7g3g-vhm6-79f3 XSS: PDF text unescaped into cleaned_html - GHSA-m446-hp3q-qfxp DOM XSS in the playground leading to API token theft
The release carries 33 commits that landed on develop since 0.9.2 in addition to the security fixes. The notes covered only the security work, which under-reported what shipped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings
mainup to v0.9.3, which is already live on PyPI.What is in this release
Security. Five coordinated-disclosure advisories. They are still drafts and
are published only after this merge, once the Docker image is out. Details go
public with the advisories.
Bug fixes. 33 commits that accumulated on
developsince 0.9.2. Mostly theDocker server, plus a Playwright driver leak on failed browser launch, the PDF
anti-bot false positive, and the overlay-removal fixes. Full list in
docs/blog/release-v0.9.3.md.Release state
v0.9.3developmainNotes for review
origin/mainis an ancestor ofdevelop.4bcd5fa.developis one commit aheadat
7245f98, a docs-only fix that expands the release notes. This is expected.crawl4ai/__version__.pyandDockerfile.SECURITY-CREDITS.md.