Tunnel Girona
+Portfolio page content that must survive overlay removal.
+Version 0.9.2 Release Highlights - Maintenance Bug Fixes
A maintenance patch release with bug fixes across the dispatcher, Docker, and GPU builds. `MemoryAdaptiveDispatcher` no longer leaks crawl tasks and browser pages when a streaming crawl is closed. Docker fixes cover the Playground "Advanced Config" 400, the Monitor WebSocket 500 under JWT auth, and Playwright headless-shell packaging. `ENABLE_GPU=true` Docker builds no longer fail on the CUDA toolkit. diff --git a/SECURITY-CREDITS.md b/SECURITY-CREDITS.md index e2ae2bc66..a1c12441c 100644 --- a/SECURITY-CREDITS.md +++ b/SECURITY-CREDITS.md @@ -18,3 +18,6 @@ We thank the following security researchers for their responsible disclosure: | UDU_RisePho | GitHub: [hoanggxyuuki](https://github.com/hoanggxyuuki) | Chromium launch-flag RCE class via extra_args (0.9.0) | 2026-06-18 | | Y4tacker | GitHub: [Y4tacker](https://github.com/Y4tacker) | Hook system exec() sandbox escape (MRO chain RCE), Chromium launch-arg injection (--utility-cmd-prefix RCE), HTTP crawler path traversal arbitrary file write | 2026-07-09 | | Rafael | GitHub: [rafaelfiguereod-stack](https://github.com/rafaelfiguereod-stack) | Reported SSRF, LLM key exfiltration, and auth gaps (already fixed / not exploitable in current code) | 2026-07-09 | +| Zhixi "Jace" Sun | Independent security researcher | Arbitrary file write via unconfined PDFContentScrapingStrategy image-write fields in untrusted config bodies (0.9.3) | 2026-08-24 | +| Nguyen Tran Thanh Lam | GitHub: [c240030](https://github.com/c240030) | SSRF via PDF download redirects, DoS via unbounded PDF size and page count, XSS via unescaped PDF text in cleaned_html (0.9.3) | 2026-07-27 | +| e1codes | GitHub: [e1codes](https://github.com/e1codes) | DOM-based XSS in the Docker Playground leading to operator API-token theft (0.9.3) | 2026-07-24 | diff --git a/crawl4ai/__version__.py b/crawl4ai/__version__.py index 505ca152c..f8f1eb5d0 100644 --- a/crawl4ai/__version__.py +++ b/crawl4ai/__version__.py @@ -1,7 +1,7 @@ # crawl4ai/__version__.py # This is the version that will be used for stable releases -__version__ = "0.9.2" +__version__ = "0.9.3" # For nightly builds, this gets set during build process __nightly_version__ = None diff --git a/crawl4ai/async_configs.py b/crawl4ai/async_configs.py index 27320fd4b..383079fef 100644 --- a/crawl4ai/async_configs.py +++ b/crawl4ai/async_configs.py @@ -214,6 +214,29 @@ class UntrustedConfigError(ValueError): "override_navigator", "magic", "process_in_browser", "shared_data", "session_id", }, + # PDFContentScrapingStrategy has no scalar allowlist (it is kept for its + # value fields), so its filesystem-write knobs must be blocked explicitly: + # image_save_dir is an unconfined write destination and save_images_locally + # turns the write on. extract_images stays allowed - it only returns the + # image bytes base64-inline in the response, no disk write. + "PDFContentScrapingStrategy": { + "image_save_dir", "save_images_locally", + }, +} + +# Field names that must NEVER be set from an untrusted body on ANY allowed +# type, checked regardless of per-type allowlist. This is the fail-closed +# backstop: the per-type maps above only cover BrowserConfig/CrawlerRunConfig, +# so a strategy that (now or in future) exposes a filesystem-write, code, or +# routing constructor arg is caught here even with no explicit allowlist. +# Presence => 400 (loud), matching js_code/extra_args behavior. +UNTRUSTED_GLOBAL_FORBIDDEN_FIELDS = { + # filesystem write / read sinks + "image_save_dir", "save_images_locally", "downloads_path", "user_data_dir", + "output_path", "save_path", "file_path", "local_path", "storage_state", + # code / command execution + "js_code", "js_code_before_wait", "c4a_script", "init_scripts", + "code", "command", "hook", "hooks", } # Scalar knobs an untrusted body MAY set, per class. A field not listed here is @@ -244,6 +267,7 @@ class UntrustedConfigError(ValueError): "fetch_ssl_certificate", # timing / waiting "wait_until", "page_timeout", "wait_for", "wait_for_timeout", + "body_visibility_timeout", "wait_for_images", "delay_before_return_html", "mean_delay", "max_range", # scrolling / rendering "ignore_body_visibility", "scan_full_page", "scroll_delay", @@ -272,6 +296,8 @@ class UntrustedConfigError(ValueError): _MAX_TIMEOUT_MS = 60_000 _MAX_SCROLL_STEPS = 1000 _MAX_VIEWPORT = 4000 +_MAX_PDF_BYTES = 100 * 1024 * 1024 +_MAX_PDF_PAGES = 2000 def _filter_untrusted_fields(type_name: str, params: dict) -> dict: @@ -280,7 +306,7 @@ def _filter_untrusted_fields(type_name: str, params: dict) -> dict: allowlist = UNTRUSTED_FIELD_ALLOWLIST.get(type_name) # None => keep all non-forbidden out = {} for key, value in params.items(): - if key in forbidden: + if key in UNTRUSTED_GLOBAL_FORBIDDEN_FIELDS or key in forbidden: raise UntrustedConfigError( f"field '{key}' is not permitted on {type_name} from an untrusted request" ) @@ -299,7 +325,7 @@ def _cap_timeout(v): return min(int(v), _MAX_TIMEOUT_MS) if type_name == "CrawlerRunConfig": - for f in ("page_timeout", "wait_for_timeout"): + for f in ("page_timeout", "wait_for_timeout", "body_visibility_timeout"): if f in params: params[f] = _cap_timeout(params[f]) if isinstance(params.get("max_scroll_steps"), int): @@ -308,6 +334,17 @@ def _cap_timeout(v): for f in ("viewport_width", "viewport_height"): if isinstance(params.get(f), int): params[f] = max(1, min(params[f], _MAX_VIEWPORT)) + elif type_name == "PDFContentScrapingStrategy": + # This class has no field allowlist, so without clamping a body could + # simply raise its own caps back to unbounded and re-open the DoS. + for f, cap in (("max_pdf_bytes", _MAX_PDF_BYTES), ("max_pdf_pages", _MAX_PDF_PAGES)): + if f in params: + v = params[f] + # <=0 or non-int would read as "no limit"; pin to the cap. + params[f] = cap if not isinstance(v, int) or v <= 0 else min(v, cap) + # Rasterizing every page is the most expensive thing this strategy can + # do, and nothing about untrusted crawling needs it. + params["extract_images"] = False return params @@ -1463,6 +1500,8 @@ class CrawlerRunConfig(): Default: False. ignore_body_visibility (bool): If True, ignore whether the body is visible before proceeding. Default: True. + body_visibility_timeout (int): Maximum time in ms to wait for the body to become visible. + Default: 30000. scan_full_page (bool): If True, scroll through the entire page to load all content. Default: False. scroll_delay (float): Delay in seconds between scroll steps if scan_full_page is True. @@ -1640,6 +1679,7 @@ def __init__( c4a_script: Union[str, List[str]] = None, js_only: bool = False, ignore_body_visibility: bool = True, + body_visibility_timeout: int = 30000, scan_full_page: bool = False, scroll_delay: float = 0.2, max_scroll_steps: Optional[int] = None, @@ -1770,6 +1810,13 @@ def __init__( self.c4a_script = c4a_script self.js_only = js_only self.ignore_body_visibility = ignore_body_visibility + if ( + not isinstance(body_visibility_timeout, (int, float)) + or isinstance(body_visibility_timeout, bool) + or body_visibility_timeout <= 0 + ): + raise ValueError("body_visibility_timeout must be a positive number") + self.body_visibility_timeout = body_visibility_timeout self.scan_full_page = scan_full_page self.scroll_delay = scroll_delay self.max_scroll_steps = max_scroll_steps @@ -2137,6 +2184,7 @@ def to_dict(self): "js_code_before_wait": self.js_code_before_wait, "js_only": self.js_only, "ignore_body_visibility": self.ignore_body_visibility, + "body_visibility_timeout": self.body_visibility_timeout, "scan_full_page": self.scan_full_page, "scroll_delay": self.scroll_delay, "max_scroll_steps": self.max_scroll_steps, diff --git a/crawl4ai/async_crawler_strategy.py b/crawl4ai/async_crawler_strategy.py index 265c376e9..6d0fb4769 100644 --- a/crawl4ai/async_crawler_strategy.py +++ b/crawl4ai/async_crawler_strategy.py @@ -812,6 +812,7 @@ async def handle_request_failed_capture(request): await page.wait_for_selector("body", state="attached", timeout=30000) # Use the new check_visibility function with csp_compliant_wait + _visibility_wait_start = time.perf_counter() is_visible = await self.csp_compliant_wait( page, """() => { @@ -823,9 +824,38 @@ async def handle_request_failed_capture(request): style.opacity !== '0'; return isVisible; }""", - timeout=30000, + timeout=config.body_visibility_timeout, ) + _visibility_wait_ms = ( + time.perf_counter() - _visibility_wait_start + ) * 1000 + + # csp_compliant_wait also returns False when the evaluation itself + # fails (page closed, context destroyed by a redirect), which costs + # no time — only warn about a wait that actually burned its budget. + if ( + not is_visible + and config.ignore_body_visibility + and _visibility_wait_ms >= config.body_visibility_timeout * 0.9 + ): + # The wait timed out and its result is about to be discarded, so + # the crawl still succeeds — just this much slower, on every crawl + # of this page. force_verbose because the whole point is that the + # delay is otherwise invisible, and the servers and batch jobs that + # most need to see it run with verbose off (see #2144). + self.logger.warning( + message=( + "Body never became visible after {elapsed}ms — the page may " + "use ng-cloak/v-cloak. This delay is added to every crawl of " + "this page; lower CrawlerRunConfig.body_visibility_timeout " + "to shorten it." + ), + tag="WARNING", + params={"elapsed": round(_visibility_wait_ms)}, + force_verbose=True, + ) + if not is_visible and not config.ignore_body_visibility: visibility_info = await self.check_visibility(page) raise Error(f"Body element is hidden: {visibility_info}") @@ -1534,7 +1564,7 @@ async def remove_overlay_elements(self, page: Page) -> None: }})() """ ) - await page.wait_for_timeout(500) # Wait for any animations to complete + await page.wait_for_timeout(600) # Wait for any animations to complete except Exception as e: self.logger.warning( message="Failed to remove overlay elements: {error}", diff --git a/crawl4ai/async_webcrawler.py b/crawl4ai/async_webcrawler.py index 8216d19bc..a2aabcc41 100644 --- a/crawl4ai/async_webcrawler.py +++ b/crawl4ai/async_webcrawler.py @@ -505,7 +505,7 @@ async def arun( # Check if blocked (skip for raw: URLs — # caller-provided content, anti-bot N/A) - if _is_raw_url: + if _is_raw_url or async_response.placeholder_html: _blocked = False _block_reason = "" else: @@ -625,7 +625,8 @@ async def arun( # empty by design, and is_blocked() would misread "0 bytes # html" as a block. _has_download = bool(getattr(crawl_result, "downloaded_files", None)) - if not _fallback_succeeded and not _is_raw_url and not _has_download: + _placeholder = bool(getattr(async_response, "placeholder_html", False)) + if not _fallback_succeeded and not _is_raw_url and not _has_download and not _placeholder: _blocked, _block_reason = is_blocked( crawl_result.status_code, crawl_result.html or "") if _blocked: diff --git a/crawl4ai/browser_manager.py b/crawl4ai/browser_manager.py index d7b8009c8..f4ab0aa32 100644 --- a/crawl4ai/browser_manager.py +++ b/crawl4ai/browser_manager.py @@ -626,7 +626,15 @@ async def acquire(cls, cdp_url: str, use_undetected: bool = False): else: from playwright.async_api import async_playwright pw = await async_playwright().start() - browser = await pw.chromium.connect_over_cdp(cdp_url) + try: + browser = await pw.chromium.connect_over_cdp(cdp_url) + except BaseException: + # Stop the driver we just started so a failed connect doesn't leak it + try: + await pw.stop() + except BaseException: + pass + raise cls._cache[cdp_url] = (pw, browser, 1) return pw, browser @@ -788,6 +796,25 @@ async def start(self): Note: This method should be called in a separate task to avoid blocking the main event loop. """ + try: + await self._start_impl() + except BaseException: + # Roll back partial startup (e.g. Playwright driver) since a failed __aenter__ never triggers __aexit__ + try: + await self.close() + except BaseException: + pass + if self.playwright is not None: + # close() intentionally skips the driver for external-CDP configs; on failed startup we still own it + try: + await self.playwright.stop() + except BaseException: + pass + self.playwright = None + self.browser = None + raise + + async def _start_impl(self): if self.playwright is not None: await self.close() diff --git a/crawl4ai/js_snippet/remove_consent_popups.js b/crawl4ai/js_snippet/remove_consent_popups.js index 9aac8d345..2f7eb0f34 100644 --- a/crawl4ai/js_snippet/remove_consent_popups.js +++ b/crawl4ai/js_snippet/remove_consent_popups.js @@ -292,6 +292,7 @@ async () => { // ========================================================================= // Phase 2: Try CMP JavaScript APIs // ========================================================================= + let apiCalled = false; // IAB TCF v2 API if (typeof window.__tcfapi === 'function') { @@ -304,6 +305,7 @@ async () => { if (typeof window.Didomi !== 'undefined') { try { window.Didomi.setUserAgreeToAll(); + apiCalled = true; } catch (e) { /* continue */ } } @@ -311,6 +313,7 @@ async () => { if (typeof window.Cookiebot !== 'undefined') { try { window.Cookiebot.submitCustomConsent(true, true, true); + apiCalled = true; } catch (e) { /* continue */ } } @@ -318,6 +321,7 @@ async () => { if (typeof window.Osano !== 'undefined') { try { window.Osano.cm.acceptAll(); + apiCalled = true; } catch (e) { /* continue */ } } @@ -325,11 +329,12 @@ async () => { if (typeof window.klaro !== 'undefined') { try { window.klaro.getManager().acceptAll(); + apiCalled = true; } catch (e) { /* continue */ } } - // Wait for CMP animations/transitions - await new Promise(r => setTimeout(r, 500)); + // Wait for CMP animations/transitions - only when a consent action actually fired + if (accepted || apiCalled) await new Promise(r => setTimeout(r, 500)); // ========================================================================= // Phase 3: Remove known CMP containers by selector diff --git a/crawl4ai/js_snippet/remove_overlay_elements.js b/crawl4ai/js_snippet/remove_overlay_elements.js index a50d94274..c134cc6b1 100644 --- a/crawl4ai/js_snippet/remove_overlay_elements.js +++ b/crawl4ai/js_snippet/remove_overlay_elements.js @@ -5,6 +5,12 @@ async () => { return style.display !== "none" && style.visibility !== "hidden" && style.opacity !== "0"; }; + // Never strip document structure , , or ; removing those tags empties the page. + const isDocumentStructure = (elem) => { + const tag = elem && elem.tagName; + return tag === "HTML" || tag === "HEAD" || tag === "BODY"; + }; + // Common selectors for popups and overlays const commonSelectors = [ // Close buttons first @@ -54,6 +60,7 @@ async () => { // Find elements with high z-index const allElements = document.querySelectorAll("*"); for (const elem of allElements) { + if (isDocumentStructure(elem)) continue; const style = window.getComputedStyle(elem); const zIndex = parseInt(style.zIndex); const position = style.position; @@ -74,6 +81,7 @@ async () => { for (const selector of commonSelectors) { const elements = document.querySelectorAll(selector); elements.forEach((elem) => { + if (isDocumentStructure(elem)) return; if (isVisible(elem)) { elem.remove(); } @@ -88,6 +96,7 @@ async () => { const removeFixedElements = () => { const elements = document.querySelectorAll("*"); elements.forEach((elem) => { + if (isDocumentStructure(elem)) return; const style = window.getComputedStyle(elem); if ((style.position === "fixed" || style.position === "sticky") && isVisible(elem)) { elem.remove(); @@ -114,7 +123,5 @@ async () => { document.body.style.paddingRight = "0px"; document.body.style.overflow = "auto"; - // Wait a bit for any animations to complete document.body.scrollIntoView(false); - await new Promise((resolve) => setTimeout(resolve, 50)); }; diff --git a/crawl4ai/models.py b/crawl4ai/models.py index 506538970..7ab3794cc 100644 --- a/crawl4ai/models.py +++ b/crawl4ai/models.py @@ -338,6 +338,7 @@ class AsyncCrawlResponse(BaseModel): mhtml_data: Optional[str] = None get_delayed_content: Optional[Callable[[Optional[float]], Awaitable[str]]] = None downloaded_files: Optional[List[str]] = None + placeholder_html: bool = False # True if html is a stand-in for the content, actual content is produced by the scraping strategy (eg. using PDFCrawlerStrategy) ssl_certificate: Optional[SSLCertificate] = None redirected_url: Optional[str] = None redirected_status_code: Optional[int] = None diff --git a/crawl4ai/processors/pdf/__init__.py b/crawl4ai/processors/pdf/__init__.py index 69a6f75a2..9fc5ad781 100644 --- a/crawl4ai/processors/pdf/__init__.py +++ b/crawl4ai/processors/pdf/__init__.py @@ -7,7 +7,54 @@ from crawl4ai.content_scraping_strategy import ContentScrapingStrategy from .processor import NaivePDFProcessorStrategy # Assuming your current PDF code is in pdf_processor.py +# Default resource bounds for remote PDF fetches. These cap what an untrusted +# caller can make the process download and parse; see set_url_validator() for +# the destination side of the same problem. +DEFAULT_MAX_PDF_BYTES = 100 * 1024 * 1024 # 100 MiB +DEFAULT_MAX_PDF_PAGES = 2000 +DEFAULT_MAX_REDIRECTS = 5 + +_REDIRECT_STATUSES = (301, 302, 303, 307, 308) + +# Destination-policy hooks, injected by whoever embeds the library. +# +# The library deliberately has no egress policy of its own: as a plain library +# the caller already chooses the URL, so there is nothing to defend against. +# It matters when the URL comes from an untrusted API client, which is the +# Docker server's situation -- deploy/docker/server.py installs the egress +# broker here at boot so the PDF path enforces the same non-global-IP policy +# the browser path already gets via enforce_egress(). +# +# _url_validator runs on every hop (initial URL and each redirect Location). +# _peer_ip_validator runs on the IP actually connected to; without it a name +# that passed validation could still resolve to an internal address on the +# second lookup requests performs (DNS rebinding). +# Both must raise to block; returning anything is treated as "allowed". +_url_validator = None +_peer_ip_validator = None + + +def set_url_validator(fn): + """Install a per-hop destination validator: fn(url) -> None, raises to block.""" + global _url_validator + _url_validator = fn + + +def set_peer_ip_validator(fn): + """Install a connected-peer check: fn(ip: str) -> None, raises to block.""" + global _peer_ip_validator + _peer_ip_validator = fn + + class PDFCrawlerStrategy(AsyncCrawlerStrategy): + """Crawler strategy for PDF documents. + + This strategy does not fetch or parse anything itself — it returns a + placeholder response (``placeholder_html=True``). It MUST be paired with + ``PDFContentScrapingStrategy`` (via ``CrawlerRunConfig.scraping_strategy``), + which performs the actual PDF download and content extraction. With any + other scraping strategy the result will contain only the placeholder text. + """ def __init__(self, logger: AsyncLogger = None): self.logger = logger @@ -16,7 +63,8 @@ async def crawl(self, url: str, **kwargs) -> AsyncCrawlResponse: return AsyncCrawlResponse( html="Scraper will handle the real work", # Scraper will handle the real work response_headers={"Content-Type": "application/pdf"}, - status_code=200 + status_code=200, + placeholder_html=True, # HTML is a placeholder for the actual content, which will be produced by the PDFContentScrapingStrategy ) async def close(self): @@ -53,18 +101,30 @@ class PDFContentScrapingStrategy(ContentScrapingStrategy): ) """ - def __init__(self, + def __init__(self, save_images_locally : bool = False, extract_images : bool = False, image_save_dir : str = None, batch_size: int = 4, - logger: AsyncLogger = None): + max_pdf_bytes: int = DEFAULT_MAX_PDF_BYTES, + max_pdf_pages: int = DEFAULT_MAX_PDF_PAGES, + max_redirects: int = DEFAULT_MAX_REDIRECTS, + logger: AsyncLogger = None, + url_validator=None): self.logger = logger + # Per-instance validator, set by deploy/docker/api.py on the strategy it + # builds for a request. It runs in addition to the module-level policy + # installed at server boot, not instead of it. + self.url_validator = url_validator + self.max_pdf_bytes = max_pdf_bytes + self.max_pdf_pages = max_pdf_pages + self.max_redirects = max_redirects self.pdf_processor = NaivePDFProcessorStrategy( save_images_locally=save_images_locally, extract_images=extract_images, image_save_dir=image_save_dir, - batch_size=batch_size + batch_size=batch_size, + max_pages=max_pdf_pages ) self._temp_files = [] # Track temp files for cleanup @@ -138,55 +198,159 @@ async def ascrap(self, url: str, html: str, **kwargs) -> ScrapingResult: return await asyncio.to_thread(self.scrap, url, html, **kwargs) + def _discard_temp_file(self, path: str) -> None: + Path(path).unlink(missing_ok=True) + if path in self._temp_files: + self._temp_files.remove(path) + + @staticmethod + def _peer_ip(response): + """Best-effort peer address of a streaming response, or None. + + The socket lives in a different place across urllib3 majors, and on a + bodyless response (a redirect) it is already released -- hence None + rather than an exception, and hence `strict` in _check_peer_ip. + """ + for get_sock in ( + lambda: response.raw._fp.fp.raw._sock, # urllib3 2.x + lambda: response.raw._original_response.fp.raw._sock, # urllib3 1.x + lambda: response.raw._connection.sock, + ): + try: + sock = get_sock() + if sock is not None: + return sock.getpeername()[0] + except Exception: + continue + return None + + def _check_peer_ip(self, response, strict: bool = True) -> None: + """Validate the address actually connected to. + + Validating the URL only proves the *name* resolved somewhere allowed; + requests resolves it again when it dials, so a hostname with a short + TTL can point somewhere internal by then (DNS rebinding). Checking the + connected peer is what closes that window. + + `strict` is False for redirect hops, where urllib3 has already released + the socket and no peer is observable. Those hops are still covered by + the per-hop URL validation; what strict mode protects is the response + whose body we are about to read back to the caller, which is where a + rebound connection would actually exfiltrate something. + """ + if not _peer_ip_validator: + return + + peer = self._peer_ip(response) + if peer is None: + if not strict: + return + # Fail closed: a policy was installed, so an unverifiable peer on + # the response we are about to read is not something to wave through. + response.close() + raise RuntimeError("Could not determine peer address for PDF download") + + try: + _peer_ip_validator(peer) + except Exception: + response.close() + raise + + def _fetch_with_redirect_checks(self, requests, url: str): + """GET `url`, validating every hop against the injected destination policy. + + Redirects are followed by hand because requests would otherwise chase a + Location into a private address without consulting the validator at all + -- a public URL that 302s to 169.254.169.254 was the reported SSRF. + Returns an open streaming response; the caller closes it. + """ + from urllib.parse import urljoin + + current = url + # One Session for the whole chain: a bare requests.get() per hop starts + # with an empty cookie jar, so a host that sets a cookie and then + # redirects (common for gated or CDN-signed PDFs) would never get it + # back. allow_redirects=True used to carry cookies for us. + session = requests.Session() + for _ in range(self.max_redirects + 1): + if self.url_validator: + self.url_validator(current) + if _url_validator: + _url_validator(current) + + # Connection timeout: 20s, Read timeout: 600s (for large PDFs) + response = session.get( + current, stream=True, timeout=(20, 60 * 10), allow_redirects=False + ) + is_redirect = response.status_code in _REDIRECT_STATUSES + self._check_peer_ip(response, strict=not is_redirect) + + if is_redirect: + location = response.headers.get('location') + response.close() + if not location: + raise RuntimeError(f"Redirect without Location header from {current}") + # Relative Locations are legal, so resolve against the current + # URL before validating -- the validator needs an absolute URL. + current = urljoin(current, location) + continue + + response.raise_for_status() + return response + + raise RuntimeError( + f"Too many redirects (>{self.max_redirects}) downloading PDF from {url}" + ) + def _get_pdf_path(self, url: str) -> str: if url.startswith(("http://", "https://")): import tempfile import requests - + # Create temp file with .pdf extension temp_file = tempfile.NamedTemporaryFile(suffix='.pdf', delete=False) temp_file.close() # Close handle immediately; file persists due to delete=False self._temp_files.append(temp_file.name) - + try: if self.logger: self.logger.info(f"Downloading PDF from {url}...") - - # Download PDF with streaming and timeout - # Connection timeout: 10s, Read timeout: 300s (5 minutes for large PDFs) - response = requests.get(url, stream=True, timeout=(20, 60 * 10)) - response.raise_for_status() - - # Get file size if available - total_size = int(response.headers.get('content-length', 0)) + + response = self._fetch_with_redirect_checks(requests, url) + + # content-length is a hint only: it is attacker-controlled and + # may be absent or understated, so the running total is what + # enforces the cap. + total_size = int(response.headers.get('content-length', 0) or 0) downloaded = 0 - - # Write to temp file - with open(temp_file.name, 'wb') as f: - for chunk in response.iter_content(chunk_size=8192): - f.write(chunk) - downloaded += len(chunk) - if self.logger and total_size > 0: - progress = (downloaded / total_size) * 100 - if progress % 10 < 0.1: # Log every 10% - self.logger.debug(f"PDF download progress: {progress:.0f}%") - + + with response: + with open(temp_file.name, 'wb') as f: + for chunk in response.iter_content(chunk_size=8192): + downloaded += len(chunk) + if downloaded > self.max_pdf_bytes: + raise RuntimeError( + f"PDF from {url} exceeds max_pdf_bytes " + f"({self.max_pdf_bytes} bytes)" + ) + f.write(chunk) + if self.logger and total_size > 0: + progress = (downloaded / total_size) * 100 + if progress % 10 < 0.1: # Log every 10% + self.logger.debug(f"PDF download progress: {progress:.0f}%") + if self.logger: self.logger.info(f"PDF downloaded successfully: {temp_file.name}") - + return temp_file.name - + except requests.exceptions.Timeout as e: - # Clean up temp file if download fails - Path(temp_file.name).unlink(missing_ok=True) - self._temp_files.remove(temp_file.name) + self._discard_temp_file(temp_file.name) raise RuntimeError(f"Timeout downloading PDF from {url}: {str(e)}") except Exception as e: - # Clean up temp file if download fails - Path(temp_file.name).unlink(missing_ok=True) - self._temp_files.remove(temp_file.name) + self._discard_temp_file(temp_file.name) raise RuntimeError(f"Failed to download PDF from {url}: {str(e)}") - + elif url.startswith("file://"): return url[7:] # Strip file:// prefix diff --git a/crawl4ai/processors/pdf/processor.py b/crawl4ai/processors/pdf/processor.py index 1af25c05c..96ff0d260 100644 --- a/crawl4ai/processors/pdf/processor.py +++ b/crawl4ai/processors/pdf/processor.py @@ -55,8 +55,9 @@ def process(self, pdf_path: Path) -> PDFProcessResult: pass class NaivePDFProcessorStrategy(PDFProcessorStrategy): - def __init__(self, image_dpi: int = 144, image_quality: int = 85, extract_images: bool = True, - save_images_locally: bool = False, image_save_dir: Optional[Path] = None, batch_size: int = 4): + def __init__(self, image_dpi: int = 144, image_quality: int = 85, extract_images: bool = True, + save_images_locally: bool = False, image_save_dir: Optional[Path] = None, batch_size: int = 4, + max_pages: Optional[int] = None): # Import check at initialization time try: import pypdf @@ -70,8 +71,24 @@ def __init__(self, image_dpi: int = 144, image_quality: int = 85, extract_images self.save_images_locally = save_images_locally self.image_save_dir = image_save_dir self.batch_size = batch_size + # None means unbounded, which is the right default for a library caller + # opening a file they chose. Callers accepting untrusted PDFs (the + # Docker server) pass a cap so page count can't be used to burn CPU. + self.max_pages = max_pages self._temp_dir = None + def _page_limit(self, total_pages: int) -> int: + """Number of pages to actually process, honouring max_pages.""" + if self.max_pages is None: + return total_pages + capped = min(total_pages, self.max_pages) + if capped < total_pages: + logger.warning( + f"PDF has {total_pages} pages; processing first {capped} " + f"(max_pages={self.max_pages})" + ) + return capped + def process(self, pdf_path: Path) -> PDFProcessResult: # Import inside method to allow dependency to be optional try: @@ -101,7 +118,10 @@ def process(self, pdf_path: Path) -> PDFProcessResult: self._temp_dir = tempfile.mkdtemp(prefix='pdf_images_') image_dir = Path(self._temp_dir) + page_limit = self._page_limit(len(reader.pages)) for page_num, page in enumerate(reader.pages): + if page_num >= page_limit: + break self.current_page_number = page_num + 1 pdf_page = self._process_page(page, image_dir) result.pages.append(pdf_page) @@ -149,7 +169,7 @@ def process_batch(self, pdf_path: Path) -> PDFProcessResult: with pdf_path.open('rb') as file: reader = PdfReader(file) result.metadata = self._extract_metadata(pdf_path, reader) - total_pages = len(reader.pages) + total_pages = self._page_limit(len(reader.pages)) # Handle image directory setup image_dir = None diff --git a/crawl4ai/processors/pdf/utils.py b/crawl4ai/processors/pdf/utils.py index 3dc0e739d..e7cc6328b 100644 --- a/crawl4ai/processors/pdf/utils.py +++ b/crawl4ai/processors/pdf/utils.py @@ -97,10 +97,14 @@ def flush_paragraph(): para = ' '.join(current_paragraph) para = re.sub(r'\s+', ' ', para).strip() if para: - # escaped_para = html.escape(para) - escaped_para = para - # escaped_para = re.sub(r'\.\n', '.\n\n', escaped_para) - # Split escaped_para by <|break|> to avoid HTML escaping + # Paragraph text comes verbatim from the PDF and is attacker- + # controlled. Escape it so markup liketag escaped_para = [f'
{part}
' for part in escaped_para] diff --git a/deploy/docker/.llm.env.example b/deploy/docker/.llm.env.example index 012435d83..6f84f1ff1 100644 --- a/deploy/docker/.llm.env.example +++ b/deploy/docker/.llm.env.example @@ -1,3 +1,6 @@ +# Optional: enable declarative hooks support (disabled by default) +# CRAWL4AI_HOOKS_ENABLED=true + # LLM Provider Keys OPENAI_API_KEY=your_openai_key_here DEEPSEEK_API_KEY=your_deepseek_key_here diff --git a/deploy/docker/MIGRATION.md b/deploy/docker/MIGRATION.md index 753823060..bcd2097c7 100644 --- a/deploy/docker/MIGRATION.md +++ b/deploy/docker/MIGRATION.md @@ -25,11 +25,24 @@ loopback by default and will not expose itself without a credential. export CRAWL4AI_API_TOKEN="$(openssl rand -hex 32)" ``` +With `docker compose`, run the export in the same shell before +`docker compose up`; the compose file passes the token into the container. +For a persistent setup, put the `CRAWL4AI_API_TOKEN=...` line in a `.env` +file in the project root instead — compose reads it automatically (a shell +export still takes precedence). + +> ⚠️ For plain `docker run`, pass it explicitly: +> `-e CRAWL4AI_API_TOKEN="$CRAWL4AI_API_TOKEN"` (the value-less shorthand +> `-e CRAWL4AI_API_TOKEN` silently passes empty from a shell where the variable +> isn't set). + - With a token set, you may expose the server (put a TLS-terminating reverse proxy in front) and must send `Authorization: Bearer🔥 Stress Test
} } + // Build a useful error message from a failed HTTP response + function httpErrorMessage(response, data) { + let detail = data && (data.detail || data.error); + // FastAPI 422s send detail as an array of error objects; other + // non-strings would render as [object Object]. + if (Array.isArray(detail)) { + detail = detail.map(e => e && e.msg ? `${(e.loc || []).join('.')}: ${e.msg}` : JSON.stringify(e)).join('; '); + } else if (detail && typeof detail !== 'string') { + detail = JSON.stringify(detail); + } + let msg = detail || `HTTP ${response.status}`; + if (response.status === 401) { + msg += getToken() + ? ' — token rejected; check the API token in the token bar (top right)' + : ' — set your API token in the token bar (top right)'; + } + return msg; + } + // Generate code snippets function generateSnippets(api, payload, method = 'POST') { // Python snippet @@ -747,11 +766,11 @@🔥 Stress Test
method: 'GET', headers: { 'Accept': 'application/json' } }); - responseData = await response.json(); + responseData = await response.json().catch(() => ({})); const time = Math.round(performance.now() - startTime); if (!response.ok) { updateStatus('error', time); - throw new Error(responseData.error || 'Request failed'); + throw new Error(httpErrorMessage(response, responseData)); } updateStatus('success', time); document.querySelector('#response-content code').textContent = JSON.stringify(responseData, null, 2); @@ -765,6 +784,12 @@🔥 Stress Test
body: JSON.stringify(payload) }); + if (!response.ok) { + const errData = await response.json().catch(() => ({})); + updateStatus('error', Math.round(performance.now() - startTime)); + throw new Error(httpErrorMessage(response, errData)); + } + const reader = response.body.getReader(); let text = ''; let maxMemory = 0; @@ -804,12 +829,12 @@🔥 Stress Test
body: JSON.stringify(payload) }); - responseData = await response.json(); + responseData = await response.json().catch(() => ({})); const time = Math.round(performance.now() - startTime); if (!response.ok) { updateStatus('error', time); - throw new Error(responseData.error || 'Request failed'); + throw new Error(httpErrorMessage(response, responseData)); } updateStatus( @@ -975,9 +1000,11 @@🔥 Stress Test
// Save current scroll position (important for large code blocks) const scrollTop = element.parentElement.scrollTop; - // Reset the element - const text = element.textContent; - element.innerHTML = text; + // Reset highlight.js state. NOTE: do NOT round-trip through + // innerHTML here — the response text contains attacker-controlled + // crawled content and `element.innerHTML = element.textContent` + // re-parses it as live HTML (DOM-based XSS). highlight.js renders + // safely from textContent and emits escaped markup. element.removeAttribute('data-highlighted'); // Reapply highlighting diff --git a/deploy/docker/supervisord.conf b/deploy/docker/supervisord.conf index 91b4b2538..d726e2c6c 100644 --- a/deploy/docker/supervisord.conf +++ b/deploy/docker/supervisord.conf @@ -6,7 +6,7 @@ logfile_maxbytes=0 [program:redis] ; Loopback-only and password-protected. REDIS_PASSWORD is exported by ; entrypoint.sh from the mounted secret; the app must supply it to connect. -command=/usr/bin/redis-server --loglevel notice --bind 127.0.0.1 ::1 --requirepass "%(ENV_REDIS_PASSWORD)s" --dir /var/lib/redis +command=/usr/bin/redis-server --loglevel notice --bind 127.0.0.1 -::1 --requirepass "%(ENV_REDIS_PASSWORD)s" --dir /var/lib/redis user=appuser ; Run redis as our non-root user autorestart=true priority=10 @@ -31,4 +31,4 @@ stdout_logfile_maxbytes=0 stderr_logfile=/dev/stderr ; Redirect gunicorn stderr to container stderr stderr_logfile_maxbytes=0 -# Optional: Add filebeat or other logging agents here if needed \ No newline at end of file +# Optional: Add filebeat or other logging agents here if needed diff --git a/deploy/docker/tests/requirements.txt b/deploy/docker/tests/requirements.txt index 5f7a842fe..b0206020f 100644 --- a/deploy/docker/tests/requirements.txt +++ b/deploy/docker/tests/requirements.txt @@ -1,2 +1,3 @@ httpx>=0.25.0 docker>=7.0.0 +pyyaml>=6.0 diff --git a/deploy/docker/tests/test_api_crawl_failures.py b/deploy/docker/tests/test_api_crawl_failures.py new file mode 100644 index 000000000..04b7037c1 --- /dev/null +++ b/deploy/docker/tests/test_api_crawl_failures.py @@ -0,0 +1,58 @@ +from types import SimpleNamespace + +import pytest +from fastapi import HTTPException + +from api import _raise_for_crawl_failure + + +def test_crawl_failure_is_reported_as_bad_gateway(): + result = SimpleNamespace(success=False, error_message="Blocked by anti-bot protection: challenge") + + with pytest.raises(HTTPException) as raised: + _raise_for_crawl_failure(result) + + assert raised.value.status_code == 502 + assert raised.value.detail == result.error_message + + +@pytest.mark.parametrize( + ("method", "path", "payload"), + [ + ("post", "/md", {"url": "https://example.com", "f": "raw"}), + ("get", "/llm/example.com?q=summarize", None), + ], +) +def test_single_url_crawl_failure_reaches_client( + stock_client, server_module, monkeypatch, method, path, payload +): + error_message = "Blocked by anti-bot protection: challenge" + failed_result = SimpleNamespace(success=False, error_message=error_message) + + class FailedCrawler: + async def arun(self, *args, **kwargs): + return failed_result + + async def get_failed_crawler(*args, **kwargs): + return FailedCrawler() + + async def release_crawler(*args, **kwargs): + return None + + import api + import crawler_pool + from auth import create_access_token + + monkeypatch.setattr(api, "validate_url_destination", lambda url: None) + monkeypatch.setattr(crawler_pool, "get_crawler", get_failed_crawler) + monkeypatch.setattr(crawler_pool, "release_crawler", release_crawler) + + token = create_access_token({"sub": "test@example.com"}) + request = getattr(stock_client, method) + kwargs = {"json": payload} if payload is not None else {} + response = request( + path, headers={"Authorization": f"Bearer {token}"}, **kwargs + ) + + assert response.status_code == 502 + assert response.json() == {"detail": error_message} diff --git a/deploy/docker/tests/test_crawl_failure_response.py b/deploy/docker/tests/test_crawl_failure_response.py new file mode 100644 index 000000000..188da944a --- /dev/null +++ b/deploy/docker/tests/test_crawl_failure_response.py @@ -0,0 +1,28 @@ +def test_all_failed_crawl_returns_results(stock_client, server_module, monkeypatch): + async def failed_crawl(**kwargs): + return { + "success": True, + "results": [ + { + "url": "https://example.com", + "success": False, + "error_message": "Wait condition failed: selector not found", + } + ], + } + + monkeypatch.setattr(server_module, "handle_crawl_request", failed_crawl) + + from auth import create_access_token + + token = create_access_token({"sub": "test@example.com"}) + response = stock_client.post( + "/crawl", + json={"urls": ["https://example.com"]}, + headers={"Authorization": f"Bearer {token}"}, + ) + + assert response.status_code == 200 + result = response.json()["results"][0] + assert result["success"] is False + assert "Wait condition failed" in result["error_message"] diff --git a/deploy/docker/tests/test_legacy_compat.py b/deploy/docker/tests/test_legacy_compat.py new file mode 100644 index 000000000..a8a4039e3 --- /dev/null +++ b/deploy/docker/tests/test_legacy_compat.py @@ -0,0 +1,253 @@ +""" +Behavioral tests for 0.9.x legacy-compatibility handling: + + * UI redirects - "/" and "/monitor" are public and redirect to + /playground and /dashboard instead of dying in the auth + gate with a bare 401; the /monitor/* API routes and the + data routes stay gated. + * output_path - /screenshot and /pdf still accept the 0.8.x output_path + field but return a warning saying no file was written, + instead of silently dropping it. + * legacy hooks - hooks.code (removed 0.8.x inline Python) is captured, + never executed, and reported as status "ignored" with a + warning when hooks are enabled; any hooks payload is + still refused (403) while hooks are disabled. + +(The compose PID-cap check lives in test_security_container_posture.py.) + +These exercise the running app via TestClient (no browser / Redis needed); +crawl internals are stubbed where a handler would otherwise need a browser. +""" + +import base64 +from types import SimpleNamespace + +import pytest + +from auth import create_access_token # noqa: E402 + + +def _bearer() -> dict: + return {"Authorization": f"Bearer {create_access_token({'sub': 'user@x.com'}, scope='data')}"} + + +# ───────────────────────── UI redirects ───────────────────────── + + +class TestUiRedirects: + def test_root_is_public_and_redirects_to_playground(self, stock_client): + r = stock_client.get("/", follow_redirects=False) + assert r.status_code in (302, 307), ( + f"GET / returned {r.status_code}; expected a redirect. The auth " + f"gate must allow the exact path '/' so the redirect route runs." + ) + assert r.headers["location"] == "/playground" + + def test_monitor_redirects_to_dashboard(self, stock_client): + # Pre-0.9 docs sent people to /monitor for the dashboard UI. That exact + # path redirects to /dashboard (UI-public) instead of dead-ending in the + # auth gate with a bare 401. + r = stock_client.get("/monitor", follow_redirects=False) + assert r.status_code in (302, 307), ( + f"GET /monitor returned {r.status_code}; expected a redirect. The " + f"auth gate must allow the exact path '/monitor' so the route runs." + ) + assert r.headers["location"] == "/dashboard" + + @pytest.mark.parametrize( + "method,path", + [ + ("get", "/monitor/health"), + ("get", "/monitor/requests"), + ("get", "/monitor/browsers"), + ("get", "/monitor/timeline"), + ("get", "/monitor/logs/errors"), + ("post", "/monitor/actions/cleanup"), + ("post", "/monitor/actions/kill_browser"), + ("post", "/monitor/stats/reset"), + ], + ) + def test_monitor_api_routes_stay_gated(self, stock_client, method, path): + # The redirect above is exact-path only. Making the /monitor *prefix* + # public would expose request logs, browser state and the destructive + # admin actions without a credential. + assert getattr(stock_client, method)(path).status_code == 401 + + def test_monitor_websocket_stays_gated(self, stock_client): + # /monitor/ws is where live stats stream; it must not open unauthenticated. + with pytest.raises(Exception): + with stock_client.websocket_connect("/monitor/ws"): + pass + + def test_data_routes_stay_gated(self, stock_client): + assert stock_client.post("/crawl", json={"urls": ["https://x"]}).status_code == 401 + + +# ───────────────────────── output_path warning ───────────────────────── + + +@pytest.fixture +def stub_crawler(server_module, monkeypatch): + """Stub the crawler pool + artifact store so /screenshot and /pdf run + without a browser. Returns the fake artifact dict for assertions.""" + art = {"artifact_id": "a1", "url": "/artifacts/a1", "mime": "x", "size": 1} + png_b64 = base64.b64encode(b"fake-png").decode() + + fake_result = SimpleNamespace(success=True, screenshot=png_b64, pdf=b"fake-pdf") + + class _FakeCrawler: + async def arun(self, url, config): + return [fake_result] + + async def fake_get_crawler(cfg): + return _FakeCrawler() + + async def fake_release_crawler(crawler): + pass + + monkeypatch.setattr(server_module, "get_crawler", fake_get_crawler) + monkeypatch.setattr(server_module, "release_crawler", fake_release_crawler) + monkeypatch.setattr(server_module, "_store_artifact", lambda kind, data: dict(art)) + return art + + +class TestOutputPathWarning: + @pytest.mark.parametrize("endpoint,payload_key", [("/screenshot", "screenshot"), ("/pdf", "pdf")]) + def test_output_path_accepted_with_warning(self, stock_client, stub_crawler, endpoint, payload_key): + r = stock_client.post( + endpoint, + json={"url": "https://example.com", "output_path": "/tmp/x.bin"}, + headers=_bearer(), + ) + assert r.status_code == 200 + body = r.json() + assert body["success"] is True + assert "warning" in body, "output_path must not be silently dropped" + assert "no file was written" in body["warning"] + assert body["artifact_id"] == stub_crawler["artifact_id"] + + @pytest.mark.parametrize("endpoint", ["/screenshot", "/pdf"]) + def test_no_warning_without_output_path(self, stock_client, stub_crawler, endpoint): + r = stock_client.post(endpoint, json={"url": "https://example.com"}, headers=_bearer()) + assert r.status_code == 200 + assert "warning" not in r.json() + + @pytest.mark.parametrize("endpoint", ["/screenshot", "/pdf"]) + def test_output_path_is_never_written(self, stock_client, stub_crawler, endpoint, tmp_path): + """Security tripwire for the 0.8.x arbitrary-write vuln: the handler + runs for real here (only crawler/artifact store are stubbed), so any + reintroduced write of body.output_path creates this file and fails.""" + target = tmp_path / "out.bin" + r = stock_client.post( + endpoint, json={"url": "https://example.com", "output_path": str(target)}, + headers=_bearer(), + ) + assert r.status_code == 200 + assert not target.exists(), "output_path must never be written to disk" + + +# ───────────────────────── legacy hooks.code ───────────────────────── + +LEGACY_HOOKS = {"code": {"before_goto": "async def hook(p, c, u, **kw): return p"}} +DECLARATIVE_HOOKS = {"hooks": [{"action": "scroll_to_bottom", "params": {"max_steps": 2}}]} + + +class TestLegacyHookCode: + def test_hook_config_captures_code_field(self): + """The legacy field must be parsed (not dropped) so it can be reported.""" + from schemas import CrawlRequestWithHooks + + req = CrawlRequestWithHooks(urls=["https://x"], hooks=LEGACY_HOOKS) + assert req.hooks.code == LEGACY_HOOKS["code"] + assert req.hooks.hooks == [] + + @pytest.mark.parametrize("code", [ + "def hook(): ...", # bare string + {"before_goto": {"nested": "dict"}}, # non-string values + ["a", "list"], # wrong container entirely + ]) + def test_hook_code_accepts_any_legacy_shape(self, code): + """The 0.8.x wire shape was never pinned; a 422 on a field we only + report about would be worse than the silent drop it replaces.""" + from schemas import CrawlRequestWithHooks + + req = CrawlRequestWithHooks(urls=["https://x"], hooks={"code": code}) + assert req.hooks.code == code + + # Any hooks payload is refused while hooks are disabled, but the detail + # must not mislead legacy callers: enabling the flag would not run + # hooks.code (removed in 0.9.0), so payloads carrying it get a removal + # message instead of the generic 'set CRAWL4AI_HOOKS_ENABLED' hint. + @pytest.mark.parametrize("hooks_payload,expected_detail", [ + (LEGACY_HOOKS, "removed in 0.9.0"), # code-only + ({**DECLARATIVE_HOOKS, **LEGACY_HOOKS}, "removed in 0.9.0"), # mixed + (DECLARATIVE_HOOKS, "Set CRAWL4AI_HOOKS_ENABLED=true"), # declarative-only + ]) + def test_any_hooks_payload_403_when_disabled( + self, server_module, monkeypatch, stock_client, hooks_payload, expected_detail + ): + monkeypatch.setattr(server_module, "HOOKS_ENABLED", False) + r = stock_client.post( + "/crawl", + json={"urls": ["https://example.com"], "hooks": hooks_payload}, + headers=_bearer(), + ) + assert r.status_code == 403 + assert expected_detail in r.json()["detail"] + + def _stub_crawl(self, server_module, monkeypatch, results): + async def fake_handle_crawl_request(**kwargs): + return dict(results) + + monkeypatch.setattr(server_module, "handle_crawl_request", fake_handle_crawl_request) + + def test_legacy_code_warned_and_ignored_when_enabled(self, server_module, monkeypatch, stock_client): + monkeypatch.setattr(server_module, "HOOKS_ENABLED", True) + # Empty declarative specs -> api.py reports a vacuous success + self._stub_crawl( + server_module, monkeypatch, + {"success": True, "results": [{"success": True}], + "hooks": {"status": "success", "attached": []}}, + ) + r = stock_client.post( + "/crawl", + json={"urls": ["https://example.com"], "hooks": LEGACY_HOOKS}, + headers=_bearer(), + ) + assert r.status_code == 200 + hooks = r.json()["hooks"] + assert hooks["status"] == "ignored", "vacuous 'success' must be rewritten" + assert hooks["attached"] == [] + assert "NOT executed" in hooks["warning"] + + def test_mixed_request_keeps_declarative_status_and_warns(self, server_module, monkeypatch, stock_client): + monkeypatch.setattr(server_module, "HOOKS_ENABLED", True) + self._stub_crawl( + server_module, monkeypatch, + {"success": True, "results": [{"success": True}], + "hooks": {"status": "success", "attached": ["before_retrieve_html"]}}, + ) + r = stock_client.post( + "/crawl", + json={"urls": ["https://example.com"], + "hooks": {**DECLARATIVE_HOOKS, **LEGACY_HOOKS}}, + headers=_bearer(), + ) + assert r.status_code == 200 + hooks = r.json()["hooks"] + assert hooks["status"] == "success", "real declarative execution must not be relabeled" + assert hooks["attached"] == ["before_retrieve_html"] + assert "NOT executed" in hooks["warning"] + + def test_no_hooks_response_untouched(self, server_module, monkeypatch, stock_client): + self._stub_crawl( + server_module, monkeypatch, + {"success": True, "results": [{"success": True}]}, + ) + r = stock_client.post("/crawl", json={"urls": ["https://example.com"]}, headers=_bearer()) + assert r.status_code == 200 + assert "hooks" not in r.json() + + +# The compose PID-cap check lives in +# test_security_container_posture.py::test_pids_limit (YAML-parsed). diff --git a/deploy/docker/tests/test_security_2026_04.py b/deploy/docker/tests/test_security_2026_04.py index 4109de046..e015274b6 100644 --- a/deploy/docker/tests/test_security_2026_04.py +++ b/deploy/docker/tests/test_security_2026_04.py @@ -82,23 +82,29 @@ def validate_webhook_url(url): # ============================================================================ class TestOutputPathRemoved(unittest.TestCase): - """output_path is gone; the server owns paths via the artifact store. + """output_path never reaches the filesystem; the server owns paths via the + artifact store. The old string-only validate_output_path was bypassable (symlink/TOCTOU, sibling-prefix '...-evil') -> arbitrary write -> RCE. The fix is to never - accept a caller path at all. Behavioral artifact-store coverage (O_NOFOLLOW, - O_EXCL, hex id, TTL, quota) lives in test_security_artifact_store.py. + use a caller path: output_path exists only as a deprecated no-op for 0.8.x + compatibility and must stay that way. Artifact-store coverage lives in + test_security_artifact_store.py; warning behavior in test_legacy_compat.py. """ - def test_screenshot_request_has_no_output_path(self): + def test_screenshot_output_path_is_deprecated_noop(self): sys.path.insert(0, DEPLOY_DIR) from schemas import ScreenshotRequest - self.assertNotIn("output_path", ScreenshotRequest.model_fields) + field = ScreenshotRequest.model_fields["output_path"] + self.assertTrue(field.deprecated, "output_path must be marked deprecated") + self.assertIsNone(field.default) - def test_pdf_request_has_no_output_path(self): + def test_pdf_output_path_is_deprecated_noop(self): sys.path.insert(0, DEPLOY_DIR) from schemas import PDFRequest - self.assertNotIn("output_path", PDFRequest.model_fields) + field = PDFRequest.model_fields["output_path"] + self.assertTrue(field.deprecated, "output_path must be marked deprecated") + self.assertIsNone(field.default) def test_validate_output_path_deleted(self): sys.path.insert(0, DEPLOY_DIR) @@ -110,18 +116,21 @@ def test_validate_output_path_deleted(self): class TestPydanticPathValidator(unittest.TestCase): - """output_path (and its traversal validator) are gone entirely. + """The traversal validator is gone entirely. Traversal rejection used to be the mitigation; the real fix is that no - caller path is accepted at all, so there is nothing to traverse. The - sandboxed artifact store owns all paths now. + caller path is ever *used*, so there is nothing to traverse. output_path + survives only as a deprecated no-op field (see TestOutputPathRemoved); the + sandboxed artifact store owns all paths. """ - def test_no_output_path_field_on_request_models(self): + def test_output_path_is_inert_on_request_models(self): sys.path.insert(0, DEPLOY_DIR) from schemas import ScreenshotRequest, PDFRequest - self.assertNotIn("output_path", ScreenshotRequest.model_fields) - self.assertNotIn("output_path", PDFRequest.model_fields) + for model in (ScreenshotRequest, PDFRequest): + field = model.model_fields["output_path"] + self.assertTrue(field.deprecated) + self.assertIsNone(field.default) def test_traversal_validator_removed(self): # No reject_traversal validator should remain registered on the models. diff --git a/deploy/docker/tests/test_security_container_posture.py b/deploy/docker/tests/test_security_container_posture.py index 871e642de..eb5df6153 100644 --- a/deploy/docker/tests/test_security_container_posture.py +++ b/deploy/docker/tests/test_security_container_posture.py @@ -78,8 +78,8 @@ class TestSupervisord: def test_redis_requires_password(self, supervisord): assert "--requirepass" in supervisord - def test_redis_bound_loopback(self, supervisord): - assert "--bind 127.0.0.1" in supervisord + def test_redis_bound_loopback_with_optional_ipv6(self, supervisord): + assert "--bind 127.0.0.1 -::1" in supervisord def test_gunicorn_bind_is_env_driven(self, supervisord): # entrypoint.sh resolves GUNICORN_BIND (loopback unless a credential). @@ -104,7 +104,14 @@ def test_no_host_dev_shm_bind(self, compose): assert "shm_size" in compose def test_pids_limit(self, compose): - assert "pids_limit" in compose + # Parse, don't grep: a raw-text search matched the word "pids_limit" + # inside a comment and guarded nothing. The cap lives under + # deploy.resources.limits (not pids_limit) for Compose v5 compatibility. + import yaml + + base = yaml.safe_load(compose)["x-base-config"] + assert "pids_limit" not in base + assert base["deploy"]["resources"]["limits"]["pids"] == 512 def test_read_only_runtime_tmpfs_are_appuser_owned(self, compose): assert "/var/lib/redis:uid=999,gid=999,mode=0700" in compose diff --git a/deploy/docker/tests/test_security_egress_proxy.py b/deploy/docker/tests/test_security_egress_proxy.py index 03ebae1ea..9dc377e25 100644 --- a/deploy/docker/tests/test_security_egress_proxy.py +++ b/deploy/docker/tests/test_security_egress_proxy.py @@ -20,6 +20,18 @@ pytestmark = pytest.mark.posture +_PROXY_ENV = ( + "CRAWL4AI_UPSTREAM_PROXY", "HTTP_PROXY", "http_proxy", + "HTTPS_PROXY", "https_proxy", "NO_PROXY", "no_proxy", +) + + +@pytest.fixture(autouse=True) +def _clear_proxy_env(monkeypatch): + # Keep the suite deterministic on dev machines that sit behind a proxy. + for name in _PROXY_ENV: + monkeypatch.delenv(name, raising=False) + async def _fake_upstream(): async def handle(reader, writer): @@ -121,6 +133,160 @@ async def test_malformed_connect_400(self): await proxy.stop() +async def _fake_corporate_proxy(seen): + """Minimal HTTP proxy: records the CONNECT request line, replies 200, then + answers any tunneled bytes with TUNNEL-OK.""" + async def handle(reader, writer): + line = await reader.readline() + seen.append(line) + while True: # drain CONNECT headers + h = await reader.readline() + if h in (b"\r\n", b"\n", b""): + break + writer.write(b"HTTP/1.1 200 Connection established\r\nVia: fake\r\n\r\n") + await writer.drain() + await reader.read(65536) + writer.write(b"TUNNEL-OK") + await writer.drain() + writer.close() + server = await asyncio.start_server(handle, "127.0.0.1", 0) + return server, server.sockets[0].getsockname()[1] + + +@pytest.mark.asyncio +class TestUpstreamChaining: + async def test_chained_connect_pins_ip_and_blocks_before_upstream(self, monkeypatch): + """The chained-CONNECT security contract: the upstream receives the + PINNED IP (never a hostname to resolve), its response headers do not + leak into the tunnel, and a blocked target produces an opaque 403 + with zero upstream traffic.""" + seen = [] + corp, corp_port = await _fake_corporate_proxy(seen) + monkeypatch.setenv("HTTPS_PROXY", f"http://127.0.0.1:{corp_port}") + + def fake_pin(url): + if "internal.example" in url: + raise EgressBlocked() + return PinnedTarget("https", "good.example", 443, "203.0.113.7") + monkeypatch.setattr(egress_proxy, "resolve_and_pin", fake_pin) + + proxy = PinningProxy() + await proxy.start() + try: + r, w = await asyncio.open_connection(proxy.bound_host, proxy.bound_port) + w.write(b"CONNECT good.example:443 HTTP/1.1\r\n\r\n") + await w.drain() + status = await asyncio.wait_for(r.readline(), timeout=5) + assert b"200" in status + await r.readline() # blank line after the 200 + w.write(b"hello") + await w.drain() + body = await asyncio.wait_for(r.read(100), timeout=5) + # Upstream's Via header must NOT leak into the tunnel. + assert body == b"TUNNEL-OK" + w.close() + + r, w = await asyncio.open_connection(proxy.bound_host, proxy.bound_port) + w.write(b"CONNECT internal.example:443 HTTP/1.1\r\n\r\n") + await w.drain() + status = await asyncio.wait_for(r.readline(), timeout=5) + assert b"403" in status + w.close() + finally: + await proxy.stop() + corp.close() + # The upstream saw ONLY the pinned IP of the allowed target. + assert seen == [b"CONNECT 203.0.113.7:443 HTTP/1.1\r\n"] + + async def test_chained_plain_http_pinned_absolute_form_no_smuggling(self, monkeypatch): + """Plain HTTP via upstream: the request is re-issued in absolute form + against the PINNED IP (no name for the upstream to resolve), carries + Connection: close, and a reused client connection cannot smuggle a + second, unvalidated request upstream.""" + lines = [] + + async def handle(reader, writer): + req = b"" + while b"\r\n\r\n" not in req: + req += await reader.read(4096) + lines.append(req) + writer.write(b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\nConnection: close\r\n\r\nhi") + await writer.drain() + writer.close() + corp = await asyncio.start_server(handle, "127.0.0.1", 0) + corp_port = corp.sockets[0].getsockname()[1] + monkeypatch.setenv("HTTP_PROXY", f"http://127.0.0.1:{corp_port}") + + def fake_pin(url): + return PinnedTarget("http", "plain.example", 80, "203.0.113.7") + monkeypatch.setattr(egress_proxy, "resolve_and_pin", fake_pin) + + proxy = PinningProxy() + await proxy.start() + try: + r, w = await asyncio.open_connection(proxy.bound_host, proxy.bound_port) + w.write(b"GET http://plain.example/ HTTP/1.1\r\n" + b"Host: plain.example\r\nConnection: keep-alive\r\n\r\n") + await w.drain() + first = await asyncio.wait_for(r.read(200), timeout=5) + assert b"200" in first + # Attempt to smuggle an unvalidated request on the same connection. + w.write(b"GET http://rebind.evil/ HTTP/1.1\r\nHost: rebind.evil\r\n\r\n") + await w.drain() + leftover = await asyncio.wait_for(r.read(200), timeout=5) + assert leftover == b"" # upstream closed; nothing came back + w.close() + finally: + await proxy.stop() + corp.close() + sent = b"".join(lines) + assert sent.startswith(b"GET http://203.0.113.7:80/ HTTP/1.1\r\n") + assert b"Connection: close" in sent + assert b"keep-alive" not in sent + assert b"rebind.evil" not in sent # the smuggled request never got upstream + + +def test_upstream_proxy_env_parsing(monkeypatch): + assert egress_proxy.upstream_proxy() is None + monkeypatch.setenv("HTTP_PROXY", "http://192.168.180.254:56560") + assert egress_proxy.upstream_proxy() == ("192.168.180.254", 56560, None) + monkeypatch.setenv("HTTPS_PROXY", "http://user:p%40ss@10.0.0.1:8080") + host, port, auth = egress_proxy.upstream_proxy() + assert (host, port) == ("10.0.0.1", 8080) + import base64 + assert base64.b64decode(auth.split(b" ")[-1].strip()) == b"user:p@ss" + # scheme-aware selection: http targets prefer HTTP_PROXY + assert egress_proxy.upstream_proxy("http") == ("192.168.180.254", 56560, None) + monkeypatch.setenv("CRAWL4AI_UPSTREAM_PROXY", "proxy.corp:3128") + assert egress_proxy.upstream_proxy() == ("proxy.corp", 3128, None) + # whitespace-only env var means unset, not a proxy named " " + monkeypatch.setenv("CRAWL4AI_UPSTREAM_PROXY", " ") + monkeypatch.delenv("HTTP_PROXY") + monkeypatch.delenv("HTTPS_PROXY") + assert egress_proxy.upstream_proxy() is None + # a junk/unsupported candidate falls through to a valid fallback + monkeypatch.setenv("HTTP_PROXY", "http://good:3128") + monkeypatch.setenv("HTTPS_PROXY", "http://") + assert egress_proxy.upstream_proxy() == ("good", 3128, None) + monkeypatch.setenv("HTTPS_PROXY", "https://tls-proxy.corp") # unsupported scheme + assert egress_proxy.upstream_proxy() == ("good", 3128, None) + monkeypatch.delenv("CRAWL4AI_UPSTREAM_PROXY") + monkeypatch.delenv("HTTP_PROXY") + assert egress_proxy.upstream_proxy() is None # https:// alone -> refused, not mis-dialed + # non-latin-1 credentials must not raise (encoded as UTF-8) + monkeypatch.setenv("HTTPS_PROXY", "http://u:%E5%AF%86%E7%A0%81@10.0.0.1:8080") + assert egress_proxy.upstream_proxy()[2] is not None + # NO_PROXY routing: suffix and CIDR entries force a direct dial + pin = PinnedTarget("https", "site.corp.example", 443, "203.0.113.7") + assert egress_proxy._use_upstream(pin) is not None + monkeypatch.setenv("NO_PROXY", ".corp.example") + assert egress_proxy._use_upstream(pin) is None + monkeypatch.setenv("NO_PROXY", "203.0.113.0/24") + assert egress_proxy._use_upstream(pin) is None + monkeypatch.setenv("NO_PROXY", "site.corp.example:443") # host:port form + assert egress_proxy._use_upstream(pin) is None + + class TestEnforceEgressWiring: def test_enforce_egress_sets_proxy(self, monkeypatch): import egress_broker diff --git a/deploy/docker/tests/test_security_pdf_image_write.py b/deploy/docker/tests/test_security_pdf_image_write.py new file mode 100644 index 000000000..4f171ce4a --- /dev/null +++ b/deploy/docker/tests/test_security_pdf_image_write.py @@ -0,0 +1,83 @@ +#!/usr/bin/env python3 +""" +Regression test for the PDF-image arbitrary-write via untrusted config body +(reported by sec-reex). + +Root cause: PDFContentScrapingStrategy is an UNTRUSTED_ALLOWED_TYPE but had no +field allowlist, so _filter_untrusted_fields fell open and kept its +filesystem-write knobs (image_save_dir / save_images_locally). A request body +the API loads as UNTRUSTED could then steer the PDF image writer to an +attacker-chosen directory. + +These tests hit the real deserialization path (from_serializable_dict / +CrawlerRunConfig.load with provenance=UNTRUSTED), not a copy. +""" + +import os +import sys +import unittest + +sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", ".."))) + +from crawl4ai.async_configs import ( + CrawlerRunConfig, + Provenance, + UntrustedConfigError, + from_serializable_dict, +) + + +def _pdf_strategy(params: dict) -> dict: + return {"type": "PDFContentScrapingStrategy", "params": params} + + +class TestPdfImageWriteGate(unittest.TestCase): + def test_untrusted_image_save_dir_is_rejected(self): + """image_save_dir on an untrusted body must raise (loud 400), not pass.""" + body = _pdf_strategy({"save_images_locally": True, "image_save_dir": "/home/appuser/.ssh"}) + with self.assertRaises(UntrustedConfigError): + from_serializable_dict(body, provenance=Provenance.UNTRUSTED) + + def test_untrusted_save_images_locally_is_rejected(self): + """The boolean that turns the write on is forbidden on untrusted bodies too.""" + body = _pdf_strategy({"save_images_locally": True}) + with self.assertRaises(UntrustedConfigError): + from_serializable_dict(body, provenance=Provenance.UNTRUSTED) + + def test_untrusted_rejected_when_nested_in_crawlerrunconfig(self): + """The real server path: PDF strategy nested under CrawlerRunConfig.scraping_strategy.""" + body = { + "type": "CrawlerRunConfig", + "params": { + "scraping_strategy": _pdf_strategy( + {"save_images_locally": True, "image_save_dir": "/tmp/c4_oob"} + ) + }, + } + with self.assertRaises(UntrustedConfigError): + CrawlerRunConfig.load(body, provenance=Provenance.UNTRUSTED) + + def test_untrusted_extract_images_still_allowed(self): + """Negative control: extract_images returns bytes base64-inline, no disk + write, so it must NOT be blocked (feature preserved over the API).""" + body = _pdf_strategy({"extract_images": True}) + obj = from_serializable_dict(body, provenance=Provenance.UNTRUSTED) + self.assertEqual(type(obj).__name__, "PDFContentScrapingStrategy") + + def test_trusted_path_unchanged(self): + """No regression: the in-process SDK (TRUSTED) may still set image_save_dir.""" + body = _pdf_strategy({"save_images_locally": True, "image_save_dir": "/tmp/mine"}) + obj = from_serializable_dict(body, provenance=Provenance.TRUSTED) + self.assertEqual(type(obj).__name__, "PDFContentScrapingStrategy") + + def test_global_backstop_covers_future_types(self): + """The global forbidden set catches a write-arg regardless of per-type map. + LXMLWebScrapingStrategy has no path arg today, but a smuggled downloads_path + must still be rejected by the fail-closed backstop.""" + body = {"type": "LXMLWebScrapingStrategy", "params": {"downloads_path": "/etc"}} + with self.assertRaises(UntrustedConfigError): + from_serializable_dict(body, provenance=Provenance.UNTRUSTED) + + +if __name__ == "__main__": + unittest.main(verbosity=2) diff --git a/docker-compose.yml b/docker-compose.yml index 1cd87ad93..02cbe0584 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,21 +1,23 @@ -version: '3.8' - # Shared configuration for all environments x-base-config: &base-config ports: - "11235:11235" # Gunicorn port env_file: - - .llm.env # API keys (create from .llm.env.example) - # Uncomment to set default environment variables (will overwrite .llm.env) - # environment: - # - OPENAI_API_KEY=${OPENAI_API_KEY:-} - # - DEEPSEEK_API_KEY=${DEEPSEEK_API_KEY:-} - # - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY:-} - # - GROQ_API_KEY=${GROQ_API_KEY:-} - # - TOGETHER_API_KEY=${TOGETHER_API_KEY:-} - # - MISTRAL_API_KEY=${MISTRAL_API_KEY:-} - # - GEMINI_API_KEY=${GEMINI_API_KEY:-} - # - LLM_PROVIDER=${LLM_PROVIDER:-} # Optional: Override default provider (e.g., "anthropic/claude-3-opus") + # API keys (create from .llm.env.example); optional so a fresh clone runs. + - path: .llm.env + required: false + environment: + # Auth token passthrough from the host shell (overwrites .llm.env). + - CRAWL4AI_API_TOKEN=${CRAWL4AI_API_TOKEN:-} + # Uncomment to set default environment variables (will overwrite .llm.env) + # - OPENAI_API_KEY=${OPENAI_API_KEY:-} + # - DEEPSEEK_API_KEY=${DEEPSEEK_API_KEY:-} + # - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY:-} + # - GROQ_API_KEY=${GROQ_API_KEY:-} + # - TOGETHER_API_KEY=${TOGETHER_API_KEY:-} + # - MISTRAL_API_KEY=${MISTRAL_API_KEY:-} + # - GEMINI_API_KEY=${GEMINI_API_KEY:-} + # - LLM_PROVIDER=${LLM_PROVIDER:-} # Optional: Override default provider (e.g., "anthropic/claude-3-opus") # Chromium needs shared memory but the host /dev/shm bind was a shared, # writable mount; use a private sized tmpfs instead. shm_size: "1gb" @@ -24,7 +26,6 @@ x-base-config: &base-config - ALL security_opt: - no-new-privileges:true - pids_limit: 512 # Read-only root filesystem; only these paths are writable (tmpfs). read_only: true tmpfs: @@ -39,6 +40,9 @@ x-base-config: &base-config resources: limits: memory: 4G + # PID cap; expressed here (not as pids_limit) so the file stays valid + # on Compose v5+, which rejects pids_limit alongside a limits block. + pids: 512 reservations: memory: 1G restart: unless-stopped diff --git a/docs/blog/release-v0.9.3.md b/docs/blog/release-v0.9.3.md new file mode 100644 index 000000000..256c4b3fc --- /dev/null +++ b/docs/blog/release-v0.9.3.md @@ -0,0 +1,149 @@ +# Crawl4AI v0.9.3: Security Release + +*August 2026 - 4 min read* + +--- + +I'm releasing Crawl4AI v0.9.3, a security release that closes five coordinated-disclosure advisories. It also carries the 33 bug fixes that piled up on `develop` since 0.9.2, most of them in the Docker server. No new features, no breaking changes. + +Four of the five are in the PDF path, and they share one root cause. `PDFContentScrapingStrategy` can be selected from an untrusted Docker API request body, and it downloads with `requests` rather than through the browser. Every control the server applies on the Chromium side (destination pinning, resource caps) simply did not reach it. The fifth is a DOM-based XSS in the Playground UI that could hand an operator's API token to an attacker. + +If you self-host the Docker server and accept crawl requests from clients you do not fully trust, upgrade. If you use the pip library and open PDFs from sources you do not control, upgrade. + +## What's fixed at a glance + +- **Arbitrary file write**: an untrusted request body could choose where extracted PDF images were written +- **SSRF**: the PDF download followed redirects into internal addresses +- **Denial of service**: remote PDFs were downloaded and parsed with no size or page cap +- **XSS**: PDF text was written into `cleaned_html` without escaping +- **DOM XSS**: the Playground result viewer re-parsed crawled content as live HTML +- **33 other bug fixes**: Docker server, crawler, and PDF handling, listed further down + +## Security fixes + +### Arbitrary file write via PDF image-write fields + +**GHSA-xpp7-j28w-2gvx, CWE-22, high.** Credit: Zhixi "Jace" Sun, independent security researcher. + +`PDFContentScrapingStrategy` was in the untrusted-allowed type list but had no field allowlist of its own. A request body could therefore set `save_images_locally: true` and `image_save_dir` to any path, and the server would write extracted PDF images there. + +Both fields are now filtered out of untrusted bodies at the trust boundary, and `extract_images` is forced off for them. Rasterizing every page of a caller-supplied PDF is the most expensive thing this strategy can do, and nothing about untrusted crawling needs it. + +### SSRF via PDF download redirects + +**GHSA-q5rj-45vw-vp2g, CWE-918, high.** Credit: Nguyen Tran Thanh Lam ([c240030](https://github.com/c240030)). + +The PDF download let `requests` follow redirects on its own. A public, allowed URL that returned a 302 to `169.254.169.254` or any other internal address was fetched without the destination ever being checked. + +Redirects are now resolved by hand, one hop at a time, with the destination policy consulted before each hop is fetched and a cap of five hops. The peer IP of the response whose body is actually read back is validated too, which closes DNS rebinding: passing the URL check only proves the *name* resolved somewhere allowed, and `requests` resolves it again when it dials. + +The library ships with no egress policy of its own, because a plain library caller already chose the URL. The Docker server installs its existing `egress_broker` policy into the PDF path at boot, so that path now gets the same non-global-IP rule the browser path already had. + +### Denial of service via unbounded PDF size and page count + +**GHSA-v2rm-hvrj-2x9q, CWE-400, medium.** Credit: Nguyen Tran Thanh Lam ([c240030](https://github.com/c240030)). + +A remote PDF was streamed to a temp file with no cap and then parsed with no page limit. One request pointing at a large or many-page PDF could exhaust disk, CPU, and bandwidth on a shared worker. + +Downloads now stop at `max_pdf_bytes`, 100 MiB by default. The cap is enforced on the running byte total, not on the `content-length` header, because that header is attacker-controlled and may be absent or understated. Parsing stops at `max_pdf_pages`, 2000 by default, in both the single and batch paths. Untrusted request bodies have both caps clamped, so a client cannot raise its own limits back to unbounded. + +The Docker config also ships a non-zero `limits.wall_clock_s` of 300 seconds. It was `0`, meaning no per-crawl deadline at all. + +### XSS via unescaped PDF text in cleaned_html + +**GHSA-7g3g-vhm6-79f3, CWE-79, medium.** Credit: Nguyen Tran Thanh Lam ([c240030](https://github.com/c240030)). + +`clean_pdf_text_to_html()` escapes every sink it writes to except one: paragraph text. The `html.escape()` call on that path had been commented out. Markup embedded in a PDF therefore survived verbatim into `cleaned_html` and executed wherever the result was rendered. + +The escape is restored. + +### DOM-based XSS in the Playground leading to API token theft + +**GHSA-m446-hp3q-qfxp, CWE-79, high.** Credit: [e1codes](https://github.com/e1codes). + +The Playground result viewer reset syntax highlighting with: + +```js +const text = element.textContent; +element.innerHTML = text; +``` + +That round trip re-parses the text as live HTML. The text is crawl output, so it is whatever the crawled page contained. Script in a crawled page could therefore run in the operator's Playground session and read the API token held there. + +The round trip is removed. highlight.js renders from `textContent` and emits escaped markup on its own, so it was never needed. + +## Bug fixes + +Everything below landed on `develop` between 0.9.2 and this release. None of it is security related. + +### Docker server + +- **PDF works out of the box**: PDF scraping is supported by default, and a request that selects `PDFContentScrapingStrategy` is now routed to `PDFCrawlerStrategy` automatically. Before this you had to pair them by hand or get a placeholder back. (#2094, #2150) +- **Upstream proxy chaining**: the egress proxy now chains through `HTTP_PROXY` / `HTTPS_PROXY` instead of ignoring them, so the server works behind a corporate proxy. (#2142) +- **Proxy env handling**: junk values in the proxy environment fall through to the next candidate rather than failing the crawl, and non-http proxy schemes are refused up front. (#2094) +- **Compose v5**: compatibility fixes, clearer warnings when a legacy field is used, and better error handling in the playground. (#2094) +- **Token and LLM env**: `CRAWL4AI_API_TOKEN` is forwarded through compose, and `.llm.env` is optional instead of required. (#2094) +- **Clearer 403 on hooks**: the disabled-hooks response now says when the request used the removed `hooks.code` field, rather than giving one generic refusal. (#2094) +- **`output_path`**: declared a deprecated no-op instead of being silently ignored. (#2094) +- **`GET /monitor`**: redirects to the dashboard UI instead of returning nothing useful. (#2157, issue #2091) +- **Failed crawls are reported**: failure details are preserved for both batch and single-URL requests instead of being dropped. (#2094, #2134, issue #2133) +- **IPv6 loopback**: an unavailable IPv6 loopback no longer breaks startup. (#2081, issue #2078) +- **MCP pinning**: `mcp` is capped below 2, keeping the v1 low-level API that `mcp_bridge` depends on. (#2148, thanks @weike-zhang) +- **Compose formatting**: commented environment variable lines are aligned with the environment list. (#2156) + +### Crawler and core + +- **Playwright driver leak**: when a browser failed to launch inside `__aenter__`, the Playwright driver subprocess was left running. Repeated failed launches leaked one process each time. It is now cleaned up. (#2160) +- **PDF crawls wrongly flagged as blocked**: `PDFCrawlerStrategy` returns a deliberately near-empty placeholder response, and the anti-bot check read that as a block. Every PDF crawl failed, burned the whole retry budget, and then called the fallback fetch. The placeholder is now recognised. (#2138, issue #2135) +- **Cookies across PDF redirects**: redirects are followed by hand, which meant each hop started with an empty cookie jar. A host that set a cookie and then redirected never got it back, so gated and CDN-signed PDFs failed. One session now covers the whole chain. (#2159) +- **Overlay removal hang**: the overlay and consent removal scripts waited on unconditional `setTimeout` calls, which could hang a crawl under a restrictive CSP. The waits are gone. (#2139) +- **`remove_overlay_elements` deleting the page**: a page whose `` carried a global popup class had its entire body removed, leaving an empty result. (#2163, thanks @Nalhin) +- **Body-visibility timeout**: now configurable and validated, and the timeout warning is emitted even when `verbose=False`. (#2117, #2131, #2145, issues #2116, #2129, #2144) + +### Documentation + +- Self-hosting and migration guides updated for 0.9.x. (#2093) +- The `PDFCrawlerStrategy` plus `PDFContentScrapingStrategy` pairing requirement is now documented. + +## Tests + +- `tests/unit/test_pdf_download_limits.py`, 22 tests: per-hop destination validation, DNS rebinding, redirect bounds, byte and page caps, untrusted-body clamping, and temp-file cleanup on abort. +- `tests/unit/test_pdf_html_escaping.py`, 7 tests: escaping of PDF paragraph text in `cleaned_html`. +- `deploy/docker/tests/test_security_pdf_image_write.py`, 6 tests: rejection of image-write fields from untrusted bodies. +- Docker endpoint coverage for crawl failures and for the per-URL `crawler_configs` PDF guard. + +## Breaking changes + +None. + +Two defaults changed, which is worth knowing if you process large PDFs in a self-hosted deployment: + +- `PDFContentScrapingStrategy` now caps downloads at 100 MiB and parsing at 2000 pages. Raise them with `max_pdf_bytes` and `max_pdf_pages` when you call it from the SDK. Requests arriving over the Docker API cannot raise them past those caps. +- `limits.wall_clock_s` in `deploy/docker/config.yml` is now `300` instead of `0`. Set it back to `0` if you deliberately want no per-crawl deadline. + +## Upgrade + +```bash +pip install -U crawl4ai +crawl4ai-doctor # verify installation +``` + +Docker users: pull the latest image once the Docker release workflow finishes. + +```bash +docker pull unclecode/crawl4ai:0.9.3 +``` + +## Acknowledgments + +Thank you to Zhixi "Jace" Sun, Nguyen Tran Thanh Lam ([c240030](https://github.com/c240030)), and [e1codes](https://github.com/e1codes) for reporting these issues privately and giving us time to fix them before disclosure. All reporters are listed in [SECURITY-CREDITS.md](https://github.com/unclecode/crawl4ai/blob/main/SECURITY-CREDITS.md). + +Thanks to the community contributors behind the bug fixes in this release: @nightcityblade (#2130, #2131, #2117, #2134, #2081), @weike-zhang (#2148), and @Nalhin (#2163). + +If you find a security issue in Crawl4AI, please report it privately. See [SECURITY.md](https://github.com/unclecode/crawl4ai/blob/main/SECURITY.md). + +## Support & Resources + +- [Documentation](https://docs.crawl4ai.com) +- [GitHub Issues](https://github.com/unclecode/crawl4ai/issues) +- [Discord Community](https://discord.gg/crawl4ai) diff --git a/docs/md_v2/advanced/pdf-parsing.md b/docs/md_v2/advanced/pdf-parsing.md index 909c0dd17..3d2e7cf71 100644 --- a/docs/md_v2/advanced/pdf-parsing.md +++ b/docs/md_v2/advanced/pdf-parsing.md @@ -7,6 +7,8 @@ Crawl4AI provides specialized strategies for handling and extracting content fro ### Overview `PDFCrawlerStrategy` is an implementation of `AsyncCrawlerStrategy` designed specifically for PDF documents. Instead of interpreting the input URL as an HTML webpage, this strategy treats it as a pointer to a PDF file. It doesn't perform deep crawling or HTML parsing itself but rather prepares the PDF source for a dedicated PDF scraping strategy. Its primary role is to identify the PDF source (web URL or local file) and pass it along the processing pipeline in a way that `AsyncWebCrawler` can handle. +> **Must be paired with `PDFContentScrapingStrategy`.** `PDFCrawlerStrategy` fetches and parses nothing on its own — it returns a placeholder response and lets the scraping strategy do the download and extraction. Always set `CrawlerRunConfig(scraping_strategy=PDFContentScrapingStrategy())`. With any other scraping strategy the crawl still reports `success=True`, but the result contains only the placeholder text instead of your PDF content. + ### When to Use Use `PDFCrawlerStrategy` when you need to: - Process PDF files using the `AsyncWebCrawler`. @@ -20,8 +22,9 @@ Use `PDFCrawlerStrategy` when you need to: - **`async crawl(self, url: str, **kwargs) -> AsyncCrawlResponse`**: - This method is called by the `AsyncWebCrawler` during the `arun` process. - It takes the `url` (which should point to a PDF) and creates a minimal `AsyncCrawlResponse`. - - The `html` attribute of this response is typically empty or a placeholder, as the actual PDF content processing is deferred to the `PDFContentScrapingStrategy` (or a similar PDF-aware scraping strategy). + - The `html` attribute of this response is a short placeholder string, as the actual PDF content processing is deferred to the `PDFContentScrapingStrategy` (or a similar PDF-aware scraping strategy). - It sets `response_headers` to indicate "application/pdf" and `status_code` to 200. + - It also sets `placeholder_html=True` on the response. This tells `AsyncWebCrawler` that the `html` is a stand-in, so the anti-bot content heuristics skip it — otherwise the short placeholder would be misread as a blocked, near-empty page and the crawl would be marked failed. Any custom crawler strategy that defers content extraction to its scraping strategy should set the same flag. - **`async close(self)`**: - A method for cleaning up any resources used by the strategy. For `PDFCrawlerStrategy`, this is usually minimal. - **`async __aenter__(self)` / `async __aexit__(self, exc_type, exc_val, exc_tb)`**: @@ -37,8 +40,8 @@ async def main(): # Initialize the PDF crawler strategy pdf_crawler_strategy = PDFCrawlerStrategy() - # PDFCrawlerStrategy is typically used in conjunction with PDFContentScrapingStrategy - # The scraping strategy handles the actual PDF content extraction + # PDFCrawlerStrategy must be paired with PDFContentScrapingStrategy — + # the scraping strategy is what actually downloads and extracts the PDF pdf_scraping_strategy = PDFContentScrapingStrategy() run_config = CrawlerRunConfig(scraping_strategy=pdf_scraping_strategy) diff --git a/docs/md_v2/api/parameters.md b/docs/md_v2/api/parameters.md index 568e14c30..f9f759a58 100644 --- a/docs/md_v2/api/parameters.md +++ b/docs/md_v2/api/parameters.md @@ -159,6 +159,7 @@ Use these for controlling whether you read or write from a local content cache. | **`c4a_script`** | `str or list[str]` (None) | C4A script that compiles to JavaScript. Alternative to writing raw JS. | | **`js_only`** | `bool` (False) | If `True`, indicates we're reusing an existing session and only applying JS. No full reload. | | **`ignore_body_visibility`** | `bool` (True) | Skip checking if `` is visible. Usually best to keep `True`. | +| **`body_visibility_timeout`** | `int` (30000) | Maximum time in milliseconds to wait for `` to become visible. Must be positive. | | **`scan_full_page`** | `bool` (False) | If `True`, auto-scroll the page to load dynamic content (infinite scroll). | | **`scroll_delay`** | `float` (0.2) | Delay between scroll steps when scanning the full page (`scan_full_page=True`) or capturing full-page screenshots. | | **`max_scroll_steps`** | `int or None` (None) | Maximum number of scroll steps during full page scan. If None, scrolls until entire page is loaded. | diff --git a/docs/md_v2/complete-sdk-reference.md b/docs/md_v2/complete-sdk-reference.md index aa0517b2d..1f3299075 100644 --- a/docs/md_v2/complete-sdk-reference.md +++ b/docs/md_v2/complete-sdk-reference.md @@ -1791,6 +1791,7 @@ run_cfg = CrawlerRunConfig( | **`js_code_before_wait`** | `str or list[str]` (None) | JavaScript to run **before** `wait_for`. Use for triggering loading that `wait_for` then checks. | | **`js_only`** | `bool` (False) | If `True`, indicates we're reusing an existing session and only applying JS. No full reload. | | **`ignore_body_visibility`** | `bool` (True) | Skip checking if `` is visible. Usually best to keep `True`. | +| **`body_visibility_timeout`** | `int` (30000) | Maximum time in milliseconds to wait for `` to become visible. Must be positive. | | **`scan_full_page`** | `bool` (False) | If `True`, auto-scroll the page to load dynamic content (infinite scroll). | | **`scroll_delay`** | `float` (0.2) | Delay between scroll steps when scanning the full page (`scan_full_page=True`) or capturing full-page screenshots. | | **`process_iframes`** | `bool` (False) | Inlines iframe content for single-page extraction. | diff --git a/docs/md_v2/core/self-hosting.md b/docs/md_v2/core/self-hosting.md index 1420774d1..68a8a1065 100644 --- a/docs/md_v2/core/self-hosting.md +++ b/docs/md_v2/core/self-hosting.md @@ -1,13 +1,17 @@ # Self-Hosting Crawl4AI 🚀 -> **🔐 0.9.0 is secure-by-default (breaking changes).** The self-hosted Docker -> server now requires authentication by default, binds to loopback unless you +> **🔐 0.9.0+ is secure-by-default (breaking changes).** The self-hosted Docker +> server requires authentication by default, binds to loopback unless you > set a token, validates request bodies against a strict trust boundary, uses > declarative hooks instead of inline Python, and returns artifact ids for > screenshot/pdf. If you are upgrading from 0.8.x, read the > [migration guide](https://github.com/unclecode/crawl4ai/blob/main/deploy/docker/MIGRATION.md) -> first. Some examples below are being updated for 0.9.0; the migration guide is -> the authoritative reference for the new defaults. +> first. +> +> **The single most important thing to know:** without a `CRAWL4AI_API_TOKEN`, +> the server binds loopback **inside the container** — published ports will +> answer with *connection reset* even though the container reports healthy. +> Every quickstart below therefore starts by setting a token. **Take Control of Your Web Crawling Infrastructure** @@ -58,7 +62,7 @@ When you self-host, you can scale from a single container to a full browser infr ## Prerequisites Before we dive in, make sure you have: -- Docker installed and running (version 20.10.0 or higher), including `docker compose` (usually bundled with Docker Desktop). +- Docker installed and running (version 20.10.0 or higher), including `docker compose` v2.24+ (usually bundled with Docker Desktop). - `git` for cloning the repository. - At least 4GB of RAM available for the container (more recommended for heavy use). - Python 3.10+ (if using the Python SDK). @@ -76,13 +80,13 @@ Pull and run images directly from Docker Hub without building locally. #### 1. Pull the Image -Our latest release is `0.8.0`. Images are built with multi-arch manifests, so Docker automatically pulls the correct version for your system. +Our latest release is `0.9.2`. Images are built with multi-arch manifests, so Docker automatically pulls the correct version for your system. -> 💡 **Note**: The `latest` tag points to the stable `0.8.0` version. +> 💡 **Note**: The `latest` tag points to the most recent stable version. ```bash # Pull the latest version -docker pull unclecode/crawl4ai:0.8.0 +docker pull unclecode/crawl4ai:0.9.2 # Or pull using the latest tag docker pull unclecode/crawl4ai:latest @@ -121,7 +125,23 @@ EOL ``` > 🔑 **Note**: Keep your API keys secure! Never commit `.llm.env` to version control. -#### 3. Run the Container +#### 3. Set an API Token (Required to Reach the Server) + +Since 0.9.0 the server is secure-by-default: **without a token it binds +loopback inside the container**, so the published port answers with +*connection reset* — even though `docker ps` shows the container healthy and +the port mapped. Generate a token first: + +```bash +export CRAWL4AI_API_TOKEN="$(openssl rand -hex 32)" +``` + +> ⚠️ Use the explicit form `-e CRAWL4AI_API_TOKEN="$CRAWL4AI_API_TOKEN"` below. +> The shorthand `-e CRAWL4AI_API_TOKEN` (no value) forwards the variable from +> the *current shell* — in a shell where it isn't set, it silently passes an +> empty value and the server starts in loopback mode with no host-side warning. + +#### 4. Run the Container * **Basic run:** ```bash @@ -129,6 +149,7 @@ EOL -p 11235:11235 \ --name crawl4ai \ --shm-size=1g \ + -e CRAWL4AI_API_TOKEN="$CRAWL4AI_API_TOKEN" \ unclecode/crawl4ai:latest ``` @@ -140,12 +161,30 @@ EOL --name crawl4ai \ --env-file .llm.env \ --shm-size=1g \ + -e CRAWL4AI_API_TOKEN="$CRAWL4AI_API_TOKEN" \ unclecode/crawl4ai:latest ``` -> The server will be available at `http://localhost:11235`. Visit `/playground` to access the interactive testing interface. +The server will be available at `http://localhost:11235` after a short startup +(allow ~10 seconds; the healthcheck allows up to 40). Verify: -#### 4. Stopping the Container +```bash +curl http://localhost:11235/health # no auth needed for /health +``` + +Every other endpoint requires `Authorization: Bearer $CRAWL4AI_API_TOKEN`. +Visit `/playground` for the interactive testing interface and `/dashboard` for +the monitoring UI — both have an **API token** bar in the top navigation; paste +your token there and click **Set**. + +> 💡 **Troubleshooting "connection reset":** during the ~10s startup window the +> published port also answers with connection reset — the same symptom as the +> missing-token failure. If resets persist after 20s, check +> `docker logs crawl4ai` for `binding loopback only`: that means no token +> reached the container (note the `127.0.0.1` in that log line is the +> *container's* loopback, not your host's — the port mapping cannot reach it). + +#### 5. Stopping the Container ```bash docker stop crawl4ai && docker rm crawl4ai @@ -154,7 +193,7 @@ docker stop crawl4ai && docker rm crawl4ai #### Docker Hub Versioning Explained * **Image Name:** `unclecode/crawl4ai` -* **Tag Format:** `LIBRARY_VERSION[-SUFFIX]` (e.g., `0.8.0`) +* **Tag Format:** `LIBRARY_VERSION[-SUFFIX]` (e.g., `0.9.2`) * `LIBRARY_VERSION`: The semantic version of the core `crawl4ai` Python library * `SUFFIX`: Optional tag for release candidates (``) and revisions (`r1`) * **`latest` Tag:** Points to the most recent stable version @@ -171,17 +210,39 @@ git clone https://github.com/unclecode/crawl4ai.git cd crawl4ai ``` -#### 2. Environment Setup (API Keys) +#### 2. Environment Setup (Required) -If you plan to use LLMs, copy the example environment file and add your API keys. This file should be in the **project root directory**. +Export an API token in your shell — the compose file passes it into the +container. **Required**, or the server will be unreachable (loopback-only, +published port → connection reset): + +```bash +export CRAWL4AI_API_TOKEN="$(openssl rand -hex 32)" +``` + +Prefer a file? Put the same line in a `.env` file in the project root — +compose reads it automatically on every run, no export needed (if both are +set, the shell export wins): + +```bash +echo "CRAWL4AI_API_TOKEN=$(openssl rand -hex 32)" > .env +``` + +If you use LLMs, also create `.llm.env` in the **project root directory** with +your API keys (optional — compose starts fine without it): ```bash # Make sure you are in the 'crawl4ai' root directory cp deploy/docker/.llm.env.example .llm.env -# Now edit .llm.env and add your API keys +# Now edit .llm.env and add your LLM API keys ``` +> ⚠️ Run the export in the **same shell** you run `docker compose up` from. +> With compose, only the shell export or a `.env` line carries the token — a +> `CRAWL4AI_API_TOKEN=` line in `.llm.env` is overridden by the compose +> passthrough. + **Flexible LLM Provider Configuration:** The Docker setup now supports flexible LLM provider configuration through a hierarchical system: @@ -249,7 +310,9 @@ The `docker-compose.yml` file in the project root provides a simplified approach ENABLE_GPU=true docker compose up --build -d ``` -> The server will be available at `http://localhost:11235`. +> The server will be available at `http://localhost:11235` (allow ~10 seconds +> for startup). All endpoints except `GET /health` require +> `Authorization: BearerLocked body
+Body is position:fixed while a modal is open; it must not be removed.
+