Problem
_clamp_untrusted caps page_timeout and wait_for_timeout at _MAX_TIMEOUT_MS = 60_000 for any config arriving over HTTP. The Docker server passes Provenance.UNTRUSTED at every /crawl call site, so a client asking for more is silently given 60s — no warning, no field in the response. A page that legitimately takes longer always fails with Page.goto: Timeout 60000ms exceeded, quoting a number the caller never sent.
The cap itself is reasonable for a public server. Not being able to change it is the problem: it isn't in config.yml, no env var overrides it, and the value is a module-level literal.
crawler.base_config looks like the intended escape hatch and isn't — it's applied post-deserialization with setattr, genuinely bypassing the clamp, but only for fields that are None or ''. page_timeout always carries its 60000 default, so setting it there does nothing, silently.
Expected
An operator running their own container can raise the ceiling — an env var, a limits: entry alongside wall_clock_s, or a base_config that applies to defaults too. Failing that, a clamped request should say so rather than report a timeout the caller didn't ask for.
Reproduce (docker, 0.9.2)
A server that answers after 90s:
import time
from http.server import BaseHTTPRequestHandler, HTTPServer
BODY = b"<html><body><h1>Slow</h1><p>" + b"x"*600 + b"</p></body></html>"
class H(BaseHTTPRequestHandler):
def do_GET(self):
time.sleep(90)
self.send_response(200); self.send_header("Content-Length", str(len(BODY))); self.end_headers()
self.wfile.write(BODY)
HTTPServer(("0.0.0.0", 8080), H).serve_forever()docker network create repro
docker run -d --name slowsrv --network repro -v $PWD/slow_server.py:/srv/s.py:ro python:3.12-slim python /srv/s.py
docker run -d --name c4ai --network repro -p 11235:11235 --shm-size=2gb \
-e CRAWL4AI_API_TOKEN=dev-token -e CRAWL4AI_ALLOW_INTERNAL_URLS=true unclecode/crawl4ai:0.9.2
curl -X POST http://localhost:11235/crawl -H 'authorization: Bearer dev-token' \
-H 'content-type: application/json' \
-d '{"urls":["http://slowsrv:8080/"],"crawler_config":{"page_timeout":300000,"wait_until":"domcontentloaded"}}'Before — asked for 300s:
HTTP 500 in 60.7s
Page.goto: Timeout 60000ms exceeded
After — same request, against an image built with _MAX_TIMEOUT_MS = 600_000:
HTTP 200 in 90.9s
success: true, markdown: "# Slow page\nxxxx…"
Suggested patch
Make the ceiling configurable rather than removing it:
_MAX_TIMEOUT_MS = int(os.environ.get("CRAWL4AI_MAX_TIMEOUT_MS", 60_000))An env var keeps the safe default for a public deployment while letting an operator raise it for a trusted one. A limits.max_timeout_ms key in config.yml, next to wall_clock_s, would fit the existing "0 = unbounded" convention equally well.
Separately, base_config skipping any field with a non-empty default is surprising enough to be worth documenting.
Problem
_clamp_untrustedcapspage_timeoutandwait_for_timeoutat_MAX_TIMEOUT_MS = 60_000for any config arriving over HTTP. The Docker server passesProvenance.UNTRUSTEDat every/crawlcall site, so a client asking for more is silently given 60s — no warning, no field in the response. A page that legitimately takes longer always fails withPage.goto: Timeout 60000ms exceeded, quoting a number the caller never sent.The cap itself is reasonable for a public server. Not being able to change it is the problem: it isn't in
config.yml, no env var overrides it, and the value is a module-level literal.crawler.base_configlooks like the intended escape hatch and isn't — it's applied post-deserialization withsetattr, genuinely bypassing the clamp, but only for fields that areNoneor''.page_timeoutalways carries its 60000 default, so setting it there does nothing, silently.Expected
An operator running their own container can raise the ceiling — an env var, a
limits:entry alongsidewall_clock_s, or abase_configthat applies to defaults too. Failing that, a clamped request should say so rather than report a timeout the caller didn't ask for.Reproduce (docker, 0.9.2)
A server that answers after 90s:
Before — asked for 300s:
After — same request, against an image built with
_MAX_TIMEOUT_MS = 600_000:Suggested patch
Make the ceiling configurable rather than removing it:
An env var keeps the safe default for a public deployment while letting an operator raise it for a trusted one. A
limits.max_timeout_mskey inconfig.yml, next towall_clock_s, would fit the existing "0 = unbounded" convention equally well.Separately,
base_configskipping any field with a non-empty default is surprising enough to be worth documenting.