docs: Add cookies management guide - #2097
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds a new “Cookie management” guide to the Python Crawlee docs, aimed at clarifying how cookies are stored on sessions and how to keep cookies/session state across retries (addressing #2060). It also cross-links the new guide from existing session, headers, and login documentation.
Changes:
- Added a new
Cookie managementguide covering reading/setting cookies, seeding cookies in new sessions, Playwright cookie syncing, retry patterns, and persistence across runs. - Added runnable Python code examples for the cookie guide (read/write, initial cookies, Playwright sync, retry strategies, persistence).
- Added references to the new cookie guide from existing guides (session management, HTTP headers, logging in).
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| docs/guides/session_management.mdx | Adds a cross-link to the new cookie guide from session management. |
| docs/guides/http_headers.mdx | Links cookie header discussion to the new cookie guide. |
| docs/guides/crawler_login.mdx | Links authentication/session guidance to the new cookie guide. |
| docs/guides/cookie_management.mdx | New guide documenting cookie behavior, retries, Playwright syncing, and persistence. |
| docs/guides/code_examples/cookie_management/read_write_cookies.py | Example for reading and setting session cookies. |
| docs/guides/code_examples/cookie_management/initial_cookies.py | Example for seeding cookies via create_session_settings. |
| docs/guides/code_examples/cookie_management/playwright_cookies.py | Example showing Playwright ↔ session cookie syncing behavior. |
| docs/guides/code_examples/cookie_management/retry_single_session.py | Example for preserving cookies across retries with a single-session pool. |
| docs/guides/code_examples/cookie_management/retry_pinned_session.py | Example for “retrying” by re-enqueueing a request pinned to a session. |
| docs/guides/code_examples/cookie_management/retry_restore_cookies.py | Example for restoring cookies via use_state + pre_navigation_hook. |
| docs/guides/code_examples/cookie_management/persist_cookies.py | Example for persisting cookies across runs via session pool persistence. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
21351a7 to
3ff1281
Compare
vdusek
left a comment
vdusek
left a comment
There was a problem hiding this comment.
LGTM, the text and examples are great, thank you.
Leaving here a few comments from Claude to consider, once they are resolved, I'll merge it (you can either address them or leave it as it is and just mark as resolved, up to you, they are mostly nits).
|
|
||
| ## Reading and setting cookies | ||
|
|
||
| Access the jar through <ApiLink to="class/Session#cookies">`context.session.cookies`</ApiLink>. Use <ApiLink to="class/SessionCookies#set">`set`</ApiLink> to add a cookie, iterate the jar to read all cookies, and index by name to read one value. |
There was a problem hiding this comment.
Note: SessionCookies.__getitem__ raises KeyError for a missing cookie rather than returning None (the str | None return type is about Cookie.value, not absence). Since readers will naturally index cookies the server may or may not have set, one clause noting that would help.
|
|
||
| ::: | ||
|
|
||
| A cookie has a `name`, a `value`, and optional parameters such as `domain`. If a cookie has no `domain`, it applies to any domain. For the full set of parameters, see <ApiLink to="class/CookieParam">`CookieParam`</ApiLink>. To dump the whole jar, use <ApiLink to="class/SessionCookies#get_cookies_as_dicts">`get_cookies_as_dicts()`</ApiLink>, and to load several cookies at once, use <ApiLink to="class/SessionCookies#set_cookies">`set_cookies()`</ApiLink>. |
There was a problem hiding this comment.
Suggestion: the claim is accurate, but worth stating the consequence in a guide that opens by naming login state and CSRF tokens. I confirmed against http.cookiejar that a domain-less cookie goes to every host:
https://httpbingo.org/x -> nodomain=1; scoped=2
https://evil.example.com/x -> nodomain=1
The examples already do the right thing by passing an explicit domain. The prose just never says why that matters.
| token = context.session.cookies['csrf_token'] | ||
| context.log.info(f'CSRF token: {token}') | ||
|
|
||
| # The server sets a `session_id` cookie and echoes the request cookies back. |
There was a problem hiding this comment.
Nit: the handler never reads the response body, so the "echoes the request cookies back" half describes something the reader can't observe here.
| # The server sets a `session_id` cookie and echoes the request cookies back. | |
| # The server sets a `session_id` cookie on the response. |
| return | ||
|
|
||
| # First attempt: establish cookies, then raise to trigger a retry. | ||
| if not context.session.cookies.get_cookies_as_dicts(): |
There was a problem hiding this comment.
Suggestion: SessionCookies implements __bool__, so the jar can be tested directly instead of building the whole list of dicts just to check emptiness.
| if not context.session.cookies.get_cookies_as_dicts(): | |
| if not context.session.cookies: |
| # Read a session before crawling. On the first run its jar is empty. On | ||
| # later runs the cookies from the previous run are already restored. | ||
| session = await session_pool.get_session() | ||
| print(f'Cookies before run: {session.cookies.get_cookies_as_dicts()}') |
There was a problem hiding this comment.
Nit: since this is a RunnableCodeBlock, a single "Run on Apify" click only ever prints an empty jar. The point lands on the second run. The prose does explain it, so fine as-is, just noting the button is a little anticlimactic here.
|
|
||
| ### Re-apply cookies in a pre-navigation hook | ||
|
|
||
| Keeps rotation and the raise-to-retry flow, and shares the cookies across every session. Snapshot the cookies after the setup step into the crawler-wide <ApiLink to="class/UseStateFunction">`use_state`</ApiLink> store. Then re-apply them onto whichever session handles the request in a <ApiLink to="class/AbstractHttpCrawler#pre_navigation_hook">`pre_navigation_hook`</ApiLink>, which runs before navigation and receives `context.session`. |
There was a problem hiding this comment.
Note: use_state is marked experimental. UseStateFunction's docstring warns that "the behavior and interface may change in future versions" (src/crawlee/_types.py:594). This guide is the only page in docs/ that mentions it, and it presents the approach as an equal peer to two stable ones. A half-sentence or a :::note would set expectations.
|
|
||
| ## Cookies with PlaywrightCrawler | ||
|
|
||
| <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> keeps the session jar and the browser context in sync. Before navigation, the session's cookies are loaded into the browser context. After the handler returns, cookies from the browser context are merged back onto the session. The merge captures whatever the page picked up through `Set-Cookie`, JavaScript, or redirects. |
There was a problem hiding this comment.
Note: the merge is broader than "cookies from the browser context". _get_cookies calls page.context.cookies() with no URL filter (_playwright_crawler.py:610), so with the default shared context it pulls in every cookie in that context, including ones other sessions' pages set for unrelated domains. The use_incognito_pages paragraph covers the outbound direction; this inbound leak onto the session jar is the more surprising half and isn't mentioned.
Description
Issues