fix(httpx): honor session cookies across HTTP client request paths

[Ayush7614]

Summary

• Httpx: send_request / stream / crawl share _build_request for outbound session cookies. Clients stay shared per proxy (not per session); cookies are re-applied in _HttpxTransport on every hop so they survive redirects without pinning a client to each jar.
• Impit (default client): honors persist_cookies_per_session. When persistence is enabled, the session jar is attached to the client; when disabled, existing cookies are sent via a Cookie header so the shared client stays cached. Clients are cached by (proxy, CookieJar) identity.
• Httpx fingerprints: Accept / Accept-Language / User-Agent come from a single fingerprint profile. HeaderGenerator.get_common_headers / get_random_user_agent_header are deprecated in favor of get_specific_headers.

Why

context.send_request() silently dropped session cookies under Httpx, breaking auth that worked for navigation (including after redirects). Impit's documented persist_cookies_per_session=False never took effect because the session jar was always attached and mutated in place.

Test plan

• tests/unit/http_clients/test_http_clients.py — shared cookie send/persist/redirect coverage across clients
• tests/unit/http_clients/test_httpx.py — single-generate() fingerprint headers; proxy-only client cache across sessions
• tests/unit/http_clients/test_impit.py — Cookie-header path when persist is disabled

[Copilot]

Pull request overview

This PR fixes inconsistent cookie handling across HTTP client request paths so that session cookies are reliably sent (and optionally persisted) whether requests go through crawler navigation (crawl) or handler-level calls (send_request/stream). It also makes Httpx’s fingerprint-derived headers internally consistent by sourcing Accept, Accept-Language, and User-Agent from a single generated fingerprint profile.

Changes:

• Httpx: send_request/stream now attach outbound session cookies via the shared request-building path (matching crawl behavior).
• Impit: implements persist_cookies_per_session semantics, caches clients by (proxy, cookie-jar identity), and adds client closing on cleanup/eviction.
• Tests: adds coverage for cookie sending/persistence toggles, single-fingerprint headers, and Impit cleanup cache reset.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
tests/unit/http_clients/test_http_clients.py	Adds unit coverage for session cookie sending in send_request/stream, persistence on/off behavior, single-fingerprint headers, and Impit cache cleanup.
src/crawlee/http_clients/_impit.py	Honors cookie persistence flag by using a resolved jar (shared vs copy), introduces client caching keyed by proxy + cookie jar identity, and closes cached clients on cleanup/eviction.
src/crawlee/http_clients/_httpx.py	Ensures handler-level requests attach session cookies and derives Accept/Accept-Language/User-Agent from one fingerprint profile.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Mantisus]

Thank you for your contributions!

When persist_cookies_per_session is False, _resolve_cookie_jar builds a fresh jar for every request, and since the cache key includes the jar identity, that means a new AsyncClient per request. Consider building the Cookie header directly instead of passing a CookieJar to AsyncClient, then the client can stay cached and shared.

For example:

import urllib.request
from http.cookiejar import CookieJar


def get_cookie_header(jar: CookieJar, url: str, headers: HttpHeaders | None = None) -> str:
    request = urllib.request.Request(url, headers=dict(headers) if headers else {})
    jar.add_cookie_header(request)
    return request.get_header('Cookie', '')

[Ayush7614]

Thanks for the review @Mantisus!

Addressed in dccf1e7:

• When persist_cookies_per_session=False, cookies are now sent via a Cookie header (using CookieJar.add_cookie_header) instead of attaching a fresh jar — so the Impit client stays cached/shared by proxy.
• LRU eviction now uses cachetools.LRUCache.popitem().
• Removed the fire-and-forget create_task(close...) path (and the __aexit__ close helper) since Impit’s Rust side already handles resource cleanup well.

[Mantisus]

Thanks for the update, just a few more points

[Ayush7614]

Thanks @Mantisus — addressed in 086b115:

• get_header('Cookie', '')
• simplified Cookie-header merge via walrus + _get_cookie_header
• dropped redundant or None and manual popitem
• moved Httpx/Impit-specific tests into test_httpx.py / test_impit.py

[Mantisus]

Please update the PR description to match the current state, otherwise LGTM

[Ayush7614]

Updated the PR description to match the current implementation. Thanks @Mantisus!

[Ayush7614]

Thanks @vdusek

Addressed in d4f5fae / 6aaadc4:

• Httpx: attach session cookie jar on the client so cookies survive redirects; crawl reuses _build_request; shared redirect cookie coverage in test_session_cookies_survive_redirect.
• Impit: cache keyed by (proxy_url, CookieJar) identity; Cookie-header path when persist_cookies_per_session=False, covered by test_persist_false_still_sends_session_cookies.
• Deprecated HeaderGenerator.get_common_headers / get_random_user_agent_header.
• Fingerprint test now mocks _generator.generate and asserts a single call (so a dual-generate() regression would fail).

[Ayush7614]

Type-check failures on the previous run came from test_client_cache_is_shared_across_sessions: async with HttpxHttpClient() as client widens to HttpClient, so accessing _client_by_proxy_url failed under ty.

Fixed in bf9a8d0 by binding the HttpxHttpClient first (same pattern as the Impit cache test). Local ty check is clean.

[Ayush7614]

cc: @vdusek

[Mantisus]

With this new approach that uses transport, cookies=session.cookies.jar doesn't make sense

Also, the session is already being passed to the transport. Use the session's jar no need to pass it separately.