refactor(request-loaders)!: Add pacing signals to RequestManager

docs/guides/request_loaders.mdx

@@ -25,7 +25,7 @@ The [`request_loaders`](https://github.com/apify/crawlee-python/tree/master/src/
 The [`request_loaders`](https://github.com/apify/crawlee-python/tree/master/src/crawlee/request_loaders) sub-package introduces the following abstract classes:
 
 - <ApiLink to="class/RequestLoader">`RequestLoader`</ApiLink>: The base interface for reading requests in a crawl.
-- <ApiLink to="class/RequestManager">`RequestManager`</ApiLink>: Extends `RequestLoader` with write capabilities.
+- <ApiLink to="class/RequestManager">`RequestManager`</ApiLink>: Extends `RequestLoader` with write capabilities, and with the pacing signals a crawler reports back.
 - <ApiLink to="class/RequestManagerTandem">`RequestManagerTandem`</ApiLink>: Combines a read-only `RequestLoader` with a writable `RequestManager`.
 
 And specific request loader implementations:
@@ -71,6 +71,7 @@ class RequestManager {
     + add_request()
     + add_requests_batched()
     + reclaim_request()
+    + record_pacing_signal()
     + purge()
     + drop()
 }
@@ -156,6 +157,22 @@ When using persistence with `SitemapRequestLoader`, make sure to use the context
 
 The <ApiLink to="class/RequestManager">`RequestManager`</ApiLink> extends `RequestLoader` with write capabilities. In addition to reading requests, a request manager can add and reclaim them. This is essential for dynamic crawling projects where new URLs may emerge during the crawl process, or when certain requests fail and need to be retried. For more details, refer to the <ApiLink to="class/RequestManager">`RequestManager`</ApiLink> API reference.
 
+### Pacing signals
+
+A request manager decides when each request goes out, so the crawler reports back to it what a site said about the pace it wants. The crawler does so through <ApiLink to="class/RequestManager#record_pacing_signal">`RequestManager.record_pacing_signal`</ApiLink>, with one of two signals:
+
+- <ApiLink to="class/RateLimitedSignal">`RateLimitedSignal`</ApiLink>: A site refused a request because the crawler was going too fast. It carries the wait the site asked for, if any.
+- <ApiLink to="class/MinIntervalSignal">`MinIntervalSignal`</ApiLink>: A site declared a minimum interval between requests, such as a robots.txt `Crawl-delay`.
+
+Each signal also has a `scope`, which says how much of the URL space it covers. A manager may apply a signal to a wider scope than it was given, but never to a narrower one, which would leave part of what the signal covers unpaced. A manager that can't honor a scope raises an error.
+
+To implement `record_pacing_signal` in your own request manager:
+
+- Return `False` when the manager doesn't pace its requests, as <ApiLink to="class/RequestQueue">`RequestQueue`</ApiLink> does. The crawler then logs that the signal was dropped. New signals may be added in a minor release, so also return `False` for a signal you don't handle.
+- Return `True` for a `RateLimitedSignal` only if the URL won't be dispatched again until a positive delay has passed. The crawler defers such a request without spending a retry.
+- Treat a `RateLimitedSignal` for a URL that's already held back as part of the same refusal. The crawler may report one refusal more than once, and the repeats carry no `wait`.
+- If the manager wraps another one, forward every signal to it without inspecting it, as <ApiLink to="class/RequestManagerTandem">`RequestManagerTandem`</ApiLink> does. Otherwise a pacer nested inside it never receives the signal.
+
 ## Request manager tandem
 
 The <ApiLink to="class/RequestManagerTandem">`RequestManagerTandem`</ApiLink> class allows you to combine the read-only capabilities of a `RequestLoader` (like <ApiLink to="class/RequestList">`RequestList`</ApiLink>) with the read-write capabilities of a `RequestManager` (like <ApiLink to="class/RequestQueue">`RequestQueue`</ApiLink>). This is useful for scenarios where you need to load initial requests from a static source (such as a file or database) and dynamically add or retry requests during the crawl. Additionally, it provides deduplication capabilities, ensuring that requests are not processed multiple times.

docs/guides/request_throttling.mdx

@@ -42,6 +42,8 @@ To use request throttling, create a <ApiLink to="class/ThrottlingRequestManager"
 
 5. **Cooldown handling**: While a domain is in a cooldown, its queued requests don't count as dispatchable, so the crawler's autoscaled pool idles instead of keeping a worker slot blocked. The requests still count towards completion, so the crawl waits for them and finishes only once every one has been handled. A domain that never stops rate-limiting doesn't keep the crawl running forever. For details, see [Persistent rate limiting](#persistent-rate-limiting).
 
+The crawler reports 429 responses and crawl-delays through <ApiLink to="class/RequestManager#record_pacing_signal">`RequestManager.record_pacing_signal`</ApiLink>, which <ApiLink to="class/RequestManagerTandem">`RequestManagerTandem`</ApiLink> forwards to the manager it wraps, so a `ThrottlingRequestManager` also paces its domains as the writable side of a tandem. For details, see [Pacing signals](./request-loaders#pacing-signals).
+
 ## Persistent rate limiting
 
 A 429 from a throttled domain doesn't fail the request. The request goes back to its queue and waits out the backoff, while the crawler keeps crawling other domains. The retry doesn't count towards `max_request_retries` or `max_session_rotations`, and the session isn't marked as bad. The 429 doesn't reach `error_handler` or `failed_request_handler`. Setting `ignore_http_error_status_codes={429}` doesn't pass the response to your request handler either, because the request is retried once the backoff ends.

docs/guides/scrapy_migration.mdx

@@ -212,7 +212,7 @@ Both Scrapy and Crawlee have settings to influence the scraping throughput, but
 
 The closest built-in analog to the `AutoThrottle` extension is the <ApiLink to="class/ThrottlingRequestManager">`ThrottlingRequestManager`</ApiLink>, covered in the [Request throttling guide](./request-throttling). It differs in what drives the backoff. Scrapy's `AutoThrottle` adapts to measured latency, while Crawlee's manager reacts to explicit signals: HTTP 429 replies and `robots.txt` crawl-delay directives. It wraps a <ApiLink to="class/RequestQueue">`RequestQueue`</ApiLink> and applies the backoff per domain. You list the domains to watch, and the crawler reports the signals to the manager on its own.
 
-Crawl-delay works only when the crawler fetches `robots.txt`. Projects generated by `scrapy startproject` obey `robots.txt` out of the box, but the default for Crawlee is to ignore it. Passing <ApiLink to="class/BasicCrawler#__init__">`respect_robots_txt_file=True`</ApiLink> tells Crawlee to enforce the `Disallow` rules. Together with <ApiLink to="class/ThrottlingRequestManager">`ThrottlingRequestManager`</ApiLink> it also reads crawl-delay, otherwise it logs a warning.
+Crawl-delay works only when the crawler fetches `robots.txt`. Projects generated by `scrapy startproject` obey `robots.txt` out of the box, but the default for Crawlee is to ignore it. Passing <ApiLink to="class/BasicCrawler#__init__">`respect_robots_txt_file=True`</ApiLink> tells Crawlee to enforce the `Disallow` rules. Together with <ApiLink to="class/ThrottlingRequestManager">`ThrottlingRequestManager`</ApiLink> it also enforces crawl-delay. For a domain the manager doesn't pace, it logs a warning instead.
 
 <Tabs groupId="scrapy-migration-throttling">
     <TabItem value="scrapy" label="Scrapy">

docs/upgrading/upgrading_to_v2.md

@@ -264,6 +264,32 @@ A `None` from `fetch_next_request` no longer means the loader is finished. It on
 
 The crawler now reads a stall from the status, so it also sees a `ThrottlingRequestManager` wrapped in a `RequestManagerTandem`. Such a crawl now ends with `PersistentRateLimitError` once a domain stalls, as it already did with a `ThrottlingRequestManager` passed directly. Before, the crawler didn't detect the stall.
 
+## `record_pacing_signal` replaces the `ThrottlingRequestManager` delay methods
+
+`RequestManager` has a new abstract method, `record_pacing_signal`. The crawler reports HTTP 429 responses and robots.txt `Crawl-delay` directives through it, so a `ThrottlingRequestManager` now receives them wherever it sits, including inside a `RequestManagerTandem`. In v1, the crawler reached the throttler only when it was passed directly.
+
+`ThrottlingRequestManager.record_domain_delay` and `set_crawl_delay` are replaced by `record_pacing_signal`, which takes a `RateLimitedSignal` or a `MinIntervalSignal`. `is_throttled` and `record_success` are removed without a replacement. The manager resets the backoff on its own once a domain stops rate-limiting.
+
+```python
+# v1
+throttler.record_domain_delay(url, retry_after=timedelta(seconds=30))
+throttler.set_crawl_delay(url, 5)
+
+# v2
+from crawlee.request_loaders import MinIntervalSignal, RateLimitedSignal
+
+throttler.record_pacing_signal(RateLimitedSignal(url, wait=timedelta(seconds=30)))
+throttler.record_pacing_signal(
+    MinIntervalSignal(url, interval=timedelta(seconds=5), scope='hostname')
+)
+```
+
+If you have a custom request manager, implement `record_pacing_signal`. Return `False` if the manager doesn't pace requests or doesn't handle the signal's `reason`, and forward every signal if it wraps another manager. For details, see [Pacing signals](../guides/request-loaders#pacing-signals).
+
+`ThrottlingRequestManager` now rejects a `base_delay` or `max_delay` that isn't positive, so a zero `base_delay` no longer works as a way to honor only `Retry-After`.
+
+A `RequestThrottledError` you raise yourself now starts the domain's backoff, and the request is deferred if the request manager paces its domain. In v1, it was an ordinary error unless you had called `record_domain_delay` first. A deferred request never fails, so raise the error only when the server has refused the request. In `AdaptivePlaywrightCrawler`, such an error from the static run now defers the request instead of falling back to the browser.
+
 ## HTTP clients
 
 `HttpClient` follows the design of Crawlee JS v4. A client sends a single request, and the base class handles redirects and cookies the same way for every client.
@@ -381,6 +407,10 @@ The ERROR message the crawler logs before raising `PersistentRateLimitError` now
 
 The two DEBUG messages that `RequestQueue.is_finished` logged, "Background add requests tasks are still in progress." and "The request queue is finished.", are removed together with the method.
 
+The warnings about an HTTP 429 that the request manager doesn't pace are reworded and logged once per domain. The warning `run()` logged at the start about `respect_robots_txt_file` without a `ThrottlingRequestManager` is replaced by a new one, logged once per domain whose robots.txt `Crawl-delay` the request manager doesn't pace. The warning about a `RequestThrottledError` handled as an ordinary error is reworded, and a new ERROR is logged when the request manager fails to record that rate limit. Another new ERROR is logged once per domain when the request manager fails to record a robots.txt `Crawl-delay`.
+
+The `ThrottlingRequestManager` warning about capping a delay now calls it a "requested wait" delay instead of a "Retry-After header" delay. Its DEBUG messages about a 429 ignored during an active backoff and about resetting a domain's rate limit state are removed, and its DEBUG message about a domain's crawl-delay shows the delay with one decimal place.
+
 ## Session retirement is permanent
 
 In v1, `Session.retire()` only added `max_error_score` to the error score, so a later `mark_good()` could lower it again and make the retired session usable. In v2, a separate `retired` flag tracks retirement, exposed as the `Session.retired` property, and a retired session is never usable again. Create a new session to replace it.

src/crawlee/crawlers/_abstract_http/_abstract_http_crawler.py

@@ -315,7 +315,7 @@ async def _handle_status_code_response(
             context: The current crawling context containing the HTTP response.
 
         Raises:
-            RequestThrottledError: If the response is a 429 from a domain throttled by a `ThrottlingRequestManager`.
+            RequestThrottledError: If the response is a 429 from a domain the crawler's request manager paces.
             SessionError: If the status code indicates the session is blocked.
             HttpStatusCodeError: If the status code represents a server error or is explicitly configured as an error.
             HttpClientStatusCodeError: If the status code represents a client error.

src/crawlee/crawlers/_adaptive_playwright/_adaptive_playwright_crawler.py

@@ -401,7 +401,7 @@ async def _run_request_handler(self, context: BasicCrawlingContext) -> None:
                 if (
                     isinstance(static_run.exception, (ContextPipelineInitializationError, RequestHandlerError))
                     and isinstance(static_run.exception.wrapped_exception, RequestThrottledError)
-                    and self._is_held_back(context.request)
+                    and await self._report_rate_limit(context.request)
                 ):
                     raise static_run.exception
                 if static_run.exception: