Update scrapingbee-cli package to support Header based Authorization

.agents/skills/scrapingbee-cli/reference/amazon/product.md

@@ -21,6 +21,7 @@ scrapingbee amazon-product --output-file product.json B0DPDRNSXV --domain com
 | `--language` | string | e.g. en_US, es_US, fr_FR. |
 | `--currency` | string | USD, EUR, GBP, etc. |
 | `--add-html` | true/false | Include full HTML. |
+| `--autoselect-variant` | true/false | Auto-select the default/most-popular variant (undocumented API param, verified accepted). |
 | `--light-request` | true/false | Light request. |
 | `--screenshot` | true/false | Take screenshot. |
 | `--tag` | string | Optional label included in API response headers. |

.agents/skills/scrapingbee-cli/reference/google/overview.md

@@ -18,7 +18,8 @@ scrapingbee google --output-file serp.json "pizza new york" --country-code us
 | `--country-code` | string | ISO 3166-1 (e.g. us, gb, de). |
 | `--device` | string | `desktop` or `mobile`. |
 | `--page` | int | Page number (default 1). |
-| `--pages` | int | Consecutive pages to fetch starting at `--page` (default 1, max 10; 3 or fewer recommended). Combined into one response; cost is per fetched page. |
+| `--pages` | int | Consecutive pages to fetch starting at `--page` (default 1, max 10; 3 or fewer recommended). Combined into one response; cost is flat per request (10 light / 15 rendered) regardless of page count. |
+| `--nb-results` | int | Requested results per page (undocumented API param, verified accepted; Google may return more or fewer). |
 | `--language` | string | Language code (e.g. en, fr, de). |
 | `--date-range` | string | `past-hour`, `past-day`, `past-week`, `past-month`, `past-year`. Restrict results by recency. |
 | `--nfpr` | true/false | Disable autocorrection. |

.agents/skills/scrapingbee-cli/reference/scrape/options.md

@@ -71,7 +71,7 @@ Blocked? See [reference/proxy/strategies.md](reference/proxy/strategies.md).
 |-----------|------|-------------|
 | `--device` | desktop \| mobile | Device type (CLI validates). |
 | `--timeout` | int | Timeout ms (1000–140000). Scrape job timeout on ScrapingBee. The CLI sets the HTTP client (aiohttp) timeout to this value in seconds plus 30 s (for send/receive) so the client does not give up before the API responds. |
-| `--custom-google` / `--transparent-status-code` | — | Google (15 credits), target status. |
+| `--custom-google` / `--transparent-status-code` | — | Google (20 credits), target status. |
 | `--tag` | string | Optional label included in API response headers. |
 | `--mode` | auto | Auto-Mode: API picks the cheapest config that succeeds; charged only for the winning config. GET only. See [Auto-Mode](#auto-mode). |
 | `--max-cost` | int | Cap credits a request may cost (≥ 1). Requires `--mode auto`; omit = uncapped. |

.agents/skills/scrapingbee-cli/reference/youtube/subtitles.md

@@ -14,7 +14,7 @@ scrapingbee youtube-subtitles --output-file subtitles.json dQw4w9WgXcQ
 
 | Flag | Values | Notes |
 |------|--------|-------|
-| `--language` | ISO language code (`en`, `fr`, ...) | A language with no matching subtitles returns 404. |
+| `--language` | ISO language code (`en`, `fr`, ...) | A language with no matching subtitles returns HTTP 200 with an empty `subtitles` object (still 5 credits); the CLI prints a warning. |
 | `--subtitle-origin` | `auto-generated`, `uploader-provided` | Filter by subtitle source. |
 
 Plus global flags (`--output-file`, `--verbose`, `--output-dir`, `--concurrency`, `--retries`, `--backoff`).

.github/skills/scrapingbee-cli/reference/amazon/product.md

@@ -21,6 +21,7 @@ scrapingbee amazon-product --output-file product.json B0DPDRNSXV --domain com
 | `--language` | string | e.g. en_US, es_US, fr_FR. |
 | `--currency` | string | USD, EUR, GBP, etc. |
 | `--add-html` | true/false | Include full HTML. |
+| `--autoselect-variant` | true/false | Auto-select the default/most-popular variant (undocumented API param, verified accepted). |
 | `--light-request` | true/false | Light request. |
 | `--screenshot` | true/false | Take screenshot. |
 | `--tag` | string | Optional label included in API response headers. |

.github/skills/scrapingbee-cli/reference/google/overview.md

@@ -18,7 +18,8 @@ scrapingbee google --output-file serp.json "pizza new york" --country-code us
 | `--country-code` | string | ISO 3166-1 (e.g. us, gb, de). |
 | `--device` | string | `desktop` or `mobile`. |
 | `--page` | int | Page number (default 1). |
-| `--pages` | int | Consecutive pages to fetch starting at `--page` (default 1, max 10; 3 or fewer recommended). Combined into one response; cost is per fetched page. |
+| `--pages` | int | Consecutive pages to fetch starting at `--page` (default 1, max 10; 3 or fewer recommended). Combined into one response; cost is flat per request (10 light / 15 rendered) regardless of page count. |
+| `--nb-results` | int | Requested results per page (undocumented API param, verified accepted; Google may return more or fewer). |
 | `--language` | string | Language code (e.g. en, fr, de). |
 | `--date-range` | string | `past-hour`, `past-day`, `past-week`, `past-month`, `past-year`. Restrict results by recency. |
 | `--nfpr` | true/false | Disable autocorrection. |

.github/skills/scrapingbee-cli/reference/scrape/options.md

@@ -71,7 +71,7 @@ Blocked? See [reference/proxy/strategies.md](reference/proxy/strategies.md).
 |-----------|------|-------------|
 | `--device` | desktop \| mobile | Device type (CLI validates). |
 | `--timeout` | int | Timeout ms (1000–140000). Scrape job timeout on ScrapingBee. The CLI sets the HTTP client (aiohttp) timeout to this value in seconds plus 30 s (for send/receive) so the client does not give up before the API responds. |
-| `--custom-google` / `--transparent-status-code` | — | Google (15 credits), target status. |
+| `--custom-google` / `--transparent-status-code` | — | Google (20 credits), target status. |
 | `--tag` | string | Optional label included in API response headers. |
 | `--mode` | auto | Auto-Mode: API picks the cheapest config that succeeds; charged only for the winning config. GET only. See [Auto-Mode](#auto-mode). |
 | `--max-cost` | int | Cap credits a request may cost (≥ 1). Requires `--mode auto`; omit = uncapped. |

.github/skills/scrapingbee-cli/reference/youtube/subtitles.md

@@ -14,7 +14,7 @@ scrapingbee youtube-subtitles --output-file subtitles.json dQw4w9WgXcQ
 
 | Flag | Values | Notes |
 |------|--------|-------|
-| `--language` | ISO language code (`en`, `fr`, ...) | A language with no matching subtitles returns 404. |
+| `--language` | ISO language code (`en`, `fr`, ...) | A language with no matching subtitles returns HTTP 200 with an empty `subtitles` object (still 5 credits); the CLI prints a warning. |
 | `--subtitle-origin` | `auto-generated`, `uploader-provided` | Filter by subtitle source. |
 
 Plus global flags (`--output-file`, `--verbose`, `--output-dir`, `--concurrency`, `--retries`, `--backoff`).

.kiro/skills/scrapingbee-cli/reference/amazon/product.md

@@ -21,6 +21,7 @@ scrapingbee amazon-product --output-file product.json B0DPDRNSXV --domain com
 | `--language` | string | e.g. en_US, es_US, fr_FR. |
 | `--currency` | string | USD, EUR, GBP, etc. |
 | `--add-html` | true/false | Include full HTML. |
+| `--autoselect-variant` | true/false | Auto-select the default/most-popular variant (undocumented API param, verified accepted). |
 | `--light-request` | true/false | Light request. |
 | `--screenshot` | true/false | Take screenshot. |
 | `--tag` | string | Optional label included in API response headers. |

.kiro/skills/scrapingbee-cli/reference/google/overview.md

@@ -18,7 +18,8 @@ scrapingbee google --output-file serp.json "pizza new york" --country-code us
 | `--country-code` | string | ISO 3166-1 (e.g. us, gb, de). |
 | `--device` | string | `desktop` or `mobile`. |
 | `--page` | int | Page number (default 1). |
-| `--pages` | int | Consecutive pages to fetch starting at `--page` (default 1, max 10; 3 or fewer recommended). Combined into one response; cost is per fetched page. |
+| `--pages` | int | Consecutive pages to fetch starting at `--page` (default 1, max 10; 3 or fewer recommended). Combined into one response; cost is flat per request (10 light / 15 rendered) regardless of page count. |
+| `--nb-results` | int | Requested results per page (undocumented API param, verified accepted; Google may return more or fewer). |
 | `--language` | string | Language code (e.g. en, fr, de). |
 | `--date-range` | string | `past-hour`, `past-day`, `past-week`, `past-month`, `past-year`. Restrict results by recency. |
 | `--nfpr` | true/false | Disable autocorrection. |

.kiro/skills/scrapingbee-cli/reference/scrape/options.md

@@ -71,7 +71,7 @@ Blocked? See [reference/proxy/strategies.md](reference/proxy/strategies.md).
 |-----------|------|-------------|
 | `--device` | desktop \| mobile | Device type (CLI validates). |
 | `--timeout` | int | Timeout ms (1000–140000). Scrape job timeout on ScrapingBee. The CLI sets the HTTP client (aiohttp) timeout to this value in seconds plus 30 s (for send/receive) so the client does not give up before the API responds. |
-| `--custom-google` / `--transparent-status-code` | — | Google (15 credits), target status. |
+| `--custom-google` / `--transparent-status-code` | — | Google (20 credits), target status. |
 | `--tag` | string | Optional label included in API response headers. |
 | `--mode` | auto | Auto-Mode: API picks the cheapest config that succeeds; charged only for the winning config. GET only. See [Auto-Mode](#auto-mode). |
 | `--max-cost` | int | Cap credits a request may cost (≥ 1). Requires `--mode auto`; omit = uncapped. |

.kiro/skills/scrapingbee-cli/reference/youtube/subtitles.md

@@ -14,7 +14,7 @@ scrapingbee youtube-subtitles --output-file subtitles.json dQw4w9WgXcQ
 
 | Flag | Values | Notes |
 |------|--------|-------|
-| `--language` | ISO language code (`en`, `fr`, ...) | A language with no matching subtitles returns 404. |
+| `--language` | ISO language code (`en`, `fr`, ...) | A language with no matching subtitles returns HTTP 200 with an empty `subtitles` object (still 5 credits); the CLI prints a warning. |
 | `--subtitle-origin` | `auto-generated`, `uploader-provided` | Filter by subtitle source. |
 
 Plus global flags (`--output-file`, `--verbose`, `--output-dir`, `--concurrency`, `--retries`, `--backoff`).

.opencode/skills/scrapingbee-cli/reference/amazon/product.md

@@ -21,6 +21,7 @@ scrapingbee amazon-product --output-file product.json B0DPDRNSXV --domain com
 | `--language` | string | e.g. en_US, es_US, fr_FR. |
 | `--currency` | string | USD, EUR, GBP, etc. |
 | `--add-html` | true/false | Include full HTML. |
+| `--autoselect-variant` | true/false | Auto-select the default/most-popular variant (undocumented API param, verified accepted). |
 | `--light-request` | true/false | Light request. |
 | `--screenshot` | true/false | Take screenshot. |
 | `--tag` | string | Optional label included in API response headers. |

.opencode/skills/scrapingbee-cli/reference/google/overview.md

@@ -18,7 +18,8 @@ scrapingbee google --output-file serp.json "pizza new york" --country-code us
 | `--country-code` | string | ISO 3166-1 (e.g. us, gb, de). |
 | `--device` | string | `desktop` or `mobile`. |
 | `--page` | int | Page number (default 1). |
-| `--pages` | int | Consecutive pages to fetch starting at `--page` (default 1, max 10; 3 or fewer recommended). Combined into one response; cost is per fetched page. |
+| `--pages` | int | Consecutive pages to fetch starting at `--page` (default 1, max 10; 3 or fewer recommended). Combined into one response; cost is flat per request (10 light / 15 rendered) regardless of page count. |
+| `--nb-results` | int | Requested results per page (undocumented API param, verified accepted; Google may return more or fewer). |
 | `--language` | string | Language code (e.g. en, fr, de). |
 | `--date-range` | string | `past-hour`, `past-day`, `past-week`, `past-month`, `past-year`. Restrict results by recency. |
 | `--nfpr` | true/false | Disable autocorrection. |

.opencode/skills/scrapingbee-cli/reference/scrape/options.md

@@ -71,7 +71,7 @@ Blocked? See [reference/proxy/strategies.md](reference/proxy/strategies.md).
 |-----------|------|-------------|
 | `--device` | desktop \| mobile | Device type (CLI validates). |
 | `--timeout` | int | Timeout ms (1000–140000). Scrape job timeout on ScrapingBee. The CLI sets the HTTP client (aiohttp) timeout to this value in seconds plus 30 s (for send/receive) so the client does not give up before the API responds. |
-| `--custom-google` / `--transparent-status-code` | — | Google (15 credits), target status. |
+| `--custom-google` / `--transparent-status-code` | — | Google (20 credits), target status. |
 | `--tag` | string | Optional label included in API response headers. |
 | `--mode` | auto | Auto-Mode: API picks the cheapest config that succeeds; charged only for the winning config. GET only. See [Auto-Mode](#auto-mode). |
 | `--max-cost` | int | Cap credits a request may cost (≥ 1). Requires `--mode auto`; omit = uncapped. |

.opencode/skills/scrapingbee-cli/reference/youtube/subtitles.md

@@ -14,7 +14,7 @@ scrapingbee youtube-subtitles --output-file subtitles.json dQw4w9WgXcQ
 
 | Flag | Values | Notes |
 |------|--------|-------|
-| `--language` | ISO language code (`en`, `fr`, ...) | A language with no matching subtitles returns 404. |
+| `--language` | ISO language code (`en`, `fr`, ...) | A language with no matching subtitles returns HTTP 200 with an empty `subtitles` object (still 5 credits); the CLI prints a warning. |
 | `--subtitle-origin` | `auto-generated`, `uploader-provided` | Filter by subtitle source. |
 
 Plus global flags (`--output-file`, `--verbose`, `--output-dir`, `--concurrency`, `--retries`, `--backoff`).

CHANGELOG.md

@@ -5,16 +5,27 @@ All notable changes to this project are documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [1.6.0] - TBD
+## [1.6.0] - 2026-08-24
 
 ### Added
 
 - **Auto-Mode on `scrape` (`--mode auto`)** — the API picks the cheapest scraping config that succeeds (tries cheap → expensive, stops at the first success) and charges only for the winning config (0 credits if all fail). GET only. Forwarded to the API as `mode=auto` when set, omitted otherwise. Cannot be combined with `--render-js`, `--premium-proxy`, `--stealth-proxy`, or `--transparent-status-code` (Auto-Mode selects these itself) — the CLI rejects such combinations before making a request.
 - **`--max-cost` on `scrape`** — cap the credits a request may cost (integer ≥ 1). Requires `--mode auto`; omit for an uncapped budget. Forwarded to the API as `max_cost` when set, omitted otherwise.
 - The verbose output (`-v`) now surfaces the `Spb-auto-cost` response header as `Auto Credit Cost` (the credits actually charged for the winning Auto-Mode config), alongside the existing `Credit Cost`.
 - **`youtube-subtitles` command** — fetch video captions/transcripts from the YouTube Subtitles API (5 credits per request). Accepts a video ID or full YouTube URL, `--language` (ISO code) and `--subtitle-origin` (`auto-generated` / `uploader-provided`), and supports batch via `--input-file` like the other YouTube commands.
-- **`--pages` on `google`** — fetch up to 10 consecutive result pages starting at `--page` in a single combined response (3 or fewer recommended; cost is per fetched page).
+- **`--pages` on `google`** — fetch up to 10 consecutive result pages starting at `--page` in a single combined response (3 or fewer recommended). Cost is flat per request — 10 credits light / 15 rendered — regardless of page count.
 - **`--search-type ads` on `google`** — classic-result structure optimized for paid-ad visibility.
+- **`--nb-results` on `google`** — requested number of results per page. Undocumented API parameter, verified accepted by the API (Google may return more or fewer results than requested).
+- **`--autoselect-variant` on `amazon-product`** — auto-select the default/most-popular product variant, matching the existing `amazon-search` flag. Undocumented API parameter, verified accepted by the API.
+
+### Changed
+
+- **Header-based authorization** — all API requests now authenticate via the `Authorization: Bearer` header instead of the deprecated `api_key` query parameter, so the key no longer appears in request URLs (or anything that logs them). `crawl` is the one exception: its Scrapy middleware (`scrapy-scrapingbee`) still builds `api_key` URLs and will migrate separately.
+
+### Fixed
+
+- **`-H` headers were silently dropped on POST/PUT** — custom headers are now `Spb-`-prefixed on every method (idempotently), which is the only form the API forwards to the target. Previously the prefix was only added on GET, so POST/PUT headers never reached the target — and a user `Authorization` header could clobber the CLI's own API authentication. Already-prefixed headers are passed through unchanged, so `-H "Spb-X: 1"` no longer double-prefixes.
+- **Empty subtitles warned about** — `youtube-subtitles` with a `--language`/`--subtitle-origin` that matches nothing returns HTTP 200 with an empty `subtitles` object (not 404) and still charges 5 credits; the CLI now prints a warning instead of silent empty JSON.
 
 ## [1.5.1] - 2026-07-20
 

plugins/scrapingbee-cli/skills/scrapingbee-cli/reference/amazon/product.md

@@ -21,6 +21,7 @@ scrapingbee amazon-product --output-file product.json B0DPDRNSXV --domain com
 | `--language` | string | e.g. en_US, es_US, fr_FR. |
 | `--currency` | string | USD, EUR, GBP, etc. |
 | `--add-html` | true/false | Include full HTML. |
+| `--autoselect-variant` | true/false | Auto-select the default/most-popular variant (undocumented API param, verified accepted). |
 | `--light-request` | true/false | Light request. |
 | `--screenshot` | true/false | Take screenshot. |
 | `--tag` | string | Optional label included in API response headers. |

plugins/scrapingbee-cli/skills/scrapingbee-cli/reference/google/overview.md

@@ -18,7 +18,8 @@ scrapingbee google --output-file serp.json "pizza new york" --country-code us
 | `--country-code` | string | ISO 3166-1 (e.g. us, gb, de). |
 | `--device` | string | `desktop` or `mobile`. |
 | `--page` | int | Page number (default 1). |
-| `--pages` | int | Consecutive pages to fetch starting at `--page` (default 1, max 10; 3 or fewer recommended). Combined into one response; cost is per fetched page. |
+| `--pages` | int | Consecutive pages to fetch starting at `--page` (default 1, max 10; 3 or fewer recommended). Combined into one response; cost is flat per request (10 light / 15 rendered) regardless of page count. |
+| `--nb-results` | int | Requested results per page (undocumented API param, verified accepted; Google may return more or fewer). |
 | `--language` | string | Language code (e.g. en, fr, de). |
 | `--date-range` | string | `past-hour`, `past-day`, `past-week`, `past-month`, `past-year`. Restrict results by recency. |
 | `--nfpr` | true/false | Disable autocorrection. |

plugins/scrapingbee-cli/skills/scrapingbee-cli/reference/scrape/options.md

@@ -71,7 +71,7 @@ Blocked? See [reference/proxy/strategies.md](reference/proxy/strategies.md).
 |-----------|------|-------------|
 | `--device` | desktop \| mobile | Device type (CLI validates). |
 | `--timeout` | int | Timeout ms (1000–140000). Scrape job timeout on ScrapingBee. The CLI sets the HTTP client (aiohttp) timeout to this value in seconds plus 30 s (for send/receive) so the client does not give up before the API responds. |
-| `--custom-google` / `--transparent-status-code` | — | Google (15 credits), target status. |
+| `--custom-google` / `--transparent-status-code` | — | Google (20 credits), target status. |
 | `--tag` | string | Optional label included in API response headers. |
 | `--mode` | auto | Auto-Mode: API picks the cheapest config that succeeds; charged only for the winning config. GET only. See [Auto-Mode](#auto-mode). |
 | `--max-cost` | int | Cap credits a request may cost (≥ 1). Requires `--mode auto`; omit = uncapped. |

plugins/scrapingbee-cli/skills/scrapingbee-cli/reference/youtube/subtitles.md

@@ -14,7 +14,7 @@ scrapingbee youtube-subtitles --output-file subtitles.json dQw4w9WgXcQ
 
 | Flag | Values | Notes |
 |------|--------|-------|
-| `--language` | ISO language code (`en`, `fr`, ...) | A language with no matching subtitles returns 404. |
+| `--language` | ISO language code (`en`, `fr`, ...) | A language with no matching subtitles returns HTTP 200 with an empty `subtitles` object (still 5 credits); the CLI prints a warning. |
 | `--subtitle-origin` | `auto-generated`, `uploader-provided` | Filter by subtitle source. |
 
 Plus global flags (`--output-file`, `--verbose`, `--output-dir`, `--concurrency`, `--retries`, `--backoff`).