docs(ruby-gem): collected updates for gem v0.23.0

src/content/docs/creating-custom-feeds.mdx

@@ -80,9 +80,9 @@ This is where you tell the html2rss engine exactly what to find on the page. You
       selector: "article.post"
     title:
       selector: "h2 a"
-    link:
+    url:
       selector: "h2 a"
-      attribute: href
+      extractor: "href"
   `}
   lang="yaml"
 />
@@ -100,20 +100,19 @@ This says: "Find each article, get the title from the h2 anchor, and get the lin
 **Step 2:** Create a file called `example.com.yml` with this basic structure:
 
 <Code
-  code={
-    "channel:\n" +
-    "  url: https://example.com/blog\n" +
-    "  title: My Blog\n" +
-    "\n" +
-    "selectors:\n" +
-    "  items:\n" +
-    '    selector: "article.post"\n' +
-    "  title:\n" +
-    '    selector: "h2 a"\n' +
-    "  link:\n" +
-    '    selector: "h2 a"\n' +
-    "    attribute: href"
-  }
+  code={`
+  channel:
+    url: https://example.com/blog
+    title: My Blog
+  selectors:
+    items:
+      selector: "article.post"
+    title:
+      selector: "h2 a"
+    url:
+      selector: "h2 a"
+      extractor: "href"
+  `}
   lang="yaml"
 />
 
@@ -148,21 +147,27 @@ html2rss supports many configuration options:
 
 <Code code={`html2rss feed your-config.yml`} lang="bash" />
 
-3. **Test with `html2rss-web`:** Add your config to the `feeds.yml` file and restart your instance
+3. **Or test against a locally saved HTML file without network requests:**
+
+<Code code={`html2rss feed your-config.yml --input sample.html`} lang="bash" />
+
+4. **Test with `html2rss-web`:** Add your config to the `feeds.yml` file and restart your instance
 
-4. **Check the output:** Make sure all items have titles, links, and descriptions
+5. **Check the output:** Make sure all items have titles, links, and descriptions
 
 ### Useful CLI flags when a site is difficult
 
 Some sites need a little more request budget than the defaults.
 
 - Use `--max-redirects` when the site bounces through several canonicalization or tracking redirects before the real page loads.
 - Use `--max-requests` when your config needs more than one request, for example pagination or other follow-up fetches.
+- Use `--input` to supply a local HTML file to inspect extraction offline.
 
 <Code
   code={`
   html2rss feed your-config.yml --max-redirects 10 && \
   html2rss feed your-config.yml --max-requests 5 && \
+  html2rss feed your-config.yml --input /path/to/page.html && \
   html2rss auto https://example.com/blog --max-redirects 10 --max-requests 5
 `}
   lang="bash"

src/content/docs/index.mdx

@@ -19,7 +19,7 @@ The canonical path to your first feed:
 
 ### Features & Operations
 
-- **[Use included configs](/web-application/guides/use-included-configs/)**: Use the Feed Directory for popular sites.
+- **[Use the Feed Directory](/web-application/guides/use-the-feed-directory/)**: Use the Feed Directory for popular sites.
 - **[Monitoring](/web-application/reference/monitoring/)**: Keep your instance healthy with probes and Sentry.
 - **[Reference](/web-application/reference/)**: Environment variables and versioning strategy.
 

src/content/docs/ruby-gem/guides/advanced-content-extraction.mdx

@@ -7,8 +7,8 @@ While basic selectors are straightforward, you can achieve very precise content
 
 ## Extractors
 
-Learn how to extract specific attributes (like `src` for images) or static values. See [Extractors](/ruby-gem/reference/selectors/#extractors/).
+Learn how to extract specific attributes (like `src` for images) or static values. See [Extractors](/ruby-gem/reference/selectors/#extractors).
 
 ## Post Processors
 
-Manipulate extracted text, sanitize HTML, convert Markdown, or apply custom logic. See [Post Processors](/ruby-gem/reference/selectors/#post-processors/).
+Manipulate extracted text, sanitize HTML, convert Markdown, or apply custom logic. See [Post Processors](/ruby-gem/reference/selectors/#post-processors).

src/content/docs/ruby-gem/guides/backward-compatibility.mdx

@@ -1,54 +1,69 @@
 ---
-title: "Backward Compatibility"
-description: "html2rss maintains backward compatibility with older configuration formats and attribute names."
+title: "Backward Compatibility & Migrations"
+description: "Migration guide and backward-compatibility reference for html2rss configuration formats."
 ---
 
 import { Code } from "@astrojs/starlight/components";
 
-html2rss maintains backward compatibility with older configuration formats and attribute names.
+This page outlines recent breaking changes, purged legacy aliases, and migration steps for older feed configurations.
 
-## Renamed Attributes
+## Removed Legacy Selector Aliases
 
-Some attribute names have been renamed for clarity, but the old names still work:
+In previous versions, `html2rss` accepted legacy selector names with a deprecation warning. These shims have been removed:
 
-| Current Name   | Legacy Names         | Description                  |
-| -------------- | -------------------- | ---------------------------- |
-| `published_at` | `updated`, `pubDate` | Publication date of the item |
+| Removed Name | Required Name  | Description           |
+| :----------- | :------------- | :-------------------- |
+| `pubDate`    | `published_at` | Item publication date |
+| `updated`    | `published_at` | Item publication date |
 
-### Example
+### Migration Example
 
-Both of these configurations work identically:
+Update any occurrences of `pubDate` or `updated` in your `selectors` block to `published_at`:
 
 <Code
   code={`
-  # Current format (recommended)
+  # Deprecated / Removed
   selectors:
-    published_at:
+    updated:
       selector: ".date"
 
-# Legacy format (still supported)
+# Current & Required
 
 selectors:
-updated:
+published_at:
 selector: ".date"
-`} lang="yaml"
+`}
+lang="yaml"
 />
 
-## Migration Guide
+## Removed Channel Attributes
 
-If you're upgrading from an older version of html2rss:
+`strategy` and `headers` were previously tolerated inside the `channel` block. These legacy shims have been removed. Both keys must now be placed at the top level of your feed configuration:
 
-1. **Update attribute names**: Replace `updated` with `published_at` in your configurations
-2. **Test your feeds**: Verify that all feeds still work correctly after the update
-
-## Deprecated Features
-
-The following features are deprecated but still supported:
-
-- **Legacy attribute names**: While still supported, use the current names for new configurations
+<Code
+  code={`
+  # Deprecated / Removed
+  channel:
+    url: "https://example.com/articles"
+    strategy: browserless
+    headers:
+      User-Agent: "CustomAgent/1.0"
+
+# Current & Required
+
+strategy: browserless
+headers:
+User-Agent: "CustomAgent/1.0"
+channel:
+url: "https://example.com/articles"
+`}
+lang="yaml"
+/>
 
-## Getting Help
+## Migration Checklist
 
-If you encounter issues with backward compatibility:
+When upgrading to modern `html2rss` releases:
 
-- **Report issues**: Open an issue if you find compatibility problems
+1. **Rename date selectors**: Ensure date selectors use `published_at` rather than `updated` or `pubDate`.
+2. **Move channel-level transport keys**: Ensure `strategy` and `headers` are defined at the top level of the YAML file.
+3. **Validate configurations**: Run `html2rss validate config.yml` to ensure your YAML conforms to the current schema.

src/content/docs/ruby-gem/index.mdx

@@ -10,8 +10,11 @@ This section provides comprehensive documentation for the `html2rss` Ruby gem.
 If you are looking for the stable machine-readable contract for config authoring, use the JSON Schema exported by the core repo:
 
 - Repository file: [**`html2rss-config.schema.json`**](https://github.com/html2rss/html2rss/blob/master/schema/html2rss-config.schema.json)
-- CLI export: `html2rss schema`
-- Runtime validation: `html2rss validate config.yml`
+- CLI export: `html2rss schema` (also `--write`, `--no-pretty`)
+- Runtime validation: `html2rss validate config.yml` or `Html2rss::Config.validate`
+- Ruby discovery: `Html2rss::Config.json_schema` / `Html2rss::Config.schema_path`
+
+The exported schema covers client-side validation. Runtime validation remains authoritative for dynamic cross-field checks such as selector-key references. Contributors regenerating the checked-in artifact should run `bundle exec rake config:schema` in the core repo.
 
 ## Getting Started
 

src/content/docs/ruby-gem/reference/auto-source.mdx

@@ -23,12 +23,18 @@ To enable it, add `auto_source: {}` to your configuration:
 `auto_source` uses the following strategies to find content:
 
 1.  **`wordpress_api`:** Detects the `<link rel="https://api.w.org/">` tag used by WordPress and pulls posts from the REST API without parsing article HTML. See [WordPress API](/ruby-gem/reference/wordpress-api/).
-2.  **`schema`:** Parses `<script type="json/ld">` tags containing structured data (e.g., [Schema.org](https://schema.org/)).
-3.  **`semantic_html`:** Searches for semantic HTML5 tags like `<article>`, `<main>`, and `<section>`.
-4.  **`html`:** Analyzes the HTML structure to find frequently occurring selectors that are likely to contain the main content.
-5.  **json_state:** Single-page applications often stash pre-rendered article data in `<script type="application/json">` tags or global variables
+2.  **`sitemap`:** Automatically locates XML sitemap documents (`<link rel="sitemap">`, `/sitemap.xml`, or `/robots.txt`), filtering entries by priority and recency, with support for Google News tags (`<news:news>`).
+3.  **`meta_oembed`:** Extracts single-item articles, video pages, and media updates from OpenGraph/Twitter meta tags and resolves JSON oEmbed endpoints (`<link rel="alternate" type="application/json+oembed">`).
+4.  **`schema`:** Parses `<script type="json/ld">` tags containing structured data (e.g., [Schema.org](https://schema.org/)), extracting authors and publishers.
+5.  **`microdata`:** Extracts HTML Microdata annotations (`itemscope itemtype`).
+6.  **`microformats2`:** Parses Microformats2 `h-entry` markup, extracting `p-name`, `e-content`, `u-url`, `dt-published`, `p-author`, `p-category`, and `u-photo` / `u-featured` media.
+7.  **`semantic_html`:** Searches for semantic HTML5 tags like `<article>`, `<main>`, and `<section>`, with fallback clustering for anchorless lists.
+8.  **`html`:** Analyzes HTML structure to find frequently occurring item selectors, with tag-structure and class-clustering fallbacks.
+9.  **`json_state`:** Single-page applications often stash pre-rendered article data in `<script type="application/json">` tags or global variables
     such as `window.__NEXT_DATA__`, `window.__NUXT__`, or `window.STATE`. The JSON-state scraper walks those blobs, finds arrays with
-    `title`/`url` pairs, and converts them into the same hashes produced by `HtmlExtractor`.
+    `title`/`url` pairs, and converts them into feed items.
+
+`auto_source` also automatically extracts and prepends kicker/teaser labels to article titles when present.
 
 **`json_state` Limitations:** the scraper requires discoverable arrays of hashes containing clear `title` and `url` fields. Minified or
 obfuscated state objects, heavily encoded values, or blobs that require executing embedded functions are ignored.
@@ -51,16 +57,28 @@ Enable or disable specific scrapers and adjust their settings:
     scraper:
       wordpress_api:
         enabled: false # default: true
+      sitemap:
+        enabled: true # default: true
+        min_priority: 0.3 # default: 0.3
+        max_age_days: 30 # default: 30
+      meta_oembed:
+        enabled: true # default: true
       schema:
         enabled: false # default: true
+      microdata:
+        enabled: true # default: true
+      microformats2:
+        enabled: true # default: true
       semantic_html:
-        enabled: false # default: true
+        enabled: true # default: true
+        fallback_anchorless: true # default: true
       json_state:
         enabled: false # default: true
       html:
         enabled: true
         minimum_selector_frequency: 3 # default: 2
         use_top_selectors: 3 # default: 5
+        fallback_anchorless: true # default: true
   `}
   lang="yaml"
 />

src/content/docs/ruby-gem/reference/cli-reference.mdx

@@ -17,21 +17,31 @@ The `html2rss` executable is the primary way to interact with the gem from your
 
 ### Auto
 
-Automatically discovers items from a page and prints the generated RSS feed to stdout.
+Automatically discovers items from a page and prints the generated RSS or JSONFeed to stdout.
 
 <Code
   code={`
   html2rss auto https://example.com/articles ; \
+  html2rss auto https://example.com/articles --format jsonfeed ; \
   html2rss auto https://example.com/app --strategy browserless --max-redirects 5 --max-requests 6 ; \
   BOTASAURUS_SCRAPER_URL="http://localhost:4010" html2rss auto https://example.com/protected --strategy botasaurus ; \
-  html2rss auto https://example.com/articles --items_selector ".post-card"
+  html2rss auto https://example.com/articles --items_selector ".post-card" ; \
+  html2rss auto --input sample.html ; \
+  html2rss auto https://example.com/articles --input sample.html
 `}
   lang="bash"
 />
 
-Command: `html2rss auto URL`
+Command: `html2rss auto [URL]`
 
-Default behavior is `--strategy auto`, which tries `faraday` then `botasaurus` then `browserless`.
+Available options:
+
+- `--strategy`: Optional request strategy (`auto`, `faraday`, `browserless`, `botasaurus`, `local_file`). Defaults to `auto`, which tries `faraday` -> `botasaurus` -> `browserless`.
+- `--format`: Output format for the auto-sourced feed (`rss` or `jsonfeed`). Defaults to `rss`.
+- `--items_selector`: Optional CSS selector hint for item extraction.
+- `--max-redirects`: Maximum redirects to follow per request.
+- `--max-requests`: Maximum requests to allow for this feed build (defaults to `4` for auto discovery).
+- `--input`: Local HTML file path to read input from without making network requests. When provided without a `URL`, `html2rss` attempts to detect the base URL from `<link rel="canonical">` or OpenGraph metadata in the HTML.
 
 #### URL Surface Guidance For `auto`
 
@@ -78,17 +88,17 @@ Start by changing the input URL to a direct listing/update page, then move to ex
 <Code
   code={`
   # 1) Start Browserless in the background
-  docker run -d --rm --name html2rss-browserless \
-    -p 3000:3000 \
-    -e "CONCURRENT=10" \
-    -e "TOKEN=6R0W53R135510" \
+  docker run -d --rm --name html2rss-browserless \\
+    -p 3000:3000 \\
+    -e "CONCURRENT=10" \\
+    -e "TOKEN=6R0W53R135510" \\
     ghcr.io/browserless/chromium
 
 # 2) Run html2rss against Browserless
 
-BROWSERLESS_IO_WEBSOCKET_URL="ws://127.0.0.1:3000" \
- BROWSERLESS_IO_API_TOKEN="6R0W53R135510" \
- html2rss auto https://example.com/updates --strategy browserless
+BROWSERLESS_IO_WEBSOCKET_URL="ws://127.0.0.1:3000" \\
+BROWSERLESS_IO_API_TOKEN="6R0W53R135510" \\
+html2rss auto https://example.com/updates --strategy browserless
 
 # 3) Stop Browserless when done
 
@@ -111,7 +121,7 @@ For custom Browserless endpoints, `BROWSERLESS_IO_API_TOKEN` is required.
 
 <Code
   code={`
-  BOTASAURUS_SCRAPER_URL="http://localhost:4010" \
+  BOTASAURUS_SCRAPER_URL="http://localhost:4010" \\
   html2rss auto https://example.com/updates --strategy botasaurus
 `}
   lang="bash"
@@ -135,18 +145,27 @@ Loads a YAML config, builds the feed, and prints the RSS XML to stdout.
   html2rss feed single.yml --strategy browserless ; \
   BOTASAURUS_SCRAPER_URL="http://localhost:4010" html2rss feed single.yml --strategy botasaurus ; \
   html2rss feed single.yml --max-redirects 5 --max-requests 6 ; \
-  html2rss feed single.yml --params id:42 foo:bar
+  html2rss feed single.yml --params id:42 foo:bar ; \
+  html2rss feed single.yml --input sample.html
 `}
   lang="bash"
 />
 
 Command: `html2rss feed YAML_FILE [feed_name]`
 
+Available options:
+
+- `--strategy`: Request strategy override (`auto`, `faraday`, `browserless`, `botasaurus`, `local_file`).
+- `--params`: Dynamic parameters passed as key-value pairs (e.g. `--params id:42 section:news`).
+- `--max-redirects`: Maximum redirects to follow per request.
+- `--max-requests`: Total request budget allowed for this feed build.
+- `--input`: Local HTML file path to read input from instead of making a network request.
+
 The CLI keeps `strategy` as a top-level override and writes runtime request limits into the generated config under `request`.
 
 ### Schema
 
-Prints the exported JSON Schema for the current gem version.
+Prints or writes the exported JSON Schema for the current gem version.
 
 <Code
   code={`
@@ -159,20 +178,30 @@ Prints the exported JSON Schema for the current gem version.
 
 Command: `html2rss schema`
 
+Available options:
+
+- `--pretty` / `--no-pretty`: Pretty-print the schema JSON (default: `true`).
+- `--write PATH`: Write the schema JSON directly to the given file path, creating any missing parent directories.
+
 ### Validate
 
 Validates a config with the runtime validator without generating a feed.
 
 <Code
   code={`
   html2rss validate single.yml ; \
-  html2rss validate feeds.yml my-first-feed
+  html2rss validate feeds.yml my-first-feed ; \
+  html2rss validate single.yml --params id:42
 `}
   lang="bash"
 />
 
 Command: `html2rss validate YAML_FILE [feed_name]`
 
+Available options:
+
+- `--params`: Dynamic parameters to interpolate before validation (default: `{}`).
+
 ### Help
 
 Displays the help message with available commands and options.