Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
239 changes: 239 additions & 0 deletions .github/description-allowlist.txt

Large diffs are not rendered by default.

60 changes: 60 additions & 0 deletions .github/workflows/lint-metadata.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
name: Lint page metadata

# Runs on every PR (no paths filter) so it can be a required status check:
# a required check that is skipped by a paths filter leaves PRs stuck pending.
# It passes fast when no pages are affected.
on: pull_request

jobs:
descriptions:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0

- name: "Require :description: on every page"
run: |
# Style guide: every page needs a :description: (search results, link
# previews, and AI discoverability all read it). Pages listed in
# .github/description-allowlist.txt are known gaps blocked on upstream
# rp-connect-docs descriptions (docs-data/overrides.json there is the
# fix path). Everything else must carry one. Shrink the list; never
# grow it.
missing=0
while IFS= read -r f; do
if ! grep -qE '^:description: .+' "$f"; then
if grep -qxF "$f" .github/description-allowlist.txt; then continue; fi
echo "::error file=$f::Missing :description: header attribute (and not in .github/description-allowlist.txt)"
missing=1
fi
done < <(find modules/*/pages -name '*.adoc' -type f)
exit $missing

- name: "Allowlist ratchet hygiene (non-blocking)"
run: |
# Allowlisted files that gained a description, or no longer exist,
# should be removed from the list. Warn only.
grep -v '^#' .github/description-allowlist.txt | while IFS= read -r f; do
[ -z "$f" ] && continue
if [ ! -f "$f" ]; then
echo "::warning::Allowlist entry no longer exists: $f - remove it from .github/description-allowlist.txt"
continue
fi
if grep -qE '^:description: .+' "$f"; then
echo "::warning file=$f::Has a description now - remove it from .github/description-allowlist.txt"
fi
done

- name: "Warn on over-length descriptions (changed files only)"
run: |
# Style guide recommends under 155 chars (search snippet cutoff).
# Warn-only, and only for files this PR touches.
git diff --name-only "${{ github.event.pull_request.base.sha }}".."${{ github.event.pull_request.head.sha }}" \
| grep -E '^modules/.*/pages/.*\.adoc$' | while IFS= read -r f; do
[ -f "$f" ] || continue
desc=$(grep -m1 '^:description: ' "$f" | sed 's/^:description: //')
if [ -n "$desc" ] && [ "${#desc}" -gt 155 ]; then
echo "::warning file=$f::Description is ${#desc} chars; style guide recommends under 155"
fi
done
1 change: 1 addition & 0 deletions modules/develop/pages/connect/components/about.adoc
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
= Components Catalog
:page-aliases: components:about.adoc, develop:connect/components/catalog.adoc, components:catalog.adoc
:description: A searchable catalog of available Redpanda Connect components.

include::connect:components:about.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= redpanda
:page-aliases: components:caches/redpanda.adoc
:description: A Kafka cache using the https://github.com/twmb/franz-go[Franz Kafka client library^].
include::connect:components:caches/redpanda.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= aws_cloudwatch_logs
:page-aliases: components:inputs/aws_cloudwatch_logs.adoc
:description: Consumes log events from AWS CloudWatch Logs.
include::connect:components:inputs/aws_cloudwatch_logs.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= aws_dynamodb_cdc
:page-aliases: components:inputs/aws_dynamodb_cdc.adoc
:description: Reads change data capture (CDC) events from DynamoDB Streams.
include::connect:components:inputs/aws_dynamodb_cdc.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
= gateway
:page-aliases: components:inputs/gateway.adoc
:description: Receive messages delivered over HTTP.

include::connect:components:partial$components/inputs/gateway.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= gcp_spanner_cdc
:page-aliases: components:inputs/gcp_spanner_cdc.adoc
:description: Creates an input that consumes from a spanner change stream.
include::connect:components:inputs/gcp_spanner_cdc.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
= jira
:page-aliases: components:inputs/jira.adoc
:description: Streams Jira issues, comments, or changelog entries via JQL with incremental polling.

include::connect:components:inputs/jira.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= microsoft_sql_server_cdc
:page-aliases: components:inputs/microsoft_sql_server_cdc.adoc
:description: Enables Change Data Capture by consuming from Microsoft SQL Server's change tables.
include::connect:components:inputs/microsoft_sql_server_cdc.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= oracledb_cdc
:page-aliases: components:inputs/oracledb_cdc.adoc
:description: Enables Change Data Capture by consuming from OracleDB.
include::connect:components:page$inputs/oracledb_cdc.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= otlp_grpc
:description: Receive OpenTelemetry traces, logs, and metrics via OTLP/gRPC protocol.

include::connect:components:inputs/otlp_grpc.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= otlp_http
:description: Receive OpenTelemetry traces, logs, and metrics via OTLP/HTTP protocol.

include::connect:components:inputs/otlp_http.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= salesforce
:page-aliases: components:inputs/salesforce.adoc
:description: Runs a SOQL query against the Salesforce REST API, paginates through all result pages, and emits one message per record.
include::connect:components:inputs/salesforce.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
= salesforce_cdc
:page-aliases: components:inputs/salesforce_cdc.adoc
:description: Subscribes to one or more Salesforce Pub/Sub topics in parallel and emits a message per event.

include::connect:components:inputs/salesforce_cdc.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
= salesforce_graphql
:page-aliases: components:inputs/salesforce_graphql.adoc
:description: Executes a GraphQL query against the Salesforce UIAPI and emits one message per record.

include::connect:components:inputs/salesforce_graphql.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= open_telemetry_collector
:page-aliases: components:metrics/open_telemetry_collector.adoc
:description: Send metrics to an https://opentelemetry.io/docs/collector/[Open Telemetry collector^].
include::connect:components:metrics/open_telemetry_collector.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= arc
:page-aliases: components:outputs/arc.adoc
:description: Writes data to an Arc database via the msgpack ingestion endpoint.
include::connect:components:outputs/arc.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= cyborgdb
:page-aliases: components:outputs/cyborgdb.adoc
:description: Inserts items into a CyborgDB encrypted vector index.
include::connect:components:outputs/cyborgdb.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= drop
:page-aliases: components:outputs/drop.adoc
:description: Drop output reference for silently discarding messages in Redpanda Connect pipelines.
include::connect:components:outputs/drop.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= gcp_bigquery_write_api
:page-aliases: components:outputs/gcp_bigquery_write_api.adoc
:description: Streams data into BigQuery using the Storage Write API.
include::connect:components:outputs/gcp_bigquery_write_api.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= iceberg
:page-aliases: components:outputs/iceberg.adoc
:description: Fan out Redpanda topics to Apache Iceberg tables using the REST catalog API.
include::connect:components:outputs/iceberg.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= otlp_grpc
:description: Send OpenTelemetry traces, logs, and metrics via OTLP/gRPC protocol.

include::connect:components:outputs/otlp_grpc.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= otlp_http
:description: Send OpenTelemetry traces, logs, and metrics via OTLP/HTTP protocol.

include::connect:components:outputs/otlp_http.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= salesforce_sink
:page-aliases: components:outputs/salesforce_sink.adoc
:description: Writes messages to Salesforce, routing each Kafka topic to its own sObject configuration.
include::connect:components:outputs/salesforce_sink.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= slack_reaction
:page-aliases: components:outputs/slack_reaction.adoc
:description: Add or remove an emoji reaction to a Slack message.
include::connect:components:outputs/slack_reaction.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= a2a_message
:page-aliases: components:processors/a2a_message.adoc
:description: Sends messages to an A2A (Agent-to-Agent) protocol agent and returns the response.
include::connect:components:partial$components/cloud-only/processors/a2a_message.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= jira
:page-aliases: components:processors/jira.adoc
:description: Queries Jira resources and returns structured data.
include::connect:components:processors/jira.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
= string_split
:page-aliases: components:processors/string_split.adoc
:description: Splits a string by a delimiter into an array. Generally, using bloblang's `split` method is preferred. In some high performance use cases this processor can be faster than the equivalent bloblang if there is no additional logic.

include::connect:components:processors/string_split.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
= try_catch
:page-aliases: components:processors/try_catch.adoc
:description: Executes a list of child `processors` on each message and, if any of them fail, executes a separate list of `catch` processors to recover from or react to the error.

include::connect:components:processors/try_catch.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= json_array
:page-aliases: components:scanners/json_array.adoc
:description: Consumes a stream of one or more JSON elements within a top level array.
include::connect:components:scanners/json_array.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= open_telemetry_collector
:page-aliases: components:tracers/open_telemetry_collector.adoc
:description: Send tracing events to an https://opentelemetry.io/docs/collector/[Open Telemetry collector^].
include::connect:components:tracers/open_telemetry_collector.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= redpanda
:page-aliases: components:tracers/redpanda.adoc
:description: Send tracing events to a Redpanda topic.
include::connect:components:tracers/redpanda.adoc[tag=single-source]
1 change: 1 addition & 0 deletions modules/develop/pages/connect/configuration/about.adoc
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= Configuration
:page-aliases: configuration:about.adoc
:description: Learn about different options for configuring Redpanda Connect.
include::connect:configuration:about.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= Windowed Processing
:page-aliases: configuration:windowed_processing.adoc
:description: Learn how to process periodic windows of messages with Redpanda Connect.
include::connect:configuration:windowed_processing.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= DynamoDB CDC Patterns
:page-aliases: cookbooks:dynamodb_cdc.adoc
:description: Learn how to capture, filter, transform, and route DynamoDB change data capture (CDC) events with Redpanda Connect.
include::connect:cookbooks:dynamodb_cdc.adoc[tag=single-source]
1 change: 1 addition & 0 deletions modules/develop/pages/connect/cookbooks/enrichments.adoc
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= Enrichment Workflows
:page-aliases: cookbooks:enrichments.adoc
:description: How to configure Redpanda Connect to process a workflow of enrichment services.
include::connect:cookbooks:enrichments.adoc[tag=single-source]
1 change: 1 addition & 0 deletions modules/develop/pages/connect/cookbooks/filtering.adoc
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= Filtering and Sampling
:page-aliases: cookbooks:filtering.adoc
:description: Configure Redpanda Connect to conditionally drop messages.
include::connect:cookbooks:filtering.adoc[tag=single-source]
3 changes: 2 additions & 1 deletion modules/develop/pages/connect/cookbooks/index.adoc
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= Cookbooks
:page-layout: index
:page-aliases: cookbooks:index.adoc
:page-aliases: cookbooks:index.adoc
:description: Step-by-step Redpanda Connect cookbooks that walk through complete, real-world pipeline builds in Redpanda Cloud.
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= Joining Streams
:page-aliases: cookbooks:joining_streams.adoc
:description: How to hydrate documents by joining multiple streams.
include::connect:cookbooks:joining_streams.adoc[tag=single-source]
1 change: 1 addition & 0 deletions modules/develop/pages/connect/guides/bloblang/about.adoc
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= Bloblang
:page-aliases: guides:bloblang/about.adoc
:description: Learn what Bloblang is and how to use the native mapping language.
include::connect:guides:bloblang/about.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= Bloblang Arithmetic
:page-aliases: guides:bloblang/arithmetic.adoc
:description: How arithmetic works within Bloblang
include::connect:guides:bloblang/arithmetic.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= Bloblang Functions
:page-aliases: guides:bloblang/functions.adoc
:description: A list of Bloblang functions
include::connect:guides:bloblang/functions.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= Bloblang Methods
:page-aliases: guides:bloblang/methods.adoc
:description: A list of Bloblang methods
include::connect:guides:bloblang/methods.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= Bloblang Walkthrough
:page-aliases: guides:bloblang/walkthrough.adoc
:description: A step by step introduction to Bloblang
include::connect:guides:bloblang/walkthrough.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= Authenticate to Amazon Aurora using IAM roles
:page-aliases: guides:cloud/aws-iam-aurora.adoc
:description: Configure IAM roles to authenticate Redpanda Connect pipelines to Amazon Aurora on AWS without static credentials.
include::connect:guides:cloud/aws-iam-aurora.adoc[tag=single-source]
1 change: 1 addition & 0 deletions modules/develop/pages/connect/guides/cloud/aws.adoc
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= Amazon Web Services
:page-aliases: guides:cloud/aws.adoc
:description: Find out about AWS components in Redpanda Connect.
include::connect:guides:cloud/aws.adoc[tag=single-source]
1 change: 1 addition & 0 deletions modules/develop/pages/connect/guides/cloud/gcp.adoc
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= Google Cloud Platform
:page-aliases: guides:cloud/gcp.adoc
:description: Find out about GCP components in Redpanda Connect.
include::connect:guides:cloud/gcp.adoc[tag=single-source]
3 changes: 2 additions & 1 deletion modules/develop/pages/connect/guides/index.adoc
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= Guides
:page-layout: index
:page-aliases: guides:index.adoc
:page-aliases: guides:index.adoc
:description: Guides for building and operating Redpanda Connect pipelines in Redpanda Cloud, from Bloblang mappings to deployment patterns.
1 change: 1 addition & 0 deletions modules/develop/pages/connect/guides/sync_responses.adoc
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= Synchronous Responses
:page-aliases: guides:sync_responses.adoc
:description: Understand synchronous response handling in Redpanda Connect, ensuring reliable and efficient data processing.
include::connect:guides:sync_responses.adoc[tag=single-source]
1 change: 1 addition & 0 deletions modules/reference/pages/glossary.adoc
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
= Glossary
:description: Definitions of the streaming, cloud, and Redpanda-specific terms used throughout the Redpanda Cloud documentation.
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= rpk cloud auth delete
:description: pass:q[Delete an `rpk` cloud authentication. Deleting a cloud authentication removes it from the `rpk.yaml` file.]

include::streaming:reference:partial$rpk-cloud/rpk-cloud-auth-delete.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= rpk cloud auth list
:description: pass:q[List `rpk` cloud authentications.]

include::streaming:reference:partial$rpk-cloud/rpk-cloud-auth-list.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= rpk cloud auth token
:description: Print the cloud auth token of the current profile. This command prints the auth token for the currently selected cloud authentication to stdout.

include::streaming:reference:partial$rpk-cloud/rpk-cloud-auth-token.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= rpk cloud auth use
:description: pass:q[Select the `rpk` cloud authentication to use. This swaps the current cloud authentication to the specified cloud authentication.]

include::streaming:reference:partial$rpk-cloud/rpk-cloud-auth-use.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= rpk cloud auth
:description: pass:q[Manage `rpk` cloud authentications. An `rpk` cloud authentication allows you to talk to Redpanda Cloud.]

include::streaming:reference:partial$rpk-cloud/rpk-cloud-auth.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= rpk cloud byoc install
:description: Install the BYOC plugin. This command downloads the BYOC managed plugin if necessary.

include::streaming:reference:partial$rpk-cloud/rpk-cloud-byoc-install.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= rpk cloud byoc uninstall
:description: Uninstall the BYOC plugin. This command deletes your locally downloaded BYOC managed plugin if it exists.

include::streaming:reference:partial$rpk-cloud/rpk-cloud-byoc-uninstall.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= rpk cloud byoc
:description: Manage a Redpanda Cloud BYOC agent. For BYOC, Redpanda installs an agent service in your owned cluster.

include::streaming:reference:partial$rpk-cloud/rpk-cloud-byoc.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= rpk cloud cluster select
:description: Update your rpk profile to talk to the requested cluster.

include::streaming:reference:partial$rpk-cloud/rpk-cloud-cluster-select.adoc[tag=single-source]
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
= rpk cloud cluster
:description: pass:q[Manage `rpk` cloud clusters. This command allows you to manage cloud clusters, as well as easily switch between which cluster you are talking to.]

include::streaming:reference:partial$rpk-cloud/rpk-cloud-cluster.adoc[tag=single-source]
Loading
Loading