Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
105 changes: 90 additions & 15 deletions pipeline/outputs/loki.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,29 +9,46 @@ There is a separate Golang output plugin provided by [Grafana](https://grafana.c
## Configuration parameters

| Key | Description | Default |
|:----|:------------|:--------|
| `host` | Loki base hostname or IP address. Don't include the sub-path, only the base hostname or URL.  | `127.0.0.1` |
| `uri` | Specify a custom HTTP URI. It must start with a forward slash (`/`).| `/loki/api/v1/push` |
| `port` | The Loki TCP port. | `3100` |
| `tls` | Use TLS authentication. | `off` |
| `http_user` | Set HTTP basic authentication user name. | _none_ |
| `http_passwd` | Set HTTP basic authentication password. | _none_ |
| :--- | :--- | :--- |
| `auto_kubernetes_labels` | If set to `true`, adds all Kubernetes labels to the stream labels. | `false` |
| `bearer_token` | Set bearer token authentication token value. | _none_ |
| `buffer_size` | Maximum HTTP response buffer size. | `512KB` |
| `compress` | Set payload compression mechanism. The only available option is `gzip`. | _none_ |
| `drop_single_key` | When set to `true` and after extracting labels only a single key remains, the log line sent to Loki will be the value of that key in `line_format`. If set to `raw` and the log line is a string, the log line will be sent unquoted. | _none_ |
| `header` | Add additional arbitrary HTTP header key/value pair. Multiple headers can be set. | _none_ |
| `tenant_id` | Tenant ID used by default to push logs to Loki. If omitted or empty it assumes Loki is running in single-tenant mode and no `X-Scope-OrgID` header is sent. | _none_ |
| `labels` | Stream labels for API request. It can be multiple comma separated of strings specifying `key=value` pairs. Allows fixed parameters, or adding custom record keys (similar to the `label_keys` property). See the Labels section. | `job=fluent-bit` |
| `host` | Loki base hostname or IP address. Don't include the sub-path, only the base hostname or URL. | `127.0.0.1` |
| `http_passwd` | Set HTTP basic authentication password. | _none_ |
| `http_user` | Set HTTP basic authentication user name. | _none_ |
| `label_keys` | (Optional.) List of record keys that will be placed as stream labels. This configuration property is for records key only. See the Labels section. | _none_ |
| `label_map_path` | Specify the label map path. The file defines how to extract labels from each record. See the Labels section. | _none_ |
| `labels` | Stream labels for API request. It can be multiple comma separated of strings specifying `key=value` pairs. Allows fixed parameters, or adding custom record keys (similar to the `label_keys` property). See the Labels section. | `job=fluent-bit` |
| `line_format` | Format to use when flattening the record to a log line. Valid values are `json` or `key_value`. If set to `json`, the log line sent to Loki will be the Fluent Bit record dumped as JSON. If set to `key_value`, the log line will be each item in the record concatenated together (separated by a single space) in the format. | `json` |
| `oauth2.audience` | Optional `OAuth 2.0` audience parameter. | _none_ |
| `oauth2.auth_method` | `OAuth 2.0` client authentication method. Supported values: `basic`, `post`, `private_key_jwt`. | `basic` |
| `oauth2.client_id` | Required when `oauth2.enable` is `true`. `OAuth 2.0` client ID. | _none_ |
| `oauth2.client_secret` | Required when `oauth2.enable` is `true` and `oauth2.auth_method` is `basic` or `post`. `OAuth 2.0` client secret. | _none_ |
| `oauth2.connect_timeout` | Connect timeout for `OAuth 2.0` token requests. | `0s` |
| `oauth2.enable` | Enable `OAuth 2.0` client credentials for outgoing requests. Incompatible with `http_user`, `http_passwd`, and `bearer_token`. See the `OAuth 2.0` authentication section. | `false` |
| `oauth2.jwt_aud` | Audience for `private_key_jwt` JSON Web Token (JWT) assertion. Defaults to the value of `oauth2.token_url` when not set. | _none_ |
| `oauth2.jwt_cert_file` | Required when `oauth2.enable` is `true` and `oauth2.auth_method` is `private_key_jwt`. Path to certificate file used by `private_key_jwt`. Fluent Bit derives the `oauth2.jwt_header` thumbprint from this certificate. | _none_ |
| `oauth2.jwt_header` | JWT header claim name for the `private_key_jwt` certificate thumbprint. Supported values are `kid` (hexadecimal SHA-1), `x5t` (base64url SHA-1), and `x5t#S256` (base64url SHA-256). Fluent Bit doesn't validate this value: an unrecognized name is sent as the claim name with a hexadecimal SHA-1 thumbprint. | `kid` |
| `oauth2.jwt_key_file` | Required when `oauth2.enable` is `true` and `oauth2.auth_method` is `private_key_jwt`. Path to PEM private key file used by `private_key_jwt`. | _none_ |
| `oauth2.jwt_ttl_seconds` | Lifetime in seconds for `private_key_jwt` JWT client assertions. | `300` |
| `oauth2.refresh_skew_seconds` | Seconds before expiry at which to refresh the access token. | `60` |
| `oauth2.resource` | Optional `OAuth 2.0` resource parameter. | _none_ |
| `oauth2.scope` | Optional `OAuth 2.0` scope. | _none_ |
| `oauth2.timeout` | Timeout for `OAuth 2.0` token requests. A value of `0s` applies no explicit response timeout. | `0s` |
Comment thread
coderabbitai[bot] marked this conversation as resolved.
| `oauth2.token_url` | Required when `oauth2.enable` is `true`. `OAuth 2.0` token endpoint URL. Must use the `http` or `https` scheme. Use `https` in production: with `http`, Fluent Bit sends the client credentials without TLS. | _none_ |
| `oauth2.user_agent` | Optional `User-Agent` header value to include in `OAuth 2.0` token requests. If omitted, no `User-Agent` header is sent. | _none_ |
| `port` | The Loki TCP port. | `3100` |
| `remove_keys` | (Optional.) List of keys to remove. | _none_ |
| `structured_metadata` | (Optional.) Comma-separated list of `key=value` strings specifying structured metadata for the log line. Like the `labels` parameter, values can reference record keys using record accessors. See [Use `structured_metadata`.](#use-structured_metadata). | _none_ |
| `structured_metadata_map_keys` | (Optional.) Comma-separated list of record key strings specifying record values of type `map`, used to dynamically populate structured metadata for the log line. Values can only reference record keys using record accessors, which should reference map values. Each entry from the referenced map will be used to add an entry to the structured metadata. See [Use `structured_metadata`.](#use-structured_metadata). | _none_ |
| `remove_keys` | (Optional.) List of keys to remove. | _none_ |
| `drop_single_key` | When set to `true` and after extracting labels only a single key remains, the log line sent to Loki will be the value of that key in `line_format`. If set to `raw` and the log line is a string, the log line will be sent unquoted. | _none_ |
| `line_format` | Format to use when flattening the record to a log line. Valid values are `json` or `key_value`. If set to `json`, the log line sent to Loki will be the Fluent Bit record dumped as JSON. If set to `key_value`, the log line will be each item in the record concatenated together (separated by a single space) in the format. | `json` |
| `auto_kubernetes_labels` | If set to `true`, adds all Kubernetes labels to the stream labels. | `false` |
| `tenant_id` | Tenant ID used by default to push logs to Loki. If omitted or empty it assumes Loki is running in single-tenant mode and no `X-Scope-OrgID` header is sent. | _none_ |
| `tenant_id_key` | Specify the name of the key from the original record that contains the Tenant ID. The value of the key is set as `X-Scope-OrgID` of HTTP header. Use to set Tenant ID dynamically. | _none_ |
| `tenant_id_key_error_handling` | Set how `tenant_id_key` split request failures affect the whole chunk. Options are `partial_success` (treat mixed success and failure as success; records for failed tenants are dropped), or `partial_error` (retry mixed success and failure; might result in duplicate records for successful tenants). | `partial_success` |
| `buffer_size` | Maximum HTTP response buffer size. | `512KB` |
| `compress` | Set payload compression mechanism. The only available option is `gzip`. | _none_ |
| `tls` | Use TLS authentication. | `off` |
| `uri` | Specify a custom HTTP URI. It must start with a forward slash (`/`). | `/loki/api/v1/push` |
| `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `0` |

## Labels
Expand Down Expand Up @@ -524,6 +541,64 @@ pipeline:
{% endtab %}
{% endtabs %}

## Use `OAuth 2.0` authentication

Set `oauth2.enable` to `true` to authenticate push requests with the `OAuth 2.0` client credentials grant. Fluent Bit requests an access token from `oauth2.token_url` and sends it as a bearer token with every push request to Loki.

Fluent Bit won't start if `oauth2.enable` is `true` and any of `http_user`, `http_passwd`, or `bearer_token` is also set. Remove those parameters when switching to `OAuth 2.0`.

Every `OAuth 2.0` configuration must set `oauth2.token_url` and `oauth2.client_id`. The remaining required parameters depend on `oauth2.auth_method`:

| `oauth2.auth_method` | Additional required parameters |
| :--- | :--- |
| `basic` (default) | `oauth2.client_secret` |
| `post` | `oauth2.client_secret` |
| `private_key_jwt` | `oauth2.jwt_key_file` and `oauth2.jwt_cert_file` |

Fluent Bit won't start if a required parameter is missing.

Fluent Bit refreshes the access token `oauth2.refresh_skew_seconds` before it expires. If Loki returns a `401 Unauthorized` response, Fluent Bit discards the cached token and retries the chunk with a new one.

The following example configuration uses the `basic` authentication method:

{% tabs %}
{% tab title="fluent-bit.yaml" %}

```yaml
pipeline:
outputs:
- name: loki
match: '*'
host: loki.example.com
port: 443
tls: on
oauth2.enable: true
oauth2.token_url: https://auth.example.com/oauth2/token
oauth2.client_id: YOUR_CLIENT_ID
oauth2.client_secret: YOUR_CLIENT_SECRET
oauth2.scope: logs.write
```

{% endtab %}
{% tab title="fluent-bit.conf" %}

```text
[OUTPUT]
Name loki
Match *
Host loki.example.com
Port 443
Tls on
oauth2.enable true
oauth2.token_url https://auth.example.com/oauth2/token
oauth2.client_id YOUR_CLIENT_ID
oauth2.client_secret YOUR_CLIENT_SECRET
oauth2.scope logs.write
```

{% endtab %}
{% endtabs %}

## Networking and TLS configuration

This plugin inherits core Fluent Bit features to customize the network behavior and optionally enable TLS in the communication channel. For more details about the specific options available, refer to the following articles:
Expand Down
Loading