feat: add environment-backed output target

.agents/skills/espipe/SKILL.md

@@ -38,7 +38,7 @@ Elasticsearch outputs:
 - `http://host:9200/index-name`
 - `https://host:9200/index-name`
 - `known-host:index-name`, resolved from `$ESPIPE_HOSTS` or `~/.espipe/hosts.yml`
-- `elasticsearch:/index-name` or `es:/index-name`, resolved with `ELASTIC_ES_URL` and optionally `ELASTIC_ES_API_KEY`
+- `env:/index-name`, resolved first from the process environment and then from `.env` using `ELASTIC_ES_URL` and optionally `ELASTIC_ES_API_KEY`
 
 Other outputs:
 
@@ -89,7 +89,7 @@ Examples:
 
 - `espipe accounts.csv records:customers`
 - `espipe users.csv https://host:9200/users`
-- `espipe --action upsert --generate-id=true 'docs/**/*.md' elasticsearch:/documents`
+- `espipe --action upsert --generate-id=true 'docs/**/*.md' env:/documents`
 - `espipe --split /hits response.json output.ndjson`
 
 Use only flags the user requests or that are required to express the destination. Do not reinterpret `--action index` as an overwrite-by-source-ID option; IDs are used only when explicit or generated according to the rules above.

CHANGELOG.md

@@ -7,6 +7,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- Added `.env` fallback for missing `ELASTIC_ES_URL` and `ELASTIC_ES_API_KEY` settings used by `env:/` outputs.
+
+### Changed
+
+- Replaced the `elasticsearch:/index` and `es:/index` environment targets with the explicit `env:/index` form.
+
 ## [0.6.1] - 2026-08-22
 
 ### Changed

Cargo.toml

@@ -31,6 +31,7 @@ anydoc = "^0.2.3"
 base64 = "^0.23.1"
 clap = { version = "^4.6.6", features = ["derive"] }
 csv = "^1.4.0"
+dotenvy = "^0.15.7"
 elasticsearch = "^9.1.0-alpha.1"
 env_logger = "^0.11.11"
 eyre = "^0.6.14"

README.md

@@ -139,6 +139,8 @@ Multi-file and glob imports log per-file read or conversion failures and continu
   Sends documents to Elasticsearch over TLS.
 - `known-host:index-name`
   Resolves `known-host` from a local hosts file and sends to the named index.
+- `env:/index-name`
+  Reads the cluster URL and optional API key from environment variables or `.env`.
 
 When writing to Elasticsearch, the output path must include an index name.
 
@@ -242,17 +244,17 @@ espipe docs.ndjson ess-cluster:my-index
 
 Known-host outputs use the authentication and TLS settings from their host entry.
 
-### Elastic CLI contexts
+### Environment targets
 
-As an [Elastic CLI extension](https://github.com/elastic/cli), `espipe` reads the active Elasticsearch context from:
+The `env:/index` output reads its connection settings from:
 
 - `ELASTIC_ES_URL` supplies the Elasticsearch base URL.
 - `ELASTIC_ES_API_KEY` supplies API-key authentication when no `--apikey`, `--username`, or `--password` option is provided.
 
-Use `elasticsearch:/index` or `es:/index` as the output. These schemes take precedence over same-named known hosts.
+Values already present in the process environment take precedence. For missing values, `espipe` searches the current directory and its parents for a `.env` file. The command fails if `ELASTIC_ES_URL` remains unset. This also works with environment variables supplied by an [Elastic CLI extension](https://github.com/elastic/cli).
 
 ```bash
-espipe docs.ndjson es:/my-index
+espipe docs.ndjson env:/my-index
 ```
 
 ## Examples
@@ -303,7 +305,7 @@ espipe docs.ndjson https://example.com:9200/my-index \
 ### Use the active Elastic CLI context
 
 ```bash
-elastic espipe docs.ndjson es:/my-index
+elastic espipe docs.ndjson env:/my-index
 ```
 
 ### Tune bulk requests

openspec/changes/archive/2026-08-26-add-elasticsearch-environment-output/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-08-26

openspec/changes/archive/2026-08-26-add-elasticsearch-environment-output/design.md

@@ -0,0 +1,51 @@
+## Context
+
+Output dispatch treats HTTP and HTTPS schemes as direct Elasticsearch targets, `file` as local output, and other schemes as configured host names. The previous environment-backed branch reserved both `es` and `elasticsearch` before configured-host lookup. Environment settings came only from the process environment. See `proposal.md` for the reason for changing that behavior.
+
+The new capability crosses command startup and output dispatch, and it adds one dependency for `.env` parsing.
+
+## Goals / Non-Goals
+
+**Goals:**
+
+- Make the configuration source visible in the output URI.
+- Keep process environment values authoritative while filling missing values from `.env`.
+- Limit `.env` loading and Elastic environment authentication to `env:/` output.
+- Preserve the existing direct URL and configured-host paths.
+
+**Non-Goals:**
+
+- Add a flag for selecting a `.env` path.
+- Add environment-backed input URIs.
+- Change the names or formats of `ELASTIC_ES_URL` and `ELASTIC_ES_API_KEY`.
+- Change known-host file discovery or authentication.
+
+## Decisions
+
+### Reserve only `env`
+
+Output dispatch checks for the exact `env` scheme before direct URL and configured-host handling. `es` and `elasticsearch` take the configured-host path. This makes the special behavior explicit and avoids permanently consuming plausible cluster aliases.
+
+Keeping the two existing schemes as deprecated aliases was considered. It would preserve compatibility, but it would keep the ambiguity and prevent those configured-host names from working.
+
+### Load `.env` only for environment output
+
+After parsing the output URI, command startup invokes `dotenvy` only when the scheme is `env`. The standard loader searches the working directory and its ancestors and does not replace variables already present in the process environment. Missing `.env` files are allowed. Parse and read errors other than absence stop startup.
+
+Loading `.env` unconditionally at process startup was considered. It could change logging or authentication for direct URL and configured-host commands, which is outside this capability.
+
+### Keep authentication precedence in command startup
+
+Command-line authentication remains authoritative. The resolved environment API key is passed to authentication setup only for `env:/` output and only when no command-line authentication option is present. This keeps output construction independent of argument precedence rules.
+
+### Validate and join the URL at the output boundary
+
+The environment output branch parses `ELASTIC_ES_URL`, requires an absolute HTTP or HTTPS URL with a host, appends the target index to the configured base path, and removes query and fragment components. It then reuses the direct Elasticsearch output builder.
+
+Building the target with string concatenation was considered, but URL parsing gives consistent validation and path handling before network work starts.
+
+## Risks / Trade-offs
+
+- [Existing commands using `es:/` or `elasticsearch:/` break] -> Document `env:/` as the migration and record the change as breaking.
+- [A `.env` file in a parent directory supplies settings unexpectedly] -> Follow `dotenvy`'s documented nearest-file search and state that behavior in the capability spec and README.
+- [Malformed `.env` content blocks an environment-backed command] -> Report the parsing error before input ingestion or network activity.

openspec/changes/archive/2026-08-26-add-elasticsearch-environment-output/proposal.md

@@ -0,0 +1,28 @@
+## Why
+
+The `es:/` and `elasticsearch:/` output schemes do not say that espipe resolves their connection settings from environment variables. They also prevent users from assigning those names to configured hosts. An explicit `env:/` target makes the configuration source clear and leaves ordinary host aliases available.
+
+## What changes
+
+- Add `env:/<index>` as the Elasticsearch output form backed by `ELASTIC_ES_URL` and the optional `ELASTIC_ES_API_KEY`.
+- Load missing environment settings from the nearest `.env` file without replacing values already present in the process environment.
+- Fail with a clear error when `ELASTIC_ES_URL` remains unset or is not an absolute HTTP or HTTPS URL.
+- Preserve explicit command-line authentication precedence over `ELASTIC_ES_API_KEY`.
+- **BREAKING**: Stop reserving `es:/` and `elasticsearch:/`; resolve them as configured host names instead.
+
+## Capabilities
+
+### New capabilities
+
+- `elasticsearch-environment-output`: Defines the environment-backed output URI, setting precedence, URL validation, authentication precedence, and configured-host namespace behavior.
+
+### Modified capabilities
+
+None.
+
+## Impact
+
+- Affects output URI dispatch and environment authentication handling in `src/main.rs` and `src/output/mod.rs`.
+- Adds the `dotenvy` runtime dependency.
+- Adds CLI coverage for process environment, `.env`, and missing-setting behavior.
+- Changes commands that used `es:/` or `elasticsearch:/` to use `env:/`.

openspec/changes/archive/2026-08-26-add-elasticsearch-environment-output/specs/elasticsearch-environment-output/spec.md

@@ -0,0 +1,99 @@
+## Purpose
+
+Define how an `env:/` Elasticsearch output resolves connection settings while preserving configured host names and explicit command-line authentication.
+
+## ADDED Requirements
+
+### Requirement: Environment output uses an explicit URI form
+
+The system SHALL reserve `env:/<index>` for an Elasticsearch output whose connection settings come from the process environment or `.env`. The URI SHALL contain one slash after `env:` and a non-empty index path. The system SHALL NOT reserve `es` or `elasticsearch` for environment-backed output.
+
+#### Scenario: Valid environment output is provided
+
+- **WHEN** the user provides `env:/logs` as the output
+- **THEN** the system selects environment-backed Elasticsearch output
+- **AND** it uses `logs` as the target index
+
+#### Scenario: Environment output omits the required index
+
+- **WHEN** the user provides `env:/` as the output
+- **THEN** startup fails with an error that identifies `env:/index` as the required form
+
+#### Scenario: Environment output uses an authority or omits the slash
+
+- **WHEN** the user provides `env://logs` or `env:logs` as the output
+- **THEN** startup fails with an error that identifies `env:/index` as the required form
+
+#### Scenario: Former environment scheme is used
+
+- **WHEN** the user provides an output whose scheme is `es` or `elasticsearch`
+- **THEN** the system resolves that scheme as a configured host name
+- **AND** it does not read Elastic environment settings for that output
+
+### Requirement: Environment settings use deterministic precedence
+
+For an `env:/` output, the system SHALL use values already present in the process environment. For each missing setting, it SHALL search the working directory and its ancestors for the nearest `.env` file and load the setting from that file. A `.env` value SHALL NOT replace a value already present in the process environment.
+
+#### Scenario: Process environment contains the URL
+
+- **WHEN** `ELASTIC_ES_URL` is present in the process environment
+- **AND** `.env` contains a different `ELASTIC_ES_URL`
+- **THEN** the system uses the value from the process environment
+
+#### Scenario: Dotenv supplies a missing URL
+
+- **WHEN** `ELASTIC_ES_URL` is absent from the process environment
+- **AND** the nearest `.env` file defines `ELASTIC_ES_URL`
+- **THEN** the system uses the value from `.env`
+
+#### Scenario: Dotenv file is malformed
+
+- **WHEN** the nearest `.env` file cannot be parsed
+- **THEN** startup fails with an error that identifies `.env` as unreadable
+
+#### Scenario: Non-environment output is selected
+
+- **WHEN** the output does not use the `env` scheme
+- **THEN** the system does not load `.env` for Elasticsearch connection settings
+
+### Requirement: Environment URL is required and valid
+
+An `env:/` output SHALL require `ELASTIC_ES_URL` after environment and `.env` resolution. The value SHALL be an absolute `http://` or `https://` URL with a host. The system SHALL append the output index to any existing base path and SHALL discard query and fragment components from the configured URL.
+
+#### Scenario: URL remains unset
+
+- **WHEN** neither the process environment nor `.env` defines `ELASTIC_ES_URL`
+- **THEN** startup fails with an error that names `ELASTIC_ES_URL`
+
+#### Scenario: URL uses an unsupported or relative form
+
+- **WHEN** the resolved `ELASTIC_ES_URL` is relative, lacks a host, or uses a scheme other than HTTP or HTTPS
+- **THEN** startup fails before sending documents
+
+#### Scenario: URL contains a base path
+
+- **WHEN** `ELASTIC_ES_URL` is `https://example.com/elasticsearch/?ignored=true#fragment`
+- **AND** the output is `env:/logs`
+- **THEN** the Elasticsearch output URL is `https://example.com/elasticsearch/logs`
+
+### Requirement: Explicit authentication takes precedence
+
+For an `env:/` output, the system SHALL use `ELASTIC_ES_API_KEY` from the process environment or `.env` when the user supplies no authentication option. An explicit `--apikey` or complete `--username` and `--password` pair SHALL take precedence over `ELASTIC_ES_API_KEY`.
+
+#### Scenario: Environment API key is the only authentication setting
+
+- **WHEN** `ELASTIC_ES_API_KEY` resolves from the process environment or `.env`
+- **AND** the user supplies no authentication option
+- **THEN** the system authenticates with the resolved API key
+
+#### Scenario: Explicit API key is provided
+
+- **WHEN** the user supplies `--apikey`
+- **AND** `ELASTIC_ES_API_KEY` is also set
+- **THEN** the system authenticates with the explicit API key
+
+#### Scenario: Explicit basic authentication is provided
+
+- **WHEN** the user supplies `--username` and `--password`
+- **AND** `ELASTIC_ES_API_KEY` is also set
+- **THEN** the system authenticates with the explicit basic credentials

openspec/changes/archive/2026-08-26-add-elasticsearch-environment-output/tasks.md

@@ -0,0 +1,11 @@
+## 1. Environment output implementation
+
+- [x] 1.1 Reserve `env:/<index>` for environment-backed Elasticsearch output and return `es` and `elasticsearch` to configured-host resolution.
+- [x] 1.2 Load missing Elastic connection settings from `.env` without replacing process environment values, and report missing or invalid URLs before ingestion.
+- [x] 1.3 Restrict environment API-key fallback to `env:/` output while preserving explicit command-line authentication precedence.
+
+## 2. Verification and documentation
+
+- [x] 2.1 Cover environment URI syntax, URL construction, and authentication precedence with unit tests.
+- [x] 2.2 Cover process environment precedence, `.env` fallback and parse failure, non-environment scoping, and missing URL failure with CLI tests.
+- [x] 2.3 Document `env:/` usage, `.env` lookup, migration from the former schemes, and the new dependency.

openspec/specs/elasticsearch-environment-output/spec.md

@@ -0,0 +1,99 @@
+## Purpose
+
+Define how an `env:/` Elasticsearch output resolves connection settings while preserving configured host names and explicit command-line authentication.
+
+## Requirements
+
+### Requirement: Environment output uses an explicit URI form
+
+The system SHALL reserve `env:/<index>` for an Elasticsearch output whose connection settings come from the process environment or `.env`. The URI SHALL contain one slash after `env:` and a non-empty index path. The system SHALL NOT reserve `es` or `elasticsearch` for environment-backed output.
+
+#### Scenario: Valid environment output is provided
+
+- **WHEN** the user provides `env:/logs` as the output
+- **THEN** the system selects environment-backed Elasticsearch output
+- **AND** it uses `logs` as the target index
+
+#### Scenario: Environment output omits the required index
+
+- **WHEN** the user provides `env:/` as the output
+- **THEN** startup fails with an error that identifies `env:/index` as the required form
+
+#### Scenario: Environment output uses an authority or omits the slash
+
+- **WHEN** the user provides `env://logs` or `env:logs` as the output
+- **THEN** startup fails with an error that identifies `env:/index` as the required form
+
+#### Scenario: Former environment scheme is used
+
+- **WHEN** the user provides an output whose scheme is `es` or `elasticsearch`
+- **THEN** the system resolves that scheme as a configured host name
+- **AND** it does not read Elastic environment settings for that output
+
+### Requirement: Environment settings use deterministic precedence
+
+For an `env:/` output, the system SHALL use values already present in the process environment. For each missing setting, it SHALL search the working directory and its ancestors for the nearest `.env` file and load the setting from that file. A `.env` value SHALL NOT replace a value already present in the process environment.
+
+#### Scenario: Process environment contains the URL
+
+- **WHEN** `ELASTIC_ES_URL` is present in the process environment
+- **AND** `.env` contains a different `ELASTIC_ES_URL`
+- **THEN** the system uses the value from the process environment
+
+#### Scenario: Dotenv supplies a missing URL
+
+- **WHEN** `ELASTIC_ES_URL` is absent from the process environment
+- **AND** the nearest `.env` file defines `ELASTIC_ES_URL`
+- **THEN** the system uses the value from `.env`
+
+#### Scenario: Dotenv file is malformed
+
+- **WHEN** the nearest `.env` file cannot be parsed
+- **THEN** startup fails with an error that identifies `.env` as unreadable
+
+#### Scenario: Non-environment output is selected
+
+- **WHEN** the output does not use the `env` scheme
+- **THEN** the system does not load `.env` for Elasticsearch connection settings
+
+### Requirement: Environment URL is required and valid
+
+An `env:/` output SHALL require `ELASTIC_ES_URL` after environment and `.env` resolution. The value SHALL be an absolute `http://` or `https://` URL with a host. The system SHALL append the output index to any existing base path and SHALL discard query and fragment components from the configured URL.
+
+#### Scenario: URL remains unset
+
+- **WHEN** neither the process environment nor `.env` defines `ELASTIC_ES_URL`
+- **THEN** startup fails with an error that names `ELASTIC_ES_URL`
+
+#### Scenario: URL uses an unsupported or relative form
+
+- **WHEN** the resolved `ELASTIC_ES_URL` is relative, lacks a host, or uses a scheme other than HTTP or HTTPS
+- **THEN** startup fails before sending documents
+
+#### Scenario: URL contains a base path
+
+- **WHEN** `ELASTIC_ES_URL` is `https://example.com/elasticsearch/?ignored=true#fragment`
+- **AND** the output is `env:/logs`
+- **THEN** the Elasticsearch output URL is `https://example.com/elasticsearch/logs`
+
+### Requirement: Explicit authentication takes precedence
+
+For an `env:/` output, the system SHALL use `ELASTIC_ES_API_KEY` from the process environment or `.env` when the user supplies no authentication option. An explicit `--apikey` or complete `--username` and `--password` pair SHALL take precedence over `ELASTIC_ES_API_KEY`.
+
+#### Scenario: Environment API key is the only authentication setting
+
+- **WHEN** `ELASTIC_ES_API_KEY` resolves from the process environment or `.env`
+- **AND** the user supplies no authentication option
+- **THEN** the system authenticates with the resolved API key
+
+#### Scenario: Explicit API key is provided
+
+- **WHEN** the user supplies `--apikey`
+- **AND** `ELASTIC_ES_API_KEY` is also set
+- **THEN** the system authenticates with the explicit API key
+
+#### Scenario: Explicit basic authentication is provided
+
+- **WHEN** the user supplies `--username` and `--password`
+- **AND** `ELASTIC_ES_API_KEY` is also set
+- **THEN** the system authenticates with the explicit basic credentials