Skip to content

feat(gateway): merge drop-in configs over the base file - #5489

Open
elezar wants to merge 6 commits into
mainfrom
feat/5446-gateway-config-dropins/elezar
Open

elezar wants to merge 6 commits into
mainfrom
feat/5446-gateway-config-dropins/elezar

Conversation

@elezar

@elezar elezar commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

Summary

The gateway now merges *.toml drop-in configs from a directory over its base config file, so operators and test harnesses can customize a package-seeded gateway.toml without editing it. Test environments use the same mechanism instead of a merge script.

Related Issue

Closes #5446

Changes

  • Add load_layered in config_file.rs: load the base file and the drop-in configs, merge them in lexical order, and validate the merged result once. Tables merge key by key; scalars and arrays replace; a drop-in cannot unset a key.
  • Require every file to declare [openshell] version = 2 as an integer, so a malformed value in one file cannot be hidden by a later file that declares 2. A drop-in may set only some keys of a table, such as one key under [openshell.gateway.tls]. Unknown keys are rejected.
  • Attribute an unknown key, a wrong type or a rejected value to the file that sets it, using the key path from serde_path_to_error. An error that no single key explains, such as a required key no file sets, or a rule over a whole table that several files contribute to, names the merged configuration and lists its files. With a single file, an error reports its line and column; with several files, it reports the key path, which holds key names and never values. Parse errors stay free of file contents.
  • Add --config-dir / OPENSHELL_GATEWAY_CONFIG_DIR. Without them, the gateway uses gateway.d/ next to the base file when it exists. An explicit directory must exist. An empty value turns drop-in configs off, including the implicit directory. A gateway.d/ with no drop-in configs and no base file counts as no configuration. --config and --config-dir compose rather than exclude each other, as in CRI-O and kubelet.
  • Follow a drop-in symlink only when it resolves to a regular file inside the drop-in directory, which covers Kubernetes ConfigMap mounts. The base file still rejects symlinks.
  • Make config preflight load the same merged result, list the files it merged, and fail on the same errors as startup. A legacy schema version still reports its version, and a malformed version still reports detected_version=unavailable.
  • Make load and preflight wrappers over the layered loader so there is one implementation and one version detector, reject a negative or oversized version as malformed, and log the merged files at startup.
  • Accept --config as an alias for config preflight --path to match the daemon flag. It is a separate commit.
  • Document the directory, order, merge, symlink and opt-out rules in configuration.mdx, the RPM guide and the man page.
  • Install tmachine test overrides as gateway.d/90-test-overrides.toml, let a package with no default config run on the drop-in alone, and remove merge-gateway-toml.py and its tests.

Not covered: the Helm ConfigMap and Secret mapping from the issue's open questions, and the e2e additional-CA overrides, which still edit the base file because e2e/rust/tests/additional_ca.rs rewrites it in place.

Implicit gateway.d/ discovery applies beside any --config path, so a drop-in directory in a shared writable location is merged unless discovery is turned off with an empty --config-dir. The compatibility review recommends explicit opt-in instead. This PR keeps implicit discovery so packaged gateway.d/ overrides work without extra flags.

Release Notes

Gateway configuration can be split across a base gateway.toml and *.toml drop-in configs in gateway.d/. Later files override earlier ones, and config preflight reports the merged result. A gateway.d/ directory next to an existing --config file is now merged. Set OPENSHELL_GATEWAY_CONFIG_DIR= (empty) or pass --config-dir "" to turn this off.

Testing

  • Checks appropriate to the affected code and behavior pass: cargo clippy -p openshell-server --all-targets is clean and the openshell-server lib tests pass (2717 tests); the pre-commit lint hook passed on each commit.
  • Unit tests added/updated: override, ordering, array replace, partial nested tables, unknown keys, wrong types, missing required keys, whole-table errors across files, secrets, version handling including non-integer versions, cross-file rules, symlinks inside and outside the directory, the no-base case, the empty --config-dir, the --config alias and layered preflight.
  • E2E tests added/updated (if applicable): the tmachine Ansible changes were not run, only parsed as YAML.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

@github-actions

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. cli — No explicit risk acceptance found

For openshell-gateway config preflight --path FILE containing [openshell] version = 'two', detected_version=unavailable becomes detected_version=2, even without fragments.

  • Impact: Automation matching the stable category=malformed detected_version=unavailable response no longer recognizes malformed schema declarations.
  • Action: Preserve existing preflight detected_version semantics on this no-break train by deriving diagnostics from the actual merged version value rather than assuming schema v2.
  • PR: feat(gateway): merge drop-in configs over the base file #5489
Review details
  • Execution: complete
  • Candidate: PR #5489 (31ffaa182e4e3e9658590e33a9893ea83ee8123b)
  • Baseline: 1580589082f693306abd5b63a2d9fc945b0e9caa (1580589082f693306abd5b63a2d9fc945b0e9caa)
  • Reviewed: rust-sdk, python-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

@elezar elezar added the retired:area:gateway Retired; do not apply. See #5151 for item-specific migration to the current taxonomy. label Nov 12, 2026
@elezar
elezar force-pushed the feat/5446-gateway-config-dropins/elezar branch from 69a79e1 to 6903a84 Compare November 12, 2026 15:57

`name` assigns an operator-facing identity to the gateway installation. Set it with `[openshell.gateway].name`, `--name`, or `OPENSHELL_GATEWAY_NAME`. It defaults to `openshell`; the Helm chart defaults it to the chart fullname so all replicas in one installation share a name. Chart fullnames are only unique within their Kubernetes namespace, so set `server.name` explicitly when one collector receives telemetry from multiple namespaces or clusters. This identity is independent of client-side gateway aliases, TLS names, and `gateway_jwt.gateway_id`.

## Drop-In Fragments

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is Drop-In fragments how this is referred to in other ecosystems?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. CRI-O, containerd, kubelet and systemd all call these drop-ins (*.d directories of override files). Renamed to "drop-in configs" throughout the code, docs, man page and tests.

@TaylorMutch TaylorMutch left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two error-attribution bugs remain in the layered loader. Both affect the requirement in #5446 that errors name the file that caused them.

Error attribution can report an error already fixed by a fragment

attribute_to_layer returns the first invalid intermediate configuration without checking whether its error is the error in the final merged configuration:

if let Err(layer_error) = finish_layered(candidate) {
return match layer_error {
ConfigFileError::Parse { detail, .. } => ConfigFileError::Parse {
path: path.clone(),
detail,
},
other => other,
};

Reproduction:

# gateway.toml
[openshell]
version = 2
[openshell.gateway]
database_max_connections = "bad"
# gateway.d/10-fix.toml
[openshell.gateway]
database_max_connections = 10
# gateway.d/20-bad.toml
[openshell.gateway]
not_a_field = 1

The final merged configuration has a valid pool size and an unknown field from 20-bad.toml. Instead, startup reports gateway.toml: invalid type: a string, expected u32. Operators are directed to fix a value that has already been overridden, and the actual failure is hidden. Attribute the error in the final merged configuration rather than returning an unrelated prefix error.

Semantic failures lose the offending fragment's path

attribute_to_layer only attaches a path to Parse errors. InvalidValue and SecretInFile retain no source path, so preflight_layered falls back to the base file:

let path = match &source {
ConfigFileError::Io { path, .. }
| ConfigFileError::NonRegular { path, .. }
| ConfigFileError::Parse { path, .. } => path.clone(),
_ => base.or(dir).map(Path::to_path_buf).unwrap_or_default(),

With a valid base containing [openshell] version = 2 and this fragment:

# gateway.d/20-bad.toml
[openshell.gateway]
database_max_connections = 0

Preflight reports path='.../gateway.toml' category=malformed detected_version=2, although the invalid value comes from 20-bad.toml. Startup's semantic error also omits the fragment path. Before layering, the single selected file made the source unambiguous; with drop-ins, operators are sent to the wrong file. Preserve the originating path for semantic validation errors as well as parse errors.

Verification: both behaviors reproduced in an isolated probe using this commit's loader, merge, and attribution functions with a reduced schema. The full server tests could not run offline because hickory-proto v0.26.2 was not cached. The Ansible integration path was inspected but not executed.

@matthewgrossman matthewgrossman left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the merged-validation design and the tmachine migration. The approach is sound; the notes below are mostly about error attribution and untested paths. Inline comments follow.

path: path.clone(),
detail,
},
other => other,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only Parse errors get re-pathed here; every other variant (SecretInFile, semantic validation errors) falls through to other => other with no file name. Before this PR an operator with one file got a path-specific error. With several fragments, they now see e.g. a secret rejection but can't tell which fragment holds it, yet configuration.mdx says "An error names the first file that causes it." The tests only assert attribution for Parse (layered_secret_in_fragment_is_rejected checks the variant, not the file). Either attribute the other variants too or soften the docs.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. Validation now runs once on the merged table, and an error is attributed by key path to the file that sets the key. SecretInFile, InvalidValue, driver-table and isolation-backend errors name the file as well as parse errors, and the docs say which errors name the merged configuration instead. layered_secret_in_drop_in_is_rejected and layered_semantic_error_names_the_drop_in assert the file name. In 4b7ba0c.

merged
.try_into()
.map_err(|error: toml::de::Error| ConfigFileError::Parse {
path: PathBuf::new(),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a merged-result Parse error is never attributed to a layer (the cumulative replay in attribute_to_layer passes for every prefix), it keeps this empty PathBuf::new(), and preflight_layered forwards that path into the stable failure message, so the operator sees a blank path. Consider falling back to the base or directory path here.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. A failure with no single owning file falls back to the base file or directory, and the merged-configuration error lists every file. layered_missing_required_key_names_the_merged_config checks that no empty path appears. In 4b7ba0c.

///
/// Hidden files and entries without a `.toml` extension are ignored so editor
/// backups and package-manager leftovers never take effect.
pub fn fragment_paths(dir: &Path) -> Result<Vec<PathBuf>, ConfigFileError> {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fragments are read through read_config_contents, so O_NOFOLLOW and the non-regular-file check now apply to every file in gateway.d. A symlinked fragment, which is common in /etc/*.d setups, will fail startup. That matches the base file, but the docs don't say so for fragments; worth one sentence in configuration.mdx.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A drop-in symlink is now followed only when it resolves to a regular file inside the drop-in directory, which covers ConfigMap-style mounts; a link that leaves the directory fails startup, and the base file still rejects symlinks. Documented in configuration.mdx, with tests for links inside and outside the directory. In 4b7ba0c.

};
anchor
.parent()
.map(|parent| parent.join("gateway.d"))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before this PR, a gateway.d directory next to an explicit --config file had no effect. Now any *.toml in it is merged silently, so a leftover or unrelated directory can change gateway behavior on upgrade. This is the stated design and preflight lists merged files, but it's worth a line in the release note.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added to the release note in the PR description. A gateway.d beside an explicit --config is merged, and --config-dir "" or an empty OPENSHELL_GATEWAY_CONFIG_DIR turns drop-ins off (b651d44).

- openshell_gateway_overrides.stat.exists
- openshell_gateway_merge_default | default('') | length > 0

- name: Write a minimal base for a package with no default configuration

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This task only runs when openshell_gateway_merge_default is set and the file is missing. I couldn't find where that variable is set from this diff. If an installer (deb, binaries) leaves it unset and no base file exists, the fragment lands with no base and the gateway fails with a missing version. The PR notes the Ansible changes were only parsed as YAML, so please run at least one packaged tmachine environment before merging.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The openshell_gateway_merge_default variable and the minimal-base task are gone: a package with no default config now runs on the drop-in alone, and the unit with an explicit --config always has its templated base. I have not run a packaged tmachine environment yet. I have applied test:e2e so CI runs them, and will report back here with the result.

- Hidden files and files without a `.toml` extension are ignored.
- A fragment can omit `[openshell] version`. The merged result must declare version 2.

The gateway validates the merged result once, including unknown fields and values that must come from the environment. An error names the first file that causes it. The precedence is: command-line flag, then `OPENSHELL_*` environment variable, then fragments, then the base file, then the built-in default.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See the comment on attribute_to_layer: this sentence is only true for parse errors today.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed together with the attribute_to_layer thread: the docs now say what is attributed to a file and what names the merged configuration. In 4b7ba0c.

@elezar
elezar force-pushed the feat/5446-gateway-config-dropins/elezar branch from 6903a84 to 22fb7a9 Compare November 13, 2026 13:43
@github-actions

Copy link
Copy Markdown

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. configuration — No explicit risk acceptance found

openshell-gateway --config FILE and config preflight --path FILE previously read only FILE but now automatically merge adjacent gateway.d/*.toml files.

  • Impact: Existing operators with previously ignored sibling files can receive different settings or startup/preflight failures without changing their invocation or selected configuration.
  • Action: For v0.1.9, preserve existing single-file discovery and require explicit --config-dir or OPENSHELL_GATEWAY_CONFIG_DIR opt-in for overlays.
  • PR: feat(gateway): merge drop-in configs over the base file #5489
Review details
  • Execution: complete
  • Candidate: PR #5489 (87061c2d25ddd45f13412fd2b9e3dbb255fc6d97)
  • Baseline: b4a6541d870f186ad5f5224be1f8b9ac9c872e20 (b4a6541d870f186ad5f5224be1f8b9ac9c872e20)
  • Reviewed: rust-sdk, python-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

@purp purp added area:platform SDKs, CLIs, APIs, gateway infrastructure, persistence, configuration, and session management. topic:operability Installing/configuring deployments, upgrades, backup/restore, diagnostics, and maintenance. and removed retired:area:gateway Retired; do not apply. See #5151 for item-specific migration to the current taxonomy. labels Nov 13, 2026
@purp purp added the topic:ux CLI commands/flags, help, errors/exit codes, human-readable output, TUI, and first-run experience. label Nov 15, 2026

@SDAChess SDAChess left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My agent found two small bugs. Otherwise directionaly I think this looks very good.

version: u32::try_from(*version).unwrap_or(u32::MAX),
})
}
Some(_) => Ok(()),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we require the version to be the integer 2 in each layer? This branch also accepts values such as version = "two". A later drop-in declaring version = 2 then overwrites that value, allowing startup and preflight to succeed.

I reproduced this with both a malformed base and a malformed earlier drop-in. Rejecting noninteger versions before merging would keep validation consistent with the requirement that every file declares schema version 2.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. Each layer must now declare [openshell] version as an integer, so "two" in the base or an earlier drop-in is rejected and names the file even if a later file declares 2. Preflight still reports detected_version=unavailable, category malformed. Test: layered_a_non_integer_version_is_rejected_even_when_a_later_file_overrides_it. In b651d44.

let owner = if message.starts_with("missing field") {
None
} else {
locate(layers, &path, unknown.is_some())

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we also account for custom validation errors that identify a whole table? For example, if the base sets rotation = "never" and max_files = 7 under [openshell.gateway.ocsf_log], a drop-in changing only ocsf_log.path gets named in the error about max_files requiring daily rotation.

The deserializer reports the containing table, so locate selects the last file touching it. Falling back to a merged-configuration error when the offending keys cannot be identified would help operators avoid investigating the wrong file.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. When an error is about a whole table that more than one file contributes to, it now reports the merged configuration and lists the files instead of naming the last file that touches the table. If a single file owns the table, that file is named. Test: layered_whole_table_error_across_files_names_the_merged_config. In b651d44.

@elezar
elezar force-pushed the feat/5446-gateway-config-dropins/elezar branch from 45f403e to b651d44 Compare November 16, 2026 14:58
@elezar elezar changed the title feat(gateway): merge drop-in config fragments over the base file feat(gateway): merge drop-in configs over the base file Nov 16, 2026
pimlock added a commit that referenced this pull request Nov 16, 2026
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
@elezar elezar added the test:e2e Requires end-to-end coverage label Nov 17, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for b651d44. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@elezar

elezar commented Nov 17, 2026

Copy link
Copy Markdown
Member Author

Thanks for the review, @TaylorMutch. Both attribution bugs are fixed in 4b7ba0c. Validation now runs once on the merged table and an error is attributed by key path, so a value that a later drop-in overrides is not reported (layered_overridden_bad_value_is_not_an_error_and_a_later_unknown_key_is reproduces your example and names 20-bad.toml). Semantic and secret errors keep the originating file, so preflight no longer falls back to the base file (layered_semantic_error_names_the_drop_in).

@rhuss rhuss left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc-review Summary

What Went Well

  • read_drop_in_contents canonicalizes a drop-in symlink and follows it only when it stays inside the drop-in directory, then still opens it with O_NOFOLLOW, so ConfigMap mounts work and a link cannot pull in an outside file.
  • check_version checks every layer before merging, so a later version = 2 cannot hide a malformed version in an earlier file.
  • resolve_config_sources is shared by startup and preflight, so both resolve the same base file and drop-in directory.
  • Validation runs once on the merged table, and attribute_semantic names the file whose value wins without echoing the value (including SecretInFile).
  • merge-settings.yaml replaces the 165-line TOML rewriter with a drop-in copy, so the tmachine tests exercise the shipped merge code.

Findings

Severity File Description Source
Minor crates/openshell-server/src/cli.rs:459 Startup keeps only .file from config_file::load_layered(...) and drops LayeredConfig.source... production
Minor crates/openshell-server/src/cli.rs:1469 No test covers the documented rule that an explicit config directory must exist, and no test par... test-quality
Minor crates/openshell-server/src/cli.rs:1475 The implicit gateway.d next to an explicit --config file is merged without an ownership or p... gator
Minor crates/openshell-server/src/config_file.rs:1872 declared_version/table_version/check_version add a second version-detection path next to ``... architecture
Minor crates/openshell-server/src/config_file.rs:1895 check_version turns an out-of-range integer (for example version = -1) into UnsupportedVers... correctness
Minor crates/openshell-server/src/config_file.rs:1977 No test loads a configuration with no base file. Every load_layered and preflight_layered te... test-quality
Important crates/openshell-server/src/config_file.rs:2063 Config type and value errors no longer say where the bad value is. finish_layered deserializes... gator
Minor crates/openshell-server/src/config_file.rs:4376 In the semantic attribution tests, only one file sets the offending key, so they cannot tell "la... test-quality
Minor deploy/man/openshell-gateway.8.md:17 The SYNOPSIS lists --path PATH (alias --config) | --config-dir DIR as alternatives, although t... goal-alignment

Review Details

These findings were produced by cc-review's reviewer agents and posted without a per-finding review. Failure scenarios were derived from reading the code and were not executed.

  • Findings posted: 9
  • Findings reviewed and not posted: 0
  • Gate outcome: FAIL
  • Participating reviewers: architecture, codex, correctness, gator, goal-alignment, production, test-quality

Coverage note: one of the configured review tools (CodeRabbit) could not run, so this review is not a complete-coverage baseline.

Comment thread crates/openshell-server/src/cli.rs Outdated
Comment on lines 459 to 467
let mut file: Option<ConfigFile> = if sources.is_empty() {
None
} else {
Some(
config_file::load_layered(sources.base.as_deref(), sources.dir.as_deref())
.map_err(|e| miette::miette!("{e}"))?
.file,
)
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: Startup keeps only .file from config_file::load_layered(...) and drops LayeredConfig.sources, so the daemon never logs which base file and gateway.d drop-ins it merged.

Why this matters: A gateway.d/ next to an explicit --config file is now merged without any flag. If a leftover drop-in changes driver or auth settings, journalctl shows no trace of it, and only config preflight (which prints config: merged <path>) reveals the merged files.

Suggested fix: Carry the sources in ServerStartupConfig (for example config_sources: Vec<PathBuf>) and log them in run_from_args after tracing_setup::install, next to the existing TLS and OIDC startup lines.

Source: production agent (also flagged by: architecture agent)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 27e2d7f. The merged files (base first, then drop-in configs) are carried in the startup config and logged once at startup.

Comment on lines +1469 to +1473
// An explicit directory must exist, and an empty one turns drop-in configs
// off. Otherwise use `gateway.d` beside the base file (or beside the
// default location) when it is there.
let dir = if let Some(dir) = dir_override.or_else(|| args.config_dir.clone()) {
(!dir.as_os_str().is_empty()).then_some(dir)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: No test covers the documented rule that an explicit config directory must exist, and no test parses config preflight --config-dir.

Why this matters: If a later change treats any drop_in_paths error as "no drop-ins" (for example with unwrap_or_default()), a mistyped --config-dir or OPENSHELL_GATEWAY_CONFIG_DIR would be ignored silently and no test would fail. Every preflight test builds ConfigPreflightArgs with ..Default::default(), so --config-dir D and --config-dir "" on preflight are never parsed.

Suggested fix: Add a test that passes --config base --config-dir <tmp>/missing and asserts that startup and preflight fail with an error naming the directory, plus a parse test for config preflight --path X --config-dir "" and --config-dir D.

Source: test-quality agent

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 27e2d7f: an_explicit_config_dir_that_does_not_exist_fails_and_names_the_directory covers startup and preflight, and preflight_parses_config_dir_alongside_path parses --config-dir D and --config-dir "" together with --path.

Comment on lines +1475 to +1482
let anchor = match base.clone() {
Some(path) => path,
None => defaults::default_gateway_config_path()?,
};
anchor
.parent()
.map(|parent| parent.join("gateway.d"))
.filter(|dir| dir.is_dir())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: The implicit gateway.d next to an explicit --config file is merged without an ownership or permission check.

Why this matters: If the base file sits in a shared writable directory, another local user can create gateway.d/00.toml and override auth, TLS or driver tables. This needs an unusual layout, and the base file has no ownership check either, so this is hardening rather than a defect.

Suggested fix: Optionally skip, with a warning, a gateway.d or drop-in file that is not owned by the gateway user or root, or that is group- or world-writable (similar to sshd StrictModes).

Source: gator agent

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed that this is hardening and not a defect, and the base file has no such check either. I will leave it out of this PR. The opt-out for operators in a shared directory is an empty --config-dir or OPENSHELL_GATEWAY_CONFIG_DIR. I will file a follow-up for ownership and permission checks.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Filed as #5704.

Comment on lines +1872 to +1904
fn declared_version(table: &toml::Table) -> Option<&toml::Value> {
table
.get("openshell")
.and_then(toml::Value::as_table)
.and_then(|openshell| openshell.get("version"))
}

/// Version declared by `table`, in the form preflight reports it.
fn table_version(table: &toml::Table) -> String {
declared_version(table).map_or_else(
|| "missing".to_string(),
|version| {
version
.as_integer()
.map_or_else(|| "unavailable".to_string(), |v| v.to_string())
},
)
}

/// Check the declared version before deserializing so a legacy schema
/// reports its version instead of tripping over fields it no longer has.
fn check_version(table: &toml::Table) -> Result<(), ConfigFileError> {
match declared_version(table) {
None => Err(ConfigFileError::MissingVersion),
Some(toml::Value::Integer(version)) if *version != i64::from(SCHEMA_VERSION) => {
Err(ConfigFileError::UnsupportedVersion {
version: u32::try_from(*version).unwrap_or(u32::MAX),
})
}
Some(toml::Value::Integer(_)) => Ok(()),
// A string or table here could be overwritten by a later file that
// declares the integer, so each file must declare it itself.
Some(_) => Err(ConfigFileError::InvalidVersion),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: declared_version/table_version/check_version add a second version-detection path next to detected_version, and the two disagree. config_file::load and config_file::preflight no longer have production callers.

Why this matters: For openshell = 5, detected_version returns "unavailable" (category malformed), while the layered path reports missing_version. Two copies of the stable preflight category logic can drift further, and the validate_loaded comment ("Shared by the single-file and layered loaders") is now stale.

Suggested fix: Keep one version-detection implementation (for example, build detected_version on table_version and return "unavailable" when openshell is not a table). Route load and preflight through load_layered(Some(path), None) and preflight_layered(Some(path), None), or remove them and port their tests.

Source: architecture agent

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 27e2d7f. load and preflight are now thin wrappers over load_layered(Some(path), None) and preflight_layered(Some(path), None), parse_and_validate and detected_version are gone, and table_version is the one version detector (openshell = 5 is unavailable/malformed on every path). The stale comment on validate_loaded is fixed. The existing single-file tests now run against the shipped loader.

Comment on lines +1895 to +1900
None => Err(ConfigFileError::MissingVersion),
Some(toml::Value::Integer(version)) if *version != i64::from(SCHEMA_VERSION) => {
Err(ConfigFileError::UnsupportedVersion {
version: u32::try_from(*version).unwrap_or(u32::MAX),
})
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: check_version turns an out-of-range integer (for example version = -1) into UnsupportedVersion { version: u32::MAX } through u32::try_from(*version).unwrap_or(u32::MAX).

Why this matters: config preflight now reports future_version for version = -1, where it used to report malformed. The startup error names version 4294967295, a value the operator never wrote. This changes the stable preflight category for single-file configs too.

Suggested fix: Only build UnsupportedVersion when the value fits in u32, and map everything else to InvalidVersion, for example match u32::try_from(*v) { Ok(SCHEMA_VERSION) => Ok(()), Ok(version) => Err(UnsupportedVersion { version }), Err(_) => Err(InvalidVersion) }. A preflight test for version = -1 that asserts malformed would pin it.

Source: correctness agent (also flagged by: gator agent, codex)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 27e2d7f. An integer that is negative or does not fit in u32 is InvalidVersion (malformed) and no longer UnsupportedVersion { version: u32::MAX }. layered_an_out_of_range_or_misplaced_version_is_malformed covers -1, 4294967296 and openshell = 5.

Comment on lines +1977 to +2004
fn load_layered_inner(
base: Option<&Path>,
dir: Option<&Path>,
) -> Result<LayeredConfig, LayerFailure> {
let origin = base.or(dir).map(Path::to_path_buf).unwrap_or_default();
let mut layers = Vec::new();
if let Some(base) = base {
layers.push(load_layer(base, None)?);
}
if let Some(dir) = dir {
let listed = drop_in_paths(dir).map_err(|error| LayerFailure {
error,
path: dir.to_path_buf(),
version: "unavailable".to_string(),
})?;
for path in listed {
layers.push(load_layer(&path, Some(dir))?);
}
}

let mut merged = toml::Table::new();
for layer in &layers {
merge_tables(&mut merged, layer.table.clone());
}
let version = table_version(&merged);
let file = finish_layered(&layers, merged).map_err(|error| LayerFailure {
path: error.file().map_or(origin, Path::to_path_buf),
error,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: No test loads a configuration with no base file. Every load_layered and preflight_layered test passes Some(&base).

Why this matters: The deb tmachine flow now runs on the drop-in alone, and the origin = base.or(dir) fallback is what keeps preflight from printing an empty path. The deleted merge_gateway_toml_test.py covered the no-base case before, and nothing replaces it.

Suggested fix: Add a load_layered(None, Some(&dir)) test with one 10-a.toml that asserts the merged value and sources, and a preflight_layered(None, Some(&dir)) test for a merged-only error that asserts the message names the directory.

Source: test-quality agent

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 27e2d7f: layered_with_no_base_runs_on_the_drop_ins_alone loads with load_layered(None, Some(dir)) and asserts the merged value and sources, and checks that a preflight failure no single file explains names the directory and never an empty path.

Comment on lines +2063 to +2138
let detail = without_quoted_strings(message);
let mut path: Vec<String> = error
.path()
.iter()
.map_while(|segment| match segment {
serde_path_to_error::Segment::Map { key } => Some(key.clone()),
_ => None,
})
.collect();
let unknown = message
.strip_prefix("unknown field `")
.and_then(|rest| rest.split('`').next());
if let Some(name) = unknown
&& path.last().map(String::as_str) != Some(name)
{
path.push(name.to_string());
}
// A missing key is a property of the merged table, not of one file. So is
// an error about a whole table that several files contribute to, such as a
// rule over two of its keys.
let spans_files = unknown.is_none()
&& value_at(merged, &path).is_some_and(toml::Value::is_table)
&& layers
.iter()
.filter(|layer| has_path(&layer.table, &path))
.count()
> 1;
let owner = if message.starts_with("missing field") || spans_files {
None
} else {
locate(layers, &path, unknown.is_some())
};
match owner {
Some(path) => ConfigFileError::Parse {
path: path.to_path_buf(),
detail,
},
None => ConfigFileError::MergedParse {
files: describe_layers(layers),
detail,
},
}
}

/// Report a validation error against the file that sets the offending key.
fn attribute_semantic(layers: &[Layer], error: ConfigFileError) -> ConfigFileError {
let to_path = |segments: &[&str]| segments.iter().map(ToString::to_string).collect::<Vec<_>>();
let path = match &error {
ConfigFileError::SecretInFile { field, .. } => {
Some(to_path(&["openshell", "gateway", field]))
}
ConfigFileError::InvalidValue { field, .. } => {
Some(field.split('.').map(str::to_string).collect())
}
ConfigFileError::InvalidDriverTable { name } => {
Some(to_path(&["openshell", "drivers", name]))
}
ConfigFileError::InvalidIsolationBackend { name, .. } => Some(to_path(&[
"openshell",
"drivers",
name,
"isolation_backend",
])),
_ => None,
};
match path.and_then(|path| locate(layers, &path, false).map(Path::to_path_buf)) {
Some(owner) => error.in_file(&owner),
None => error,
}
}

/// Deserialize and validate the merged table.
fn finish_layered(layers: &[Layer], merged: toml::Table) -> Result<ConfigFile, ConfigFileError> {
check_version(&merged)?;
let file: ConfigFile = serde_path_to_error::deserialize(toml::Value::Table(merged.clone()))
.map_err(|error| attribute_parse(layers, &merged, &error))?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important: Config type and value errors no longer say where the bad value is. finish_layered deserializes a toml::Value::Table, which carries no source span, and attribute_parse uses error.path() only to pick the file, then builds the message from error.inner().message() alone. Startup (cli.rs:463) and preflight (cli.rs:1128) now use this path even for a single gateway.toml with no drop-ins.

Why this matters: Before this PR, a wrongly typed value such as headers = { x-tenant = 42 } under [openshell.gateway.otlp] produced an error with its line and column (from parse_error_detail). The same file now produces only invalid type: integer ..., expected a string, with no line and no key name, so operators of every deployment lose the location in a large gateway.toml. A missing required key in a single file is also reported as invalid merged gateway config although nothing was merged. The location tests (rejects_unknown_policy_validation_failure_mode, parse_errors_locate_the_problem_without_quoting_the_file) keep passing because they still call config_file::load, which has no production caller now, and the PR description does not mention this change.

Suggested fix: Prefix the detail with the key path, which holds key names and never values, for example format!("{}: {}", error.path(), without_quoted_strings(message)) when the path is non-empty, or recover the line from the owning layer's original contents. Then point the two location tests at load_layered (or have load delegate to it) so they cover the shipped loader, and consider plain file wording instead of MergedParse when only one layer was loaded.

Source: gator agent (also flagged by: goal-alignment agent)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 27e2d7f. With a single file, the error is reported against that file with its line and column again (the file is reread with spans), and a missing required key names the file instead of "merged". With several files, the detail is prefixed with the key path, which holds key names and never values. Tests: a_single_file_reports_the_line_and_column_of_a_wrong_value, a_single_file_missing_a_required_key_names_the_file, layered_a_wrong_value_across_files_reports_the_key_path_and_hides_the_value. The existing location tests pass through load, which now uses the layered loader, so they cover the shipped path. The docs describe the new error format.

Comment on lines +4376 to +4391
fn layered_semantic_error_names_the_drop_in() {
let (_dir, base, frags) = layered_dir(
BASE,
&[(
"20-bad.toml",
"[openshell]\nversion = 2\n[openshell.gateway]\ndatabase_max_connections = 0\n",
)],
);
let error = load_layered(Some(&base), Some(&frags)).unwrap_err();
assert!(error.to_string().contains("20-bad.toml"), "{error}");
let preflight = preflight_layered(Some(&base), Some(&frags))
.unwrap_err()
.to_string();
assert!(preflight.contains("20-bad.toml"), "{preflight}");
assert!(preflight.contains("detected_version=2"), "{preflight}");
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: In the semantic attribution tests, only one file sets the offending key, so they cannot tell "last file wins" from "first file wins" in attribute_semantic.

Why this matters: Changing locate(layers, &path, false) to true, or reversing the iteration, would still pass every test. The common case is a base that sets database_max_connections = 5 and a drop-in that overrides it with 0, where the error must name the drop-in.

Suggested fix: Let the base set database_max_connections = 5 and 20-bad.toml set 0, then assert the error contains 20-bad.toml and not gateway.toml. A case with [openshell.drivers]\npodman = 1 in a drop-in would also cover InvalidDriverTable attribution.

Source: test-quality agent

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 27e2d7f: layered_the_drop_in_that_sets_the_bad_value_is_named_when_the_base_sets_a_good_one (base sets 5, 20-bad.toml sets 0, and the error names the drop-in and not gateway.toml) and layered_an_invalid_driver_table_names_the_drop_in_that_sets_it.

Comment thread deploy/man/openshell-gateway.8.md Outdated
**openshell-gateway** \[*OPTIONS*\]

**openshell-gateway** **config preflight** [**--path** *PATH* | **--** *GATEWAY_ARGS*...]
**openshell-gateway** **config preflight** [**--path** *PATH* (alias **--config**) | **--config-dir** *DIR* | **--** *GATEWAY_ARGS*...]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: The SYNOPSIS lists --path PATH (alias --config) | --config-dir DIR as alternatives, although the two combine, and the CONFIGURATION PREFLIGHT synopsis at line 176 leaves out --config-dir entirely.

Why this matters: In ConfigPreflightArgs, config_dir conflicts only with gateway_args. An operator reading the man page would think they cannot check a base file together with a custom drop-in directory, and would not learn from it that preflight merges gateway.d.

Suggested fix: Use [--path PATH (alias --config)] [--config-dir DIR] | -- GATEWAY_ARGS... on both line 17 and line 176, and add one sentence saying that preflight merges drop-in configs and prints the files it merged.

Source: goal-alignment agent (also flagged by: architecture agent)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 27e2d7f. Both synopses now read [--path PATH (alias --config)] [--config-dir DIR] | -- GATEWAY_ARGS..., and the section says that preflight merges the drop-in configs, prints the files it merged, and fails on the same errors as startup.

Move `config_file.rs` to `config_file/mod.rs` with no change to its
contents, so the code that loads layered configs can live in its own file.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
…light

Select, read and fold the configuration file into the daemon arguments in
one function that both startup and preflight call, so they cannot drift
apart on which file they read. Preflight still reports failures in its
stable format.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
…effects

Move the assembly of the gateway's runtime configuration out of
`prepare_server_config_with_drivers` into `assemble_server_config`, which
takes the effective arguments and the configuration file and has no side
effects. Startup keeps the steps that create directories, probe drivers
and stage artifacts, and behaves as before.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
Load `*.toml` drop-in configs from a directory and merge them over the
base config in lexical order, so operators and packages can customize a
gateway without editing the base file. Select the directory with
`--config-dir` or `OPENSHELL_GATEWAY_CONFIG_DIR`, or let the gateway use
`gateway.d` beside the base file. An empty value turns drop-in configs
off. `--config` and `--config-dir` compose.

Tables merge key by key; scalars and arrays replace; a drop-in cannot
unset a key. Every file must declare `[openshell] version = 2` as an
integer, but a drop-in may set only some keys of a table. The merged
result is validated once. An error names the file that sets the
offending key, with its line and column for a single file and its key
path for several. Errors that no single key explains, such as a missing
required key or a rule over a table that several files set, name the
merged configuration and its files. Unknown keys are rejected.

Follow a drop-in symlink only when it resolves inside the drop-in
directory, which covers Kubernetes ConfigMap mounts, and keep the base
file strict. Treat a `gateway.d` with no drop-in configs and no base
file as no configuration. `load` and `preflight` use the same loader,
`config preflight` lists the files it merged, and the gateway logs them
at startup.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
Install an environment's gateway overrides as `gateway.d/90-test-overrides.toml`
next to the gateway's own config instead of merging them into it with a
script. A package with no default config runs on the drop-in config alone.
The override files declare `[openshell] version = 2`.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
Match the daemon flag so the same name selects the base config file in
both places. Keep --path for compatibility.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
@elezar
elezar force-pushed the feat/5446-gateway-config-dropins/elezar branch from 539999a to 90a42e6 Compare November 18, 2026 13:24

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:platform SDKs, CLIs, APIs, gateway infrastructure, persistence, configuration, and session management. test:e2e Requires end-to-end coverage topic:operability Installing/configuring deployments, upgrades, backup/restore, diagnostics, and maintenance. topic:ux CLI commands/flags, help, errors/exit codes, human-readable output, TUI, and first-run experience.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(gateway): support drop-in configuration fragments (gateway.d/*.toml)

6 participants