Skip to content

fix(network): refresh expired policy DNS mappings on connect - #5650

Merged
drew merged 4 commits into
mainfrom
fix/4674-refresh-expired-dns/drew
Nov 18, 2026
Merged

drew merged 4 commits into
mainfrom
fix/4674-refresh-expired-dns/drew

Conversation

@drew

@drew drew commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Refresh expired policy DNS mappings when a new transparent connection uses a cached synthetic address. Clients can reconnect after the resolver TTL without performing another DNS lookup, while resolver failures and rejected destinations still fail closed.

Related Issue

Closes #4674

Changes

  • Retain expired mapping identities and reuse the trusted DNS publication and destination-validation path on connect through PolicyDnsService::lookup_for_connect, which returns the resulting mapping and owns refresh coordination; the store stays synchronous.
  • Reauthorize mapped connections at dispatch and rebuild connectors from the latest validated pins.
  • Retain authored refresh-eligible ports independently of the contracts satisfied by the latest DNS answer.
  • Combine concurrent refreshes for each synthetic address and back off for one second after failure. Start the fresh mapping lease after resolution completes.
  • Preserve process/port authorization and policy-generation checks; policy-generation invalidation remains tracked separately in bug(network): every policy generation change invalidates all policy DNS mappings #5366.
  • Document refresh behavior and the mapping-denial reason, and exercise cached-address expiry in the existing transparent TCP E2E test.

Release Notes

Clients that cache DNS answers can open new connections after a synthetic mapping expires. OpenShell refreshes and validates the destination before connecting.

Testing

  • Networking crate: 1,680 tests passed, 2 ignored.
  • Dispatch regression verifies a real TCP connection to the fresh destination after expiry; partial-mapping regression verifies recovery of a missing port after expiry.
  • Focused concurrency regression passed with a resolver slower than the one-second TTL floor.
  • Clippy for all networking crate targets, Rust formatting, and mise run docs passed.
  • Repository pre-commit checks passed, including workspace and additional test/example Clippy checks, formatting, Markdown, license, Helm, and SDK checks.
  • Docker sandbox E2E local_container_native_tcp_uses_policy_dns_and_fails_closed passed with freshly built supervisor and sandbox images, including a connection to a cached numeric address after 31 seconds.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Relevant implementation and published documentation updated

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
@github-actions github-actions Bot added the state:accepted A maintainer decided OpenShell should pursue this issue label Nov 17, 2026
@github-actions

Copy link
Copy Markdown

@drew drew added area:runtime Sandbox execution, isolation, supervision, policy enforcement, networking, and provider lifecycle. topic:documentation Architecture docs, READMEs, project website updates, and automated documentation generation. topic:networking General network transport, connectivity, and proxy behavior. topic:policy Policy authoring, validation, proving, distribution, enforcement, ergonomics, and hardening. topic:reliability High availability, recovery, retries, graceful degradation, and correctness across restarts. labels Nov 17, 2026
@johntmyers johntmyers added the test:e2e Requires end-to-end coverage label Nov 17, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for e66f4e3. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@johntmyers johntmyers left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gator-agent

PR Review Status

The code review found no blocking issues. Expired mappings refresh through the trusted resolver and destination-validation path, while process, port, endpoint, and policy-generation checks remain in place. The docs explain refresh behavior and failed-refresh denials.

Blocking findings: None.

Carried findings: None.

Gator metadata
  • Validation: Focused network reliability fix implementing accepted issue #4674.
  • Docs: Published network-policy and logging pages updated.
  • Checks: Branch checks remain pending; Helm lint passed and deployment scanning was not applicable.
  • E2E: test:e2e applied; current-head Branch E2E Checks attempt 2 running for the standard suite, including the cached-address expiry regression.
  • Head SHA: e66f4e3ef69851e0ab0ae4e6860632574e020555
  • Base SHA: 624aceea32934f73c350ede3f2185880816a030c
  • Merge base SHA: 2e5589b2180dd432320adda124c4f7bcede119b5
  • Patch ID: 5893900c887c164a917e8afbb8d4fbcd9a9ef6b2
  • Gator payload: 13
  • Review mode: initial
  • Previous reviewed SHA: none
  • Review budget exhausted: no
  • Maintainer decision required: no
  • Next state: gator:watch-pipeline

@johntmyers johntmyers added the gator:watch-pipeline Gator is monitoring PR CI/CD status label Nov 17, 2026

@TaylorMutch TaylorMutch left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed e66f4e3 locally. I found two reproducible correctness issues, detailed inline.

Validation: 71 existing policy-DNS tests passed. Two additional focused regressions reproduced the findings below, including the dispatch case with real TTL expiry. The tests ran in a temporary copy of this exact commit.

@johntmyers johntmyers added gator:in-review Gator is reviewing or awaiting PR review feedback and removed gator:watch-pipeline Gator is monitoring PR CI/CD status labels Nov 17, 2026
Signed-off-by: Drew Newberry <anewberry@nvidia.com>

@johntmyers johntmyers left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gator-agent

PR Review Status

The follow-up review found no new blocking issues. Thanks @TaylorMutch: I checked both reported correctness problems against the new code and regressions. Dispatch now reauthorizes and rebuilds connectors from current pins, and refresh eligibility retains authored ports when the latest DNS answer satisfies only some ports.

@shiju-nv, I also checked your requests for service-owned refresh orchestration, simpler resolver fixtures, and a compact rejection test. Those remain open advisory review topics; this commit retains the existing structure. I have left your threads untouched.

Blocking findings: None.

Carried Gator findings: None.

Required branch checks and the standard E2E suite are running on this head. Gator will monitor their results; maintainer review is still required.

Gator metadata
  • Validation: Focused network reliability fix implementing accepted issue #4674.
  • Docs: Previously reviewed published network-policy and logging updates remain accurate; delta adds implementation details to the crate README.
  • Checks: DCO, Helm Lint, and Trivy Changes passed; Branch Checks pending.
  • E2E: test:e2e retained; current-head Branch E2E Checks run 44156946578 is queued with standard-suite jobs running. No rerun needed for this head.
  • Head SHA: 97e8a42fc96be51179c31eb19ed2691bc2cd2100
  • Base SHA: 624aceea32934f73c350ede3f2185880816a030c
  • Merge base SHA: 2e5589b2180dd432320adda124c4f7bcede119b5
  • Patch ID: b10279719bf29e9a09ebfad4b49dfd2b9da29bff
  • Gator payload: 13
  • Review mode: follow_up
  • Previous reviewed SHA: e66f4e3ef69851e0ab0ae4e6860632574e020555
  • Review budget exhausted: no
  • Maintainer decision required: no convergence decision required; open advisory human threads remain for normal review.
  • Next state: gator:watch-pipeline

@johntmyers johntmyers added gator:watch-pipeline Gator is monitoring PR CI/CD status and removed gator:in-review Gator is reviewing or awaiting PR review feedback gator:watch-pipeline Gator is monitoring PR CI/CD status labels Nov 17, 2026
@johntmyers johntmyers added the gator:approval-needed Gator completed review; maintainer approval needed label Nov 17, 2026
TaylorMutch
TaylorMutch previously approved these changes Nov 17, 2026

@TaylorMutch TaylorMutch left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rechecked 97e8a42. Both previously reported correctness issues are fixed: dispatch reauthorizes and rebuilds connectors from current DNS pins, and refresh eligibility preserves authored ports independently of the latest usable contracts. No new correctness findings. Validation: 72 policy-DNS tests and the real-TCP dispatch regression passed locally; Branch Checks and E2E are green. The open architecture and test-simplification suggestions are advisory.

@johntmyers johntmyers added gator:merge-ready and removed gator:approval-needed Gator completed review; maintainer approval needed labels Nov 17, 2026
Signed-off-by: Drew Newberry <anewberry@nvidia.com>

@johntmyers johntmyers left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gator-agent

PR Review Status

The follow-up review found no new blocking issues. Thanks @shiju-nv: I checked the service-owned lookup_for_connect, synchronous store, simpler resolver fixtures, and compact rejection test against your requests. The refactor addresses them while preserving fresh-pin dispatch and authored-port refresh eligibility from the fixes @TaylorMutch verified.

Blocking findings: None.

Carried Gator findings: None.

Branch Checks and the standard E2E suite are running on this head. Maintainer approval is required again after the new commit; Gator will check the pipeline results next cycle.

Gator metadata
  • Validation: Focused network reliability fix implementing accepted issue #4674.
  • Docs: Previously reviewed published network-policy and logging updates remain accurate; crate README reflects service-owned refresh.
  • Checks: DCO, Helm Lint, Trivy Changes, and issue acceptance passed. Branch Checks pending. Previous approval dismissed; current review decision is REVIEW_REQUIRED.
  • E2E: test:e2e retained; current-head Branch E2E Checks run 44207058317 is queued with build jobs running. No rerun or mirror refresh required for this head.
  • Head SHA: 3b684da721e3388f4158becd639af31cb46bf0b6
  • Base SHA: 624aceea32934f73c350ede3f2185880816a030c
  • Merge base SHA: 2e5589b2180dd432320adda124c4f7bcede119b5
  • Patch ID: 4467d0d025f72556a9610303e080a1e2cf6863e7
  • Gator payload: 13
  • Review mode: follow_up
  • Previous reviewed SHA: 97e8a42fc96be51179c31eb19ed2691bc2cd2100
  • Review budget exhausted: no
  • Maintainer decision required: no convergence decision required; no open Gator obligations or qualifying scope growth.
  • Next state: gator:watch-pipeline

@johntmyers johntmyers added gator:watch-pipeline Gator is monitoring PR CI/CD status and removed gator:merge-ready labels Nov 18, 2026
Signed-off-by: Drew Newberry <anewberry@nvidia.com>

@johntmyers johntmyers left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gator-agent

PR Review Status

The Critical-only follow-up review found no new blocking issues in the test consolidation. Production code is unchanged. @shiju-nv, I checked the compact refresh-rejection coverage against your earlier request: the combined test still checks failed refresh, backoff, recovery, and reuse. Prior review dispositions remain intact.

Blocking findings: None.

Carried Gator findings: None.

Branch Checks and the standard E2E suite are running on this head. Maintainer review remains required; Gator will inspect the check results next cycle.

Gator metadata
  • Validation: Focused networking reliability fix implementing accepted issue #4674.
  • Docs: Previously reviewed published network-policy and logging updates remain applicable; latest delta only consolidates tests.
  • Checks: DCO, issue acceptance, Helm Lint, and Trivy Changes passed. Branch Checks pending.
  • E2E: test:e2e retained; current-head Branch E2E Checks run 44208805557 has queued and running build jobs. The old label-help rerun request applied to an earlier head; no rerun or mirror refresh is needed for this head.
  • Head SHA: 94f67ac99e5a03c3ffda2fef18076f4843125651
  • Base SHA: 624aceea32934f73c350ede3f2185880816a030c
  • Merge base SHA: 2e5589b2180dd432320adda124c4f7bcede119b5
  • Patch ID: f54d2a98c9b17d5e682d9ae6946309b8a263fec4
  • Gator payload: 13
  • Review mode: critical_only
  • Previous reviewed SHA: 3b684da721e3388f4158becd639af31cb46bf0b6
  • Review budget exhausted: yes
  • Maintainer decision required: no; no unresolved Gator obligations, qualifying scope growth, or proposed Warning requiring authorization.
  • Next state: gator:watch-pipeline

@johntmyers johntmyers added gator:approval-needed Gator completed review; maintainer approval needed and removed gator:watch-pipeline Gator is monitoring PR CI/CD status labels Nov 18, 2026
@johntmyers johntmyers added gator:merge-ready and removed gator:approval-needed Gator completed review; maintainer approval needed labels Nov 18, 2026
@drew
drew added this pull request to the merge queue Nov 18, 2026
Merged via the queue into main with commit 4b433e6 Nov 18, 2026
165 of 169 checks passed
@drew
drew deleted the fix/4674-refresh-expired-dns/drew branch November 18, 2026 05:34
@johntmyers

Copy link
Copy Markdown
Collaborator

gator-agent

Monitoring Complete

Monitoring is complete because this PR has merged.

Final status: Gator had reached merge-ready, with no blocking findings remaining in its latest review. Existing review history is preserved.

I removed the active gator:* label because there is nothing left for gator to monitor on this PR.

Gator metadata
  • Head SHA: 94f67ac99e5a03c3ffda2fef18076f4843125651
  • Gator payload: 13

cv added a commit to NVIDIA/NemoClaw that referenced this pull request Nov 18, 2026
)

Adds a **Last verified** column to every table in [Current
Limits](https://github.com/NVIDIA/NemoClaw/blob/v1/docs/resources/limits.md),
so readers can see which `v1` commit each limit was last checked against
and what was checked, and rechecks every row against
[`cb8ef25`](https://github.com/NVIDIA/NemoClaw/tree/cb8ef25252ffffbca051787adb4711c96ebfcf28)
and its pins: OpenShell 0.1.9 (`ce58cf8e`), NeMo Fabric `135147a0`,
Hermes Agent `b200f5bc`, and OpenClaw 2026.9.34.

## Problem

The limits page didn't say when a row was last true, or against which
source tree. Pins move often (OpenShell 0.1.9 and Fabric `135147a0`
landed on 2026-11-17), so a reader couldn't tell a current limit from a
stale one. Two rows also linked issues that don't track them: the policy
DNS row linked an OpenShell issue closed today, and the StorageClass row
linked an issue about a different change.

## Change

- Every table gets a **Last verified** column. Each cell links the `v1`
commit the row was checked against, then keys for what was checked, such
as ``[`cb8ef25`](…): code, issue``. All rows checked here use `cb8ef25`,
the branch's merge base with `v1`. A short legend under the intro
explains the column and defines the keys: **code** (NemoClaw's source,
tests and E2E cases), **OpenShell**, **Fabric**, **Hermes** and
**OpenClaw** (the source or image that commit pins in its
`versions.json` and `image/fabric/Dockerfile`), **issue** (the linked
issues are open and no fix has landed in those pins), and **docs** (the
linked page). A row marked **not verified** names no commit.
- The source uses one reference-style link definition at the end of the
page; the docs build turns it into an inline link in each cell.
- Each row was checked against NemoClaw's code and tests, the pinned
upstream source (OpenShell's chart and server, Fabric's adapters and
runtime handle, Hermes's endpoint and context checks), and the linked
issues' state.

### Rows removed

None. Every row still holds on `v1` with today's pins.

### Rows narrowed or reworded

- **Podman beyond a local rootless Linux gateway**: "managed inference"
now reads "model services on the Podman host", and the row says a Podman
gateway can use one on a Docker host through [SSH
placement](https://github.com/NVIDIA/NemoClaw/blob/v1/docs/agents-inference/remote-service.md).
The vLLM and Ollama services take SSH Docker placement with a Podman
gateway, as
[`usage.md`](https://github.com/NVIDIA/NemoClaw/blob/v1/docs/build-deploy/usage.md#use-a-managed-podman-gateway)
already says.
- **LoadBalancer Service**: the row now says `expose.service` takes only
`NodePort`, which is why a LoadBalancer can't be selected; the chart's
reasons are unchanged.
- **`gateway.kubernetes.expose` on a cluster other than kind**: now says
this includes any OpenShift Route.
#14522 added `expose.route`, and
no Route has run live.
- **AMD64 Nemotron image and GPU inference**: now names and links [the
Nemotron example for an AMD64 GPU
host](https://github.com/NVIDIA/NemoClaw/blob/v1/docs/agents-inference/models.md#configure-nemotron-on-an-amd64-gpu-host).
- **Managed MCP servers**: now says NemoClaw passes the agent's `mcp`
section to Fabric but doesn't run the servers or grant them network
access or credentials.
- **OpenClaw with an `anthropic` provider**: drops "OpenShell 0.1.8 and
later", since the pin is 0.1.9, which still refuses the bearer header.

### Links fixed

- **Policy DNS expiry**: NVIDIA/OpenShell#4674
closed on 2026-11-18 when NVIDIA/OpenShell#5650
merged, after 0.1.9; the fix isn't in the pinned `ce58cf8e`. The row
stays, and its issue cell now says it's fixed after OpenShell 0.1.9 and
links the fix.
- **Gateway StorageClass**:
#13829 tracks the tagged
`target` change; its review defers chart-only settings such as
`server.dataStorageClass` to an issue that hasn't been filed. The cell
now reads "No open issue".
- Kept: "Not planned" links to
#14307 and
#12651, which record those
decisions, and the reference to
#13546 as the record of the
in-cluster vLLM run.

### Rows marked not verified

- **The headless OpenClaw request hasn't been rehearsed as written**:
checking it needs the
[procedure](https://github.com/NVIDIA/NemoClaw/blob/v1/docs/agents-inference/agents.md#run-one-headless-openclaw-request)
run as written against the applied first-deployment example, with an
OpenClaw image and a model.

## Beyond CI

No live run: each cell names the source or issue state it was checked
against.

---------

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:runtime Sandbox execution, isolation, supervision, policy enforcement, networking, and provider lifecycle. state:accepted A maintainer decided OpenShell should pursue this issue test:e2e Requires end-to-end coverage topic:documentation Architecture docs, READMEs, project website updates, and automated documentation generation. topic:networking General network transport, connectivity, and proxy behavior. topic:policy Policy authoring, validation, proving, distribution, enforcement, ergonomics, and hardening. topic:reliability High availability, recovery, retries, graceful degradation, and correctness across restarts.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A policy DNS address stops taking new connections after the trusted resolver's TTL, one second when that TTL is 0

4 participants