feat(attest): accept several collateral endpoints instead of one

[kvinwang]

Draft — the mechanism works and is tested, but two decisions below are worth settling before this is worth merging.

Problem

Verification depends on collateral that only the vendor publishes, and each source was configured as exactly one URL. That makes every verifier a single point of failure downstream of a service nobody here operates.

AMD KDS is the sharp case. It is a single global endpoint with no official mirror, rate limited to roughly one identical request per ten seconds, and it was completely unreachable for an entire 0.6.0-rc0 test round — TCP timeouts and 100% ICMP loss from five vantage points across four autonomous systems, while every other external service answered normally. For as long as that lasts, no KMS can verify an SEV-SNP quote, so no AMD guest can obtain keys. The rate limit bites in normal operation too: two cold-process verifications of the same chip return 429, which reads exactly like a verification failure.

There is no host-side escape. normalize_kernel_cert_table can read ASK and VCEK out of the SNP extended report, but stock QEMU has no certs-path-style property on sev-snp-guest and upstream psp-sev.h has no SNP_SET_EXT_CONFIG, so guests ask and receive an empty table.

Change

pccs and amd_kds under [core.attestation.urls] (and [attestation.urls] for dstack-verifier) now accept either a string or a list:

[core.attestation.urls]
pccs = ["https://pccs.internal", "https://pccs.phala.network"]
amd_kds = ["https://kds-cache.internal/vcek/v1", "https://kdsintf.amd.com/vcek/v1"]

Endpoints are tried in order, first answer wins. This covers KMS, dstack-verifier, gateway, and the guest's own HostApi path that verifies the SGX key-provider quote.

Backward compatibility is deliberate rather than incidental: a bare string still parses, the legacy top-level pccs_url still wins when nothing else is set, and a one-element list serializes back out as a bare string — guests hash the sys-config they were handed, so a newer binary must not rewrite its shape.

Why this is not a hole in the trust model

Collateral is vendor-signed and verified against roots compiled into the binary. For AMD the fetched ARK is discarded outright (let (_fetched_ark, ask) = ...). An endpoint in this list can be slow, stale, or absent; it cannot forge. Which endpoint answered has no bearing on whether the signature checks pass — which is what makes a cache or an internal mirror a legitimate entry rather than a downgrade.

Two different failover rules, on purpose

• AMD KDS is in-tree, so it classifies. Transport failures, 408, 429 and 5xx describe the endpoint and move on. A 404 describes the chip — every mirror repeats it — so it surfaces as-is instead of being retried into a pile of identical errors.
• PCCS goes through dcap-qvl, which returns anyhow::Error with no status code to classify on, so it fails over on any error. Matching on error strings to do better would be worse than the coarse rule it replaces.

Tests

sev-snp-qvl/src/failover_tests.rs drives real sockets rather than asserting on the shape of the code:

• a 429 hands off, and the second endpoint serves the answer
• a refused connection hands off
• a 404 stops, and the second endpoint is never contacted
• an all-fail run names every endpoint it tried

Plus round-trip tests in dstack-types pinning the string/list duality and the legacy-field precedence.

Verified against the real thing

Not just unit tests. A Cloudflare Worker caching mirror in front of KDS, with amd_kds pointed at it:

• four consecutive cold-process dstack-verifier --verify runs of one real SEV-SNP attestation all pass; the same sequence run directly against KDS fails on the second with the 429
• a KMS configured this way released keys to an AMD SEV-SNP guest end to end on real hardware, through gateway registration and public traffic

That exercised a single mirror URL rather than the failover path, so the ordering logic here rests on the socket tests above.

Open questions

1. Should the list be shuffled? kms_urls and gateway_urls have shuffle_* flags to spread load. Strict ordering is right for "prefer my cache, fall back to the vendor", wrong for "three equal mirrors". Adding shuffle_collateral_urls is easy; I did not want to guess.
2. Should vmm.toml pass a list through to guests? cvm.pccs_url is still a single String at vmm/src/app.rs, so a guest cannot be given more than one PCCS today even though SysConfig can now carry several. Straightforward follow-up, kept out of this diff to keep it reviewable.

Related: #1140 — nothing here checks collateral freshness, and on SEV-SNP nothing checks certificate validity at all, so an arbitrarily old cached VCEK is indistinguishable from a fresh one. Worth settling separately, but it is the reason to be careful about how long a mirror caches.