Add versioned v1 backfill artifact contracts

[cotti]

Summary

Adds the versioned v1 contracts for the six document families the changelog backfill pipeline passes between its stages, so components can interoperate deterministically and plans can be content-addressed. Contracts and serialization only — no CLI surface changes (docs/cli-schema.json is untouched), no behavior changes to existing changelog code, and everything is new files under:

• src/services/Elastic.Changelog/Backfill/ — the contracts (namespace Elastic.Changelog.Backfill), with a README.md giving a plain-language tour
• tests/Elastic.Changelog.Tests/Backfill/ — the tests

The six document families

Family	What it is
inventory	The census: which products and release-note sources exist, where they came from, and what we decided about each (classification, cutoff, adoption state, allowlist status, substitutions, filename convention, unresolved items)
overrides	Manual corrections an operator feeds into planning, each with a required reason
semantic-model	The release notes reduced to their meaning (ProductRelease shape from the epic: category family, optional precise type / breaking-change subtype, title, description, impact/action, highlight, product references, canonical PR/issue URLs, areas, source locations) plus structured triage diagnostics
plan	Exactly what we intend to create in S3 (create-bundle, create-amend, create-supplemental-bundle, skip-existing, manual-review, conflict), pinned to all inputs: source commits, input document hashes, current-state snapshot, optional enrichment snapshot, and the deployed scrubber allowlist identity (content SHA-256 and/or deployment commit)
provenance	The evidence trail: which field of which entry got which value, from which rung of the evidence ladder, at what confidence
ledger	What actually happened when a plan was applied: plan hash, input refs, created-object hashes, every attempted action with its outcome, registry state, verification result

Identity types are first-class: canonical PR URL, canonical issue URL, and synthetic file identity (stable name + checksum, the thing amend retractions match on). PR/issue references must always be full canonical URLs — bare numbers are rejected by validation.

Envelopes and schema versioning

Every persisted document is wrapped in an envelope carrying artifact (the family name) and schema_version. BackfillDocuments.Deserialize<T> checks both before parsing the payload and throws BackfillDocumentException with an actionable message on: invalid JSON, a non-object top level, a missing header field, an unknown artifact name, a different document kind than requested, or a schema version other than the one this code supports (exact match per family, all currently 1). There is no silent best-effort parsing. Deserialized documents are treated as external input: required fields are enforced by the serializer, semantic validation runs on every read and write and reports all problems at once with their location (e.g. releases[0]: entries[2]: …), and documents over a 64M-character safety bound are refused.

Canonicalization and hashing

A document's identity is SHA-256 over the UTF-8 bytes of the canonical form of its envelope, written sha256: + 64 lower-case hex. Canonical form (CanonicalJson):

• object keys sorted by ordinal comparison; duplicate keys rejected
• no insignificant whitespace
• \r\n / \r inside strings normalized to \n
• null object properties dropped (absent ≡ null); null array items kept
• array order preserved (it is meaning); dictionaries serialize as JSON objects, so key sorting makes insertion order irrelevant
• numbers kept verbatim (all contract numbers are integers); dates/timestamps are invariant ISO strings, ledger timestamps must be UTC

The pretty-printed on-disk form never affects the hash: reading a file back and re-hashing gives the same answer.

AOT

All serialization goes through the source-generated BackfillJsonContext (snake_case properties, kebab-case string enums via JsonStringEnumMemberName, nulls omitted). No reflection; the AOT publish of docs-builder is warning-free with these types included.

Test plan

All verified locally:

• Round-trip serialize → deserialize → equivalent (and hash-equal) for all six families with representative fixtures
• Determinism: same document always hashes the same; dictionary insertion order and formatting/line-ending differences do not change the hash; any semantic change does; list order remains meaningful; algorithm pinned against a known SHA-256 vector
• Version mismatch (newer and older), wrong document kind, unknown kind, missing header fields, invalid JSON, oversized input — all fail explicitly with actionable errors
• Envelope/field validation: missing required JSON fields, bare PR numbers, malformed hashes, truncated commit SHAs, set-override without value, duplicate override ids, amend without parent, non-UTC ledger timestamps, and more
• dotnet format clean · ./build.sh build --skip-dirty-check green · dotnet publish -c Release (AOT) zero trim/AOT warnings · tests/Elastic.Changelog.Tests: 873 passed, 0 failed (67 new) · docs/cli-schema.json untouched

Part of elastic/docs-eng-team#672

Note: issue #672 also resolves where checked-in inventories, overrides, plans, provenance, and ledgers live long term; that placement discussion can follow up on this PR without changing these contracts.

[Mpdreamz]

Request changes: this overcomplicates a one-off migration

The goal is narrow: get historical release-note bundles into S3 so the CDN has what already exists on elastic.co/docs/release-notes. This PR lands a six-family, versioned, content-addressed artifact platform (inventory / overrides / semantic-model / plan / provenance / ledger) before anything reads published notes or writes S3.

That is a lot of permanent machinery for what should be a temporary one-off CLI, removed once the upload is done — something like docs-builder changelog migrate-from-web.

What migrate-from-web needs to do (functionally)

1. Know which products/versions are in scope (flags + a small config/cutoff list is enough).
2. Fetch the published release-note content for those scopes (the live site, or the Markdown that backs it).
3. Turn that into the existing bundle YAML shape we already upload/scrub/serve (Bundle / inline entries) — not a parallel “semantic model.”
4. Upload to bundle/{product}/ with create-only semantics (never overwrite).
5. Print what was created / skipped / failed. Done. Delete the command afterward.

What we do not need from this PR

• Enveloped, schema-versioned, hashed cross-stage “documents” (and please avoid that overloaded term next to Elasticsearch).
• Overrides / plan / provenance / ledger families for a throwaway tool.
• A custom CanonicalJson + content-addressing layer so stages can pin each other.

Paper trail

The durable output is the bundles on S3. If you want an audit trail, one run report (scope, source→bundle key, created/skipped/failed + reason, object ETag/hash) attached to the issue/PR is enough — not a versioned in-repo artifact platform.

Please collapse or close this foundation until a vertical slice of migrate-from-web proves any of this surface is still justified.