Core spec: allow ai_context and custom_extensions on the document root

[eisber]

Summary

The document root is the only node in the core spec with no ai_context and no custom_extensions. Both keys appear on SemanticModel, Dataset, Field, Metric and Relationship; the root is closed (additionalProperties: false) with only version and semantic_model.

Since semantic_model is a list, guidance that governs every model in a document has nowhere to live. A producer emitting one model per data source must copy shared instructions into each model, and a consumer cannot tell those copies apart from genuinely model-specific instruction. See #322 for the full motivation.

This PR adds two optional keys to the document root:

Key	Type
ai_context	$ref: #/$defs/AIContext
custom_extensions	array of #/$defs/CustomExtension

Semantics: document-level context applies to every semantic model in the document; model-level ai_context adds to it and takes precedence where the two conflict.

Precedent

This is not a new concept for the project. An ontology document already carries ai_context on its root, and $refs this spec's definition:

"ai_context": { "$ref": "../core-spec/osi-schema.json#/$defs/AIContext" }

The core spec already defines document-wide AI context — the core document root was simply the only root that never consumed it. A test pins both roots to the same AIContext definition so they cannot drift.

Not a breaking change

• No new $defs — both keys reuse existing definitions.
• required is unchanged; both keys are optional.
• additionalProperties: false is unchanged, so the root stays closed. Unknown root keys, a wrong-typed ai_context, and an extension missing data are all still rejected.
• examples/tpcds_semantic_model.yaml validates unchanged, and absent keys do not serialize.

Interaction with other open work

• fix(converters): emit schema-conformant OSI documents (drop non-spec root dialects/vendors) #148 / Keep converter outputs aligned with root schema #306 remove root keys (dialects, vendors) that were never in the schema. This PR is compatible in substance and goes the other way round: it adds to the schema first and leaves the root closed. Under Keep converter outputs aligned with root schema #306's extra="forbid", declared fields still validate.
• docs: align spec yaml ai_context shape #221 fixes spec.yaml to show ai_context as a structured object (drift core-spec/spec.yaml declares ai_context: string but the JSON schema + canonical example use a structured object #141). The root entry added here uses docs: align spec yaml ai_context shape #221's shape, so it does not re-introduce the drift.
• does ai_context really nessary? Strengthen description and name to reduce reliance on ai_context #234 asks whether ai_context should exist at all. That is a broader question; this PR only makes the existing construct available at the one level where it is missing.

Related Issues

Refs #322

Checklist

Specification

• Spec changes are included in core-spec/ and follow the existing structure
• Spec changes have been discussed on the mailing list or in a linked issue — see Core spec: document root cannot carry ai_context or custom_extensions #322
• Breaking changes to the spec are clearly called out in the summary — there are none; see "Not a breaking change"

Ontology

• Ontology changes in ontology/ are consistent with spec changes — ontology/ is untouched; it already had root ai_context and this aligns core with it
• New or modified terms are defined and documented

Converters

• Converter logic in converters/ is updated to reflect spec or ontology changes — not applicable, both keys are optional and no converter emits them
• New converters include tests under the converter's test directory — not applicable

Validation

• Validation rules in validation/ are updated if the spec changed — validate.py is schema-driven and needed no change; verified positive, negative and back-compat cases against it
• New validation cases are covered by tests

Documentation

• docs/ is updated to reflect any user-facing changes
• New features or behaviors are documented with examples where appropriate — new "Document" section in spec.md with a worked example
• CONTRIBUTING.md is updated if the contribution process changed — not applicable

Examples

• examples/ are added or updated for any new spec constructs — no example changes; the worked example lives in spec.md, and tpcds_semantic_model.yaml is intentionally left unchanged to demonstrate back-compat

Tests

• All existing tests pass (pytest / CI green) — 9 -> 14 passing
• New functionality is covered by tests — 5 new tests, including one pinning the core and ontology roots to the same AIContext definition

Compliance

• ASF license headers are present on all new source files — no new files added
• No third-party dependencies are added without PMC/IPMC approval — none added

[Copilot]

Pull request overview

Adds support for document-scoped guidance and vendor metadata by allowing ai_context and custom_extensions on the core spec document root, aligning the core document root with patterns already used elsewhere (and with the ontology root).

Changes:

• Extend core-spec/osi-schema.json document root with optional ai_context and custom_extensions (root remains closed).
• Update the Python reference model and tests to parse/serialize ai_context / custom_extensions at the document level.
• Update spec/docs to document the new document-level fields and their precedence semantics.

Show a summary per file

File	Description
python/tests/test_models.py	Adds regression tests asserting schema/root parity and round-trip serialization for document-level ai_context and custom_extensions.
python/src/ossie/models.py	Adds ai_context and custom_extensions to the OSIDocument Pydantic model.
docs/index.md	Updates docs text to include document-level AI context.
core-spec/spec.yaml	Documents root-level ai_context / custom_extensions in the YAML spec summary.
core-spec/spec.md	Adds a new “Document” section with a worked example and precedence semantics.
core-spec/osi-schema.json	Adds the new optional root properties with descriptions; keeps additionalProperties: false.

Review details

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 6/6 changed files
• Comments generated: 2
• Review effort level: Lite

[eisber]

Confirmed — the diagnosis is right. Reproduced against this branch:

emitted root keys: ['version', 'dialects', 'vendors', 'semantic_model']
schema errors: ["Additional properties are not allowed ('dialects', 'vendors' were unexpected)"]

Two reasons I'm not making that change here, though:

1. It's pre-existing, not introduced by this PR. dialects/vendors are already on OSIDocument in main; this PR only adds ai_context and custom_extensions alongside them.

2. A removal limited to models.py would break converters. Root vendors is still emitted by the Honeydew converter (converters/honeydew/src/honeydew_osi/converter.py:605-611) and asserted by its tests (test_vendors_roundtrip). Dropping the model fields without the converter side would just relocate the failure.

That work is already in flight in two open PRs — #148 (dbt side + OSIDocument) and #306 (dbt + Honeydew + rejecting root extras). #306 in particular covers the converter half. Issue #322 calls this out explicitly: this PR "goes the other way round — it adds to the schema first and keeps the root closed", deliberately compatible with #148/#306.

So I'd rather let those land the removal than half-do it here and conflict with both. Nothing in this PR makes the situation worse: the root stays additionalProperties: false, and the two optional keys it adds are the only additions.

_{Side note for whoever picks up #148/#306: OSIVendor has no HONEYDEW member, so OSIDocument.model_validate() currently rejects the Honeydew converter's own output. Schema-side $defs/Vendor is free-form, so custom_extensions is unaffected — it's purely the root vendors field, which those PRs remove anyway.}

[eisber]

Good catch on the inconsistency — but I've fixed it in the opposite direction, because the string form is the side that's already been agreed to be wrong.

That contradiction is tracked in #141: spec.yaml says ai_context: string at every level, while osi-schema.json ($defs/AIContext) and the canonical examples/tpcds_semantic_model.yaml both use the structured object. The resolution direction there is to tighten spec.yaml up to the schema, not to push the schema's structured form down to string. Declaring the root as string here would move away from that and remove the only structured example in the file.

While checking this I did find a real problem with the block — one that's only visible with more of the file in view. The new keys had landed inside the # Enumerations section (after the --- on line 27, among dialects/datatypes/vendor_name, which are enum definitions, not root keys). Parsed, that section read:

['dialects', 'datatypes', 'vendor_name', 'ai_context', 'custom_extensions', 'semantic_model']

Fixed by giving the root keys their own --- section with a # Document root header, matching how every other section in this file is delimited. It now parses as:

0 -> ['version']
1 -> ['dialects', 'datatypes', 'vendor_name']
2 -> ['ai_context', 'custom_extensions', 'semantic_model']
3 -> ['datasets'] ...

which also pulls semantic_model out of the enum section, where it had been sitting on main.

I did take your "point readers to the authoritative definition" suggestion: the comment now states that the structured form is the one shown and refers to AIContext in osi-schema.json as authoritative for both forms. Aligning the four remaining model-level ai_context: string sites is #141's scope, so I've left those untouched here.