Skip to content

feat(python): add preserveUnknownFields option - #146

Open
2ynn wants to merge 1 commit into
qynn/fix/python-model-dump-filtersfrom
qynn/feat/python-preserve-unknown-fields
Open

2ynn wants to merge 1 commit into
qynn/fix/python-model-dump-filtersfrom
qynn/feat/python-preserve-unknown-fields

Conversation

@2ynn

@2ynn 2ynn commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Why

Pydantic drops fields a model does not declare. Python SDKs therefore lose undeclared fields in nested JSON request and response bodies: they are not sent to the API, and they disappear from response models, so a response cannot be sent back unchanged.

What changed

Add python.preserveUnknownFields (default false). When enabled:

  • generated models and error bodies allow extra fields, including schemas with additionalProperties: false; explicitly typed additional properties still validate;
  • each model serializer reads every declared field from the key Pydantic emitted (alias or field name, including the alias-keyed output a nested model is serialized from again) and removes it, then copies the remaining keys, the unknown fields, back into the output. With the option disabled the serializer reads fields as before;
  • marshal_json keeps nested nulls and excludes only a top-level None; optional fields left unset are still omitted;
  • UNSET ignores extra fields;
  • generated test assertions compare only the fields their examples declare, through a drop_unexpected_fields test helper, because mock responses can carry more. Declared additional properties keep their exact key set, and their values are compared the same way.

Known limitations

  • When a payload contains both a field's API name and an undeclared key equal to its Python name (displayName and display_name), API-name dumps (model_dump(by_alias=True), the request body) keep both keys, but Python-name dumps (the default model_dump()) show the undeclared value in place of the declared one. Plain Pydantic behaves the same with extra="allow" and populate_by_name=True.
  • Additional properties assigned under a key the model already emits replace the declared value in the output.
  • Flattened operation arguments accept only declared fields, so undeclared top-level request fields need flattenRequests: false; nested objects keep theirs. The option description states this.
  • Unknown fields are carried in JSON bodies only; form, multipart, query and header serialization use declared fields.
  • As in fix(python): honour include/exclude filters in model dumps #149, a dump that combines exclude_none=True with include, exclude, exclude_unset or exclude_defaults can still write excluded required fields back as None.

types/basemodel.py becomes a template (basemodel.py.stmpl), so the review SDK copy now goes through the Python formatter. Apart from that formatting, the review SDK gains only preserveUnknownFields: false in gen.yaml.

Test coverage

  • New unknown-fields uber fragment: an echo operation whose body nests objects, arrays, maps, discriminated and plain unions, allOf, open/typed/closed additional properties (including additional properties typed as an object schema), empty objects, open enums and nullable fields, plus a flattened variant and an error response.
  • New test ID unknown-fields-json-round-trip, skipped for other targets. The tertiary TypeScript overlay removes the echo operation because its inline success and error bodies get the same generated name in that configuration.
  • primary (option disabled): unknown fields are still dropped.
  • secondary (enabled, flattenRequests: false): request/response round trips for dicts and models, response-only and error-body extras, dump filters, typed additional-properties validation, top-level None handling.
  • quaternary (enabled, flattenRequests: true): nested unknown fields through flattened arguments, sync and async; drop_unexpected_fields ignores unknown fields inside typed additional-property values but still reports extra map entries and changed values.

Stack

  1. fix(python): serialize unrecognized open enum values #145: unrecognized enum serialization
  2. fix(python): honour include/exclude filters in model dumps #149: honour include/exclude filters in model dumps
  3. This PR: preserveUnknownFields
  4. feat(python): add modelSerializationMethods option #147: modelSerializationMethods

Testing

  • TARGET=primary make test-pythonv2 (option disabled): 723 passed.
  • TARGET=secondary make test-pythonv2: 330 passed, 1 skipped.
  • TARGET=quaternary make test-pythonv2: 256 passed, 1 skipped. The typed additional-properties helper test fails with the previous drop_unexpected_fields.
  • TARGET=review make build-pythonv2: integrity validation passes.
  • prettier passes; make check-template-pythonv2 was run on the top of the stack (feat(python): add modelSerializationMethods option #147), which contains these templates.
  • git diff --check passes.
  • TARGET=tertiary make test-pythonv2 (option disabled): 214 passed, 1 skipped.

Public-safety check

  • This change contains no credentials, customer documents, private repository URLs, private filesystem paths, or unredacted private logs.
  • Title, body, comments, and commit messages name no customers or customer-derived identifiers, private paths or trackers, or workflow provenance, and are understandable without private context (.claude/skills/public-repo-communication/SKILL.md).
  • Generated fixtures and review SDK changes are public-safe.
  • I reviewed git diff --check.

@github-actions github-actions Bot added csharp Trigger (24) snapshot tests for csharp go Trigger (90) snapshot tests for go java Trigger (40) snapshot tests for java mcp-typescript Trigger (16) snapshot tests for mcp-typescript php Trigger (22) snapshot tests for php python Trigger (106) snapshot tests for python ruby Trigger (19) snapshot tests for ruby typescript Trigger (168) snapshot tests for typescript unity labels Oct 23, 2026

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 31 files

Shadow auto-approve: would not auto-approve because issues were found.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread templates/templates/pythonv2/includes/tests.ts
Comment thread templates/templates/pythonv2/model.py.stmpl
@2ynn
2ynn force-pushed the qynn/fix/python-unrecognized-enum-serialization branch from e86efdc to dce293e Compare October 23, 2026 15:55
@2ynn
2ynn force-pushed the qynn/feat/python-preserve-unknown-fields branch from a47683e to 189b725 Compare October 23, 2026 15:55
@speakeasy-generator-bot

speakeasy-generator-bot Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Generator Snapshot Testing

Snapshot tests are running.

Snapshot results: https://github.com/speakeasy-api/openapi-generation-snapshots/issues/135#issuecomment-6372136181

@2ynn
2ynn force-pushed the qynn/feat/python-preserve-unknown-fields branch from 189b725 to 56a8cb5 Compare October 23, 2026 17:19
@2ynn
2ynn changed the base branch from qynn/fix/python-unrecognized-enum-serialization to qynn/fix/python-model-dump-filters October 23, 2026 17:19
@2ynn
2ynn marked this pull request as ready for review October 23, 2026 17:37
@2ynn
2ynn requested a review from a team as a code owner October 23, 2026 17:37
@2ynn
2ynn requested review from TristanSpeakEasy and removed request for a team October 23, 2026 17:37
Pydantic drops fields a model does not declare, so undeclared fields in
nested JSON request and response bodies were lost on validation and on
serialization back to the API.

Add python.preserveUnknownFields (default false). When enabled:

- generated models and error bodies allow extra fields, including
  schemas with additionalProperties: false; explicitly typed additional
  properties still validate
- model serializers read each declared field from the key Pydantic
  emitted (alias or field name, including the alias-keyed output a
  nested model is serialized from again) and copy the remaining keys
  back out
- marshal_json keeps nested nulls and only excludes a top-level None
- UNSET ignores extra fields
- generated test assertions compare only the fields their examples
  declare, since mock responses can carry more; declared additional
  properties keep their exact key set and are compared recursively

basemodel.py becomes a template, so the review SDK copy now passes
through the Python formatter.

The option is covered by an echo fragment exercised in the primary
(disabled), secondary and quaternary variants.
@2ynn
2ynn force-pushed the qynn/fix/python-model-dump-filters branch from 9ddfe10 to 7eca0ea Compare October 23, 2026 18:05
@2ynn
2ynn force-pushed the qynn/feat/python-preserve-unknown-fields branch from 56a8cb5 to 876df5f Compare October 23, 2026 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

csharp Trigger (24) snapshot tests for csharp go Trigger (90) snapshot tests for go java Trigger (40) snapshot tests for java mcp-typescript Trigger (16) snapshot tests for mcp-typescript php Trigger (22) snapshot tests for php python Trigger (106) snapshot tests for python ruby Trigger (19) snapshot tests for ruby typescript Trigger (168) snapshot tests for typescript unity

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant