Repository navigation
feat(codegen): partition schemas and model stubs - #880
Draft
stobrien89 wants to merge 1 commit into
Draft
stobrien89 wants to merge 1 commit into
stobrien89 wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
🔵 Needs a closer look
Separate re-exports from the same partition are skipped because paths remain in visited, causing verify_models to incorrectly report exported models as missing from models.pyi.
No open issues 🎉
What changed in this PR
This PR extends #877’s lazy operation loading by partitioning generated schemas and model stubs for large clients, while preserving stable facades and single-file output for smaller clients.
Changes:
- Split schemas above 50 shapes using source-size weights, keeping recursive components together.
- Partition large model stubs and re-export declarations through
models.pyi. - Update conformance checks and integration tests to support partitioned output.
| File | Description |
|---|---|
packages/smithy-test/tests/unit/test_conformance.py |
Tests event-stream fields with partitioned stubs. |
packages/smithy-test/src/smithy_test/_private/conformance.py |
Follows stub re-exports during field validation. |
codegen/core/src/main/java/software/amazon/smithy/python/codegen/generators/SchemaGenerator.java |
Generates weighted schema partitions and lazy facades. |
codegen/core/src/main/java/software/amazon/smithy/python/codegen/generators/ModelStubGenerator.java |
Generates stub partitions with localized type references. |
codegen/core/src/it/java/software/amazon/smithy/python/codegen/test/PythonCodegenTest.java |
Checks large-model partition output. |
codegen-output/rpcv2-cbor/src/rpcv2cbor/_private/schemas_1.py |
Adds RPC v2 CBOR schemas, including recursive shapes. |
codegen-output/rpcv2-cbor/src/rpcv2cbor/_private/schemas_0.py |
Adds the first RPC v2 CBOR schema partition. |
codegen-output/rest-xml/src/restxml/_private/schemas_3.py |
Adds REST XML schemas with deferred cross-partition references. |
codegen-output/rest-xml/src/restxml/_private/schemas_1.py |
Partitions REST XML payload and header schemas. |
codegen-output/rest-xml/src/restxml/_private/schemas_0.py |
Adds the first REST XML schema partition. |
codegen-output/rest-json-1/src/restjson/_private/schemas_7.py |
Partitions streaming, query, and sparse-collection schemas. |
codegen-output/rest-json-1/src/restjson/_private/schemas_2.py |
Partitions HTTP payload, header, and label schemas. |
codegen-output/rest-json-1/src/restjson/_private/schemas_1.py |
Partitions document and event-stream schemas. |
codegen-output/rest-json-1/src/restjson/_private/models_1.pyi |
Adds a private model-stub partition. |
codegen-output/ec2-query/src/ec2query/_private/schemas_1.py |
Adds EC2 Query schemas with deferred references. |
codegen-output/ec2-query/src/ec2query/_private/schemas_0.py |
Adds the first EC2 Query schema partition. |
codegen-output/aws-query/src/awsquery/_private/schemas_1.py |
Partitions AWS Query collection and XML schemas. |
codegen-output/aws-query/src/awsquery/_private/schemas_0.py |
Adds the first AWS Query schema partition. |
codegen-output/aws-json-1-1/src/awsjson11/_private/schemas_0.py |
Adds the first AWS JSON 1.1 schema partition. |
codegen-output/aws-json-1-0/src/awsjson10/_private/schemas_0.py |
Adds the first AWS JSON 1.0 schema partition. |
Review effort: Balanced
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue #, if available:
#797
Description
Partitions generated schemas and model stubs for large clients.
This is PR 2/2, stacked on #877 and intended to be reviewed against
lazy-operation-loading. Lazy operation loading keeps unused operation definitions out of startup, while this PR keeps their schema partitions unloaded until needed.Changes
schemas.pyandmodels.pyifacades.EC2 cold-start results
Measured on x86 with five runs and one warmup. The partitioned result includes #877.
The 50-shape threshold retains nearly all of the measured memory benefit while producing about half as many schema partitions as 25 shapes.
Validation