Fix Python schema generation for fields named int or bool - #64
Conversation
crossplane project build generates broken Python models when an XRD has
a property literally named int or bool. The generated models reference
undefined type aliases int_aliased and bool_aliased, which makes the
models unimportable:
PydanticUserError: `ObjectMeta` is not fully defined; you should
define `int_aliased`, then call `ObjectMeta.model_rebuild()`.
The undefined aliases are emitted by the pinned code generator image
docker.io/koxudaxi/datamodel-code-generator:0.31.2. The CLI worked
around this with fixAliasedTypesInFile, which text-replaced the broken
aliases, but it only ran in the OpenAPI generation path - not the
XRD/CRD path that project build uses - so XRD-derived models and the
shared meta/v1.py kept the broken references.
The underlying code generator bug is fixed upstream in 0.54.0, which
sanitizes builtin-conflicting field names by appending a trailing
underscore and preserving the wire name via a Pydantic alias. Fields
named int now generate as `int_: int | None = Field(None, alias='int')`.
This commit bumps the pinned image to 0.59.0, which fixes the broken
output at its source. With the fix in place fixAliasedTypesInFile no
longer matches anything, so this commit removes it along with the
postProcessFile wrapper, leaving both generation paths to call
adjustImportsInFile directly.
Fixes crossplane#63.
Signed-off-by: Nic Cope <nicc@rk0n.org>
|
Here are the full diffs (vs the pre-regeneration commit), for a provider CRD schema (
|
📝 WalkthroughWalkthroughThis PR upgrades the Python schema code generator Docker image to version 0.59.0 and refactors the post-processing pipeline to remove intermediate helper functions that are now handled upstream by the newer image version. ChangesPython code generator upgrade and pipeline simplification
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Possibly related issues
🚥 Pre-merge checks | ✅ 6✅ Passed checks (6 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@internal/schemas/generator/python.go`:
- Around line 577-578: The wrapped error from adjustImportsInFile drops the file
context making failure messages unhelpful; update the errors.Wrapf call in the
caller that invokes adjustImportsInFile (the block containing if err :=
adjustImportsInFile(fs, destPath); err != nil) to include destPath and a short
user-facing hint (e.g., "unable to update imports for generated file %s; check
file permissions or import paths and re-run generation") so the error becomes
errors.Wrapf(err, "unable to update imports for generated file %s; check file
permissions or import paths and re-run generation", destPath).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 195be654-f908-4931-9b7a-e23cc3fc7267
📒 Files selected for processing (1)
internal/schemas/generator/python.go
| if err := adjustImportsInFile(fs, destPath); err != nil { | ||
| return errors.Wrapf(err, "adjusting imports") |
There was a problem hiding this comment.
Restore actionable context in the wrapped import-adjustment error.
At Line 578, the wrap message dropped file context, which makes failures hard to act on during generation. Can we include destPath and a brief user-facing hint?
💡 Proposed fix
- if err := adjustImportsInFile(fs, destPath); err != nil {
- return errors.Wrapf(err, "adjusting imports")
+ if err := adjustImportsInFile(fs, destPath); err != nil {
+ return errors.Wrapf(err, "cannot adjust generated python imports for %q; verify generated model paths and try again", destPath)
}As per coding guidelines, "CRITICAL: Ensure all error messages are meaningful to end users, not just developers - avoid technical jargon, include context about what the user was trying to do, and suggest next steps when possible."
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if err := adjustImportsInFile(fs, destPath); err != nil { | |
| return errors.Wrapf(err, "adjusting imports") | |
| if err := adjustImportsInFile(fs, destPath); err != nil { | |
| return errors.Wrapf(err, "cannot adjust generated python imports for %q; verify generated model paths and try again", destPath) | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@internal/schemas/generator/python.go` around lines 577 - 578, The wrapped
error from adjustImportsInFile drops the file context making failure messages
unhelpful; update the errors.Wrapf call in the caller that invokes
adjustImportsInFile (the block containing if err := adjustImportsInFile(fs,
destPath); err != nil) to include destPath and a short user-facing hint (e.g.,
"unable to update imports for generated file %s; check file permissions or
import paths and re-run generation") so the error becomes errors.Wrapf(err,
"unable to update imports for generated file %s; check file permissions or
import paths and re-run generation", destPath).
|
It looks like datamodel-code-generator changed how it emits a field's default value. Before: providerConfigRef: Optional[ProviderConfigRef] = Field(
default_factory=lambda: ProviderConfigRef.model_validate(
{'kind': 'ClusterProviderConfig', 'name': 'default'}
)
)After: providerConfigRef: ProviderConfigRef | None = Field(
{'kind': 'ClusterProviderConfig', 'name': 'default'}, validate_default=True
)function-sdk-python serializes composed resources with
So every composed upbound (GCP/AWS) resource now has an explicit |
@negz Will functions setting the |
adamwg
left a comment
There was a problem hiding this comment.
LGTM. From the diffs, it looks to me like this shouldn't be a breaking change for any existing Python functions using the schemas.
Probably best to wait until we have some e2e tests in place to help validate changes, but I wonder if we could get renovate to automatically bump versions of the third-party images we depend on for things like this.
@adamwg I don't think it's a breaking change but there is a subtle behavior change around defaults in there. See crossplane/function-sdk-python#207 for an analysis. |
|
I'll merge this, but I've added a label to try remind us to not the behavior change in the release notes. I think it'll work best if we also bump the functions to use crossplane/function-sdk-python#208. |
crossplane 2.4.0 Created-by: HarmonybrewBot Commit-by: HarmonybrewBot Merged-by: HarmonybrewBot Description: Created by `brew bump` --- Created with `brew bump-formula-pr`.<details> <summary>release notes</summary> <pre>The `v2.4.0` release is the first Crossplane CLI release that does not correspond to a Crossplane core release. It is compatible with the currently supported minor versions of Crossplane core: v2.3, v2.2, v2.1, and v1.20. It includes a number of incremental features and improvements along with many bug fixes. ## 🚨 Installation Note Since this release does not correspond to a Crossplane core release, it will not be uploaded to [releases.crossplane.io](https://releases.crossplane.io). Its artifacts are available only from [cli.crossplane.io](https://cli.crossplane.io). The `install.sh` script will install it from the correct location. ## 🎉 Highlights * **New Commands** * `crossplane xr generate` converts a claim to a composite resource, as Crossplane would do internally when a claim is applied to a cluster. #13 * `crossplane xr patch` applies defaults from an XRD to an input XR, as the apiserver would do when an XR is applied to a cluster. #61 * `crossplane xpkg get-crds` downloads package dependencies based on a `crossplane.yaml` file or package resource and writes their CRDs to files. #26 * **Command Updates** * `crossplane project init` now allows a repository to be specified. #105 * `crossplane resource validate` now has YAML and JSON output formats. #66 * `crossplane xpkg push` now allows OCI manifest annotations to be specified with the `--oci-annotation` flag. #11 * `crossplane composition render` now supports repeating the `--required-resources` and `--extra-resources` flags. #107 * Commands that interact with a cluster (e.g., `crossplane resource trace` and `crossplane xpkg install`) now support impersonation with the `--as`, `--as-group`, and `--as-uid` flags. #114 * **Developer Experience Updates** * Projects can now include pre-built functions specified in the `crossplane-project.yaml` file, allowing functions to be built using any language, tool, or framework. #24 * Projects can now specify a subset of languages for which schemas should be generated in the `crossplane-project.yaml` file. #24 ## What's Changed * Update dependencies and renovate for v2.3.0 release (main) by @adamwg in crossplane/cli#19 * ci: Remove image promotion steps by @adamwg in crossplane/cli#22 * build: Use the shared crossplane cachix cache in nix.sh by @adamwg in crossplane/cli#17 * Add the LICENSE file by @adamwg in crossplane/cli#27 * chore(deps): update module github.com/containerd/containerd to v1.7.32 [security] (main) by @crossplane-renovate[bot] in crossplane/cli#34 * chore(deps): update module github.com/sigstore/cosign/v2 to v2.6.2 [security] (main) by @crossplane-renovate[bot] in crossplane/cli#35 * chore(deps): update module golang.org/x/crypto to v0.52.0 [security] (main) by @crossplane-renovate[bot] in crossplane/cli#36 * Improve generation of reference docs by @adamwg in crossplane/cli#30 * chore(deps): update module golang.org/x/net to v0.55.0 [security] (main) by @crossplane-renovate[bot] in crossplane/cli#37 * chore(deps): update actions/create-github-app-token digest to fee1f7d (main) by @crossplane-renovate[bot] in crossplane/cli#52 * Update CONTRIBUTING.md and add install instructions to README.md by @adamwg in crossplane/cli#49 * Add the install.sh script and upload it to S3 by @adamwg in crossplane/cli#54 * docs: Enclose the CLI version number in backticks by @adamwg in crossplane/cli#55 * fix(docs): compile the bqRE once at package level by @tampakrap in crossplane/cli#57 * chore(deps): update actions/stale digest to eb5cf3a (main) by @crossplane-renovate[bot] in crossplane/cli#58 * feat(xr): Convert a Claim to XR via `crossplane xr generate` by @tampakrap in crossplane/cli#13 * chore(deps): pin dependencies (main) by @crossplane-renovate[bot] in crossplane/cli#51 * chore(deps): update cachix/install-nix-action digest to 8aa0397 (main) by @crossplane-renovate[bot] in crossplane/cli#60 * fix(deps): update module github.com/go-git/go-git/v5 to v5.19.1 [security] (main) by @crossplane-renovate[bot] in crossplane/cli#40 * render: Sync render.proto with c/c and wire XRD through by @jcogilvie in crossplane/cli#74 * Fix Python schema generation for fields named int or bool by @negz in crossplane/cli#64 * chore(deps): update actions/checkout digest to df4cb1c (main) by @crossplane-renovate[bot] in crossplane/cli#76 * chore(deps): update codecov/codecov-action digest to 75cd116 (main) by @crossplane-renovate[bot] in crossplane/cli#77 * chore(deps): update github/codeql-action digest to 8aad20d (main) by @crossplane-renovate[bot] in crossplane/cli#80 * chore(deps): update mheap/require-checklist-action digest to 9c8100a (main) by @crossplane-renovate[bot] in crossplane/cli#81 * Support pre-built function runtimes and per-language schema generation by @negz in crossplane/cli#24 * render: Clean up unused code and duplicate consts by @adamwg in crossplane/cli#83 * render: Replace condition timestamps and sort resources to stabilize output by @adamwg in crossplane/cli#82 * feat(render): capture stderr and handle pipeline-fatal exit code in both engines by @jcogilvie in crossplane/cli#91 * chore(deps): update actions/checkout action to v6.0.3 (main) by @crossplane-renovate[bot] in crossplane/cli#84 * chore(deps): update vale-cli/vale-action action to v2.1.2 (main) by @crossplane-renovate[bot] in crossplane/cli#85 * feat(xr): Introduce `xr patch --xrd` by @tampakrap in crossplane/cli#61 * Update the CLI docs path by @adamwg in crossplane/cli#99 * fix(xr): add missing import, caused by merge of outdated base tree by @tampakrap in crossplane/cli#103 * ci: Update the CLI reference documentation on every merge by @adamwg in crossplane/cli#106 * feat(validate): structured results and --output json|yaml flag by @jcogilvie in crossplane/cli#66 * function: Add missing dependencies to the Go function template by @adamwg in crossplane/cli#109 * xrd: Opportunistically infer integer types when generating from an XR by @adamwg in crossplane/cli#108 * feat: add xpkg get-crds subcommand by @fernandezcuesta in crossplane/cli#26 * Drop the scale subresource before building OpenAPI for schema generation by @negz in crossplane/cli#119 * ci: Re-use exising docs PRs and sign off on commits by @adamwg in crossplane/cli#120 * ci: Make cli-docs-bot the author of docs update commits by @adamwg in crossplane/cli#121 * xrd: Handle v2 XRDs in `crossplane xrd convert` by @adamwg in crossplane/cli#122 * fix(render): do not overwrite function docker network if set, start crossplane-container in same network by @nkzk in crossplane/cli#65 * chore(deps): update codecov/codecov-action digest to 0fb7174 (main) by @crossplane-renovate[bot] in crossplane/cli#101 * chore(deps): update korthout/backport-action action to v4.5.2 (main) by @crossplane-renovate[bot] in crossplane/cli#102 * Expose host-native CLI binary as the default flake package by @negz in crossplane/cli#126 * chore(deps): pin dependencies (main) by @crossplane-renovate[bot] in crossplane/cli#128 * chore(deps): update renovatebot/github-action action to v46.1.15 (main) by @crossplane-renovate[bot] in crossplane/cli#129 * Decompress function runtime tarballs once when loading by @negz in crossplane/cli#127 * fix: loaded XRD must honor the composite schema by @fernandezcuesta in crossplane/cli#123 * schemas: Prime oapi-codegen's global state to make capitalization consistent by @adamwg in crossplane/cli#130 * Add repository option to `crossplane project init` by @bobh66 in crossplane/cli#105 * feat(xpkg): add --annotation flag to xpkg build and xpkg push by @chaitanyapantheor in crossplane/cli#11 * fix(render): support repeating --required-resources and --extra-resources by @chaitanyapantheor in crossplane/cli#107 * fix(deps): update module github.com/alecthomas/kong to v1.15.0 (main) by @crossplane-renovate[bot] in crossplane/cli#133 * chore(deps): update module github.com/containerd/containerd to v1.7.33 [security] (main) by @crossplane-renovate[bot] in crossplane/cli#132 * fix(deps): update module github.com/oapi-codegen/oapi-codegen/v2 to v2.7.1 (main) by @crossplane-renovate[bot] in crossplane/cli#88 * fix(deps): update module github.com/kubernetes-sigs/kro to v0.9.2 (main) by @crossplane-renovate[bot] in crossplane/cli#87 * fix(deps): update module github.com/google/go-containerregistry to v0.21.7 (main) by @crossplane-renovate[bot] in https://github.com/crossplane/ See merge request: Harmonybrew/homebrew-core!12309
Description of your changes
Fixes #63
crossplane project buildgenerates broken Python models when an XRD has a property literally namedintorbool(for example when modelling DRA'sDeviceAttribute, whose wire fields areint/bool/string/version). The generated models reference undefined type aliasesint_aliasedandbool_aliased, which makes them unimportable:The undefined aliases are emitted by the pinned code generator image
docker.io/koxudaxi/datamodel-code-generator:0.31.2. The CLI worked around this withfixAliasedTypesInFile, which text-replaced the broken aliases, but it only ran in the OpenAPI generation path — not the XRD/CRD path thatproject builduses — so XRD-derived models and the sharedmeta/v1.pykept the broken references.The underlying code generator bug is fixed upstream in koxudaxi/datamodel-code-generator#2968, first released in
0.54.0, which sanitizes builtin-conflicting field names by appending a trailing underscore and preserving the wire name via a Pydantic alias. A field namedintnow generates asint_: int | None = Field(None, alias='int').This PR bumps the pinned image to
0.59.0, fixing the broken output at its source. With the fix in placefixAliasedTypesInFileno longer matches anything, so it is removed along with thepostProcessFilewrapper, leaving both generation paths to calladjustImportsInFiledirectly.I have:
./nix.sh flake checkto ensure this PR is ready for review.Added or updated unit tests.The Python generator's image-backed output is not currently covered by automated tests; see the draft note above.Linked a PR or a docs tracking issue to document this change.Addedbackport release-x.ylabels to auto-backport this PR.Need help with this checklist? See the cheat sheet.