Commit 467e2ee
LAV-3023: Match managed Iceberg column COMMENT lifecycle (#3432)
* LAV-3023: preserve managed Iceberg column comments across DDL lifecycle
Capture real Snowflake snapshots for CREATE and ALTER column comments, including
empty and quoted values, multi-column MODIFY, and missing-column errors.
Keep inline comments in the metadata store while stripping them from heap DDL.
Parse MODIFY comment actions and parenthesized column lists, and render managed
Iceberg GET_DDL with its column comments and Iceberg properties.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
## Test matrix
CREATE unset column × metadata/GET_DDL -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_column_comment_lifecycle
CREATE inline comment × metadata/GET_DDL -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_column_comment_lifecycle
ALTER set × metadata/GET_DDL -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_column_comment_lifecycle
MODIFY replace quoted × metadata/GET_DDL -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_column_comment_lifecycle
ALTER empty × metadata/GET_DDL -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_column_comment_lifecycle
ALTER/MODIFY unset × metadata/GET_DDL -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_column_comment_lifecycle
MODIFY parenthesized multi set × metadata -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_column_comment_lifecycle
MODIFY parenthesized multi unset × metadata -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_column_comment_lifecycle
ALTER set missing column × error -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_column_comment_missing_column
ALTER unset missing column × error -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_column_comment_missing_column
ALTER comment × metadata version -> tests/queries/iceberg/test_iceberg_tables.py::test_managed_iceberg_column_comment_does_not_publish
## Deviations
The hint named the CREATE serializer and ALTER rewrite. The merged main branch
also had a partial Iceberg GET_DDL renderer; I extended it to match the new
Cloud snapshot and updated the parser fork for documented MODIFY forms.
* LAV-3023: parse every managed Iceberg multi-column comment form
Recognize UNSET COMMENT after a comma in column-action lists and accept bare
column-name UNSET COMMENT continuations in the fork parser. Capture Cloud
metadata and GET_DDL for ALTER and MODIFY, with and without COLUMN, in
parenthesized and unparenthesized lists.
Swept both parser layers for the multi-column continuation pattern; also
covered quoted identifiers, dollar-quoted values, embedded keyword strings,
and SQL comments. Existing single-column, missing-column, and metadata-version
paths remain green.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
## Test matrix
CREATE inline/unset metadata and GET_DDL -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_column_comment_lifecycle
Single-column set/replace/empty/unset metadata and GET_DDL -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_column_comment_lifecycle
Single-column missing set/unset error -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_column_comment_missing_column
Comment-only ALTER metadata version -> tests/queries/iceberg/test_iceberg_tables.py::test_managed_iceberg_column_comment_does_not_publish
MODIFY unparenthesized COLUMN set -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_multi_column_comments
MODIFY unparenthesized COLUMN unset -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_multi_column_comments
MODIFY unparenthesized bare set -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_multi_column_comments
MODIFY unparenthesized bare unset -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_multi_column_comments
MODIFY parenthesized COLUMN set -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_multi_column_comments
MODIFY parenthesized COLUMN unset -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_multi_column_comments
MODIFY parenthesized bare set -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_multi_column_comments
MODIFY parenthesized bare unset -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_multi_column_comments
ALTER unparenthesized COLUMN set -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_multi_column_comments
ALTER unparenthesized COLUMN unset -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_multi_column_comments
ALTER unparenthesized bare set -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_multi_column_comments
ALTER unparenthesized bare unset -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_multi_column_comments
ALTER parenthesized COLUMN set -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_multi_column_comments
ALTER parenthesized COLUMN unset -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_multi_column_comments
ALTER parenthesized bare set -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_multi_column_comments
ALTER parenthesized bare unset -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_multi_column_comments
Quoted identifiers, single/dollar strings, SQL comments set -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_multi_column_comments
Quoted identifiers, SQL comments unset -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_multi_column_comments
E-quoted comment value -> uncovered: Snowflake parser's comment literal grammar accepts single and dollar strings, not E-prefixed strings
Tagged-dollar comment value -> uncovered: documented Snowflake comment literal form uses $$ delimiters
* LAV-3023: investigator: final checks passed; resume reviewer after wall-clock cap
Attempt 1 exhausted wall clock immediately after the second .spiral/check.sh completed successfully. Commit 35a8c03db addresses the reviewer gap in both parser layers and adds 18 Cloud snapshot cells for ALTER/MODIFY multi-column COMMENT and UNSET COMMENT forms. The transcript records Cloud and emulator passes for the expanded test and a pass for test_managed_iceberg_column_comment_does_not_publish. Continue with reviewer/PR gate; inspect any fresh feedback before changing code.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-3023: preserve ALTER option rewrites beside column comments
Exclude the statement's opening ALTER from column-action detection. It made
ordinary ALTER TABLE/VIEW SET and UNSET bypass the option rewrite and caused
48 compat failures in CI. The parser now recognizes ALTER/MODIFY only after
the table target, including optional COLUMN and comma continuations.
Swept the column-action check's input forms for the same false positive:
ALTER TABLE and ALTER VIEW SET/UNSET, IF EXISTS, and data metric options now
pass; the managed Iceberg multi-column branch remains scoped to its column
operations. Existing snapshot tests witnessed the regression, so no test or
snapshot was changed. The Iceberg S3 tests skip in this worker's default
compat environment; the earlier branch round recorded their Cloud and
emulator passes with the backend available.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
## Test matrix
ALTER TABLE SET COMMENT, single/dollar strings -> tests/queries/test_comments.py::test_alter_table_set_comment; test_alter_table_set_comment_dollar_quoted
ALTER TABLE UNSET COMMENT -> tests/queries/test_comments.py::test_alter_table_set_comment
ALTER VIEW SET/UNSET COMMENT -> tests/queries/test_comments.py::test_alter_view_set_comment
ALTER TABLE [IF EXISTS] SET/UNSET metadata option, missing table and valid table -> tests/queries/ddl/test_alter_table.py::test_alter_table_set_metadata_option_missing_table; test_alter_table_set_unset_metadata_options_noop
ALTER TABLE SET/UNSET boolean/default collation/DCM property -> tests/queries/ddl/test_alter_table_properties.py::test_alter_table_boolean_properties; test_alter_table_iceberg_default_ddl_collation; test_alter_table_unset_dcm_project
ALTER TABLE SET data metric option -> tests/queries/functions/test_data_metric_functions.py::test_data_metric_association_options
CREATE Iceberg inline/unset column comment, metadata and GET_DDL -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_column_comment_lifecycle
ALTER/MODIFY Iceberg single-column set/replace/empty/unset, metadata and GET_DDL -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_column_comment_lifecycle
ALTER/MODIFY Iceberg multi-column set/unset, optional COLUMN, parentheses, metadata and GET_DDL -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_multi_column_comments
ALTER Iceberg missing column set/unset error -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_column_comment_missing_column
ALTER Iceberg comment-only metadata version -> tests/queries/iceberg/test_iceberg_tables.py::test_managed_iceberg_column_comment_does_not_publish
* LAV-3023: support managed Iceberg tables without BASE_LOCATION
Allow a managed CREATE with an external volume to omit BASE_LOCATION. Generate
an internal storage path, retain whether the user supplied the location, and
render BASE_LOCATION in GET_DDL only for an explicit value. Remove the three
emulator-only location clauses from the check compat test so both targets run
the Cloud-captured CREATE statements; its snapshot is unchanged.
The S3-backed replay reached CREATE but this worker's PostgreSQL process used
localhost:4566 with no LocalStack service, so it failed to publish metadata.
The normal compat invocation skips S3 tests here. make check passed, including
vendored parser tests; CI's LocalStack lane must verify the S3 path.
test_edit_waiver: tests/queries/iceberg/test_iceberg_check.py
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
## Test matrix
Managed CREATE with omitted BASE_LOCATION, valid CHECK, GET_DDL -> tests/queries/iceberg/test_iceberg_check.py::test_managed_iceberg_check_create (S3 replay unavailable locally)
Managed CREATE with omitted BASE_LOCATION, invalid CHECK -> tests/queries/iceberg/test_iceberg_check.py::test_managed_iceberg_invalid_check_is_atomic (S3 replay unavailable locally)
Managed CREATE with omitted BASE_LOCATION, catalog case variants -> tests/queries/iceberg/test_iceberg_check.py::test_managed_iceberg_check_catalog_case (S3 replay unavailable locally)
Managed CREATE with explicit BASE_LOCATION, GET_DDL -> tests/queries/iceberg/test_iceberg_interop.py::test_managed_iceberg_column_comment_lifecycle
Managed CTAS with omitted BASE_LOCATION -> uncovered: S3 replay unavailable on this worker; CI's LocalStack lane exercises managed CTAS with explicit locations
Parser missing location and external volume -> vendor/sqlparser/tests/sqlparser_snowflake.rs::test_snowflake_create_iceberg_table_without_location
## Deviations
The hints focused on the CREATE serializer and ALTER rewrite. The parser fork also rejected omitted BASE_LOCATION before the rewrite; I narrowed that guard for managed tables with an external volume.
* LAV-3023: use identical projection policy CREATE SQL on both targets
Remove the emulator-only BASE_LOCATION from the projection policy CREATE
helper. The emulator now supports omitted locations, so GET_DDL can match
the existing Cloud snapshot without suppressing user-supplied locations.
The snapshot remains unchanged.
Targeted compat replay skipped because this worker has no S3 backend.
make check and .spiral/check.sh passed.
test_edit_waiver: tests/queries/iceberg/test_iceberg_check.py tests/queries/iceberg/test_projection_policy.py
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
## Test matrix
CREATE with inline projection policies, DESC/GET_DDL/references -> tests/queries/iceberg/test_projection_policy.py::test_inline_projection_policy_spelling_and_metadata (S3 replay unavailable locally; CI exercises it)
CREATE with projection policy, INSERT/ALTER/read/errors -> tests/queries/iceberg/test_projection_policy.py::test_inline_projection_policy_enforcement_and_errors (S3 replay unavailable locally; CI exercises it)
CREATE with projection policy and CTAS, read/references -> tests/queries/iceberg/test_projection_policy.py::test_inline_projection_policy_declared_ctas (S3 replay unavailable locally; CI exercises it)
* LAV-3023: record projection policy test edit waiver for branch guard
The issue grants the projection policy test edit. The prior commit copied the combined waiver line verbatim, but the branch policy guard parses comma-separated paths and treated both space-separated paths as one name. Record the granted projection path individually so the existing test edit is recognized. No source, test, or snapshot content changes in this round.
test_edit_waiver: tests/queries/iceberg/test_projection_policy.py
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
## Test matrix
CREATE projection policy with omitted BASE_LOCATION, metadata and GET_DDL -> tests/queries/iceberg/test_projection_policy.py::test_inline_projection_policy_spelling_and_metadata (S3 replay unavailable locally; CI exercises it)
CREATE projection policy with omitted BASE_LOCATION, INSERT/ALTER/read/errors -> tests/queries/iceberg/test_projection_policy.py::test_inline_projection_policy_enforcement_and_errors (S3 replay unavailable locally; CI exercises it)
CREATE projection policy with omitted BASE_LOCATION and CTAS, read/references -> tests/queries/iceberg/test_projection_policy.py::test_inline_projection_policy_declared_ctas (S3 replay unavailable locally; CI exercises it)
---------
Co-authored-by: spiral <spiral@localhost>
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>1 parent b75c1a6 commit 467e2ee
2 files changed
Lines changed: 45 additions & 13 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3110 | 3110 | | |
3111 | 3111 | | |
3112 | 3112 | | |
3113 | | - | |
3114 | | - | |
3115 | | - | |
3116 | | - | |
| 3113 | + | |
| 3114 | + | |
3117 | 3115 | | |
3118 | 3116 | | |
3119 | 3117 | | |
| |||
3122 | 3120 | | |
3123 | 3121 | | |
3124 | 3122 | | |
| 3123 | + | |
3125 | 3124 | | |
3126 | 3125 | | |
3127 | 3126 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
11755 | 11755 | | |
11756 | 11756 | | |
11757 | 11757 | | |
| 11758 | + | |
11758 | 11759 | | |
11759 | 11760 | | |
11760 | 11761 | | |
11761 | 11762 | | |
11762 | 11763 | | |
11763 | | - | |
| 11764 | + | |
11764 | 11765 | | |
11765 | 11766 | | |
11766 | | - | |
| 11767 | + | |
| 11768 | + | |
| 11769 | + | |
| 11770 | + | |
| 11771 | + | |
11767 | 11772 | | |
11768 | 11773 | | |
11769 | 11774 | | |
| |||
12217 | 12222 | | |
12218 | 12223 | | |
12219 | 12224 | | |
12220 | | - | |
| 12225 | + | |
| 12226 | + | |
| 12227 | + | |
| 12228 | + | |
| 12229 | + | |
| 12230 | + | |
| 12231 | + | |
| 12232 | + | |
| 12233 | + | |
| 12234 | + | |
| 12235 | + | |
| 12236 | + | |
| 12237 | + | |
| 12238 | + | |
| 12239 | + | |
| 12240 | + | |
| 12241 | + | |
12221 | 12242 | | |
12222 | 12243 | | |
12223 | 12244 | | |
| |||
12248 | 12269 | | |
12249 | 12270 | | |
12250 | 12271 | | |
12251 | | - | |
12252 | | - | |
12253 | | - | |
12254 | | - | |
| 12272 | + | |
| 12273 | + | |
12255 | 12274 | | |
12256 | | - | |
12257 | | - | |
| 12275 | + | |
| 12276 | + | |
| 12277 | + | |
| 12278 | + | |
| 12279 | + | |
| 12280 | + | |
12258 | 12281 | | |
12259 | 12282 | | |
12260 | 12283 | | |
| |||
13274 | 13297 | | |
13275 | 13298 | | |
13276 | 13299 | | |
| 13300 | + | |
| 13301 | + | |
| 13302 | + | |
| 13303 | + | |
| 13304 | + | |
| 13305 | + | |
| 13306 | + | |
13277 | 13307 | | |
13278 | 13308 | | |
13279 | 13309 | | |
| 13310 | + | |
| 13311 | + | |
| 13312 | + | |
13280 | 13313 | | |
13281 | 13314 | | |
13282 | 13315 | | |
| |||
0 commit comments