Antalya 26.5: Forward port of export part and partition#1963
Open
zvonand wants to merge 11 commits into
Open
Conversation
…next commit) --- Original cherry-pick message follows: Merge pull request #1388 from Altinity/fp_antalya_26_1_export_part_partition Antalya 26.1 - Forward port of export part and partition # Conflicts: # src/Common/ErrorCodes.cpp # src/Common/ProfileEvents.cpp # src/Common/setThreadName.h # src/Core/ServerSettings.cpp # src/Core/Settings.cpp # src/Core/SettingsEnums.cpp # src/Core/SettingsEnums.h # src/Interpreters/ClientInfo.h # src/Parsers/ASTAlterQuery.cpp # src/Storages/IPartitionStrategy.cpp # src/Storages/IPartitionStrategy.h # src/Storages/MergeTree/IMergeTreeDataPart.cpp # src/Storages/MergeTree/MergeTreeData.cpp # src/Storages/MergeTree/MergeTreeData.h # src/Storages/MergeTree/ReplicatedMergeTreeRestartingThread.cpp # src/Storages/ObjectStorage/StorageObjectStorage.cpp # src/Storages/ObjectStorage/StorageObjectStorageCluster.cpp # src/Storages/ObjectStorage/StorageObjectStorageConfiguration.cpp # src/Storages/ObjectStorage/StorageObjectStorageConfiguration.h # src/Storages/StorageMergeTree.cpp # src/Storages/StorageReplicatedMergeTree.h # src/Storages/System/StorageSystemMerges.cpp # src/Storages/System/attachSystemTables.cpp # tests/queries/0_stateless/02221_system_zookeeper_unrestricted.reference # tests/queries/0_stateless/02221_system_zookeeper_unrestricted_like.reference # tests/queries/0_stateless/03413_experimental_settings_cannot_be_enabled_by_default.sql # tests/queries/0_stateless/03745_system_background_schedule_pool.reference # tests/queries/0_stateless/03745_system_background_schedule_pool.sql
Cherry-pick of #1388 onto antalya-26.5. Conflicts resolved by keeping both antalya-26.5 additions and source PR additions: - `ErrorCodes.cpp`: `STALE_VERSION` at 1005, `PENDING_MUTATIONS_NOT_ALLOWED` at 1006 - `ProfileEvents.cpp`: kept `UserThrottler*` and `ExportsThrottler*` events - `setThreadName.h`: kept `BLOB_KILLER_TASK`, `BLOB_COPIER_TASK`, `EXPORT_PART` - `ServerSettings.cpp`: added `enable_experimental_export_merge_tree_partition_feature` - `Settings.cpp`: added export settings and antalya-26.5 SMT/join settings - `SettingsEnums.{cpp,h}`: kept antalya-26.5 enums and added `MergeTreePartExportFileAlreadyExistsPolicy` - `ClientInfo.h`: added `BackgroundOperationType` enum - `ASTAlterQuery.cpp`: kept `execute_args` and added `to_table_function`/`partition_by_expr` - `IPartitionStrategy.{cpp,h}`: kept `const` qualifier, dropped removed path methods, added `Block` overload - `IMergeTreeDataPart.cpp`: kept `getMinMaxIndex`/`setMinMaxIndex`, added `MinMaxIndex::getBlock` - `MergeTreeData.{cpp,h}`: resolved 5 conflicts — includes, settings externs, changeSettings, `writePartLog` signature - `ReplicatedMergeTreeRestartingThread.cpp`: kept `deduplication_hashes_cache.start()` - `StorageObjectStorage.cpp`: kept `startup`/`shutdown`/`scheduleDataProcessingJob` - `StorageObjectStorageCluster.cpp`: kept `setConstraints`, added partition strategy block - `StorageObjectStorageConfiguration.{cpp,h}`: kept `schema_hash` handling in `getPathForWrite`, kept all four `has*Wildcard` methods - `StorageMergeTree.cpp`: kept both error code externs - `StorageReplicatedMergeTree.h`: added export-related includes - `StorageSystemMerges.cpp`: kept antalya-26.5 includes - `attachSystemTables.cpp`: kept both `StorageSystemWasmModules.h` and `StorageSystemExports.h` - Test reference files: kept both `deduplication_hashes` and `exports` entries
…next commit) --- Original cherry-pick message follows: Merge pull request #1402 from Altinity/26_1_export_improvements_test Improvements to partition export # Conflicts: # ci/jobs/scripts/integration_tests_configs.py # src/Core/Settings.cpp # tests/integration/test_export_replicated_mt_partition_to_object_storage/test.py
- `integration_tests_configs.py`: kept `antalya-26.5` iceberg entries (updated to `False` with new descriptions) and added the new `test_export_replicated_mt_partition_to_object_storage/` entry (bucket 1). - `Settings.cpp`: inserted the two new settings (`export_merge_tree_partition_lock_inside_the_task`, `export_merge_tree_partition_system_table_prefer_remote_information`) after `export_merge_tree_part_throw_on_pending_patch_parts`, before the `antalya-26.5` additions (`shared_merge_tree_sequential_consistency_*`, `max_bytes_*`). The `iceberg_partition_timezone` context from "theirs" was not on `antalya-26.5` and was correctly excluded. - `test.py`: took "theirs" — added `@pytest.mark.parametrize` decorator and `skip_if_remote_database_disk_enabled` call to `test_kill_export` (bucket 1).
add setting to define filename pattern for part exports
…eduled_after_scheduling Bump scheduled exports count only in case it has been scheduled
…ing_the_lock Export Partition - release the part lock when the query is cancelled
…next commit) --- Original cherry-pick message follows: Merge pull request #1618 from Altinity/export_partition_iceberg Export partition to apache iceberg # Conflicts: # src/Core/ServerSettings.cpp # src/Storages/MergeTree/MergeTreeData.cpp # src/Storages/ObjectStorage/DataLakes/IDataLakeMetadata.h # src/Storages/ObjectStorage/DataLakes/Iceberg/Constant.h # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.h # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.h # src/Storages/ObjectStorage/DataLakes/Iceberg/MetadataGenerator.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/MultipleFileWriter.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/MultipleFileWriter.h # src/Storages/ObjectStorage/DataLakes/Iceberg/Utils.cpp # src/Storages/ObjectStorage/StorageObjectStorage.cpp # src/Storages/ObjectStorage/StorageObjectStorageCluster.cpp # src/Storages/ObjectStorage/StorageObjectStorageCluster.h # tests/integration/test_export_replicated_mt_partition_to_object_storage/test.py
Resolve conflict markers left in the previous commit. Key adaptations for the forward port to antalya-26.5: - `StorageObjectStorageCluster`: drop methods that depend on `pure_storage` (prerequisite PR #1390 not present on antalya-26.5); adapt `MergeTreeData.cpp` to cast to `StorageObjectStorage` instead of `StorageObjectStorageCluster` - Drop `object_storage_list_objects_*` settings from `ServerSettings.cpp` (prerequisite PR #1405 not present on antalya-26.5) - `MultipleFileWriter`: adapt `startNewFile` and `finalize` to use `IcebergPathFromMetadata` + `IcebergPathResolver::resolve` (antalya-26.5 API) instead of `FileNamesGenerator::Result` (source-branch API) - `generateManifestFile` / `generateManifestList`: update callers in `Compaction.cpp`, `Mutations.cpp`, `IcebergMetadata.cpp` to the new signature (remove `data_file_row_counts`/`data_file_byte_counts`, use `std::vector<String>` for data file names, use `IcebergSerializedFileStats`) - `commitImportPartitionTransactionImpl`: adapt all `FileNamesGenerator` calls to antalya-26.5 API (`generateMetadataPathWithInfo`, `generateManifestEntryName`, `IcebergPathFromMetadata::serialize`) - `MetadataGenerator::generateNextMetadata`: add `is_truncate` parameter introduced by this PR
…next commit) --- Original cherry-pick message follows: Merge pull request #1713 from Altinity/export_partition_remove_lock_in_task Remove lock inside the task export partition # Conflicts: # src/Storages/MergeTree/ExportPartFromPartitionExportTask.cpp
zvonand
added
releasy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Forward port of export part and partition #1041, #1083, #1086, #1090, #1106, #1124, #1144, #1147, #1150, #1157, #1158, #1161, #1167, #1229, #1294, #1320, #1324 and #1330 (#1388 by @arthurpassos, #1402 by @arthurpassos, #1490 by @arthurpassos, #1499 by @arthurpassos, #1593 by @arthurpassos, #1618 by @arthurpassos, #1713 by @arthurpassos).
CI/CD Options
Exclude tests:
Regression jobs to run:
Combined port of 7 PR(s) (group
pr-1713). Cherry-picked from #1388, #1402, #1490, #1499, #1593, #1618, #1713.