Add jobs testing for RTAS - #684
Conversation
| Table replaced = ops.getTable(tableName); | ||
| Assertions.assertNull( | ||
| replaced.schema().findField("datepartition"), | ||
| "Replace should have dropped the column the in-flight job was launched with"); |
There was a problem hiding this comment.
Previous table property is considered invalid and we need to validate it when RTAS occurs. RTAS can leave the table in an invalid state. I see the fix as either proactive validation, or that we don't allow RTAS until you disable retention, or we restirc the column modifications that orther features are depending on
There was a problem hiding this comment.
Let me reject users RTAS request when the retention policy contradicts with the schema.
There was a problem hiding this comment.
Added the validation.
| "Current behavior: a delete planned pre-RTAS rebases onto the replacement"); | ||
|
|
||
| // It lands as a snapshot that deletes nothing. The files it planned to remove belong to the | ||
| // pre-RTAS table and are not part of the replacement, so the filter matches nothing. |
There was a problem hiding this comment.
Its the opposite condition I am worried about where a newly RTAS'd table immediately purges newly written data. WE might be fine becuase this is implemented as a logical operation but if they drop a partition spec what was a cheap metadata op becomes an expensive query.
can we also test the removal of partitioning where otherwise the data was fine?
There was a problem hiding this comment.
It should be fine since today the retention column is also not required to be a partition column.
There was a problem hiding this comment.
Added a tests on this.
| "Replace should have moved the table off the snapshot the delete was planned on"); | ||
| verifyRowCount(ops, tableName, 2); | ||
|
|
||
| // The retention delete commits onto the staled snapshot instead of failing. The writer |
There was a problem hiding this comment.
Let me verify it again, spark delete does not seem to be a newDelete() operation.
There was a problem hiding this comment.
Turns out spark delete doesn't go through this route. Added other 3 tests. It will delete new data but it is a valid operation.
480da07 to
5ee8372
Compare
5ee8372 to
bbb89f5
Compare
2 participants
Summary
Added a test suite for running jobs on RTAS'ed tables.
Also starts to reject RTAS command when the new schema mismatch the retention config.
Changes
For all the boxes checked, please include additional details of the changes made in this pull request.
Testing Done
For all the boxes checked, include a detailed description of the testing done for the changes made in this pull request.
Additional Information
For all the boxes checked, include additional details of the changes made in this pull request.