Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -199,7 +199,8 @@ object IcebergWriteProtoTranslation {
* zstd (parquet-rs default 1, parquet-mr default 3). To produce files the same size as
* iceberg-java would, substitute parquet-mr's 3 when zstd is in use and the user did not set an
* explicit level. gzip defaults to 6 on both sides; snappy and lz4 have no level concept;
* compressed bytes are an accepted divergence regardless (see iceberg-writes.md).
* compressed bytes are an accepted divergence regardless (see "Accepted divergences" under
* Writes in `docs/source/user-guide/latest/iceberg.md`).
*/
private def resolveCompressionLevel(
props: Map[String, String],
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4901,7 +4901,8 @@ class CometIcebergWriteActionSuite
// What is asserted here is determinism, not parity: iceberg-java's fanout writer iterates its
// own `StructLikeMap`, so the two writers only agree where that map order and path order happen
// to coincide -- which they do for the ascending partition values the upstream test uses. See
// the fanout entry under accepted divergences in `iceberg-writes.md`.
// the fanout ordering entry in "Accepted divergences" under Writes in
// `docs/source/user-guide/latest/iceberg.md`.
//
// Eight partitions rather than the two that test uses: an unfixed writer lands in path order by
// luck 1 time in 8!, where with two it would pass half the time.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,8 +60,8 @@ import org.apache.comet.CometSparkSessionExtensions
* - The timed statement includes the driver-side Iceberg commit, which all three configurations
* pay equally.
* - The warehouse is a local temporary directory, so no object-store latency is included.
* - The two writers choose different file roll points (see the accepted divergences in
* `docs/source/user-guide/latest/iceberg-writes.md`), so the resulting file layouts are not
* - The two writers choose different file roll points (see "Accepted divergences" under Writes
* in `docs/source/user-guide/latest/iceberg.md`), so the resulting file layouts are not
* expected to match. What is checked is that every configuration writes the same rows.
* - Every group has the same row count but not the same row width, so `Rate` and `Per Row`
* compare configurations within a group, not one group with another.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,8 +60,8 @@ import org.apache.comet.CometSparkSessionExtensions
* - The timed statement includes the driver-side Iceberg commit, which all three configurations
* pay equally. It dilutes the writer's factor rather than inflating it.
* - The warehouse is a local temporary directory, so no object-store latency is included.
* - The two writers choose different file roll points (see the accepted divergences in
* `docs/source/user-guide/latest/iceberg-writes.md`), so the resulting file layouts are not
* - The two writers choose different file roll points (see "Accepted divergences" under Writes
* in `docs/source/user-guide/latest/iceberg.md`), so the resulting file layouts are not
* expected to match. Only wall clock and row counts are compared.
* - `Rate` and `Per Row` are always computed against the corpus size, so for the `DELETE` case
* they describe the rows the statement passed over, not the far smaller number it removed.
Expand Down