GH-533: Add ALP (Adaptive Lossless floating-Point) encoding specification

[prtkgaur]

Add the encoding specification for ALP (encoding value 10) to Encodings.md. ALP compresses FLOAT and DOUBLE columns by converting values to integers via decimal scaling, then applying Frame of Reference encoding and bit-packing. Values that cannot be losslessly round-tripped are stored as exceptions.

• Closes [Proposal] Add ALP encoding support in parquet file format #533

See rendered preview here: https://github.com/prtkgaur/parquet-format/blob/alpEncoding/Encodings.md#adaptive-lossless-floating-point-alp--10

The spec covers:

• Page layout: 7-byte header, offset array, compressed vectors
• Vector format: AlpInfo, ForInfo, packed values, exception data
• Encoding math: two-step multiplication for cross-language consistency
• Parameter selection, exception detection, and decoding steps

Based on the paper "ALP: Adaptive Lossless floating-Point Compression" (Afroozeh and Boncz, SIGMOD 2024). Wire format matches the C++ Arrow and Java parquet-java implementations.

Rationale for this change

What changes are included in this PR?

Do these changes have PoC implementations?

[emkornfield]

Suggested change

	All multi-byte values are little-endian.
	All multi-byte values are stored in little-endian order.

[prtkgaur]

Done

[emkornfield]

Suggested change

	| 2 | log_vector_size | 1 byte | uint8 | log2(vector\_size). Must be in \[3, 15\]. Default: 10 (vector size 1024) |
	| 2 | log_vector_size | 1 byte | uint8 | log2(vector\_size). Must be in \[3, 15\]. Recommended default: 10 (vector size 1024) |

[prtkgaur]

done

[emkornfield]

we think ALP-RD fits in well here? I forget what the extension point is and why we were OK keeping this field, but not version.

[prtkgaur]

changing the code to read ALP's newer laid out bits would be difficult to apply across all implementations. but AlpRD is a completely newer implementations.

Added a note to the field description.

[emkornfield]

Suggested change

	| 2 | log_vector_size | 1 byte | uint8 | log2(vector\_size). Must be in \[3, 15\]. Default: 10 (vector size 1024) |
	| 2 | log_vector_size | 1 byte | uint8 | log2(vector\_size). Must be in the inclusive range: \[3, 15\]. Default: 10 (vector size 1024) |

[prtkgaur]

done

[emkornfield]

bit_width isn't in the header either, so it is a little strange to call ths out here?

[alamb]

I also hihglighted this as confusing when reading the spec. I recommend removing this sentence as the packed data size is covered in the "Vector Format" section below

[prtkgaur]

comment regarding packed data size has been removed.

[emkornfield]

This seems like a little bit of a strange callout since it is covered on line 457 explicitly. and log_vector_size is stored in the header?

[alamb]

I agree this is redundant -- I think leaving the note is valuable clarification

Note: The number of elements per vector and the packed data size are NOT stored
in the header, they are derived

However the other items are not

[prtkgaur]

Removed the packed data size bullet, kept the elements-per-vector clarification.

[emkornfield]

Suggested change

	The first offset equals `num_vectors * 4` (pointing just past the offset array).
	The first offset always equals `num_vectors * 4` (pointing just past the offset array).

Lets be explicitly here that we don't support padding.

[prtkgaur]

done

[emkornfield]

Suggested change

	| PackedValues | ceil(N * bit\_width / 8) | Bit-packed delta values |
	| PackedValues | ceil(`vector_size` * bit\_width / 8) | Bit-packed delta values |

[prtkgaur]

done

[emkornfield]

Suggested change

	| ExceptionValues | num\_exceptions * sizeof(T) | Original float/double values |
	| ExceptionValues | num\_exceptions * sizeof(encoded type) (float=4 and double=8) | Original float/double values |

[prtkgaur]

done

[emkornfield]

Where does the group of 8 values come in? Wouldn't this messup the number of bytes math?

[prtkgaur]

Good catch — removed 'groups of 8' phrasing. Now simply references the same LSB-first packing order as RLE/Bit-Packing Hybrid.

(I think RleBitPackHybrid was in my mind at that point :) )

[emkornfield]

Why can't they be precomputed at runtime?

[prtkgaur]

You're right — this shouldn't mandate literals vs runtime computation. Reworded to require that encoder and decoder use identical power-of-10 values, without prescribing how they're obtained.

[emkornfield]

Suggested change

	| DOUBLE | 2^51 + 2^52 = 6,755,399,441,055,744 | `(long)((value + magic) - magic)` |
	| DOUBLE | 2^51 + 2^52 = 6,755,399,441,055,744 | `(int64_t)((value + magic) - magic)` |

[prtkgaur]

done

[emkornfield]

Suggested change

	| FLOAT | 2^22 + 2^23 = 12,582,912 | `(int)((value + magic) - magic)` |
	| FLOAT | 2^22 + 2^23 = 12,582,912 | `(int32_t)((value + magic) - magic)` |

[prtkgaur]

done

[emkornfield]

Suggested change

	| encoded = round( value * 10^e * 10^(-f) ) |
	| encoded = fast_round( value * 10^e * 10^(-f) ) |

[prtkgaur]

done

[alamb]

Thank you @prtkgaur and @emkornfield -- I started going through this proposal as well. I haven't made it through the entire thing, but I left commands on what I have made it through first.

@prtkgaur how would you like to address feedback on this PR? Would you like to process the comments? I would also be happy to make a PR with proposed edits to your branch if that would be better. Please just let me know

[alamb]

You have more summary at the end of the encoding, but I think a few more sentences in the intro would help people understand this more easily

Suggested change

	ALP works by converting floating-point values to integers using decimal scaling,
	then applying Frame of Reference (FOR) encoding and bit-packing. Values that
	cannot be losslessly converted are stored as exceptions. The encoding achieves
	high compression for decimal-like floating-point data (e.g., monetary values,
	sensor readings) while remaining fully lossless.
	ALP works by converting floating-point values to integers using decimal scaling,
	then applying Frame of Reference (FOR) encoding and bit-packing. Values that
	cannot be losslessly converted are stored as exceptions. The encoding achieves
	high compression for decimal-like floating-point data (e.g., monetary values,
	sensor readings) while remaining fully lossless. Values do not depend on
	each other, which enables quick random access and parallel encode/decode.

[prtkgaur]

Agreed — added sentence about value independence enabling random access and parallel encode/decode.

[alamb]

I agree this is redundant -- I think leaving the note is valuable clarification

Note: The number of elements per vector and the packed data size are NOT stored
in the header, they are derived

However the other items are not

[alamb]

I also hihglighted this as confusing when reading the spec. I recommend removing this sentence as the packed data size is covered in the "Vector Format" section below

[alamb]

I expect to have a second round of feedback tomorrow

[alamb]

Ok, I made it through the spec again. I think it is really quite close

Let me know what you think about the comments and what you think abotu the idea of starting to draft a Blog post.

[alamb]

Something seems strange to me about this chart. The version on main

parquet-format/Encodings.md

Lines 28 to 40 in 96edf77

	### Supported Encodings
	For details on current implementation status, see the [Implementation Status](https://parquet.apache.org/docs/file-format/implementationstatus/#encodings) page.
	| Encoding type | Encoding enum | Supported Types |
	| ------------------------------------------------ | --------------------------------------------------------- | ------------------------------------------------- |
	| [Plain](#PLAIN) | PLAIN = 0 | All Physical Types |
	| [Dictionary Encoding](#DICTIONARY) | PLAIN_DICTIONARY = 2 (Deprecated) <br> RLE_DICTIONARY = 8 | All Physical Types |
	| [Run Length Encoding / Bit-Packing Hybrid](#RLE) | RLE = 3 | BOOLEAN, Dictionary Indices |
	| [Delta Encoding](#DELTAENC) | DELTA_BINARY_PACKED = 5 | INT32, INT64 |
	| [Delta-length byte array](#DELTALENGTH) | DELTA_LENGTH_BYTE_ARRAY = 6 | BYTE_ARRAY |
	| [Delta Strings](#DELTASTRING) | DELTA_BYTE_ARRAY = 7 | BYTE_ARRAY, FIXED_LEN_BYTE_ARRAY |
	| [Byte Stream Split](#BYTESTREAMSPLIT) | BYTE_STREAM_SPLIT = 9 | INT32, INT64, FLOAT, DOUBLE, FIXED_LEN_BYTE_ARRAY |

Doesn't appear in this document.

I would expect this PR that adds the new ALP encoding to add a new entry to the existing table, rather than add an entirely new table.

Maybe we need to merge up from main to this branch (I am happy to do this and I think I have the permissions to push to your branch, but I don't want to mess things up on you)

[prtkgaur]

You're right, this branch was forked before the summary table was added in #550. Rebased onto upstream/master and added ALP as a row to the existing table.

[alamb]

I think it would help to use consistent terminology here to above.

Suggested change

	The rounding function uses a "magic number" technique for branchless rounding:
	The `fast_round` function uses a "magic number" technique for branchless rounding.
	`fast_round(value)` is defined as follows:

[prtkgaur]

Done. Updated to use fast_round consistently and added "is defined as follows".

[alamb]

I don't understand this sentence. The C++ implementation does not seem to change sign order from what I can tell:

https://github.com/apache/arrow/pull/48345/changes#diff-f9ab708cab94060b4067fff0a6739e9c3751b450422115663b2bd0badfcc748bR487-R490

 /// \brief Convert a float to an int without rounding
  static inline auto FastRound(T n) -> SignedExactType {
    n = n + Constants::kMagicNumber - Constants::kMagicNumber;
    return static_cast<SignedExactType>(n);
  }

Also the "Fast Rounding" section of the ALP paper doesn't mention sign reversal that I can find

[prtkgaur]

You're right -- the sign reversal was incorrect. The C++ implementation uses a single formula for both positive and negative values (n + magic - magic). Removed the paragraph.

[alamb]

This section has a worked example of FOR / bitpacking so perhaps we could write it like

Suggested change

	##### Frame of Reference and Bit-Packing
	##### Example: Frame of Reference and Bit-Packing

[prtkgaur]

Updated

[alamb]

Suggested change

	After decimal encoding and exception substitution:
	Given the following data after decimal encoding and exception substitution:

[prtkgaur]

Updated

[alamb]

This appears to be C/++ implementation details -- I am not sure it adds any value to the spec

[prtkgaur]

removed

[alamb]

I don't think we need to add add to the Parquet spec -- it isn't required to implement the encoder/decoder for ALP.

[prtkgaur]

removed

[alamb]

Suggested change

	enabling random access to individual vectors and parallel encode/decode.
	enabling random access to individual values and parallel encode/decode.

[prtkgaur]

done

[alamb]

I don't understand what "use identical power-of-10 values" means. ALL encoders and decoders must to use the exact same values (and floating point arithmetic) as I understand it.

[prtkgaur]

Yes, what you said. Reworded to: "All implementations MUST use the exact same floating-point arithmetic and power-of-10 constants to guarantee cross-language interoperability."

[alamb]

I think SHOULD is too strong a word here. Maybe we could soften it to a suggestion

Suggested change

	To avoid the cost of exhaustive search on every vector, implementations
	SHOULD use sampling to select up to 5 candidate (exponent, factor)
	combinations (the "encoding preset") at the start of each column chunk.
	Each vector then searches only those 5 candidates.
	To avoid the cost of exhaustive search on every vector, implementations
	can use a sampling approach. One such approach, described in the paper, is to
	select up to 5 candidate (exponent, factor) combinations (the "encoding preset")
	at the start of each column chunk, and when encoding each vector,
	test each of the 5 candidates for the fewest exceptions.

[prtkgaur]

Done

[pitrou]

The algorithm description is so long that I think it should be moved to a separate file that we would link to here. In this file we would just keeping the description that is before the Overview.

[prtkgaur]

Yes this makes sense. I too thought it became long and having a separate file would be good.
Let me take a stab at it after I address the above comments and get approval. Else the comment threads on github will get lost.

[alamb]

Maybe similar to how it is done for bloom filter: https://github.com/apache/parquet-format/blob/master/BloomFilter.md

[alamb]

I also think it would be ok to if we moved the content to a separate file as a follow on PR (after the spec change is approved) as in my mind moving the content to a separate file does not affect the content of the spec

[alamb]

It seems we should do this as a follow on PR. I can do so after this PR is merged

[iemejia]

Review focusing on spec consistency, numerical correctness under IEEE 754, and completeness for cross-language implementors.

[iemejia]

The formula shown here only covers non-negative values. For negative values, the expression value + magic may fall below the binade [2^52, 2^53) (for double), entering a region where ULP = 0.5 instead of 1.0, which produces incorrect rounding.

Known implementations (C++/DuckDB, Java) use sign branching:

if value >= 0:
    result = (int64_t)((value + magic) - magic)
else:
    result = (int64_t)((value - magic) + magic)

While the round-trip exception check prevents data corruption (incorrectly rounded values become exceptions), omitting sign branching causes a higher exception rate for datasets with negative values, degrading compression ratios.

Suggestion: document both branches in the spec table, or at minimum add a note that implementations SHOULD use sign branching for negative values.

[prtkgaur]

Good catch — you're right. I've updated the spec to document both branches:

  if value >= 0:  (intN_t)((value + magic) - magic)                                                                                                    
  if value <  0:  (intN_t)((value - magic) + magic)                                                                                                    

I also added a paragraph explaining why: the technique relies on landing in a binade where ULP = 1.0, which only works for non-negative values with +magic and negative values with -magic. Without sign branching, negative values with magnitude beyond 2^22 (float) or 2^51 (double) fall into a lower binade and round incorrectly — silently increasing the exception rate.

[alamb]

Known implementations (C++/DuckDB, Java) use sign branching:

@iemejia can you please provide example / evidence for this this statement?

The three implementatons I could find all use brancless code (I found not find any check for sign) -- they simply add then subtract the magic number

DuckDB's implementation of ALP (C++)

https://github.com/duckdb/duckdb/blob/68d73b7c3a85a8907909cf6d4b6533f83a45496d/src/include/duckdb/storage/compression/alp/algorithm/alp.hpp#L118-L134

SpiralDB's implementation of ALP (Rust)

https://github.com/spiraldb/alp/blob/d797ba56f980f83657bf6b812f88714ebe3ed674/src/alp/mod.rs#L70-L74

cwida/ALP (reference implementation from the paper authors)

https://github.com/cwida/ALP/blob/31ca0ed11c93c99d3f5b5c30e01a3e1c3832d3ce/include/alp/encoder.hpp#L87

[alamb]

Without sign branching, negative values with magnitude beyond 2^22 (float) or 2^51 (double) fall into a lower binade and round incorrectly — silently increasing the exception rate.

Can you please provide some examples where encoding using the signed version would correctly recover the intended value but the unsigned version would not?

What is the evidence that the exception rate would be higher?

[alamb]

I made a plaground to show how to test various combinations / how fast_round works: https://github.com/alamb/fast_round\

(update: I can't find any reason why we would want to add the sign test)

[iemejia]

The value 333.3 with (exponent=4, factor=3) may not survive a round-trip under strict IEEE 754 arithmetic:

• Encode: 333.3 * 10^4 * 10^(-3) = 333.3 * 10000.0 * 0.001. Since 0.001 is not exactly representable in IEEE 754 double, the product is approximately 3333.000000000000069..., which rounds to 3333.
• Decode: 3333 * 10^3 * 10^(-4) = 3333 * 1000 * 0.0001. Since 0.0001 is not exactly representable, the product may not bit-equal 333.3.

If decode(encode(333.3)) != 333.3 at the bit level, this value should be classified as an exception. The example may be using idealized math rather than actual IEEE 754 double semantics.

Suggestion: verify this specific round-trip in an actual IEEE 754 implementation, or replace 333.3 with a value that provably round-trips (e.g., 3000.0, 500.0, or any value where the scaling produces an exact result).

[prtkgaur]

Umm,

I verified this computationally — 333.3 with (e=4, f=3) round-trips exactly under IEEE 754 double arithmetic:

Encode: 333.3 * 10000.0 = 3333000.0 (exact), * 0.001 = 3333.0 (exact)                                                                                  
Decode: 3333 * 1000.0 = 3333000.0 (exact), * 0.0001 = 333.3 (bit-identical to original)                                                                

The intermediate value is exactly 3333.0, not ~3333.000000000000069 as suggested. This is precisely the property that the two-separate-multiplications design guarantees: for values with d decimal digits where d ≤ (e - f), the round-trip is exact. Here 333.3 has d=1 digit and e-f=1, so the invariant holds.

The concern that "0.001 is not exactly representable therefore the product is I think not right" doesn't follow — IEEE 754 correctly-rounded multiplication of 3333000.0 by the nearest double to 0.001 produces exactly 3333.0 in this case.

[emkornfield]

I think the concern might be 333.3 as text is not actually the value that gets stored as a double. For instance if you look at (https://www.h-schmidt.net/FloatConverter/IEEE754.html) 333.29998779296875. I think the math still ends up working out here though. Generally, ".3" is the canonical example of something that doesn't store exactly into floating point values. I think the recommendation here would be use something like 333.25 which doesn't suffer from this problem.

[prtkgaur]

Ahh got it.

Changed the worked-example value from 333.3 to 333.5. 333.5 is exactly representable in IEEE 754 (667/2), so it sidesteps the ".3 isn't stored exactly" issue entirely while still using a non-zero factor.

[iemejia]

This condition is under-specified in two ways:

1. For DOUBLE, the target integer type is int64, not int32. The condition should reference INT64_MAX for doubles.
2. The actual safe casting limit is not exactly INT32_MAX / INT64_MAX but the largest floating-point value that can be safely converted to the target integer type without undefined behavior. For float→int32 this is approximately 2147483520.0f (not 2147483647), and for double→int64 approximately 9223372036854774784.0 (not 9223372036854775807). These differ because not all integers near the max are exactly representable in the source float/double type.

Suggestion: either specify the exact limits for each type, or reword to: "the scaled value exceeds the range that can be losslessly represented in the target integer type (int32 for FLOAT, int64 for DOUBLE)."

[prtkgaur]

Good catch on both points. Fixed — the table now reads:

| Out of range | scaled value outside int32 (FLOAT) or int64 (DOUBLE) | Exceeds target integer type range

This avoids both the int32-only bug and the subtlety around exact casting limits (e.g., 2147483520.0 being the actual safe boundary for float→int32). The round-trip check remains the authoritative correctness condition regardless.

[iemejia]

This requirement is not actionable without specifying the actual constant values. Powers of 10 like 10^(-3) are not exactly representable in IEEE 754, and different methods of computing them (e.g., 1.0/1000.0 vs the compile-time literal 1e-3 vs pow(10, -3)) can produce different bit patterns.

Suggestion: either (a) provide a table of required constants with their exact IEEE 754 hex representations, or (b) specify that constants MUST match the values produced by the standard decimal-to-binary conversion of the literals 1e0, 1e1, ..., 1e18 and 1e-0, 1e-1, ..., 1e-18 as defined by IEEE 754-2008 §5.12.2. This makes the requirement unambiguous and testable across languages.

[prtkgaur]

Thanks for the review. The concern about 1.0/1000.0 vs 1e-3 producing different bit patterns is not actually valid — both operands are exactly representable in IEEE 754, and division is a required correctly-rounded operation, so the result is bit-identical to the literal. The real risk you're identifying is pow(), which is not required to be correctly rounded (it's only a "recommended" operation even in IEEE 754-2019).

That said, I agree the spec should be explicit. I've added a clarification that the constants MUST match the correctly-rounded IEEE 754 decimal-to-binary conversion of the literals 1e0..1e18 and 1e-1..1e-18 per §5.12.2, and that implementations MUST NOT compute these via pow(). This makes the requirement unambiguous and testable without needing a hex table — any conforming implementation will produce identical bit patterns from these short decimal literals.

[rdblue]

Do we have a test case or two where we know pow() produces an incorrect value in some cases? Then we could add a table of test values to help catch the issue.

[alamb]

This requirement is not actionable without specifying the actual constant values.

I don't agree that this requirement is not actionable (we have at least two implementations that didn't seem to have any trouble figuring out 1e0 and 1e2 man.

(b) specify that constants MUST match the values produced by the standard decimal-to-binary conversion of the literals 1e0, 1e1, ..., 1e18 and 1e-0, 1e-1, ..., 1e-18 as defined by

I think that would be ok, but to be honest I don't think any more clarification is needed here

[alamb]

Do we have a test case or two where we know pow() produces an incorrect value in some cases? Then we could add a table of test values to help catch the issue.

Given the spec doesn't call for pow() I the the discussion about the exact numeric representation of pow() is somewhat distracting -- I recommend we leave the spec as is

[iemejia]

In practice, minimizing exception count alone is not optimal. A parameter pair with slightly more exceptions but a much smaller bit-width (tighter FOR range) can produce smaller output. The actual optimization target should be estimated encoded size, accounting for both bit-width and exception overhead.

Since this is an encoder-only concern (decoders are agnostic to the selection strategy), the spec should clarify that any valid (e, f) pair produces a correct encoding — the choice only affects compression ratio. Suggested rewording:

The encoder SHOULD select the (exponent, factor) pair that produces the smallest encoded output. A simple heuristic is to minimize exception count; a more precise approach accounts for both bit-width and exception overhead.

[prtkgaur]

Good point — accepted. I've reworked the section to:

1. Lead with the correctness statement: any valid (e,f) pair produces a correct encoding, the decoder is agnostic, and the exception mechanism guarantees round-trip fidelity regardless.
2. Use SHOULD for the optimization guidance and frame "minimize exceptions" as one heuristic rather than the goal. The actual target is smallest encoded size, accounting for both bit-width and exception overhead.
3. Updated the sampling paragraph accordingly.

[emkornfield]

Suggested change

	| min_val = min(encoded[]) |
	| min_val = min(encoded[:]) |

small nit show that it is meant to select all values, don't feel strongly.

[prtkgaur]

Applied in both diagram and table

[emkornfield]

Suggested change

	landing in a binade where the unit in the last place (ULP) equals 1.0. For
	landing in a [binade](https://en.wikipedia.org/wiki/Binade) where the unit in the last place (ULP) equals 1.0. For

Had to look this up.

[prtkgaur]

Added Wikipedia link

[emkornfield]

should we include the hex-values here to avoid ambiguity?

[prtkgaur]

I'd lean against a hex table here. The spec already pins the constants unambiguously: they must be the correctly-rounded IEEE 754 conversions of the decimal literals 1e0..1e18 and 1e-1..1e-18 per §5.12.2, and MUST NOT be computed via pow(). Those are short literals, so every conforming decimal-to-binary conversion yields identical bit patterns — a hex table would be redundant and is itself an error-prone thing to maintain across 37 constants. If reviewers still want the exact patterns spelled out, I'd suggest an appendix/table in the eventual standalone ALP file rather than inline here. Does that work, or would you prefer the hex table now?

[alamb]

I agree adding hex values doesn't seem like it would add much

[emkornfield]

This seems complete to me, thank you for all the revisions.

[emkornfield]

We also need to update the enum in parquet.thrift in this repo.

[mapleFU]

What about FLBA and float16 datatype? Or should we even support fp8 or other types in the future? ( This can be not in this pr, just curious that)

[alamb]

It would be interesting to research as a next step in my opinion

The fixed length byte array (FLBA) seems unlikely to benefit from ALP (which is specifically designed to recognize values that are encoded using floating point that could be more efficiently represented as fixed precision decimals

FP16/FP8 could be helped, for decimal data that is stored in that type of column; However I think those types are often used for ML type usecases where the data uses the full range of the types, as I understand

[wgtmac]

I've raised the same question in the google doc earlier. FP16 and FP8 have fewer bits so the benefit may not be that significant compared to regular float and double. But I agree that we can discuss these in the future to not expand and thus block current spec design.

[pitrou]

It's not branchless if there's a sign branching involved.

We should also explain why this function is useful or necessary. What is the expected rounding mode? Halfway to even?

[alamb]

It's not branchless if there's a sign branching involved.

Yes, this is confusing. The original paper (and the last time I reviewed this spec implementation) did not have this branching. I don't the branching is necessary and the explanation is very confusing (if not incorrect -- see my comments on this thread #557 (comment))

We should also explain why this function is useful or necessary. What is the expected rounding mode? Halfway to even?

I believe the rationale is to "recover the intended integer after computing value * 10^e * 10^-f which introduces mathematical noise due to floating point arithmetic

This is not well explained in the original paper either, which perhaps lead to the confusion in this section

From my reading, this part of the spec is normative anyway. Since it isn't called on the decode path, encoders can use a different method (like the original branchless version) and still produce conformant files

Thus I don't think my concerns with this section are blocking; However, as written I now find it quite misleading and it would be good to clarify / fix this

[alamb]

I went through this again and it looks good to me. Thank you so much @prtkgaur and everyone who has spent time reviewing and iterating on this feature.

I personally think the changes to the fast_rounding section to include the signed version should be reverted as they of dubious value, but since that spec is normative (aka a suggested way to implement the encoder, and not required) and not incorrect, I don't see it as a blocker

FYI @azimafroozeh (it's happening!)

[alamb]

It would be interesting to research as a next step in my opinion

The fixed length byte array (FLBA) seems unlikely to benefit from ALP (which is specifically designed to recognize values that are encoded using floating point that could be more efficiently represented as fixed precision decimals

FP16/FP8 could be helped, for decimal data that is stored in that type of column; However I think those types are often used for ML type usecases where the data uses the full range of the types, as I understand

[alamb]

It seems we should do this as a follow on PR. I can do so after this PR is merged

[alamb]

Known implementations (C++/DuckDB, Java) use sign branching:

@iemejia can you please provide example / evidence for this this statement?

The three implementatons I could find all use brancless code (I found not find any check for sign) -- they simply add then subtract the magic number

DuckDB's implementation of ALP (C++)

https://github.com/duckdb/duckdb/blob/68d73b7c3a85a8907909cf6d4b6533f83a45496d/src/include/duckdb/storage/compression/alp/algorithm/alp.hpp#L118-L134

SpiralDB's implementation of ALP (Rust)

https://github.com/spiraldb/alp/blob/d797ba56f980f83657bf6b812f88714ebe3ed674/src/alp/mod.rs#L70-L74

cwida/ALP (reference implementation from the paper authors)

https://github.com/cwida/ALP/blob/31ca0ed11c93c99d3f5b5c30e01a3e1c3832d3ce/include/alp/encoder.hpp#L87

[alamb]

This doesn't make any sense to me. It seems like an overly complicated argument with fancy terms that ultimately isn't useful in the context of ALP and has no empirical justification

See my comment/question on the original thread here: #557 (comment)

[rdblue]

Here's an example where the branch-less fast round produces an error:

@Test
  public void testFloats() {
    float f = 6194305.5f;
    float magic = 8388608.0f;

    assertThat((int) (f + magic - magic)).isEqualTo(6194306);

    float nf = -6194305.5f;
    float nmagic = -8388608.0f;

    assertThat((int) (nf + nmagic - nmagic)).isEqualTo(-6194306);

    float umagic = 12582912.0f;

    // off by 1
    assertThat((int) (nf + umagic - umagic)).isEqualTo(-6194305);
  }

I think the issue is limited to non-integer values in the bottom half of (-2^23, -2^22]. When using the mid-point constant to avoid branching, they are not forced into the 2^23 binade. As a result, the precision is not limited and the rounding does not occur.

[alamb]

Here's an example where the branch-less fast round produces an error:

When I played around with it, I found that switching to branchless fast rounding simply trades one set of error cases for another

I made a playground to show how to test various combinations work (repo here https://github.com/alamb/fast_round)

For example here are three cases where the branchless rounding is actually better and the signed version is worse

        // (b) At and above 2^52 plain is better
        Case {
            input: -5226231736140883.0,
            correct: -5226231736140883,
            fast_round: -5226231736140883,  // exact
            sign_branch: -5226231736140884, // off by 1
            description: "[2^52,2^53): plain is exact, branch corrupts it",
        },
        Case {
            input: -12298387206974542.0,
            correct: -12298387206974542,
            fast_round: -12298387206974542,  // exact
            sign_branch: -12298387206974544, // off by 2
            description: "[2^53,2^54): plain is exact, branch corrupts it",
        },
        Case {
            input: -143122344316276784.0,
            correct: -143122344316276784,
            fast_round: -143122344316276784,  // exact
            sign_branch: -143122344316276800, // off by 16
            description: "large: plain is exact, branch corrupts it",
        },

[alamb]

It's not branchless if there's a sign branching involved.

Yes, this is confusing. The original paper (and the last time I reviewed this spec implementation) did not have this branching. I don't the branching is necessary and the explanation is very confusing (if not incorrect -- see my comments on this thread #557 (comment))

We should also explain why this function is useful or necessary. What is the expected rounding mode? Halfway to even?

I believe the rationale is to "recover the intended integer after computing value * 10^e * 10^-f which introduces mathematical noise due to floating point arithmetic

This is not well explained in the original paper either, which perhaps lead to the confusion in this section

From my reading, this part of the spec is normative anyway. Since it isn't called on the decode path, encoders can use a different method (like the original branchless version) and still produce conformant files

Thus I don't think my concerns with this section are blocking; However, as written I now find it quite misleading and it would be good to clarify / fix this

[alamb]

Without sign branching, negative values with magnitude beyond 2^22 (float) or 2^51 (double) fall into a lower binade and round incorrectly — silently increasing the exception rate.

Can you please provide some examples where encoding using the signed version would correctly recover the intended value but the unsigned version would not?

What is the evidence that the exception rate would be higher?

[alamb]

I agree adding hex values doesn't seem like it would add much

[alamb]

This requirement is not actionable without specifying the actual constant values.

I don't agree that this requirement is not actionable (we have at least two implementations that didn't seem to have any trouble figuring out 1e0 and 1e2 man.

(b) specify that constants MUST match the values produced by the standard decimal-to-binary conversion of the literals 1e0, 1e1, ..., 1e18 and 1e-0, 1e-1, ..., 1e-18 as defined by

I think that would be ok, but to be honest I don't think any more clarification is needed here

[alamb]

Do we have a test case or two where we know pow() produces an incorrect value in some cases? Then we could add a table of test values to help catch the issue.

Given the spec doesn't call for pow() I the the discussion about the exact numeric representation of pow() is somewhat distracting -- I recommend we leave the spec as is

[wgtmac]

I've raised the same question in the google doc earlier. FP16 and FP8 have fewer bits so the benefit may not be that significant compared to regular float and double. But I agree that we can discuss these in the future to not expand and thus block current spec design.

[wgtmac]

Suggested change

	enabling random access to individual values and parallel encode/decode.
	enabling random access to individual values and parallel encoding/decoding.

[wgtmac]

Just surprised that there isn't any change in the parquet.thrift?

[wgtmac]

IIUC, the informative compression description should be move to a separate doc later in the followup PR because here we just need necessary description for decoding a ALP-encoded page (i.e. the Page Layout section).

cc @alamb

[wgtmac]

Suggested change

	Byte: 0 1 2 3 4 5 6
	+----------------+---------------+--------------+----+----+----+----+
	Byte: 0 1 2 3 4 5 6
	+----------------+---------------+--------------+----+----+----+----+

Let's adjust the alignment here.

[wgtmac]

If the minimum value is 3, why not defining it as log_vector_size_minus_three so it can start with 0? Not a strong opinion since this anyway consume a byte which is large enough.

[wgtmac]

Suggested change

	| Select (exponent, factor) pair for this vector |
	| Select (exponent, factor) pair for this array |

Be consistent with Input: float/double array above.

[wgtmac]

This is incorrect when pages are compressed or have rep/def levels. I would suggest removing this sentence. If we want to emphasize random access of ALP, perhaps we should replace page_data_start with a more precise start point after decompressed.

[wgtmac]

Suggested change

	| 3 | num_elements | 4 bytes | int32 | Total number of floating-point values in the page |
	| 3 | num_elements | 4 bytes | int32 | Total number of non-null floating-point values in the page |

[wgtmac]

Why num_vectors * 4 but not num_vectors * 4 + 7 to include the header? Usually the whole buffer is stored including header, so implementations need to manually add 7 each time to get an absolute offset to each vector.

[wgtmac]

Why here we use camel case but other places use snake case?

[wgtmac]

This only applies to compression_mode=0 and integer_encoding=0, right? Should we be explicit on this? Future settings may not even have AlpInfo and ForInfo.

[wgtmac]

num_exceptions is described below so readers may have no idea where it comes from when reading this line.

[wgtmac]

Can we enhance the graph by pointing out areas of vector header and data section?

[wgtmac]

It seems that the paper has THREE authors but we have only listed TWO.

[wgtmac]

In the past, parquet-java has NaN canonicalization which is wrong. Should we clarify that these are raw bits?

[wgtmac]

We need to clarify that delta[i] is an unsigned integer to avoid signed overflow.