Hadoop Compression error

After the fixes to CVE-2026-90559 I have noticed that sometimes compressing with SnappyHadoopCompatibleOutputStream fails with the exception IllegalArgumentException("not enough space for output: need %,d bytes, but only %,d remaining");

I have the following unit test to prove this:

package org.xerial.snappy;

import org.apache.commons.io.IOUtils;
import org.apache.commons.lang3.RandomUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.io.compress.CompressionInputStream;
import org.apache.hadoop.io.compress.SnappyCodec;
import org.junit.Assert;
import org.junit.Test;

import java.io.FileOutputStream;
import java.nio.file.Files;
import java.nio.file.Path;

public class HadoopTest {
    
    @Test
    public void test() throws Exception{
        SnappyCodec hadoopCodec = new SnappyCodec();
        hadoopCodec.setConf(new Configuration());
        RandomUtils rnd = RandomUtils.insecure();
        
        for(int i=0; i < 99_000; i++){
            int size = rnd.randomInt(5_120, 512_000); // 5kb to 500kb
            byte[] input = rnd.randomBytes(size);
            
            Path tmp = Files.createTempFile("test", "test");
            FileOutputStream fileOutputStream = new FileOutputStream(tmp.toFile());
            SnappyHadoopCompatibleOutputStream snappyOutput = new SnappyHadoopCompatibleOutputStream(fileOutputStream);
            snappyOutput.write(input);
            snappyOutput.flush();
            
            
            CompressionInputStream snappyInput = hadoopCodec.createInputStream(Files.newInputStream(tmp));
            byte[] output = new byte[size];
            IOUtils.read(snappyInput, output);
            
            
            Assert.assertArrayEquals(input,output);
            Files.deleteIfExists(tmp);
        }
    }
}

I did find a fix to this by changing SnappyOutputStream::hasSufficientOutputBufferFor
It had the magic number of -4. I changed it to -7 and all tests passed.

private boolean hasSufficientOutputBufferFor(int inputSize)
    {
        int maxCompressedSize = Snappy.maxCompressedLength(inputSize);
        // I changed the magic number of 4 to 7
        return maxCompressedSize < outputBuffer.length - outputCursor - 7;
    }

I would put a PR about this, but I did not want to until I understand what -4 & -7 means.
I suspect it's something to do with hadoop not having a header but I'm not sure.

Any help would be welcome.

~~duplicated with #765?~~

seems not

-4 and -7 are the number of bytes the guard must reserve in front of the worst-case compressed payload, measured from the current outputCursor. The guard runs before the block preamble is written, so it has to cover:

maxCompressedLength(inputSize) + per-block header

The header size depends on the stream type:

stream	bytes written before the payload	correct guard
`SnappyOutputStream` (xerial framed)	4 (compressed length)	`maxComp <= len - C - 4`, i.e. `< len - C - 3`
`SnappyHadoopCompatibleOutputStream`	8 (uncompressed length + compressed length)	`maxComp <= len - C - 8`, i.e. `< len - C - 7`

Hadoop's block format is [uncompressedSize][compressedSize][data] - see BlockCompressorStream at rel/release-3.5.0, line 82: "Each block contains the uncompressed length for the block, followed by one or more length-prefixed blocks of compressed data"). SnappyHadoopCompatibleOutputStream writes the extra 4-byte uncompressed length in writeBlockPreemble() -> writeCurrentDataSize(), and that call advances outputCursor by 4, so compression actually starts at outputCursor + 8 relative to where the guard ran. The guard's 4 only accounts for the compressed-length field, which is why the Hadoop-compatible stream ends up 4 bytes short.

Why -7 rather than -8: the guard is written with a strict <, while the real requirement is <= (the bounds check throws when outputSize - outputOffset < requiredSize). Since maxComp <= X is the same as maxComp < X + 1, the header size appears as one less in the strict form. So -7 is the tightest correct value, and -8 is also safe (one byte more conservative). For the base stream the same formula gives -3; upstream's -4 is just 1 byte stricter than that stream needs, which is harmless there but wrong once the subclass adds a second header field.

Concrete numbers from a local reproduction on 1.1.10.11 (bufferLen=38281 = SnappyCodec.HEADER_SIZE 16 + 4 + maxCompressedLength(32768)=38261, outputCursor=32782, failing inputSize=4680, maxCompressedLength(4680)=5492):

guard -4: 5492 < 38281-32782-4 = 5495 -> true, no flush, then throws
actual need: 5492 <= 38281-32782-8 = 5491 -> false -> need 5,492 bytes, but only 5,491 remaining
guard -7: 5492 < 5492 -> false, flushes first, then it fits
the failing partial-block sizes are {4680, 4681, 4682} - exactly three values, matching the 4-to-7 difference

Rather than introduce a new literal, it would be cleaner to derive the reserved size so subclasses that add a preamble are handled automatically:

protected int blockHeaderSize() { return 4; }        // base
// SnappyHadoopCompatibleOutputStream: return 8;

private boolean hasSufficientOutputBufferFor(int inputSize) {
    int maxCompressedSize = Snappy.maxCompressedLength(inputSize);
    return maxCompressedSize <= outputBuffer.length - outputCursor - blockHeaderSize();
}

@pan3793

Thanks @pan3793
I have made this PR #768

Add regression test for Hadoop block header overflow (xerial#767)

Hadoop uses a blockHeaderSize of 8 instead of the normal 4 (#768)

@tomjshore

Fixed by #768 and released in v1.1.10.12, now available on Maven Central. Thanks for the report and the fix, @tomjshore!

Navigation Menu

Repository navigation

Hadoop Compression error #767

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Hadoop Compression error #767

Description

Activity

pan3793 commented on Oct 8, 2026

pan3793 commented on Oct 8, 2026

tomjshore commented on Oct 10, 2026

xerial commented on Oct 10, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions