fix: memory over allocation due to last partial block group - #832
Open
siddh34 wants to merge 1 commit into
Open
Conversation
3 tasks
dkovba
requested changes
Aug 7, 2026
Contributor
There was a problem hiding this comment.
Thank you for the contribution!
Below are the test results across the size/content matrix, cross-checked with e2fsck:
Requested Size: 32KiB
Content: none
Size: FAIL
audit.imageSize == testCase.requested = false; audit.imageSize = 134217728; testCase.requested = 32768
e2fsck: PASS
----------------------------------------
Requested Size: 64MiB
Content: none
Size: FAIL
audit.imageSize == testCase.requested = false; audit.imageSize = 134217728; testCase.requested = 67108864
e2fsck: PASS
----------------------------------------
Requested Size: 128MiB
Content: none
Size: PASS
e2fsck: PASS
----------------------------------------
Requested Size: 128MiB+4KiB
Content: none
Size: PASS
e2fsck: PASS
----------------------------------------
Requested Size: 130MiB+8KiB
Content: none
Size: PASS
e2fsck: PASS
----------------------------------------
Requested Size: 160MiB
Content: none
Size: PASS
e2fsck: PASS
----------------------------------------
Requested Size: 256MiB
Content: none
Size: PASS
e2fsck: PASS
----------------------------------------
Requested Size: 1GiB
Content: none
Size: PASS
e2fsck: PASS
----------------------------------------
Requested Size: 4GiB
Content: none
Size: FAIL
allocatedMiB <= ceilingMiB = false; allocatedMiB = 65; ceilingMiB = 32
e2fsck: PASS
----------------------------------------
Requested Size: 63x128MiB
Content: none
Size: FAIL
allocatedMiB <= ceilingMiB = false; allocatedMiB = 127; ceilingMiB = 32
e2fsck: PASS
----------------------------------------
Requested Size: 63x128MiB+4KiB
Content: none
Size: FAIL
allocatedMiB <= ceilingMiB = false; allocatedMiB = 129; ceilingMiB = 32
e2fsck: FAIL rc=4
Block bitmap differences: +(32768--32930)
Fix? no
/imgs/63x128MiB+4KiB.img: ********** WARNING: Filesystem still has errors **********
/imgs/63x128MiB+4KiB.img: 11/524288 files (0.0% non-contiguous), 32737/2064385 blocks
----------------------------------------
Requested Size: 8GiB
Content: none
Size: FAIL
allocatedMiB <= ceilingMiB = false; allocatedMiB = 129; ceilingMiB = 32
e2fsck: FAIL rc=4
Block bitmap differences: +(32768--32930)
Fix? no
/imgs/8GiB.img: ********** WARNING: Filesystem still has errors **********
/imgs/8GiB.img: 11/524288 files (0.0% non-contiguous), 32737/2097152 blocks
----------------------------------------
Requested Size: 16GiB
Content: none
Size: FAIL
allocatedMiB <= ceilingMiB = false; allocatedMiB = 258; ceilingMiB = 32
e2fsck: FAIL rc=4
Block bitmap differences: +(32768--65826)
Fix? no
/imgs/16GiB.img: ********** WARNING: Filesystem still has errors **********
/imgs/16GiB.img: 11/1048576 files (0.0% non-contiguous), 32737/4194304 blocks
----------------------------------------
Requested Size: 160MiB
Content: 10MiB
Size: PASS
e2fsck: PASS
----------------------------------------
Requested Size: 128MiB
Content: 50MiB
Size: PASS
e2fsck: PASS
----------------------------------------
Requested Size: 160MiB
Content: 120MiB
Size: PASS
e2fsck: PASS
----------------------------------------
Requested Size: 160MiB
Content: 124MiB
Size: PASS
e2fsck: FAIL rc=4
Block bitmap differences: +(32768--32806)
Fix? no
/imgs/160MiB_124MiB-content.img: ********** WARNING: Filesystem still has errors **********
/imgs/160MiB_124MiB-content.img: 135/16384 files (0.0% non-contiguous), 32737/40960 blocks
----------------------------------------
Requested Size: 160MiB
Content: 126MiB
Size: PASS
e2fsck: PASS
----------------------------------------
Requested Size: 256MiB
Content: 130MiB
Size: PASS
e2fsck: PASS
----------------------------------------
Requested Size: 1GiB
Content: 200MiB
Size: PASS
e2fsck: PASS
----------------------------------------
Requested Size: 300MiB
Content: 260MiB
Size: FAIL
audit.imageSize == testCase.requested = false; audit.imageSize = 402653184; testCase.requested = 314572800
e2fsck: PASS
----------------------------------------
Requested Size: 4GiB
Content: 260MiB
Size: PASS
e2fsck: PASS
----------------------------------------
Requested Size: 63x128MiB
Content: 500MiB
Size: PASS
e2fsck: FAIL rc=4
Block bitmap differences: +(131072--160417)
Fix? no
/imgs/63x128MiB_500MiB-content.img: ********** WARNING: Filesystem still has errors **********
/imgs/63x128MiB_500MiB-content.img: 511/516096 files (0.0% non-contiguous), 131041/2064384 blocks
----------------------------------------
Requested Size: 8GiB
Content: 300MiB
Size: PASS
e2fsck: FAIL rc=4
Block bitmap differences: +(98304--109730)
Fix? no
/imgs/8GiB_300MiB-content.img: ********** WARNING: Filesystem still has errors **********
/imgs/8GiB_300MiB-content.img: 311/524288 files (0.0% non-contiguous), 98273/2097152 blocks
----------------------------------------
Requested Size: 16GiB
Content: 1000MiB
Size: PASS
e2fsck: FAIL rc=4
Block bitmap differences: +(262144--321828)
Fix? no
/imgs/16GiB_1000MiB-content.img: ********** WARNING: Filesystem still has errors **********
/imgs/16GiB_1000MiB-content.img: 1011/1048576 files (0.0% non-contiguous), 262113/4194304 blocks
----------------------------------------
Author
|
@dkovba thanks for the tests I will update my pr in few days and will include these failed tests in it too! |
siddh34
force-pushed
the
fix/memoryOverAllocation
branch
from
August 11, 2026 03:07
98555ce to
16a7a7b
Compare
Author
|
Updates after issues discovered in new test cases as given by @dkovba
defining minimum inodes per group at line 41 This tries multiple candidate values for inodesPerGroup line 1093 it basically means smaller inode tables less metadata overhead
Now ext4 block bitmaps are per-group. Changes can be found at line 879 basically marking the packed metadata blocks that physically live inside a group as allocated in that group’s block bitmap also tested with e2fsck @dkovba please check according to your convience, thanks! Open to making any changes needed or I missed for this current pull request |
dkovba
reviewed
Aug 15, 2026
dkovba
left a comment
dkovba
left a comment
Contributor
There was a problem hiding this comment.
Thank you for updating the PR!
dkovba
requested changes
Aug 15, 2026
dkovba
left a comment
dkovba
left a comment
Contributor
There was a problem hiding this comment.
With the suggested implementation of the partial block support, we use significantly more physical space than on main and than mke2fs does.
With minimumInodesPerGroup: UInt32 = 896:
Requested Size: 16GiB
Content: none
Size: PASS
e2fsck: PASS
Logical Size vs mke2fs: PASS
Physical Size vs mke2fs: FAIL
physical <= mke2fs = false; physical = 30,457,856; mke2fs = 4,816,896
With the original block size derivation:
Requested Size: 16GiB
Content: none
Size: PASS
e2fsck: PASS
Logical Size vs mke2fs: PASS
Physical Size vs mke2fs: FAIL
physical <= mke2fs = false; physical = 269,647,872; mke2fs = 4,816,896
| // minimizes the number of blockGroups needed to its lowest value | ||
| for ipg in stride(from: inc, through: Int(self.maxInodesPerGroup), by: inc) { | ||
| let start = Int(self.minimumInodesPerGroup) // inodesPerGroup | ||
| let step = Int(self.minimumInodesPerGroup) |
Contributor
There was a problem hiding this comment.
Should we keep the original block size derivation here? With minimumInodesPerGroup: UInt32 = 896, we may run out of inodes.
↳ 256MiB image, 8000 files:
↳ write: Could not allocate inode in ext2 filesystem
↳ created == fileCount → false
↳ created → 1781
↳ fileCount → 8000
Author
There was a problem hiding this comment.
Okay got it, Would not change the original block size derivation
the idea was to keep packed-metadata inode tables small for sparse & large images
this would break the allocatedSize <= ceilingMIB
shall I remove the check for allocatedSize <= ceilingMIB?
siddh34
force-pushed
the
fix/memoryOverAllocation
branch
from
August 15, 2026 11:36
16a7a7b to
b01028b
Compare
siddh34
force-pushed
the
fix/memoryOverAllocation
branch
from
August 15, 2026 11:42
b01028b to
0594735
Compare
Author
Tested with let inc = Int(self.blockSize * self.inodeStrideKiB) / Int(EXT4.InodeSize) // minimizes the number of blockGroups needed to its lowest valuewhere self.inodeStrideKiB = 512 Results Here is my script for tests import ContainerizationArchive
import Foundation
import SystemPackage
import Testing
@testable import ContainerizationEXT4
struct E2FsckImageCase: Sendable {
let name: String
let imagePath: FilePath
let requestedSize: UInt64
let contentSize: UInt64
init(
name: String,
imagePath: FilePath,
requestedSize: UInt64,
contentSize: UInt64 = 0
) {
self.name = name
self.imagePath = imagePath
self.requestedSize = requestedSize
self.contentSize = contentSize
}
}
struct E2FsckResult: Sendable {
let exitCode: Int32
let stdout: String
let stderr: String
var passed: Bool {
exitCode == 0
}
}
struct EXT4E2FsckHarness {
let e2fsckPath: String
let mke2fsPath: String
init(
e2fsckPath: String = "/opt/homebrew/Cellar/e2fsprogs/1.47.4/sbin/e2fsck",
mke2fsPath: String = "/opt/homebrew/Cellar/e2fsprogs/1.47.4/sbin/mke2fs"
) {
self.e2fsckPath = e2fsckPath
self.mke2fsPath = mke2fsPath
}
func createReferenceImage(requestedSize: UInt64, at imagePath: FilePath) throws {
if FileManager.default.fileExists(atPath: imagePath.string) {
try FileManager.default.removeItem(at: imagePath.url)
}
FileManager.default.createFile(atPath: imagePath.string, contents: nil)
let handle = try FileHandle(forWritingTo: imagePath.url)
try handle.truncate(atOffset: requestedSize)
try handle.close()
let process = Process()
process.executableURL = URL(fileURLWithPath: mke2fsPath)
process.arguments = ["-F", "-q", "-t", "ext4", imagePath.string]
let stderrPipe = Pipe()
process.standardError = stderrPipe
process.standardOutput = Pipe()
try process.run()
process.waitUntilExit()
guard process.terminationStatus == 0 else {
let err = String(decoding: stderrPipe.fileHandleForReading.readDataToEndOfFile(), as: UTF8.self)
struct MkE2FsError: Swift.Error, CustomStringConvertible {
let description: String
}
throw MkE2FsError(description: "mke2fs failed: \(err)")
}
}
func createImage(for testCase: E2FsckImageCase) throws {
let formatter = try EXT4.Formatter(
testCase.imagePath,
minDiskSize: testCase.requestedSize
)
if testCase.contentSize > 0 {
let chunkSize = min(testCase.contentSize, 1.mib())
let chunk = Data(repeating: 0x41, count: Int(chunkSize))
let stream = RepeatingDataStream(chunk: chunk, totalBytes: testCase.contentSize)
try formatter.create(
path: FilePath("/content.bin"),
mode: EXT4.Inode.Mode(.S_IFREG, 0o644),
buf: stream
)
}
try formatter.close()
}
func runE2Fsck(on imagePath: FilePath) throws -> E2FsckResult {
let process = Process()
process.executableURL = URL(fileURLWithPath: e2fsckPath)
process.arguments = ["-fn", imagePath.string]
let stdoutPipe = Pipe()
let stderrPipe = Pipe()
process.standardOutput = stdoutPipe
process.standardError = stderrPipe
try process.run()
process.waitUntilExit()
let stdoutData = stdoutPipe.fileHandleForReading.readDataToEndOfFile()
let stderrData = stderrPipe.fileHandleForReading.readDataToEndOfFile()
return E2FsckResult(
exitCode: process.terminationStatus,
stdout: String(decoding: stdoutData, as: UTF8.self),
stderr: String(decoding: stderrData, as: UTF8.self)
)
}
func deleteImage(at imagePath: FilePath) throws {
if FileManager.default.fileExists(atPath: imagePath.string) {
try FileManager.default.removeItem(at: imagePath.url)
}
}
func imageSize(at imagePath: FilePath) throws -> UInt64 {
let handle = try FileHandle(forReadingFrom: imagePath.url)
return try handle.seekToEnd()
}
func allocatedMiB(at imagePath: FilePath) throws -> UInt64 {
let ext4 = try EXT4.EXT4Reader(blockDevice: imagePath)
let sb = ext4.superBlock
let blocksCount = UInt64(sb.blocksCountLow) | (UInt64(sb.blocksCountHigh) << 32)
let freeBlocks = UInt64(sb.freeBlocksCountLow) | (UInt64(sb.freeBlocksCountHigh) << 32)
let usedBlocks = blocksCount - freeBlocks
let usedBytes = usedBlocks * UInt64(sb.blockSize)
return usedBytes / 1024 / 1024
}
// Physical (used) and logical (file) size for any existing ext4 image, ours or a reference.
func footprint(at imagePath: FilePath) throws -> (logicalBytes: UInt64, physicalBytes: UInt64) {
let logicalBytes = try imageSize(at: imagePath)
let ext4 = try EXT4.EXT4Reader(blockDevice: imagePath)
let sb = ext4.superBlock
let blocksCount = UInt64(sb.blocksCountLow) | (UInt64(sb.blocksCountHigh) << 32)
let freeBlocks = UInt64(sb.freeBlocksCountLow) | (UInt64(sb.freeBlocksCountHigh) << 32)
let physicalBytes = (blocksCount - freeBlocks) * UInt64(sb.blockSize)
return (logicalBytes, physicalBytes)
}
}
final class RepeatingDataStream: ReadableStream {
private let chunk: Data
private let totalBytes: UInt64
private var writtenBytes: UInt64 = 0
private var chunkOffset: Int = 0
init(chunk: Data, totalBytes: UInt64) {
self.chunk = chunk
self.totalBytes = totalBytes
}
func read(_ buffer: UnsafeMutablePointer<UInt8>, maxLength len: Int) -> Int {
guard writtenBytes < totalBytes else {
return 0
}
let remaining = Int(min(UInt64(len), totalBytes - writtenBytes))
let bytes = chunk.withUnsafeBytes { chunkBytes in
guard let base = chunkBytes.baseAddress?.assumingMemoryBound(to: UInt8.self) else {
return 0
}
var copied = 0
while copied < remaining {
let available = chunk.count - chunkOffset
let toCopy = min(remaining - copied, available)
buffer.advanced(by: copied).update(from: base.advanced(by: chunkOffset), count: toCopy)
copied += toCopy
chunkOffset = (chunkOffset + toCopy) % chunk.count
}
return copied
}
writtenBytes += UInt64(bytes)
return bytes
}
}
private func formatBytes(_ bytes: UInt64) -> String {
let units: [(UInt64, String)] = [(1.gib(), "GiB"), (1.mib(), "MiB"), (1.kib(), "KiB")]
for (size, label) in units where bytes >= size && bytes % size == 0 {
return "\(bytes / size)\(label)"
}
return "\(bytes)B"
}
private func formatCount(_ n: UInt64) -> String {
let formatter = NumberFormatter()
formatter.numberStyle = .decimal
return formatter.string(from: NSNumber(value: n)) ?? "\(n)"
}
@Test func e2FsckMatrixComparesAgainstMke2fs() throws {
let harness = EXT4E2FsckHarness()
let cases: [(name: String, requested: UInt64, content: UInt64)] = [
// ("32KiB_none", 32.kib(), 0),
// ("64MiB_none", 64.mib(), 0),
("128MiB_none", 128.mib(), 0),
("1GiB_none", 1.gib(), 0),
("4GiB_none", 4.gib(), 0),
("8GiB_none", 8.gib(), 0),
("16GiB_none", 16.gib(), 0),
]
let root = FileManager.default.temporaryDirectory
.appendingPathComponent("ext4-mke2fs-compare-\(UUID().uuidString)", isDirectory: true)
try FileManager.default.createDirectory(at: root, withIntermediateDirectories: true)
defer { try? FileManager.default.removeItem(at: root) }
for testCase in cases {
let ourImage = FilePath(root.appendingPathComponent("\(testCase.name)-ours.img", isDirectory: false))
let refImage = FilePath(root.appendingPathComponent("\(testCase.name)-ref.img", isDirectory: false))
try harness.createImage(
for: E2FsckImageCase(name: testCase.name, imagePath: ourImage, requestedSize: testCase.requested, contentSize: testCase.content)
)
try harness.createReferenceImage(requestedSize: testCase.requested, at: refImage)
let fsckResult = try harness.runE2Fsck(on: ourImage)
let ours = try harness.footprint(at: ourImage)
let reference = try harness.footprint(at: refImage)
let sizePass = ours.logicalBytes == testCase.requested
let fsckPass = fsckResult.passed
let logicalPass = ours.logicalBytes == reference.logicalBytes
let physicalPass = ours.physicalBytes <= reference.physicalBytes
print("Requested Size: \(formatBytes(testCase.requested))")
print("Content: \(testCase.content > 0 ? formatBytes(testCase.content) : "none")")
print("Size: \(sizePass ? "PASS" : "FAIL")")
print("e2fsck: \(fsckPass ? "PASS" : "FAIL")")
print("Our logical size: \(formatBytes(ours.logicalBytes)), mke2fs logical size: \(formatBytes(reference.logicalBytes))")
print("Our physical size: \(formatBytes(ours.physicalBytes)), mke2fs physical size: \(formatBytes(reference.physicalBytes))")
print("Logical Size vs mke2fs: \(logicalPass ? "PASS" : "FAIL")")
print("Physical Size vs mke2fs: \(physicalPass ? "PASS" : "FAIL")")
if !physicalPass {
print("physical <= mke2fs = false; physical = \(formatCount(ours.physicalBytes)); mke2fs = \(formatCount(reference.physicalBytes))")
}
#expect(sizePass)
#expect(fsckPass)
#expect(physicalPass, "physical <= mke2fs = false; physical = \(ours.physicalBytes); mke2fs = \(reference.physicalBytes)")
}
}@dkovba please check when it is convenient for you also please reply to comment |
siddh34
force-pushed
the
fix/memoryOverAllocation
branch
from
August 15, 2026 15:25
0594735 to
c167cb3
Compare
siddh34
force-pushed
the
fix/memoryOverAllocation
branch
from
August 15, 2026 17:23
c167cb3 to
3d888ee
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This resolves issue #647
This is fix for memory over allocation due to last partial block
I have explicitly laid out metadata for extra/last groups
Modifications:
Packed-region math is computation starts at line 718
function to calculate blocks in last group at line 58
while loop here didn't made any sense so replaced it with if check line 914