chore: decompress options, customDictionary exposed

[kyr0]

@pimterry Hi Tim, this PR is not ready to merge; I just wanted to share my work results with you for general review.

The following issues in my downstream project caused it:

• customDictionary usage does not work kyr0/brotli-compress#5
• Memory leak with repeated compression kyr0/brotli-compress#2

Decompress options, Memory Management, Updated Deps and some work on customDictionary support and verification of its proper impl. solved all of my downstream issues in my fork.

I'd like to contribute whatever you find useful. Please tell me if this project is still maintained and which parts you'd like to accept. I'd willing to create actually useful small PRs by your suggestions.

Thank you for this project and your hard work.

Best,
Aron

[CLAassistant]

All committers have signed the CLA.

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	cargo/​serde@​1.0.136 ⏵ 1.0.229	+35
	cargo/​serde-wasm-bindgen@​0.6.5
	cargo/​alloc-no-stdlib@​2.0.3 ⏵ 2.0.4
	cargo/​alloc-stdlib@​0.2.1 ⏵ 0.2.4			+4
	cargo/​brotli-decompressor@​4.0.0 ⏵ 5.0.3

View full report

[pimterry]

Sure, I'm open to this. Can you split it up for review separately? I think there's at least 3 PRs in here:

• The serde package migration - fairly mechanical
• The brotli package bump - multiple major versions, so this needs some thorough testing to make sure this doesn't change anything unexpectedly
• Custom dictionary support - the big bit for the new feature

Definitely happy to include custom dictionaries, medium term that's something I'm interested in using too.

You mentioned memory management fixes, can you explain where those are in here?

[kyr0]

@pimterry Sure, it will be a pleasure. I'm trying to manage it in the upcoming days, but it could take a bit longer as I'm working on a few bigger projects in parallel currently.

As my package currently uses my fork, I can at least for that specific rev say that the memory management issues reported earlier with the old WASM build my package was using simply disappeared with the new one; I cannot pinpoint the root cause right now, but it's simply not happening anymore. It's been a few years since I first implemented my package and linked it against an ancient rev of your nice work here -- so it could have a bunch of possible root causes that the memory management now works surprisingly well (including code changes in this repo, or even low-level WASM gen/bind generation). I don't have the bandwidth right now for cross-evaluations and some kind of PCA on this; but I guess we can simply run my new memory benchmark on the bundle and call it a day after verification that it doesn't leak - even without an exact root cause known. Call me lazy but if there is not issue... :D simple script: https://github.com/kyr0/brotli-compress/blob/main/bench/memory.mjs

Btw for custom dictionary, one user already reported that it works fine now for them, providing more anecdotal evidence, underpinning my e2e tests... kyr0/brotli-compress#5 (comment)

btw thanks again for your pioneering work back in the day!

[socket-security]

Warning

Review the following alerts detected in dependencies.

According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.

Action	Severity	Alert  (click "▶" to expand/collapse)
Warn		Critical CVE: node-npm tar: Decompression/parse DoS via unlimited input CVE: GHSA-23hp-3jrh-7fpw node-tar: Decompression/parse DoS via unlimited input (CRITICAL) Affected versions: < 7.5.19 Patched version: 7.5.19 From: ? → npm/wasm-pack@0.15.0 → npm/tar@7.5.16 ℹ Read more on: This package | This alert | What is a critical CVE? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Remove or replace dependencies that include known critical CVEs. Consumers can use dependency overrides or npm audit fix --force to remove vulnerable dependencies. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/tar@7.5.16. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

View full report