feat: change retriever output to jsonl

[wes-mil]

Description

Resolves:

Type of Change

• Chore (a change that does not modify the application functionality)
• Bug fix (a change that fixes an issue)
• New feature / enhancement (a change that adds new functionality)
• Refactor (no behaviour change)
• Test coverage
• Build / CI / tooling
• Documentation

Testing

• Unit tests added / updated
• Integration tests added / updated
• Full test suite run (make test_all with CONNECTION_STRING set)

Screenshots (if appropriate):

Driver Impact

• PostgreSQL driver (drivers/pg)
• Neo4j driver (drivers/neo4j)

Checklist

• Code is formatted
• All existing tests pass
• go.mod / go.sum are up to date if dependencies changed

Summary by CodeRabbit

• New Features

  • Collection exports now use manifest-based, compressed JSONL fragments.
  • Loading validates fragment integrity, JSONL records, record counts, and graph schemas before writing data.
  • Encrypted archive loading applies the same validation as directory-based loading.

• Documentation

  • Updated Quick Start and collection format documentation with JSONL structure, validation behavior, file metadata, and testing guidance.

• Tests

  • Expanded round-trip coverage across supported backends, including restored properties, relationships, and topology.
  • Added validation coverage for malformed records, checksums, unsafe paths, and large fragments.

[coderabbitai]

Warning

Review limit reached

@wes-mil, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 55 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 79de13ca-b889-4cfd-a74b-948528a39260

📥 Commits

Reviewing files that changed from the base of the PR and between 86c9b27 and 36e9edc.

📒 Files selected for processing (3)

• cmd/retriever/README.md
• retriever/compression.go
• retriever/compression_test.go

Walkthrough

The retriever collection format now uses manifest-based compressed JSONL fragments. Dumping streams records into shards, loading performs integrity and semantic preflight validation, and tests, benchmarks, archive handling, documentation, and integration coverage are updated accordingly.

Changes

Compressed JSONL collection flow

Layer / File(s)	Summary
Manifest contract and JSONL codec retriever/types.go, retriever/compression.go, retriever/*_test.go	Manifest support is restricted to the new JSONL format; fragment records are encoded, decoded, checksummed, sized, and validated as compressed JSONL.
Streaming JSONL dump generation retriever/dump.go, retriever/dump_test.go, cmd/retriever/bench.go	Node and edge scans write records directly into sharded .jsonl fragments and produce manifest metadata when writers close.
Validated streaming load path retriever/load.go, retriever/load_test.go, retriever/load_benchmark_test.go	Loading streams node and edge records through callbacks while checking integrity, phases, identifiers, and manifest counts before database writes.
Archive, documentation, and end-to-end validation retriever/archive_*_test.go, cmd/retriever/README.md, README.md, cmd/retriever/retriever_integration_test.go	Archive behavior, format documentation, benchmarks, and backend-aware dump/load round-trip assertions are updated for JSONL collections.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant RetrieverDump
  participant CompressedJSONLinesWriter
  participant CollectionManifest
  participant VerifyCollectionFragments
  participant GraphDatabase
  RetrieverDump->>CompressedJSONLinesWriter: stream node and edge records
  CompressedJSONLinesWriter->>CollectionManifest: record path, count, sizes, checksum
  VerifyCollectionFragments->>CollectionManifest: read fragment metadata
  VerifyCollectionFragments->>GraphDatabase: stream validated records for loading

Loading

Possibly related PRs

• SpecterOps/DAWGS#103: Both changes update the retriever dump/load fragment encoding and verification pipeline.

Suggested reviewers: zinic

Poem

I’m a rabbit with JSONL cheer,
Shards hop neatly, manifest clear.
Checksums sparkle, records stream,
Loads preflight like a dream.
Nodes then edges, row by row—
Carrots for every graph we grow!

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description matches the template structure but lacks the required change summary and issue reference after 'Resolves:'.	Add a brief summary of what changed and why, and fill in a valid Resolves ticket or issue number.
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: switching retriever output to JSONL.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch retriever-jsonl

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@retriever/compression.go`:
- Around line 251-310: Bound the physical JSONL line size in
readCompressedJSONLinesFromReader so ReadBytes cannot accumulate unbounded
decompressed data; track each record’s bytes and fail immediately with a
descriptive decode error when the configured maximum is exceeded, while
preserving normal parsing and EOF handling for lines within the limit.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 721f583c-4b6a-4726-bc1d-bb56a9ef77c0

📥 Commits

Reviewing files that changed from the base of the PR and between 7f871e7 and 0cb17a9.

📒 Files selected for processing (17)

• README.md
• cmd/retriever/README.md
• cmd/retriever/bench.go
• cmd/retriever/retriever_integration_test.go
• retriever/archive_envelope.go
• retriever/archive_envelope_test.go
• retriever/archive_tar_test.go
• retriever/compression.go
• retriever/compression_test.go
• retriever/doc.go
• retriever/dump.go
• retriever/dump_test.go
• retriever/load.go
• retriever/load_benchmark_test.go
• retriever/load_test.go
• retriever/manifest_test.go
• retriever/types.go

[zinic]

This looks reasonable to me