#783 Add support for the optional copy flag in the 'raw' format, allowing files to be skipped from copying while still counted in the size stats.

[yruslan]

Overview

Release Notes

• Raw file processing now supports optional copy indicators.
• Files marked as not requiring copying are skipped, while eligible files are copied as before.
• Processing statistics now report the total source size handled.

Related

Closes #783

Summary by CodeRabbit

• New Features

  • Raw file processing now supports optional copy indicators.
  • Files marked as not requiring copying are skipped, while eligible files are copied as before.
  • Processing statistics now report the total source size handled.

• Bug Fixes

  • Copy failures continue to generate warnings without stopping other file processing.
  • Invalid raw file data is rejected with clear validation errors.

• Tests

  • Added coverage for copy flags, skipped files, copied files, and input validation.

[coderabbitai]

Warning

Review limit reached

@yruslan, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 46 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 24b53412-2d12-4a7d-88f3-7f36a9b4b71e

📥 Commits

Reviewing files that changed from the base of the PR and between de6606b and 78f218a.

📒 Files selected for processing (2)

• pramen/core/src/main/scala/za/co/absa/pramen/core/metastore/peristence/MetastorePersistenceRaw.scala
• pramen/core/src/main/scala/za/co/absa/pramen/core/metastore/peristence/RawFile.scala

Walkthrough

RawFile parses file paths and copy flags from raw DataFrames. saveTable skips files with copy = false, tracks processed source size, and retains warning handling for copy failures. Tests cover parsing, validation, filtering, and copying.

Changes

Raw file copying

Layer / File(s)	Summary
Raw file contract and parsing pramen/core/src/main/scala/za/co/absa/pramen/core/metastore/peristence/RawFile.scala, pramen/core/src/test/scala/za/co/absa/pramen/core/metastore/persistence/RawFileSuite.scala	Adds RawFile and RawFile.fromDf. The parser validates the path column, reads the optional copy column, and defaults copying to true.
Conditional persistence and statistics pramen/core/src/main/scala/za/co/absa/pramen/core/metastore/peristence/MetastorePersistenceRaw.scala, pramen/core/src/test/scala/za/co/absa/pramen/core/metastore/persistence/MetastorePersistenceRawSuite.scala	Updates saveTable to copy only selected files, track processed source size, expose RAW_COPY_FIELD_KEY, and test filtered output.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant InputDataFrame
  participant MetastorePersistenceRaw
  participant SourceFilesystem
  participant PartitionDirectory
  participant MetaTableStats
  InputDataFrame->>MetastorePersistenceRaw: call saveTable
  MetastorePersistenceRaw->>SourceFilesystem: read source file size
  MetastorePersistenceRaw->>PartitionDirectory: copy files with needsCopying=true
  MetastorePersistenceRaw->>MetaTableStats: record processed source size
  MetastorePersistenceRaw-->>InputDataFrame: complete saveTable

Loading

Poem

A rabbit checks each file with care,
Copies only those marked “share.”
The skipped ones stay where they lay,
While sizes count the source array.
Tests hop through the new pathway.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly summarizes support for the optional copy flag and its effect on copying and size statistics.
Description check	✅ Passed	The description includes the required sections, release notes, and related issue, but the Overview section is empty.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feature/783-raw-to-raw-transformer-without-additional-copy

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@pramen/core/src/main/scala/za/co/absa/pramen/core/metastore/peristence/MetastorePersistenceRaw.scala`:
- Around line 101-104: Update the pattern match in
MetastorePersistenceRaw.saveTable around fsUtilsTrg.copyFileWithRetry so every
Some(ex) result is handled, including exceptions whose getMessage is null.
Record the exception’s message when available and use a non-null fallback value
otherwise, preserving Seq.empty[String] for None and preventing MatchError.

In
`@pramen/core/src/main/scala/za/co/absa/pramen/core/metastore/peristence/RawFile.scala`:
- Around line 43-44: Update the RawFile mapping in the branch that selects
RAW_COPY_FIELD_KEY to treat null copy values as true before constructing
RawFile, matching the missing-column behavior. Add a RawFileSuite test covering
a nullable copy column with a null value and assert it produces a RawFile whose
copy flag is true.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: d66b12db-326b-4436-ba34-798a76c35c45

📥 Commits

Reviewing files that changed from the base of the PR and between c722ba8 and de6606b.

📒 Files selected for processing (4)

• pramen/core/src/main/scala/za/co/absa/pramen/core/metastore/peristence/MetastorePersistenceRaw.scala
• pramen/core/src/main/scala/za/co/absa/pramen/core/metastore/peristence/RawFile.scala
• pramen/core/src/test/scala/za/co/absa/pramen/core/metastore/persistence/MetastorePersistenceRawSuite.scala
• pramen/core/src/test/scala/za/co/absa/pramen/core/metastore/persistence/RawFileSuite.scala

[github-actions]

Unit Test Coverage

Overall Project	76.91% -0.13%	🍏
Files changed	45.45%	❌

Module	Coverage
pramen:core Jacoco Report	77.9% -0.14%	❌

Files

Module	File	Coverage
pramen:core Jacoco Report	RawFile.scala	100%	❌
	MetastorePersistenceRaw.scala	95.19% -1.47%	❌