[sureshanaparti]

Description

This PR fixes FKVM: createVolume / createTemplate from incremental snapshot fails randomly, and managesnapshot.sh skips mkdir.

Always create the snapshot backup dir, surface qemu-img and agent errors

Fixes #14485

With kvm.incremental.snapshot=true and snapshot.backup.to.secondary=false, createVolume / createTemplate from an incremental snapshot fail randomly with "Failed to copy snapshot: ... InvocationTargetException when invoking RPC callback for command: copySnapshotAsyncCallback".

Changes:

• managesnapshot.sh backup_snapshot() now always runs mkdir -p on the destination. The SSVM removes snapshots/<account>/<volume> after a temporary copy is used, and the host's NFS client can still report the
  directory as present from its dentry cache, so the old [ ! -d ] guard skipped mkdir and qemu-img failed with ENOENT. mkdir forces a server-side revalidation, so this is reliable.
• qemu-img's stderr is kept in the error returned to the management server instead of being sent to /dev/null.
• SnapshotServiceImpl.copySnapshotAsyncCallback no longer NPEs when the source snapshot has no CreateSnapshotPayload (the copy from snapshot flow). The agent's error now reaches the API response and the failure cleanup runs.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

[codecov]

Codecov Report

❌ Patch coverage is 0% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 18.32%. Comparing base (d02adcd) to head (0e55bf8).

Files with missing lines	Patch %	Lines
...oudstack/storage/snapshot/SnapshotServiceImpl.java	0.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##               4.22   #14568   +/-   ##
=========================================
  Coverage     18.32%   18.32%           
+ Complexity    16644    16643    -1     
=========================================
  Files          5939     5939           
  Lines        537768   537768           
  Branches      65929    65929           
=========================================
+ Hits          98551    98569   +18     
+ Misses       428008   427989   -19     
- Partials      11209    11210    +1     

Flag	Coverage Δ
uitests	4.08% <ø> (ø)
unittests	19.42% <0.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

🟡 Changes recommended

The change has an active moderate issue because diagnostic text is used as a printf format string, allowing percent directives in mkdir errors to corrupt the reported failure message.

1 open issue

• Print mkdir diagnostics with a constant printf format · New

What changed in this PR

This PR fixes intermittent KVM volume/template creation failures from incremental snapshots by reliably recreating backup directories, preserving qemu-img diagnostics, and handling snapshot-copy failures when no creation payload is present.

Changes:

• Always runs mkdir -p before snapshot backup.
• Propagates qemu-img and mkdir errors in backup failures.
• Handles missing CreateSnapshotPayload during asynchronous copy failures.
• Adds regression coverage for error propagation and failure cleanup.

File	Summary	Review findings
scripts/​storage/​qcow2/​managesnapshot.sh	Recreates backup directories and reports backup command errors.	Moderate (4 votes): Diagnostic messages are interpolated into printf format strings, allowing % directives to be interpreted. Moderate (1 vote): The LVM-backed branch, including its -U fallback, still discards qemu-img stderr.
engine/​storage/​snapshot/​src/​main/​java/​org/​apache/​cloudstack/​storage/​snapshot/​SnapshotServiceImpl.java	Handles failed copy callbacks when the source snapshot has no payload.	No findings.
engine/​storage/​snapshot/​src/​test/​java/​org/​apache/​cloudstack/​storage/​snapshot/​SnapshotServiceImplTest.java	Tests failure propagation and cleanup for copies without a payload.	No findings.

_{Review effort: Lite}

[sureshanaparti]

@blueorangutan package

[blueorangutan]

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19891

[Damans227]

@sureshanaparti if this copy fails during createVolume or createTemplate, does the users snapshot get deleted now? looks like the cleanup below removes the snapshot itself, not just the temp copy

[Damans227]

Tested on a KVM lab (OL9 host) with kvm.incremental.snapshot=true and snapshot.backup.to.secondary=false, and a second mount of the secondary storage on the host, same as #14485.

Test:

Take a ROOT snapshot of a running VM. Then each round take a new snapshot and run createVolume and createTemplate from it.

Before: 4.22 jar and script. createTemplate failed in 3 of 3 rounds.

errorcode 530: InvocationTargetException when invoking RPC callback for command: copySnapshotAsyncCallback
Failed to backup snaptshot: Failed to backup bfe54f89-9681-46d0-b0ab-edb5c9874dce for disk /mnt/d105083a-b130-30c5-9209-beac5f87c96d/snapshots/bfe54f89-9681-46d0-b0ab-edb5c9874dce to /mnt/a6292620-e487-3a1b-a47e-1a126d748a50/snapshots/2/3

After: 11 rounds, 0 failures on createVolume and createTemplate.

I also made the copy step on the host fail on purpose, then ran createVolume from a new snapshot. The volume fails as expected, but the snapshot is gone too. listSnapshots returns nothing, the row is marked removed, and the file is still on primary storage. On 4.22 the snapshots from the failed copies above stayed BackedUp.

agent: Failed to backup snaptshot: forced failure for test
snapshots: 17  79d36031-3591-41ab-a9d7-785a391bda68  Error  removed 2026-11-05 19:27:19
snapshot_store_ref: 42  Primary  Destroyed  /mnt/d105083a-b130-30c5-9209-beac5f87c96d/snapshots/fd972788-d100-4f37-ad6f-f21340c63da6

@sureshanaparti can we keep the snapshot when only the copy fails? thats the case from my question on line 497.