Orphaned snapshots remain after deleting volumes in CLVM_NG primary storage

[mbertolina]

problem

Issue: Orphaned snapshots remain after deleting volumes in CLVM_NG primary storage

I'm seeing an issue with CLVM_NG primary storage where snapshots persist for volumes that have already been deleted.

Reproduction steps:

1. Deploy a VM with a single root volume that has an associated volume snapshot.
2. Create a new volume from that snapshot.
3. Detach the root volume from the VM.
4. Attach the newly created volume to the VM.
5. Delete the original volume (the one with the associated snapshot).

Observed behavior:
The snapshot remains in the system, still linked to the deleted volume.

Additional observations:

• When deleting the volume, I can confirm that the LVM logical volumes (LVs) associated with both the root volume and its snapshot are deleted on the KVM hosts. So the underlying storage is cleaned up correctly.
• However, the snapshot entry is not removed from the database/GUI
• If I then manually delete the snapshot from the "Volume Snapshot" section, the operation reports success and the snapshot is no longer displayed in the GUI. This suggests the deletion works, but only when triggered explicitly against the snapshot record.

Expected behavior:
Deleting a volume should also remove (or orphan-clean) any snapshots associated with it, or at minimum the snapshot should no longer reference a non-existent volume.

versions

ACS: 4.23
Primary storage: CLVM_NG
Hypervisor: Rhel9 KVM

[mbertolina]

It may be related to #13331

[DaanHoogland]

@mbertolina , I think this is expected behaviour. After you delete a volume you can still create a new volume from any snapshot that was created from the old one, and the snapshot is expected to be deleted as a separate entity.

[mbertolina]

@DaanHoogland thanks for your reply. What is the reason of having an snapshot with a reference to a deleted volume?

Expected behavior:
Deleting a volume should also remove (or orphan-clean) any snapshots associated with it, or at minimum the snapshot should no longer reference a non-existent volume.

[DaanHoogland]

@DaanHoogland thanks for your reply. What is the reason of having an snapshot with a reference to a deleted volume?

Expected behavior: Deleting a volume should also remove (or orphan-clean) any snapshots associated with it, or at minimum the snapshot should no longer reference a non-existent volume.

that last bit could be discussed, but on the other hand a volume is re-storable if it has not yet been garbage collected. Volumes and snapshots are first class citizens. Snapshots have a lifecycle not necessarily links to the volume they where created from, but thay are still created from the volume (even if it doesn't anymore exist) If this would lead to bugs these need to be addressed of course. Did you encounter any, @mbertolina ?

[nickaccuweb]

@mbertolina

I checked the CloudStack 4.23 documentation, and this appears to be expected behavior.

CloudStack treats volumes and their snapshots as separate resources. Deleting a volume does not automatically delete its snapshots, as they may still be needed to restore data or create another volume.

The same behavior was reported in GitHub issue #14154, where a CloudStack contributor confirmed that snapshots are expected to remain available after the original volume is deleted. The issue was subsequently marked as not a bug.

Since you've confirmed that the underlying LVM volumes are being removed correctly, I would suggest checking whether the remaining snapshot is still available on secondary storage or its configured backup location.

Could you try creating a new volume from that snapshot after deleting the original volume?

If the new volume is created successfully, the behavior is consistent with CloudStack's snapshot lifecycle. You can then delete the snapshot separately from the Volume Snapshots section when it is no longer required.

However, if the snapshot cannot be restored or causes errors because the original volume no longer exists, that would indicate a separate issue worth investigating.

Reference:
#14154

https://docs.cloudstack.apache.org/en/latest/adminguide/storage.html#volume-deletion-and-garbage-collection

Nick R
AccuWeb.Cloud