create-env: failed VM recreate causes `DeleteUnused` to deregister the in-use stemcell image

[neddp]

Summary

When bosh create-env fails to boot the replacement VM, the state file (bosh-state.json) is left with current_stemcell_id: "". On the next create-env run, DeleteUnused sees an empty current pointer and treats every stemcell record as unused - including the one whose IaaS image (e.g. an AWS AMI) is still actively referenced by the live deployment. The image is deregistered from the IaaS, and all subsequent attempts to create VMs from it fail.

Steps to reproduce

1. Run bosh create-env targeting an environment whose VM needs to be recreated (e.g. a stemcell version bump or a manifest change).
2. The new VM is created in the IaaS but the BOSH agent fails to respond (agent timeout, network issue, etc.) - create-env exits with an error.
3. Inspect bosh-state.json: current_stemcell_id is now "" even though the stemcell record is still listed under stemcells[] and its IaaS image is still in use.
4. Run bosh create-env again (retry).
5. DeleteUnused runs at the end of the deploy phase and deletes the IaaS image for the previous stemcell because current_stemcell_id is empty.
6. Any subsequent create_vm CPI call that references that image fails:

CPI 'create_vm' method responded with error:
CmdError{"type":"Bosh::Clouds::CloudError","message":"could not find AMI 'ami-xxxxxxxxxxxxxxxxx'","ok_to_retry":false}

Root cause

vm.Delete() in deployment/vm/vm.go calls stemcellRepo.ClearCurrent() unconditionally whenever the old VM is deleted, including during the delete-then-recreate cycle inside create-env:

// deployment/vm/vm.go
err = vm.stemcellRepo.ClearCurrent()
if err != nil {
    return bosherr.WrapError(err, "Clearing current stemcell from stemcell repo")
}

The intended design (introduced in 2014, commit bd573fe8) was a three-step sequence:

1. vm.Delete() - clear current_stemcell_id (mark old stemcell as "not current")
2. New VM boots → cloudStemcell.PromoteAsCurrent() - set current_stemcell_id to the new stemcell
3. stemcellManager.DeleteUnused() - reap all non-current stemcell records

On the happy path the clear in step 1 is immediately overwritten by PromoteAsCurrent in step 2, so it has no observable effect. However, when step 2 fails (VM never boots, agent never responds), PromoteAsCurrent is never reached. bosh-state.json is persisted with current_stemcell_id: "" while the stemcell record and its IaaS image remain.

On the next create-env run, FindUnused in stemcell/manager.go returns all stemcell records when current_stemcell_id is empty:

// stemcell/manager.go
for _, stemcellRecord := range stemcellRecords {
    if !found || stemcellRecord.ID != currentStemcellRecord.ID {
        // 'found' is false when current_stemcell_id is "" → every record is treated as unused
        unusedStemcells = append(unusedStemcells, ...)
    }
}

DeleteUnused then calls cloud.DeleteStemcell on the in-use image, permanently deregistering it from the IaaS.

Why the clear is redundant on the success path

PromoteAsCurrent unconditionally overwrites current_stemcell_id with the new stemcell's ID after the VM boots. Whether or not vm.Delete() cleared the pointer first makes no difference on a successful deploy. The clear only has an observable effect in the failure window, and in that window it is purely destructive.

Observed state in bosh-state.json at time of failure

{
  "director_id": "...",
  "current_vm_cid": "",
  "current_stemcell_id": "",
  "current_disk_id": "dcc1b1bb-...",
  "stemcells": [],
  "releases": []
}

The current_stemcell_id being empty while bosh stemcells still shows the stemcell as currently deployed is the tell-tale symptom.

Manual recovery

Re-upload the missing stemcell with --fix to recreate its IaaS image:

bosh -e <director> upload-stemcell <url> --fix

Environment

• Observed on AWS (AMI deregistration), but the code path is IaaS-agnostic - any CPI that implements delete_stemcell is affected.
• bosh-cli versions: traced to the original bosh-micro-cli / bosh-init lineage; present in all current versions.
• The bug is more likely to surface on unattended pipelines where a failed create-env is automatically retried, vs. interactive single-shot usage where the operator would re-supply the stemcell tarball.

History

The stemcellRepo.ClearCurrent() call in vm.Delete() was introduced in commit bd573fe8 ("Delete unused stemcells", November 2014) as part of a feature to reclaim superseded stemcell images after upgrades. The intention was correct; the implementation assumed that the delete→recreate sequence always completes atomically, which is not guaranteed when VM boot fails.