From d17eccb479d2466da537ba1baeb3aa6440b96859 Mon Sep 17 00:00:00 2001 From: Russell Trow Date: Thu, 8 Oct 2026 11:02:12 +0100 Subject: [PATCH] Consolidate storage cleanup patterns into 'Delete or auto-expire unused storage' Merges delete-unused-storage-resources and set-retention-policy-on-storage-resources (same author/date, identical Assumptions) into one pattern covering manual deletion and retention policies, with a prominent data-loss safeguard section and a new Cost Impact section. Originals moved to archive; redirects and persona data updated. Part of #434 (see #385). Co-Authored-By: Claude Sonnet 5.5 --- .../delete-unused-storage-resources.md | 0 ...t-retention-policy-on-storage-resources.md | 0 .../delete-or-auto-expire-unused-storage.md | 73 +++++++++++++++++++ docusaurus.config.js | 6 +- src/data/personas.ts | 7 +- 5 files changed, 80 insertions(+), 6 deletions(-) rename {docs => archive}/operations/resource-lifecycle/delete-unused-storage-resources.md (100%) rename {docs => archive}/operations/resource-lifecycle/set-retention-policy-on-storage-resources.md (100%) create mode 100644 docs/operations/resource-lifecycle/delete-or-auto-expire-unused-storage.md diff --git a/docs/operations/resource-lifecycle/delete-unused-storage-resources.md b/archive/operations/resource-lifecycle/delete-unused-storage-resources.md similarity index 100% rename from docs/operations/resource-lifecycle/delete-unused-storage-resources.md rename to archive/operations/resource-lifecycle/delete-unused-storage-resources.md diff --git a/docs/operations/resource-lifecycle/set-retention-policy-on-storage-resources.md b/archive/operations/resource-lifecycle/set-retention-policy-on-storage-resources.md similarity index 100% rename from docs/operations/resource-lifecycle/set-retention-policy-on-storage-resources.md rename to archive/operations/resource-lifecycle/set-retention-policy-on-storage-resources.md diff --git a/docs/operations/resource-lifecycle/delete-or-auto-expire-unused-storage.md b/docs/operations/resource-lifecycle/delete-or-auto-expire-unused-storage.md new file mode 100644 index 000000000..b0b4ee6db --- /dev/null +++ b/docs/operations/resource-lifecycle/delete-or-auto-expire-unused-storage.md @@ -0,0 +1,73 @@ +--- +version: 1.0 +submitted_by: greenhsu123 +published_date: 2022-11-10 +category: Operations +description: Storage that is no longer used still occupies hardware and carries embodied carbon. Deleting unused storage, either manually or through automated retention policies, reduces the hardware needed to hold data, provided deletion is controlled so that data still needed is not lost. +tags: + - storage + - data-lifecycle + - size:small + - persona:devops-engineer + - persona:data-engineer + - persona:software-engineer +personas: DevOps Engineer, Data Engineer, Software Engineer +--- + +# Delete or auto-expire unused storage + +## Description + +Storage resources such as volumes, snapshots, backups, logs and datasets accumulate over time and are rarely removed. Once nothing reads them, they deliver no value but still occupy physical storage hardware that must be manufactured, powered and eventually replaced. Keeping everything "in case it is needed" means storing data indefinitely, so storage requirements and the embodied carbon attributed to them grow without limit. + +Cleanup is usually left until someone notices the bill, and is often done once rather than as a routine. + +## Solution + +Remove storage that is no longer used, using one or both of two techniques: + +- **Delete unused storage manually.** Periodically identify storage with no recent access or no owning workload, confirm it is not needed, and delete it. This suits one-off resources such as orphaned volumes, old snapshots and abandoned environments. +- **Expire storage automatically with a retention policy.** Define how long each class of data is kept and let the platform delete it when that period ends. This suits data that is produced continuously, such as logs, backups and temporary files, where manual cleanup does not scale. + +Whichever is used, protect against deleting data that is still needed: + +- Assign an owner and a retention period to each class of storage, based on business and legal needs. +- Put a review or approval step before manual deletion. +- Use a soft-delete or grace period before permanent removal, so mistakes can be reversed. +- Take extra care with automated expiry, since it deletes data silently and at scale. Test a new policy on non-critical data first. + +## SCI Impact + +`SCI = (E * I) + M per R` +[Software Carbon Intensity Spec](https://grnsft.org/sci) + +- `M`: Decreases. Deleting unused storage reduces the number of storage devices needed, so less embodied carbon is attributed to the workload. The effect is greatest in shared infrastructure, where freed capacity avoids new hardware. + +Energy (`E`) is also lower in principle, as less data is stored and replicated, but the effect is small compared with the embodied saving. + +## Cost Impact + +- **Storage costs decrease.** Less stored data means lower charges for capacity, and for the replicas, snapshots and backups that scale with it. +- **Operational effort is reduced by automation.** A retention policy removes the recurring effort of manual cleanup, at the cost of defining and maintaining the policy. +- **Recovery and compliance risk can add cost.** Deleting data that later proves necessary can cost far more to recreate than it saved, and some data is subject to legal retention requirements. +- **Early-deletion fees may apply.** Some storage tiers charge a minimum storage duration, so deleting data soon after it is written may not reduce cost. + +## Assumptions + +- Unused storage can be identified, for example through access logs, last-modified dates, or the absence of an owning workload. +- Each class of data has an owner who can say how long it must be kept. +- Legal, regulatory and backup requirements for retention are known and documented before any deletion rule is created. +- The storage platform supports deletion rules, versioning or a soft-delete period, if automated expiry is to be used. + +## Considerations + +- **Deletion can be permanent.** Data deleted in error may be lost for good, and automated policies can delete large volumes before anyone notices. Use grace periods and backups, and start with narrow rules. +- **"Unused" is hard to prove.** Data read rarely, such as audit records or annual reports, can look idle. Check with owners instead of relying on access time alone. +- **Retention periods need maintaining.** A policy set once can go out of date as requirements change, so review policies on a regular schedule. +- **Cold storage may be a better fit.** Where data must be kept but is rarely accessed, moving it to a lower-tier storage class retains it without deleting. This does not reduce the volume stored, so the embodied saving is smaller. +- **Dependencies may not be visible.** Snapshots, images and datasets can be referenced by other systems, so check dependencies before deleting. + +## References + +- [Hardware Efficiency Principle](https://learn.greensoftware.foundation/practitioner/hardware-efficiency) +- Related patterns: [Remove unused assets](/operations/resource-lifecycle/remove-unused-assets), [Optimise storage utilization](/operations/resource-lifecycle/optimise-storage-resource-utilisation) diff --git a/docusaurus.config.js b/docusaurus.config.js index 2604d7306..9e6b4c839 100644 --- a/docusaurus.config.js +++ b/docusaurus.config.js @@ -260,10 +260,12 @@ const config = { { from: "/catalog/cloud/use-circuit-breaker", to: "/operations/capacity-management/use-circuit-breaker" }, // Cloud → Operations: Resource Lifecycle - { from: "/catalog/cloud/delete-unused-storage-resources", to: "/operations/resource-lifecycle/delete-unused-storage-resources" }, + { from: "/operations/resource-lifecycle/delete-unused-storage-resources", to: "/operations/resource-lifecycle/delete-or-auto-expire-unused-storage" }, + { from: "/operations/resource-lifecycle/set-retention-policy-on-storage-resources", to: "/operations/resource-lifecycle/delete-or-auto-expire-unused-storage" }, + { from: "/catalog/cloud/delete-unused-storage-resources", to: "/operations/resource-lifecycle/delete-or-auto-expire-unused-storage" }, { from: "/catalog/cloud/optimise-storage-resource-utilisation", to: "/operations/resource-lifecycle/optimise-storage-resource-utilisation" }, { from: "/catalog/cloud/remove-unused-assets", to: "/operations/resource-lifecycle/remove-unused-assets" }, - { from: "/catalog/cloud/set-retention-policy-on-storage-resources", to: "/operations/resource-lifecycle/set-retention-policy-on-storage-resources" }, + { from: "/catalog/cloud/set-retention-policy-on-storage-resources", to: "/operations/resource-lifecycle/delete-or-auto-expire-unused-storage" }, // Cloud → Operations (root) { from: "/catalog/cloud/scan-for-vulnerabilities", to: "/operations/scan-for-vulnerabilities" }, diff --git a/src/data/personas.ts b/src/data/personas.ts index 9237468c2..88b2a5905 100644 --- a/src/data/personas.ts +++ b/src/data/personas.ts @@ -43,7 +43,7 @@ export const personas: Persona[] = [ { title: "Compress stored data", permalink: "/development/data-handling/compress-stored-data", description: "Storing uncompressed data wastes bandwidth and increases storage capacity requirements; applying appropriate compression reduces both storage consumption and the energy needed to read and write it.", category: "Development", tags: ["cloud","size:medium"] }, { title: "Leverage pre-trained models and transfer learning for AI/ML development", permalink: "/development/pre-trained-transfer-learning", description: "As part of your AI/ML process, you should evaluate using a pre-trained model and use transfer learning to avoid training a new model from scratch.", category: "Development", tags: ["ai","machine-learning","size:small"] }, { title: "Optimise storage utilization", permalink: "/operations/resource-lifecycle/optimise-storage-resource-utilisation", description: "It's better to maximise storage utilisation so the storage layer is optimised for the task, not only in terms of energy proportionality but also in terms of embodied carbon. Two storage units running at low utilization rates will consume more energy than one running at a high utilization rate. In addition, the unused capacity on the underutilised storage unit could be more efficiently used for another task or process.", category: "Operations", tags: ["storage","size:small"] }, - { title: "Set storage retention policies", permalink: "/operations/resource-lifecycle/set-retention-policy-on-storage-resources", description: "From an embodied carbon perspective, it's better to have an automated mechanism to delete unused storage resources so we are efficient with hardware and so that the storage layer is optimised for the task.", category: "Operations", tags: ["storage","size:small"] }, + { title: "Delete or auto-expire unused storage", permalink: "/operations/resource-lifecycle/delete-or-auto-expire-unused-storage", description: "Storage that is no longer used still occupies hardware and carries embodied carbon. Deleting unused storage, either manually or through automated retention policies, reduces the hardware needed to hold data, provided deletion is controlled so that data still needed is not lost.", category: "Operations", tags: ["storage","data-lifecycle","size:small"] }, ], }, { @@ -69,7 +69,7 @@ export const personas: Persona[] = [ { title: "Enable text compression", permalink: "/development/web-performance/enable-text-compression", description: "Web browsers often communicate with web servers in a human readable format. These can be HTML, JavaScript and/or CSS files and REST requests which can return a response in JSON. This human readable communication is redundant and, as such, can be compressed to save bandwidth.", category: "Development", tags: ["web","size:small"] }, { title: "Minimize the total number of deployed environments", permalink: "/development/minimizing-deployed-environments", description: "In a given application, there may be a need to utilize multiple environments in the application workflow. Typically, a development environment is used for regular updates, while staging or testing enviroments are used to make sure there are no issues before code reaches a production environment where users may have access. Each added environment has an increasing energy impact, which in turn creates more emissions. As such, it is important to understand the necessity of each enviroment and it's environmental impact.", category: "Development", tags: ["cloud","deployment"] }, { title: "Terminate TLS at border gateway", permalink: "/development/evaluate-whether-to-use-TLS-termination", description: "Transport Layer Security (TLS) ensures that all data passed between the web server and web browsers remain private and encrypted. However, terminating and re-establishing TLS increases CPU usage and might be unnecessary in certain architectures.", category: "Development", tags: ["cloud","security","compute","kubernetes","role:software-engineer","size:medium"] }, - { title: "Delete unused storage resources", permalink: "/operations/resource-lifecycle/delete-unused-storage-resources", description: "From an embodied carbon perspective, it's better to delete unused storage resources so we are efficient with hardware and so that the storage layer is optimised for the task.", category: "Operations", tags: ["storage","size:small"] }, + { title: "Delete or auto-expire unused storage", permalink: "/operations/resource-lifecycle/delete-or-auto-expire-unused-storage", description: "Storage that is no longer used still occupies hardware and carries embodied carbon. Deleting unused storage, either manually or through automated retention policies, reduces the hardware needed to hold data, provided deletion is controlled so that data still needed is not lost.", category: "Operations", tags: ["storage","data-lifecycle","size:small"] }, { title: "Match utilization requirements of virtual machines (VMs)", permalink: "/operations/capacity-management/match-utilization-requirements-of-vm", description: "It's better to have one VM running at a higher utilization than two running at low utilization rates, not only in terms of energy proportionality but also in terms of embodied carbon. Two servers running at low utilization rates will consume more energy than one running at a high utilization rate. In addition, the unused capacity on the underutilized server could be more efficiently used for another task or process.", category: "Operations", tags: ["compute","size:small"] }, { title: "Match utilization requirements with pre-configured servers", permalink: "/operations/capacity-management/match-utilization-requirements-with-pre-configured-server", description: "It's better to have one VM running at a higher utilization than two running at low utilization rates, not only in terms of energy proportionality but also in terms of embodied carbon. Two servers running at low utilization rates will consume more energy than one running at a high utilization rate. In addition, the unused capacity on the underutilized server could be more efficiently used for another task or process.", category: "Operations", tags: ["compute","size:small"] }, { title: "Optimise storage utilization", permalink: "/operations/resource-lifecycle/optimise-storage-resource-utilisation", description: "It's better to maximise storage utilisation so the storage layer is optimised for the task, not only in terms of energy proportionality but also in terms of embodied carbon. Two storage units running at low utilization rates will consume more energy than one running at a high utilization rate. In addition, the unused capacity on the underutilised storage unit could be more efficiently used for another task or process.", category: "Operations", tags: ["storage","size:small"] }, @@ -80,7 +80,6 @@ export const personas: Persona[] = [ { title: "Scale down kubernetes applications when not in use", permalink: "/operations/capacity-management/scale-down-kubernetes-workloads", description: "In order to reduce carbon emissions and costs, Dev&Test Kubernetes clusters can turn off nodes out of office hours. Thereby, optimization is implemented at the cluster level. For production clusters, where nodes need to stay up and running, optimization needs to be implemented at the application level.", category: "Operations", tags: ["kubernetes","size:small"] }, { title: "Scale infrastructure with user load", permalink: "/operations/capacity-management/scale-infrastructure-with-user-load", description: "Demand for resources depends on user load at any given time. However, most applications run without taking this into consideration. As a result,resources are underused and inefficient.", category: "Operations", tags: ["compute","cloud","size:medium"] }, { title: "Scale Kubernetes workloads based on relevant demand metrics", permalink: "/operations/capacity-management/scale-kubernetes-workloads-based-on-events", description: "By default, Kubernetes scales workloads based on CPU and RAM utilization. In practice, however, it's difficult to correlate your application's demand drivers with CPU and RAM utilization. Scaling your workload based on relevant demand metrics that drive scaling of your applications, such as HTTP requests, queue length, and cloud alerting events can help reduce resource utilization, and therefore also your carbon emissions.", category: "Operations", tags: ["cloud","serverless","kubernetes","role:software-engineer","role:cloud-engineer","size:medium"] }, - { title: "Set storage retention policies", permalink: "/operations/resource-lifecycle/set-retention-policy-on-storage-resources", description: "From an embodied carbon perspective, it's better to have an automated mechanism to delete unused storage resources so we are efficient with hardware and so that the storage layer is optimised for the task.", category: "Operations", tags: ["storage","size:small"] }, { title: "Time-shift Kubernetes cron jobs", permalink: "/operations/time-shift-kubernetes-cron-jobs", description: "The carbon emissions of a software system depends on the power consumed by that software, but also on the Carbon intensity of the electricity it is powered on. For this reason, running energy-efficient software on carbon intensive electricity grid, might be inefficient to reduce its global carbon emissions. Carbon aware time scheduling, is about scheduling workloads to execute, when electricity carbon intensity is low.", category: "Operations", tags: ["kubernetes","size:medium"] }, { title: "Use circuit breaker patterns", permalink: "/operations/capacity-management/use-circuit-breaker", description: "Modern applications need to communicate with other applications on a regular basis. Since these other applications have their own deployment schedule, downtimes and availability, the network connection to these application might have problems. If the other application is not reachable, all network requests against this other application will fail and future network requests are futile.", category: "Operations", tags: ["compute","size:medium"] }, { title: "Encrypt what is necessary", permalink: "/requirements/encrypt-what-is-necessary", description: "Data protection through encryption is a crucial aspect of our security measures. However, the encryption process can be resource-intensive at multiple levels.", category: "Requirements", tags: ["size:medium","cloud","security","role:cloud-engineer"] }, @@ -158,7 +157,7 @@ export const personas: Persona[] = [ { title: "Reduce transmitted data", permalink: "/development/data-handling/reduce-transmitted-data", description: "From an energy-efficiency perspective, it's better to minimize the size of the data transmitted so that less energy is required because the network traffic is reduced.", category: "Development", tags: ["networking","size:small"] }, { title: "Use Asynchronous network calls instead of synchronous", permalink: "/development/use-async-instead-of-sync", description: "When making calls across process boundaries to either databases or file systems or REST APIs, relying on synchronous calls can cause the calling thread to become blocked, putting additional load on the CPU", category: "Development", tags: ["cloud","size:medium"] }, { title: "Use compiled languages", permalink: "/development/media-and-code-efficiency/use-compiled-languages", description: "Interpreted languages re-parse and compile code on every execution, consuming more energy than pre-compiled binaries, which perform compilation once and run more efficiently at runtime.", category: "Development", tags: ["cloud","size:small","programming-language"] }, - { title: "Delete unused storage resources", permalink: "/operations/resource-lifecycle/delete-unused-storage-resources", description: "From an embodied carbon perspective, it's better to delete unused storage resources so we are efficient with hardware and so that the storage layer is optimised for the task.", category: "Operations", tags: ["storage","size:small"] }, + { title: "Delete or auto-expire unused storage", permalink: "/operations/resource-lifecycle/delete-or-auto-expire-unused-storage", description: "Storage that is no longer used still occupies hardware and carries embodied carbon. Deleting unused storage, either manually or through automated retention policies, reduces the hardware needed to hold data, provided deletion is controlled so that data still needed is not lost.", category: "Operations", tags: ["storage","data-lifecycle","size:small"] }, { title: "Remove unused assets", permalink: "/operations/resource-lifecycle/remove-unused-assets", description: "Unused cloud resources such as databases, storage buckets, and compute instances continue to consume energy and generate embodied carbon; identifying and decommissioning them eliminates unnecessary waste.", category: "Operations", tags: ["cloud","size:small"] }, { title: "Use circuit breaker patterns", permalink: "/operations/capacity-management/use-circuit-breaker", description: "Modern applications need to communicate with other applications on a regular basis. Since these other applications have their own deployment schedule, downtimes and availability, the network connection to these application might have problems. If the other application is not reachable, all network requests against this other application will fail and future network requests are futile.", category: "Operations", tags: ["compute","size:medium"] }, { title: "Match your service level objectives to business needs", permalink: "/requirements/match-slo", description: "If service downtimes are acceptable it's better to not strive for highest availability but to design the solution according to real business needs. Lower availability guarantees can help reduce energy consumption by using less infrastructure components.", category: "Requirements", tags: ["Cloud","Reliability/Availability","Monitoring"] },