data-lifecycle

A curated list of awesome open source tools and commercial products to catalog, version, and manage data 🚀

🦀 A smart, persistent key-value store in Rust for managing data with a lifecycle. Features atomic TTLs, frequency counting, and is built on sled for performance and durability.

KVKK & GDPR compliant data lifecycle management system with automated retention tracking, controlled destruction approval, PDF record generation, and audit logging. Built with FastAPI and React.

Alchemist is an intelligent data transformation engine that turns raw, messy data into clean, curated, and governed datasets. It automates and simplifies the data engineering lifecycle with a focus on quality, lineage, and operational excellence across the modern data ecosystem.

A coursera course on Machine Learning in Production

Free, hands-on data-literacy lab for non-coders — 68 interactive demos across the full data lifecycle: cleaning, analysis, machine learning, generative AI, and the statistical traps that fool decision-makers. Bilingual EN/العربية. No signup, open-source, citable (DOI).

ARGUS Fraud Radar: Enterprise real-time banking fraud detection platform (PySpark, Medallion Architecture, LightGBM, FastAPI & Streamlit). STUCOM Pelai · DAW 2 · UNITA Erasmus+ BIP.

File system reporting for research storage: what is stored, how much, how old, what kind, and who owns it, for multi-petabyte file systems with billions of files. Age maps, change over time, cleanup plans, chargeback, and capacity forecasts, built on fs-inventory scans. Python, FastAPI, DuckDB.

📊 Analyze ClickHouse usage to uncover table relationships, discover safe cleanup options, and generate insightful visual reports effortlessly.

Offline SQLite-backed evaluation suite for agent memory retention, expiration, deletion, tenant separation and conflicting facts.

Practical, code-first guides for automating InfluxDB at IoT scale: task scheduling & orchestration, downsampling pipelines, retention and data-lifecycle management, and external workflow engines (Airflow, Prefect, Dagster, Kubernetes).

RDM course with tactical solutions for researchers and data professionals alike

Enterprise IAM governance pipeline — maps 75 applications to IAM products, classifies risk, detects governance gaps, and produces management dashboards with remediation queues and department scorecards.

The objective of this project was to develop a revenue forecasting solution for a digital wallet company. The primary goal was to analyze historical transaction data, create a predictive model to forecast future revenue, and present these insights through an interactive dashboard.

Chuck Lifecycle Service is the Go based microservice for managing the lifecycle of data chunks across distributed workflows, from creation and validation to processing, tracking, completion, expiration, and cleanup.

Hands-on data technologies administration labs using PostgreSQL and dbt to teach data governance, modeling, lifecycle, security, quality, monitoring, and architecture.

Práctica 1. Tipología y Ciclo de Vida de los Datos. Caso práctico de Web Scraping orientado a aprender a identificar los datos relevantes por un proyecto analítico y usar las herramientas de extracción de datos.

Storage tiering CLI for InfluxDB: archive aged time-series windows to compressed Parquet on S3/R2/MinIO with checksummed manifests, deep verification, guarded expiry and lossless rehydrate.

“A full-stack data lifecycle project for stock market data using Python, MySQL, Feature Engineering, and EDA, focused on FAANG companies.”