A practical, engineering-first playbook for storing, compressing, and querying spatial data at cloud scale.
🌐 Read it at data-compression.net »
data-compression.net is a free, deeply-researched knowledge base for the people who move geospatial data for a living — GIS data engineers, Python backend developers, and cloud architects. It answers the questions that decide whether a spatial platform is fast and cheap or slow and expensive:
- Which format should this dataset actually live in — GeoParquet, FlatGeobuf, Cloud-Optimized GeoTIFF, or something legacy?
- What ZSTD level, row-group size, and space-filling-curve ordering keep queries fast without inflating storage bills?
- How do you migrate thousands of Shapefiles and GeoJSON files to columnar formats reliably, with schema mapping, metadata preservation, validation, and safe failure handling?
- How do you query the result in place with DuckDB, AWS Athena, and Trino / Presto — and keep the bytes scanned (and the invoice) under control?
Every guide is written to a production standard: typed, version-pinned Python you can run, reference tables built around real workloads, hand-drawn diagrams, and honest trade-off analysis instead of marketing.
The site is organised into four connected tracks:
| Track | What you'll learn |
|---|---|
| Storage Fundamentals & Format Comparison | Columnar vs streaming formats, WKB geometry encoding, CRS handling, the limits of Shapefile and GeoJSON, Cloud-Optimized GeoTIFF for raster, and cloud cost & storage-lifecycle modelling. |
| Compression, Chunking & Spatial Indexing | Tuning ZSTD levels, sizing row groups, dictionary encoding, quadtree indexing, and Hilbert vs Z-order space-filling curves for fast spatial pruning. |
| Data Conversion & Migration Pipelines | Batch converting legacy data to GeoParquet with Python — schema mapping, metadata preservation, parallelism with Dask, CI/CD validation, cost & observability, and resilient failure routing. |
| Query Engines & Cloud Analytics | Querying GeoParquet directly with the DuckDB spatial extension, AWS Athena, and Trino / Presto — predicate pushdown, partitioning, and controlling bytes scanned. |
- Runnable, not hand-wavy. Code blocks use typed signatures, pinned library versions, and real error handling.
- Workload-first tables. Every comparison table maps a setting to the workload it fits, so you can match your case immediately.
- Custom diagrams. Architecture, data-flow, and decision diagrams are hand-authored SVGs that adapt to light and dark themes.
- Fast and private. The site is a static, offline-ready progressive web app — no trackers, no paywall, instant loads.
- Eleventy (11ty) static site generator
- Markdown content with a custom Markdown-It pipeline (Prism syntax highlighting, accessible FAQ accordions, auto breadcrumbs)
- Hand-authored inline SVG diagrams and icons
- Deployed on Cloudflare Pages as an installable, offline-capable PWA
Issues and suggestions are welcome via the issue tracker.
Note on authorship: all commits to this repository are made solely under the maintainer account (
geospatial-data-compression). Contributions are not attributed to co-authors, and commit metadata carries no additional authors or trailers.
© Geospatial Data Compression. All rights reserved unless stated otherwise.