From 07f0f3969bdc2f3326378fa8343b2573f3d9e39b Mon Sep 17 00:00:00 2001 From: d33bs Date: Wed, 19 Aug 2026 16:28:23 -0600 Subject: [PATCH] add outline --- README.md | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 57 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 282d6a9..d30b94b 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,58 @@ # tesserift -Measure how bioimaging storage layouts shape performance, analysis, and biological discovery. + +Bioimaging data can follow a format specification and still perform poorly. + +We will study around **six public datasets from three sources**, with two datasets from each source: + +* **Image Data Resource (IDR):** established OME bioimaging datasets with different image shapes and workloads. +* **SSBD:** multidimensional OME-Zarr data, including 3D and time-series images. +* **Cell Painting Gallery / JUMP:** large-scale imaging data with associated metadata and feature tables. + +Two datasets from each source will help us measure variation both **within** and **between** public data services. + +Unlike general-purpose Zarr/Parquet benchmarks, we tie storage properties to bioimaging-specific workloads and test causality directly, not just correlation. + +The work will focus on four goals: + +* **Measure format skew.** Compare chunking, compression, file counts, metadata, schemas, row groups, and related storage properties. +* **Find costly patterns.** Identify storage properties that predict slow access or analysis. +* **Measure real workloads.** Test image reads, table scans, filtering, joins, and related bioimaging tasks. +* **Turn evidence into rules.** Add tested performance rules to `zarr-lint` and `arrow-lint`. + +Cost means wall-clock latency and throughput, measured on a fixed compute and network setup. + +We will not search for many examples of the same problem. +When we find an important pattern, we will create controlled versions of a dataset that change that property. + +This lets us test whether the storage difference actually causes the observed performance change. + +The main question is: + +**How much do storage layouts vary across public bioimaging resources, and can that variation predict the cost of accessing and analyzing the data?** + +The manuscript will also examine how these costs affect interactive exploration and large-scale biological analysis. + +## Operational outline + +Keep the complete study in one public GitHub repository. + +* Write the manuscript with **Quarto Manuscripts**. +* Keep analysis and benchmark code beside the manuscript. +* Store generated figures and tables in the repository. +* Make benchmarks reproducible from scripts or workflows. +* Record dataset versions, inputs, and benchmark settings. +* Develop the manuscript and analysis together. +* Publish the completed manuscript as a **bioRxiv preprint**. +* Select a **right-size peer-reviewed journal** after the results and scope are clear. + +The repository becomes the manuscript, analysis, figures, and reproducibility record. + +## Timeline + +Rough estimate, ~9 months total: + +* Months 1-2: select datasets, set up benchmarking infrastructure. +* Months 3-4: measure format skew and workload cost. +* Months 5-6: build controlled dataset variants, test causality. +* Months 7-8: draft the manuscript, publish the bioRxiv preprint. +* Month 9: submit to a right-size journal.