Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 57 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,58 @@
# tesserift
Measure how bioimaging storage layouts shape performance, analysis, and biological discovery.

Bioimaging data can follow a format specification and still perform poorly.

We will study around **six public datasets from three sources**, with two datasets from each source:

* **Image Data Resource (IDR):** established OME bioimaging datasets with different image shapes and workloads.
* **SSBD:** multidimensional OME-Zarr data, including 3D and time-series images.
* **Cell Painting Gallery / JUMP:** large-scale imaging data with associated metadata and feature tables.

Two datasets from each source will help us measure variation both **within** and **between** public data services.

Unlike general-purpose Zarr/Parquet benchmarks, we tie storage properties to bioimaging-specific workloads and test causality directly, not just correlation.

The work will focus on four goals:

* **Measure format skew.** Compare chunking, compression, file counts, metadata, schemas, row groups, and related storage properties.
* **Find costly patterns.** Identify storage properties that predict slow access or analysis.
* **Measure real workloads.** Test image reads, table scans, filtering, joins, and related bioimaging tasks.
* **Turn evidence into rules.** Add tested performance rules to `zarr-lint` and `arrow-lint`.

Cost means wall-clock latency and throughput, measured on a fixed compute and network setup.

We will not search for many examples of the same problem.
When we find an important pattern, we will create controlled versions of a dataset that change that property.

This lets us test whether the storage difference actually causes the observed performance change.

The main question is:

**How much do storage layouts vary across public bioimaging resources, and can that variation predict the cost of accessing and analyzing the data?**

The manuscript will also examine how these costs affect interactive exploration and large-scale biological analysis.

## Operational outline

Keep the complete study in one public GitHub repository.

* Write the manuscript with **Quarto Manuscripts**.
* Keep analysis and benchmark code beside the manuscript.
* Store generated figures and tables in the repository.
* Make benchmarks reproducible from scripts or workflows.
* Record dataset versions, inputs, and benchmark settings.
* Develop the manuscript and analysis together.
* Publish the completed manuscript as a **bioRxiv preprint**.
* Select a **right-size peer-reviewed journal** after the results and scope are clear.

The repository becomes the manuscript, analysis, figures, and reproducibility record.

## Timeline

Rough estimate, ~9 months total:

* Months 1-2: select datasets, set up benchmarking infrastructure.
* Months 3-4: measure format skew and workload cost.
* Months 5-6: build controlled dataset variants, test causality.
* Months 7-8: draft the manuscript, publish the bioRxiv preprint.
* Month 9: submit to a right-size journal.