Skip to content

Compatibility matrix: write from sqlflow, read back from the vendor's engine, in CI #210

Description

@turbolytics

Part of Iceberg v1. The proof. A matrix, not a tutorial.

Why

"Proven with X" is only a claim until the direction is shown: sqlflow writes through the vendor's catalog, and the vendor's own engine reads the rows back. That artifact is what a provider or a partner evaluates, and nobody in the competitive set publishes one.

What

  • A CI workflow, separate from the PR suite, that runs one job per catalog with the vendor account's credentials in repository secrets. Scheduled nightly and on release tags, not on every PR.
  • Each job: create or truncate a table, run a fixed pipeline against a fixed fixture, then query the table from the vendor's engine (Snowflake SQL, Athena or the S3 Tables read path, Databricks SQL) and assert row count and checksum.
  • Versions pinned and recorded in the result: sqlflow, iceberg-go, the catalog's API version, the engine version that read it back.
  • The result rendered as a page on turbolytics.io: catalog, auth mode, partition transforms exercised, read-back engine, versions, last green date. Red rows stay on the page.

Each catalog is its own issue so a red one does not block the others.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions