delta-harness: add OpenHouse behavioral test matrix by mkuchenbecker · Pull Request #682 · linkedin/openhouse

mkuchenbecker · 2026-08-13T20:08:33Z

Summary

Add a self-contained Scala behavioral test harness for OpenHouse and Apache Iceberg.
Exercise DDL, DML, maintenance, branching, WAP, streaming, CDC, undrop, and negative operations across table layouts and data formats.
Localize every case preparation, action, and assertion while preserving the exact 2,574-case catalog.
Run the same scenario sources locally against embedded OpenHouse and expose a publishable module for downstream acceptance tests.

Test plan

Run the delta-harness Gradle module tests on JDK 17.
Verify 2,574 unique case IDs and catalog fingerprint 9e5ec513f2bbc775469154c8d1cf45e14654af2fca0e0f29b4bba6acae286a0a.
Run the targeted embedded surface, fork, and encryption slices.

Adds a self-contained Scala behavioral test harness that characterizes OpenHouse + Apache Iceberg table behavior end-to-end. The harness crosses a large matrix of table layouts (partitioning, MoR/CoW, ordered writes, nested types) with DDL, DML, maintenance, branching/WAP, streaming, and negative-path operations, asserting deltas against observed pre-state so each case holds under any layout. It runs locally against a real embedded OpenHouse catalog (harness/openhouse/Env.scala boots OpenHouseLocalServer + the OpenHouse Spark catalog; see run-openhouse.sh and HARNESS-GUIDE.md). The scenario and framework sources are also structured as a publishable Gradle library module (openhouse-spark-delta-harness_2.12) that excludes the embedded-only Env so downstream environments can supply their own adapter. Genuine product or upstream bugs are tagged in Plan.knownBugs with a prose explanation and skipped rather than silently passed, so the suite stays green while documenting the defect. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Adds TESTING-MATRIX.md, a living reference that explains the harness as a cross product of independent axes (operation family, data file format, partitioning, write mode, schema, preparation lineage, and reference routing). Documents how a case id reads, the CoreTable/NestedTypesTable/TypesTable schemas, the table layouts, the preparation lineages, and each operation family including the DDL sub-families. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Move each test's preparation, action, and assertions into its scenario file so the complete behavior is readable in one place. Keep reusable preparation recipes while creating a fresh table for every case. Preserve the exact 2,574-case catalog, ordering, and known-bug behavior with regression tests for the catalog fingerprint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Explain the scenario-owned test structure, immutable preparations, and fresh-table isolation used by the localized test cases. Describe the matrix as living documentation for the current harness architecture. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Remove the harness guide and testing matrix from the implementation PR so the documentation can be reviewed in a separate stacked change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mkuchenbecker · 2026-08-26T16:20:56Z

+          TablePreparation(
+            layout.label,
+            createAndSeed(layout, 3)
+              .sql("ddl")(table => s"ALTER TABLE $table ADD COLUMN cc int")(),


what ddl does is unclear here. ITs a function.

mkuchenbecker · 2026-08-26T16:21:21Z

+          layout.label.endsWith("/orc"))
+      .flatMap { layout =>
+        val preparations = List(
+          TablePreparation(


Each preparation needs a description.

mkuchenbecker · 2026-08-26T16:21:46Z

+        layout.label.endsWith("/parquet") ||
+          layout.label.endsWith("/orc"))


Why not just ahve a list of layouts?

mkuchenbecker · 2026-08-26T16:22:15Z

+            "ddlConsume:writeOrder."),
+          TablePreparation(
+            layout.label,
+            createAndSeed(layout, 3)


Create and seed should be separate calls. What is seeded should be visible or standard.

Separate table preparations from DML operations so each case shows its starting state, mutation, and relative assertions in one place. Keep feature-owned scenarios in removable RTAS, merge-on-read, and branch layers while preserving the exact ordered 2,572-case catalog. Run the same published sources through the local Gradle task and the acceptance adapter. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Keep the standard branch focused on copy-on-write behavior, shared table preparations, bespoke DDL coverage, and the local execution framework. Remove RTAS, merge-on-read, branch, and WAP scenario ownership from this layer. Pin the resulting ordered standard catalog at 1,181 cases so each child branch can add one reviewable feature delta. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mkuchenbecker · 2026-08-26T23:05:10Z

+    // (b) toJson takes no format-version argument, so the key serializes the same regardless of format version.
+    // (c) The value round-trips through fromJson then toJson.
+    val reparsed = org.apache.iceberg.SchemaParser.fromJson(json)
+    val json2 = org.apache.iceberg.SchemaParser.toJson(reparsed)


Just import. Apply this across all of these deep calls.

Keep runtime case metadata limited to stable identifiers and execution state. Put preparation and test explanations beside their Scala behavior so reviewers can read each case without tracing string registries. Generate a fresh UUID for every table and begin cleanup only after the preparation creates it, which preserves any pre-existing table on a name conflict. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Extract the owned-table cleanup state machine behind a package-private boundary so its failure paths can be tested without starting Spark. Pin conflict preservation, successful cleanup, and suppression of cleanup failure behind the primary test failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Pin the remaining ownership outcome: when the test body succeeds and cleanup fails, the cleanup failure must surface to the runner. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mkuchenbecker and others added 5 commits August 13, 2026 12:47

docs(delta-harness): move guides to stack

4064251

Remove the harness guide and testing matrix from the implementation PR so the documentation can be reviewed in a separate stacked change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mkuchenbecker changed the title ~~delta-harness: OpenHouse Iceberg behavioral test harness + guide~~ delta-harness: add OpenHouse behavioral test matrix Aug 25, 2026

mkuchenbecker commented Aug 26, 2026

View reviewed changes

Comment thread integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlScenarios.scala

mkuchenbecker and others added 2 commits August 26, 2026 12:57

This was referenced Aug 26, 2026

test(delta-harness): add merge-on-read matrix mkuchenbecker/openhouse#38

Draft

test(delta-harness): add RTAS matrix mkuchenbecker/openhouse#39

Draft

test(delta-harness): add branch and WAP matrix mkuchenbecker/openhouse#40

Draft

mkuchenbecker commented Aug 26, 2026

View reviewed changes

mkuchenbecker and others added 3 commits August 27, 2026 18:02

test(delta-harness): cover cleanup failure

808fcc9

Pin the remaining ownership outcome: when the test body succeeds and cleanup fails, the cleanup failure must surface to the runner. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Navigation Menu

Uh oh!

delta-harness: add OpenHouse behavioral test matrix - #682

delta-harness: add OpenHouse behavioral test matrix#682
mkuchenbecker wants to merge 10 commits into
linkedin:mainfrom
mkuchenbecker:mkuchenbecker/delta-harness-oss

mkuchenbecker commented Aug 13, 2026 •

edited

Loading

Uh oh!

mkuchenbecker Aug 26, 2026 •

edited

Loading

Uh oh!

mkuchenbecker Aug 26, 2026

Uh oh!

mkuchenbecker Aug 26, 2026

Uh oh!

mkuchenbecker Aug 26, 2026

Uh oh!

Uh oh!

mkuchenbecker Aug 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		layout.label.endsWith("/parquet") \|\|
		layout.label.endsWith("/orc"))

Uh oh!

Conversation

mkuchenbecker commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

mkuchenbecker Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mkuchenbecker Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

mkuchenbecker Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

mkuchenbecker Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mkuchenbecker Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

mkuchenbecker commented Aug 13, 2026 •

edited

Loading

mkuchenbecker Aug 26, 2026 •

edited

Loading