bdq_workbench

BDQ Workbench is a Java 17 application for policy-driven Biodiversity Data Quality (BDQ) execution over Darwin Core Archives (DwC-A) and Darwin Core Data Packages.

BDQ Workbench takes DarwinCore Archive files or Darwin Core Data Package files as input, identifies tests that apply to a bdqffdq:UseCase (purpose to which data are to be put and need to have fitness for) that are available in the implemntation, then runs those tests on the data (in pre-amendment, amendment, and post-amendment phases) and produces a data quality report (in several formats). It also evaluates the binding of information elements in the input data with the test implementations, can provide parameters to parameterized tests, and can run tests from a use case individually.

See: https://bdq.tdwg.org/

Build and test

mvn -q clean package

mvn -q test

Run integration tests only (failsafe):

mvn -q verify -Prelease

Run

mvn -q package
java -jar target/bdq_workbench-0.1.0-SNAPSHOT.jar

Launching the jar without options opens a desktop GUI for entering parameters and monitoring execution. The startup screen includes a dataset file picker, use-case selection, and advanced options for custom use-case/test-definition/ontology sources, discovery packages, and threads. By default the GUI caches:

• https://bdq.tdwg.org/draft/dist/bdquc.xml (use cases)
• https://bdq.tdwg.org/draft/dist/bdqtest.ttl (test definitions)
• https://bdq.tdwg.org/draft/vocabulary/bdqffdq.ttl (ontology)

Use-case and RDF definition inputs support RDF/XML, Turtle, and JSON-LD serializations. Logging is configured to the console with a default DEBUG root level in src/main/resources/logback.xml.

Show command-line help:

java -jar target/bdq_workbench-0.1.0-SNAPSHOT.jar --help

Configuration defaults are in src/main/resources/application.properties and can be overridden with CLI options, for example:

java -jar target/bdq_workbench-0.1.0-SNAPSHOT.jar --dataset path/to/dataset.zip

Configuration precedence is:

1. command-line or GUI-supplied overrides
2. src/main/resources/application.properties
3. built-in fallback defaults in ConfigLoader

Pre-execution record filtering can be configured from the CLI with repeatable --record-filter flags, or in the GUI with the Build Record Filters... dialog after selecting a dataset. Filter syntax is:

java -jar target/bdq_workbench-0.1.0-SNAPSHOT.jar \
  --record-filter 'dwc:genus=Abies|Pinus' \
  --record-filter dwc:country=Canada

Within a record-filter, search terms are related by OR; across record-filters, terms are related by AND. The example above is interpreted as dwc:genus=(Abies OR Pinus) AND dwc:country=Canada. Field names match case-insensitively by full name or local name (for example dwc:country and country), while search term values are matched exactly and case-sensitively against the canonicalized input values.

Processing pipeline and flow-control options

The current desktop/CLI flow can be summarized as:

Dataset input
   |
   +--> Load DwC-A zip / Data Package CSV
   |
   +--> [optional] Build record filters from dataset terms/values (GUI)
   |
   +--> [optional] Apply record filters
   |
   +--> Resolve use case / policy tests
   |
   +--> Discover implementations on classpath
   |
   +--> Bind tests + validate parameter mappings
   |       |
   |       +--> unresolved tests may still be shown in preflight
   |       +--> GUI can continue with runnable tests only ("Start Available Tests")
   |       +--> GUI can edit/load/save parameter values before execution
   |
   +--> [optional] Distinct-value aggregation (`bdq.execution.dedup=true`)
   |
   +--> PRE_AMENDMENT tests
   |
   +--> AMENDMENT tests
   |
   +--> POST_AMENDMENT tests
   |
   +--> Post-process responses
   |       |
   |       +--> built-in COUNT multi-record measures
   |       +--> synthesized `UNABLE_TO_RUN` responses for unresolved/unbound tests
   |
   +--> Export text, RDF/Turtle, XLSX, and unresolved-response XLSX reports

Current flow-control options are intentionally modest:

• Dataset choice: choose a DwC-A zip or Darwin Core Data Package.
• Record filtering: optional exact-match filtering before binding or execution; in the GUI the filter builder inspects the selected dataset and lets the user choose only from terms present in the data.
• Use case/test-definition/ontology sources: GUI advanced options and CLI/config can override the default RDF sources.
• Implementation packages: CLI/config/GUI advanced options control the discovery package roots.
• Thread count: CLI/config/GUI advanced options control the worker pool size.
• Continue with unresolved tests: GUI preflight can proceed with runnable tests even when some policy or implementation bindings remain unresolved.
• Parameter overrides: GUI preflight supports per-test parameter editing plus saving/loading parameter settings.
• Isolated test execution: GUI preflight/debug tools can run one bound test independently against the prepared dataset.
• Distinct-value reduction: CLI/config bdq.execution.dedup and the GUI's Reduce repeated test calls by distinct input values checkbox toggle whether eligible bindings run once per distinct input-value group instead of once per record.
• Not currently supported: there is still no user-facing phase skip/select control, and the main run UI still does not expose cancellation.

Architecture overview

The codebase is organized under org.filteredpush.bdq_workbench with explicit module boundaries:

• app: bootstrap, configuration loading, orchestration, exception handling
• model: domain model (UseCase, Policy, TestDefinition, ImplementationBinding, Phase, Response)
• ingest: DwC-A and Data Package ingestion into canonical records
• rdf_policy: use-case/policy/test RDF resolution
• test_discovery: annotation-based discovery (@Provides, @Validation, @Issue, @Measure, @Amendment, etc.) and binding
• execution: parallel phase orchestration (pre-amendment, amendment, post-amendment) with deterministic ordering and distinct-value test call reduction
• reporting: summary output, normalized response stream export, RDF export, and XLSX spreadsheet export (via kurator-ffdq)

Extension points are interfaces for discovery, binding, execution adapters, and report exporters.

Input binding and parameter handling

Execution binding is reflection-driven and annotation-aware:

• @ActedUpon("dwc:term") and @Consulted("dwc:term") are matched against canonical record fields.
• Matching is deterministic across exact values, dwc:prefix values, and local-name forms such as eventDate.
• @Parameter(name = "bdq:...") values come from the selected test definition / UI parameter editor.
• Legacy implementations that accept (Map record) or (Map record, Map parameters) are still supported for backward compatibility.

For every candidate implementation method the workbench records:

• implementation status: FOUND, MISSING, or deterministic resolution of an ambiguous set
• binding status: BOUND, PARTIAL, or UNBOUND
• per-parameter diagnostics for missing Darwin Core terms, missing user parameters, or unsupported parameter types

When both default and parameterized implementations exist for the same test, the workbench prefers:

1. the parameterized method when the user provides parameter values
2. the default method when no parameter values are provided

The preflight grid shows the chosen method, parameterization capability, and whether the selected run is using default values.

Response stream semantics

Each execution result is normalized into a response stream entry with:

• record id
• test id and test type
• implementation class/method provenance
• phase (PRE_AMENDMENT, AMENDMENT, POST_AMENDMENT)
• parameter values used for the invocation
• responseStatus
• responseResult
• comment
• amendment payload, when present

DQResponse objects are adapted reflectively by reading getResultState(), getValue().getObject(), and getComment(). Amendment results are preserved as normalized amendment maps and then applied to the amendment working copy before post-amendment execution.

Reports include:

• reports/bdq-report-summary.txt Human readable summary of test execution results.
• reports/bdq-report-responses.txt Human readable list of test execution Response values.
• reports/bdq-report-rdf.ttl RDF test responses serialized as Turtle.
• reports/bdq-report-xls.xlsx Spreadsheet report produced via kurator-ffdq's XLSXPostProcessor (see below).
• reports/bdq-report-xls-unresolved.xlsx Spreadsheet companion listing unresolved, unbound, and other sentinel-record responses excluded from the main per-record workbook.

Distinct-value execution (test call reduction)

A BDQ test is specified as a pure function of the Darwin Core terms it declares as input (@ActedUpon/@Consulted), so records that share identical values for exactly those terms must produce identical results. Rather than invoking a test once per record, ParallelPhaseExecutionService partitions each phase's records into distinct-value groups per binding (via RecordGroupPartitioner), invokes the test once against one representative record per group, and copies that one response to every record in the group — the final response list has exactly the same shape (one response per record per test) as running per-record would, just with fewer real invocations.

A few behaviors worth knowing about:

• Grouping is shared across tests, not just per test. Two bindings that happen to declare the same set of term names — regardless of test type, or whether a term is ACTED_UPON for one and CONSULTED for the other — share the same partitioning work for a phase via a PhaseGroupCache, rather than each recomputing it from scratch. E.g. ten different validations that each act upon only dwc:country all reuse one partition of the dataset's distinct country values.
• Grouping is exact-match only. Term values are compared with plain string equality — no case-folding or other normalization — since many BDQ tests are sensitive to exact formatting.
• Amendments are sequenced correctly within the AMENDMENT phase. Since one amendment test's output can change values a later amendment test in the same phase groups or reads by, AMENDMENT-phase bindings are processed one at a time: each binding's groups are computed, invoked, and its resulting amendments applied to every group member, and any cached partition touching the changed fields is discarded, before the next binding's groups are computed. PRE_AMENDMENT and POST_AMENDMENT never mutate records mid-phase, so their bindings' groups are all computed and submitted together.
• Not every binding is eligible. Implementations bound via the legacy (Map record)/ (Map record, Map parameters) signatures read the whole record or parameter map rather than specific declared terms, so the workbench can't know what subset of fields they actually depend on — these always run once per record.
• Configurable via bdq.execution.dedup (CLI --dedup true|false, default true). Disabling it runs every binding once per record exactly as if none were dedup-eligible, useful for debugging or comparing behavior against the pre-reduction execution path.

Spreadsheet (XLSX) report export

XlsxReportExporter builds an in-memory kurator-ffdq FFDQModel directly from the run's ExecutionSummary — one data resource per input record and one response per Response — and streams it directly to bdq-report-xls.xlsx with kurator-ffdq's XLSXPostProcessor, which produces Summary, Initial Values, Final Values, Measures, Validations, Amendments, and Issues sheets, with per-record rows color-coded by outcome.

A few behaviors worth knowing about:

• Missing information elements are padded, not omitted. If a use case's tests expect a Darwin Core term as an input information element (acted upon or consulted) and that term isn't present in the input data at all, it still appears as a column in the report, with an empty value for every record — rather than being silently missing from the spreadsheet. This comes from the run's test/implementation bindings (ExecutionSummary.bindings()), not from re-resolving the ratified ontology, so it reflects exactly what the bound implementations look for.
• Responses that don't apply to one record — built-in multi-record measures and synthesized unresolved/unbound placeholder responses — are excluded from bdq-report-xls.xlsx and instead listed in their own small companion workbook, bdq-report-xls-unresolved.xlsx (via UnresolvedResponsesExporter), since kurator-ffdq's per-record model has no place for them. This is a separate file rather than an extra sheet appended afterward: appending would require reopening the (potentially very large) written workbook as a plain XSSFWorkbook, which for a large dataset can exceed Apache POI's single-zip-entry read cap and fail with a RecordFormatException. Keeping the two files separate means the main export is a pure, one-pass stream straight to disk regardless of dataset size.
• Issues sheet coloring is a known gap. kurator-ffdq's Issue context class lacks a no-arg constructor (unlike Measure/Validation/Amendment), which breaks RDFBeans deserialization if one is attached to a saved response. XlsxReportExporter leaves it unset, so ISSUE-type responses still get their row and every field column, just without per-cell acted-upon/consulted coloring.
• Build dependency: this depends on kurator-ffdq's "restored and productized" XLSXPostProcessor, which as of this writing only exists in a 3.3.0-SNAPSHOT build (the pom.xml dependency is pinned there, with a comment to move it to a released 3.3.0 once one is cut). Building this project currently requires that SNAPSHOT installed locally.

Multi-record measure preparation

ExecutionSummary exposes filtering and counting helpers over the normalized response stream so downstream multi-record measure work can count and filter by:

• phase
• test type
• response status
• response result

This is the initial plumbing layer for multi-record calculations. Full multi-record execution remains a follow-up item, but downstream code can now consume the normalized response stream instead of raw input rows.

GUI workflow

The desktop GUI supports:

1. selecting a dataset and use case
2. optionally opening Build Record Filters... to inspect dataset terms and common values, then assembling exact-match filters
3. running a preflight review that loads the dataset, applies filters, discovers implementations, and populates a test grid
4. reviewing binding status, method selection, parameterization capability, and normalized filter counts before execution
5. editing parameter values or keeping defaults before execution
6. saving and loading parameters for parameterized tests
7. continuing with runnable tests only when some tests remain unresolved (Start Available Tests)
8. running a bound test in isolation
9. monitoring a simple stage list plus live per-phase progress and response/result counters
10. reviewing a post-run summary and saved output locations

The execution phases are fixed (PRE_AMENDMENT, AMENDMENT, POST_AMENDMENT) and are not currently user-skippable from either the CLI or the GUI.

When policy resolution or implementation binding cannot produce a runnable test, the workbench still emits synthesized UNABLE_TO_RUN responses so those tests appear in the final summary and unresolved workbook outputs.

Development

AI-assisted development disclosure

This project has used GitHub Copilot and Claude Code as AI coding assistants during development.

Copilot and Claude contributions are limited to suggested code and documentation text.
All accepted changes were reviewed, edited as needed, and validated by human maintainers before inclusion in the master branch.

Provenance and responsibility

• Human maintainers are responsible for all design decisions, semantics, and released content.
• AI-generated suggestions are treated as draft material and may contain errors.
• Ontology-aligned terminology and normative language in this project are curated by the tdwg/bdq maintainers.