Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
285 changes: 253 additions & 32 deletions docs/data-tests/with-context-tests.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,27 @@ A set of generic tests that extend common dbt, dbt-utils, and dbt-expectations t
When a test fails, the failing rows are returned together with the columns you care about — making it much easier to investigate the root cause directly from the test results.

If `context_columns` is omitted, **all columns** are returned alongside failing rows.
A single column can be given as a plain string instead of a list.

<Note>
No extra packages are required. These tests ship with the Elementary dbt
package and do not depend on `dbt_utils` or `dbt_expectations` being installed,
even where they mirror a test from one of those packages.
</Note>

<Note>
If a column listed in `context_columns` does not exist on the model, a warning
is logged and that column is skipped. The test continues and will not error.
</Note>

<Note>
To test only some of the rows, use dbt's
[`where` config](https://docs.getdbt.com/reference/resource-configs/where).
Earlier versions of some of these tests took a `row_condition` argument
instead; it was removed in favour of `where`, which does the same thing and
works on every dbt test.
</Note>

---

## not_null_with_context
Expand Down Expand Up @@ -53,47 +68,86 @@ models:

---

## accepted_range_with_context
## expression_is_true_with_context

`elementary.accepted_range_with_context`
`elementary.expression_is_true_with_context`

Validates that a SQL expression holds for every row. Extends `dbt_utils.expression_is_true`.

Validates that column values fall within an accepted range. Extends `dbt_utils.accepted_range`.
This is the most flexible test in the set: any condition you can write in SQL, including range checks and comparisons across columns.

<Note>
This test is table-level, so it has no tested column of its own. List every
column you want in the sample under `context_columns`, including the ones your
expression references.
</Note>

### Parameters

| Parameter | Required | Default | Description |
| ----------------- | -------- | ------- | ------------------------------------------------------------------------------------ |
| `column_name` | Yes | — | The column to test. |
| `min_value` | No* | `none` | Minimum accepted value (inclusive by default). At least one bound must be provided. |
| `max_value` | No* | `none` | Maximum accepted value (inclusive by default). At least one bound must be provided. |
| `inclusive` | No | `true` | Whether the bounds are inclusive. |
| `context_columns` | No | `none` | List of additional columns to return with failing rows. Omit to return all columns. |
| Parameter | Required | Default | Description |
| ----------------- | -------- | ------- | ------------------------------------------------------------------------------- |
| `expression` | Yes | — | SQL condition that must hold for every row. Rows where it is false are returned. |
| `column_name` | No | `none` | Apply the expression to a single column, as in `column_name: amount` with `expression: "> 0"`. |
| `context_columns` | No | `none` | List of additional columns to return with failing rows. Omit to return all columns. |

<RequestExample>

```yml Range check
models:
- name: orders
data_tests:
- elementary.expression_is_true_with_context:
expression: "amount >= 0 and amount <= 10000"
context_columns: [amount, order_id, customer_id]
```

```yml Comparison across columns
models:
- name: subscriptions
data_tests:
- elementary.expression_is_true_with_context:
expression: "end_date > start_date"
context_columns: [subscription_id, start_date, end_date]
```

</RequestExample>

\* At least one of `min_value` or `max_value` must be provided.
---

## not_empty_string_with_context

`elementary.not_empty_string_with_context`

Validates that a column contains no empty strings. Extends `dbt_utils.not_empty_string`.

### Parameters

| Parameter | Required | Default | Description |
| ------------------ | -------- | ------- | ------------------------------------------------------------------------------- |
| `column_name` | Yes | — | The column to test. |
| `trim_whitespace` | No | `true` | Whether to trim the value before testing, so whitespace-only values also fail. |
| `context_columns` | No | `none` | List of additional columns to return with failing rows. Omit to return all columns. |

<RequestExample>

```yml With context columns
models:
- name: orders
- name: customers
columns:
- name: amount
- name: name
data_tests:
- elementary.accepted_range_with_context:
min_value: 0
max_value: 10000
context_columns: [order_id, customer_id, order_date]
- elementary.not_empty_string_with_context:
context_columns: [customer_id, email, created_at]
```

```yml Min bound only
```yml Treat whitespace-only values as valid
models:
- name: orders
- name: customers
columns:
- name: amount
- name: notes
data_tests:
- elementary.accepted_range_with_context:
min_value: 0
inclusive: false
- elementary.not_empty_string_with_context:
trim_whitespace: false
```

</RequestExample>
Expand All @@ -111,19 +165,19 @@ Expects column values to not be null. Extends `dbt_expectations.expect_column_va
| Parameter | Required | Default | Description |
| ----------------- | -------- | ------- | ------------------------------------------------------------------------------- |
| `column_name` | Yes | — | The column to test for null values. |
| `row_condition` | No | `none` | Optional SQL filter applied before testing (e.g. `"status = 'active'"`). |
| `context_columns` | No | `none` | List of additional columns to return with failing rows. Omit to return all columns. |

<RequestExample>

```yml With row condition and context columns
```yml With a row filter and context columns
models:
- name: subscriptions
columns:
- name: end_date
data_tests:
- elementary.expect_column_values_to_not_be_null_with_context:
row_condition: "status = 'active'"
config:
where: "status = 'active'"
context_columns: [subscription_id, customer_id, start_date]
```

Expand All @@ -137,12 +191,13 @@ models:

Expects column values to be unique. Returns all duplicate rows (not just a count), so you can see the full context of each duplicate. Extends `dbt_expectations.expect_column_values_to_be_unique`.

Null values are ignored rather than treated as duplicates of one another, matching dbt's built-in `unique` test.

### Parameters

| Parameter | Required | Default | Description |
| ----------------- | -------- | ------- | ------------------------------------------------------------------------------- |
| `column_name` | Yes | — | The column to test for uniqueness. |
| `row_condition` | No | `none` | Optional SQL filter applied before testing. |
| `context_columns` | No | `none` | List of additional columns to return with failing rows. Omit to return all columns. |

<RequestExample>
Expand All @@ -161,24 +216,91 @@ models:

---

## expect_compound_columns_to_be_unique_with_context

`elementary.expect_compound_columns_to_be_unique_with_context`

Expects a combination of columns to be unique. Returns every duplicate row, so you can see the full context of each collision. Extends `dbt_expectations.expect_compound_columns_to_be_unique`.

Rows where every listed column is null are ignored rather than treated as duplicates of one another.

Use this when no single column identifies a row, for example when uniqueness is on a customer and date pair.

### Parameters

| Parameter | Required | Default | Description |
| ----------------- | -------- | ------- | ------------------------------------------------------------------------------- |
| `column_list` | Yes | — | List of columns whose combination must be unique. |
| `context_columns` | No | `none` | List of additional columns to return with failing rows. Omit to return all columns. |

<RequestExample>

```yml With context columns
models:
- name: daily_metrics
data_tests:
- elementary.expect_compound_columns_to_be_unique_with_context:
column_list: [customer_id, metric_date]
context_columns: [metric_value, loaded_at]
```

</RequestExample>

---

## expect_column_pair_values_A_to_be_greater_than_B_with_context

`elementary.expect_column_pair_values_A_to_be_greater_than_B_with_context`

Expects the values of one column to be greater than another. Extends `dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B`.

Both compared columns are always returned with failing rows, before any `context_columns`.

### Parameters

| Parameter | Required | Default | Description |
| ----------------- | -------- | ------- | ------------------------------------------------------------------------------- |
| `column_A` | Yes | — | The column expected to hold the greater value. |
| `column_B` | Yes | — | The column compared against. |
| `or_equal` | No | `false` | Whether equal values pass. |
| `context_columns` | No | `none` | List of additional columns to return with failing rows. Omit to return all columns. |

<RequestExample>

```yml With context columns
models:
- name: orders
data_tests:
- elementary.expect_column_pair_values_A_to_be_greater_than_B_with_context:
column_A: total_amount
column_B: discount_amount
or_equal: true
context_columns: [order_id, customer_id]
```

</RequestExample>

---

## expect_column_values_to_match_regex_with_context

`elementary.expect_column_values_to_match_regex_with_context`

Expects column values to match a given regular expression. Extends `dbt_expectations.expect_column_values_to_match_regex`.

<Info>
Requires `dbt_expectations` to be installed in your project.
</Info>
<Warning>
Not supported on SQL Server or Microsoft Fabric. T-SQL has no regular
expression functions, so this test raises a clear compilation error on those
platforms rather than running.
</Warning>

### Parameters

| Parameter | Required | Default | Description |
| ----------------- | -------- | ------- | ------------------------------------------------------------------------------- |
| `column_name` | Yes | — | The column to test. |
| `regex` | Yes | — | The regular expression pattern to match. |
| `row_condition` | No | `none` | Optional SQL filter applied before testing. |
| `is_raw` | No | `false` | Whether the regex is a raw string. |
| `is_raw` | No | `false` | Whether the regex is a raw string. Honored on Snowflake, BigQuery, Spark and Databricks; ignored elsewhere. |
| `flags` | No | `""` | Optional regex flags (adapter-dependent). |
| `context_columns` | No | `none` | List of additional columns to return with failing rows. Omit to return all columns. |

Expand All @@ -199,6 +321,60 @@ models:

---

## expect_column_values_to_match_regex_list_with_context

`elementary.expect_column_values_to_match_regex_list_with_context`

Expects column values to match any or all of a list of regular expressions. Extends `dbt_expectations.expect_column_values_to_match_regex_list`.

<Warning>
Not supported on SQL Server or Microsoft Fabric. T-SQL has no regular
expression functions, so this test raises a clear compilation error on those
platforms rather than running.
</Warning>

### Parameters

| Parameter | Required | Default | Description |
| ----------------- | -------- | ------- | ------------------------------------------------------------------------------- |
| `regex_list` | Yes | — | List of regular expression patterns. |
| `match_on` | No | `"any"` | `"any"` passes a row that matches at least one pattern, `"all"` requires every pattern. Any other value raises an error. |
| `is_raw` | No | `false` | Whether the patterns are raw strings. Honored on Snowflake, BigQuery, Spark and Databricks; ignored elsewhere. |
| `flags` | No | `""` | Optional regex flags. Flags the platform does not support are dropped with a warning. |
| `context_columns` | No | `none` | List of additional columns to return with failing rows. Omit to return all columns. |

<RequestExample>

```yml Match any pattern
models:
- name: contacts
columns:
- name: phone
data_tests:
- elementary.expect_column_values_to_match_regex_list_with_context:
regex_list:
- "^\\+1[0-9]{10}$"
- "^\\+44[0-9]{10}$"
match_on: any
context_columns: [contact_id, country]
```

```yml Require every pattern
models:
- name: products
columns:
- name: sku
data_tests:
- elementary.expect_column_values_to_match_regex_list_with_context:
regex_list: ["^SKU-", "-[0-9]+$"]
match_on: all
context_columns: [product_id, name]
```

</RequestExample>

---

## relationships_with_context

`elementary.relationships_with_context`
Expand Down Expand Up @@ -240,3 +416,48 @@ models:
```

</RequestExample>


---

## accepted_range_with_context

`elementary.accepted_range_with_context`

<Warning>
Deprecated, and scheduled for removal in the next release. Use
`dbt_utils.accepted_range` instead. Running it logs a warning.
</Warning>

`dbt_utils.accepted_range` selects every column already, so unlike the other
tests on this page this one cannot add context to a stored sample. The only
thing it can do is narrow the sample to a chosen subset, which is not what
`context_columns` is for.

If you were using it to keep columns out of a stored sample, there is no direct
replacement. Use the sampling controls instead: the `show_sample_rows` and PII
tags, `disable_test_samples`, or `test_sample_row_count`.

### Migration

```yml Before
models:
- name: orders
columns:
- name: amount
data_tests:
- elementary.accepted_range_with_context:
min_value: 0
max_value: 10000
```

```yml After
models:
- name: orders
columns:
- name: amount
data_tests:
- dbt_utils.accepted_range:
min_value: 0
max_value: 10000
```
Loading