semantics.md

Semantics

Purpose

Semantic specs define Gold datasets for analytics, controls, and reporting.

The framework intentionally implements a small semantic layer rather than a full semantic serving platform.

That layer is strong enough to express:

• curated Gold dimensions
• approved measures
• joins to conformed reference data
• reusable shared measures
• certification state metadata

Main Types

• SemanticViewSpec
• SemanticDimension
• SemanticMeasure
• SemanticMeasureDefinition
• SemanticFilter
• SemanticJoin

What a Semantic Spec Actually Does

A semantic spec compiles into one SQL definition for a view or a table.

It controls:

• source relation
• joins
• grouping grain
• projected dimensions
• aggregated measures
• filtering
• materialization type
• semantic metadata

Dimensions

Dimensions can come from:

• a source column
• a SQL expression

Column-backed dimension

dimensions:
  - name: status
    column: status

Expression-backed dimension

dimensions:
  - name: business_date
    expression: CAST(accounts.valid_from AS DATE)

Expression-backed dimensions are useful for:

• date derivation
• normalized code values
• joined reference labels

Measures

Measures are SQL aggregation expressions.

Examples:

• count(*)
• sum(amount)
• sum(CASE WHEN status = 'ACTIVE' THEN amount ELSE 0 END)

Inline measure example

measures:
  - name: active_amount
    expression: sum(CASE WHEN accounts.status = 'ACTIVE' THEN accounts.amount ELSE 0 END)

Shared Measures

Shared measures let you define reusable approved aggregations inside the same semantic spec.

Example:

shared_measures:
  - name: account_count
    expression: count(*)
  - name: total_amount
    expression: sum(accounts.amount)

measure_refs:
  - account_count
  - total_amount

Why this is useful:

• keeps Gold configs shorter
• makes measure naming more deliberate
• avoids repeated SQL for common measures

Filters

Filters are applied before aggregation.

Example:

filters:
  - column: is_current
    operator: "="
    value_sql: "true"

That is the normal pattern when Gold should reflect only current Silver state.

Joins

Semantic joins are explicit and governed by the spec.

Example:

joins:
  - source_table: main.refdata.currency_dim
    alias: currency_dim
    join_type: left
    condition: accounts.currency = currency_dim.currency_code

This is useful for:

• currency reference data
• product hierarchies
• conformed descriptive dimensions

Materialization

Gold outputs can be published as:

• view
• table

Use view when

• the output is exploratory
• the underlying source is already stable
• you want lightweight publication

Use table when

• downstream consumers need stable materialized output
• controls must reconcile against a fixed Gold object
• the output is intended to behave like a published data product

Certification State

Semantic specs can carry a certification_state.

Example:

certification_state: validated

Current supported states are metadata states such as:

• draft
• validated
• certified

The framework persists that state into table/view properties so consumers can distinguish exploratory from more trusted outputs.

Full Example

semantic_spec:
  name: customer_accounts_summary
  source_table: main.banking.customer_accounts_silver AS accounts
  joins:
    - source_table: main.refdata.currency_dim
      alias: currency_dim
      join_type: left
      condition: accounts.currency = currency_dim.currency_code
  grain:
    - source_system
  dimensions:
    - name: business_date
      expression: CAST(accounts.valid_from AS DATE)
    - name: status
      expression: accounts.status
    - name: currency
      expression: accounts.currency
    - name: currency_name
      expression: currency_dim.currency_name
  shared_measures:
    - name: account_count
      expression: count(*)
    - name: total_amount
      expression: sum(accounts.amount)
    - name: active_account_count
      expression: sum(CASE WHEN accounts.status = 'ACTIVE' THEN 1 ELSE 0 END)
    - name: active_amount
      expression: sum(CASE WHEN accounts.status = 'ACTIVE' THEN accounts.amount ELSE 0 END)
  measure_refs:
    - account_count
    - total_amount
    - active_account_count
    - active_amount
  filters:
    - column: is_current
      operator: "="
      value_sql: "true"
  materialization: table
  certification_state: validated

This produces a Gold output that:

• reads current Silver state
• joins currency reference data
• groups by curated dimensions
• computes approved measures
• publishes a stable Gold table

Conceptual SQL Shape

The above spec renders to SQL in this shape:

SELECT
    source_system,
    CAST(accounts.valid_from AS DATE) AS business_date,
    accounts.status AS status,
    accounts.currency AS currency,
    currency_dim.currency_name AS currency_name,
    count(*) AS account_count,
    sum(accounts.amount) AS total_amount,
    sum(CASE WHEN accounts.status = 'ACTIVE' THEN 1 ELSE 0 END) AS active_account_count,
    sum(CASE WHEN accounts.status = 'ACTIVE' THEN accounts.amount ELSE 0 END) AS active_amount
FROM main.banking.customer_accounts_silver AS accounts
LEFT JOIN main.refdata.currency_dim AS currency_dim
  ON accounts.currency = currency_dim.currency_code
WHERE is_current = true
GROUP BY
    source_system,
    CAST(accounts.valid_from AS DATE),
    accounts.status,
    accounts.currency,
    currency_dim.currency_name

What the Semantic Layer Is Not Yet

It is not yet:

• a full enterprise semantic platform
• a metric dependency engine
• a semantic API/query service
• a conformed entity registry
• a time-grain intelligence framework

It is still very useful for:

• governed Gold tables
• finance/risk summaries
• control outputs
• small cross-domain curated datasets

Practical Guidance

Use the semantic layer when:

• you need a stable Gold output
• measures and dimensions should be reviewed explicitly
• a Gold dataset must be understandable from config alone

Do not use it as a place to hide complex unreviewed SQL logic that should really live in curated Silver modeling first.