Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 49 additions & 0 deletions iotdb-thingsboard-table/CI-NOTES.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
<!--

Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.

-->

# CI Notes

The `iotdb-extras` parent reactor builds and tests this module only on JDK 17+:
the root pom adds it to `<modules>` through a profile activated by
`<jdk>[17,)</jdk>` (it compiles with Java 17 language features), so the root
build compiles and tests it on the 17/21 jobs and skips it on the 8/11 jobs. This
file is a developer reference of the local checks for the `iotdb-thingsboard-table`
module; it is not itself a GitHub Actions workflow.

## Candidate Checks

- Compile from the standalone module directory:
`mvn compile -DskipTests`
- Run unit tests:
`mvn test`
- Validate the local stack file:
`docker compose -f docker-compose.test.yml config`
- Run Docker-backed integration tests only when Docker is available:
`mvn -Piotdb-table-it verify`
- Start the optional local stack only when required environment values are set:
`TB_POSTGRES_USER=<postgres-user> TB_POSTGRES_PASSWORD=<postgres-password> IOTDB_USERNAME=<iotdb-user> IOTDB_PASSWORD=<iotdb-password> docker compose -f docker-compose.test.yml up -d`

## Notes

- Keep this file inside the module. Do not copy it to `.github/workflows`.
- Do not store passwords, tokens, or local hostnames in CI configuration.
- Keep the Docker image tags aligned with the versions exercised by this module's
integration-test profile.
162 changes: 162 additions & 0 deletions iotdb-thingsboard-table/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
<!--

Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.

-->

# IoTDB ThingsBoard Table

## Overview

`iotdb-thingsboard-table` is a ThingsBoard historical-telemetry DAO backend
built on Apache IoTDB 2.0.8 Table Mode. It lets a ThingsBoard deployment store
and serve time-series telemetry through IoTDB's table-session API instead of the
default Cassandra/SQL backends. The module targets ThingsBoard v4.3.1.2. Because
it compiles with Java 17 language features (records and others), the
`iotdb-extras` parent reactor builds and tests it only on JDK 17+: the root pom
adds it to `<modules>` through a profile activated by `<jdk>[17,)</jdk>`, so the
JDK 8/11 reactor jobs skip it while the 17/21 jobs build it as part of the root
project.

## ThingsBoard SPI surface (Strategy F)

The ThingsBoard DAO SPI and value types (`org.thingsboard.*`) are not published
to Maven Central, so they cannot be a normal compile dependency. Strategy F
treats them as a **compile-only source surface** under `src/provided/java`: just
enough of the ThingsBoard interfaces and value objects to compile against. The
maven-jar-plugin excludes `org/thingsboard/**` from the built jar, so these
compile-only types never ship and never shadow the real ThingsBoard classes. At
runtime the actual ThingsBoard classpath supplies them. This keeps the module
buildable in isolation while binding to the genuine ThingsBoard types on a real
deployment.

The compile-only surface under `src/provided/java` was manually verified against
ThingsBoard `v4.3.1.2` (commit `c37fb509`): the `TimeseriesDao` SPI methods the
DAO consumes and the value-object accessors it reads were checked against the
upstream sources. A fully-automated check against the upstream artifact is not
possible because ThingsBoard's `dao`/`common-data` modules are not published to
Maven Central (the reason for Strategy F). As a guard against silent drift,
`StrategyFContractTest` pins the exact `TimeseriesDao` SPI method signatures the
DAO depends on, so any accidental edit to the local surface fails the build.

## Scope

This initial module delivers an inert-by-default foundation: `IoTDBTableBaseDao`
(session-pool lifecycle, schema/table bootstrap) and the
`IoTDBTableTimeseriesDao` write path (`save`), raw read, and delete. To exercise
it, set both `database.ts.type=iotdb-table` and
`iotdb.ts.experimental-raw-only=true`. Aggregation, latest telemetry, and
attribute/label DAOs are outside the current scope.

> **This is an incremental / experimental backend.** Explicitly enabling it
> routes ThingsBoard historical telemetry through IoTDB Table Mode for **raw read
> + write + delete only**. **Time-bucketed aggregation is NOT implemented yet**;
> aggregation, latest telemetry, and attributes are outside the current scope.

## Known limitations

**Same-timestamp type change across separate flushes.** The writer collapses
duplicate `(tenant, entity, key, timestamp)` saves *within a single flush* so the
last write wins, but it does not yet defend against a same-`(tenant, entity, key,
timestamp)` save whose value *type* changes between two **separate** flushes (for
example a `LONG` written in one flush and a `STRING` written at the same
timestamp in a later flush). Because each typed value lands in its own column
(`long_v`, `str_v`, ...), that single point can end up with two non-null typed
columns.

This is a deliberate current-scope decision: the cleanup that would prevent it (a
delete-then-insert overwrite on every save) is outside the current scope. The
behavior is **fail-fast, not silent**: a raw read of that one poisoned point
throws an `IllegalStateException` (the single-typed-column invariant enforced in
`IoTDBTableBaseDao`) rather than returning a wrong value. Every other point is
unaffected. This documented behavior is pinned by an integration test
(`IoTDBTableTimeseriesDaoIT`).

## Configuration

The backend is bound from `iotdb.*` Spring properties (see `IoTDBTableConfig`).
Key activation and operational flags:

| Property | Default | Meaning |
| --- | --- | --- |
| `database.ts.type` | _(unset)_ | Set to `iotdb-table` as the ThingsBoard historical-timeseries backend selector. |
| `iotdb.ts.experimental-raw-only` | `false` | Explicit opt-in for this initial raw-only backend. Must be `true` together with `database.ts.type=iotdb-table`; write, raw read, and delete are implemented, while time-bucketed aggregation is outside the current scope. |
| `iotdb.host` / `iotdb.port` | `127.0.0.1` / `6667` | IoTDB node address. |
| `iotdb.username` / `iotdb.password` | `root` / `root` | IoTDB credentials. |
| `iotdb.database` | `thingsboard` | Target IoTDB database. |
| `iotdb.session-pool-size` | `8` | Table session pool size. |
| `iotdb.schema.bootstrap` | `true` | When `true`, the module runs an idempotent startup bootstrap that reads `schema-iotdb-table.sql` from the classpath and creates the `telemetry` / `entity_attributes` tables (and database) on a fresh IoTDB before the first write. Set to `false` if you manage the schema out-of-band. |

The module is a Spring Boot **auto-configuration**
(`IoTDBTableConfiguration`), registered via
`META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports`
(and `META-INF/spring.factories` as a fallback), so it activates in a real
ThingsBoard deployment without the host application having to component-scan
`org.apache.iotdb.extras`.

## Build

The module builds from the repository root as part of the reactor:

```bash
# from the iotdb-extras repository root
mvn -pl iotdb-thingsboard-table -am clean test
```

It can also be built standalone from the module directory:

```bash
cd iotdb-thingsboard-table
mvn compile -DskipTests
```

## Test

Run the Java test scaffold from the module directory:

```bash
mvn test
```

Run the Docker-backed integration tests only when required:

```bash
mvn -Piotdb-table-it verify
```

Start the local integration stack with explicit environment values:

```bash
TB_POSTGRES_USER=<postgres-user> TB_POSTGRES_PASSWORD=<postgres-password> \
IOTDB_USERNAME=<iotdb-user> IOTDB_PASSWORD=<iotdb-password> \
docker compose -f docker-compose.test.yml up -d
```

Stop and remove the local stack:

```bash
docker compose -f docker-compose.test.yml down -v
```

## Status

Initial module status: `IoTDBTableBaseDao` plus the `IoTDBTableTimeseriesDao`
write, raw-read, and delete paths are implemented behind
`database.ts.type=iotdb-table` and `iotdb.ts.experimental-raw-only=true`.
Without both properties, the module is inert. Aggregation, latest telemetry, and
attributes are outside the current scope.
94 changes: 94 additions & 0 deletions iotdb-thingsboard-table/docker-compose.test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#

services:
iotdb:
image: apache/iotdb:2.0.8-standalone
container_name: iotdb-table-test
environment:
IOTDB_USERNAME: ${IOTDB_USERNAME:?set IOTDB_USERNAME}
IOTDB_PASSWORD: ${IOTDB_PASSWORD:?set IOTDB_PASSWORD}
ports:
- "${IOTDB_RPC_PORT:-6667}:6667"
networks:
- tb-iotdb-test
healthcheck:
test: ["CMD-SHELL", "bash -ec ': >/dev/tcp/127.0.0.1/6667'"]
interval: 10s
timeout: 5s
retries: 12
start_period: 30s

postgres:
image: postgres:15
container_name: tb-postgres-test
environment:
POSTGRES_DB: ${TB_POSTGRES_DB:-thingsboard}
POSTGRES_USER: ${TB_POSTGRES_USER:?set TB_POSTGRES_USER}
POSTGRES_PASSWORD: ${TB_POSTGRES_PASSWORD:?set TB_POSTGRES_PASSWORD}
ports:
- "${TB_POSTGRES_PORT:-5432}:5432"
networks:
- tb-iotdb-test
healthcheck:
test:
[
"CMD-SHELL",
"pg_isready -U \"$${POSTGRES_USER}\" -d \"$${POSTGRES_DB}\"",
]
interval: 10s
timeout: 5s
retries: 12
start_period: 10s

thingsboard:
image: thingsboard/tb-node:4.3.1.2
container_name: thingsboard-table-test
depends_on:
iotdb:
condition: service_healthy
postgres:
condition: service_healthy
environment:
TB_QUEUE_TYPE: ${TB_QUEUE_TYPE:-in-memory}
SPRING_DATASOURCE_URL: jdbc:postgresql://postgres:5432/${TB_POSTGRES_DB:-thingsboard}
SPRING_DATASOURCE_USERNAME: ${TB_POSTGRES_USER:?set TB_POSTGRES_USER}
SPRING_DATASOURCE_PASSWORD: ${TB_POSTGRES_PASSWORD:?set TB_POSTGRES_PASSWORD}
DATABASE_TS_TYPE: ${DATABASE_TS_TYPE:-sql}
DATABASE_TS_LATEST_TYPE: ${DATABASE_TS_LATEST_TYPE:-sql}
IOTDB_HOST: ${IOTDB_HOST:-iotdb}
IOTDB_PORT: ${IOTDB_PORT:-6667}
IOTDB_USERNAME: ${IOTDB_USERNAME:?set IOTDB_USERNAME}
IOTDB_PASSWORD: ${IOTDB_PASSWORD:?set IOTDB_PASSWORD}
INSTALL_TB: ${INSTALL_TB:-true}
LOAD_DEMO: ${LOAD_DEMO:-false}
ports:
- "${TB_HTTP_PORT:-8080}:8080"
networks:
- tb-iotdb-test
healthcheck:
test: ["CMD-SHELL", "bash -ec ': >/dev/tcp/127.0.0.1/8080'"]
interval: 15s
timeout: 5s
retries: 20
start_period: 90s

networks:
tb-iotdb-test:
driver: bridge
Loading