python(feat): add client side detect config for parquet by alden-sift · Pull Request #819 · sift-stack/sift

alden-sift · 2026-10-07T21:52:12Z

Summary

Flat dataset Parquet config detection fails for wide files (many columns and/or long column names) because the client sends the Parquet footer to the server's DetectConfig endpoint, and footers over the server's 20 MiB gRPC receive limit are rejected with RESOURCE_EXHAUSTED.

This PR detects PARQUET_FLATDATASET configs on the client from the footer schema with pyarrow, the same way HDF5, TDMS, ULog, and MCAP configs are detected. Single channel per row Parquet still uses server detection.

detect_config and import_from_path now also accept relative_start_time so files with relative time can be detected and imported without editing the config.

Also refactored by moving helper functions either into _internal module or moved them into ImportConfig as methods.

Dependency change: flat dataset Parquet detection now requires pyarrow. Install it with pip install sift-stack-py[parquet].

Validation

Recreation

detect_config on a Parquet file with 20,000 columns and long column names:

import os

import dotenv
import dotenv
from sift_client import SiftClient
from sift_client._internal.util.file import extract_parquet_footer
from sift_client.sift_types.data_import import DataTypeKey

dotenv.load_dotenv()
pq_file = "data/super-wide-and-big-name.parquet"
client = SiftClient(
    api_key=os.getenv("SIFT_API_KEY"),
    grpc_url=os.getenv("SIFT_GRPC_URL"),
    rest_url=os.getenv("SIFT_REST_URL"),
)
footer_bytes, _ = extract_parquet_footer(pq_file)
print(f"footer size: {len(footer_bytes)} bytes")
client.data_import.import_from_path(
    pq_file, asset="test-long-parquet-1", data_type=DataTypeKey.PARQUET_FLATDATASET
)

Before this change:

grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
        status = StatusCode.RESOURCE_EXHAUSTED
        details = "grpc: received message larger than max (29943806 vs. 20971520)"

After this change:

alden@MacBook-Pro eng-12274-test % /Users/alden/projects/eng-12274-test/.venv/bin/python /Users/alden/projects/eng-12274-test/main.py
size of footer bytes: 29943832
Upload [super-wide-and-big-name.parquet] |████████████████████████████████████████| 32.3MB/32.3MB [100%] in 0.7s (46.70MB/s)

Behavior validation

Automated test: ran the old (server side) and the new (client side) config detection on Parquet files covering every time type, scalar and complex types, structs, durations, and embedded configs to ensure identical behavior.
Manual test: Parquet files imported through the client and through the webapp produced identical runs.
Unit tests: _tests/_internal/test_parquet.py.
Integration tests

github-actions · 2026-10-07T21:53:29Z

Python docs preview: https://sift-stack.github.io/sift/python/pr-819/

Deployed from 7622ccd. The link may take up to a minute to become live as GitHub Pages propagates.

Detect PARQUET_FLATDATASET import configs locally from the Parquet footer schema with pyarrow instead of calling the server DetectConfig endpoint. Detection matches the server: struct fields are flattened, the first top-level time typed column is the time column, and configs embedded by Sift exports are used directly. The time column is excluded from the data columns, and an INT64/UINT64 column named ts, timestamp, or time is used when no time typed column exists. Add relative_start_time to detect_config and import_from_path. It applies to Parquet only and warns when ignored for other formats or absolute time formats. Add a parquet extra for pyarrow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Python 3.8 resolves pyarrow 17, which cannot build float16 arrays from Python floats or write view types to Parquet. Build the float16 column from numpy values and move the view types into a test that is skipped when the installed pyarrow cannot write them.

alexluck-sift · 2026-10-10T19:18:08Z

+    if proto.time_column.format == TimeFormatProto.TIME_FORMAT_UNSPECIFIED:
+        raise ValueError(
+            "timestamp format is not supported. Please specify the time format explicitly."
+        )


Why check the proto format here at all? detect_config already falls back to the caller's time_format, then ABSOLUTE_UNIX_NANOSECONDS (lines 416-422), but this raises before either runs. Making CsvTimeColumn.format optional like ParquetTimeColumn.format would let that fallback handle it.

CsvImportConfig._from_proto calls TimeFormat(proto.time_column.format) as the format of the CsvImportConfig. TimeFormat does not have a TIME_FORMAT_UNSPECIFIED member, so we run into unhandled errors. This all happens during the _detect_config_for_type call on line 415, before the fallback.

Fixed in commit 11c7d84, where TimeFormat gains the UNSPECIFIED member and is handled with the existing fallback.

Let's not add UNSPECIFIED to the public enum, since users could then pass it

removed UNSPECIFIED from public enum and instead checks for valid enum membership in CsvImportConfig._from_proto in commit 906b4ad

alexluck-sift · 2026-10-10T19:18:08Z

+    if not isinstance(
+        config, (ParquetFlatDatasetImportConfig, ParquetSingleChannelPerRowImportConfig)
+    ):


For single-channel-per-row the server-detected time column is built in ParquetTimeColumn._from_proto, and if the server reports a relative format the validator raises there, before this function can apply the caller's start time. I'd pass relative_start_time into the server path's _from_proto too, or scope the docstring to flat datasets.

scopped to flat dataset for now, implemented in commit 9a4b196

alexluck-sift · 2026-10-10T19:18:08Z

+from sift.common.type.v1.channel_config_pb2 import ChannelConfig as ChannelConfigProto
+
+from sift_client._internal.util.file import extract_parquet_footer
+from sift_client.resources.data_imports import _infer_time_column


nit: this is the only detector in _internal/util that reaches up into resources/, and for a private helper. I'd move _infer_time_column and its two constants into a pyarrow-free _internal/util module and import it from both places.

implemented in commit c54c883

alexluck-sift · 2026-10-10T19:18:08Z

+        )
+        return
+    time_format = config.time_column.format
+    if time_format is None or not time_format.name.startswith("RELATIVE_"):


nit: this is the third name.startswith("RELATIVE_") check (also data_import.py:143 and :837). An is_relative property on TimeFormat would keep them from drifting.

implemented in commit 9dbd258

alexluck-sift · 2026-10-10T19:18:08Z

+parquet = ["pyarrow>=17.0.0"]

 [tool.sift.extras.combine]
 # Combined extras configurations and aliases can be defined here


make sure to update this to reflect any new extras. parquet isn't in file-imports (line 281).

fixed in a0cf8a4 by adding parquet to file-imports.

alexluck-sift · 2026-10-10T19:18:08Z

+            relative_start_time: Start time for a relative time format.
+                Parquet only; ignored with a warning for other formats or


why is this parquet only? Pretty sure it is supported in other formats? CsvTimeColumn, TdmsImportConfig, and Hdf5ImportConfig all have relative_start_time.

implemented in f731566

alexluck-sift · 2026-10-10T19:18:08Z

        data_type: DataTypeKey | None = None,
        time_format: TimeFormat | None = None,
+        *,
+        relative_start_time: datetime | None = None,


datetime does not have nanosecond precision. Does the backend not support ns precision? if not, we really need to support it to ensure no data loss here. The proto field is a google.protobuf.Timestamp, which carries nanoseconds.

fixed in commit a6c19e3 . Opting for pandas Timestamp class as it is more ergonomic than the protobuf Timestamp class, and as it subclasses datetime, keeps the code backwards compatible/non breaking.

alexluck-sift · 2026-10-10T19:18:08Z


        data_type_key = _resolve_data_type_key(path.suffix.lower(), data_type)
-        config = await self._detect_config_for_type(path, data_type_key)
+        config = await self._detect_config_for_type(path, data_type_key, relative_start_time)


nit: recommend using key-value args in almost all situations to avoid issues with ordering

implemented in commit 0b773e1

alexluck-sift · 2026-10-10T19:18:08Z

        config.time_format = time_format


+def _apply_relative_start_time(config: ImportConfig, relative_start_time: datetime) -> None:


nit: recommend moving all of these private functions to _internal since they are not class members. This would be really messy for a user to read

moved helpers into _internal, and moved some config-specific helpers into ImportConfig methods in efe86ef

Co-authored-by: Alex Luck <luck@siftstack.com>

…me test

Import config fields accept a pd.Timestamp alongside datetime, and configs read from protos return a pd.Timestamp so nanoseconds survive a round trip. detect_config and import_from_path also accept an ISO 8601 string, parsed with pd.Timestamp.

TimeFormat gains UNSPECIFIED, matching the proto's TIME_FORMAT_UNSPECIFIED. A CSV DetectConfig response without a detected format no longer raises; detect_config applies the caller's time_format or ABSOLUTE_UNIX_NANOSECONDS instead. Importing with an unspecified format raises the existing "format must be set" errors.

… classification

Drop UNSPECIFIED from the public TimeFormat enum. CsvImportConfig maps the proto's TIME_FORMAT_UNSPECIFIED to None, matching Parquet, TDMS, and HDF5. TimeColumnBase.format now defaults to None, so CSV and Parquet time columns share the same rule: optional at construction, required at import.

…nto import configs

test

35d9e86

alden-sift requested a review from wei-sift October 7, 2026 21:52

alden-sift and others added 3 commits October 9, 2026 14:43

python(fix): lint pyproject.toml and update uv.lock

63ec9f2

alden-sift marked this pull request as ready for review October 9, 2026 22:39

alden-sift requested a review from alexluck-sift October 9, 2026 23:25

alexluck-sift reviewed Oct 10, 2026

View reviewed changes

alden-sift and others added 7 commits October 12, 2026 10:03

Update python/lib/sift_client/sift_types/data_import.py

27dc582

Co-authored-by: Alex Luck <luck@siftstack.com>

python(fix): scope relative_start_time to flat dataset Parquet configs

9a4b196

python(refactor): move infer_time_column into _internal.util

c54c883

python(refactor): make optional Parquet detection args keyword-only

0b773e1

python(build): include parquet in the file-imports extra

a0cf8a4

python(test): drop redundant single channel per row relative_start_ti…

bca2990

…me test

alden-sift self-assigned this Oct 12, 2026

alden-sift added 8 commits October 12, 2026 12:52

python(refactor): added is_relative property to TimeFormat for easier…

9dbd258

… classification

python(feat): support relative_start_time for all import formats

f731566

python(fix): raise when CSV detection finds no time column

69dda85

python(docs): clarify relative_start_time semantics for MCAP

f8ed1c6

python(refactor): move time format and relative_start_time handling o…

efe86ef

…nto import configs

python(build): define data-review extra as an alias of parquet

c456562

		relative_start_time: Start time for a relative time format.
		Parquet only; ignored with a warning for other formats or

		config.time_format = time_format


		def _apply_relative_start_time(config: ImportConfig, relative_start_time: datetime) -> None:

Conversation

alden-sift commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Recreation

Behavior validation

Uh oh!

github-actions Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alden-sift Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alden-sift Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alden-sift Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alden-sift Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alden-sift Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alden-sift Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alden-sift Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alden-sift commented Oct 7, 2026 •

edited

Loading

github-actions Bot commented Oct 7, 2026 •

edited

Loading

alden-sift Oct 12, 2026 •

edited

Loading

alden-sift Oct 12, 2026 •

edited

Loading

alden-sift Oct 12, 2026 •

edited

Loading

alden-sift Oct 12, 2026 •

edited

Loading

alden-sift Oct 12, 2026 •

edited

Loading

alden-sift Oct 12, 2026 •

edited

Loading

alden-sift Oct 12, 2026 •

edited

Loading