Skip to content

Add support for long types - #243

Merged
dlindhol merged 7 commits into
mainfrom
long
Aug 5, 2026
Merged

Add support for long types#243
dlindhol merged 7 commits into
mainfrom
long

Conversation

@dlindhol

@dlindhol dlindhol commented Feb 4, 2026

Copy link
Copy Markdown
Member

We've had a number of datasets where we use longs and it would be a shame to prevent them from hapi access. There is the risk of overflow, however. One that we've ignored in other cases. The hapi spec (https://github.com/hapi-server/data-specification/blob/master/hapi-3.3.1/HAPI-data-access-spec-3.3.1.md) says thjis about data types:

Note that there are only a few supported data types: isotime, string, integer, and double. This is intended to keep the client code simple in terms of dealing with the data stream. However, the spec may be expanded to include other types, such as 4-byte floating-point values (which would be called float), or 2-byte integers (which would be called short).

Not even mentioning 64-bit integers (i.e. longs). We do support floats but since doubles are supported, we only risk precision noise.

Maybe we could convert to 32-bit integers, maybe using fill values for those that exceed the max int?

I still need to test the behavior of the 3 output formats.

@dlindhol
dlindhol requested a review from lindholc February 4, 2026 16:25
@dlindhol

dlindhol commented Feb 4, 2026

Copy link
Copy Markdown
Member Author

Add longs to the type conversion operation that is applied before encoding and streaming. Disable filtering out datasets that have long data types from the catalog.
@dlindhol

dlindhol commented Jul 28, 2026

Copy link
Copy Markdown
Member Author

This only addresses the ability to serve LaTiS datasets that have variables defined as longs. The ConvertHapiTypes operation will convert these to 32-bit integers (with a risk of overflow'd values) before being sent to the BinaryEncoder so the hapi.DataCodec does not need to deal with longs.

This will also allow LaTiS datasets with long type variables to be in the catalog of supported datasets.

@dlindhol
dlindhol marked this pull request as ready for review July 28, 2026 18:54

@lindholc lindholc left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am somewhat uneasy about letting things silently overflow.

What if we 1) only converted variables that specify fill/missing/whatever values and use those values if the conversion fails or 2) building on 1, if no fill value is specified, use max int or min int or something and set that as the fill value.

Comment thread src/main/scala/latis/ops/ConvertHapiTypes.scala Outdated
@dlindhol

dlindhol commented Aug 4, 2026

Copy link
Copy Markdown
Member Author

Overflow is unfortunately common. Even Scala's toInt does it. I would like to do better. I was thinking of an operation, used as a processing instruction, to convert types where we could centrally handle such cases. We would need the ability to push some operation like time selections before it. That's a bigger task.

The current reality is that we have data sources with variables defined as longs. I see it as a shortcoming that HAPI chose not to include support for 64-bit integers.

Comment thread src/main/scala/latis/service/hapi/HapiService.scala Outdated
Comment thread src/main/scala/latis/ops/ConvertHapiTypes.scala Outdated
@dlindhol
dlindhol merged commit 559d969 into main Aug 5, 2026
2 checks passed
@dlindhol
dlindhol deleted the long branch August 5, 2026 19:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants