Conversation
|
Should also update the binary codec: https://github.com/latis-data/latis3-hapi/blob/master/src/main/scala/latis/util/hapi/DataCodec.scala#L24 |
Add longs to the type conversion operation that is applied before encoding and streaming. Disable filtering out datasets that have long data types from the catalog.
|
This only addresses the ability to serve LaTiS datasets that have variables defined as longs. The This will also allow LaTiS datasets with long type variables to be in the catalog of supported datasets. |
lindholc
left a comment
There was a problem hiding this comment.
I am somewhat uneasy about letting things silently overflow.
What if we 1) only converted variables that specify fill/missing/whatever values and use those values if the conversion fails or 2) building on 1, if no fill value is specified, use max int or min int or something and set that as the fill value.
|
Overflow is unfortunately common. Even Scala's The current reality is that we have data sources with variables defined as longs. I see it as a shortcoming that HAPI chose not to include support for 64-bit integers. |
We've had a number of datasets where we use longs and it would be a shame to prevent them from hapi access. There is the risk of overflow, however. One that we've ignored in other cases. The hapi spec (https://github.com/hapi-server/data-specification/blob/master/hapi-3.3.1/HAPI-data-access-spec-3.3.1.md) says thjis about data types:
Not even mentioning 64-bit integers (i.e. longs). We do support floats but since doubles are supported, we only risk precision noise.
Maybe we could convert to 32-bit integers, maybe using fill values for those that exceed the max int?
I still need to test the behavior of the 3 output formats.