Skip to content

handles ndarray type parsing for types in short notation with bytes - #9

Merged
ctrueden merged 7 commits into
apposed:mainfrom
gletort:ndarray-bytsize
Sep 25, 2026
Merged

ctrueden merged 7 commits into
apposed:mainfrom
gletort:ndarray-bytsize

Conversation

@gletort

@gletort gletort commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

In NDArray() function, dtype str can be a short notation in bytes instead of bits (more standard). Handles that case by not dividing by 8 when it's not a standard notation.

@gletort
gletort marked this pull request as draft August 24, 2026 09:16
@gletort
gletort marked this pull request as ready for review August 24, 2026 09:21
ctrueden added a commit to apposed/appose-java that referenced this pull request Sep 23, 2026
DType.fromLabel now accepts NumPy-style short forms (e.g. u2, f4, |u1,
=c8) in addition to standard labels, consistent with appose-python.
Explicit byte orders (< or >) are rejected, since array data is always
in native byte order. Labels are now matched case-sensitively.

Also add DType.FLOAT16.

See also apposed/appose-python#9.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
NDArray now accepts NumPy-style short forms (e.g. u2, f4, |u1, =c8)
in addition to standard names, normalizing them to the standard name
(e.g. uint16), so that only standard names cross the IPC channel.

Only platform-independent types are supported, via a fixed lookup
table, so that parsing behaves the same in every environment. Explicit
byte orders (< or >) are rejected: array data is always native order.

See also apposed/appose-java@655db12.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ctrueden

ctrueden commented Sep 23, 2026 •

Copy link
Copy Markdown
Member

@gletort Thanks for the detailed report on Zulip as well as this PR, and apologies for the delay in response. Your diagnosis was spot on: short-form dtypes like >u2 count bytes rather than bits, so Appose was allocating 1/8 of the needed memory.

While digging in, I decided dtype handling needed a more thorough fix across both appose-python and appose-java, so Claude and I built on your branch:

  • NumPy-style short forms (u2, f4, |u1, =c8, ...) are now accepted and normalized to standard names (uint16, ...), with a fixed table of platform-independent types, so parsing behaves identically everywhere. float16 is now supported too.

  • Appose arrays are always in native byte order (shared memory never crosses machines), so an explicit </> prefix is now rejected with a clear error rather than silently misinterpreted.

  • We also added a new convenience for exactly your use case:

    shared = appose.NDArray.from_ndarray(img)

    It allocates the shared memory and copies the data in one step. Your big-endian image works directly: NumPy byte-swaps the values during the copy, so no
    newbyteorder conversion is needed.

One gotcha: str(img.dtype) gives '>u2' for non-native arrays, whereas img.dtype.name always gives the standard name ('uint16'). So if you construct an NDArray yourself, use dtype.name.

ctrueden and others added 2 commits September 23, 2026 15:47
It allocates shared memory matching the array's dtype and shape, then
copies the data by assignment, so that NumPy converts values into the
native byte order and C-ordered layout. This handles e.g. big-endian
images from file readers, whose str(dtype) is '>u2' and thus rejected.

Add numpy as a dev dependency, for testing it. As a consequence, the
worker in test_crash_with_active_task now warns that numpy is installed
but not imported, polluting the stderr under test; so import it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a dtype like '>u2' is rejected, the error now names the standard
dtype to use instead (e.g. 'uint16'), and points to arr.dtype.name and
NDArray.from_ndarray. Also document that str(arr.dtype) is not suitable
for non-native arrays, whereas arr.dtype.name is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gletort

gletort commented Sep 24, 2026

Copy link
Copy Markdown
Contributor Author

On my use case, switching the str(img.dytpe) to str(img.dtype.name) works well now, without needing anymore the newbyteorder. But I don't see the appose.NDArray.from_ndarray fonction ? It's in the shm.py file ? Maybe I messed up my pull of this branch

@ctrueden

Copy link
Copy Markdown
Member

I don't see the appose.NDArray.from_ndarray fonction ? It's in the shm.py file ?

Ahh, I forgot to push. 🤦 Done now! I'll merge it as soon as tests pass.

Name the NumPy-to-Appose copy after ShmImg.copyOf in imglib2-appose.
For Appose-to-NumPy, implement __array__ so numpy.asarray(nda) wraps
the shared memory without copying, and deprecate ndarray() in favor of it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ctrueden

ctrueden commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

OK, I lied: I decided to fix the horribly confusing naming of NDArray.ndarray() returning a NumPy array. It is now more aligned with appose-java and imglib2-appose:


Copy a native array into new shared memory

  • appose-python: NDArray.copy_of(arr)
  • imglib2-appose: ShmImg.copyOf(rai)
  • appose-java: —

View shared memory as a native array (no copy)

  • appose-python: numpy.asarray(nda)
  • imglib2-appose: NDArrays.asArrayImg(nda)
  • appose-java: nda.buffer()

Get an NDArray from a native array (view if it wraps one, otherwise copy)

  • appose-python: —
  • imglib2-appose: NDArrays.asNDArray(rai)
  • appose-java: —

@ctrueden
ctrueden merged commit 4ce9aec into apposed:main Sep 25, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants