Pass pixel masks as a list of files and add a NeXus mask format by SimonHeybrock · Pull Request #861 · scipp/ess

SimonHeybrock · 2026-11-09T12:25:10Z

Summary

Workflows now take a list of mask files, PixelMaskFilenames, instead of mapping the workflow over filenames with with_pixel_mask_filenames. This applies to the SANS and powder workflows. A single provider reads all files into PixelMasks, a dict by mask name that covers the whole instrument. It can also be set directly from memory. A separate provider converts it to DetectorMasks for one detector.

# before
wf = sans.with_pixel_mask_filenames(wf, filenames)
# after
wf[PixelMaskFilenames] = filenames

This is a breaking change: with_pixel_mask_filenames, PixelMaskFilename and MaskedDetectorIDs are removed.

Why

Map/reduce changes the graph depending on the number of files, and we are migrating away from it (#746). Reading is independent of the run and the bank, so it runs once. Only the conversion depends on the bank. With sciline.Stage, DetectorMasks stays a flat dict[str, sc.Variable], and the bank level exists only in the file content. It also fixes two existing bugs: an empty list of files crashed in SANS, and LokiAtLarmorTutorialWorkflow assigned a list to a single-filename parameter. Sans2d now applies mask files together with its computed edge and sample holder masks.

Masks from NeXus files are named by their dataset names. Masks from ID files are named by the file name without its directory, since directories differ between hosts. The graph does not depend on the number or the type of mask files.

Mask file format

The second commit adds a format for masks of detector banks, needed for masks of tubes or straws (#767). It can be split into its own PR if reviewers prefer. Mask files are archived for years, so the format uses plain HDF5 types and the group structure of NeXus, and not the scipp HDF5 layout:

/entry                         NXentry
    instrument                 NXinstrument
        loki_detector_0        NXdetector
            masks              NXcollection
                broken_tubes   uint8, HDF5 dimension labels ("tube",)
                beamstop       uint8, HDF5 dimension labels ("layer", "tube", "straw")

The dataset name is the mask name, and each mask has its own dimensions. A file can hold all masks of an instrument, or masks can be split into files, for example by validity period. The ADR in this PR records the layout and the rejected alternatives, including the NeXus pixel_mask field.

ISIS XML files and scipp HDF5 files with a list of detector IDs are still read. Powder workflows accept only these ID-based files for now, because they do not pass a bank name when converting masks.

Workflows do not check validity dates. Selecting the mask files valid for a run is left to the caller that sets the list of filenames.

Before merging

esssans and essdiffraction import ess.reduce.masking, so they need an essreduce release that contains it. The essreduce>=26.10.1 pin is not raised yet, because that release does not exist yet.

esslivedata and essapps call the removed API and need updating after the release.

Follow-up

DetectorMasks[RunType], in a separate PR.

Test plan

Unit tests for reading, writing and converting masks in essreduce, and workflow tests for LoKI, Sans2d and powder, including a test that a NeXus mask file gives the same mask as the Larmor ID file.
On Larmor data, the list parameter gives I(Q) identical to the map/reduce version.
Ran the loki-iofq, dream-powder-reduction and esssans zoom notebooks. The other edited notebooks were not run, and the docs were not built.

🤖 Generated with Claude Code

Workflows now take PixelMaskFilenames and read all mask files in one provider, instead of mapping over the files with sciline's map/reduce. This is the change that #746 lists for esssans and essdiffraction. The result does not depend on the number of files, so an empty list needs no special case. The reader and the conversion to boolean masks move to ess.reduce.masking, so that esssans and essdiffraction share them. The reader selects the format by file suffix (ISIS XML or scipp HDF5). Masks are named after the file without its directory, since directories differ between hosts. Two files with the same name raise an error, because one mask would otherwise replace the other. Breaking: with_pixel_mask_filenames, PixelMaskFilename and MaskedDetectorIDs are removed. esssans and essdiffraction need a release of essreduce that contains ess.reduce.masking. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A NeXus file can now hold a mask for each detector bank, in the logical dimensions of the bank. This allows small masks of tubes or straws (#767), which a list of detector IDs cannot express without listing every pixel. Masks use the pixel_mask field of NXdetector, with HDF5 dimension labels. Mask files are archived for years, so they do not use the scipp HDF5 layout, which is defined only by the code of scipp. The ADR records the layout and the reasons. Masks are checked when converted, because scipp silently ignores a mask with a dimension that the data lacks. A key that is not a bank of the instrument raises, since a misspelled name would otherwise mask nothing. The SANS provider now needs the bank name. Zoom has none, so ISIS gets a provider without it, and a per-bank file raises there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The value does not have to come from a file: it can be set directly, as the BEER workflow does. It holds the pixel masks of the whole instrument, and DetectorMasks holds them for one detector. to_pixel_masks is renamed to to_detector_masks, since it returns what the SANS workflows call DetectorMasks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

PixelMasks can be set directly, so its values are not limited to what the file readers produce. A boolean variable was treated as a list of detector IDs, so that [True] masked the pixel with ID 1, and an integer mask for a bank was passed on as a mask. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Sans2d builds DetectorMasks from its edge and sample holder masks only, so setting PixelMaskFilenames had no effect and gave no error. It now combines both. The list of files defaults to empty, so that the workflow works without mask files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The reader looks only at /entry. Any other entry would be ignored without an error, which would leave pixels unmasked. This also applies if files with several entries are supported later, and an older reader gets one of them. The ADR describes that possible extension. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A mask file may use any order of dimensions, so DetectorMasks could have a different layout depending on the file. It now always follows the detector. Add tests for masks with transposed dimensions or a subset of them, and for files whose sizes or labels do not fit the detector, including a comparison of I(Q) with an equivalent list of detector IDs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

SimonHeybrock · 2026-11-09T13:53:28Z

@nvaytet Can you have a look at this, starting with the ADR and proposed mask file format, since it is related to your work in #834.

nvaytet · 2026-11-09T19:43:40Z

+    return PixelMasks(masks)
+
+
+def to_detector_masks(


Do we have to do this because the ISIS format is to list masks for a flat detector pixel list?
I would have expected/hoped we could just keep the logical dimensions of the detectors in the masks.

It should preferably not be that formats from other facilities force us to do extra work, but instead do extra work when working with data from other facilities.

Do we have to do this

Is "this" referring to the entire function?

Stripping XML support completely would be an option, but I did not want to make the decision here. Since it is already supported there was no reason to remove it now.

By "this" I meant having to map a mask with logical dimensions to detector numbers. I would have hoped the RawDetector is a DataArray with logical dims (folded after loading), and the mask also have logical dims that are compatible with the RawDetector.

Good question. The code made this hard to see. Masks per bank (the NeXus format) are matched to the folded detector by logical dims only, with no detector-number lookup. The ID lookup is used only for files that list detector IDs, such as ISIS XML. to_detector_masks used to take only the IDs, which suggested every mask went through them. It now takes the detector and uses detector_number only for ID-list masks (de65bac).

Mask files hold any number of named masks, as datasets in /entry/instrument/<bank>/masks (NXcollection), instead of one NeXus pixel_mask bit field per bank. The dataset name is the mask name, a nonzero value masks a pixel, and each mask has its own dimensions. This allows one file for all masks of an instrument, and tube and straw masks no longer have to be combined into one array. Masks with different validity periods go into separate files; workflows use the masks of all files and raise on duplicate names. The writer stores uint8, since HDF5 has no boolean type. The reader accepts any integer type and the h5py boolean enumeration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

SimonHeybrock · 2026-11-10T09:58:46Z

@nvaytet Thanks for the review. Your comments convinced us that following the NeXus pixel_mask definition costs more than it gives: one unnamed mask per bank, many files per instrument, and tube and straw masks merged into one array. The format is now:

/entry                         NXentry
    instrument                 NXinstrument
        loki_detector_0        NXdetector
            masks              NXcollection
                broken_tubes   uint8, dims ("tube",)
                beamstop       uint8, dims ("layer", "tube", "straw")

The dataset name is the mask name. There is no title, and NeXus masks are not named after the file.
Each mask has its own dims.
One file can hold all masks of an instrument. Several files still work, for example for masks with different validity periods. Workflows use the masks of all files, and duplicate names raise.
A pixel is masked if its value is not zero. There is no bit field.

The ADR gives the reasoning, including why the masks sit in an NXcollection and not directly in NXdetector. This settles the threads on pixel_mask, pixel_mask_N and the OR question, so I resolved them. I also expanded the to_detector_masks docstring: masks per bank keep their logical dims, and only masks given as detector ID lists (ISIS XML) are looked up pixel by pixel.

LokiAtLarmorTutorialWorkflow sets PixelMaskFilenames to a list of paths. Creating its widget failed on str.join, and the parameter box then showed no parameters at all, which broke the LoKI widget notebook. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

nvaytet

Looks good now 👍

nvaytet · 2026-11-10T16:20:58Z

-  `ess.reduce.masking.write_pixel_masks` sets bit 8, "user defined mask".
- Dimensions of `pixel_mask` are named with HDF5 dimension labels.
+- The name of a dataset in `masks` is the name of the mask in the reduced data.
+  Datasets with the same name in several banks are one mask.


What do you mean exactly by "are one mask"? Do they get loaded as a single data object that spans multiple NXdetectors?

nvaytet · 2026-11-10T16:22:04Z

-            pixel_mask             int32, HDF5 dimension labels ("layer", "tube")
+            masks                  NXcollection
+                broken_tubes       uint8, HDF5 dimension labels ("tube",)
+                beamstop           uint8, HDF5 dimension labels ("layer", "tube", "straw")


Maybe I missed it but should we explain why it's unit8 and not booleans?

HDF5 does not support bool.

Yes, I remembered that. I was asking if we could explain this in the ADR, so that the next person coming along doesn't ask the same question?

Masks given per bank match the detector by its logical dimensions and never use detector IDs. Taking only the IDs hid this, and suggested that every mask goes through an ID lookup. The function now takes the detector data array: masks per bank use its sizes, and only masks given as a list of IDs read the ID coordinate. Its name is a parameter, since ISIS data from Mantid and POWGEN data do not use 'detector_number'. The SANS DetectorIDs key existed only to feed masking, so it is removed and the mask providers take EmptyDetector[SampleRun]. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

jl-wynen · 2026-11-11T10:38:45Z

+  NeXus defines them as the start and end of a measurement, which can differ from the validity of a mask.
+  For example, a mask derived from a run in November can be valid from the day in October when a tube broke.
+  A reader of an archived file would also take them for measurement times.
+  `end_time` is usually unknown when a file is archived, since a static mask stops being valid when the hardware changes, and the file cannot be edited then.


Even if there is no validity encoded, we should still add a creation time. That way, you can at least get a rough idea of whether a file might still be valid.

Good point. write_pixel_masks now writes the NeXus root attributes file_time and creator, the same as kafka-to-nexus does for raw files.

jl-wynen · 2026-11-11T10:53:20Z

+```text
+/entry                             NXentry
+    valid_from                     "2026-11-01T00:00:00Z", not read by workflows
+    instrument                     NXinstrument


I'd add a name dataset to the instrument since the detector names are not always so easy to recognise. (Same as raw data files)

Good point, maybe also facility or similar fields we have in raw NeXus files?

Seems there is no facility in the NeXus files, so not doing that for now (only sth. part of NXsource), I think the instrument name should suffice.

write_pixel_masks takes the instrument name and writes it to /entry/instrument/name, as in raw data files, since bank names do not always identify the instrument. It also writes the NeXus root attributes file_time and creator, so a file without a validity date still shows roughly when it was made. Workflows do not read these. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

jl-wynen · 2026-11-11T13:15:04Z

-        instrument = entry.create_class('instrument', snx.NXinstrument)
+    with h5py.File(filename, 'w') as h5file:
+        h5file.attrs['file_time'] = datetime.now(UTC).isoformat(timespec='seconds')
+        h5file.attrs['creator'] = f'essreduce {__version__}'


NeXus uses a separate creator and version:

@creator: (optional) [NX_CHAR](https://manual.nexusformat.org/nxdl-types.html#nx-char) facility or program where file originated @creator_version: (optional) [NX_CHAR](https://manual.nexusformat.org/nxdl-types.html#nx-char) Version of facility or program used in writing the file

NeXus defines the root attributes `creator` (program that wrote the file) and `creator_version` (its version) as separate fields. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

SimonHeybrock and others added 2 commits November 9, 2026 10:52

SimonHeybrock self-assigned this Nov 9, 2026

github-actions Bot added essdiffraction Issues for essdiffraction. essreduce Issues for essreduce. esssans Issues for esssans. labels Nov 9, 2026

SimonHeybrock and others added 8 commits November 9, 2026 12:47

Use the name to_detector_masks in test names

5607075

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Record why mask files do not use NeXus start_time and end_time

2a4b6c7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Record why mask files do not use the NeXus axes attribute

f557f63

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

SimonHeybrock marked this pull request as ready for review November 9, 2026 13:53

SimonHeybrock requested a review from nvaytet November 9, 2026 13:53

nvaytet reviewed Nov 9, 2026

View reviewed changes

nvaytet approved these changes Nov 10, 2026

View reviewed changes

SimonHeybrock and others added 2 commits November 11, 2026 07:06

Clarify masks across banks and why mask files use uint8

6f9e2ab

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

jl-wynen reviewed Nov 11, 2026

View reviewed changes

Write creator and creator_version separately in mask files

83d325e

NeXus defines the root attributes `creator` (program that wrote the file) and `creator_version` (its version) as separate fields. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

jl-wynen approved these changes Nov 12, 2026

View reviewed changes

SimonHeybrock added this pull request to the merge queue Nov 12, 2026

Merged via the queue into main with commit 8b24894 Nov 12, 2026
24 checks passed

SimonHeybrock deleted the pixel-mask-filenames-list branch November 12, 2026 08:54

Conversation

SimonHeybrock commented Nov 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

Mask file format

Before merging

Follow-up

Test plan

Uh oh!

SimonHeybrock commented Nov 9, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SimonHeybrock commented Nov 10, 2026

Uh oh!

nvaytet left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

SimonHeybrock commented Nov 9, 2026 •

edited

Loading