LoKI: handle all detector banks in the specs and bindings #166

The specs and bindings in ess.instruments.loki handle one detector bank, because they are bound to LokiAtLarmorWorkflow. LoKI has 9 banks. This issue records the plan for handling all of them. Part of #24.

Design

There is one CALIBRATED_DETECTOR record per run, and the banks are inside its value. So a request has the same shape for 1 or 9 banks:

class DetectorImage(BaseModel, arbitrary_types_allowed=True):
    banks: dict[str, sc.DataArray]   # banks differ in shape, so they can't be one array
    sample_position: sc.Variable
    source_position: sc.Variable

class PixelMasks(BaseModel, arbitrary_types_allowed=True):
    masks: dict[str, dict[str, sc.Variable]]   # bank -> mask name -> bool over any of the bank's dims

detector = client.submit(CALIBRATED_DETECTOR, params={'run': file(run=60339)})
masks = client.submit(MASKS, params={'detector': detector.ref('detector'),
                                     'mask_files': [file(path=mask_file)]})
centre = client.submit(BEAM_CENTRE, params={'detector': detector.ref('detector'),
                                            'masks': masks.ref('masks')})
iofq = client.submit(IOFQ, params={'banks': ['loki_detector_0', ...], ...})

Per-bank I(Q), as asked for in docs/requirements/sans.md, is a batch of IOFQ records with banks=[name], all referencing the same masks and centre.

Plot(inputs=...), Listing.fitting, describe, and references between records need no change. A nested per-bank value of this shape went through an in-process client unchanged.

Mask files

One file is one mask, named by its file, as now. It maps the name of a bank to that bank's part of the mask. A tube index only means something within its bank, so the bank has to be in the file. The file format and the naming inside a file are under discussion in scipp/ess#834, and this plan depends only on the following:

• A mask has only the dims it needs, such as a tube. It is not always per pixel.
• A bank not listed has no mask from that file. A beam-stop mask only concerns the rear bank.
• A bank name the detector does not have is refused, and so is a mask that does not fit its bank's dims and sizes. Both fail in MASKS, not during a long I(Q). This is why MASKS keeps its detector reference.
• The XML files of pixel IDs stay readable if we still need them: their reader builds the same mapping from each bank's detector_number. Detector numbers do not overlap between banks in loki-coda-5-pulses.hdf.

def compute_masks(params: MasksParams) -> MasksOutputs:
    masks = {bank: {} for bank in params.detector.banks}
    for path in params.mask_files:
        for bank, mask in _read_mask(path, params.detector).items():  # one reader per format
            if bank not in masks:
                raise ValueError(f'{path.name}: unknown bank {bank}')
            params.detector.banks[bank].assign_masks(check=mask)  # raises if dims don't fit
            masks[bank][path.name] = mask
    return MasksOutputs(masks=PixelMasks(masks=masks))

Plots

on_detector makes one x-y histogram, which mixes banks at different distances. detector_image and masks_on_image need one figure per bank or a mosaic. centre_on_image should draw only the bank the centre came from, since the cross is valid in the plane at centre.z.

The generic view in panel/outputs.py needs a change too. Outputs._plot draws only arrays at the top level of an output, so it would show nothing for banks, where it now shows counts with sliders. _brief would print a raw repr of the dict.

Blocked by

ess.loki processes one bank at a time. ess/sans/workflow.py on main says so, because esssans has no way to merge the I(Q) of banks. Merging before normalization is step 1 of the plan in scipp/ess#763. esssans is meant to handle several banks by default.

• IOFQ is blocked: how _iofq_pipeline selects banks and gives each its masks depends on the interface esssans ends up with.
• CALIBRATED_DETECTOR, MASKS, CREATE_MASK, and BEAM_CENTRE could loop over NeXusDetectorName today, but we would be guessing the same interface. The only full reductions we test on have one bank (Larmor). loki-coda-5-pulses.hdf has 9 banks, but gives NaN for the centre of mass.

So this waits for esssans.

Discarded alternatives

• One record per bank. The file is read 9 times, and each downstream request needs lists of references (detectors: [ref × 9], masks: [ref × 9]), which the form shows as 9 pickers each. Nothing stops pairing the masks of one bank with the detector of another. This is also against ADR 0006: structure below a row belongs to the driver.
• detector: dict[str, DetectorImage] with keyed references. Listing.fitting offers only whole outputs and read_inputs refuses keyed references, and the keys of the detector and the masks would have to agree.
• A file per bank, with the bank in the params or the file's metadata. One mask becomes up to 9 files, or the per-bank lists come back into every request.

Open questions

• The file format, and whether a bank can hold several arrays in one file, for example one by tube and one by straw.
• Mask dim names are those of DETECTOR_BANK_SIZES. A file written with other names is refused.
• BEAM_CENTRE and each plot read all banks to use one: about 140 MB for 3.2 million pixels, assuming float64 counts and variances, not measured.
• Do we still need the XML pixel-ID files for LoKI at ESS?