[temiroff]

Description

Cameras can now run tensor processors on their captures. CameraCfg.tensor_processors lists TensorProcessor classes (#8693) with named inputs and outputs:

camera = replace(
    front_camera,
    data_types=["rgb"],
    tensor_processors=[
        TensorProcessorCfg(func=ShadeDepth, inputs=["distance_to_image_plane"], outputs=["rgb"]),
    ],
)

• Renderer requests. The renderer is asked for the processors' inputs that no earlier processor produces, and not for outputs a processor produces. Here it renders only distance_to_image_plane.
• Running. The camera constructs each processor at the first published capture with its configuration and the named camera buffers, then runs the processors in order on every new capture and adds their outputs to camera.data.output and camera.data.info, so they appear from the first published capture on. A processor can read earlier processors' outputs and can replace a render output it reads.
• Lifecycle. Camera resets call reset(env_ids) with only the reset environments; teardown calls close() on every processor and still releases the renderer data if one fails.
• env_mask. With update_period > 0, environments that reset at different times capture on different steps, so a call can mix new rows with rows that still hold the previous capture. A processor that lists "env_mask" in its inputs receives a boolean mask of the environments with a new capture. The name is reserved, and the mask is not requested from the renderer or published. It describes the current capture request, so processors reading it need synchronous rendering; the camera raises an error with async_rendering=True.
• Published captures. BaseRenderer.read_output, RenderContext.render_into_camera and render_into_cameras return whether a capture was published, so processors run once per new capture. Isaac RTX, OVRTX and Newton Warp return it.

This is the camera part of #8834, split out so it can merge independently of moving PPISP onto it.

Compatibility

• Custom renderers should return True from read_output after publishing a capture; a renderer returning None publishes as before but runs no processors.
• Cameras without tensor_processors behave as before.

Validation

Run on Windows with a CUDA GPU, Newton Warp renderer:

• source/isaaclab/test/renderers/test_camera_output_contract.py, test_simulation_render_context.py, and source/isaaclab_ppisp/test: 72 passed. This covers processor chaining and replacement, env_mask delivery and exclusion, the asynchronous rendering error, partial resets reaching only the reset rows, closing on teardown including a failing close(), and per-request published flags.
• A Cosmos camera integration built on this branch (depth in, generated rgb out, through a real Newton camera): 138 passed.
• isaaclab --format and the changelog check: passed.
• Not run locally: Isaac RTX and OVRTX camera tests, which need Isaac Sim; CI runs them.

Type of change

• New feature (non-breaking change which adds functionality)
• Documentation update

Checklist

• I have read and understood the contribution guidelines
• I have run the pre-commit checks with uv run isaaclab --format
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package
• I have added my name to the CONTRIBUTORS.md or my name already exists there

[greptile-apps]

Confidence Score: 4/5

[Medium impact] The changes affect camera capture workflows across the supported renderer backends.

Fix the asynchronous capture mask before merging so stateful processors update the right environments.

Findings

1.  Processors update the wrong rows ▶
2.  Failed close skips camera cleanup ▶

Summary

Adds ordered camera tensor processors, including renderer-input selection, per-capture execution, selected-row resets, and teardown. Renderers now report whether they published a capture.

• Camera captures can now flow through ordered tensor processors.
• Renderers now report whether each camera capture is new.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Resolve processor inputs] --> B[Request renderer buffers]
    B --> C[Submit camera capture]
    C --> D{Capture published?}
    D -- No --> E[Keep previous processor outputs]
    D -- Yes --> F[Run processors in order]
    F --> G[Publish named outputs]
    H[Camera reset] --> I[Reset selected processor rows]
    J[Camera teardown] --> K[Close processors]

Loading

_{Reviews (1) · Last reviewed commit: "Run tensor processors on camera captures" · Reviewed by Greptile}

[greptile-apps]

Processors update the wrong rows

With OVRTX async_rendering=True and staggered captures, env_mask describes the current request rather than the image just published. OVRTX publishes the previous pending capture, but this line passes the current _is_outdated flags. If the previous request updated row 0 and the current request updates row 1, a stateful processor skips row 0 and advances row 1 using the older image.

Save the mask with each submitted capture and pass the published capture’s mask to processors.

Knowledge Base Used: Omniverse backend

[temiroff]

Fixed in ad3516e: a processor that lists env_mask now requires synchronous rendering, and the camera raises a ValueError saying so. The mask marks the current capture request, while asynchronous rendering publishes the previous request's image. Processors without env_mask still work with asynchronous rendering. Covered in test_capture_mask_is_not_requested_from_the_renderer.

[greptile-apps]

Failed close skips camera cleanup

If a processor’s close() raises, _invalidate_initialize_callback skips later processors, renderer cleanup, the parent callback, and the frame view. The destructor also skips later processors and renderer cleanup. This can leave resources open and the camera only partly shut down.

Isolate processor close failures and use finally blocks so the remaining camera cleanup still runs.

[temiroff]

Fixed in ad3516e and aecf237: the camera closes every processor, then releases the render data, the parent state and the frame view in finally, and raises the close failure afterwards (one error as is, several as a RuntimeError from the first, as RenderContext does). Covered in test_failed_processor_close_still_releases_the_camera.

[isaaclab-review-bot]

Isaac Lab Review Bot

Needs attention: Design and architecture (1), API (1), Implementation (1), Style consistency (3), Test quality (1). Other reviewed categories look good.

• PR description: The title and body match the feature and validation scope, but the non-breaking classification is inaccurate because BaseRenderer.read_output changes a public extension contract without a deprecation bridge.
• Design and architecture: CameraData.info registers processor outputs before CameraData.output contains them, breaking their mirrored key contract until the first published capture; publish both key sets together.
• Scope and over-engineering: No over-engineering issues.
• API: RenderContext declares boolean returns but forwards legacy None values unchanged; normalize read_output results or explicitly type and handle the compatibility value.
• Compatibility and deprecation: No breaking changes identified.
• Implementation: Each published capture allocates a mask proxy and repeatedly rebuilds merged mappings; cache the proxy and reuse one working buffer mapping.
• Style consistency: Camera configuration crosses module-private boundaries and probes an undeclared backend field; expose the helper appropriately, declare async_rendering on RendererCfg, and move CAPTURE_MASK_NAME above the class.
• Test quality: The integration tests cover distinct processor contracts, but the renderer-request test drives a private helper through stubs; validate real CameraCfg instances instead.

Significant concerns. Posted 7 inline findings.

Automated review; human maintainers own approval decisions.

[isaaclab-review-bot]

🟡 Warning · Api — Declared list[bool] can contain None

The PR explicitly supports renderers that still return None, so entries here and the value from render_into_camera can be None despite the declared bool types. Wrap with bool(renderer.read_output(...)).

[temiroff]

Fixed in c09e597: render_into_cameras wraps each result in bool(...), so a renderer that still returns None reads as "not published".

[isaaclab-review-bot]

🟡 Warning · Style Consistency — Private helper imported across modules

_render_data_types is underscore-private, meaning internal to camera_cfg.py, yet camera.py and the new test import it. Give it a public name or move it into camera.py.

[temiroff]

Kept: _render_data_types is shared by CameraCfg.validate_config and Camera, so both check the same rule. It stays private to the camera package instead of becoming public API.

[isaaclab-review-bot]

🟡 Warning · Implementation — Per-capture proxy and dict allocations

Each published capture wraps _is_outdated in a new ProxyArray and merges two dicts per processor. Build the mask proxy once in _create_buffers and reuse a single working mapping.

[temiroff]

Kept: this allocates one small proxy and a few dict references per published capture, next to a full render; caching them would add state that must be rebuilt when the camera reinitializes.

[isaaclab-review-bot]

🟡 Warning · Design Architecture — info lists outputs before they exist

Processor names enter data.info here but data.output only after the first published capture, so with async rendering data.output[name] raises KeyError. Seed or defer both key sets together.

[temiroff]

Fixed in c09e597: processor outputs enter CameraData.info when they are first published, in the same call that adds them to CameraData.output, so the two key sets stay mirrored before the first capture.

[isaaclab-review-bot]

🟡 Warning · Style Consistency — Constant declared after the class

The prescribed file order puts constants before classes, as _DEPRECATED_RENDERER_FIELD_DEFAULTS already does in this module. Move CAPTURE_MASK_NAME into that block above CameraCfg.

[temiroff]

Fixed in c09e597: CAPTURE_MASK_NAME moved to the constants above CameraCfg.

[isaaclab-review-bot]

🟡 Warning · Style Consistency — getattr default probes backend capability

getattr(cfg.renderer_cfg, "async_rendering", False) hides a backend-specific field behind a default, against the direct-attribute rule. Declare the capability on RendererCfg and read it directly.

[temiroff]

Kept: only the OVRTX renderer has async_rendering. Declaring it on RendererCfg would add a base-class field for this one check.

[isaaclab-review-bot]

🟡 Warning · Test Quality — Test drives private helper via stubs

This asserts private _render_data_types against SimpleNamespace stubs, so CameraCfg field changes go unnoticed. Assert the same reserved-name and async errors through validate() on a real CameraCfg.

[temiroff]

Kept: validate_config returns before this rule for renderers without output specs, so the stubs test the rule directly. The real-camera path is covered by test_camera_tensor_processors_chain_and_replace_render_outputs.