Repository navigation
perf(rtx): parallelize MDL material loading - #379
Merged
Merged
Conversation
Record the Coordinator+Pool architecture for parallel MDL loading (ADR 0009) and the vocabulary it introduces (Coordinator, Module Preload, Compile Job, Compile Backend, Staged Decode) in the libmdl glossary.
FindMDL_SDK extracts MI_NEURAYLIB_PRODUCT_VERSION_STRING from version.h and fails configure below 2023.1.0 -- the release that introduced multiple parallel database transactions, which the coordinator+pool loading (ADR 0009) depends on. No functional change; a too-old SDK now fails loud at configure instead of racing at runtime.
add_material() writes function_index/return_code back into each Target_function_description entry, so the function-static array was shared mutable state. Harmless while compilation is single-threaded, but a data race once compile jobs run in parallel (ADR 0009). Make it a per-call local.
TestMdlConcurrentCompile drives many distinct `mdl` materials through a single flush -- the batch the coordinator+pool loading (ADR 0009) will fan out -- and pins the semantics parallelism must preserve: N distinct sources yield exactly N registry slots and a lit render; N distinct + N duplicates still dedup to N; a broken source in the batch falls back without taking its neighbours down. Green serially now; the same assertions gate the parallel path when run under VISRTX_MDL_COMPILE_THREADS>1. Also gitignore the Testing/ ctest scratch dir.
…er pool Move MDL material loading off the render/commit thread and compile a whole flush of materials in parallel (ADR 0009). MdlCompileCoordinator owns one dedicated coordinator thread plus a worker pool (VISRTX_MDL_COMPILE_THREADS, default 1). run() executes registry / MDL-SDK bookkeeping on the coordinator thread and blocks the caller, so the registry is touched by exactly one thread -- confinement instead of per-container locks. submit() runs the heavy, registry-free compile on a pool worker and returns a future. Teardown drains the workers (which reach back through run()) before the coordinator, and run()/submit() after stop() execute inline so device teardown never deadlocks. MaterialRegistry is split so the parallel and serial parts are separable: preloadModule (coordinator: load + commit a module so worker transactions read it without racing to reload it), compileMaterial (worker: own transaction, create_compiled_material + translate_link_unit + PTX stitch + emission IR into a transaction-independent CompileProduct, no registry mutation), and insertCompiled (coordinator: dedup by compiled-material hash, slot assignment in submission order, map + timestamp updates). acquireMaterialAsync chains these across the pool and the coordinator. MDL::commitParameters starts the compile (beginAsyncAcquire) and holds the future; finalize's syncSource consumes it when the resolved source still matches, releasing a superseded prefetch so it never leaks a slot, and the destructor drains a pending future for a material dropped mid-flush. MDLE and error paths also compile on pool workers; the commit thread waits for those requests, so a one-worker pool serializes every compile. All MaterialRegistry read/write sites (MDL.cpp, Renderer.cpp, VisRTXDevice.cpp) route through the coordinator. Validated: 35/35 MDL/emissive/MaterialX tests at VISRTX_MDL_COMPILE_THREADS=1 and =8, including TestMdlConcurrentCompile (24 materials compiling in parallel), plus 5x repeat runs of the concurrency-sensitive tests at 8 threads.
…ty gate Add the mdlPtxFingerprint device property -- an order-independent content hash of every compiled material's PTX -- and TestMdlPtxIdentity, which runs the same scene under VISRTX_MDL_COMPILE_THREADS=1 and =8 and asserts the fingerprints match. This is the silent-miscompile gate for ADR 0009: PTX corruption from the parallel compile path that never changes a rendered image still changes the fingerprint. Passing confirms concurrent translate_link_unit produces byte-identical code on the current SDK. Reject unexpected ANARI errors and missing requested materials before publishing the fingerprint. Negative tests cover total and partial fallback.
initOptixPipeline created each material's OptiX module serially. Create them on the compile pool instead (optixModuleCreate on one context is thread-safe -- the built-in renderer modules already build this way), each task with its own log buffer. SBT assembly stays a serial second pass, so callable order and every material's SBT offset are byte-for-byte the serial layout. Validated: 37/37 MDL tests at VISRTX_MDL_COMPILE_THREADS=8 and the PTX identity gate.
Default VISRTX_MDL_COMPILE_THREADS to hardware_concurrency (capped at 8 so a many-core host does not oversubscribe the shared SDK compiler); set it to 1 to force serial. The PTX-identity gate and the full suite at 8 threads validate the parallel path, so it is safe to enable by default. TestMdlCompileBenchmark times compiling a batch of compile-heavy materials in one flush and prints the elapsed time and thread count. Measured on this host: 64 materials compile in 16.3 s serially vs 2.3 s at 8 threads (7.1x).
compileMaterial ran the shared MDL entity resolver (Core::resolveResource) directly on the pool worker, so parallel compiles hit it concurrently (ADR 0009 requires texture resolution to be coordinator-serial). Route each resolveResource through coordinator.run when compiling on a worker; it runs inline on the coordinator thread (the synchronous MDLE/fallback path), where resolution is already serial. Caught by code review; validated on the MaterialX/textured path at VISRTX_MDL_COMPILE_THREADS=8.
…taged decode) A material can reference many texture files, and stb decode is pure CPU work done serially on the commit thread. Split loadFromImage into decodeStb (pure) and createFromStb (device/helium work), add a mutex-guarded staging map, and a stageDecodeBatch that finalize calls to fan the decodes out across the compile pool and wait on the commit thread (never on a pool worker, so a task never waits on its own pool). acquireSampler consumes the staged pixels, falling back to the inline path for BSDF-data, .dds (handed to the device as-is) and any decode that failed. stageDecodeBatch runs on the commit thread and reads the sampler cache (its sole mutator, blocked in the wait meanwhile) to skip already-cached textures, and de-duplicates repeated keys, so a texture referenced N times -- in one material or shared across the flush -- is decoded exactly once. Validated on the MaterialX/textured suite at VISRTX_MDL_COMPILE_THREADS=8.
Keep PTX snapshots owned and version-consistent. Confine module loading to the coordinator and expose only pool-based acquisition. Allocate worker storage before starting threads. Add regression coverage for serial fallback/MDLE compilation, snapshot lifetime, module handoff, and allocation failure.
tarcila
requested review from
jeffamstutz
and removed request for
jeffamstutz
October 23, 2026 15:46
Separate serial-lane and worker-pool execution from MDL configuration. Keep executor ownership and teardown ordering in the MDL state. Make worker count explicit, rename run() to invokeSerial(), and build standalone executor tests without MDL support.
jeffamstutz
approved these changes
Oct 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Move expensive
MDLcompilation, texture decoding, andOptiXmodule creation onto a worker pool while preserving blockingcommit/finalizesemantics.Default to hardware concurrency, capped at 8 workers. This can be changing using the environment variable
VISRTX_MDL_COMPILE_THREADS. Setting it to 1 serializes all compilation.Loading the NVIDIA OldAttic: