On TensorRT-RTX 1.5.0.114 (core reports [TensorRT v10500] [b114]), an FP8 (E4M3)
quantized convolution engine builds and deserializes successfully but crashes
the process the moment it executes the first FP8 conv kernel, with Windows exit status
0xC0000094 (STATUS_INTEGER_DIVIDE_BY_ZERO). No TensorRT/CUDA error is emitted. The
equivalent FP16 engine runs fine. Reproduces with a minimal 3-conv network.It always crashes with exit code 127 = 0xC0000094 = STATUS_INTEGER_DIVIDE_BY_ZERO.
make_repro.py builds a 3-layer conv net (Conv 3->64, Conv 64->64, Conv 64->3, ReLU),
exports FP32 ONNX, and FP8-quantizes it (random calibration data, so all scales are
well-conditioned and non-zero). Produces tiny_fp8.onnx.
- generate the FP8 onnx (deps: torch, onnx, nvidia-modelopt[onnx])
- AOT-build the engine -> SUCCEEDS
tensorrt_rtx --onnx=tiny_fp8.onnx --saveEngine=tiny_fp8.trt \
--minShapes=input:1x3x64x64 --optShapes=input:1x3x256x256 --maxShapes=input:1x3x512x512
- load only, no inference -> PASSES (exit 0)
tensorrt_rtx --loadEngine=tiny_fp8.trt --shapes=input:1x3x256x256 --skipInference
- run inference -> CRASHES, exit 0xC0000094 (STATUS_INTEGER_DIVIDE_BY_ZERO)
tensorrt_rtx --loadEngine=tiny_fp8.trt --shapes=input:1x3x256x256 --noDataTransfers --duration=3 --warmUp=500
Please see attached all the info, like a minimal python that reproduces the problem and the ONNX files. I have the fp32, (with its .data file), the fp16 which works (check the file tiny_fp16_run.log) and the fp8 model. I cannot upload here files, the link to DropBox:
https://www.dropbox.com/scl/fi/03fiu4tvr9bh58b7j3a8d/minrepro.rar?rlkey=115j9nitx29jju72jvkychoe4&st=zg513yuz&dl=0
Expected behavior
Expecting it to not crash.
Environment
- CPU: AMD Ryzen 7800X3D
- GPU: NVIDIA GeForce RTX 4070 SUPER (Ada, compute capability 8.9, 12 GB)
- Driver: 610.52
- TensorRT-RTX: 1.5.0.114 (engine core:
[TensorRT v10500] [b114])
- OS: Windows 11 (x64)
- CUDA: 13.3 (see attached full
nvidia-smi output)
- Quantizer: NVIDIA TensorRT ModelOpt 0.44.0 (
modelopt.onnx.quantization.quantize, fp8, calibration_method="max")
Additional context
- --skipInference succeeds (exit 0), the fault is at execution.
- The FP16 engine of the identical network runs fine on the same GPU/shape.
- Data-independent: random-uniform calibration (guaranteed non-zero, well-conditioned amax
scales) crashes again.
- Not OOM: the minimal net allocates ~100 MB scratch and a larger real model peaked at 3.4 GB of
12 GB before crashing.
- Fault type is an integer*divide-by-zero (0xC0000094) and not a float divide (a bad scale would be 0xC000008E I suppose, maybe not?).
- Reproduces across unrelated and various architectures (this 3-conv net, a compact SR SRVGGNet, and a
U-Net denoiser).
On TensorRT-RTX 1.5.0.114 (core reports [TensorRT v10500] [b114]), an FP8 (E4M3)
quantized convolution engine builds and deserializes successfully but crashes
the process the moment it executes the first FP8 conv kernel, with Windows exit status
0xC0000094 (STATUS_INTEGER_DIVIDE_BY_ZERO). No TensorRT/CUDA error is emitted. The
equivalent FP16 engine runs fine. Reproduces with a minimal 3-conv network.It always crashes with exit code 127 = 0xC0000094 = STATUS_INTEGER_DIVIDE_BY_ZERO.
make_repro.py builds a 3-layer conv net (Conv 3->64, Conv 64->64, Conv 64->3, ReLU),
exports FP32 ONNX, and FP8-quantizes it (random calibration data, so all scales are
well-conditioned and non-zero). Produces tiny_fp8.onnx.
Please see attached all the info, like a minimal python that reproduces the problem and the ONNX files. I have the fp32, (with its .data file), the fp16 which works (check the file tiny_fp16_run.log) and the fp8 model. I cannot upload here files, the link to DropBox:
Expected behavior
Expecting it to not crash.
Environment
[TensorRT v10500] [b114])nvidia-smioutput)modelopt.onnx.quantization.quantize, fp8, calibration_method="max")Additional context
scales) crashes again.
12 GB before crashing.
U-Net denoiser).