oneAPI-aware MPI - #914
Conversation
|
I have no idea what is failing here. Anyhow, before I add |
|
The failure looks to be unrelated. Is is possible for you to test your changes by adding a buildkite pipeline? For an example, you can see how oneAPI.jl runs their tests: https://github.com/JuliaGPU/oneAPI.jl/blob/a00fad6d0532ab7548f236b9293e9dac5845fd0e/.buildkite/pipeline.yml |
|
I've now tried building MPICH with I've added a |
|
Sounds good! Let us know what the MPICH team says. I am happy to merge if building a oneAPI-aware MPICH is too much trouble. |
|
@lcw, builds are currently failing on hardware without native FP64 support, such as the integrated GPU used in our Buildkite oneAPI runner. The MPICH OpenCL implementation has extensive FP64 dependencies; when ocloc attempts to compile these for non-FP64 devices, it triggers a segfault. This persists even when applying ocloc emulation flags. I don't have a Intel GPU with FP64 support that I could add to buildkite. Given these hardware constraints, I suggest we merge this as an experimental feature for now. |
Build MPICH with Level Zero support and run the test suite against it with `--backend=oneAPI` on the oneapi queue. The Level Zero loader, headers and ocloc all come from the same JLLs oneAPI.jl uses, so that libmpi and oneAPI.jl share a single loader in the test process. yaksa compiles its Level Zero kernels to (device independent) SPIR-V with `ocloc -device skl`, but Skylake was dropped from recent releases of the Intel compute runtime, so retarget the kernels at a device ocloc still knows. Integrated Intel GPUs generally have no native Float64 support and `oneArray` refuses to hold element types the device cannot handle, so drop Float64 and ComplexF64 from the tested types there. Also report oneAPI in `MPI.has_gpu()` and add `MPI.has_oneapi` to the manual, which Documenter requires as `checkdocs` defaults to `:all`.
The GitHub checks are named after the step label, so reusing the CUDA and ROCm one made the oneAPI results indistinguishable on the pull request.
|
@lcw I fiddled with the buildkite runner. The oneAPI.jl tests seem to pass now. Can you take a look? Would this be sufficient to get this merged? |
|
@michel2323 Can you please point me where to look to see them passing? I still see failing tests for |
MPICH 4.3.2 and 5.0.1 both intermittently deliver small device-buffer messages with the head intact and the tail stale or scrambled on the Iris Xe CI machine. Disabling the fast-memcpy path and switching to immediate command lists did not help; the alltoall tests trip over it near-deterministically and any other test can be struck occasionally. Exclude the alltoall tests and soft-fail the step until the upstream bug is fixed.
|
@lcw Sorry about that. The single tests showed as passed intermittently before then flipping to fail. I compile MPICH 5.x on the oneAPI Buildkite runner, and this revealed a bug for that particular driver (i915). More recent Intel GPUs use the Xe driver, where I cannot reproduce this issue. Most tests pass, however, the issue is non-deterministic and can affect any test. So I marked the oneAPI.jl tests as On Aurora, the tests pass, and it would be great to have this in a released MPI.jl. I could set up a runner that uses the ALCF CI at Argonne through GitLab, as I did for oneAPI.jl. What do you recommend? |
|
Thanks for all your effort! I'll merge this and make a release. |
|
@lcw Oh, no worries. As Claude would say, "I liked your pushback" with the runners, etc. Just to be clear, the tests are not hot. They fail silently until the issue with MPICH and the i915 on the Buildkite runner is fixed. Let me know if you want anything changed. |
2 participants
I can only test this on Aurora at ANL. I could provide an MPICH_jll with support. How should I proceed?