docs: KubeVirt with DRA

[mikemckiernan]

Review HTML: https://nvidia.github.io/cloud-native-docs/review/pr-473/gpu-operator/latest/gpu-operator-kubevirt-dra.html

[github-actions]

Documentation preview

https://nvidia.github.io/cloud-native-docs/review/pr-473

[varunrsekar]

This does not exist.
You may use this example but : https://github.com/NVIDIA/go-nvfm/blob/main/examples/list-partitions/main.go
The steps are:

git clone https://github.com/NVIDIA/go-nvfm && cd go-nvfm
make example-list-partitions
kubectl cp ./list-partitions -n gpu-operator <driver-pod>:/run/nvidia/list-partitions
kubectl exec -n gpu-operator <driver-pod> -- bash -c "NVFM_UNIX_SOCKET_PATH=/run/nvidia-fabricmanager/socket /run/nvidia/list-partitions"

But I don't think we should include this validation step. cc: @shivamerla

[shivamerla]

@varunrsekar this could be useful for savvy admins who want to check that the partitions are activated. Anyway the repo is public, they will report if any issues with it. We cannot use this directly? https://github.com/NVIDIA/Fabric-Manager-Client

[mikemckiernan]

Very grateful to you both for the review and letting me know the client isn't automatically installed. (Why?)

I revised to group the two verification/informational commands under an optional step that mentions "requires additional software." It's honest, hopefully prevents surprise, and still helps a determined admin gain more knowledge. Well, imo.

If it's still problematic, plmk and thanks again!

[varunrsekar]

We cannot use this directly? https://github.com/NVIDIA/Fabric-Manager-Client

No there's a couple of issues with it:

1. A bug that prevents it from working with unix socket
2. Building the binary require installing jsoncpp and the FM SDK

The go-nvfm examples have no dependeicies so would be easier to build.

this could be useful for savvy admins who want to check that the partitions are activated

And yeah its good if they know what they're doing. We're injecting a binary into the driver container and running something inside it so I was skeptical. Maybe we should come up with instructions using a debug container?

[varunrsekar]

@mikemckiernan FYI The verification steps require the client to be run inside the driver container to work.

[bmozaffa]

This looks like a command to copy and paste... but I think the comment would break it?

[mikemckiernan]

I believe it does, but I was willing to include it because:

A) I didn't want two copy-paste commands that are identical except for one line--confusing, imo.
B) If I add "For NVLink5 or later...add the "FabricManagerPartitioning":true feature gate" above or below the code block, I fear it'll get missed. Conversely, if I put a "If you host isn't, remove..." above or below, also might not get read and I'm unsure if that was tested or could cause a problem.

My thinking is that nearly cut-n-paste that is clear and has the added bonus of failing-command-attention-getting behavior was the right choice.

If you have a suggestion, please share.

[bmozaffa]

I think we need another warning here, perhaps with input from engineering, about our race condition between GPU and VFIO allocation

[mikemckiernan]

@shivamerla, @varunrsekar, plmk if you agree with Babak's concern. I believe Babak is referring to the second bullet from the KubeVirt VFIO limitations and considerations in the DRA driver docs.

[shivamerla]

Yes Mike, better to highlight that limitation here as well as mixed workloads are allowed now by default with DRA.