GPU not released on VM stop #30
Comments
itzsimpl
changed the title
|
I think you want to post this at https://github.com/canonical/lxd instead. |
|
I'm happy to still keep this one open as Incus is very likely to have this exact same issue given where we're at with the fork. |
|
Just to let you know, I've opened the issue also on on Canonical/lxd, and there is a little bit more info (additional tests that I made on different GPUs and with vGPU drivers), see canonical/lxd#12128. |
stgraber
changed the title
|
Going to poke at that one tomorrow. Sadly the only system I have with multiple NVIDIA GPUs is a box where I have no intention to ever install the binary NVIDIA driver :) But I do have our other test system which has a single NVIDIA GPU and where I don't mind installing the NVIDIA drivers on the host, so I'm hoping I can reproduce what you're seeing on that one. |
|
I'm unable to reproduce the described issue with current Incus: @itzsimpl can you confirm that this is still an issue on current Incus? |
|
@stgraber thank you for starting to look into this. Unfortunately, I do not have a system with the same setup available at the moment. Based on the experiments when we first saw the issue it may be limited to "older" and "non-datacenter" GPUs, as these load/unload more devices (eg. Quadro RTX 6000 in our case, see canonical/lxd#12128 (comment)). FWW. We also noticed issues with unloading of vGPU drivers. The only workaround that we managed to setup was to remove the devices and rescan the PCI once the VM shuts down, but that does not work with vGPU drivers (see canonical/lxd#12128 (comment)). |
|
Okay, so we're going to need to get access to a system with such a GPU to be able to reproduce the issue and look for a fix. Having multiple devices in the group definitely sounds like it may be the problem but we don't have anything in our lab that behaves that way. Similarly for vGPU, we only have the A100 for that and it uses mdev which doesn't have any such issues. |
|
FWW. vis vGPU, we had mdev as well, the Quadro RTX 6000 is on the list of supported GPUs (https://docs.nvidia.com/grid/gpus-supported-by-vgpu.html), but the result was that on VM shutdown some vGPUs did not get released properly, so VM shutdown and startup eventually drained the GPU memory, only a reboot helped (canonical/lxd#12128 (comment)). The drivers were 535.54.03, this is all I can remember or have on file from then, sorry. |
Following lxc/lxc#4332 (comment) I'm opening the issue here.
Required information
Click to see full
lxc-start --versionlxc-checkconfiguname -acat /proc/self/cgroupcat /proc/1/mountsIssue description
I have an Ubuntu:22.04 system with multiple GPUs, NVIDIA drivers 535 server installed, persistence mode is off. I pass individual GPUs to a VM as PCI passthrough. When I pass a single GPU to the VM, start it and then stop, the GPU is not returned to the host system (i.e.
nvidia-smidoes not show it anymore). When I pass multiple GPUs to the VM, start it and then stop, the GPU with the lowest PCI address on the host is not returned to the host system (i.e.nvidia-smidoes not show it anymore), but the other GPUs get returned just fine.Restarting the VMs again the GPUs are visible inside the VM, but if I start a container with nvidia-driver passthrough, only the GPUs that are currently visible on the host (i.e. all installed minus those that were not returned from the VMs earlier) are visible in the container. The only info I can find is that syslog says "Failed to stop device".
Steps to reproduce
Information to attach
Click to see full
lxc info --show-log vm2)syslog), the single GPU casesyslog), two GPU caseThe text was updated successfully, but these errors were encountered: