Skip to content

tuh8888/libvirt_win10_vm

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

24 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Virtual Machine with vGPU Unlock for single GPU desktop

This document catalogs my working setup for configuring an Arch Linux host OS to run a Windows 10 VM while passing through a virtualized GPU.

Sources

The resources I used during my own install. This document is a compilation of the instructions and files provided by these people.

Guides

Sources

Configuration

The what and how. Mostly described in suggested installation order unless otherwise stated.

Host

Operating system

Configuration of the host OS for virtualization using KVM and device passthrough using OVMF, IOMMU, and VFIO.

Arch Linux
Kernel 5.12.15-arch1-1
Kernel modules
  • vfio
  • vfio_iommu_type1
  • vfio_pci
  • vfio_virqfd
How to add
  1. Edit /etc/mkinitcpio.conf
  2. Add MODULES=(... vfio vfio_iommu_type1 vfio_pci vfio_virqfd ...) to the MODULES line.
  3. Add HOOKS=(... modconf ...) to the HOOKS line.
  4. Update update-initramfs -P (use mkinitpico -P for manjaro)
Boot parameters
  • amd_iommu=on
  • iommu=pt
  • vfio_iommu_type1.allow_unsafe_interrupts=1
  • kvm.ignore_msrs=1
  • blacklist=nouveau
How to add
  1. edit /etc/default/grub
  2. Add to GRUB_CMDLINE_LINUX_DEFAULT:
GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 blacklist=nouveau ..."
  1. Update GRUB update-grub

Hardware

The physical devices present on my system.

  • CPU: AMD Ryzen 7 5800X (16) @ 3.800GHz
  • GPU: NVIDIA GeForce GTX 1060 6GB
  • Memory: 16MiB
  • 1440p Monitor

Software

What needs to be installed.

Arch Packages
Packages installed using pacman.
linux

The OS kernel.

  • Version: 5.12.15
How to install: First option
  1. Install downgrade from AUR (using yay or other AUR helper)
  2. downgrade linux
  3. Select 5.12.15
How to install: From Arch archive
  1. Download package curl -O https://archive.archlinux.org/packages/l/linux/linux-5.12.15.arch1-1-x86_64.pkg.tar.zst
  2. Install with pacman -U linux-5.12.15.arch1-1-x86_64.pkg.tar.zst
linux-headers

Needed to build kernel modules, particularly the Nvidia driver.

  • Version: 5.12.15
How to install: First option
  1. Install downgrade from AUR (using yay or other AUR helper)
  2. downgrade linux-headers
  3. Select 5.12.15
From Arch archive
  1. Download package curl -O https://archive.archlinux.org/packages/l/linux-headers/linux-headers-5.12.15.arch1-1-x86_64.pkg.tar.zst
  2. Install with pacman -U linux-headers-5.12.15.arch1-1-x86_64.pkg.tar.zst
virt-manager

Used to create and run virtual machines.

libvirt
kvm

(Kernel-based Virtual Machine). A hypervisor. Used for virtualization.

qemu
looking-glass

Used to view VM in graphical display.

iptables-nft

For networking with the VM.

edk2-ovmf
dnsmasq

Used for VM networking.

mdevctl

Used to create vGPU devices

scream

Used to read audio stream from network.

How to setup
  1. Install using package manager.
  2. Create systemd-file
    [Unit]
    Description=Scream network audio client
    
    [Unit]
    Description=Scream Receiver
    After=pulseaudio.service network-online.target
    Wants=pulseaudio.service
    
    [Service]
    Type=simple
    ExecStartPre=/bin/sleep 3
    ExecStart=scream -i virbr0
    Restart=always
    RestartSec=1
    
    [Install]
    WantedBy=default.target
          
Python packages
Installed with pip.
frida
How to installIf using conda
  1. Install python-pip using pacman.
  2. Run /bin/pip install frida
Otherwise
  1. pip install frida
Graphics setup
Nvidia Driver
Version: 460.73.01 merged and pre-unlocked This driver is a merged version of Nvidia’s vGPU driver and their standard driver. It allows for the host graphics to be displayed while the vGPU’s are active.Components
  1. merged driver
  2. vgpu_unlock
  3. vgpu_unlock_5.12 patch
How to installMake sure all above arch packages and python packages are installed.Download thecomponents using curl -O the_linkMake the driver executable. chmod +x NVIDIA-Linux-x86_64-460.73.01-grid-vgpu-kvm-v5.runExtract it. ./NVIDIA-Linux-x86_64-460.73.01-grid-vgpu-kvm-v5.run -xcd NVIDIA-Linux-x86_64-640.73.01-grid-vgpu-kvm-v5Apply the 5.12 patch patch -p0 < ../twelve.patchDisable graphics sudo systemctl isolate multi-user.targetRe-login as root and navigate back to the driver directory
cd /home/user/Downloads/NVIDIA-Linux-x86_64-460.73.01-grid-vgpu-kvm-v5
  
Install the driver
./nvidia-installer --dkms
  
Add vgpu_unlock to service files.
nano /lib/systemd/system/nvidia-vgpud.service
  

Replace ExecStart with:

ExecStart=/root/vgpu_unlock/vgpu_unlock /usr/bin/nvidia-vgpud
  
nano /lib/systemd/system/nvidia-vgpu-mgr.service
  

Replace ExecStart with:

ExecStart=/root/vgpu_unlock/vgpu_unlock /usr/bin/nvidia-vgpu-mgr
  
systemctl daemon-reload
  
Add vgpu_unlock to source files.
nano /usr/src/nvidia-460.73.01/nvidia/os-interface.c
  

Under #include “nv-time.h” insert this line

#include "/root/vgpu_unlock/vgpu_unlock_hooks.c"
  
nano /usr/src/nvidia-460.73.01/nvidia/nvidia.Kbuild
  

Add this to the bottom of the file.

ldflags-y += -T /root/vgpu_unlock/kern.ld
  
Remove and reinstall the Nvidia dkms module
dkms remove -m nvidia -v 460.73.01 --all
dkms install -m nvidia -v 460.73.01
  
Make all of vgpu_unlock executable

chmod -R +x vgpu_unlock

Reboot
vGPU Profiles
  • nvidia-48 GRID P40-3Q
How to installChoose profile
mdevctl types
  

I chose nvidia-47 (GRID P40-3Q) because it has 3Q at the end of its name. This means I can have two 3GB vGPU’s running.

Determine GPU PCI address
lspci | grep VGA
PCI_ADDRESS=$(lspci | grep VGA | grep -Po '\d\w:\d\d.\d')
echo $PCI_ADDRESS
  
Generate UUIDs for each vGPU
UUID1=$(uuidgen)
UUID2=$(uuidgen)
echo $UUID1 $UUID2
  
Create vGPUs
mdevctl start -u $UUID1 -p 0000:$PCI_ADDRESS -t nvidia-48
mdevctl start -u $UUID2 -p 0000:$PCI_ADDRESS -t nvidia-48

mdevctl define -a -u $UUID1
mdevctl define -a -u $UUID2
  

VM Image

The virtual machine configuration.

  • Chipset: Q35
  • Firmware: UEFI x86_64: /usr/share/edk2-ovmf/x64/OVMF_CODE.fd
  • Hypervisor KVM
  • Spice Server
  • Video QXL
  • MDEV
  • shmem looking-glass
How to install
In virt-manager, create virtual machine.
Apply above settings minus “MDEV” and “shmem looking-glass”.
Start VM and install OS.
Shutdown guest.
Add looking-glass shmem device.

The size is determined like this.

<shmem name='looking-glass'>
  <model type='ivshmem-plain'/>
  <size unit='M'>64</size>
</shmem>

Make a file /etc/tmpfiles.d/10-looking-glass.conf

#Type Path               Mode UID  GID Age Argument

f /dev/shm/looking-glass 0660 user kvm -
Remove tablet device
Add MDEV device.

Where $UUID1 is the UUID of one of the mdev vGPU’s created earlier.

<hostdev mode="subsystem" type="mdev" managed="no" model="vfio-pci" display="on">
  <source>
    <address uuid="$UUID1"/>
  </source>
</hostdev>
Install all guest software.
Shutdown guest.
Either set Video model to “none” or install guest Nvidia driver​.
How to run
  1. Start VM. (May need to click run several times if error related to “vfio-<bus>” occurs.
  2. Run looking-glass client
    export SDL_VIDEO_X11_VISUALID=
    looking-glass-client
        

Guest

Inside the VM.

Operating system

  • Windows 10

To download (within guest)

Software

Looking Glass Host
How to install
  1. Download components.
  2. Update driver in Device Manager->System Devicess->PCI standard RAM Controller to virtio PCI RAM controller driver.
  3. Install virtio guest tools and spice guest tools.
  4. Install looking glass host by running looking-glass-host-setup.exe as administrator.
  5. VM.
How to install
  1. Run Nvidia GRID driver installer.
  2. Reboot.
  3. In Nvidia Console set resolution. My monitor used 2560x1440.
  4. Run vGPU_LicenseBypass.
How to install
  1. Install driver.
  2. Change sound source to Scream (WDM).

My files

Suggestions

Since there is a lot of messing with the kernel and graphics drivers, it’s important to have a plan for recovering your host OS.

  • Use timeshift or other auto-snapshotting of the host.
  • Have a recovery drive available.

Maintenance

Preventing accidental upgrades

In order to avoid accidentally upgrading these carefully chosen package versions, make sure to ignore updates to them:

IgnorePkg = linux linux-headers nvidia-dkms nvidia nvidia-utils nvidia-settings opencl-nvidia libxnvctrl

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Contributors 4

  •  
  •  
  •  
  •  

Languages