vGPU (SR-IOV) with Intel 12th Gen iGPU [Updated Nov 2023]

[Update 2023-11-01] The following guide has been updated to reflect some detailed steps and up-to-date info with Proxmox kernel 6.2

Contents
[Part I: Foreword]
[Part II: Prerequisite]
[Part III: Proxmox Host Installation]
[Part IV: Linux VM Usage]
[Part V: Windows VM Usage]
[Part VI: LXC Usage - Jellyfin HWA (Breaking Change)]
[Part VII: Conclusion]

Part I: Foreword:

I’ve always dreamed about using my hardware as efficiently as possible. In today’s PC master race, it almost felt like we are only using less than 20% of what our PCs are really capable of. For example, Intel ships most of its CPUs with a built-in iGPU. Although not a 3D Mark champ, it still packs a punch when it comes to media decoding and encoding. Previously mentioned in my NAS build, I went with an Intel 12th gen i3-12100. It has an Intel UHD 730 iGPU which does wonders for media encoding and decoding, especially the support for AV1 decode, which is absent on all other GPUs on the market, other than the Intel ARC and NVIDIA RTX 40 series.

So naturally, I started looking for hacks that can enable me to virtualize this iGPU for various VM uses, such as Parsec on Windows guest VMs, and hardware accelerated media transcoding/decoding with Plex or Jellyfin.

This guide will walk you through all the necessary steps to enable Intel iGPU for virtualization by modifying the kernel and driver using DKMS, and create virtual functions to pass these virtual GPUs to VMs.

Special thanks to:

Strongz for this DKMS module that made it all possible (https://github.com/strongtz/i915-sriov-dkms)

Tteck for Proxmox Helper Scripts (https://tteck.github.io/Proxmox/)

Foxi (https://foxi.buduanwang.vip/virtualization/pve/2695.html/)

WARNING:

This guide is highly experimental. If you are trying to use this guide on your production machine, DON'T DO IT. Test it with a spare system first, work out all the kinks before you move on to your production environment. From my experience, even if it seems to be running fine, there could be a lot of instabilities that we just don't know about yet.

Part II: Prerequisite:

Motherboard: Turn on SRIOV and VT-d (IOMMU) in BIOS.

Proxmox Virtual Environment 7+

6.1 or newer Kernel (6.2 has been released by Proxmox, nothing has changed) 5.19 Kernel

You can update the PVE kernel to 6.2 5.19 using these commands:

1. Disable enterprise repo.
   nano /etc/apt/sources.list

  Then use the following repos from Proxmox:

 Alternatively, just use Tteck's Proxmox Helper Script (PVE 7 ONLY):

1. Install 6.1 kernel (or 6.2).
     apt install pve-kernel-6.1
2. Update apt and install 6.1 headers.
     apt update && apt install pve-headers-$(uname -r)
3. Update initramfs and reboot.
     update-initramfs -u
     reboot

Part III.1: Proxmox Host Installation

After rebooting your system, do a kernel name check to see if 6.1 kernel has applied.

uname -r

If you can see this output, it means your kernel has been updated to 6.1 (or 6.2, it works).

6.1.10-1-pve

Now, we can proceed to installing the DKMS module.

1. Install DKMS, Unzip and build tools.
   apt install git dkms build-* -y
2. Git clone this repo, then edit the dkms.conf file. Replace "PACKAGE_NAME" with "i915-sriov-dkms" and "PACKAGE_VERSION" with "6.2" (so it can be correctly differentiated with older versions, totally cosmetic):

git clone https://github.com/strongtz/i915-sriov-dkms
cd i915-sriov-dkms

nano dkms.conf

[REPLACE THESE TWO LINES]
PACKAGE_NAME="@_PKGBASE@" ## "i915-sriov-dkms"
PACKAGE_VERSION="@PKGVER@" ## "6.2"

3.  Copy the folder contents to /usr/src, then install it:

mkdir /usr/src/i915-sriov-dkms-6.2

(assuming you're still in the package folder)
cp * /usr/src/i915-sriov-dkms-6.2

dkms install --force -m i915-sriov-dkms -v 6.2

When that’s done, use the following command to verify the installation:

dkms status

The correct output should look something like this:

i915-sriov-dkms, 6.1, 6.1.10-1-pve, x86_64: installed

Part III.2: Host Kernel Command Line Modification

Now the DKMS module is loaded, we will need to edit GRUB to enable the virtual functions.

1. Edit GRUB:
     nano /etc/default/grub
2. Look for “GRUB_CMDLINE_LINUX_DEFAULT” line, and add the following:
     intel_iommu=on i915.enable_guc=3 i915.max_vfs=7  
3. Your line could look like this: GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on i915.enable_guc=3 i915.max_vfs=7 iommu=pt quiet"
4. Update initramfs and grub:
     update-grub update-initramfs -u
  pve-efiboot-tool refresh
5. The third command is probably redundant but I am including it anyway.
6. Install sysfsutils and configure sysfs.conf:
     apt install sysfsutils -y
  echo "devices/pci0000:00/0000:00:02.0/sriov_numvfs = 7" > /etc/sysfs.conf
7. Please note, this is assuming your iGPU’s PCI address is the same as mine (0000:00:02) and most likely it would be. If not, you will have to change the command appropriately. If you are not sure which address your iGPU is assigned to, you can use the following command to verify:
     lspci | grep "VGA"
8. If you have a discrete GPU installed, you may see two or more outputs. Identify the Intel iGPU address, and use that for the command above.

Part III.3: Download iGPU Firmware (Optional)

1. Check if the firmware exists on your system:
   ls /lib/firmware/i915/tgl_guc_70.1.1.bin
2. If the output is empty, download the firmware from here:
   wget https://mirrors.apqa.cn/d/proxmox-edge/intel_gpu_sriov/i915-firmware.tar.gz
3. Then: tar -xf i915-firmware.tar.gz
4. Finally, copy to the firmware folder: cp ./firmware/* /lib/firmware/i915/
5. Note: this is the firmware for 12th/13th gen iGPUs (Alder Lake, Raptor Lake). If you are using 11th gen, you have to source your own firmware.
6. Reboot your system
   reboot

Part III.4: Verification

After a system reboot, check dmesg to see if your iGPU has virtual functions enabled, or use the following:

lspci  | grep VGA

You should see a similar output under PCIe address your iGPU is assigned to:

00:02.0 VGA compatible controller: Intel Corporation Device 4692 (rev 0c)
00:02.1 VGA compatible controller: Intel Corporation Device 4692 (rev 0c)
00:02.2 VGA compatible controller: Intel Corporation Device 4692 (rev 0c)
00:02.3 VGA compatible controller: Intel Corporation Device 4692 (rev 0c)
...

If you don't see any VFs, do dmesg | grep i915 to see what kind of error messages you're getting.

Part IV: Linux VM Usage

If you can see the VFs are enabled, congratulations! Now we can move on to setting up the hosts to use these virtual GPUs.

If you are using the VFs in Linux VMs, you are in luck, because you have already practices every necessary steps before in the Host setup. Simply put:

1. Use Q35+OVMF machine type for the Linux guest. Do not add the VF first, we will do that during step 7.
2. After the OS has been installed and is requesting a reboot, proceed with the reboot. However, during POST, enter VM's UEFI settings, and turn off secure boot.
3. Upgrade the Linux Guest to 6.2 Kernel along with the headers, which Intel has already mainlined many SRIOV features,
4. Download and compile that DKMS module we have used in the Proxmox Host.
5. When it comes to modifying the command line defaults, use i915.enable_nuc=3 instead of i915.enable_nuc=7.
6. Update GRUB and initramfs, then shutdown the VM.
7. Now you can add the VF, disable VM's emulated graphics, and start up the VM again to see if everything's working.

To verify the Guest VM has been configured correctly, do dmesg | grep i915, you should see an output that is similar to this:

Furthermore, we can verify by installing vainfo. Here's a sample output:

And - last but not least, clinfo.  Before you can use OpenCL, you have to add Intel's repository to apt, then install the required packages. Here's how:

1. Install wget and gpg-agent:
   apt install wget gpg-agent

2. Download the keys:
   wget -qO - https://repositories.intel.com/graphics/intel-graphics.key | sudo gpg --dearmor --output /usr/share/keyrings/intel-graphics.gpg

3. Add the Repo:
   echo 'deb [arch=amd64,i386 signed-by=/usr/share/keyrings/intel-graphics.gpg] https://repositories.intel.com/graphics/ubuntu jammy arc' | sudo tee /etc/apt/sources.list.d/intel.gpu.jammy.list

4. Install Run-time Packages:
   apt install -y intel-opencl-icd intel-level-zero-gpu level-zero intel-media-va-driver-non-free libmfx1 libmfxgen1 libvpl2 libegl-mesa0 libegl1-mesa libegl1-mesa-dev libgbm1 libgl1-mesa-dev libgl1-mesa-dri libglapi-mesa libgles2-mesa-dev libglx-mesa0 libigdgmm12 libxatracker2 mesa-va-drivers mesa-vdpau-drivers mesa-vulkan-drivers va-driver-all

5. Install Developer Packages:
   apt install -y libigc-dev intel-igc-cm libigdfcl-dev libigfxcmrt-dev level-zero-dev clinfo

6. To use the VF for render/computing, you will need to be in the render user-group.
   stat -c "%G" /dev/dri/render*
groups ${USER}

   If a group is listed for the render node which isn’t listed for the user, you will need to add the user to the group using gpasswd.
   sudo gpasswd -a ${USER} render
newgrp render

Finally, if all things went well, you should be able to use clinfo to see a similar output to this:

That's all there is to it. Have fun!

Part V: Windows VM Usage

You can now create a VM and use one of the virtual functions inside the VM. For example, this is my Windows 10 Pro VM configuration. It is necessary to use "Host" as the Processor Type:

Inside “PCI Device”, I assigned one of the virtual functions (virtualized GPU) to it:

It looks like this:

I won’t go into too much detail of how to create a Windows VM. For short:

1. Machine type must be q35 with OVMF BIOS (UEFI boot), CPU Type set to "Host".
2. Boot the VM with default Proxmox emulated graphics, then enable Remote Desktop.
3. Shut down the VM, remove the Proxmox graphics (set it to "none"), then add the virtual function iGPU.
4. Boot the VM, RDP to it, then install Intel official drivers. I am using gfx_win_101.4146 Intel Arc driver directly downloaded from the Intel website, without the dreaded “Code 43” error.

This is my another successful setup on an i5-13600K. Handbrake Nightly running a transcoding with QuickSync encoder, on Windows 11 Pro VM with iGPU SR-IOV Virtual Function. (Sorry Lost the Image)

Running Furmark. Like I said, this iGPU is not going to win any benchmark trophies.

Running Unigine Heaven. 720p, Low Preset. iGPU is pegged at 100%. Parsec encode has slowed down significantly, too. From less than 4ms per frame up to almost 10-12 ms per frame.

Let's throw Jellyfin Hardware Transcoding into the mix, with Furmark and Parsec running on the Tiny11 VM:

Part VI: LXC - Jellyfin HWA (Breaking Change)

The Tteck Jellyfin LXC would no longer work under this new configuration (Unless the newer Jellyfin-FFmpeg is used, see above). So we have to start with a vanilla LXC. In this example, I used Ubuntu 22.04 and did a distro upgrade to 23.04. Also, this is a priviledged container that has root access.

After the vanilla LXC is setup, time for some configuration.

1. Out of all the VFs that you have, pick one. Let's say we are going with No.4, which has the PCIe Address of 02.4 (02.0 being the Primary Function, which is the iGPU itself. Never pass this to any VM).
2. In Proxmox console, type ls -l /dev/dri/by-path to find out which devices are associated with your VF. In this example, I can see VF4 is associated with card4 and renderD132.

3.  Then, edit the Jellyfin LXC container config. nano /etc/pve/lxc/<JELLYFIN LXC ID>.conf

4.  Comment out these two lines: lxc.cgroup2.devices.allow%3A c 29%3A0 rwm  lxc.mount.entry%3A /dev/fb0 dev/fb0 none bind,optional,create=file

5.  Edit this line: lxc.cgroup2.devices.allow: c 226:4 rwm where the '4' in '226:4' is the VF number you have picked. If you picked '1', then it should read '226:1'.

6.  Edit this line: lxc.cgroup2.devices.allow: c 226:132 rwm where the '132' in '226:132' should be the renderD number we obtained earlier with ls -l /dev/dri/by-path

7.  Edit this line: lxc.mount.entry: /dev/dri/card4 dev/dri/card0 none bind,optional,create=file where the first argument /dev/dri/card4 is the VF you picked, the second argument maps it to card0 in the LXC container. I've tried leaving it the same as the host but after trial and error, only mapping it to card0 will work.

8. Edit this line: lxc.mount.entry: /dev/dri/renderD132 dev/dri/renderD128 none bind,optional,create=file pretty straight forward, we are mapping renderD132 from the host (Proxmox) to renderD128 in the LXC container.

SAMPLE CONFIG, FOR REFRENCE ONLY

arch: amd64
cores: 2
hostname: vaapi-test
memory: 512
net0: name=eth0,bridge=vmbr0,firewall=1,gw=[],hwaddr=[],type=veth
ostype: ubuntu
rootfs: local:123/vm-123-disk-0.raw,size=12G
swap: 512

lxc.cgroup2.devices.allow: c 226:4 rwm
lxc.cgroup2.devices.allow: c 226:132 rwm
lxc.mount.entry: /dev/dri/card4 dev/dri/card0 none bind,optional,create=file
lxc.mount.entry: /dev/dri/renderD132 dev/dri/renderD128 none bind,optional,create=file

## the last four lines are what matters the most, they map the VF to the LXC as main GPU

9.  Save the file, reboot the container, and install FFmpeg:

apt install ffmpeg

10. Then, install vainfo, clinfo, and intel-opencl-icd. In some cases, you would also need the intel-media-va-driver-non-free package

apt install vainfo clinfo intel-opencl-icd intel-media-va-driver-non-free

11. You can examine the output of vainfo and clinfo to see if the VF is being identified correctly:

12. If the outputs above are similar to what I have shown, chances are you are good to go. Last but not least, install Jellyfin via it's official source, and set the Jellyin FFmpeg path to the output of which ffmpeg, most likely it would be /usr/bin/ffmpeg. Also, set the harware acceleration to "VAAPI" and use /dev/dri/card0 as the VAAPI Device.

13. To test if the device can be used by FFmpeg, you can use this command to initialize the device:

ffmpeg -v verbose -init_hw_device vaapi=va -init_hw_device opencl@va

Additionally, this is the output I get if I run the command with Jellyfin-FFmpeg:

/usr/lib/jellyfin-ffmpeg/ffmpeg -init_hw_device vaapi=va

So, simply put, use the official FFmpeg instead of the one shipped with Jellyfin.

Part VII. Conclusion

With some tweaks and commands, we successfully used the iGPU's virtual functions in both a Windows VM and a Jellyfin server. I am really glad that I didn’t cheap out on the $15 difference by getting the i3-12100 for my NAS, instead of the i3-12100F. However, after discovering the UHD 730 on this CPU only has one media engine, I do have some buyer's remorse of not going up to i5-12500 for the UHD 770 iGPU. I mean, this iGPU takes up none of the PCIe slots, and provides a really nice functionality that we can only dream on with other higher end graphics cards. Granted, by using vgpu-unlock, we can virtualize an NVIDIA card (Ampere and up excluded) as well. However, for my application, the simplicity of Intel iGPU is hard to beat.

I am expecting an Intel ARC A380 in the next couple of days. I will try to use this DKMS module to see if I can get the A380 to enable virtual functions as well. Until then, good luck tinkering. Also, I am putting together a test bench to see how the iGPU performs under virtual functions versus just single PCIe passthrough. Stay tuned.

UPDATE 2023-03-06: The Intel Arc A380 is a NO GO. The PVE kernel is just not new enough to include the correct i915 driver for the A380. Even if it does, say, when Proxmox get Linux Kernel 6.2, Intel has confirmed that Alchemist WILL NOT SUPPORT SRIOV.