When it comes to my home lab, I’m a big believer in running my virtualization platforms, routers, storage servers, and other essential systems separately on different machines. So instead of consolidating my NAS server and my productivity-driven virtual guests on the same Proxmox Guest, I have relied on a basic platform for my TrueNAS workloads for a long time. After all, considering the wacky experiments I participate in, I’d say it’s pretty justified to keep my PVE nodes away from my central backup server.
That being said, I recently worked on a project where I deployed TrueNAS inside a Proxmox virtual machine and it worked surprisingly well. Of course, if my Proxmox node failed, my TrueNAS virtual machine would also fail. But as long as I don’t go too far with my experiments, I have to admit that iXsystems’ recommended method for a virtualized TrueNAS instance is reliable enough for a hybrid TrueNAS workstation powered by Proxmox.
Deploying a TrueNAS virtual machine like a conventional Linux distribution is a recipe for disaster
Even if it works at first, this NAS setup would be quite risky
Let’s say I want to spin up any Linux-flavored virtual machine that isn’t a ZFS-powered storage platform like TrueNAS. Well, the procedure is quite simple: you would give the new VM a name and assign it the correct CPU, memory and storage resources through the virtual machine creation wizard in Proxmox. If I wanted to assign more storage disks to it, I would simply assign them to the VM as virtual disks from the Hardware tab and call it a day. Or, in case I wanted more space, I would go to the Proxmox shell and run some commands to grant the entire disk to the VM.
While this procedure would work even for development virtual machines, it is far from ideal for a TrueNAS virtual machine. You see, TrueNAS uses the super powerful ZFS as the underlying file system, which requires direct control over the physical disks. Assigning them to a TrueNAS virtual machine would add additional virtualization layers to the mix, which could break write caching. And if the order of write operations gets mixed up during a failure, the data integrity of my storage pools is likely to be compromised.
Heck, these virtualization layers are still present even if I were to push raw disks to my TrueNAS VM via shell commands, and my virtualized backup system wouldn’t be able to track SMART statistics, verify disk information, or flush the disk cache properly. Sure, things might work fine at first, but without hardware-level disk access, you could end up with broken storage pools on a TrueNAS virtual machine. Fortunately, there is still a way to get a virtualized TrueNAS instance up and running on Proxmox…
The PCI(e) step is the trick that makes this project possible
It’s pretty simple once you know what you’re going for.
Since exposing only the drives to TrueNAS involves additional hypervisor layers, the safest way to get a Proxmox-powered NAS server involves moving the entire SATA controller to the VM. That way my TrueNAS VM (and by extension, ZFS) would have full access to my hard drives and wouldn’t have to deal with the quirks of HDD-based virtual storage pools. The caveat, of course, is that it may seem a little complicated for beginners. But if you’re somewhat versed in PCI passthrough on Proxmox, passing the SATA controller shouldn’t take much time.
For reference, I performed this experiment on a TerraMaster F4-425 Prowhich already has a good CPU and enough RAM for a Proxmox-based hybrid TrueNAS setup. I have created Dedicated guides on PCI passthroughbut here is a short version for those who want to follow along. First, I edited the GRUB_CMDLINE_LINUX_DEFAULT=”silent” line inside the /etc/default/grub configuration file adding intel_iommu=enabled just after the don’t worry flag and ran update-grub to enable IOMMU provisions. from the /etc/modules The file also needed a couple of additional parameters, I added the following code to it:
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
With that, editing the /etc/modprobe.d/vfio.conf All that was left was a file to include the device IDs of my SATA controllers. So, I ran the lspci -nn command to check the device IDs and entered them into /etc/modprobe.d/vfio.conf in it options vfio-pci ids = Device_1, Device_2 format. Now that the PCI step for my SATA controllers was complete, I quickly headed to the Hardware tab of TrueNAS VM and added them as PCI devices.
TrueNAS VM also requires a lot of RAM
I blame that on memory-hungry ZFS clusters.
My TrueNAS virtual machine still needed some memory tweaking to get it ready for battle. ZFS is famous for its tendency to hog memory, so I allocated 8GB of my NAS’s 16GB of RAM to the VM. I have a couple more virtual guests running on this system, so I will probably have to upgrade the amount of RAM in the future to avoid over-provisioning my memory resources.
All that said, I’ve had this virtualized NAS setup up and running for over a month now and it hasn’t let me down. I’ll still have to curb my tinkering impulses, as I don’t want to mess up the TrueNAS setup with the Proxmox host. But I have to say that running a Proxmox + TrueNAS hybrid setup on the same machine is pretty impressive, even if the general home lab community frowns upon it a bit.
