Using TrueNAS as an iSCSI storage backend for libvirt
Technical howto article
I set up a hypervisor on a Lenovo ThinkCentre I bought on eBay. I use libvirt on Ubuntu 22.04 to control my virtual machines. I wanted to use my TrueNAS box as a storage backend for persistent data volumes.
In case you are still interested in using iSCSI, I’ll describe the setup details. But first, here’s a description of the problems I encountered.
Random iSCSI disconnect issues
On my home network with a 1 Gbps Ethernet dumb switch, the iSCSI connection would randomly drop about once or twice per week.
Logs on TrueNAS:
Apr 18 15:49:15 freenas.alexware.deverteuil.net WARNING: 192.168.13.184 (iqn.1993-08.org.debian:01:e0741deca62): no ping reply (NOP-Out) after 5 seconds; dropping connection
Logs on the Ubuntu host (
2022-04-18 15:49:21 Kernel reported iSCSI connection 9:0 error (1022 - Invalid or unknown error code) state (3)
2022-04-18 15:49:31 connection9:0 is operational after recovery (1 attempts)
Logs in the Ubuntu VM (kernel log):
2022-04-18 15:49:26 WARNING: Pool 'zpool-01' has encountered an uncorrectable I/O failure and has been suspended.
The data volume is ZFS formatted. Whenever the iSCSI connection dropped, the ZPOOL went in a degraded state and all write operations were blocked.
The workaround is to SSH into the VM and issue a
zpool clear <pool> command
to reset the ZPOOL to a normal state.
- iSCSI is super finicky and not a great protocol unless network and storage are provisioned appropriately.
- Some consideration for vdevs performance
Here is the relevant documentation for libvirt and TrueNAS if you wish to set up an iSCSI connection.
Set up iSCSI block shares on TrueNAS
Create a dataset to organize ZVOLs. In my ZPOOL, I create a
virtual-machines-externaldataset where I create ZVOLs for VMs hosted on my dedicated virtualisation node.
Make sure the iSCSI service is started.
Go to Sharing > Block Shares (iSCSI). I leave the default parameters intact, it works for me.
Create a Portal. A portal is an iSCSI service listening on an interface. Use
0.0.0.0:3260to listen on all IP addresses.
Create an Initiator Group. Initiator Groups will be used when defining an Authorized Access. I define my Initiator Group by local network subnet.
Create an Authorized Access. This is where you set a user name and password, if you want to. For simplicity, I didn’t configure authentication yet.
Create a Target. A target is a group of iSCSI resources. libvirt will connect to this Target to discover the logical units it can attach. A target has a name, a Portal ID, and an Initiator Group ID.
Add Extents. Extents have a name within the iSCSI service, and they point to a ZVOL.
Create Associated Targets. This is where Extents are added as LUNs to the iSCSI target.
Create an iSCSI storage pool in libvirt
I’m going to use virt-manager to add the TrueNAS iSCSI Target as storage pool.
- In virt-manager, open the Connection Details and go to the Storage tab.
- Click on the
+button to add a pool.
- Give it a name.
- Select the “iscsi: iSCSI Target” type.
- Leave the Target Path as
- Type in the host name of your TrueNAS box.
- Use the following format for the Source IQN:
<iscsi share base name>:<target name>.
- Click Finish.
- You should see the available LUNs. If not, you can refresh the pool to re-discover the LUNs.
- You can then attach those LUNs to your VMs.