Using TrueNAS as an iSCSI storage backend for libvirt

I set up a hypervisor on a Lenovo ThinkCentre I bought on eBay. I use libvirt on Ubuntu 22.04 to control my virtual machines. I wanted to use my TrueNAS box as a storage backend for persistent data volumes.

In case you are still interested in using iSCSI, I’ll describe the setup details. But first, here’s a description of the problems I encountered.

Random iSCSI disconnect issues

On my home network with a 1 Gbps Ethernet dumb switch, the iSCSI connection would randomly drop about once or twice per week.

Logs on TrueNAS:

Apr 18 15:49:15 freenas.alexware.deverteuil.net WARNING: 192.168.13.184 (iqn.1993-08.org.debian:01:e0741deca62): no ping reply (NOP-Out) after 5 seconds; dropping connection

Logs on the Ubuntu host (iscsid.service):

2022-04-18 15:49:21 Kernel reported iSCSI connection 9:0 error (1022 - Invalid or unknown error code) state (3)
2022-04-18 15:49:31 connection9:0 is operational after recovery (1 attempts)

Logs in the Ubuntu VM (kernel log):

2022-04-18 15:49:26 WARNING: Pool 'zpool-01' has encountered an uncorrectable I/O failure and has been suspended.

The data volume is ZFS formatted. Whenever the iSCSI connection dropped, the ZPOOL went in a degraded state and all write operations were blocked.

The workaround is to SSH into the VM and issue a zpool clear <pool> command to reset the ZPOOL to a normal state.

Research:

• iSCSI is super finicky and not a great protocol unless network and storage are provisioned appropriately.
  https://www.truenas.com/community/threads/ctld-continually-exits-read-connection-lost-bug.80933/post-561248
• Some consideration for vdevs performance
  https://www.truenas.com/community/threads/some-differences-between-raidz-and-mirrors-and-why-we-use-mirrors-for-block-storage.44068/

Documentation

Here is the relevant documentation for libvirt and TrueNAS if you wish to set up an iSCSI connection.

• TrueNAS iSCSI share
• libvirt iSCSI storage backend

Set up iSCSI block shares on TrueNAS

1. Create a dataset to organize ZVOLs. In my ZPOOL, I create a virtual-machines-external dataset where I create ZVOLs for VMs hosted on my dedicated virtualisation node.

   

2. Make sure the iSCSI service is started.

   

3. Go to Sharing > Block Shares (iSCSI). I leave the default parameters intact, it works for me.

   

4. Create a Portal. A portal is an iSCSI service listening on an interface. Use 0.0.0.0:3260 to listen on all IP addresses.

   

5. Create an Initiator Group. Initiator Groups will be used when defining an Authorized Access. I define my Initiator Group by local network subnet.

   

6. Create an Authorized Access. This is where you set a user name and password, if you want to. For simplicity, I didn’t configure authentication yet.

   

7. Create a Target. A target is a group of iSCSI resources. libvirt will connect to this Target to discover the logical units it can attach. A target has a name, a Portal ID, and an Initiator Group ID.

   

8. Add Extents. Extents have a name within the iSCSI service, and they point to a ZVOL.

   

9. Create Associated Targets. This is where Extents are added as LUNs to the iSCSI target.

   

Create an iSCSI storage pool in libvirt

I’m going to use virt-manager to add the TrueNAS iSCSI Target as storage pool.

1. In virt-manager, open the Connection Details and go to the Storage tab.
2. Click on the + button to add a pool.

   

3. Give it a name.
4. Select the “iscsi: iSCSI Target” type.
5. Leave the Target Path as /dev/disk/by-path.
6. Type in the host name of your TrueNAS box.
7. Use the following format for the Source IQN:
   <iscsi share base name>:<target name>.
   For example: iqn.2005-10.org.freenas.ctl:fixion-storage.
8. Click Finish.
9. You should see the available LUNs. If not, you can refresh the pool to re-discover the LUNs.

   

10. You can then attach those LUNs to your VMs.