Using TrueNAS as an iSCSI storage backend for libvirt

Technical howto article

I set up a hypervisor on a Lenovo ThinkCentre I bought on eBay. I use libvirt on Ubuntu 22.04 to control my virtual machines. I wanted to use my TrueNAS box as a storage backend for persistent data volumes.

I ran this for a couple of months, but I quickly encountered issues. The iSCSI connection would drop randomly, and the read/write speed wasn’t good. After some research, I gave up on using iSCSI because my network is just not fast and reliable enough. Now I use local storage for my VMs, and replicate the data to the TrueNAS box using rsync or ZFS replication tasks.

In case you are still interested in using iSCSI, I’ll describe the setup details. But first, here’s a description of the problems I encountered.

Random iSCSI disconnect issues

On my home network with a 1 Gbps Ethernet dumb switch, the iSCSI connection would randomly drop about once or twice per week.

Logs on TrueNAS:

Apr 18 15:49:15 freenas.alexware.deverteuil.net WARNING: 192.168.13.184 (iqn.1993-08.org.debian:01:e0741deca62): no ping reply (NOP-Out) after 5 seconds; dropping connection

Logs on the Ubuntu host (iscsid.service):

2022-04-18 15:49:21 Kernel reported iSCSI connection 9:0 error (1022 - Invalid or unknown error code) state (3)
2022-04-18 15:49:31 connection9:0 is operational after recovery (1 attempts)

Logs in the Ubuntu VM (kernel log):

2022-04-18 15:49:26 WARNING: Pool 'zpool-01' has encountered an uncorrectable I/O failure and has been suspended.

The data volume is ZFS formatted. Whenever the iSCSI connection dropped, the ZPOOL went in a degraded state and all write operations were blocked.

The workaround is to SSH into the VM and issue a zpool clear <pool> command to reset the ZPOOL to a normal state.

Research:

Documentation

Here is the relevant documentation for libvirt and TrueNAS if you wish to set up an iSCSI connection.

Set up iSCSI block shares on TrueNAS

  1. Create a dataset to organize ZVOLs. In my ZPOOL, I create a virtual-machines-external dataset where I create ZVOLs for VMs hosted on my dedicated virtualisation node.

  2. Make sure the iSCSI service is started.

  3. Go to Sharing > Block Shares (iSCSI). I leave the default parameters intact, it works for me.

  4. Create a Portal. A portal is an iSCSI service listening on an interface. Use 0.0.0.0:3260 to listen on all IP addresses.

  5. Create an Initiator Group. Initiator Groups will be used when defining an Authorized Access. I define my Initiator Group by local network subnet.

  6. Create an Authorized Access. This is where you set a user name and password, if you want to. For simplicity, I didn’t configure authentication yet.

  7. Create a Target. A target is a group of iSCSI resources. libvirt will connect to this Target to discover the logical units it can attach. A target has a name, a Portal ID, and an Initiator Group ID.

  8. Add Extents. Extents have a name within the iSCSI service, and they point to a ZVOL.

  9. Create Associated Targets. This is where Extents are added as LUNs to the iSCSI target.

Create an iSCSI storage pool in libvirt

I’m going to use virt-manager to add the TrueNAS iSCSI Target as storage pool.

  1. In virt-manager, open the Connection Details and go to the Storage tab.
  2. Click on the + button to add a pool.
  3. Give it a name.
  4. Select the “iscsi: iSCSI Target” type.
  5. Leave the Target Path as /dev/disk/by-path.
  6. Type in the host name of your TrueNAS box.
  7. Use the following format for the Source IQN:
    <iscsi share base name>:<target name>.
    For example: iqn.2005-10.org.freenas.ctl:fixion-storage.
  8. Click Finish.
  9. You should see the available LUNs. If not, you can refresh the pool to re-discover the LUNs.
  10. You can then attach those LUNs to your VMs.
Alexandre de Verteuil
Alexandre de Verteuil
Solutions Architect

I teach people how to see the matrix metrics.
Monkeys and sunsets make me happy.

Related