Documenting Problems That Were Difficult To Find The Answer To

Installing Xubuntu 14.04 Trusty On ZFS With LUKS Encryption


The aim was to get ZFS installed on a Linux system with 2 large spinning hard drives both encrypted with ZFS on top with ZFS filesystems mounted at / and /var with a swap volume. The /boot partition would be on its own USB drive.

Target ZFS setup

Target ZFS setup

Health Warning

This took me over 10 hours to get right. Most of the parts were available at other sites as linked throughout this article. But little things – like getting multiple LUKS disks mounted before ZFS – was very difficult to solve. When I say take snapshots take snapshots to save yourself a world of delays when something goes wrong.

Also note that ZFS appears, initially, to take excessively long times to write many small files. The time it takes to install a linux-headers package can be between five and ten minutes.



I downloaded the Xubuntu 14.04 Truty Tahr desktop image (torrent index) and used Unetbootin to install it as an ISO image on an 8GB USB flash drive (with a 4GB working area).

Then I booted my server into “Try Xubuntu without installing” from the grub boot menu on the USB flash drive.

Once the desktop had loaded I opened a terminal window and became root by simply typing sudo bash.

Next I installed ZFS by following the instructions at this page:

# apt-add-repository --yes ppa:zfs-native/stable
# apt-get update
# apt-get install debootstrap ubuntu-zfs

Boot USB Formatting

I followed the instructions at this blog post to format my boot USB drive.

Hard Drive Encryption

Following the instructions at this blog post I set off to encrypt my two spinning hard drives.

My two hard drives were located at /dev/sda and /dev/sdb. To set up the encryption:

# cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha256 /dev/sda
# cryptsetup luksOpen /dev/sda crypt1

# cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha256 /dev/sdb
# cryptsetup luksOpen /dev/sdb crypt2

Note that aes-xts-plain64 should be used for drives exceeding 2TB in size, aes-xts-plain is adequate for drives less than 2TB. Note that aes-xts-plain can still be used for drives exceeding 2TB but numbers will roll over due to the 32-bit counter used.

Also sha256 is the default preference now instead of sha1 for the hash.

Creating zpool

Next I set up the zpool mirroring the encrypted devices:

# zpool create -o ashift=12 rpool mirror /dev/mapper/crypt1 /dev/mapper/crypt2
# zpool list
# zpool status rpool

Note! The Linux boot process requires the root pool to be named rpool – I tried something different in spite of reading warnings and in the end I had to rename my pool on boot into the new system. So call it something else if you want – but you will end up in a BusyBox console eventually and have to import your pool as name rpool.

Creating Filesystems

# zfs create rpool/ROOT
# zfs create rpool/VAR
# zfs set compression=lz4 rpool/ROOT
# zfs set compression=lz4 rpool/VAR
# zfs create -V 32G -b 4096 \
        -o compression=off \
        -o primarycache=metadata -o secondarycache=none \
        -o sync=always \

Base Installation

First unmount the ZFS pool and map the target mount points of the filesystems. Then export the pool so we can re-import it at a different location.

# zfs unmount -a
# zfs set mountpoint=/ rpool/ROOT
# zfs set mountpoint=/var rpool/VAR
# zpool set bootfs=rpool/ROOT rpool
# zpool export rpool

You can find what zfs pools are available to import:

# zpool import
   pool: rpool
     id: 15624077173946208759
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.

        rpool       ONLINE
          mirror-0  ONLINE
            crypt1  ONLINE
            crypt2  ONLINE

Import the ZFS pool into a mounted location:

# zpool import -R /mnt/rpool rpool

Also mount the USB stick boot partition:

# mkdir /mnt/rpool/boot
# mount /dev/sdc1 /mnt/rpool/boot

Finally do the bootstrap which will download and install a base system (this will take several minutes, if it is exceeding ten and approaching twenty then maybe you accidentally started writing to your flash drive instead of your mounted ZFS filesystem):

# debootstrap trusty /mnt/rpool
I: Base system installed successfully.

You may want to take a snapshot of this install given it took such a long time!

# zfs snapshot rpool/ROOT@after-base-install
# zfs snapshot rpool/VAR@after-base-install
# zfs list -t snapshot
NAME                            USED  AVAIL  REFER  MOUNTPOINT
rpool/ROOT@after-base-install      0      -   136K  -
rpool/VAR@after-base-install       0      -  54.9M  -

Base Cleanup

Edit the hostname:

# echo "bigguns" >/mnt/rpool/etc/hostname

Also add the line to /etc/hosts: bigguns     # or whatever hostname you chose

Now we need to set up the /mnt/rpool/etc/fstab file with the location of the USB boot stick. First we need to find the ID of the USB boot stick:

# blkid
/dev/sdc1: LABEL="usb_boot" UUID="3a12cd22-2280-4913-8662-bf4321db5423" TYPE="ext4"
/dev/sdc2: LABEL="USB_FAT32" UUID="1357-C12F" TYPE="vfat"

Then take the UUID of the boot partition and add to the /mnt/rpool/etc/fstab file:

/dev/disk/by-uuid/3a12cd22-2280-4913-8662-bf4321db5423 /boot/grub auto defaults 0 1

We will also need to set up LUKS. Make a note of the encrypted devices:

# blkid |grep LUKS
/dev/sda: UUID="e61cb221-08e6-48e1-bd41-1b157751b444" TYPE="crypto_LUKS"
/dev/sdb: UUID="6c8b8364-7a5b-492e-9cee-474fa1ba3fd2" TYPE="crypto_LUKS"

Add these to /mnt/rpool/etc/crypttab:

crypt1 UUID=e61cb221-08e6-48e1-bd41-1b157751b444 none luks
crypt2 UUID=6c8b8364-7a5b-492e-9cee-474fa1ba3fd2 none luks


This section is taken largely from this page.

Prepare virtual filesystems from LiveCD into new system:

# mount --bind /dev  /mnt/rpool/dev
# mount --bind /dev/pts  /mnt/rpool/dev/pts
# mount --bind /proc /mnt/rpool/proc
# mount --bind /sys  /mnt/rpool/sys
# chroot /mnt/rpool /bin/bash --login

Install PPA support in the chroot environment:

# locale-gen en_US.UTF-8 # always add this even if you want another language
# locale-gen en_GB.UTF-8
# apt-get update
# apt-get install ubuntu-minimal software-properties-common

Install cryptsetup or you may not be able to unlock your disks the next boot:

# apt-get install cryptsetup

Symlink your LUKS container devices. Without this update-grub will complain that it cannot find the canonical path and error. This tip found at this page.

# ln -s /dev/mapper/crypt1 /dev/crypt1
# ln -s /dev/mapper/crypt2 /dev/crypt2

Assure that future kernel updates will succeed by always creating the symbolic link:

# echo 'ENV{DM_NAME}=="crypt1", SYMLINK+="crypt1"' > /etc/udev/rules.d/99-local-crypt.rules
# echo 'ENV{DM_NAME}=="crypt2", SYMLINK+="crypt2"' >> /etc/udev/rules.d/99-local-crypt.rules

Add the following lines to /etc/fstab:

/dev/mapper/crypt1 / zfs defaults 0 0
/dev/mapper/crypt2 / zfs defaults 0 0
/dev/zvol/rpool/SWAP none swap defaults 0 0

Install ZFS into the chroot environment:

# apt-add-repository --yes ppa:zfs-native/stable
# apt-get update
# apt-get install ubuntu-zfs
# apt-get install grub2-common grub-pc
# apt-get install zfs-initramfs
# apt-get dist-upgrade

Add boot=zfs to grub:

# vi /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="boot=zfs nosplash"

Reinstall grub, but before you do ensure you have an image installed (e.g. linux-image-generic and linux-image-headers) otherwise update-initramfs will silently do nothing and grub will have nothing to boot. The headers are necessary otherwise apt-get will complain about unconfigured packages. Not pretty but has to be done to make the dependencies happy.

# apt-get install --no-install-recommends linux-image-generic linux-headers-generic

At this point let’s take another snapshot!

# zfs snapshot rpool/ROOT@before-bootstrap
# zfs snapshot rpool/VAR@before-bootstrap

Also note that update-initramfs isn’t quite intelligent enough to figure out that you need two encrypted disks to be unlocked before the mirror can be available to ZFS. To that end we need to update a hook named cryptroot. Edit the file /usr/share/initramfs-tools/hooks/cryptroot and in the function get_root_device() comment out the return in the loop:

while read device mount type options dump pass; do
  if [ "$mount" = "/" ]; then
    device=$(canonical_device "$device") || return 0
    echo "$device"
    #return -- COMMENT OUT THIS LINE

Now you can run update-initramfs.

# update-initramfs -c -k all
# update-grub
# grub-install /dev/sdc

Set a root password for the new system:

# passwd root


Exit out of chroot:

# exit

Unmount and export zpool or system may fail to start.

# umount /mnt/rpool/boot
# umount /mnt/rpool/dev/pts
# umount /mnt/rpool/dev
# umount /mnt/rpool/proc
# umount /mnt/rpool/sys
# zfs umount -a
# zpool export rpool

And reboot.

# shutdown -r now

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: