newspaint

Documenting Problems That Were Difficult To Find The Answer To

Category Archives: Technology

Speeding Up ddrescue Kernel Timeouts on Bad Blocks on Ubuntu 16.04

I was having trouble recovering data from an old hard drive. But the worst part was that ddrescue appeared to freeze for 3 minutes trying to access a bad block.

I would see messages like the following in /var/log/syslog:

Sep 17 14:13:25 localhost kernel: [607885.497842] INFO: task ddrescue:14310 blocked for more than 120 seconds.
Sep 17 14:13:25 localhost kernel: [607885.497845]       Tainted: P           OE   4.4.0-93-generic #116-Ubuntu
Sep 17 14:13:25 localhost kernel: [607885.497845] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 17 14:13:25 localhost kernel: [607885.497847] ddrescue        D ffff8802850db9e8     0 14310  14309 0x00000004
Sep 17 14:13:25 localhost kernel: [607885.497849]  ffff8802850db9e8 ffff8802850dba10 ffff8807f9f4aa00 ffff8807c0fdb800
Sep 17 14:13:25 localhost kernel: [607885.497851]  ffff8802850dc000 ffff88081dd96e00 7fffffffffffffff ffff880710370c00
Sep 17 14:13:25 localhost kernel: [607885.497852]  ffff8800d8972300 ffff8802850dba00 ffffffff8183f0d5 0000000000000000
Sep 17 14:13:25 localhost kernel: [607885.497853] Call Trace:
Sep 17 14:13:25 localhost kernel: [607885.497859]  [] schedule+0x35/0x80
Sep 17 14:13:25 localhost kernel: [607885.497861]  [] schedule_timeout+0x1b5/0x270
Sep 17 14:13:25 localhost kernel: [607885.497864]  [] ? __blk_run_queue+0x33/0x40
Sep 17 14:13:25 localhost kernel: [607885.497866]  [] ? queue_unplugged+0x2a/0xb0
Sep 17 14:13:25 localhost kernel: [607885.497867]  [] io_schedule_timeout+0xa4/0x110
Sep 17 14:13:25 localhost kernel: [607885.497870]  [] do_blockdev_direct_IO+0x19e2/0x3410
Sep 17 14:13:25 localhost kernel: [607885.497872]  [] ? I_BDEV+0x20/0x20
Sep 17 14:13:25 localhost kernel: [607885.497874]  [] __blockdev_direct_IO+0x43/0x50
Sep 17 14:13:25 localhost kernel: [607885.497875]  [] blkdev_direct_IO+0x58/0x80
Sep 17 14:13:25 localhost kernel: [607885.497877]  [] generic_file_read_iter+0x4e0/0x670
Sep 17 14:13:25 localhost kernel: [607885.497879]  [] ? fsnotify+0x307/0x4a0
Sep 17 14:13:25 localhost kernel: [607885.497880]  [] blkdev_read_iter+0x35/0x40
Sep 17 14:13:25 localhost kernel: [607885.497883]  [] new_sync_read+0x94/0xd0
Sep 17 14:13:25 localhost kernel: [607885.497885]  [] __vfs_read+0x26/0x40
Sep 17 14:13:25 localhost kernel: [607885.497886]  [] vfs_read+0x86/0x130
Sep 17 14:13:25 localhost kernel: [607885.497888]  [] SyS_read+0x55/0xc0
Sep 17 14:13:25 localhost kernel: [607885.497890]  [] entry_SYSCALL_64_fastpath+0x16/0x71
Sep 17 14:13:53 localhost kernel: [607913.902354] sd 38:0:0:0: timing out command, waited 180s
Sep 17 14:13:53 localhost kernel: [607913.902361] sd 38:0:0:0: [sdf] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Sep 17 14:13:53 localhost kernel: [607913.902362] sd 38:0:0:0: [sdf] tag#0 Sense Key : Hardware Error [current] 
Sep 17 14:13:53 localhost kernel: [607913.902364] sd 38:0:0:0: [sdf] tag#0 Add. Sense: Logical unit communication CRC error (Ultra-DMA/32)
Sep 17 14:13:53 localhost kernel: [607913.902366] sd 38:0:0:0: [sdf] tag#0 CDB: Read(10) 28 00 07 bf 7b 00 00 00 80 00
Sep 17 14:13:53 localhost kernel: [607913.902367] blk_update_request: I/O error, dev sdf, sector 129989376

Clearly waiting 3 minutes for every bad block was unacceptable – especially given the possible extra damage being done to the hard drive.

The article at this link and the article at this link had a solution:

~# cat /sys/block/sdf/device/eh_timeout
10
~# cat /sys/block/sdf/device/timeout
30

~# echo -n "2" >/sys/block/sdf/device/eh_timeout
~# echo -n "3" >/sys/block/sdf/device/timeout

~# ddrescue -n -d -r1 /dev/sdf image-sdf-80G.bin image-sdf-80G.map
GNU ddrescue 1.19
Press Ctrl-C to interrupt
Initial status (read from logfile)
rescued:    24643 MB,  errsize:  23029 kB,  errors:     398

Current status
rescued:    80025 MB,  errsize:   79872 B,  current rate:        0 B/s
   ipos:    20015 MB,   errors:      32,    average rate:    9196 kB/s
   opos:    20015 MB, run time:    1.67 h,  successful read:      42 s ago
Retrying bad sectors... Retry 1 (forwards)

This greatly reduced the time before ddrescue gave up on a block to several seconds (~20 seconds) from several minutes.

How to Mount Partition from Disk Image

Adopted from this article.

First identify the starting offset of the partition you want to mount. To do this use the parted print command to list the partitions:

~$ parted image-sdf-80G.bin unit B print
Model:  (file)
Disk /tmp/image-sdf-80G.bin: 80025280000B
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start         End           Size          Type      File system  Flags
 1      32256B        2146798079B   2146765824B   primary   fat32        boot
 2      2146798080B   80015523839B  77868725760B  extended               lba
 5      2146830336B   38074821119B  35927990784B  logical   ntfs
 6      38074853376B  80015523839B  41940670464B  logical   ntfs

With this you can loop mount the file using the offset of the partition from the beginning of the file, e.g.:

~$ mount -o ro,loop,offset=2146830336 image-sdf-80G.bin /mnt/wantedpartition

Getting LetsEncrypt Working With Exim

LetsEncrypt offers free SSL certificates if you use the tool provided. However the tricky part is validation which expects to be able to open a listening webserver port on your server.

In my case I had put Exim on a virtual server and didn’t want to open access to the outside world for validation purposes.

On Ubuntu 16.04 (Xenial) I installed certbot:

~$ sudo apt-get install software-properties-common
~$ sudo add-apt-repository ppa:certbot/certbot
~$ sudo apt-get update
~$ sudo apt-get install certbot

Then I requested a manual validation for a certificate request:

~$ sudo certbot -d mail01.newspaint.wordpress.com --manual --preferred-challenges dns certonly

This requested I add a TXT record to my domain name server (which you have to do in a separate terminal window because certbot waits for you to complete this before requesting you to hit a key to continue). For example I added the following to my zone file for Bind 9 / named:

_acme-challenge.mail01 TXT ( "483OWAC8DAb5iT7BCBxzZ0Qyqqbzh_PFeWiauprsY3C" )

An easy mistake is to simply add _acme-challenge without the subdomain (mail01) or fully qualifying the domain name without the trailing dot (i.e. _acme-challenge.mail01.newspaint.wordpress.com.). You have to ensure you update the serial number of your zone file and reload it.

The LetsEncrypt generated certificates/keys are placed in /etc/letsencrypt/archive/domain_name/ and the most recent certificate is symlinked from the /etc/letsencrypt/live/domain_name/ directory.

Now add the following to your Exim configuration file:

tls_certificate = /etc/letsencrypt/live/mail01.newspaint.wordpress.com/fullchain.pem
tls_privatekey = /etc/letsencrypt/live/mail01.newspaint.wordpress.com/privkey.pem

It’s very important to examine the default permissions certbot creates for those directories because this affects Exim:

drwxr-xr-x 23 root root 4096 Aug 29 00:55 /
drwxr-xr-x 97 root root 4096 Sep 10 03:34 /etc/
drwxr-xr-x  8 root root 4096 Sep 16 00:11 /etc/letsencrypt/
drwx------  3 root root 4096 May 21 08:29 /etc/letsencrypt/archive/
drwxr-xr-x  2 root root 4096 Jul 20 12:18 /etc/letsencrypt/archive/mail01.newspaint.wordpress.com/
-rw-r--r--  1 root root 3550 May 21 08:29 /etc/letsencrypt/archive/mail01.newspaint.wordpress.com/fullchain1.pem
-rw-r--r--  1 root root 1704 May 21 08:29 /etc/letsencrypt/archive/mail01.newspaint.wordpress.com/privkey1.pem
drwx------  3 root root 4096 May 21 08:29 /etc/letsencrypt/live/
drwxr-xr-x  2 root root 4096 Jul 20 12:18 /etc/letsencrypt/live/mail01.newspaint.wordpress.com/

When Exim runs on Ubuntu it runs as the user Debian-exim. And it doesn’t read the certificate/key until an SMTP connection is made to it – so unlike other daemons that start as root, read SSL certificates, then drop into a less privileged user, Exim doesn’t read the certificate until it needs it when it is already an unprivileged user.

As you can see above there are two directories an ordinary user cannot see:

  • /etc/letsencrypt/archive/
  • /etc/letsencrypt/live/

Both of these must be made visible to the Debian-exim user. Your choices are:

  • chmod 755 /etc/lets/encrypt/{archive,live} #(but that lets anybody read the files)
  • chmod 755 /etc/lets/encrypt/{archive,live}; chgrp Debian-exim /etc/lets/encrypt/{archive,live} #(much better)

If you don’t do this you’ll get the following error from Exim in /var/log/exim4/mainlog:

2017-09-15 23:53:07 TLS error on connection from mail-it0-f44.google.com [209.85.214.44] (cert/key setup: cert=/etc/letsencrypt/live/mail01.newspaint.wordpress.com/fullchain.pem key=/etc/letsencrypt/live/mail01.newspaint.wordpress.com/privkey.pem): Error while reading file.

Using PHP as CGI in Apache 2.4

I wanted to use PHP in CGI mode on a webserver. Granted – it’s not as convenient or fast as mod_php. I was using this to allow myself to run phpMyAdmin on a low memory server.

I created an .htaccess in my phpMyAdmin path:

# create an action called "dophp" which calls /phpMyAdmin/dophp.cgi
Action dophp /phpMyAdmin/dophp.cgi

# allow files ending in .cgi to be executed (our script dophp.cgi)
<FilesMatch "\.cgi$">
  Options +ExecCGI
  SetHandler cgi-script
</FilesMatch>

# all PHP files should be handled by a call to /phpMyAdmin/dophp.cgi
<FilesMatch "\.php$">
  SetHandler dophp
</FilesMatch>

Next I created the file dophp.cgi in the same directory:

#!/bin/sh

exec /usr/bin/php-cgi

I ensured that my Apache configuration had enabled modules actions and either cgid (if you’re not using the prefork MPM) or cgi (if you are using the prefork MPM).

Lastly I needed to edit /etc/php/7.0/cgi/php.ini and change the following line:

cgi.force_redirect = 0

Otherwise I’d get the error in my Apache log:

malformed header from script 'dophp.cgi': Bad header: <b>Security Alert!</b> The PHP CGI cannot be accessed directly.

Installing ZFS on LUKS on Ubuntu 16.04 on Hetzner Dedicated Server

Hetzner, in Germany, offer dedicated servers on auction.

My goal was to set up an Ubuntu 16.04 server with ECC (error correcting code) memory and two hard drives in a mirror arrangement running ZFS.

I wanted the hard drives to be encrypted using LUKS (Linux unified key setup) (with the exception of the boot partition).

The setup would look like this:

 ____________     _____________    __________
|            |   |             |  |          |
| rpool/ROOT |   | (zvol swap) |  | mounted  |
|   ZFS fs   |   |             |  |   /boot  |
|____________|   |_____________|  |__________|
 _____|_________________|______         |
|                              |        |
|     zpool mirror "rpool"     |        |
|______________________________|        |
 _____|______     _____|______          |
|            |   |            |         |
| LUKS crypt |   | LUKS crypt |         |
| /dev/mapper|   | /dev/mapper|         |
|     /crypt1|   |     /crypt2|         |
|____________|   |____________|         |
 _____|______     _____|______     _____|______
|            |   |            |   |            |
| partition  |   | partition  |   | partition  |
| /dev/sda2  |   | /dev/sdb2  |   | /dev/sda1  |
|____________|   |____________|   |____________|

The Hetzner dedicated server I tried this on did not have built-in KVM – so it was necessary to find a method of allowing LUKS encrypted drives to be unlocked/opened prior to booting Linux – and the solution appeared to put dropbear in initramfs so one could SSH during the boot phase for the purpose of unlocking the partitions prior to mounting the root ZFS filesystem.

Before you do this please note – you will require some files from another Xenial (Ubuntu 16.04) distribution! See bootstrap for more information.

Start by booting into the Rescue Linux image (which happens to be Jessie Debian as of this article).

Enable Rescue Mode from Hetzner Control Panel

Enable Rescue Mode from Hetzner Control Panel

After enabling Rescue Mode from the control panel you can then select “Reset” from the control panel to boot into the Rescue image.

Health Warning

This process took me 12 hours to iron out issues I had. It wasn’t helped by the fact that I didn’t have a KVM (until near the end) to indicate what was going wrong during boot, and when I finally did get a KVM attached for an hour most keys I typed were duplicated by the KVM which made any kind of debugging almost impossible. Writing this article has taken me 6 additional hours.

It should be more straight forward now I’ve documented the process but you may find yourself tripped up by something small which is annoyingly difficult to diagnose blind as you are across the Internet.

Every Time You Boot Into Rescue Mode

You’re going to have to enable ZFS which isn’t part of the Rescue image.

This is agonisingly slow taking around 6-10 minutes, it can be slightly sped up (down to 3 minutes) by removing whatever kernel image has been installed but isn’t being currently used, e.g.:

-------------------------------------------------------------------

  Welcome to the Hetzner Rescue System.

  This Rescue System is based on Debian 8.0 (jessie) with a newer
  kernel. You can install software as in a normal system.

  To install a new operating system from one of our prebuilt
  images, run 'installimage' and follow the instructions.

  More information at http://wiki.hetzner.de

-------------------------------------------------------------------

Hardware data:

Network data:

root@rescue ~ # cat /proc/version
Linux version 4.10.16 (build@build.rzse.hetzner.de) (gcc version 4.9.2 (Debian 4.9.2-10) ) #25 SMP Tue May 16 12:37:37 CEST 2017

root@rescue ~ # apt-get remove -y linux-image-4.12.4 linux-headers-4.12.4
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  linux-headers-4.12.4 linux-image-4.12.4
0 upgraded, 0 newly installed, 2 to remove and 0 not upgraded.
After this operation, 121 MB disk space will be freed.
(Reading database ... 76476 files and directories currently installed.)
Removing linux-headers-4.12.4 (4.12.4-6) ...
Removing linux-image-4.12.4 (4.12.4-6) ...

root@rescue ~ # time apt-get install -t jessie-backports zfs-dkms
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  dh-python dkms libmpdec2 libnvpair1linux libpython3-stdlib
  libpython3.4-minimal libpython3.4-stdlib libuutil1linux libzfs2linux
  libzpool2linux python3 python3-minimal python3.4 python3.4-minimal spl
  spl-dkms sudo zfs-zed zfsutils-linux
Suggested packages:
  python3-apport menu python3-doc python3-tk python3-venv python3.4-venv
  python3.4-doc binfmt-support nfs-kernel-server samba-common-bin
  zfs-initramfs zfs-dracut
The following NEW packages will be installed:
  dh-python dkms libmpdec2 libnvpair1linux libpython3-stdlib
  libpython3.4-minimal libpython3.4-stdlib libuutil1linux libzfs2linux
  libzpool2linux python3 python3-minimal python3.4 python3.4-minimal spl
  spl-dkms sudo zfs-dkms zfs-zed zfsutils-linux
0 upgraded, 20 newly installed, 0 to remove and 84 not upgraded.
Need to get 8,085 kB of archives.
After this operation, 37.4 MB of additional disk space will be used.

real    2m48.082s
user    2m56.348s
sys     0m18.100s
root@rescue ~ #

Great! Now you can use ZFS in the rescue system.

Partitioning Hard Drives

I wanted 3 partitions on each hard drive. One for /boot (unencrypted), the largest for my ZFS root which will be encrypted, and the final (which would be put at the front of the disk) for a GRUB BIOS partition.

The disk would be partitioned as follows:

 ___________ ___________ ___________
|           |           |           |
| /dev/sda3 | /dev/sda1 | /dev/sda2 |
| GRUB BIOS |   /boot   |   LUKS    |
|     1 GiB |     4 GiB | remainder |
|___________|___________|___________|

I used parted to create these partitions:

root@rescue ~ # parted /dev/sda mklabel gpt # make disk GPT
root@rescue ~ # parted /dev/sda mkpart myboot 1GiB 5GiB
root@rescue ~ # parted /dev/sda mkpart mycrypt 5GiB 100%
root@rescue ~ # parted /dev/sda mkpart mygrub 2048s 1GiB
root@rescue ~ # parted /dev/sda set 3 bios_grub on
root@rescue ~ # parted /dev/sda align-check opt 1 # ensure optimal
root@rescue ~ # parted /dev/sda align-check opt 2 # ensure optimal
root@rescue ~ # parted /dev/sda print # review

Disk /dev/sda: 3001GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name     Flags
 3      1049kB  1074MB  1073MB               mygrub   bios_grub
 1      1074MB  5369MB  4295MB               myboot
 2      5369MB  3001GB  2995GB               mycrypt

root@rescue ~ # parted /dev/sdb mklabel gpt # make disk GPT
root@rescue ~ # parted /dev/sdb mkpart myboot 1GiB 5GiB
root@rescue ~ # parted /dev/sdb mkpart mycrypt 5GiB 100%
root@rescue ~ # parted /dev/sdb mkpart mygrub 2048s 1GiB
root@rescue ~ # parted /dev/sdb set 3 bios_grub on
root@rescue ~ # parted /dev/sdb align-check opt 1 # ensure optimal
root@rescue ~ # parted /dev/sdb align-check opt 2 # ensure optimal
root@rescue ~ # parted /dev/sdb print # review

Disk /dev/sdb: 3001GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name     Flags
 3      1049kB  1074MB  1073MB               mygrub   bios_grub
 1      1074MB  5369MB  4295MB               myboot
 2      5369MB  3001GB  2995GB               mycrypt

root@rescue ~ #

Encrypting Paritions

The Linux root directory will be put on ZFS which will be mirrored off the /dev/sda2 and /dev/sdb2 partitions. So these partitions need to be encrypted:

root@rescue ~ # cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha256 /dev/sda2
WARNING!
========
This will overwrite data on /dev/sda2 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter passphrase: 
Verify passphrase: 
root@rescue ~ # cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha256 /dev/sdb2
WARNING!
========
This will overwrite data on /dev/sdb2 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter passphrase: 
Verify passphrase: 
root@rescue ~ #

Then opened:

root@rescue ~ # cryptsetup luksOpen /dev/sda2 crypt1
Enter passphrase for /dev/sda2:
root@rescue ~ # cryptsetup luksOpen /dev/sdb2 crypt2
Enter passphrase for /dev/sdb2:
root@rescue ~ #

Setting up ZFS on Encrypted Partitions

First create a ZFS pool, titled “rpool” (because that’s what Linux used to need, it’s more convention than anything, but if you choose something else be warned sometimes scripts are hard-coded to expect “rpool” and you might run into trouble – it’s a lot safer just to call it “rpool”).

It will be a mirror (you can choose something else at your own risk) – so if data is corrupted on one partition ZFS should be able to automatically recover it from the other.

root@rescue ~ # zpool create -o ashift=12 rpool mirror /dev/mapper/crypt1 /dev/mapper/crypt2
root@rescue ~ # zpool status
  pool: rpool
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            crypt1  ONLINE       0     0     0
            crypt2  ONLINE       0     0     0

errors: No known data errors
root@rescue ~ #

Create a swap volume that is the same size as the memory in your dedicated server (so if you have 16G of ram add the -V 16G parameter). Format the swap. Then create a root ZFS filesystem and turn on compression of the filesystem.

root@rescue ~ # zfs create -V 32G -b 4096 -o compression=off -o primarycache=metadata -o secondarycache=none -o sync=always rpool/SWAP
root@rescue ~ # mkswap /dev/zvol/rpool/SWAP
Setting up swapspace version 1, size = 33554428 KiB
no label, UUID=7e475689-c8c1-2a41-552c-8ff37cec3be1

root@rescue ~ # zfs create rpool/ROOT
root@rescue ~ # zfs set compression=lz4 rpool/ROOT

Unmount the ZFS filesystem (just unmount all ZFS filesystems) and configure the mount point of the root ZFS filesystem. Tell the pool that it should boot into the root ZFS filesystem. Finally export the pool so we can import it again later at a temporary location.

root@rescue ~ # zfs unmount -a # unmount all ZFS filesystems
root@rescue ~ # zfs set mountpoint=/ rpool/ROOT

root@rescue ~ # zpool set bootfs=rpool/ROOT rpool

root@rescue ~ # zpool export rpool # in preparation for dummy mount
root@rescue ~ # zpool import # confirm pool is available to import
   pool: rpool
     id: 14246658913528246541
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

        rpool       ONLINE
          mirror-0  ONLINE
            crypt1  ONLINE
            crypt2  ONLINE

root@rescue ~ # zpool import -R /mnt/rpool rpool # import to different mount

From this point forward the Ubuntu system we will build will go into /mnt/rpool/. There is another “rpool” directory under this – ignore that – that is the pool’s directory, but the filesystem’s directory (rpool/ROOT) is actually the temporary mount point /mnt/rpool/.

Create and Mount Boot Partition

For completeness format both /dev/sda1 and /dev/sdb1 (boot partitions). But we’ll only mount (and install to) one of them. You may want to back up the /dev/sda1 boot partition when this is all done.

root@rescue ~ # mkfs.ext4 -L "sda_boot" /dev/sda1
root@rescue ~ # mkfs.ext4 -L "sdb_boot" /dev/sdb1

root@rescue ~ # mkdir /mnt/rpool/boot # mount point
root@rescue ~ # mount /dev/sda1 /mnt/rpool/boot

Bootstrap

This is the fun part! Bootstrapping the initial operating system into the target location (our ZFS root filesystem and mounted /boot partition).

But there’s a problem. Jessie doesn’t come with support for bootstrapping Ubuntu Xenial (16.04). In order to do this you will have to find an existing Xenial system and copy the following files onto your rescue host:

  • /usr/share/keyrings/ubuntu-archive-keyring.gpg
  • /usr/share/debootstrap/scripts/xenial

You should now have these files on your rescue system, if you don’t you cannot proceed (or you’ll have to choose a different distribution).

root@rescue ~ # ls /usr/share/keyrings/ubuntu-archive-keyring.gpg
/usr/share/keyrings/ubuntu-archive-keyring.gpg
root@rescue ~ # ls /usr/share/debootstrap/scripts/xenial
/usr/share/debootstrap/scripts/xenial

Now do the bootstrap! Optionally, after it is complete, take a ZFS snapshot of the filesystem in case we want to roll back later.

root@rescue ~ # time debootstrap --arch=amd64 xenial /mnt/rpool
I: Base system installed successfully.

real    3m13.981s
user    0m30.020s
sys     0m9.184s

root@rescue ~ # zfs snapshot rpool/ROOT@after-base-install # optional

Chroot Into Image

From this point forward we will be in a chroot which will feel like we’re in the newly installed operating system. We aren’t, really, but it will feel that way. And we can make changes necessary before the first boot.

Set up the bind mounts (required for the chroot to function properly), copy resolv.conf to the /run directory (required to resolve domain names inside the chroot) and chroot.

root@rescue ~ # mkdir /run/resolvconf
root@rescue ~ # cp /etc/resolv.conf /run/resolvconf/

root@rescue ~ # for i in dev dev/pts proc sys run; do echo ==$i==; mount --bind /$i /mnt/rpool/$i; done
root@rescue ~ # chroot /mnt/rpool /bin/bash --login
root@rescue:/#

Chroot: Hostname

Edit /etc/hostname to contain your desired hostname.

Add a line to /etc/hosts that contains the text “127.0.1.1 bigguns” (where “bigguns” should be your_hostname, the same as you put into /etc/hostname in the step above).

Chroot: Use Hetzner Mirror

Optional, but if you want to use the local mirror, edit /etc/apt/sources.list and use the Xenial sources specified in the Hetzner Ubuntu Aptitude mirror document:

deb http://mirror.hetzner.de/ubuntu/packages xenial           main restricted universe multiverse
deb http://mirror.hetzner.de/ubuntu/packages xenial-updates   main restricted universe multiverse
deb http://mirror.hetzner.de/ubuntu/packages xenial-backports main restricted universe multiverse
deb http://mirror.hetzner.de/ubuntu/packages xenial-security  main restricted universe multiverse

After doing this do an apt update.

root@rescue:/# apt-get update
Fetched 16.9 MB in 2s (7286 kB/s)               
Reading package lists... Done

Chroot: Set Up Filesystem Tables

Find out the UUID of the boot device and add it to /etc/fstab:

root@rescue:/# blkid |grep /dev/sda1
/dev/sda1: LABEL="sda_boot" UUID="4049ab3c-a015-46b3-a594-d23ac8218c62" TYPE="ext4" PARTLABEL="myboot" PARTUUID="43150b47-89df-4765-825a-edd22898605d"

root@rescue:/# echo "/dev/disk/by-uuid/4049ab3c-a015-46b3-a594-d23ac8218c62 /boot auto defaults 0 1" >>/etc/fstab

Add more lines to /etc/fstab:

root@rescue:/# echo "/dev/mapper/crypt1 / zfs defaults 0 0" >>/etc/fstab
root@rescue:/# echo "/dev/mapper/crypt2 / zfs defaults 0 0" >>/etc/fstab
root@rescue:/# echo "/dev/zvol/rpool/SWAP none swap defaults 0 0" >>/etc/fstab

If you’re wondering why there are two root directory entries there – they are so cryptsetup can find them while running initramfs hooks and ensures that, during boot, you are offered to provide the password to decrypt both LUKS encrypted partitions before continuing.

Add LUKS encrypted partitions to /etc/crypttab:

root@rescue:/# blkid |grep LUKS
/dev/sda2: UUID="fcc1c632-0a5d-4061-be4e-18d2097e91b0" TYPE="crypto_LUKS" PARTLABEL="mycrypt" PARTUUID="809fb832-db3a-47a6-8e0d-b7d0491011e8"
/dev/sdb2: UUID="51451730-00d5-48e5-a9ed-bd6f9d0260cb" TYPE="crypto_LUKS" PARTLABEL="mycrypt" PARTUUID="9877c3c6-5b21-4b69-bc5a-e346ae1567ca"

root@rescue:/# echo "crypt1 UUID=fcc1c632-0a5d-4061-be4e-18d2097e91b0 none luks" >>/etc/crypttab
root@rescue:/# echo "crypt2 UUID=51451730-00d5-48e5-a9ed-bd6f9d0260cb none luks" >>/etc/crypttab

Symlink LUKS container devices or update-grub will complain it cannot find canonical path and error (later during installation).

root@rescue:/# ln -s /dev/mapper/crypt1 /dev/crypt1
root@rescue:/# ln -s /dev/mapper/crypt2 /dev/crypt2

Assure that future kernel updates will succeed by always creating the symbolic link:

root@rescue:/# echo 'ENV{DM_NAME}=="crypt1", SYMLINK+="crypt1"' > /etc/udev/rules.d/99-local-crypt.rules
root@rescue:/# echo 'ENV{DM_NAME}=="crypt2", SYMLINK+="crypt2"' >> /etc/udev/rules.d/99-local-crypt.rules

Chroot: Ensure Apt Packages for Filesystems Present

Install PPA support first.

root@rescue:/# locale-gen en_US.UTF-8 # always add this even if you want another language
root@rescue:/# locale-gen en_GB.UTF-8
root@rescue:/# apt-get install ubuntu-minimal software-properties-common

Get packages related to encryption, ZFS, and GRUB.

root@rescue:/# apt-get install cryptsetup # or may not be able to unlock at boot
root@rescue:/# apt-get install zfs-initramfs zfs-dkms # we need ZFS
root@rescue:/# apt-get install grub2-common grub-pc # add GRUB for booting

As GRUB is installing an interactive choice may be given for which devices to install to, choose /dev/sda and /dev/sdb.

Ensure GRUB boots into ZFS by adding “boot=zfs” to /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="boot=zfs nosplash"

Make ZFS calm down (it will access the disks every 5 seconds by default flushing buffers) to access the disks once every 30 seconds by editing (or creating) /etc/modprobe.d/zfs.conf and adding the line:

options zfs zfs_txg_timeout=30

Update /usr/share/initramfs-tools/hooks/cryptroot and uncomment out the return in function get_fs_devices() so that all the encrypted drives will be unlocked during boot (or else just the first drive will, which is no good for a mirror).

get_fs_devices() {
  local device mount type options dump pass
  local wantmount="$1"

  if [ ! -r /etc/fstab ]; then
    return 1
  fi

  grep -s '^[^#]' /etc/fstab | \
    while read device mount type options dump pass; do
      if [ "$mount" = "$wantmount" ]; then
        local devices
        if [ "$type" = "btrfs" ]; then
          for dev in $(btrfs filesystem show $(canonical_device "$device" --no-simplify) 2>/dev/null | sed -r -e 's/.*devid .+ path (.+)/\1/;tx;d;:x') ; do
            devices="$devices $(canonical_device "$dev")"
          done
        else
          devices=$(canonical_device "$device") || return 0
        fi
        echo "$devices"
        #return ## COMMENT OUT THIS LINE
      fi
    done
}

Install a Linux image. This can take a while so turning off sync on the filesystem can speed it up because of the number of files in the headers package.

root@rescue:/# zfs get sync rpool/ROOT
NAME        PROPERTY  VALUE     SOURCE
rpool/ROOT  sync      standard  default

root@rescue:/# zfs set sync=disabled rpool/ROOT

root@rescue:/# time apt-get install linux-image-generic linux-headers-generic
real    4m11.613s
user    2m23.124s
sys     0m27.472s

root@rescue:/# zfs set sync=standard rpool/ROOT

Chroot: Use Dropbear to Allow Disk Unlocking During Boot Via SSH

Thanks go to this article which describes the process of adding dropbear to the boot process allowing decryption of disks over SSH without the need for a KVM.

Install packages.

root@rescue:/# apt-get install dropbear busybox

Update /etc/initramfs-tools/initramfs.conf and ensure the following lines are present:

BUSYBOX=y
DROPBEAR=y

Create dropbear keys:

root@rescue:/# mkdir /etc/initramfs-tools/root
root@rescue:/# mkdir /etc/initramfs-tools/root/.ssh
root@rescue:/# dropbearkey -t rsa -f /etc/initramfs-tools/root/.ssh/id_rsa.dropbear

Convert dropbear key to openssh format:

root@rescue:/# /usr/lib/dropbear/dropbearconvert dropbear openssh /etc/initramfs-tools/root/.ssh/id_rsa.dropbear /etc/initramfs-tools/root/.ssh/id_rsa

Extract public key from /etc/initramfs-tools/root/.ssh/id_rsa:

root@rescue:/# dropbearkey -y -f /etc/initramfs-tools/root/.ssh/id_rsa.dropbear | grep "^ssh-rsa " > /etc/initramfs-tools/root/.ssh/id_rsa.pub

Add public key to authorized_keys file:

root@rescue:/# cat /etc/initramfs-tools/root/.ssh/id_rsa.pub >> /etc/initramfs-tools/root/.ssh/authorized_keys
root@rescue:/# chmod 600 /etc/initramfs-tools/root/.ssh/authorized_keys

To enable the start of dropbear add or update /etc/default/dropbear:

NO_START=0

Copy into /etc/initramfs-tools/hooks/crypt_unlock.sh script from this link.

root@rescue:/# chmod 755 /etc/initramfs-tools/hooks/crypt_unlock.sh

Disable dropbear service on boot so openssh is used after partition is decrypted.

root@rescue:/# update-rc.d dropbear disable

TAKE A COPY OF /etc/initramfs-tools/root/.ssh/id_rsa (the private key)! You’ll need it to ssh into dropbear later!

Chroot: Add OpenSSH, Set Root Password, Deal With Ethernet, Rebuilt initramfs

Install OpenSSH server and change the port to something obscure.

root@rescue:/# apt-get install openssh-server

root@rescue:/# perl -p -i.bak -e 's/^Port.*$/Port 222/' /etc/ssh/sshd_config

Set root password (or add your own public key to /root/.ssh/authorized_keys, but if you do ensure you set permissions to 600).

root@rescue:/# passwd root

Now for the very tricky part. Ubuntu has changed the names of Ethernet ports from eth0, eth1 etc to tricky things like eno1, ens1, enp4s0.

The only way to really know what your Ethernet interface will be named is to reboot, fall into busybox, and type dmesg |grep -i eth – but that involves a KVM being installed on your server which is difficult and potentially expensive.

In my case I’m guessing my Ethernet name (enp4s0) was gathered from:

root@rescue:/# dmesg |grep eth0
[    1.459268] e1000e 0000:04:00.0 eth0: Intel(R) PRO/1000 Network Connection

But I may be completely wrong about that.

Why is this important? Because you want to be able to create a file like the following:

root@rescue:/# echo -en "auto enp4s0\niface enp4s0 inet dhcp\n" >/etc/network/interfaces.d/enp4s0

Now update initramfs or you may be missing important services (like cryptsetup) on boot.

root@rescue:/# update-initramfs -c -k all
update-initramfs: Generating /boot/initrd.img-4.4.0-93-generic
cryptsetup: WARNING: could not determine root device from /etc/fstab

root@rescue:/# update-grub
Generating grub configuration file ...
Warning: Setting GRUB_TIMEOUT to a non-zero value when GRUB_HIDDEN_TIMEOUT is set is no longer supported.
Found linux image: /boot/vmlinuz-4.4.0-93-generic
Found initrd image: /boot/initrd.img-4.4.0-93-generic
done

Chroot: Dist Upgrade and Reboot

root@rescue:/# apt-get update
root@rescue:/# apt-get dist-upgrade

Escape from chroot.

root@rescue:/# exit
root@rescue ~ # for i in run sys proc dev/pts dev; do echo ==$i==; umount /mnt/rpool/$i; done
root@rescue ~ # umount /mnt/rpool/boot
root@rescue ~ # zfs umount -a
root@rescue ~ # zpool export rpool

root@rescue ~ # shutdown -r now

Miscellaneous

Clean Up on Boot

After a successful boot into Linux I had some /lib/cryptsetup/askpass processes lying around. So I added the following lines to my /etc/rc.local file:

/bin/ps ax |/bin/grep /lib/cryptsetup/askpass |/usr/bin/cut -c1-5 |/usr/bin/xargs /bin/kill
sleep 5
/bin/ps ax |/bin/grep /lib/cryptsetup/askpass |/usr/bin/cut -c1-5 |/usr/bin/xargs /bin/kill -9

Setting NTP

Ubuntu 16.04 uses timesyncd for synchronisation.

If desired Hetzner’s own NTP servers can be used.

Edit the file /etc/systemd/timesyncd.conf and add the lines (if missing):

[Time]
NTP=ntp1.hetzner.de ntp2.hetzner.com ntp3.hetzner.net
FallbackNTP=0.de.pool.ntp.org 1.de.pool.ntp.org 2.de.pool.ntp.org 3.de.pool.ntp.org

Check the service is running:

root@myhost:/# service systemd-timesyncd status
 systemd-timesyncd.service - Network Time Synchronization
   Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; vendo
r preset: enabled)
  Drop-In: /lib/systemd/system/systemd-timesyncd.service.d
           └─disable-with-time-daemon.conf
   Active: active (running) since Thu 2017-09-07 03:13:33 UTC; 1h 59min ago
     Docs: man:systemd-timesyncd.service(8)
 Main PID: 1851 (systemd-timesyn)
   Status: "Synchronized to time server 91.189.91.157:123 (ntp.ubuntu.com)."
   CGroup: /system.slice/systemd-timesyncd.service
           └─1851 /lib/systemd/systemd-timesyncd

Sep 07 03:13:33 lalor systemd[1]: Starting Network Time Synchronization...
Sep 07 03:13:33 lalor systemd[1]: Started Network Time Synchronization.
Sep 07 03:14:04 lalor systemd-timesyncd[1851]: Synchronized to time server 91.189.91.157:123 (ntp.ubuntu.com).

root@myhost:/# service systemd-timesyncd restart
root@myhost:/# service systemd-timesyncd status
 systemd-timesyncd.service - Network Time Synchronization
   Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; vendo
r preset: enabled)
  Drop-In: /lib/systemd/system/systemd-timesyncd.service.d
           └─disable-with-time-daemon.conf
   Active: active (running) since Thu 2017-09-07 05:15:18 UTC; 4s ago
     Docs: man:systemd-timesyncd.service(8)
 Main PID: 2754 (systemd-timesyn)
   Status: "Synchronized to time server 213.239.239.164:123 (ntp1.hetzner.de)."
   CGroup: /system.slice/systemd-timesyncd.service
           └─2754 /lib/systemd/systemd-timesyncd

Sep 07 05:15:18 lalor systemd[1]: Starting Network Time Synchronization...
Sep 07 05:15:18 lalor systemd[1]: Started Network Time Synchronization.
Sep 07 05:15:18 lalor systemd-timesyncd[2754]: Synchronized to time server 213.239.239.164:123 (ntp1.hetzner.de).

You may also want to change the system timezone to Europe/Berlin (or whatever preference you have):

root@myhost:/# dpkg-reconfigure tzdata

Recovery in Rescue

More likely than not something went wrong. To get back into your chroot in Rescue mode here’s the cheatsheet.

Entering Chroot

root@rescue ~ # apt-get remove -y linux-image-4.12.4 linux-headers-4.12.4
root@rescue ~ # time apt-get install -t jessie-backports zfs-dkms

root@rescue ~ # cryptsetup luksOpen /dev/sda2 crypt1
root@rescue ~ # cryptsetup luksOpen /dev/sdb2 crypt2

root@rescue ~ # zfs unmount -a # unmount all ZFS filesystems
root@rescue ~ # zpool export rpool # in preparation for dummy mount
root@rescue ~ # zpool import -R /mnt/rpool rpool # import to different mount
root@rescue ~ # mount /dev/sda1 /mnt/rpool/boot

root@rescue ~ # mkdir /run/resolvconf
root@rescue ~ # cp /etc/resolv.conf /run/resolvconf/

root@rescue ~ # for i in dev dev/pts proc sys run; do echo ==$i==; mount --bind /$i /mnt/rpool/$i; done
root@rescue ~ # chroot /mnt/rpool /bin/bash --login

And inside the chroot:

root@rescue:/# ln -s /dev/mapper/crypt1 /dev/crypt1
root@rescue:/# ln -s /dev/mapper/crypt2 /dev/crypt2

After Leaving Chroot

root@rescue ~ # for i in run sys proc dev/pts dev; do echo ==$i==; umount /mnt/rpool/$i; done
root@rescue ~ # umount /mnt/rpool/boot
root@rescue ~ # zfs umount -a
root@rescue ~ # zpool export rpool

Installing Selenium for Perl on Ubuntu 16.04

First I had to download the “Selenium Standalone Server” (a Java .jar file) from the Selenium download page (version 3.5.3 as of writing this article).

Then I ensured I had a JRE (Java run-time environment) by executing:

~$ sudo apt-get install openjdk-9-jre

I also downloaded the Mozilla GeckoDriver from the download page specified above (version 0.18.0 as of writing this article). I then extracted geckodriver-v0.18.0-linux64.tar.gz (the geckodriver executable) into the same folder as the Java Selenium Standalone Server .jar.

~$ cd /opt/selenium
~$ tar -xvjf geckodriver-v0.18.0-linux64.tar.gz

I started up the Java Selenium Standalone Server in a different terminal window by running:

~$ cd /opt/selenium
~$ $ java -jar selenium-server-standalone-3.5.3.jar
2017-09-02 04:35:00.772:INFO::main: Logging initialized @601ms to org.seleniumhq.jetty9.util.log.StdErrLog
2017-09-02 04:35:00.973:INFO:osjs.Server:main: jetty-9.4.5.v20170502
2017-09-02 04:35:00.991:WARN:osjs.SecurityHandler:main: ServletContext@o.s.j.s.ServletContextHandler@77f1baf5{/,null,STARTING} has uncovered http methods for path: /
2017-09-02 04:35:00.995:INFO:osjsh.ContextHandler:main: Started o.s.j.s.ServletContextHandler@77f1baf5{/,null,AVAILABLE}
2017-09-02 04:35:01.012:INFO:osjs.AbstractConnector:main: Started ServerConnector@87a85e1{HTTP/1.1,[http/1.1]}{0.0.0.0:4444}
2017-09-02 04:35:01.012:INFO:osjs.Server:main: Started @845ms

Next I installed a package for Perl support of Selenium by downloading the CPAN package Selenium-Remote-Driver v1.20 (25 May 2017) and extracting the file Selenium-Remote-Driver-1.20.tar.gz and running the following commands:

~$ sudo apt-get install libwww-perl libarchive-zip-perl libfile-which-perl libio-string-perl libjson-perl libmoo-perl libxml-simple-perl libtry-tiny-perl libsub-install-perl libtest-longstring-perl libnamespace-clean-perl make
~$ cd /opt/selenium
~$ tar -xvjf Selenium-Remote-Driver-1.20.tar.gz
~$ cd Selenium-Remote-Driver-1.20
~$ perl -w Makefile.PL
~$ make

Then I wrote a test script:

#!/usr/bin/perl -w

use lib '/opt/selenium/Selenium-Remote-Driver-1.20/lib';
use Selenium::Remote::Driver;

use strict;

print( "- Connecting to Selenium Server Standalone (Java)...\n" );
my $driver = Selenium::Remote::Driver->new(
  'remote_server_addr' => "localhost",
  'port' => 4444,
  'browser_name' => 'firefox',
);

print( "- Opening Google webpage...\n" );
$driver->get( "http://www.google.com/" );

print( "- Display webpage title on console...\n" );
print $driver->get_title() . "\n";

sub find_and_visit() {
  print( "- Search for 'newspaint'...\n" );
  my $elem = $driver->find_element( "//input[\@name='q']" );
  if ( ! $elem ) {
    print( "Could not find input element with name 'q'.\n" );
    return;
  }

  $elem->send_keys( "newspaint" );
  sleep( 2 );

  my $submit = $driver->find_element(
    "//input[\@value='Google Search']"
  );
  if ( ! $submit ) {
    print( "Could not find Google Search button.\n" );
    return;
  }

  $submit->click();
  sleep( 5 );

  my @anchors = $driver->find_elements( "//a" );
  foreach my $anchor ( @anchors ) {
    my $href = $anchor->get_attribute( "href" );
    next if ( ! $href );

    if ( $href eq "https://newspaint.wordpress.com/" ) {
      print( "Clicking on newspaint blog link...\n" );
      $anchor->click();
      sleep( 5 );
      last;
    }
  }
}

find_and_visit();

print( "Quitting...\n" );
$driver->quit();

Running AUSkey on Ubuntu 16.04 in 2017

Getting the Australian government’s authentication mechanism “AUSkey” working on Linux is no easy feat thanks to its rather poor choice of Java plugins to do the job – now that browsers have discontinued support for Java plugins that becomes considerably more difficult.

I created a LXC container using instructions from this post to get a virtual machine with X11 over SSH support going – to completely insulate what I was doing from the rest of my operating system. In addition I had to sudo apt-get install xubuntu-icon-theme for completeness with the icons Firefox would expect.

Then I used the instructions at this forum post to get Firefox working with AUSkey.

  • download an “Extended Support Release” of Firefox (version 52.3.0) from https://www.mozilla.org/en-US/firefox/organizations/all/ for your OS/language
  • extract the .tar.bz2 file to /opt/
  • based on the system requirements execute the statement sudo apt-get install libgtk-3-0 libdbus-glib-1-2 libxt6
  • install the Java Runtime Environment (JRE) browser plugin using the OpenJDK package by first closing down your browser, then running sudo apt-get install icedtea-plugin, then starting the browser again and visiting about:plugins to confirm that the IcedTea-Web Plugin is in “State: Enabled” (should be the case without any additional changes)
  • now visit the activation link you were given in your activation e-mail (looks like https://auskey.abr.gov.au/Activate.aspx?code=TaRaQui37jag&verify=Y&pid=71), choose “Install AUSkey”, and then the browser may ask at the top of the screen “Allow https://auskey.abr.gov.au to run IcedTea-Web?”, press “Allow…”
  • you may get a security warning “The website’s HTTPS certificate cannot be verified. Do you want to continue?”, tick “Always trust content from this publisher” and press “Yes”
  • you’ll get a message saying “The application’s digital signature has been verified. Do you want to run the application? It will be granted unrestricted access to your computer.” Choose “Run”.
  • you’ll get a message saying “The application AUSkey from … uses resources from the following remote locations. They looks ok. Are you sure you want to run this application?”. Tick “Remember this option?” and then choose “Proceed”.
  • bizarrely, at this point, you might just have to go through a cycle of pasting your activation link into your browser and going through the motions again and again (each time ending up at the “Setting up AUSkey troubleshooting” page) until, maybe on the tenth time, it requires you to enter your activation code, confirm your details, and then choose a password.
  • It may be that when you access an AUSkey protected site you may have to reload the page a few times before it recognises the fact you have an AUSkey.

    The process seems very buggy.

OpenWRT/LEDE Buffalo WZR-HP-AG300H Getting 5GHz Radio Working

I had a problem with my Buffalo WZR-HP-AG300H, I couldn’t get the 5GHz radio wireless interface working along with the 2.5GHz radio.

In the end I used the following /etc/config/wireless configuration:

config wifi-device 'radio0'
        option type 'mac80211'
        option phy 'phy0'
        option txpower '7'
        option country 'GB'
        option hwmode '11g'
        option channel '7'
        option htmode 'HT20'

config wifi-device 'radio1'
        option type 'mac80211'
        option phy 'phy1'
        option txpower '9'
        option country 'GB'
        option hwmode '11a'
        option channel '120'
        option htmode 'HT40'

config wifi-iface
        option device 'radio0'
        option mode 'ap'
        option ssid 'my2500KHz'
        option network 'wlan'
        option encryption 'psk2'
        option key 'password'
        option wmm '0'

config wifi-iface
        option device 'radio1'
        option mode 'ap'
        option ssid 'my5GHz'
        option network 'wlan'
        option encryption 'psk2'
        option key 'password'
        option wmm '0'

This configuration seemed to work for me only after I rebooted the router.

LXC Container Reports PTY allocation request failed on channel 0 On SSH Connection

I tried upgrading my LXC from Ubuntu Trusty 14.04 by running sudo apt-get install lxc because, by default, the lxc package was not being upgraded.

But I then had problems getting consoles/terminals with my existing LXC containers.

This problem exhibits itself when attempting to ssh to a LXC container with the following message:

# ssh ubuntu@10.0.3.201
ubuntu@10.0.3.201's password: 
PTY allocation request failed on channel 0

It also exhibits itself when attempting to lxc-console a LXC container:

# sudo lxc-console -n mycontainer
lxc-console: commands.c: lxc_cmd_console: 722 Console -1 invalid, busy or all consoles busy.

(although a workaround is to connect using sudo lxc-console -n mycontainer -t 0).

The issue is that every container config file needs to have some extra lines added:

# required for lxc-console to work
lxc.tty = 4

# requires for interactive SSH to work
lxc.pts = 1024

One other issue I came across was that I would get the following errors when trying to start a container:

# sudo lxc-start -F -n mycontainer
Failed to mount cgroup at /sys/fs/cgroup/systemd: Permission denied
[!!!!!!] Failed to mount API filesystems, freezing.
Freezing execution.

This was bypassed by adding the following to the container’s config file:

# disable apparmour restrictions on container
lxc.aa_profile = unconfined

VLC on Ubuntu 16.04 with NVidia Graphics Card – Divx Video Playback Blank

I attempted to play a divx-encoded video on VLC 2.2.2 running on Ubuntu 16.04 LTS. The video was blank, although if, while the video was playing, I selected Video > Video Track > Disable, then select Video > Video Track > Track 1 from the menu it would display the current frame as a still image.

I opened up the messages window by selecting Tools > Messages from the menu. I then altered the Verbosity from 0 (errors) to 1 (warnings). Then I pressed play on the video for a short period to capture the warnings:

It displayed messages like:

avi warning: multiple riff -> OpenDML ?
avi warning: detected OpenDML file
avcodec info: Using NVIDIA VDPAU Driver Shared Library 384.59 Wed Jul 19 23:45:51 PDT 2017 for hardware decoding.
avcodec warning: cannot decode one frame (337 bytes)
core warning: VoutDisplayEvent 'pictures invalid'
core warning: VoutDisplayEvent 'pictures invalid'
avcodec warning: cannot decode one frame (337 bytes)
avcodec warning: cannot decode one frame (190 bytes)
avcodec warning: cannot decode one frame (190 bytes)

This led me to thinking the VDPAU driver was maybe failing.

A simple fix (although possibly not efficient) is select Tools > Preferences from the menu, select “Input / Codecs” from the icons at the top of the Simple Preferences dialog box, and change the first option, “Hardware-accelerated decoding” from “Automatic” to “Disable”. Then clicking “Save” at the bottom of the dialog box.

USB Tethering From CyanogenMod Android to Ubuntu Trusty 14.04

My laptop could not connect to the hotel’s WiFi but my mobile phone could. So I went into my phone settings, selected “…More”, selected “Tethering & portable hotspot”, and enabled “USB tethering”. This was while my phone was configured to be in “charge only” mode on USB.

My phone was attached to my Ubuntu computer by USB cable. And if I clicked on the Network Manager applet on my start bar (using Xubuntu) it showed me the option of “Ethernet Network (my phone model)” but it was greyed out. So Ubuntu had detected the phone had tethering turned on but wasn’t able to connect to it.

Automatic Option

Click on the Network Manager applet. At the bottom of the menu choose “Edit”.

Press “Add” to add a network connection.

Choose a connection type of “Ethernet” from the drop-down and press the “Create…” button.

Give the connection a name, e.g. “Tethering My Phone USB”. Select your USB interface from the drop down list of “Device MAC address” on the “Ethernet” tab (which is opened by default).

Choose “Save…” and the tethered network should automatically begin to work.

Manual Option (if all else fails)

The solution was to open a terminal and run:

$ sudo ifconfig usb0 up
$ sudo dhclient usb0

Now I had an IP address assigned to my usb0 interface and a default route.