Documenting Problems That Were Difficult To Find The Answer To

Category Archives: Linux

Bash Script to Run Apt-Get Upgrades on All LXC Running Containers

If you have several LXC containers on your system that you want to upgrade in a loop you could use a sensible infrastructure management platform like Ansible.

Here, however, is a simple BASH script for logging into each running LXC host one at a time and letting you answer yes or no to the apt-get upgrade taking place:


# declare an array variable
declare -a LINES

# store each line about running LXC containers into array
while IFS= read -r line; do
  echo "  processing LINENUM=$LINENUM, $line"
done <<< "$(sudo lxc-ls -f)"

# iterate through each line in the array
while [[ $INDEX -lt $LINENUM ]]; do
  # squash up the spaces and delete commas from line
  CLEANLINE=`echo "${LINES[$INDEX]}" |tr -s ' ' |tr -d ,`

  # extract the hostname and ipaddress fields
  HNAME=`echo "$CLEANLINE" |cut '-d ' -f1`
  IPADDR=`echo "$CLEANLINE" |cut '-d ' -f5`

  # ssh into host if a valid IP address is present
  if [ `echo "$IPADDR" |grep -E '^([0-9]+[.]){3}([0-9]+)$'` ]; then
    echo "== $HNAME ($IPADDR) =="
    ssh -t ubuntu@$IPADDR 'echo "== $(hostname) =="; sudo apt-get update; echo "== $(hostname) =="; sudo apt-get upgrade;'

  INDEX=$((INDEX + 1))

This script makes use of BASH arrays. In addition it was discovered that attempting to change variables inside a read loop that was piped was fruitless because the pipe was run inside its own shell and thus the variable changes were lost.

Conditionally adding a directory to PATH if not already in PATH

In BASH it is possible to check that a directory is not already in the PATH environment variable before adding it.

The basic string-in-string search function was adopted from this forum post.

stringContains() { [ -z "${1##*$2*}" ]; }
stringBegins()   { [ -z "${1##$2*}"  ]; }
stringEnds()     { [ -z "${1##*$2}"  ]; }

# call as inPath "$PATH" "/my/new/path"
inPath() {
  if stringBegins   "$1" "$2:";  then return 0; fi
  if stringEnds     "$1" ":$2";  then return 0; fi
  if stringContains "$1" ":$2:"; then return 0; fi
  if [ "$1" == "$2" ];           then return 0; fi
  return 1;

If you wanted to, say, add “/home/myuser/bin” if it didn’t already exist, you could add the above functions and the following:

if inPath "$PATH" "/home/myuser/bin"; then
  : # do nothing

Converting XPS to PDF using MuPDF on Ubuntu Linux

I had a recently generated “.xps” file sent to me and needed it in PDF form for convenient viewing. The following took place on Xubuntu Trusty Tahr 14.04 LTS.

The “xpstopdf” tool failed me with a poorly converted document (just a single rectangle, no text).

Then I installed “mupdf”:

~$ sudo apt-get install mupdf
~$ mupdf mydocument.xps

This it rendered to my satisfaction to the screen. Next I wanted a PDF file!

But when I tried using “mudraw“, the associated tool installed with the “mupdf-tools” Ubuntu package, I got the following error:

~$ sudo apt-get install mupdf-tools
~$ mudraw -o /tmp/mydocument.pdf /tmp/mydocument.xps
error: empty page tree, cannot insert page
error: cannot draw '/tmp/mydocument.xps'

In the end I decided to try compiling the latest version available from MuPDF‘s download page (I downloaded “mupdf-1.9a-source.tar.gz” published 2016-04-21).

~$ wget ''
~$ tar -xvzf mupdf-1.9a-source.tar.gz
~$ cd mupdf-1.9a-source
~$ sudo apt-get install mesa-common-dev libxcursor-dev libxshmfence-dev libxext-dev libxrandr-dev libxinerama-dev
~$ nice make -j 3
~$ ./build/release/mutool draw -o /tmp/mydocument.pdf /tmp/mydocument.xps

This time my PDF was rendered without any errors!

Want to Slow Linux ZFS Flushing/Syncing to Disk Every 5 Seconds?

So you’ve got ZFS on Linux and you notice your hard drive LED light up every 5 seconds. How can you slow that down?

The zfs_txg_timeout parameter is described as:

~# modinfo zfs |grep zfs_txg_timeout
parm:           zfs_txg_timeout:Max seconds worth of delta per txg (int)

You can check the current value of this parameter:

~# cat /sys/module/zfs/parameters/zfs_txg_timeout

To verify this is the parameter you want to change, set this to 10 and count the number of seconds between the hard drive light coming on:

~# echo 10 >/sys/module/zfs/parameters/zfs_txg_timeout

Once you know this is the parameter causing the hard drive activity you can tell the ZFS module the default you want it to start with on next boot by editing /etc/modprobe.d/zfs.conf and adding a line similar to the following:

options zfs zfs_txg_timeout=30

Take care when changing this number as it may result in a higher probability of lost data in the event of a power outage.

Disabling the Alarm/Beep on APC UPS Back-UPS 1400 on Ubuntu 14.04

Note this was also done on a APC UPS Back-UPS 700 as well as a APC UPS Back-UPS 1400.

So you don’t want to be interrupted by long piercing beeps from your APC-branded UPS every minute when the power goes out. That’s fair. Perhaps it is night time and the power goes out occasionally. You just want your server to keep humming as long as it can. If the power stays off for two hours straight, then fine, it can die, but if the power recovers in that time you just don’t want to know about it. You bought your un-interruptable power supply (UPS) to increase reliability in, perhaps, an area with an unreliable electricity supply.

Note that for some people the alarm is important to them: they must take action in the event of a power outage. This guide is not for you.

If you have Ubuntu Linux 14.04 Trusty Tahr then it is actually very easy to disable the alarm on your UPS.

Attach USB cable from computer to UPS

Attach USB cable from computer to UPS

First connect your UPC to your PC (or laptop) via USB cable.

Connect the USB cable to the back of the UPS

Connect the USB cable to the back of the UPS

Confirm that it is found by running lsusb:

user@host:~$ lsusb
Bus 003 Device 013: ID 051d:0002 American Power Conversion Uninterruptible Power Supply

Next install apcupsd via apt-get:

user@host:~$ sudo apt-get install apcupsd
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
Suggested packages:
The following NEW packages will be installed:
  apcupsd apcupsd-doc
0 upgraded, 2 newly installed, 0 to remove and 12 not upgraded.
Need to get 891 kB of archives.
After this operation, 2,378 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 trusty/universe apcupsd-doc all 3.14.10-2build1 [608 kB]
Get:2 trusty/universe apcupsd amd64 3.14.10-2build1 [282 kB]
Fetched 891 kB in 0s (4,839 kB/s)
Selecting previously unselected package apcupsd-doc.
(Reading database ... 378817 files and directories currently installed.)
Preparing to unpack .../apcupsd-doc_3.14.10-2build1_all.deb ...
Unpacking apcupsd-doc (3.14.10-2build1) ...
Selecting previously unselected package apcupsd.
Preparing to unpack .../apcupsd_3.14.10-2build1_amd64.deb ...
Unpacking apcupsd (3.14.10-2build1) ...
Processing triggers for doc-base (0.10.5) ...
Processing 1 added doc-base file...
Processing triggers for man-db ( ...
Processing triggers for ureadahead (0.100.0-16) ...
Setting up apcupsd-doc (3.14.10-2build1) ...
Setting up apcupsd (3.14.10-2build1) ...
update-rc.d: warning:  start runlevel arguments (1 2 3 4 5) do not match apcupsd Default-Start values (2 3 4 5)
update-rc.d: warning:  stop runlevel arguments (0 6) do not match apcupsd Default-Stop values (0 1 6)
Please check your configuration ISCONFIGURED in /etc/default/apcupsd
Processing triggers for ureadahead (0.100.0-16) ...

Now you’ll have to sudo vi /etc/apcupsd/apcupsd.conf and comment out the following lines and replace them with the following:


#UPSTYPE apcsmart
#DEVICE /dev/ttyS0

Now you’re ready to run apctest which is documented on the apcupsd page:

user@host:~$ sudo apctest
2016-07-16 18:48:25 apctest 3.14.10 (13 September 2011) debian
Checking configuration ...
Attached to driver: usb
sharenet.type = Network & ShareUPS Disabled
cable.type = USB Cable
mode.type = USB UPS Driver
Setting up the port ...
Doing prep_device() ...

You are using a USB cable type, so I'm entering USB test mode
Hello, this is the apcupsd Cable Test program.
This part of apctest is for testing USB UPSes.

Getting UPS capabilities...SUCCESS

Please select the function you want to perform.

1)  Test kill UPS power
2)  Perform self-test
3)  Read last self-test result
4)  View/Change battery date
5)  View manufacturing date
6)  View/Change alarm behavior
7)  View/Change sensitivity
8)  View/Change low transfer voltage
9)  View/Change high transfer voltage
10) Perform battery calibration
11) Test alarm
12) View/Change self-test interval
 Q) Quit

Select function number: 6

Current alarm setting: ENABLED
 E to Enable alarms
 D to Disable alarms
 Q to Quit with no changes
Your choice: Select function: D

New alarm setting: DISABLED

1)  Test kill UPS power
2)  Perform self-test
3)  Read last self-test result
4)  View/Change battery date
5)  View manufacturing date
6)  View/Change alarm behavior
7)  View/Change sensitivity
8)  View/Change low transfer voltage
9)  View/Change high transfer voltage
10) Perform battery calibration
11) Test alarm
12) View/Change self-test interval
 Q) Quit

Select function number: Q


Fairly straightforward.

Note you may want to uninstall the apcupsd package from your system now.

user@host:~$ sudo apt-get remove apcupsd
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
Use 'apt-get autoremove' to remove them.
The following packages will be REMOVED:
0 upgraded, 0 newly installed, 1 to remove and 12 not upgraded.
After this operation, 749 kB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 379108 files and directories currently installed.)
Removing apcupsd (3.14.10-2build1) ...
Please check your configuration ISCONFIGURED in /etc/default/apcupsd
Please check your configuration ISCONFIGURED in /etc/default/apcupsd
Processing triggers for ureadahead (0.100.0-16) ...
Processing triggers for man-db ( ...

Lenovo TS-140 Ethernet Card Halt

I had my Ethernet interface effectively just die after 33 days uptime in Linux and running continuously for many, many months.

What was particularly bizarre was that I had an identical Lenovo TS-140 running beside it attached to the same Ethernet switch – that was running a GUI and it completely froze at this point. At least with the first console server I was able to access it and make a copy of the logs for later analysis after rebooting.

From /var/log/dmesg I had the following:

[2931914.307645] ------------[ cut here ]------------
[2931914.307663] WARNING: CPU: 0 PID: 0 at /build/linux-Mxzr_W/linux-3.13.0/net/sched/sch_generic.c:264 dev_watchdog+0x276/0x280()
[2931914.307668] NETDEV WATCHDOG: eth0 (e1000e): transmit queue 0 timed out
[2931914.307671] Modules linked in: btrfs raid6_pq xor ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs libcrc32c nf_conntrack_netlink nfnetlink_queue nfnetlink_log nfnetlink bluetooth xt_LOG xt_limit ts_bm xt_comment xt_string xt_conntrack xt_HL xt_nat veth xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_tcpudp bridge stp llc iptable_filter ip_tables x_tables x86_pkg_temp_thermal intel_powerclamp coretemp snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec kvm snd_hwdep snd_pcm mei_me snd_page_alloc serio_raw snd_timer mei snd shpchp lpc_ich soundcore mac_hid nf_nat_sip nf_conntrack_sip nf_nat nf_conntrack zfs(POX) zunicode(POX) zcommon(POX) znvpair(POX) spl(OX) zavl(POX) hid_generic usbhid hid dm_crypt usb_storage crct10dif_pclmul crc32_pclmul i915 aesni_intel aes_x86_64 lrw e1000e gf128mul psmouse glue_helper ablk_helper i2c_algo_bit cryptd ptp drm_kms_helper pps_core drm ahci libahci video wmi
[2931914.307793] CPU: 0 PID: 0 Comm: swapper/0 Tainted: P           OX 3.13.0-85-generic #129-Ubuntu
[2931914.307797] Hardware name: LENOVO ThinkServer TS140/ThinkServer TS140, BIOS FBKT82AUS 04/02/2014
[2931914.307800]  0000000000000000 ffff88051ea03d98 ffffffff8172b6a7 ffff88051ea03de0
[2931914.307808]  0000000000000009 ffff88051ea03dd0 ffffffff810699cd 0000000000000000
[2931914.307814]  ffff8800361a0000 ffff8804fc73e880 0000000000000001 0000000000000000
[2931914.307820] Call Trace:
[2931914.307824]  <IRQ>  [<ffffffff8172b6a7>] dump_stack+0x64/0x82
[2931914.307845]  [<ffffffff810699cd>] warn_slowpath_common+0x7d/0xa0
[2931914.307851]  [<ffffffff81069a3c>] warn_slowpath_fmt+0x4c/0x50
[2931914.307863]  [<ffffffff8164ef86>] dev_watchdog+0x276/0x280
[2931914.307870]  [<ffffffff8164ed10>] ? dev_graft_qdisc+0x80/0x80
[2931914.307878]  [<ffffffff81076956>] call_timer_fn+0x36/0x150
[2931914.307884]  [<ffffffff8164ed10>] ? dev_graft_qdisc+0x80/0x80
[2931914.307892]  [<ffffffff8107798f>] run_timer_softirq+0x21f/0x310
[2931914.307900]  [<ffffffff8106f00c>] __do_softirq+0xfc/0x310
[2931914.307908]  [<ffffffff8106f595>] irq_exit+0x105/0x110
[2931914.307919]  [<ffffffff8173e755>] smp_apic_timer_interrupt+0x45/0x60
[2931914.307926]  [<ffffffff8173d0dd>] apic_timer_interrupt+0x6d/0x80
[2931914.307929]  <EOI>  [<ffffffff815dc5e2>] ? cpuidle_enter_state+0x52/0xc0
[2931914.307946]  [<ffffffff815dc5d8>] ? cpuidle_enter_state+0x48/0xc0
[2931914.307954]  [<ffffffff815dc72c>] cpuidle_idle_call+0xdc/0x220
[2931914.307963]  [<ffffffff8101e4de>] arch_cpu_idle+0xe/0x30
[2931914.307971]  [<ffffffff810c1eb5>] cpu_startup_entry+0xc5/0x2b0
[2931914.307980]  [<ffffffff81719777>] rest_init+0x77/0x80
[2931914.307990]  [<ffffffff81d34f70>] start_kernel+0x438/0x443
[2931914.307998]  [<ffffffff81d34941>] ? repair_env_string+0x5c/0x5c
[2931914.308006]  [<ffffffff81d34120>] ? early_idt_handler_array+0x120/0x120
[2931914.308014]  [<ffffffff81d345ee>] x86_64_start_reservations+0x2a/0x2c
[2931914.308021]  [<ffffffff81d34733>] x86_64_start_kernel+0x143/0x152
[2931914.308026] ---[ end trace 7c85c7d5a955f5e4 ]---
[2931914.308063] e1000e 0000:00:19.0 eth0: Reset adapter unexpectedly
[2931918.468625] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[2931938.327046] e1000e 0000:00:19.0 eth0: Reset adapter unexpectedly
[2931942.327873] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx

Almost exactly the same set of messages described in this bug thread (but with no solution at time of writing).

A few solutions were proposed. This one proposed disabling TSO, GSO and GRO using ethtool:

ethtool -K eth0 gso off gro off tso off

But I decided to try turning active power state management off in the kernel after seeing the following in /var/log/dmesg:

[    0.114082] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
[    0.147241] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
[    0.147621] acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration

So I followed the recommendation in this post by adding pcie_aspm=off to /etc/default/grub as follows:

GRUB_CMDLINE_LINUX_DEFAULT="pcie_aspm=off nosplash"

… and then re-ran sudo update-grub.

Note that I cannot tell you if this definitively works. This Ethernet crash only happened once in the 14 months I’ve had the server. Hopefully it won’t happen again.

Audacious on Ubuntu Doesn’t Play the Next Song in the Playlist

Found that Audacious only plays one song and then stops?

According to this bug report Audacious disables playlist advance if you press the ctrl+N toggle.

Sometimes I think my Firefox session has the focus and I press ctrl+N for a new window when, in fact, Audacious was the last window I had focus in – which trips up this issue.

You can edit the ~/.config/audacious/config file and change:




Xubuntu and System Freeze/Hang After Entering Password On Wake From Suspend

On an Asus S400C running Xubuntu 14.04 Trusty Tahr I was having a problem after waking from suspend. The computer would wake up, I would tap a key to bring up the display from blank, and it would offer me to type in my session password.

If I typed a session password wrong it would tell me, but if I typed the session password right the system would freeze (the fan would turn on indicating maybe heavy CPU use) and no key combination would do anything (including ctrl-alt-F2 to switch to another terminal).

Well, I don’t know why this was happening, but it seems to have been down to the light-locker package. So I removed it.

$ sudo apt-get remove light-locker light-locker-settings
$ sudo apt-get install gnome-screensaver

The computer now allows me to log in after suspend but I have to type in the session password twice.

Cloned My Drive, Now Grub Cannot Find LS Command

You’re sitting looking at a basic grub prompt. You type in “ls” in the hope you see something:

grub> ls

Error 27: Unrecognized command


Well, I don’t know how you can fix this (get the ls command back). But I did figure out how to boot my cloned drive. And the tab key is your friend. If you type “root” you can find out information about your current drive, but the key is to hit tab twice after entering slash after a command such as “kernel”.

You hit the following commands:

grub> kernel /vm[tab]linuz-3.13.0-[tab][tab]
vmlinuz-3.13.0-55-generic  vmlinuz-3.13.0-86-generic
grub> kernel /vmlinuz-3.13.0-86-generic
grub> initrd /initrd.img-3.13.0-86-generic
grub> boot

Now if the boot seems to go successfully enough but you’re using crypto/encryption/dm-crypt/LUKS – then you might get a BusyBox prompt after you successfully enter your crypt partition password:


If this happens then you need to specify, to grub, on the kernel command line, where your root is. In my case root is found on the /dev/mapper/crypt device after decryption which must be subsequently mounted by Linux.

So to boot successfully I reboot and enter the following into grub:

grub> kernel /vmlinuz-3.13.0-86-generic root=/dev/mapper/crypt
grub> initrd /initrd.img-3.13.0-86-generic
grub> boot

Restoring SELinux Labels After Restoring From Data Backup To Android

So I made a major error of judgement and tried to put Marshmallow CyanogenMod 13 onto my LG G3 (d855). Sure, for a while it raced along nice and fast, but in the end turned into an utter disaster when, overnight, it decided to go into an endless CPU loop resulting in rebooting every 10 minutes or so back into another CPU loop. Making the phone very hot and essentially useless. Well enough of that long story. I tried to downgrade using a backup I made earlier back into Lollypop CyanogenMod 12.1.

But K-9 mail was not starting.

I ran:

# adb root
# adb shell
root@d855:/ # logcat |grep com.fsck.k9
W/com.fsck.k9(13997): type=1400 audit(0.0:2058): avc: denied { write } for name="preferences_storage" dev="mmcblk0p43" ino=530653 scontext=u:r:untrusted_app:s0 tcontext=u:object_r:system_data_file:s0 tclass=file
E/SQLiteDatabase(13997): Failed to open database '/data/data/com.fsck.k9/databases/preferences_storage'.
E/SQLiteDatabase(13997): 	at com.fsck.k9.preferences.Storage.openDB(
E/SQLiteDatabase(13997): 	at com.fsck.k9.preferences.Storage.loadValues(
E/SQLiteDatabase(13997): 	at com.fsck.k9.preferences.Storage.(
E/SQLiteDatabase(13997): 	at com.fsck.k9.preferences.Storage.getStorage(
E/SQLiteDatabase(13997): 	at com.fsck.k9.Preferences.(
E/SQLiteDatabase(13997): 	at com.fsck.k9.Preferences.getPreferences(
E/SQLiteDatabase(13997): 	at com.fsck.k9.K9.onCreate(
E/AndroidRuntime(13997): Process: com.fsck.k9, PID: 13997
E/AndroidRuntime(13997): java.lang.RuntimeException: Unable to create application com.fsck.k9.K9: android.database.sqlite.SQLiteException: not an error (code 0): Could not open the database in read/write mode.

Turns out that first line from the log there is SELinux denying write access to the file.

I discovered I needed to run the restorecon command which would apply the necessary labellings for the files in /data/data using the rules in /file_contexts.

# adb root
# adb shell
root@d855:/ # restorecon -Rv /data/data/com.fsck.k9

But there is another reason why an application may be unable to access its database. Wrong user. What user does an application run as? That is found in /data/system/packages.list:

root@d855:/ # grep com.fsck.k9 /data/system/packages.list
com.fsck.k9 10076 0 /data/data/com.fsck.k9 default 3003,1028,1015

So with that information I could set the owner of the data files to that required by the application:

root@d855:/ # chown -R 10076 /data/data/com.fsck.k9