Documenting Problems That Were Difficult To Find The Answer To

UK Government Gateway Supports TOTP/Google Authenticator

Well I use FreeOTP but it still works. Government Gateway wants you to install an HMRC app, but you can use Google Authenticator variants such as FreeOTP.

If you choose the HMRC app as your “backup” authentication option from Government Gateway it will show a QR code. This can be scanned directly into the FreeOTP app, if you want to know what the QR code contains then apt-get install zbar-tools, take a screen shot, and run:

me@myhost:~$ zbarimg /tmp/screenshot.png

You will want to make a safe copy of this QR image or the URL it represents in case you need to move that secret to another device for a 30 second TOTP code.

Copy Metadata From Input to Output in FFmpeg

The documentation for the -map_metadata option for FFmpeg reads:

-map_metadata[:metadata_spec_out] infile[:metadata_spec_in] (output,per-metadata)

Set metadata information of the next output file from infile. Note that those are file indices (zero-based), not filenames. Optional metadata_spec_in/out parameters specify, which metadata to copy. A metadata specifier can have the following forms:

  • g – global metadata, i.e. metadata that applies to the whole file
  • s[:stream_spec] – per-stream metadata. stream_spec is a stream specifier as described in the Stream specifiers chapter. In an input metadata specifier, the first matching stream is copied from. In an output metadata specifier, all matching streams are copied to.
  • c:chapter_index – per-chapter metadata. chapter_index is the zero-based chapter index.
  • p:program_index – per-program metadata. program_index is the zero-based program index.
  • If metadata specifier is omitted, it defaults to global.

    By default, global metadata is copied from the first input file, per-stream and per-chapter metadata is copied along with streams/chapters. These default mappings are disabled by creating any mapping of the relevant type. A negative file index can be used to create a dummy mapping that just disables automatic copying.

    For example to copy metadata from the first stream of the input file to global metadata of the output file:

    ffmpeg -i in.ogg -map_metadata 0:s:0 out.mp3

    To do the reverse, i.e. copy global metadata to all audio streams:

    ffmpeg -i in.mkv -map_metadata:s:a 0:g out.mkv

    Note that simple 0 would work as well in this example, since global metadata is assumed by default.

So let’s say we had three input streams, 0:0 (video), 0:1 (audio), and 0:2 (subtitle) and we wanted to copy the metadata for all three streams and the global metadata. We could use:

ffmpeg -i -map 0:0 -map 0:1 -map 0:2
  -map_metadata:g 0:g # take file 0's global metadata and copy to this' global
  -map_metadata:s:0 0:s:0 # take file 0's stream 0 metadata and copy to this' stream 0
  -map_metadata:s:1 0:s:1 # take file 0's stream 1 metadata and copy to this' stream 1
  -map_metadata:s:2 0:s:2 # take file 0's stream 2 metadata and copy to this' stream 2

Subtitle Codec 94213 is Not Supported in FFmpeg

I was attempting to convert a video file from an Ambarella A7L dashcam to the Matroska (mkv) file format but encountered the following error:

    Stream #0:2(eng): Subtitle: mov_text (text / 0x74786574), 0 kb/s (default)
      creation_time   : 2018-06-01 13:05:26
      handler_name    : Ambarella EXT
[matroska @ 0x4754d40] Subtitle codec 94213 is not supported.

In order to retain such subtitle codec (using -c:s copy option) better to use the mp4 file format which will allow this subtitle format.

Or, to retain the subtitle information in the Matroska container, it can be converted to the subrip format using the -c:s srt option.

Setting Up OpenVPN Server for Rooted LineageOS Phone

Installation Steps

Install OpenVPN Server

me@server:~$ sudo apt-get install openvpn
The following NEW packages will be installed:
  liblzo2-2 libpkcs11-helper1 openvpn
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 513 kB of archives.
After this operation, 1,353 kB of additional disk space will be used.
me@server:~$ sudo apt-get install openssl
The following NEW packages will be installed:
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 492 kB of archives.
After this operation, 956 kB of additional disk space will be used.

Choose a routed VPN (rather than a bridged). Use username/password authentication (still requires server certificate authority (CA) and server certificate.

Edit /etc/default/openvpn and add:

# Ensure tunnel device is available
if [ ! -d /dev/net ]; then
  mkdir /dev/net

if [ -d /dev/net ]; then
  if [ ! -e /dev/net/tun ]; then
    mknod /dev/net/tun c 10 200
    chmod 666 /dev/net/tun

Create a tunnel device:

me@server:~# openvpn --mktun --dev tun0

Create Certificate Authority

me@server:~# mkdir /etc/openvpn/ssl
me@server:~# cd /etc/openvpn/ssl
me@server:/etc/openvpn/ssl# openssl genrsa -out ca.key 2048
Generating RSA private key, 2048 bit long modulus
e is 65537 (0x10001)
me@server:/etc/openvpn/ssl# openssl req -new -x509 -key ca.key -out ca.crt

Now we have ca.crt (the certificate authority certificate) and ca.key (the key for the certificate authority).

You can get the text form of the certificate (used later for the client configuration) by issuing the command:

me@server:/etc/openvpn/ssl# openssl x509 -in ca.crt -text

… and extracting the portion between (and including) the BEGIN CERTIFICATE and END CERTIFICATE lines.

Create Server Certificate and Sign Using Certificate Authority

me@server:/etc/openvpn/ssl# openssl genrsa -out signing.key 2048
Generating RSA private key, 2048 bit long modulus
e is 65537 (0x10001)
me@server:/etc/openvpn/ssl# openssl rsa -in signing.key -pubout -out
writing RSA key
me@server:/etc/openvpn/ssl# openssl req -new -key signing.key -out request.csr

me@server:/etc/openvpn/ssl# openssl x509 -req -days 1500 -in request.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt
Signature ok
Getting CA Private Key
me@server:/etc/openvpn/ssl# openssl x509 -in server.crt -noout -text

Now we have server.crt (the server certificate).

Create Difie-Hellman Key

me@server:/etc/openvpn/ssl# openssl dhparam -out dh1024.pem 1024

Create Static Shared Key

me@server:/etc/openvpn/ssl# openvpn --genkey --secret static.key


(taken from the example at /usr/share/doc/openvpn/examples/sample-scripts/

#!/usr/bin/perl -w

use strict;

# Allowed username/password combinations
my %users = (
    'jim' => 'letmein',
    'mary' => 'password123',

# Get username/password from file

my $ARG;
if ($ARG = shift @ARGV) {
    if (!open (UPFILE, "<$ARG")) {
        print "Could not open username/password file: $ARG\n";
        exit 1;
} else {
    print "No username/password file specified on command line\n";
    exit 1;

my $username = <UPFILE>;
my $password = <UPFILE>;

if (!$username || !$password) {
    print "Username/password not found in file: $ARG\n";
    exit 1;

chomp $username;
chomp $password;

close (UPFILE);

if ( $users{$username} && ( $password eq $users{$username} ) ) {
    exit 0;
} else {
    print "Auth '$username' failed.\n";
    exit 1;

Create server.conf

port 1194
proto udp
dev tun
mssfix 576
cipher AES-256-CBC

ca /etc/openvpn/ssl/ca.crt
cert /etc/openvpn/ssl/server.crt
key /etc/openvpn/ssl/signing.key
dh /etc/openvpn/ssl/dh1024.pem

tls-auth /etc/openvpn/ssl/static.key 0

script-security 2 # necessary for auth-user-pass-verify
auth-user-pass-verify /etc/openvpn/ via-file

server # subnet for clients
keepalive 10 120
status /var/log/openvpn-status.log
log /var/log/openvpn.log
verb 3

push "redirect-gateway def1"
push "dhcp-option DNS" # cloudflare DNS

Enabling/Starting/Stopping OpenVPN

me@server:~# systemctl enable openvpn@server.service
me@server:~# systemctl start openvpn@server.service
me@server:~# systemctl stop openvpn@server.service

Create .ovpn Configuration File for OpenVPN Android Client

remote # the IP address of my OpenVPN server
port 1194
proto udp
key-direction 1
mssfix 576
cipher AES-256-CBC


-----BEGIN OpenVPN Static key V1-----
-----END OpenVPN Static key V1-----

On The Phone

Install “OpenVPN Connect – Fast & Safe SSL VPN Client” by “OpenVPN”.

Then add .ovpn file created above.

Install iptables Scripts

me@server:~# apt-get install iptables-persistent
The following NEW packages will be installed:
  iptables iptables-persistent libnfnetlink0 netfilter-persistent
0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 292 kB of archives.
After this operation, 1,804 kB of additional disk space will be used.

Edit /etc/iptables/rules.v4:

# iptables IPv4 rules for reload on start-up
# To Reload:
#   service netfilter-persistent restart
# To Test:
#   iptables-restore -t </etc/iptables/rules.v4







external program fork failed

If you see something like the following:

Wed Jun 20 03:33:31 2018 WARNING: External program may not be called unless '--script-security 2' or higher is enabled. See --help text or man page for detailed info.
Wed Jun 20 03:33:31 2018 WARNING: Failed running command (--auth-user-pass-verify): external program fork failed

… then you probably have a “script-security 1” parameter somewhere in your configuration (after any other script-security 2 directives). Grep for it and comment it out so that your “script-security 2” directive can take effect.

See Also

Changing Encrypted Partition Password on LineageOS 14.1

This was done on a ZTE Axon 7 running Android 7.1.2 (LineageOS 14.1).

I booted into the LineageOS 14.1 operating system, connected using ADB.

WARNING: the encryption password can only be changed in LineageOS itself, not TWRP, so if you get this wrong you will have an irrecoverable encrypted data partition. To protect yourself ensure you verify the new password before rebooting.

NOTE: this does not change the screen lock code, it only changes the password for decrypting the phone on boot (the first password you’re required to enter after a cold boot).

~$ adb root
~$ adb shell
axon7:/ # vdc cryptfs verifypw 1234 # wrong password
200 8421 -1
axon7:/ # vdc cryptfs verifypw 1111 # wrong password
200 8543 -1
axon7:/ # vdc cryptfs verifypw 4444 # correct password
200 8558 0
axon7:/ # vdc cryptfs changepw help
Usage: cryptfs changepw default|password|pin|pattern [[currentpasswd] newpasswd]

axon7:/ # vdc cryptfs changepw pin 4444 912541324 # change password
200 8703 0

axon7:/ # vdc cryptfs verifypw 4444 # no longer correct password
200 8761 -1
axon7:/ # vdc cryptfs verifypw 912541324 # new correct password
200 8770 0

Thanks to the information in this forum post for assistance in how to do this.

Exim: Helo Command Rejected Need Fully-Qualified Hostname

I’ve been running my Exim configuration for years when received a mail delivery failure message with the error:

504 5.5.2 <myhost>: Helo command rejected: need fully-qualified hostname

I’d not come across this issue before and my Exim set-up had been happily introducing itself to other mail servers as myhost which was the unqualified hostname of the virtual machine Exim was running on.

So – what to do?

The helo_data command, when added to a transport configuration, could be set to whatever string you want sent by Exim with the HELO message during the SMTP handshake phase of a connection.

But rather than just simply give a fixed string – I wanted my configuration to be a little more clever – and determine the MX host from the sent e-mail’s from domain.

To do this the ${lookup dnsdb} function was used.

The configuration setting I use is as follows, which I’ll break down and explain afterwards:

helo_data = ${extract{2}{,}{${extract{1}{:}{${sort {${lookup dnsdb{>:,,mx=$sender_address_domain}}} {<} {${extract{1}{,}{$item}}}}}}}{$value}{unknown}}

If I expand this out and add comments it may aid in understanding, I’ll start from the centre and work my way out.

First the sender’s domain lookup:

# Do a DNS lookup of the MX (mail exchange) records of the sender's domain
# EXAMPLE OUTPUT (for ""):
#   40,,,,,

${lookup dnsdb{>:,,mx=$sender_address_domain}}

#   records are not given in any particular order, and are colon-separated

Obviously having a list of MX hosts is too much, we just want one, and we want the highest priority host (with the smallest number). So we have to sort the list using the ${sort} function:

# Sort the list of MX records in order of priority from smallest to largest
# EXAMPLE OUTPUT (for ""):
#   10,,,,,

  { ${lookup dnsdb {>:,,mx=$domain} } }  # text to sort
  { < }                                  # comparison operator
  { ${extract {1} {,} {$item} } }        # fn to run on each item

#   each item to be sorted must have the number extracted

Okay now we have a sorted list. We just want the first entry in the list and to do this we use the ${extract} function.

# Sort the list of MX records in order of priority from smallest to largest
# and select the first record in the list
# EXAMPLE OUTPUT (for ""):
#   10,

  {1}                                       # select first field
  {:}                                       # separator is colon
  {                                         # text to extract from
      { ${lookup dnsdb {>:,,mx=$domain} } }
      { ${extract {1} {,} {$item} } }

Finally we just want the hostname which is the second field in a comma-separated list of that string.

We can provide a default value in the event nothing is extracted. Which leads us to our final formula:

# Sort the list of MX records in order of priority from smallest to largest
# and select the first record in the list, then select the second column
# which is the hostname, otherwise return "unknown"
# EXAMPLE OUTPUT (for ""):

  {2}                                           # select 2nd field
  {,}                                           # separator is comma
  {                                             # text to extract from
          { ${lookup dnsdb {>:,,mx=$domain} } }
          { ${extract {1} {,} {$item} } }
  {$value}                                      # result on success
  {unknown}                                     # result on failure

Selecting a column in tab-delimited text in Splunk

So you have a raw field that looks something like this:

field1	field2	field3	field4	field5

… where the gaps between fields are tab characters (“\t” or ASCII character 9).

You might think the way to select the 3rd field would be as follows:

index=myindex mysearchterm
  |eval fields=split(_raw, "\t" )
  |eval desiredfield=mvindex(fields,3)

But no. The Splunk split command does not recognise “\t” as the tab character.

A work around is to replace all the tabs with a unique string and split on this instead. So the above could be re-written as:

index=myindex mysearchterm
  |eval myraw=_raw
  |rex mode=sed field=myraw "s/\t/MYUNIQUESEPARATOR/g"
  |eval fields=split(myraw, "MYUNIQUESEPARATOR" )
  |eval desiredfield=mvindex(fields,3)

Xubuntu 16.04 and XFCE Not Allowing Monitors to Sleep or Power Off

The answer is that Google Chrome disables (“inhibits”) sleeping when doing WebRTC communications. Unbelievable, right? Anyway. Here’s how to discover the cause.

First kill any instance of xfce4-power-manager running.

Then in a command window run:

me@myhost:~$ xfce4-power-manager --debug --no-daemon

TRACE[xfpm-dpms.c:75] xfpm_dpms_set_timeouts(): Settings dpms: standby=1200 suspend=0 off=3600

Now, try opening a Chrome browser, say, version 60.0.3112.90 (Official Build) (64-bit). Next open up a WebRTC webpage such as and click Start on the demo page.

You will probably see the following appear in your command window:

TRACE[xfpm-inhibit.c:343] xfpm_inhibit_inhibit(): Inhibit send application name=/usr/bin/google-chrome-stable reason=WebRTC has active PeerConnections sender=:1.2387
TRACE[xfpm-inhibit.c:101] xfpm_inhibit_has_inhibit_changed(): Inhibit added

You can confirm that Chrome is using WebRTC for this page by visiting chrome://webrtc-internals/ and seeing the webpage(s) using WebRTC.

And if you close the offending tab you will see the following in your command window:

TRACE[xfpm-inhibit.c:357] xfpm_inhibit_un_inhibit(): UnHibit message received
TRACE[xfpm-inhibit.c:95] xfpm_inhibit_has_inhibit_changed(): Inhibit removed

At this point you have two choices:

  • stop using Chrome altogether
  • stop visiting that particular page that uses WebRTC in Chrome

The issue was discussed on this forum page and at this page.

BASH /usr/bin/adb No Such File or Directory

Using Xubuntu (Ubuntu) 16.04.3 I encountered a strange problem. I’d removed the Android debugging tools using the command:

my@myhost:~$ sudo apt-get remove android-tools-adb

I did this because I installed the latest platform tools manually from Google directly.

So next time I went to run adb I didn’t expect to see the following:

my@myhost:~$ adb
bash: /usr/bin/adb: No such file or directory

The answer was found in this forum post. I did not realise this but BASH actually caches executables found in the $PATH to avoid having to perform future searches.

The type command can reveal whether an executable has been stored in the cache (or “hashed”):

my@myhost:~$ type adb
adb is hashed (/usr/bin/adb)

To clear the entire cache the following command can be run:

my@myhost:~$ hash -r

.. but it might be preferable to only clear that command that has been removed from the cache:

my@myhost:~$ hash -d adb

How to Migrate WhatsApp From One Rooted Android Phone to Another Rooted Android Phone

If you are using CyanogenMod or LineageOS or similar and want to copy WhatsApp from one phone to the other here are the steps that may do what you want.

Firstly decide where you’re going to back your WhatsApp data to, in my case I wanted to write to the /storage/sdcard1/ folder on my old phone in which was inserted a MicroSD card.

Start off by backing up the app, data, and media files from the existing (old) phone:

me@server:~$ adb root
me@server:~$ adb shell
root@oldphone:/ # tar -cvpJf /storage/sdcard1/data-whatsapp.tar.xz /data/data/com.whatsapp
root@oldphone:/ # tar -cvpJf /storage/sdcard1/app-whatsapp.tar.xz /data/app/com.whatsapp-3
root@oldphone:/ # tar -cvpJf /storage/sdcard1/media-whatsapp.tar.xz /data/media/0/WhatsApp
root@oldphone:/ # exit

me@server:~$ adb pull /storage/sdcard1/data-whatsapp.tar.xz
me@server:~$ adb pull /storage/sdcard1/app-whatsapp.tar.xz
me@server:~$ adb pull /storage/sdcard1/media-whatsapp.tar.xz

An alternate method to avoid storing on the device’s flash is the following:

me@server:~$ adb root
me@server:~$ adb exec-out "tar -cvpJf - /data/data/com.whatsapp 2>/dev/null" >/tmp/data-whatsapp.tar.xz
me@server:~$ adb exec-out "tar -cvpJf - /data/app/com.whatsapp-3 2>/dev/null" >/tmp/app-whatsapp.tar.xz
me@server:~$ adb exec-out "tar -cvpJf - /data/media/0/WhatsApp 2>/dev/null" >/tmp/media-whatsapp.tar.xz

Now unplug the existing (old) phone. Plug in the new phone. Choose where you want to upload the backups to for extraction later (I chose /storage/0000-0000 as that was my MicroSD card on the new phone).

me@server:~$ adb root
me@server:~$ adb push data-whatsapp.tar.xz /storage/0000-0000/
me@server:~$ adb push app-whatsapp.tar.xz /storage/0000-0000/
me@server:~$ adb push media-whatsapp.tar.xz /storage/0000-0000/

me@server:~$ adb shell
root@newphone:/ # cd /
root@newphone:/ # tar -xvpJf /storage/0000-0000/data-whatsapp.tar.xz
root@newphone:/ # tar -xvpJf /storage/0000-0000/app-whatsapp.tar.xz
root@newphone:/ # tar -xvpJf /storage/0000-0000/media-whatsapp.tar.xz

At this point everything is where it needs to be. However there’s a big problem. Every app has a unique userid/groupid (uid/gid) – but uid=gid for each application. The application we just copied won’t be able to access its own data and user files because the files it needs will have a different uid/gid on the other phone.

First, soft restart the phone so the newly copied application can be found by Android.

If you fail to change the uid/gid of the data files for the application you will see messages like the following in logcat:

me@server:~$ adb root
me@server:~$ adb shell
root@newphone:/ # logcat |grep -i whatsapp
03-14 15:38:26.368 22689 22689 W pool-6-thread-1: type=1400 audit(0.0:1111): avc: denied { getattr } for path="/data/data/com.whatsapp/databases/_jobqueue-WhatsAppJobManager" dev="dm-0" ino=134223 scontext=u:r:untrusted_app:s0:c512,c768 tcontext=u:object_r:system_data_file:s0 tclass=file permissive=0

03-14 15:38:26.371 22689 22689 W pool-6-thread-1: type=1400 audit(0.0:1112): avc: denied { read } for name="_jobqueue-WhatsAppJobManager" dev="dm-0" ino=134223 scontext=u:r:untrusted_app:s0:c512,c768 tcontext=u:object_r:system_data_file:s0 tclass=file permissive=0

This article illustrates how to find the uid/gid of an app.

me@server:~$ adb root
me@server:~$ adb shell
root@newphone:/ # cat /data/system/packages.xml |grep '<package' |grep whatsapp
    <package name="com.whatsapp" ... userId="10053" ...

Armed with knowing what the new uid/gid should be we can go change the data from the existing uid/gid.

me@server:~$ adb root
me@server:~$ adb shell
root@newphone:/ # ls -alnd /data/data/com.whatsapp/
drwxr-x--x 10 10065 10065 4096 2018-03-14 16:11 /data/data/com.whatsapp/

root@newphone:/ # find /data/data/com.whatsapp -user 10065 -exec chown -h 10053 {} \;
root@newphone:/ # find /data/data/com.whatsapp -group 10065 -exec chgrp -h 10053 {} \;

If you’re using LineageOS 14.1 (“Nougat”) then you will also have to explicitly give your app permission to read storage (for the media that has been stored). Otherwise you might get messages like the following in logcat if you try and open a picture:

03-18 15:10:09.137 xxxxx xxxxx E BitmapFactory: Unable to decode stream: /storage/emulated/0/WhatsApp/Media/WhatsApp Images/Sent/IMG-20180122-WA0000.jpg (Permission denied)

To do this on your phone:

  • open Settings
  • find and open Apps
  • navigate to your app (e.g. “WhatsApp”) and open
  • open Permissions
  • enable Storage permission
  • (you may want to enable other permissions, too, depending on what you want your app to be able to do)

This should largely do it. Maybe you’ll need to soft reboot your phone. If WhatsApp works after this then you may want to consider going to Google Play and attempting to install WhatsApp – it should recognise that the app is already installed and offer to upgrade – so upgrade.