Ubuntu Xenial Booting and Seeing Cryptsetup: LVM Is Not Available

I was in the process of replacing a hard drive on my root pool (rpool) ZFS zpool. I had taken out one of the hard drives my system relied upon when booting, it was a drive for which cryptsetup expected me to type a password when booting.

For several minutes I saw the following messages during boot (and finally a BusyBox shell):

cryptsetup: lvm is not available
cryptsetup: lvm is not available
cryptsetup: lvm is not available
cryptsetup: lvm is not available
cryptsetup: lvm is not available
cryptsetup: lvm is not available
cryptsetup: lvm is not available
cryptsetup: lvm is not available
cryptsetup: lvm is not available
cryptsetup: lvm is not available
cryptsetup: lvm is not available
cryptsetup: lvm is not available
cryptsetup: lvm is not available
cryptsetup: lvm is not available
cryptsetup: lvm is not available
cryptsetup: lvm is not available
cryptsetup: lvm is not available
done.
  ALERT! /dev/disk/by-uuid/... does not exist,
        Check cryptopts=source= bootarg: cat /proc/cmdline
        or missing modules, devices: cat /proc/modules; ls /dev
-r Dropping to a shell. Will skip /dev/disk/by-uuid/... if you can't fix.
/scripts/panic/plymouth: line 18: /bin/plymouth: not found

BusyBox v1.22.1 (Ubuntu 1:1.22.0-15ubuntu1.4) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs)

To boot into my rpool I did the following:

(initramfs) # figure out the existing device to decrypt
(initramfs) cat /conf/conf.d/cryptroot
target=crypt1,source=UUID=...,key=none,rootdev
target=crypt2,source=UUID=...,key=none,rootdev

(initramfs) # find out what device has the UUID I'm looking for
(initramfs) blkid
/dev/sda: UUID="..." TYPE="crypto_LUKS"
/dev/sdb: UUID="..." TYPE="crypto_LUKS"

(initramfs) # open the encrypted disk that should still be working
(initramfs) cryptsetup luksOpen /dev/sdb crypt2
Enter passphrase for /dev/sdb: ********

(initramfs) # enable ZFS, and import the root rpool
(initramfs) /sbin/modprobe zfs
(initramfs) zpool import
  pool: rpool
    id: ...
 state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
       devices and try again.
  see: http://zfsonlinux.org/msg/ZFS-8000-6X
config:
       rpool      UNAVAIL  missing device
         mirror-0 DEGRADED
           ...    UNAVAIL
           crypt2 ONLINE

(initramfs) zpool import -f -R / -m -N rpool
(initramfs) exit

At this point I was offered to unlock my other disks as per the usual boot sequence and the system booted into a degraded root zpool successfully.

Using Dnsmasq as Caching Nameserver on Ubuntu Xenial

Setting up dnsmasq as a caching nameserver locally on Ubuntu Xenial (16.04.6 LTS) can speed up the Internet experience as, by default, Linux queries a nameserver every time a domain name is connected to – and this usually involves the round-trip time to the configured nameserver. It is so much quicker to have a response locally if it is cached.

First, install dnsmasq:

$ sudo apt-get install dnsmasq
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  dns-root-data dnsmasq-base libnetfilter-conntrack3
The following NEW packages will be installed:
  dns-root-data dnsmasq dnsmasq-base libnetfilter-conntrack3
0 upgraded, 4 newly installed, 0 to remove and 7 not upgraded.
Need to get 353 kB of archives.
After this operation, 972 kB of additional disk space will be used.
Do you want to continue? [Y/n] y

You may also want the lookup tool “dig” to test the dnsmasq install:

$ sudo apt-get install dnsutils
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  bind9-host libbind9-140 libdns162 libisc160 libisccc140 libisccfg140
  liblwres141
Suggested packages:
  rblcheck
The following NEW packages will be installed:
  bind9-host dnsutils libbind9-140 libdns162 libisc160 libisccc140
  libisccfg140 liblwres141
0 upgraded, 8 newly installed, 0 to remove and 7 not upgraded.
Need to get 1,338 kB of archives.
After this operation, 6,059 kB of additional disk space will be used.
Do you want to continue? [Y/n] y

Once dnsmasq has been installed create a custom cache configuration in the /etc/dnsmasq.d/ subdirectory:

# http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html

# Listen on the given IP address(es).
listen-address=127.0.0.1,::1

# Listen on <port> instead of the standard DNS port (53).
port=53

# Force dnsmasq to really bind only the interfaces it is listening on.
bind-interfaces

# Log the results of DNS queries handled by dnsmasq.
# Enable a full cache dump on receipt of SIGUSR1.
# If the argument "extra" is supplied, ie --log-queries=extra then the
# log has extra information at the start of each line. This consists of
# a serial number which ties together the log lines associated with an
# individual query, and the IP address of the requestor.
log-queries

# If the facility given contains at least one '/' character, it is
# taken to be a filename, and dnsmasq logs to the given file, instead
# of syslog. If the facility is '-' then dnsmasq logs to stderr.
log-facility=/var/log/dnsmasq.log

# Tells dnsmasq to never forward A or AAAA queries for plain names,
# without dots or domain parts, to upstream nameservers. If the name is
# not known from /etc/hosts or DHCP then a "not found" answer is
# returned.
domain-needed

# All reverse lookups for private IP ranges (ie 192.168.x.x, etc) which
# are not found in /etc/hosts or the DHCP leases file are answered with
# "no such domain" rather than being forwarded upstream.
bogus-priv

# Don't read the hostnames in /etc/hosts.
#no-hosts

# Set the maximum number of concurrent DNS queries.
dns-forward-max=150

# Set the size of dnsmasq's cache.
# Setting the cache size to zero disables caching.
cache-size=250

# Disable negative caching.
#no-negcache

# This option gives a default value for time-to-live (in seconds) which
# dnsmasq uses to cache negative replies even in the absence of an SOA
# record.
neg-ttl=120

# Read the IP addresses of the upstream nameservers from <file>,
# instead of /etc/resolv.conf.
resolv-file=/etc/resolv.dnsmasq

# Don't poll /etc/resolv.conf for changes.
#no-poll

# Specify time-to-live for information from /etc/hosts.
local-ttl=15

# Set a maximum TTL value for entries in the cache.
max-cache-ttl=300

# Setting this flag forces dnsmasq to try each query with each server
# strictly in the order they appear in /etc/resolv.conf
#strict-order

Next, create a custom resolv.conf file for dnsmasq to use:

# Google secondary DNS
nameserver 8.8.4.4

# Cloudflare secondary DNS
nameserver 1.0.0.1

We’re not finished! If we want to use our own resolv.conf file then we have to modify the defaults file for dnsmasq:

IGNORE_RESOLVCONF=yes

Alright, now we’re ready to start dnsmasq. Well it might already be running:

$ sudo systemctl status dnsmasq
* dnsmasq.service - dnsmasq - A lightweight DHCP and caching DNS server
   Loaded: loaded (/lib/systemd/system/dnsmasq.service; enabled; vendor preset: enabled)
  Drop-In: /run/systemd/generator/dnsmasq.service.d
           `-50-dnsmasq-$named.conf, 50-insserv.conf-$named.conf
   Active: active (running) since Mon 2019-08-26 06:13:09 UTC; 1h 4min ago
  Process: 7170 ExecStop=/etc/init.d/dnsmasq systemd-stop-resolvconf (code=exited, status=0/SUCCESS)
  Process: 7224 ExecStartPost=/etc/init.d/dnsmasq systemd-start-resolvconf (code=exited, status=0/SUCCESS)
  Process: 7212 ExecStart=/etc/init.d/dnsmasq systemd-exec (code=exited, status=0/SUCCESS)
  Process: 7209 ExecStartPre=/usr/sbin/dnsmasq --test (code=exited, status=0/SUCCESS)
 Main PID: 7223 (dnsmasq)
   CGroup: /system.slice/dnsmasq.service
           `-7223 /usr/sbin/dnsmasq -x /var/run/dnsmasq/dnsmasq.pid -u dnsmasq -7 /etc/dnsmasq.d,.dpkg-dist,.dpkg-old,.dpkg-new --local-service

Aug 26 06:13:08 myhost systemd[1]: Starting dnsmasq - A lightweight DHCP and caching DNS server...
Aug 26 06:13:08 myhost dnsmasq[7209]: dnsmasq: syntax check OK.
Aug 26 06:13:09 myhost systemd[1]: Started dnsmasq - A lightweight DHCP and caching DNS server.

Either way, start or restart the dnsmasq daemon:

$ sudo systemctl stop dnsmasq
$ sudo systemctl start dnsmasq

We can view the dnsmasq log:

$ cat /var/log/dnsmasq.log
Aug 26 07:21:10 dnsmasq[8118]: started, version 2.75 cachesize 250
Aug 26 07:21:10 dnsmasq[8118]: compile time options: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth DNSSEC loop-detect inotify
Aug 26 07:21:10 dnsmasq[8118]: reading /etc/resolv.dnsmasq
Aug 26 07:21:10 dnsmasq[8118]: using nameserver 8.8.4.4#53
Aug 26 07:21:10 dnsmasq[8118]: using nameserver 1.0.0.1#53
Aug 26 07:21:10 dnsmasq[8118]: read /etc/hosts - 4 addresses

How about testing with a looking?

$ dig @localhost -p 53 www.wikipedia.org
; <> DiG 9.10.3-P4-Ubuntu <> @localhost -p 53 www.wikipedia.org
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29520
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1452
;; QUESTION SECTION:
;www.wikipedia.org.             IN      A

;; ANSWER SECTION:
www.wikipedia.org.      9864    IN      CNAME   dyna.wikimedia.org.
dyna.wikimedia.org.     266     IN      A       91.198.174.192

;; Query time: 7 msec
;; SERVER: ::1#53(::1)
;; WHEN: Mon Aug 26 07:24:27 UTC 2019
;; MSG SIZE  rcvd: 91

$ tail /var/log/dnsmasq.log
Aug 26 07:24:27 dnsmasq[8118]: query[A] www.wikipedia.org from ::1
Aug 26 07:24:27 dnsmasq[8118]: forwarded www.wikipedia.org to 8.8.4.4
Aug 26 07:24:27 dnsmasq[8118]: forwarded www.wikipedia.org to 1.0.0.1
Aug 26 07:24:27 dnsmasq[8118]: reply www.wikipedia.org is 
Aug 26 07:24:27 dnsmasq[8118]: reply dyna.wikimedia.org is 91.198.174.192

Looks to be working. Interestingly, by default, dnsmasq queries all name servers simultaneously, at first, to determine which is responding the quickest, and will then tend to just query that one for a while, until it tries all the name servers again.

A few more things to finish up. Let’s tell Linux to use localhost to do DNS lookups in future:

# The primary network interface
auto ens5
iface ens5 inet static
        # dns-* options are implemented by the resolvconf package, if installed
        dns-nameservers 127.0.0.1

And to make the change by hand until the next reboot you can edit /etc/resolv.conf directly to use 127.0.0.1 as the only nameserver.

You may want to also add a logrotate configuration:

/var/log/dnsmasq.log {
        size 20M
        rotate 50
        compress
        missingok
        notifempty
        create 644 dnsmasq root
        prerotate
                if systemctl status dnsmasq >/dev/null; then
                        systemctl stop dnsmasq >/dev/null;
                        touch /tmp/logrotate-dnsmasq-stopped.tmp;
                fi
        endscript
        postrotate
                if [ -e /tmp/logrotate-dnsmasq-stopped.tmp ]; then
                        rm /tmp/logrotate-dnsmasq-stopped.tmp;
                        systemctl start dnsmasq >/dev/null;
                fi
        endscript
}

Panasonic KX-UT113 VoIP Phone Displays Auto Ans

If you see the display showing “AUTO ANS” on your Panasonic KX-UT113 VoIP phone then it means that incoming calls will be picked up after a number of rings with the speakerphone enabled.

Panasonic KX-UT113 Displaying AUTO ANS

 

To disable (or enable) this function while the phone is on-the-hook (hung-up) press the button that doubles as the mute as well as auto-answer button.

Panasonic KX-UT113 Auto Answer Button

If With Multiple Conditions in Splunk Eval

A common task one desires to do with the if() command in Splunk is to perform multiple tests. Unfortunately this is very poorly documented on the Splunk website.

You can use the AND and OR keywords (as opposed to the && or || you might have expected).

e.g.:

index=os sourcetype=ps host=myhost01
  |rex field=CPUTIME "^((?\d+)-)?(?\d\d):(?\d\d):(?\d\d)$"
  |eval cpusec=if( isnum(c_days), c_days, 0)*24*3600 + (c_hours*3600) + (c_mins*60) + c_secs
  |eval aa_inrange=if( (cpusec>1000) AND (cpusec<2000), 1, 0 )
  |search aa_inrange=1

Converting Raw Bytes Into Int32, Int16, UInt32, UInt16

If you find yourself processing raw bytes and want to decode signed and unsigned integers then the following routines will help.

Network-Ordered (Big Endian) Encoded Numbers

Note that the following code is for network-ordered (big endian) numbers.

In order to verify the routines coded are correct we can specify tests (test-driven development) to check they do what we expect – particularly at the boundary conditions:

use Test::More;

is( str_to_uint32( "\x00\x00\x00\x01" ),          1, "uint32 1" );
is( str_to_uint32( "\xFF\xFF\xFF\xFF" ), 4294967295, "uint32 4294967295" );
is( str_to_uint16( "\x00\x01" ),                  1, "uint16 1" );
is( str_to_uint16( "\xFF\xFF" ),              65535, "uint16 65535" );
is( str_to_int32( "\x00\x00\x00\x01" ),           1, "int32 1" );
is( str_to_int32( "\xFF\xFF\xFF\xFF" ),          -1, "int32 -1" );
is( str_to_int32( "\x7F\xFF\xFF\xFF" ),  2147483647, "int32 2147483647" );
is( str_to_int32( "\x80\x00\x00\x00" ), -2147483648, "int32 -2147483648" );
is( str_to_int16( "\x00\x01" ),                   1, "int16 1" );
is( str_to_int16( "\xFF\xFF" ),                  -1, "int16 -1" );
is( str_to_int16( "\x7F\xFF" ),               32767, "int16 32767" );
is( str_to_int16( "\x80\x00" ),              -32768, "int16 -32768" );

Decoding network-ordered unsigned integers is very easy in Perl with the unpack() function. For documentation see the pack() function and perlpacktut tutorial pages.

sub str_to_uint32 {
  return unpack( "N", $_[0] ); # "N" for "Network" order (big-endian)
}

sub str_to_uint16 {
  return unpack( "n", $_[0] ); # "n" for "Network" order (big-endian)
}

Decoding network-ordered signed integers is slightly more difficult as the pack() function does not appear to directly support such encodings. Instead we can convert a decoded unsigned representation and, if negative, apply twos’ compliment. From the Wikipedia page:

Conveniently, another way of finding the two’s complement of a number is to take its ones’ complement and add one.

The corollary is that to decode a negative number we take the ones’ compliment and subtract one. We use the pragma use integer; to ensure that bit flipping is done on integral values, not floating-point.

sub str_to_int32 {
  use integer;

  my $num = str_to_uint32( $_[0] );
  if ( $num & 0x80000000 ) {
    $num = 0 - ( ( ( ~ $num ) & 0xFFFFFFFF ) + 1 );
  }

  return $num;
}

sub str_to_int16 {
  use integer;

  my $num = str_to_uint16( $_[0] );
  if ( $num & 0x8000 ) {
    $num = 0 - ( ( ( ~ $num ) & 0xFFFF ) + 1 );
  }

  return $num;
}

Upon running our tests we get the following output:

me@myhost:~ $ perl -w test_functions.pl
ok 1 - uint32 1
ok 2 - uint32 4294967295
ok 3 - uint16 1
ok 4 - uint16 65535
ok 5 - int32 1
ok 6 - int32 -1
ok 7 - int32 2147483647
ok 8 - int32 -2147483648
ok 9 - int16 1
ok 10 - int16 -1
ok 11 - int16 32767
ok 12 - int16 -32768

Little Endian Encoded Numbers

The same routines can be used with a minor differences. Firstly setting up our tests:

use Test::More;

is( str_to_uint32( "\x01\x00\x00\x00" ),          1, "uint32 1" );
is( str_to_uint32( "\xFF\xFF\xFF\xFF" ), 4294967295, "uint32 4294967295" );
is( str_to_uint16( "\x01\x00" ),                  1, "uint16 1" );
is( str_to_uint16( "\xFF\xFF" ),              65535, "uint16 65535" );
is( str_to_int32( "\x01\x00\x00\x00" ),           1, "int32 1" );
is( str_to_int32( "\xFF\xFF\xFF\xFF" ),          -1, "int32 -1" );
is( str_to_int32( "\xFF\xFF\xFF\x7F" ),  2147483647, "int32 2147483647" );
is( str_to_int32( "\x00\x00\x00\x80" ), -2147483648, "int32 -2147483648" );
is( str_to_int16( "\x01\x00" ),                   1, "int16 1" );
is( str_to_int16( "\xFF\xFF" ),                  -1, "int16 -1" );
is( str_to_int16( "\xFF\x7F" ),               32767, "int16 32767" );
is( str_to_int16( "\x00\x80" ),              -32768, "int16 -32768" );

All we have to do is simply change two functions:

sub str_to_uint32 {
  return unpack( "V", $_[0] ); # "V" for "VAX" order (little-endian)
}

sub str_to_uint16 {
  return unpack( "v", $_[0] ); # "v" for "VAX" order (little-endian)
}

Upon running our tests we get the following output:

me@myhost:~ $ perl -w test_functions.pl
ok 1 - uint32 1
ok 2 - uint32 4294967295
ok 3 - uint16 1
ok 4 - uint16 65535
ok 5 - int32 1
ok 6 - int32 -1
ok 7 - int32 2147483647
ok 8 - int32 -2147483648
ok 9 - int16 1
ok 10 - int16 -1
ok 11 - int16 32767
ok 12 - int16 -32768

Debugging LWP::UserAgent in Perl 5

Over the years the way of dumping the sent and received headers of HTTP requests changed. It used to be the following line added to your script would result in headers being dumped:

use LWP::Debug qw(+);

But more recently the approach has been to add handlers to the user agent, e.g.:

$ua->add_handler(
  "request_send",
  sub {
    my $msg = shift;              # HTTP::Message
    $msg->dump( maxlength => 0 ); # dump all/everything
    return;
  }
);

$ua->add_handler(
  "response_done",
  sub {
    my $msg = shift;                # HTTP::Message
    $msg->dump( maxlength => 512 ); # dump max 512 bytes (default is 512)
    return;
  }
);

The dump() function is documented in the HTTP::Message module.

The dump() function has a few options that can be provided:

• maxlength – maximum number of bytes to display, zero for unrestricted (all)
• no_content – a string to replace “(no content)” with
• prefix – a string to prepend to each line dumped

The handlers available are documented in the LWP::UserAgent module and are:

• request_preprepare
• request_prepare
• request_send
• response_header
• response_data
• response_done
• response_redirect