newspaint

Documenting Problems That Were Difficult To Find The Answer To

Monthly Archives: September 2012

Setting Up A Socks Proxy Tunnelling Through Multiple SSH Servers

In this example we will connect:

 +-----------+             +---------+
 | localhost |             | server1 |
 |>1234      |-----> 11000 |>11000   |
 +-----------+             +---------+
                             |
                             V
                           12000
 +-----------+             +---------+
 | server3   |             |>12000   |
 |     SOCKS<| SOCKS <---- |server2  |
 +-----------+             +---------+
   |
   V
 INTERNET

…using this command…

  ssh -A -t -L1234:localhost:11000 me@server1 \
    'ssh -A -t -L11000:localhost:12000 me@server2 \
      ''ssh -A -t -D12000 me@server3'' \
    '

Note that you may want to add -v and -p <port_num> flags for verbose and port numbers as required.

This example sets up a localhost socks proxy on port 1234. And this will result in all traffic going to that proxy to come out at the Internet from server3.

Actually – the quoting isn’t strictly necessary. The above command could also be expressed as:

  ssh -A -t -L1234:localhost:11000 me@server1 \
    ssh -A -t -L11000:localhost:12000 me@server2 \
      ssh -A -t -D12000 me@server3

Some Assistance To Understanding

We know that -D port_num means that a SOCKS proxy will be listening on port_num on the local host and that traffic will be pushed through the SSH tunnel to the Internet from the connected-to SSH server.

In order to tunnel a port through to the end SSH server attached to the Internet we need to use a point-to-point port-forwarding tunnel through the intermediate SSH hosts. This is where the -L command comes in. By saying -L localport:endhost:endport we are saying that any TCP connections made to the localhost on localport will be forwarded through the SSH tunnel where the remote SSH host will attempt to send the connection to endhost:endport. In this example we want to eventually tunnel a port through to the SOCKS proxy port on the final SSH server. Hence at each stage endhost is always localhost.

Starting With Putty

Imagine you have a Windows PC. You want to set up a SOCKS proxy port on port 1234 on your PC (and set your Firefox SOCKS proxy to point at 127.0.0.1:1234) but have this tunnelled through two other Linux computer before your packets enter the Internet.

 +-----------+             +---------+
 | localhost |             | server1 |
 |>1234      |-----> 11000 |>11000   |
 +-----------+             +---------+
                             |
                             V
                           SOCKS
                           +---------+
                           |>SOCKS   |
            INTERNET <---- |server2  |
                           +---------+

Set up your Putty connection to server 1 but configure your tunnel to that server from localhost port 1234 to port 11000 on server 1 as follows:

Set Up A Local Tunnel From Your Local SOCKS Proxy Port To Port On First Linux Server

Set Up A Local Tunnel From Your Local SOCKS Proxy Port To Port On First Linux Server

When you press the Add button your configuration screen will look like the following:

Putty Tunnelling Screen After Adding Local Tunnel Rule

Putty Tunnelling Screen After Adding Local Tunnel Rule

Save your connection settings (as you would usually). Then when you open up your session type the following into your Linux server:

ssh -D11000 me@server2

This will establish a connection to server 2 (the server you want your packets to be exposed to the Internet). This command will cause server 1 to listen on port 11000 (which is the port packets from port 1234 on your local computer spill out to) and forward these packets to server 2 which will distribute them out onto the Internet.

In truth server 2 is the SOCKS proxy. You’ve just made a long tunnel from your local host to server 2 to reach the SOCKS proxy.

Trying Out OpenWRT 12.09-Beta On Buffalo WZR-HP-G300NH

Update: Note that as of 2012-11-22 there is a release candidate available at openwrt-ar71xx-generic-wzr-hp-g300nh-jffs2-sysupgrade.bin with checksum a56542d15a1109b92b2afbfcd9198a9d (“OpenWrt Attitude Adjustment 12.09-rc1 / LuCI 0.11 Branch (0.11+svn9425)”).


I downloaded openwrt-ar71xx-generic-wzr-hp-g300nh-jffs2-sysupgrade.bin from the OpenWRT website.

I then used the web interface to flash the new firmware to the router (after backing up my previous settings of course).

During the flashing process the front red diag LED will go on and stay fixed for a while (several minutes – be patient).

Eventually the log-in screen should display.

Log In Screen Of Attitude Adjustment 12.09 Beta

Log In Screen Of Attitude Adjustment 12.09 Beta

I’ve not noticed – straight away – any significant changes in the web interface.

The ssh login screen is a little different:

root@myserver:~# ssh -p22 192.168.1.1


BusyBox v1.19.4 (2012-08-26 12:49:54 UTC) built-in shell (ash)
Enter 'help' for a list of built-in commands.

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 ATTITUDE ADJUSTMENT (12.09-beta, r33312)
 -----------------------------------------------------
  * 1/4 oz Vodka      Pour all ingredients into mixing
  * 1/4 oz Gin        tin with ice, strain into glass.
  * 1/4 oz Amaretto
  * 1/4 oz Triple sec
  * 1/4 oz Peach schnapps
  * 1/4 oz Sour mix
  * 1 splash Cranberry juice
 -----------------------------------------------------

The /etc/config/wireless appears to have lost the “hwmode” option so I added it by putting the following line in:

config 'wifi-device' 'radio0'
  ...
  option 'hwmode' '11ng'
  ...

… this made my Wireless interface top speed 78Mbit/s.

Full Disk Encryption on Xubuntu Precise 12.04

The Xubuntu 12.04 live disk (which I ran on a USB stick installed using the UNetbootin tool) does not provide an installer capable of creating full disk encryption out of the box.

Instead one must pre-format the destination hard drive and create the encrypted partitions before installing the operating system.

The instructions I followed were taken from this blog post however I used a different cipher (aes-cbc-essiv:sha256). Actually for some reason the Ubuntu install disk doesn’t have sha256 support so I used aes-xtc-plain:sha512 which is arguably more secure.

I missed one essential element and this appeared to manifest itself in bug report launchpad bug 1003309 with the symptoms in this bug similar to mine (no enter-password screen on boot). The solution is below.

I will reproduce a large amount of that blog post because I feel that the information is so valuable that I wouldn’t want it to disappear off the Internet. But make no mistake: apart from the solution to preparing for a kernel upgrade (which I discovered) the instructions for encrypting during the install are not my own work.

Encrypting The Disk And Then Installation

Installation occurs after you have encrypted the disk, not before!

Download an Ubuntu or Xubuntu live disk and boot it (“try Ubuntu/Xubuntu without installing” from the boot menu).

Partitions

From the live desktop run gparted. Create a small (say 5GB) primary partition near the beginning of the disk – specify a type of ext4 and set the mount point to /boot. This partition must not become encrypted – it is necessary for launching the cryptography tools and the operating system.

Then create a large partition – I recommend primary but not strictly necessary – but leave some space at the end (2 x your RAM size, e.g. 16GB). Specify a type of ext4 and set the mount point to /. This will be the root encrypted partition.

Finally create a small partition – I recommend an extended but not strictly necessary – and specify a type of Linux swap.

Prepare For Encryption

Open a terminal in your live desktop and install the packages for encryption:

sudo -i
apt-get install lvm2

Encrypt The Root Partition

In this example my root partition was on /dev/sda2 while my boot partition was on /dev/sda1.

cryptsetup luksFormat -c aes-xts-plain -s 512 -h sha512 /dev/sda2
cryptsetup luksOpen /dev/sda2/ crypt
mkfs.ext4 /dev/mapper/crypt

Format The Boot Partition

mkfs.ext4 /dev/sda1

Install Ubuntu

From the desktop click on the installer icon.

When it asks if you want to use the whole disk (one of the first few choices you are given) click “do something else” and specify your partitions manually. Don’t forget to specify the three partitions! Root (/), boot (/boot), and swap (presumably /dev/sda3).

When installation is complete, don’t reboot! Click “continue testing!”.

Prepare Installed System To Use Encrypted Drive

Open a terminal window and mount the newly-installed partitions.

cd /media
mkdir /media/root
mount /dev/mapper/crypt /media/root
mount /dev/sda1 /boot

Now type sudo blkid and make a note of the UUID for the partition of /dev/sda2 (the root encrypted partition). Then paste it into /etc/crypttab – e.g.

# <target name> <source device>         <key file>      <options>
crypt UUID=176881fa-546a-492e-a3b6-dfbfbfb6a77a none luks

Next run the following command to set up the boot partition to find the encrypted disk.

sudo update-initramfs -u

If you get the error “update-initramfs is disabled since running on read-only media” then continue on and we’ll deal with this later.

Set Up An Encrypted Swap Partition

This will cause the swap to be encrypted with a random seed/key on every boot.

cryptsetup -d /dev/urandom create cryptswap /dev/sda3
mkswap -f /dev/mapper/cryptswap -v1

Then update /etc/crypttab:

# <target name> <source device>         <key file>      <options>
crypt UUID=176881fa-546a-492e-a3b6-dfbfbfb6a77a none luks
cryptswap /dev/sda3 /dev/urandom swap

Open /etc/fstab and change it to specify the UUID of the /boot partition – and specify /dev/mapper/crypt as the root partition – and add the swap partition, too:

# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    nodev,noexec,nosuid 0       0
/dev/mapper/crypt /               ext4    errors=remount-ro 0       1
UUID=3cd4ecfa-d494-469d-978b-327aa095a087 /boot           ext4    defaults        0       2
/dev/mapper/cryptswap   none    swap    sw      0       0

Re-run update-initramfs to be doubly sure and then reboot.

If you get the error “update-initramfs is disabled since running on read-only media” then type the following:

/media/root/usr/sbin/update-initramfs

(because we mounted the encrypted Linux partition to /media/root and the update-initramfs that came on the USB stick was always going to be disabled).

Preparing For A Kernel Upgrade

(You do want to be able to unlock the disk by being offered an opportunity to enter your password, right?)

After following the above instructions I could reboot the computer, enter my password for the disk encryption, and boot into Xubuntu.

As most people do I issued an apt-get update and apt-get dist-upgrade which resulted in a new kernel being installed. When I rebooted I got a plain black screen – and it would echo characters I typed but do nothing else. When I rebooted into recovery mode it would get as far as detecting my USB devices and then seem to hang. I was able (by pressing the tab key during boot) to get grub to boot me into my original kernel and that asked me for my disk encryption password as expected and I could boot that way.

The clue to solving this problem was in this Ubuntu help page about encrypted filesystems which specified that the cryptsetup package is not installed by default with modern Ubuntu.

When I ran a dpkg -l I could see I had the cryptsetup-bin package installed but not cryptsetup. So I installed it and this was the produced output:

root@mybox:/# apt-get install cryptsetup
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  linux-headers-3.2.0-29 linux-headers-3.2.0-29-generic python-support
Use 'apt-get autoremove' to remove them.
Suggested packages:
  busybox
The following NEW packages will be installed
  cryptsetup
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 80.0 kB of archives.
After this operation, 316 kB of additional disk space will be used.
Get:1 http://gb.archive.ubuntu.com/ubuntu/ precise/main cryptsetup amd64 2:1.4.1-2ubuntu4 [80.0 kB]
Fetched 80.0 kB in 0s (2,760 kB/s)  
Preconfiguring packages ...
Selecting previously unselected package cryptsetup.
(Reading database ... 172939 files and directories currently installed.)
Unpacking cryptsetup (from .../cryptsetup_2%3a1.4.1-2ubuntu4_amd64.deb) ...
Processing triggers for ureadahead ...
ureadahead will be reprofiled on next reboot
Processing triggers for man-db ...
Setting up cryptsetup (2:1.4.1-2ubuntu4) ...
update-initramfs: deferring update (trigger activated)
WARNING: you need to set all of cipher, hash and size for the plain dm-crypt mapping cryptswap in /etc/crypttab.
Processing triggers for initramfs-tools ...
update-initramfs: Generating /boot/initrd.img-3.2.0-30-generic

Then when I rebooted I found the new kernel asked me for my disk encryption password and booted normally.

Changing the Encrypted Partition’s Password

First find out which partition you want to modify. You can do this operation while the partition is mounted – so, thankfully, no need to reboot.

cat /etc/crypttab
blkid

List what keys you presently have for that partition:

root@myserver:~# cryptsetup luksDump /dev/sda2
LUKS header information for /dev/sda2

Version:       	1
Cipher name:   	aes
Cipher mode:   	cbc-essiv:sha256
Hash spec:     	sha512
Payload offset:	4096
MK bits:       	256
MK digest:     	22 14 a2 bb 34 13 23 65 af f2 ca ad 22 20 04 18 19 d2 2e 2f 
MK salt:       	5a 36 77 17 ca a4 12 02 ac 30 0f b7 52 5c ac 48 
               	7e 3a 57 8c 7f e8 86 9f c1 1a 07 2d 22 19 71 15 
MK iterations: 	14625
UUID:          	314281fa-546a-522e-a716-efa1bfb6ef1a

Key Slot 0: ENABLED
	Iterations:         	58833
	Salt:               	33 5a ca ca 72 01 19 00 12 fc c3 63 79 44 85 5b 
	                      	f1 0c 0d bb 6b 98 93 c0 86 4a ac 12 7e 71 bd 2e 
	Key material offset:	8
	AF stripes:            	4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED

Then add a key (you can have up to 8 per partition using luks):

root@myserver:~# cryptsetup luksAddKey /dev/sda2
Enter any passphrase: enter_your_old_password
Enter new passphrase for key slot: enter_new_password
Verify passphrase: enter_new_password

Now if the existing key was in slot 0 (zero) then you’ll have to delete the existing key from that slot. But you’ll have to provide a different key (e.g. the new key you just added) before you’ll be able to complete the operation.

root@myserver:~# cryptsetup luksKillSlot /dev/sda2 0
Enter any remaining LUKS passphrase: enter_new_password

All done.

Double-Check the Encrypted Partition’s Password

If you’re paranoid (and I’m paranoid) that you might have forgotton what your disk’s password is then you can type:

root@myserver:~# cryptsetup luksChangeKey /dev/sda2
Enter LUKS passphrase to be changed: 
Enter new LUKS passphrase: ^C

root@myserver:~#

If you type it wrong you’ll get:

root@myserver:~# cryptsetup luksChangeKey /dev/sda2
Enter LUKS passphrase to be changed: 
No key available with this pass-phrase.

Setting Linux Time Without NTP Using HTTP

Overview

The motivation behind this post is because I couldn’t access UDP NTP time servers from behind a corporate firewall (and UDP cannot be tunnelled through a SSH tunnel).

If you put the below into a script (and call it, say, get_time_from_http.pl) and call it as follows:

date `perl get_time_from_http.pl http://www.bbc.co.uk/`

…this will set the time on your computer wherever you are in the world (give or take one second plus the round-trip latency to the website you specified).

Why You Can Use The BBC Website From Anywhere In The World

The BBC website returns a date stamp in the following format:

Date: Fri, 21 Sep 2012 07:43:37 GMT

As the timestamp is in GMT the Perl functions Time::Local::timegm and localtime can be used to convert that time to a local time.

The output of this script is a string of the form MMDDhhmmCCYY.ss as per the man page for date.

get_time_from_http.pl

#!/usr/bin/perl -w

use Time::Local;
use LWP::UserAgent;
use Getopt::Std;

use strict;

my %opts = ();
Getopt::Std::getopts( 'c', \%opts );

my $url = shift or die( "Must provide URL to sync to" );

my $ua = LWP::UserAgent->new();
my $response = $ua->get( $url );
if ( ! $response->is_success() ) {
    die( "Failed to fetch URL: " . $response->status_line() );
}

my $datestr = $response->header( 'Date' );
die( "No date found" ) if ( ! $datestr );

sub getdatestr {
    my ( $epoch ) = @_;

    my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime($epoch);
    #my $str = sprintf( "%04d-%02d-%02d %02d:%02d:%02d", $year + 1900, $mon + 1, $mday, $hour, $min, $sec );
    my $str = sprintf( "%02d%02d%02d%02d%04d.%02d", $mon + 1, $mday, $hour, $min, $year + 1900, $sec );
    return( $str );
}

if ( $datestr =~ m{(\d+)\s+(\S+)\s+(\d{4})\s+(\d\d):(\d\d):(\d\d)(\s+(\S+))?} ) {
    my ( $weekday, $monthstr, $year, $hour, $min, $sec ) = ( $1, $2, $3, $4, $5, $6 );
    my ( $tzone ) = $8;

    my %months = qw(jan 1 feb 2 mar 3 apr 4 may 5 jun 6 jul 7 aug 8 sep 9 oct 10 nov 11 dec 12);
    my $month = $months{lc($monthstr)};

    my $tfunc = $tzone eq "GMT" ? \&Time::Local::timegm : \&Time::Local::timelocal;
    my $epoch = $tfunc->( $sec, $min, $hour, $weekday, $month - 1, $year );

    printf( "%s\n", getdatestr( $epoch ) );
} else {
    die( "Unknown date format \"" . $datestr . "\"" );
}

Monitoring the Billion 7404VGO-M ADSL Router

I have had, for some time, a Billion BiPAC 7404VGO-M ADSL router bought in Australia. I doubt the model is still available however I am documenting the following anyway in case anybody else is using such a (or similar) model.

Billion BiPAC 7404VGO-M ADSL2+ Router

Billion BiPAC 7404VGO-M ADSL2+ Router

It should be worth noting that this ADSL router works just fine in the UK using the Australian software. I am presently getting 15Mb/s downstream and 1Mb/s upstream.

This has been a rock-solid router for 4 years now. I have been very, very happy with it. It wasn’t the cheapest when I bought it but I have had no problems with it over the years.

What else do I like about this router? Monitoring it using SNMP. I presently monitor a number of parameters:

  • attenuation up/down stream
  • signal-to-noise ratio up/down stream
  • line speed up/down stream
  • data transferred up/down stream

Monitoring Script

I am posting my quick-and-dirty Perl script used for monitoring this router. Note that I have the router on IP address 192.168.1.254 (I have long forgotten what the default is for this router).

The script is run by cron every 5 minutes on a Debian/Ubuntu server and requires the rrdtool and snmp packages to be installed.

#!/usr/bin/perl -w

use strict;

print( scalar(localtime()) . " ---- \n" );

###
# THE following commands should be run manually at a Linux command line
# before running this script for the first time. It creates the data files
# (databases) into which monitoring data will be placed.
#
# rrdtool create adsl-speed.rrd -s 300 DS:up:GAUGE:600:0:999999999 DS:down:GAUGE:600:0:999999999 RRA:AVERAGE:0.5:1:2016 RRA:MIN:0.5:1:2016 RRA:MAX:0.5:1:2016 RRA:AVERAGE:0.5:12:8760 RRA:MIN:0.5:12:8760 RRA:MAX:0.5:12:8760
# rrdtool create adsl-snr.rrd -s 300 DS:up:GAUGE:600:0:9999 DS:down:GAUGE:600:0:9999 RRA:AVERAGE:0.5:1:2016 RRA:MIN:0.5:1:2016 RRA:MAX:0.5:1:2016 RRA:AVERAGE:0.5:12:8760 RRA:MIN:0.5:12:8760 RRA:MAX:0.5:12:8760
# rrdtool create adsl-att.rrd -s 300 DS:up:GAUGE:600:0:9999 DS:down:GAUGE:600:0:9999 RRA:AVERAGE:0.5:1:2016 RRA:MIN:0.5:1:2016 RRA:MAX:0.5:1:2016 RRA:AVERAGE:0.5:12:8760 RRA:MIN:0.5:12:8760 RRA:MAX:0.5:12:8760
# rrdtool create adsl-transfer.rrd -s 300 DS:up:DERIVE:600:0:99999999 DS:down:DERIVE:600:0:99999999 RRA:AVERAGE:0.5:1:2016 RRA:MIN:0.5:1:2016 RRA:MAX:0.5:1:2016 RRA:AVERAGE:0.5:12:8760 RRA:MIN:0.5:12:8760 RRA:MAX:0.5:12:8760

my $routerip = "192.168.1.254";
my $rrdspeed = "adsl-speed.rrd";
my $rrdsnr = "adsl-snr.rrd";
my $rrdatt = "adsl-att.rrd";
my $rrdtransfer = "adsl-transfer.rrd";

my %oid = (
  upspeed => '.1.3.6.1.2.1.10.94.1.1.5.1.2.3',
  downspeed => '.1.3.6.1.2.1.10.94.1.1.4.1.2.3',
  upsnr => '.1.3.6.1.2.1.10.94.1.1.3.1.4.3',
  downsnr => '.1.3.6.1.2.1.10.94.1.1.2.1.4.3',
  upattenuation => '.1.3.6.1.2.1.10.94.1.1.3.1.5.3',
  downattenuation => '.1.3.6.1.2.1.10.94.1.1.2.1.5.3',
  ifInOctets => '.1.3.6.1.2.1.2.2.1.10',
  ifOutOctets => '.1.3.6.1.2.1.2.2.1.16',
);

###
# GET LINESPEED
#

my $cmd_speed = "snmpget -v 1 $routerip -c public " . $oid{downspeed} . " " . $oid{upspeed};
my $result = `$cmd_speed`;

my %var = ();
if (
    $result =~ m{
        \A[^\r\n]+\s+([0-9]+)[\r\n]+
        [^\r\n]+\s+([0-9]+)[\r\n]+
    }sx
) {
    $var{downspeed} = $1;
    $var{upspeed} = $2;
}

if ( $var{downspeed} && $var{upspeed} ) {
    my $cmd_update = "rrdtool update $rrdspeed --template up:down N:" . join( ":", $var{upspeed}, $var{downspeed} );
    print( $cmd_update . "\n" );
    my $result2 = `$cmd_update`;
}

###
# GET SIGNAL-TO-NOISE RATIO
#

my $cmd_snr = "snmpget -v 1 $routerip -c public " . $oid{downsnr} . " " . $oid{upsnr};
$result = `$cmd_snr`;

if (
    $result =~ m{
        \A[^\r\n]+\s+([0-9]+)[\r\n]+
        [^\r\n]+\s+([0-9]+)[\r\n]+
    }sx
) {
    $var{downsnr} = $1 / 10;
    $var{upsnr} = $2 / 10;
}

if ( $var{downsnr} && $var{upsnr} ) {
    my $cmd_update = "rrdtool update $rrdsnr --template up:down N:" . join( ":", $var{upsnr}, $var{downsnr} );
    print( $cmd_update . "\n" );
    my $result2 = `$cmd_update`;
}

###
# GET LINE ATTENUATION
#

my $cmd_att = "snmpget -v 1 $routerip -c public " . $oid{downattenuation} . " " . $oid{upattenuation};
$result = `$cmd_att`;

if (
    $result =~ m{
        \A[^\r\n]+\s+([0-9]+)[\r\n]+
        [^\r\n]+\s+([0-9]+)[\r\n]+
    }sx
) {
    $var{downatt} = $1 / 10;
    $var{upatt} = $2 / 10;
}

if ( $var{downatt} && $var{upatt} ) {
    my $cmd_update = "rrdtool update $rrdatt --template up:down N:" . join( ":", $var{upatt}, $var{downatt} );
    print( $cmd_update . "\n" );
    my $result2 = `$cmd_update`;
}

###
# GET DATA TRANSFERRED
#

my $iface = "6";
my $cmd_transfer = "snmpget -v 1 $routerip -c public " . $oid{ifInOctets} . ".$iface " . $oid{ifOutOctets} . ".$iface";
$result = `$cmd_transfer`;

if (
    $result =~ m{
        \A[^\r\n]+\s+([0-9]+)[\r\n]+
        [^\r\n]+\s+([0-9]+)[\r\n]+
    }sx
) {
    $var{inoctets} = $1;
    $var{outoctets} = $2;
}

if ( defined($var{inoctets}) && defined($var{outoctets}) ) {
    my $cmd_update = "rrdtool update $rrdtransfer --template up:down N:" . join( ":", $var{outoctets}, $var{inoctets} );
    print( $cmd_update . "\n" );
    my $result2 = `$cmd_update`;
}

Graphing Script

This script processes the RRD data files and creates PNG images in the local directory. It then copies the images to the /var/www/monitor subdirectory and changes the permissions on the files (to ensure they can be overwritten in the future).

The script is run by cron every 5 minutes on a Debian/Ubuntu server and requires the rrdtool package to be installed.

#!/usr/bin/perl -w

use strict;

print( scalar(localtime()) . "\n" );
my $now = scalar(localtime());
$now =~ s/:/\\:/g;

my $rrdspeed = "adsl-speed.rrd";
my $rrdsnr = "adsl-snr.rrd";
my $rrdatt = "adsl-att.rrd";
my $rrdtransfer = "adsl-transfer.rrd";

my $width = 800;
my $height = 380;

foreach my $settings (
    [ $height, $width, "365d" ],
    [ $height, $width, "31d" ],
    [ $height, $width, "7d", ],
    [ $height, $width, "24h", ],
) {
    graph_them(
        $settings->[0],
        $settings->[1],
        $settings->[2],
    );
}

sub graph_them {
    my ( $height, $width, $period ) = @_;

    my $cmd_rrdgraph = "rrdtool graph adsl-speed-${period}.png";
    $cmd_rrdgraph .= " -s 'now-$period'";
    $cmd_rrdgraph .= " -e now";
    $cmd_rrdgraph .= " -t 'ADSL Router Speed'";
    $cmd_rrdgraph .= " -v  bps";
    $cmd_rrdgraph .= " -w $width -h $height";
    $cmd_rrdgraph .= " --lower-limit 0";
    $cmd_rrdgraph .= " DEF:up=$rrdspeed:up:AVERAGE";
    $cmd_rrdgraph .= " DEF:down=$rrdspeed:down:AVERAGE";
    $cmd_rrdgraph .= " 'LINE2:up#f02020:  upstream'";
    $cmd_rrdgraph .= " 'GPRINT:up:MIN:Min %8.0lf bps'";
    $cmd_rrdgraph .= " 'GPRINT:up:AVERAGE:Avg %8.0lf bps'";
    $cmd_rrdgraph .= " 'GPRINT:up:MAX:Max %8.0lf bps'";
    $cmd_rrdgraph .= " 'GPRINT:up:LAST:Last %8.0lf bps'";
    $cmd_rrdgraph .= " 'COMMENT:\\l'";
    $cmd_rrdgraph .= " 'LINE2:down#1040e8:downstream'";
    $cmd_rrdgraph .= " 'GPRINT:down:MIN:Min %8.0lf bps'";
    $cmd_rrdgraph .= " 'GPRINT:down:AVERAGE:Avg %8.0lf bps'";
    $cmd_rrdgraph .= " 'GPRINT:down:MAX:Max %8.0lf bps'";
    $cmd_rrdgraph .= " 'GPRINT:down:LAST:Last %8.0lf bps'";
    $cmd_rrdgraph .= " 'COMMENT:\\l'";
    $cmd_rrdgraph .= " 'COMMENT:Created $now\\r'";

    print `$cmd_rrdgraph`;

    $cmd_rrdgraph = "rrdtool graph adsl-snr-${period}.png";
    $cmd_rrdgraph .= " -s 'now-$period'";
    $cmd_rrdgraph .= " -e now";
    $cmd_rrdgraph .= " -t 'ADSL Signal-to-Noise Ratio'";
    $cmd_rrdgraph .= " -v  dB";
    $cmd_rrdgraph .= " -w $width -h $height";
    $cmd_rrdgraph .= " --lower-limit 0";
    $cmd_rrdgraph .= " 'DEF:up=$rrdsnr:up:AVERAGE'";
    $cmd_rrdgraph .= " 'DEF:down=$rrdsnr:down:AVERAGE'";
    $cmd_rrdgraph .= " 'LINE2:up#f02020:  upstream'";
    $cmd_rrdgraph .= " 'GPRINT:up:MIN:Min %5.1lf dB'";
    $cmd_rrdgraph .= " 'GPRINT:up:AVERAGE:Avg %5.1lf dB'";
    $cmd_rrdgraph .= " 'GPRINT:up:MAX:Max %5.1lf dB'";
    $cmd_rrdgraph .= " 'GPRINT:up:LAST:Last %5.1lf dB\\l'";
    $cmd_rrdgraph .= " 'LINE2:down#1040e8:downstream'";
    $cmd_rrdgraph .= " 'GPRINT:down:MIN:Min %5.1lf dB'";
    $cmd_rrdgraph .= " 'GPRINT:down:AVERAGE:Avg %5.1lf dB'";
    $cmd_rrdgraph .= " 'GPRINT:down:MAX:Max %5.1lf dB'";
    $cmd_rrdgraph .= " 'GPRINT:down:LAST:Last %5.1lf dB\\l'";
    $cmd_rrdgraph .= " 'COMMENT:Created $now\\r'";

    print `$cmd_rrdgraph`;

    $cmd_rrdgraph = "rrdtool graph adsl-att-${period}.png";
    $cmd_rrdgraph .= " -s 'now-$period'";
    $cmd_rrdgraph .= " -e now";
    $cmd_rrdgraph .= " -t 'ADSL Line Attenuation'";
    $cmd_rrdgraph .= " -v  dB";
    $cmd_rrdgraph .= " -w $width -h $height";
    $cmd_rrdgraph .= " --lower-limit 0";
    $cmd_rrdgraph .= " DEF:up=$rrdatt:up:AVERAGE";
    $cmd_rrdgraph .= " DEF:down=$rrdatt:down:AVERAGE";
    $cmd_rrdgraph .= " 'LINE2:up#f02020:  upstream'";
    $cmd_rrdgraph .= " 'GPRINT:up:MIN:Min %5.1lf dB'";
    $cmd_rrdgraph .= " 'GPRINT:up:AVERAGE:Avg %5.1lf dB'";
    $cmd_rrdgraph .= " 'GPRINT:up:MAX:Max %5.1lf dB'";
    $cmd_rrdgraph .= " 'GPRINT:up:LAST:Last %5.1lf dB\\l'";
    $cmd_rrdgraph .= " 'LINE2:down#1040e8:downstream'";
    $cmd_rrdgraph .= " 'GPRINT:down:MIN:Min %5.1lf dB'";
    $cmd_rrdgraph .= " 'GPRINT:down:AVERAGE:Avg %5.1lf dB'";
    $cmd_rrdgraph .= " 'GPRINT:down:MAX:Max %5.1lf dB'";
    $cmd_rrdgraph .= " 'GPRINT:down:LAST:Last %5.1lf dB\\l'";
    $cmd_rrdgraph .= " 'COMMENT:Created $now\\r'";

    print `$cmd_rrdgraph`;

    $cmd_rrdgraph = "rrdtool graph adsl-transfer-${period}.png";
    $cmd_rrdgraph .= " -s 'now-$period'";
    $cmd_rrdgraph .= " -e now";
    $cmd_rrdgraph .= " -t 'ADSL Data Transferred (5 min average)'";
    $cmd_rrdgraph .= " -v 'Bytes/s'";
    $cmd_rrdgraph .= " -w $width -h $height";
    $cmd_rrdgraph .= " --lower-limit 0";
    $cmd_rrdgraph .= " DEF:up=$rrdtransfer:up:AVERAGE";
    $cmd_rrdgraph .= " DEF:down=$rrdtransfer:down:AVERAGE";
    $cmd_rrdgraph .= " CDEF:cup=up,1024,1024,1024,*,*,/";
    $cmd_rrdgraph .= " CDEF:cdown=down,1024,1024,1024,*,*,/";
    $cmd_rrdgraph .= " VDEF:tup=cup,TOTAL";
    $cmd_rrdgraph .= " VDEF:tdown=cdown,TOTAL";
    $cmd_rrdgraph .= " 'LINE2:up#f02020:  upstream'";
    $cmd_rrdgraph .= " 'GPRINT:up:MIN:Min %8.0lf B/s'";
    $cmd_rrdgraph .= " 'GPRINT:up:AVERAGE:Avg %8.0lf B/s'";
    $cmd_rrdgraph .= " 'GPRINT:up:MAX:Max %8.0lf B/s'";
    $cmd_rrdgraph .= " 'GPRINT:up:LAST:Last %8.0lf B/s'";
    $cmd_rrdgraph .= " 'GPRINT:tup:Total %7.3lf GBytes'";
    $cmd_rrdgraph .= " 'COMMENT:\\l'";
    $cmd_rrdgraph .= " 'LINE2:down#1040e8:downstream'";
    $cmd_rrdgraph .= " 'GPRINT:down:MIN:Min %8.0lf B/s'";
    $cmd_rrdgraph .= " 'GPRINT:down:AVERAGE:Avg %8.0lf B/s'";
    $cmd_rrdgraph .= " 'GPRINT:down:MAX:Max %8.0lf B/s'";
    $cmd_rrdgraph .= " 'GPRINT:down:LAST:Last %8.0lf B/s'";
    $cmd_rrdgraph .= " 'GPRINT:tdown:Total %7.3lf GBytes'";
    $cmd_rrdgraph .= " 'COMMENT:\\l'";
    $cmd_rrdgraph .= " 'COMMENT:Created $now\\r'";

    print `$cmd_rrdgraph`;

    for my $graph ( qw(att snr speed transfer) ) {
        `cp "adsl-${graph}-${period}.png" "/var/www/monitor/"`;
        `chmod 666 "/var/www/monitor/adsl-${graph}-${period}.png"`;
    }
}

Resulting Graphs

Real-world example of signal-to-noise ratio over a year

Real-world example of signal-to-noise ratio over a year

The above is the output from my server showing the changes in signal-to-noise ratio from the same location over the period of a full year.

OpenVPN Adventures Today

Setting Up OpenWRT As An OpenVPN Client

I wanted to set up my wireless router so that all LAN (including wi-fi) connections to it would be tunnelled through OpenVPN to a remote OpenVPN server on the Internet. This turned out to be a lot more time consuming than I wanted it to be.

OpenVPN Client With OpenWRT Router

OpenVPN Client With OpenWRT Router

Above is a diagram of the set-up I wanted to establish.

Set Up OpenVPN

Packages you’re going to want to install: openvpn, ip. The ip package is very important because it has the ip route command which sets the default route after a successful tunnel has been established.

Here is my OpenVPN client settings in file /etc/config/openvpn on my OpenWRT router:

config openvpn sample_client        
        option enable 1 # Set to 1 to enable
        option client 1
        option pull 1

        # choice of tap/tun must be the same as server
        option dev tap
#       option dev tun

        # I started with TCP but moved to UDP
#       option proto tcp
        option proto udp

        # UDP MUST be fragmented
        option tun-mtu 1200
        option fragment 1000

        # The hostname/IP and port of the server.
        # You can have multiple remote entries
        # to load balance between the servers.
        list remote "1.2.3.4 1195" # myopenvpnserver.com

        # Keep trying indefinitely to resolve the
        # host name of the OpenVPN server.
        option resolv_retry infinite

        option nobind 1 # no need to bind to specific local port

        # Try to preserve some state across restarts.
        option persist_key 1
        option persist_tun 1

        # SSL/TLS parms.
        option ca /etc/openvpn/ca.crt
        option cert /etc/openvpn/client-homerouter.crt
        option key /etc/openvpn/client-homerouter.key

        # If a tls_auth key is used on the server
        # then every client must also have the key.
        option tls_auth "/etc/openvpn/ta.key 1"
        option dh "/etc/openvpn/dh2048.pem"

        # Enable compression on the VPN link.
        # Don't enable this unless it is also
        # enabled in the server config file.
        option comp_lzo 0

        option verb 3 # Set log file verbosity.

        option status /tmp/openvpn-status.log

Your OpenVPN configuration may be somewhat different depending on what certificates you use, whether you use TA or DH, whether you use compression, or TCP/UDP (but I’ll discuss TCP/UDP a little later).

Now for OpenVPN to start automatically you need to run the following command on the server:

cd /etc/rc.d
ln -s ../init.d/openvpn /etc/rc.d/S46openvpn

This will ensure OpenVPN is started upon reboot after the network has been configured.

Finally – if you use UDP then (as I describe further down) you need to establish a static route to your VPN server out the eth1 (WAN) port so that those packets can still go where they need to go after the VPN sets a default route to itself. You can run:

ip route add 1.2.3.4/32 via 192.168.0.1

… where 1.2.3.4 should be replaced with your OpenVPN server address, and 192.168.0.1 should be replaced with your upstream router IP address.

However if you want this to happen automatically on boot then add the following to your /etc/config/network file:

config 'route'
        option 'interface' 'wan'
        option 'target' '1.2.3.4'
        option 'netmask' '255.255.255.255'
        option 'gateway' '192.168.0.1'

Network

It becomes necessary to edit /etc/config/network so that the following section is added:

config 'interface' 'vpn'
        option 'ifname' 'tap0'
        option '_orig_ifname' 'tap0'
        option '_orig_bridge' 'false'
        option 'proto' 'none'

Also you will have to add the following line to your config 'interface' 'wan' block:

        option 'metric' '20'

This is so that the WAN default route becomes lower in priority to the VPN default route when the VPN is established.

Firewall

This is the trickiest bit. And for me didn’t actually work until I did a reboot!

In this example the file /etc/config/firewall is being edited:

I added a zone:

config 'zone'
        option 'name' 'vpn'
        option 'input' 'ACCEPT'
        option 'output' 'ACCEPT'
        option 'forward' 'ACCEPT'
        option 'network' 'vpn'
        option 'masq' '1'

Note the ‘masq’ option. It is important that this setting is present in both the ‘vpn’ and ‘wan’ zones but not the ‘lan’ zone.

Next I added two forwarding rules:

config 'forwarding'
        option 'src' 'vpn'
        option 'dest' 'lan'

config 'forwarding'
        option 'src' 'lan'
        option 'dest' 'vpn'

Okay time to reboot router!

At The OpenVPN Server Side

I was running my OpenVPN on an Ubuntu server. There were a few things I had to do before everything would go smoothly, however, mostly involving the firewall and iptables.

Firstly I needed a MASQUERADE rule. My OpenVPN server was configured to be 172.17.0.1/16 (that’s the IP address to which all clients would forward packets). So anything coming from that address destined for the Internet (eth0) needed to be MASQUERADE-d. In my /var/lib/iptables/active file (which I restore using iptables-restore on boot I added:

*nat
:POSTROUTING ACCEPT [0:0]

-A POSTROUTING -s 172.17.0.0/16 -o eth+ -j MASQUERADE

COMMIT

I also needed, in the *filter tables, rules to permit forwarding to/from the TAP interface:

-A FORWARD -i tap+ -j ACCEPT
-A FORWARD -o tap+ -j ACCEPT

TCP vs UDP

I started with a TCP OpenVPN client connection – this would be established fine. But I had an infuriating problem. The connection kept dying after 1-4 minutes. I would get log entries in the server like this:

Sun Sep  2 17:34:07 2012 client-homerouter/2.4.6.8:35955 [client-homerouter] Inactivity timeout (--ping-restart), restarting
Sun Sep  2 17:34:07 2012 client-homerouter/2.4.6.8:35955 SIGUSR1[soft,ping-restart] received, client-instance restarting
Sun Sep  2 17:34:07 2012 TCP/UDP: Closing socket

I never resolved this. The closest I got was pulling out tshark on the server and tcpdump on the OpenWRT router and could see that TCP-retransmission packets were being sent back to the client but would never get a response from the router. No idea what the problem was.

This seems to have been resolved with switching to UDP – at least with UDP there are no retransmissions if a packet gets dropped. However UDP simply won’t work unless data is fragmented (see below).

Errors Encountered

FRAG_IN error flags=0xffffffff: FRAG_TEST not implemented

This was turning up in my OpenVPN server log after I switched from TCP to UDP. I’d set the tun-mtu and fragment options in my server.conf – but on my Android phone I had not set these values. By editing my OpenVPN connection on my Android phone, I then selected menu->Advanced and then at the bottom added the extra arguments:

--tun-mtu 1200 --fragment 1000

.. and this stopped the above errors in my server log.

read UDPv4 [EHOSTUNREACH]: No route to host (code=148)

On my OpenWRT router, which was acting as an OpenVPN client, I would be getting these errors – and this happened when I switched from TCP to UDP.

The problem was this: the connection would establish fine – I could see it established in my server log. But then something curious happened. The server would send UDP traffic to the client (my router) but my client (the router) would never send any more UDP packets after the connection was established.

Eventually I realised this: when an OpenVPN client establishes a connection I make it set the default route to be through that VPN tunnel. However the UDP packets the client was sending – which were being sent to the IP address of the server – were now being sucked into the new default route (in to itself) – hence they disappeared.

This didn’t happen with a TCP connection because a connection is established once: at the beginning of the connection. Hence it didn’t matter if the default route changed mid-way through the connection – because the connection had already been established. But UDP packets are independent entities – every single one of them. So the instant the default route changed the outgoing client packets were just going into the VPN tunnel instead of out the WAN link.

This was fixed by adding another specific route. E.g. if my VPN server was 14.1.2.3 then I should add the command:

ip route add 14.1.2.3/32 via 192.168.0.1

..(where 192.168.0.1 is my upstream router/WAN gateway).

read TCPv4_CLIENT []: No route to host (code=148)

This is the error I’d receive when my TCP-based OpenVPN connection would stop working. Looking back I suspect this is identical to the UDPv4 problem above – just that it would take a few minutes before causing the problem rather than right away as in the case of UDP. The solution, of course, is to add a static route to the OpenVPN host through the WAN port.

Notes For Upgrading To Attitude Adjustment 12.09

After I upgraded the router to OpenWRT Attitude Adjustment 12.09 Beta I had to re-establish a few things:

opkg install openvpn
opkg install ip

cd /etc/rc.d
ln -s ../init.d/openvpn /etc/rc.d/S46openvpn

While this will set up the tunnel I don’t seem to be getting the default route being set to point at the tunnel anymore. I can still manually force this by issuing the following command:

ip route add 0.0.0.0/0 via x.x.x.x

…where x.x.x.x is the next-hop address of the OpenVPN server.

Cheap Car Video Camera From Nowkin

I recently purchased a video camera for my car. I bought it from Amazon from this link for around £25 – however when I purchased it the company selling it was NowAdvisorUK not Loftk as more recently shown. Also the camera I purchased looked different; it arrived after 7 or 8 days from Hong Kong (I used standard shipping) to the UK in a package with http://www.nowkin.com printed on it. A link to the device I had delivered from NowKin is here.

The car camera I have been using

The car camera I have been using

The device came with suction cup mount (works great) and slim profile device with monitor that can be flipped out. It had 6 infra-red (IR) LEDs on the front. It came with a short mini-USB to PC-USB cable, and a long cabled mini-USB to car-adapter. No SD card included.

The device has absolutely no brand marks, no model numbers, no manufacturer stamps, nothing on the case. The software identifies itself as version 2012-05-21SD which doesn’t give much away. When plugged into a Linux server I can identify the device as “Sunplus Technology Co., Ltd SPCA1527A/SPCA1528 SD card camera”.

The software is pretty straight-forward to use. Put in a class-4 SD-card and a red icon will appear. Put in a class-6 SD-card and a white icon will appear. It appears that you only want to put in class-6 or higher (faster) cards. Class-4 cards will record but the motion might be jerky.

The timestamp needs setting at first. You can only set up to the minute, not the second, but maybe this will be fixed in a later release.

If you have this connected to your car charger the camera will automatically begin recording when power is applied to the USB socket. And the camera will switch off within 1-2 seconds of the power being switched off. There is no 3 or 5-minute stay-on period after turning off – in spite of a menu option provided for this. You can initiate recording without USB power by turning on then pressing the OK/record button – which may be useful for recording outside of a vehicle (and experimenting indoors when you first get it).

There’s very little (if any) break between recorded files. I know this because I’ve taken a lot of the files and joined them together and it isn’t usually noticeable when going from one 5-minute period to another.

The IR LEDs are not very useful – truth be told. They are highly focussed, too. Best to switch them off and just let the camera’s reasonable exposure handling at night do its work showing you what you see when your headlights are on.

The view is very wide. I record in 1280×720 mode. The camera supports 1920×1080 and 640×480 but don’t feel either of these modes are for me. As a result number plates are only visible when almost directly behind another vehicle at the lights.

Audio pick up is good. It might be quiet but consistently clear – so if you turn up the volume you’ll hear yourself and your car stereo and indicator all quite clearly. The track is in mono.

Video files are large. Around 616MB every 5 minutes. My 8GB flash card lasts about an hour before being overwritten. Bear in mind that flash isn’t designed to be abused like this – most flash cards will last between 1,000 and 10,000 complete writes – so expect your flash card to eventually die after that many hours of video (I’m not 100% on this but just saying what I expect).

Device comes with a BL-5C flat rectangular cellphone battery (quite clever really). The internal clock will survive a brief removal of the cellphone battery (haven’t tried taking it out for a long period though).

I’m quite satisfied with the result. I certainly enjoy using FFMPEG to concatenate video files together and speed them up 20x for my friends. I just wish the device was labelled somehow so I could do software updates and/or recommend the device to my friends who could knowingly get exactly the same device.

Videos

I have uploaded two videos I have taken – with a mix of sped up and normal speed – to demonstrate the camera image quality in low light conditions without the IR LEDs (and really the IR LEDs add little to nothing). There is a video taken in the evening, and another taken at night.