newspaint

Documenting Problems That Were Difficult To Find The Answer To

Monthly Archives: January 2020

Configuring RAS Daemon for Lenovo ThinkServer TS140

The RAS daemon (rasdaemon) can be installed on Ubuntu Linux 16.04 using the command:

$ sudo apt-get install rasdaemon

Once this is installed the ThinkServer TS140 requires a custom file to be created in /etc/ras/dimm_labels.d/ with the following file format:

# Vendor: 
#   Model: 
#     :  ...

… although it seems the old Edac label format is also valid:

# Vendor: 
#   Model: 
#     :  ..

In order to discover the values required one can run the following commands:

$ for i in /sys/devices/system/edac/mc/mc0/rank*; do \
  echo ==$i==; \
  cat "$i/dimm_location"; echo -ne "\n"; \
done

==/sys/devices/system/edac/mc/mc0/rank0==
csrow 0 channel 0 
==/sys/devices/system/edac/mc/mc0/rank1==
csrow 0 channel 1 
==/sys/devices/system/edac/mc/mc0/rank2==
csrow 1 channel 0 
==/sys/devices/system/edac/mc/mc0/rank3==
csrow 1 channel 1 
==/sys/devices/system/edac/mc/mc0/rank4==
csrow 2 channel 0 
==/sys/devices/system/edac/mc/mc0/rank5==
csrow 2 channel 1 
==/sys/devices/system/edac/mc/mc0/rank6==
csrow 3 channel 0 
==/sys/devices/system/edac/mc/mc0/rank7==
csrow 3 channel 1 

The board name and vendor can also be found:

$ cat /sys/devices/virtual/dmi/id/board_vendor
LENOVO
$ cat /sys/devices/virtual/dmi/id/board_name
ThinkServer TS140

Then a file (e.g. “lenovo-thinkserver-ts140.txt”) can be created in /etc/ras/dimm_labels.d/:

Vendor: LENOVO
  Model: ThinkServer TS140
    DIMM1_0: 0.0.0
    DIMM1_1: 0.0.1
    DIMM2_0: 0.1.0
    DIMM2_1: 0.1.1
    DIMM3_0: 0.2.0
    DIMM3_1: 0.2.1
    DIMM4_0: 0.3.0
    DIMM4_1: 0.3.1

And the rasdaemon restarted:

$ sudo systemctl status rasdaemon
$ sudo systemctl restart rasdaemon

Then the script can be run:

$ ras-mc-ctl --layout
          +-----------------------------------------------+
          |                      mc0                      |
          |  csrow0   |  csrow1   |  csrow2   |  csrow3   |
----------+-----------------------------------------------+
channel1: |  4096 MB  |  4096 MB  |  4096 MB  |  4096 MB  |
channel0: |  4096 MB  |  4096 MB  |  4096 MB  |  4096 MB  |
----------+-----------------------------------------------+

$ ras-mc-ctl --print-labels
LOCATION                            CONFIGURED LABEL     SYSFS CONTENTS      
mc0 csrow 0 channel 0               DIMM1_0              DIMM1_0             
mc0 csrow 0 channel 1               DIMM1_1              DIMM1_1             
mc0 csrow 1 channel 0               DIMM2_0              DIMM2_0             
mc0 csrow 1 channel 1               DIMM2_1              DIMM2_1             
mc0 csrow 2 channel 0               DIMM3_0              DIMM3_0             
mc0 csrow 2 channel 1               DIMM3_1              DIMM3_1             
mc0 csrow 3 channel 0               DIMM4_0              DIMM4_0             
mc0 csrow 3 channel 1               DIMM4_1              DIMM4_1    

The first time you do this the new labels may not be registered, e.g. you see the following:

$ ras-mc-ctl --print-labels
LOCATION                            CONFIGURED LABEL     SYSFS CONTENTS      
mc0 csrow 0 channel 0               DIMM1_0              mc#0csrow#0channel#0
mc0 csrow 0 channel 1               DIMM1_1              mc#0csrow#0channel#1
mc0 csrow 1 channel 0               DIMM2_0              mc#0csrow#1channel#0
mc0 csrow 1 channel 1               DIMM2_1              mc#0csrow#1channel#1
mc0 csrow 2 channel 0               DIMM3_0              mc#0csrow#2channel#0
mc0 csrow 2 channel 1               DIMM3_1              mc#0csrow#2channel#1
mc0 csrow 3 channel 0               DIMM4_0              mc#0csrow#3channel#0
mc0 csrow 3 channel 1               DIMM4_1              mc#0csrow#3channel#1

The “SYSFS” contents can be found:

$ for i in /sys/devices/system/edac/mc/mc0/rank*; \
  do echo "$i/dimm_label: $(cat $i/dimm_label)"; \
done
/sys/devices/system/edac/mc/mc0/rank0/dimm_label: mc#0csrow#0channel#0
...
/sys/devices/system/edac/mc/mc0/rank7/dimm_label: mc#0csrow#3channel#1

So you can register the configured labels by running:

$ sudo ras-mc-ctl --register-labels
$ ras-mc-ctl --print-labels
LOCATION                            CONFIGURED LABEL     SYSFS CONTENTS      
mc0 csrow 0 channel 0               DIMM1_0              DIMM1_0
...

In Ubuntu the rasdaemon service is defined in /etc/systemd/system/multi-user.target.wants/rasdaemon.service and contains the line:

ExecStart=/usr/sbin/rasdaemon -f -r

This instructs the daemon to start in foreground mode and to record events to a SQLite3 database.

The SQLite3 database is located at /var/lib/rasdaemon/ras-mc_event.db:

$ sqlite3 /var/lib/rasdaemon/ras-mc_event.db .tables
aer_event     extlog_event  mc_event      mce_record  

$ sqlite3 /var/lib/rasdaemon/ras-mc_event.db ".schema mc_event"
CREATE TABLE mc_event (
  id INTEGER PRIMARY KEY,
  timestamp TEXT,
  err_count INTEGER,
  err_type TEXT,
  err_msg TEXT,
  label TEXT,
  mc INTEGER,
  top_layer INTEGER,
  middle_layer INTEGER,
  lower_layer INTEGER,
  address INTEGER,
  grain INTEGER,
  syndrome INTEGER,
  driver_detail TEXT
);

Add LetsEncrypt to Reverse Proxy in Ubuntu Xenial

A rough picture of what is being attempted here is as follows (although I haven’t split out SSL and non-SSL virtual hosts here):

              ____________          ______________
             |            |        |              |
Internet --> | Rev Proxy  | - / -> |   Backend    |
             | 172.16.5.5 |        | 172.16.11.11 |
             |____________|        |______________|
                   |
             /.well-known/
              _____V_____
             | port 9876 |
             | 127.0.0.1 |
             |  certbot  |
             |___________|

First the certbot utility needs to be installed. To do this add the certbot repository to the /etc/apt/sources.list file:

deb http://ppa.launchpad.net/certbot/certbot/ubuntu xenial main

Attempt to update the apt repository.

$ sudo apt-get update
Get:5 http://ppa.launchpad.net/certbot/certbot/ubuntu xenial InRelease [24.3 kB]
Ign:5 http://ppa.launchpad.net/certbot/certbot/ubuntu xenial InRelease         
Get:10 http://ppa.launchpad.net/certbot/certbot/ubuntu xenial/main amd64 Packages [18.6 kB]
Get:11 http://ppa.launchpad.net/certbot/certbot/ubuntu xenial/main Translation-en [10.9 kB]
Fetched 3,495 kB in 1s (2,800 kB/s)                                
Reading package lists... Done
W: GPG error: http://ppa.launchpad.net/certbot/certbot/ubuntu xenial InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 8C47BE8E75BCA694
W: The repository 'http://ppa.launchpad.net/certbot/certbot/ubuntu xenial InRelease' is not signed.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.

We need to import the key to be able to install from this repository.

$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 8C47BE8E75BCA694
Executing: /tmp/tmp.X7oOFSoctO/gpg.1.sh --keyserver
keyserver.ubuntu.com
--recv-keys
8C47BE8E75BCA694
gpg: requesting key 75BCA694 from hkp server keyserver.ubuntu.com
gpg: key 75BCA694: public key "Launchpad PPA for certbot" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)

Next, install certbot.

$ sudo apt-get install certbot
0 to upgrade, 31 to newly install, 0 to remove and 3 not to upgrade.
Need to get 2,142 kB of archives.
After this operation, 11.4 MB of additional disk space will be used.
Do you want to continue? [Y/n] y

Now, update the Apache site configuration to proxy-pass (reverse proxy) certbot validation requests to a port that certbot will listen to during certificate requests. For example, I put the following into /etc/apache2/sites-enabled/000-default.conf:

  ProxyPass "/.well-known/" "http://127.0.0.1:9876/.well-known/"
  ProxyPassReverse "/.well-known/" "http://127.0.0.1:9876/.well-known/"

If you haven’t already you’ll need to include the proxy modules into Apache:

$ cd /etc/apache2/mods-enabled
$ sudo ln -s ../mods-available/proxy.* .
$ sudo ln -s ../mods-available/proxy_http.* .
$ sudo service apache2 restart

Now we can do a dry-run of certbot to test everything is functioning:

$ sudo certbot certonly \
    --preferred-challenges http-01 --http-01-port=9876 \
    --no-verify-ssl --dry-run \
    --domain www.mydomain.com.au \
    -v

Saving debug log to /var/log/letsencrypt/letsencrypt.log

How would you like to authenticate with the ACME CA?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Spin up a temporary webserver (standalone)
2: Place files in webroot directory (webroot)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1

IMPORTANT NOTES:
 - The dry run was successful.
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.

Now create the certificate (do the same but without a dry run):

$ sudo certbot certonly \
    --preferred-challenges http-01 --http-01-port=9876 \
    --no-verify-ssl \
    --domain www.mydomain.com.au \
    -v

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/www.mydomain.com.au/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/www.mydomain.com.au/privkey.pem
   Your cert will expire on 2020-03-31. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

Next we’ve got to add the created certificate to our Apache site configuration for the SSL virtual server, in my case this is file /etc/apache2/sites-enabled/001-default-ssl.conf:

    SSLCertificateFile /etc/letsencrypt/live/www.mydomain.com.au/cert.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/www.mydomain.com.au/privkey.pem
    SSLCertificateChainFile /etc/letsencrypt/live/www.mydomain.com.au/fullchain.pem

As for the reverse proxy part – you can reverse proxy to a SSL website internally using something like the following for your SSL virtual server configuration (within the <VirtualHost> tags).

    # disable SSL certificate checks for back-end server
    SSLProxyEngine on
    SSLProxyCheckPeerName off
    SSLProxyCheckPeerExpire off
    SSLProxyVerify none
    SSLProxyCheckPeerCN off

    # set X-Forwarded-For header to track actual IP of incoming request
    ProxyPreserveHost On

    ProxyPass "/" "https://172.16.11.11/"
    ProxyPassReverse "/" "https://172.16.11.11/"

On the backend server you’ll probably want to log the actual IP of the external request, rather than the IP of the reverse proxy.

To do this enable the mod_remoteip module to your enabled modules:

$ cd /etc/apache2/mods-enabled
$ sudo ln -s ../mods-available/remoteip.* .
$ sudo service apache2 restart

Configure your virtual server to use the remoteip module as well as update the custom logging configuration:

  # note %h is usually present in LogFormat for host, but use %a instead for mod_remoteip
  LogFormat "%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined_forwarded

  # use the address from X-Forwarded-For in access log instead of the reverse proxy
  CustomLog /var/log/apache2/access_ssl.log combined_forwarded

  # use the IP address from the X-Forwarded-For from requests from reverse proxy
  RemoteIPHeader X-Forwarded-For
  RemoteIPTrustedProxy 172.16.5.5