Documenting Problems That Were Difficult To Find The Answer To

Add LetsEncrypt to Reverse Proxy in Ubuntu Xenial

A rough picture of what is being attempted here is as follows (although I haven’t split out SSL and non-SSL virtual hosts here):

              ____________          ______________
             |            |        |              |
Internet --> | Rev Proxy  | - / -> |   Backend    |
             | |        | |
             |____________|        |______________|
             | port 9876 |
             | |
             |  certbot  |

First the certbot utility needs to be installed. To do this add the certbot repository to the /etc/apt/sources.list file:

deb xenial main

Attempt to update the apt repository.

$ sudo apt-get update
Get:5 xenial InRelease [24.3 kB]
Ign:5 xenial InRelease         
Get:10 xenial/main amd64 Packages [18.6 kB]
Get:11 xenial/main Translation-en [10.9 kB]
Fetched 3,495 kB in 1s (2,800 kB/s)                                
Reading package lists... Done
W: GPG error: xenial InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 8C47BE8E75BCA694
W: The repository ' xenial InRelease' is not signed.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.

We need to import the key to be able to install from this repository.

$ sudo apt-key adv --keyserver --recv-keys 8C47BE8E75BCA694
Executing: /tmp/tmp.X7oOFSoctO/ --keyserver
gpg: requesting key 75BCA694 from hkp server
gpg: key 75BCA694: public key "Launchpad PPA for certbot" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)

Next, install certbot.

$ sudo apt-get install certbot
0 to upgrade, 31 to newly install, 0 to remove and 3 not to upgrade.
Need to get 2,142 kB of archives.
After this operation, 11.4 MB of additional disk space will be used.
Do you want to continue? [Y/n] y

Now, update the Apache site configuration to proxy-pass (reverse proxy) certbot validation requests to a port that certbot will listen to during certificate requests. For example, I put the following into /etc/apache2/sites-enabled/000-default.conf:

  ProxyPass "/.well-known/" ""
  ProxyPassReverse "/.well-known/" ""

If you haven’t already you’ll need to include the proxy modules into Apache:

$ cd /etc/apache2/mods-enabled
$ sudo ln -s ../mods-available/proxy.* .
$ sudo ln -s ../mods-available/proxy_http.* .
$ sudo service apache2 restart

Now we can do a dry-run of certbot to test everything is functioning:

$ sudo certbot certonly \
    --preferred-challenges http-01 --http-01-port=9876 \
    --no-verify-ssl --dry-run \
    --domain \

Saving debug log to /var/log/letsencrypt/letsencrypt.log

How would you like to authenticate with the ACME CA?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Spin up a temporary webserver (standalone)
2: Place files in webroot directory (webroot)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1

 - The dry run was successful.
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.

Now create the certificate (do the same but without a dry run):

$ sudo certbot certonly \
    --preferred-challenges http-01 --http-01-port=9876 \
    --no-verify-ssl \
    --domain \

 - Congratulations! Your certificate and chain have been saved at:
   Your key file has been saved at:
   Your cert will expire on 2020-03-31. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:
   Donating to EFF:          

Next we’ve got to add the created certificate to our Apache site configuration for the SSL virtual server, in my case this is file /etc/apache2/sites-enabled/001-default-ssl.conf:

    SSLCertificateFile /etc/letsencrypt/live/
    SSLCertificateKeyFile /etc/letsencrypt/live/
    SSLCertificateChainFile /etc/letsencrypt/live/

As for the reverse proxy part – you can reverse proxy to a SSL website internally using something like the following for your SSL virtual server configuration (within the <VirtualHost> tags).

    # disable SSL certificate checks for back-end server
    SSLProxyEngine on
    SSLProxyCheckPeerName off
    SSLProxyCheckPeerExpire off
    SSLProxyVerify none
    SSLProxyCheckPeerCN off

    # set X-Forwarded-For header to track actual IP of incoming request
    ProxyPreserveHost On

    ProxyPass "/" ""
    ProxyPassReverse "/" ""

On the backend server you’ll probably want to log the actual IP of the external request, rather than the IP of the reverse proxy.

To do this enable the mod_remoteip module to your enabled modules:

$ cd /etc/apache2/mods-enabled
$ sudo ln -s ../mods-available/remoteip.* .
$ sudo service apache2 restart

Configure your virtual server to use the remoteip module as well as update the custom logging configuration:

  # note %h is usually present in LogFormat for host, but use %a instead for mod_remoteip
  LogFormat "%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined_forwarded

  # use the address from X-Forwarded-For in access log instead of the reverse proxy
  CustomLog /var/log/apache2/access_ssl.log combined_forwarded

  # use the IP address from the X-Forwarded-For from requests from reverse proxy
  RemoteIPHeader X-Forwarded-For

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: