Let’s Encrypt is awesome! Not only is it more secure than your existing certificate authority. It's also reliable, scalable, fully automated — and free!

letsencrypt-logo-horizontal


Have you ever had a certificate renewal go wrong?

It can be pretty painful.

One of the benefits of a load balancer, is that you can renew the certificates in the cluster one node at a time — with no downtime. And if you terminate on the load balancer you can use a test cluster for the new certificate, before using it in production.

But, if you've ever had to renew a large number of certificates — then you'll understand just how awesome automating the process with Let's Encrypt can be.

Aaron previously described our quick and dirty version of Let's Encrypt automation. With a few tweaks it's been working surprisingly well for customers with thousands of certificates.

So we decided to go ahead with full support in our product, along with a more resilient script to manage your certificates on the load balancer.

Why are SSL certificates becoming more important?

A couple of years ago it was acceptable to have a web site running in plain HTTP mode i.e. un-encrypted. If you were handling credit cards or needed a secure login area you could just encrypt that section of the web site with an HTTPS/SSL certificate.

However things have changed, with the increased focus on privacy — ALL communications should really be encrypted i.e. better safe than sorry.

Google warned us this was coming. Chrome now flags up a "Not secure" for any every website that uses un-secured HTTP.

chrome68-https

Sites that encrypt traffic with HTTPS will have their URL displayed normally in the address bar.

There has been a significant push by Google and Mozilla to always HTTPS, secure or bust (or a big red flag from Google, if rumours are to be believed), with the aim to achieve privacy and data integrity.

The necessity for simple SSL certification deployment and renewal is more important than ever, and Let's Encrypt makes it possible through freely available open certificate management.

Let's Encrypt is a zero cost certificate authority for HTTPS encryption, now trusted by all major root programs, including Google, Microsoft, Apple, Mozilla and Oracle. Used in conjunction with freely available tools it provides automatic enrolment/renewal, simple cert creation, negating validation emails and manual configuration.

One drawback is that they only offer Domain Vaildation (DV), not Organisation Validation (OV) or Extended Validation (EV) because it cannot automate that high level of certificate assurance. Let's Encrypt will validate that you are connecting to a certain domain securely, but it won't guarantee the ownership.

But then we all know that OV/EV was just a money making scam :-)

If your website is processing credit cards, transmitting sensitive data or email encryption, you may want a traditional EV/OV certificate provider.

SSL

Why is automation of certificate renewal important?

For enhanced security the Let's Encrypt certificate is only valid for 90 days, so an automated process for renewal is essential. We have you covered with an automatic renewal...read on.

So how do you configure Let's Encrypt on the Loadbalancer.org appliance?

Let's Encrypt is fully supported from version 8.3.4 onwards. You control the configuration through a simple command line script. So what does the script do?

  • It automates the process of validating the Domain and IP.
  • Downloads and moves the new certificates to correct location.
  • Automatically checks and renews the certificates.
  • Optional e-mail alerts, because we don't like surprises.
  • Batch Mode, because one is never enough.

The initial set up is somewhat involved — but it's worth it!

We need a special backend configured, just to talk to the Let's Encrypt servers. So you'll need to configure Layer 7 VIP with manual backend in the WebUI, then run the script to get the certificate and finally apply the new cert to the VIP via the WebUI (this can be also done via the API of course).

Prerequisites as follows:

  • A working Layer 7 VIP, HTTP mode using port 80
  • A working Stunnel termination
  • Port 80 must be open to the internet to allow access to Let's Encrypt servers
  • Must be able to access WebUI via localhost/127.0.01, not impeded by firewall rules.
  • Minimum version is 8.3.4

1. Configure the WebUI L7 Manual configuration

On the active load balancer, navigate via the WebUI to Cluster Configuration -> Layer 7 Manual Configuration

Add the following:

backend letsencrypt
server letsencrypt 127.0.0.1:8888

So it should look something like this:

ls-HAProxy_man-2

Then click the "Update" button.

2. Add an ACL

Via the WebUI Cluster Configuration -> Layer 7 - Virtual Services -> Modify -> Edit ACL Rules

le-acl-001

(from the script)

Rule Type  Boolean  URL Text/IP                   Action      Redirection Loc
path beg   Equals   /.well-known/acme-challenge/   Backend     letsencrypt

Then click on "Add" and "Save" on the ACL pop-up, then the "Update" button at the bottom of the Layer 7 Modify Virtual Service page and then reload HAProxy when prompted.

3. Enable lb-letsencrypt

Now, using lb-letsencrypt.sh, install and upgrade the acme.sh script. Open a ssh client to the Master load balancer, run the following command (use sudo if on AWS or Azure) :

lb-letsencrypt.sh -u

4. Lets get a certificate!

So, you have domain name, a working dns record forwarded to the VIP of the load balancer, let's connect the dots...

Before fetching the new certificate, you will have to allow port 80 open to access the internet, Let's Encrypt do not publish a list of their issuing servers so you'll have to allow a while performing this.

Also, there is a fair amount of rate limiting if there are many failed validations, somewhere in the range of 5 failures per account, per hostname, per hour.

The basic command structure is: Issue Cert (-i) on VIP port (-p) to the ACME.sh port (-h) for the domain (-d).

lb-letsencrypt.sh -i -p 443 -h 8888 -d foo.moo.com

And it looks like this:

[root@lbmaster lbuser]# lb-letsencrypt.sh -i -p 443 -h 8888 -d foo.moo.com
Process Stunnel is running, continuing.
we have a slave
Active appliance, continuing...
WUI Username:
loadbalancer
WUI Password:

We are asked for the WebUI user and password here, if you're AWS then remember that the password is the instance-id.

If it completes successfully, you'll see the certificate being issued to various locations, and the important bit:

Succesfully issued and uploaded new cert foo.moo.com, finished.
[root@lbmaster lbuser]#

5. Apply the new cert via the WebUI.

Apply the new certificate the Stunnel configuration via the WebUI -> SSL Terminations -> Modify:

le-ssl-cert-stun_834

Click the "Update" button, reload Stunnel when prompted, and, you're done.

Automatically check for certificate renewal:

By default, a crontab entry is added to automatically update the certificate within seven days of the expiration date, you can check it with crontab -l:

Script examples:

The script supports the following options:

-i = Issue Cert (Use either issue or batch not both!)
-b = Batch Mode Issue Cert "-b "
-D = Remove certificate config and cron entry.
-u = Install / Upgrade ACME.sh
-p = VIP Port
-h = ACME.sh Port (Example suggests 8888 but you can use any free port)
-d = Domain (You may provide multiple domains by providing -d multiple times like -d dom1.com -d dom2.com)
-f = Alert From Email Address
-t = Alert To Address
-r = SMTP Relay
-x = SMTP Port

Use lb-letsencrypt --help for help

You can add an email alert by issuing the script like this:

lb-letsencrypt -i -p 443 -h 8888 -d foo.moo.com -f lb1@gmail.com -t rob@gmail.com -r aspmx.l.google.com -x 25 

Multiple domains are not a problem:

lb-letsencrypt -i -p 443 -h 8888 -d foo.foo.com -d moo.moo.com -d boo.boo.com 

Deleting a certificate (delete it from the WebUI first):

lb-letsencrypt.sh -D -d foo.moo.com

Batch Mode

With batch mode you can run in a .CSV file with multiple lines of single domains, very useful for SNI configs using a cert for each domain or subdomain.

**The CSV file must contain one domain per line, as follows:**
DOMAIN,VIPPort,ACMEPort,FROMAddr,TOAddr,RELAYAddr,RELAYPort

And to execute:

lb-letsencrypt.sh -b import.csv

What does it do?

At the heart is the acme.sh shell script, probably one of the easiest and smartest client scripts around, to automatically issue & renew free certificates from Let's Encrypt, using the Automatic Certificate Management Environment (ACME) protocol.

There are two stages, first the agent proves to the Certificate Authority that the load balancer controls one (or more) domains, and then it requests or renews certificates against that domain. So, it verifies that you control a certain domain, and then issues a X.509 certificate, which is processed to the following locations in the load balancer:

Your cert is in /root/.acme.sh/foo.moo.com/foo.moo.com.cer
Your cert key is in /root/.acme.sh/foo.moo.com/foo.moo.com.key
The intermediate CA cert is in /root/.acme.sh/foo.moo.com/ca.cer
And the full chain certs is here: /root/.acme.sh/foo.moo.com/fullchain.cer

With lb-letsencrypt.sh, we provide a wrapper for acme.sh that gives the essential error-checking of the Loadbalancer.org config, the VIP and Stunnel information and specifies other utilities (email, batch, cron update).

Logs

You can see the changes being made in /var/log/lbadmin.log

Here is an example of a sucessful certificate provision:

Aug 28 14:15:01 lbmaster lb-letsencrypt: Successfully issued and uploaded new cert foo.moo.com, finished 

And heres the result of not allowing port 80 open to the internet:

Aug 28 13:54:38 lbmaster lb-letsencrypt: Failed to issue certificate for foo.moo.com, check external access on port 80 and that DNS has been configured. 

So what's next with Let's Encrypt?

We plan to make this a lot easier with a web interface based wizard. But the current system works well and we highly recommend that you try it.

Please leave a comment below to suggest what else we could do with the awesome Let's Encrypt — thank you.