Oh SNAPT! Where's my load balancer?

I was genuinely shocked and a little sad hearing the news that SNAPT has gone bust. They were a great company, with a nice product and more importantly — nice people. Our support team has been busy helping very stressed customers convert production sites to our commercial offering. But what if you don't have the budget for that?

Have you recently been left high and dry by your previous service provider, with just an HAProxy configuration file and nowhere to go? In this article, we will show you how easy it is to get an HAProxy-based replacement for SNAPT up and running in minutes, with just your config file to hand.

How to move from SNAPT to HAProxy for free:

Firstly, before we can get started, we'll need a Linux-based server which will act as our host for HAProxy. For this blog, we'll be using Ubuntu Server 22.04, a Debian-based Linux distribution, with HAProxy 2.0 available within its software repositories.

Installing HAProxy

To install the HAProxy service, first connect to the server over SSH and issue the following command:

sudo apt install haproxy

You should then see something similar to:

Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed
  haproxy
0 to upgrade, 1 to newly install, 0 reinstalled, 0 to remove and 93 not to upgrade.
Need to get 1,519 kB of archives.
After this operation, 3,288 KB of additional disk space will be used.
Get:1 http://gb.archive.ubuntu.com/ubuntu focal-updates/main amd64 haproxy amd64 2.0.13-2ubuntu0.5 [1,519 kB]
Fetched 1,519 kB in 0s (21.7 MB/s)
Selecting previously unselected package haproxy.
(Reading database ... 365637 files and directories currently installed.)
Preparing to unpack .../haproxy_2.0.13-2ubuntu0.5_amd64.deb ...
Unpacking haproxy (2.0.13-2ubuntu0.5) over (2.0.13-2ubuntu0.5) ...
Setting up haproxy (2.0.13-2ubuntu0.5) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for rsyslog (8.2001.0-1ubuntu1.3) ...
Processing triggers for systemd (245.4-4ubuntu3.17) ...

To ensure the service is enabled, issue the following command:

sudo systemctl enable haproxy.service

To check the service is running, use:

sudo systemctl status haproxy.service

...which will output something similar to:

● haproxy.service - HAProxy Load Balancer
     Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-08-31 13:02:59 BST; 1h 16min ago
       Docs: man:haproxy(1)
             file:/usr/share/doc/haproxy/configuration.txt.gz
   Main PID: 1699154 (haproxy)
      Tasks: 7 (limit: 18885)
     Memory: 2.9M
     CGroup: /system.slice/haproxy.service
             ├─1699154 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock -sf 1699155 -x /run/haproxy/admin.sock
             └─1780375 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock -sf 1699155 -x /run/haproxy/admin.sock

Your HAProxy configuration

In this example, we were able to obtain our HAProxy configuration from the Nova appliance using:

cat /etc/nova/haproxy/adcs/http451.cfg

...which will show your HAProxy configuration as follows (example shown):

frontend nova-http-451
bind 0.0.0.0:443
maxconn 100000
tcp-request content accept if { src -f /etc/nova/rulesets/whitelist.lst }
tcp-request content reject if { src -f /etc/nova/rulesets/blacklist.lst }
http-request set-header X-Nova-SRC %[src]
http-request set-header X-Nova-Country %[src,map_ip(/etc/nova/rulesets/ip2country.map)]
option httplog
log 127.0.0.1:12346 local0
capture request header Host len 64
capture request header User-Agent len 200
capture response header Server len 20
capture response header Content-Type len 64
mode http
option forwardfor
option http-server-close
timeout http-request 15s
http-response set-header Server NOVA
http-request track-sc0 src table per_ip_rates
http-request track-sc1 url32+src table per_ip_and_url_rates unless { path_end .css .js .png .jpeg .gif .woff .jpg }
http-request allow if { src -f /etc/nova/rulesets/whitelist.lst }
default_backend int-be-http-451-test
backend int-be-http-451-test
mode http
option accept-invalid-http-response
balance roundrobin
server test0 52.15.129.74:443  weight 10 check rise 5 fall 3 inter 2000

Importing your HAProxy configuration

Now that we have our HAProxy configuration, we can now import this into our new HAProxy service, that is running on our Ubuntu server.

The default HAProxy config file can be found in /etc/haproxy/haproxy.cfg. By using nano, we can edit the default haproxy config file and append the above to our existing configuration.

First, take a back-up the default config file:

sudo cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.backup

Then edit the haproxy.cfg file and append the Nova config to the end of it:

sudo nano /etc/haproxy/haproxy.cfg

After we've finished editing the haproxy.cfg file, adding our Nova config, use CTRL+X to save the file with the changes made.

Validating the HAProxy configuration

Before we restart the HAProxy service, we can check that the updated configuration file passes the validation. To do this we can issue the following command:

haproxy -f /etc/haproxy/haproxy.cfg -c

But wait, I have validation errors!

After we run the validation tool, we may encounter some errors, for example:

# haproxy -f /etc/haproxy/haproxy.cfg -c

[ALERT] 242/115957 (1626912) : parsing [/etc/haproxy/haproxy.cfg:43]: 'http-request set-header': failed to parse sample expression <src,map_ip(/etc/nova/rulesets/ip2country.map)> : invalid args in converter 'map_ip' : failed to open pattern file </etc/nova/rulesets/ip2country.map>.
[ALERT] 242/115957 (1626912) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg

Let's fix this...

There are a few things we need to do to fix this, as there are a number of files missing after we've added the Nova config to our new HAProxy installation.

Let's create the missing folders:

sudo mkdir -p /etc/nova/rulesets

Now we create the files that are referenced within out HAProxy config file:

sudo touch /etc/nova/rulesets/ip2country.map
sudo touch /etc/nova/rulesets/whitelist.lst
sudo touch /etc/nova/rulesets/blacklist.lst

Missing backends

In addition to resolving the missing files, there are a couple of backend declarations that are missing from the config file, which are referenced within the Nova configuration. These are mainly to do with Bot Protection and will address rate-limiting and automated content scrapers to name a few. You can read more about this, here: Bot Protection with HAProxy.

Add the following to the end of the haproxy.cfg file:

backend per_ip_and_url_rates
    stick-table type binary len 8 size 1m expire 24h store http_req_rate(24h)

backend per_ip_rates
    stick-table type ip size 1m expire 24h store gpc0,gpc0_rate(30s)

The complete configuration file should now look like this:

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-C>
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

#
# Nova HAProxy Config.
#
frontend nova-http-451
  bind 0.0.0.0:443
  maxconn 100000

  tcp-request content accept if { src -f /etc/nova/rulesets/whitelist.lst }
  tcp-request content reject if { src -f /etc/nova/rulesets/blacklist.lst }

  http-request set-header X-Nova-SRC %[src]
  http-request set-header X-Nova-Country %[src,map_ip(/etc/nova/rulesets/ip2country.map)]

  option httplog
  log 127.0.0.1:12346 local0
  capture request header Host len 64
  capture request header User-Agent len 200
  capture response header Server len 20
  capture response header Content-Type len 64
  mode http
  option forwardfor
  option http-server-close
  timeout http-request 15s
  http-response set-header Server NOVA
  http-request track-sc0 src table per_ip_rates
  http-request track-sc1 url32+src table per_ip_and_url_rates unless { path_end .css .js .png .jpeg .gif .woff .jpg }
  http-request allow if { src -f /etc/nova/rulesets/whitelist.lst }

default_backend int-be-http-451-test

backend int-be-http-451-test
 mode http
 option accept-invalid-http-response
 balance roundrobin
 server test0 52.15.129.74:443  weight 10 check rise 5 fall 3 inter 2000


backend per_ip_and_url_rates
  stick-table type binary len 8 size 1m expire 24h store http_req_rate(24h)

backend per_ip_rates
  stick-table type ip size 1m expire 24h store gpc0,gpc0_rate(30s)

Now we re-run the validation tool, to see if it passes validation:

haproxy -f /etc/haproxy/haproxy.cfg -c

We should see the following "Configuration file is valid" message:

# haproxy -f /etc/haproxy/haproxy.cfg -c
Configuration file is valid
# 

Restarting the HAProxy service

Now that we've successfully validated the configuration file, we can safely restart the HAProxy service:

sudo systemctl restart haproxy.service

Let's check the status, with:

sudo systemctl status haproxy.service

...which should output something similar to the following:

● haproxy.service - HAProxy Load Balancer
     Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-08-31 14:23:35 BST; 2s ago
       Docs: man:haproxy(1)
             file:/usr/share/doc/haproxy/configuration.txt.gz
    Process: 1789309 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exited, status=0/SUCCESS)
   Main PID: 1789310 (haproxy)
      Tasks: 7 (limit: 18885)
     Memory: 2.5M
     CGroup: /system.slice/haproxy.service
             ├─1789310 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
             └─1789311 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock

Further down, we can also see our frontend and backend HAProxy services have successfully started:

Aug 31 14:23:35 haproxy[1789310]: Proxy nova-http-451 started.
Aug 31 14:23:35 haproxy[1789310]: Proxy nova-http-451 started.
Aug 31 14:23:35 haproxy[1789310]: [NOTICE] 242/142335 (1789310) : New worker #1 (1789311) forked
Aug 31 14:23:35 haproxy[1789310]: Proxy int-be-http-451-test started.
Aug 31 14:23:35 systemd[1]: Started HAProxy Load Balancer.
Aug 31 14:23:35 haproxy[1789310]: Proxy int-be-http-451-test started.
Aug 31 14:23:35 haproxy[1789310]: Proxy per_ip_and_url_rates started.
Aug 31 14:23:35 haproxy[1789310]: Proxy per_ip_and_url_rates started.
Aug 31 14:23:35 haproxy[1789310]: Proxy per_ip_rates started.
Aug 31 14:23:35 haproxy[1789310]: Proxy per_ip_rates started.

Firewall Considerations

It is also worth checking that your iptables/firewalld rules are configured correctly to allow any inbound traffic you are loadbalancing with HAProxy.

Conclusion

And that's all there is to it! As you can see, it's a very straightforward process to get a vanilla HAProxy service up and running, using a Nova configuration file.

If you're needing to transition asap please do reach out. We've helped many others over the last few weeks in exactly the same situation.