A couple of our customers have asked if our appliances would do G-Zip compression in the past we haven't given it much thought. Then out of the blue a company offered us a card to test with aha.com and some of us in the office welcoming the opportunity to meddle with anything new jumped at the chance.
Why have we not bothered much with compression?
Well because all modern web servers automatically apply compression when required. And our load balancer can easily be configured to do compression (but why would you do it twice?). Although, saying that if you are running a WAF (Web Application Firewall) then you might want to unzip files on the fly to inspect them for malicious content. And then you might run into performance issues, and hence the need for a hardware accelerator.
So Lets have a play with a hardware aceelerator for G-Zip compression!
They provided us with a AHA360-PCIe 2.5 Gbits/sec GZIP Compression/Decompression Accelerator
Features
- Open standard compression algorithm (GZIP/Deflate)
- Performs both compression and decompression
- Compresses and decompresses at a throughput up to 2.5 Gbits/sec
- Uses the AHA3610
- x4 PCI-Express interface
- ASCII text compresses greater than 3.0:1
- 3.6:1 Canterbury corpus compression ratio
- Scatter Gather DMA
- 64-bit OS support
- Linux and Windows reference drivers with source code
- SMP and Hypterthreading support
- 32-bit and 64-bit Linux kernel support
As we all like stats here are a few numbers from the initial performance of the card.
Firstly I compared the G-Ziping (im sure thats not a real word) of a 533MB iso file with the same process running through the card. The numbers are quite good, as you can see it almost entirely offloaded the process to the card and ran the compression though in half the time it took the SW version of G-Zip
time ./ahazip -f=/usr/src/file31.iso -o=/usr/src/compressed.gz
real 0m15.566s
user 0m0.015s
sys 0m1.674s
[root@lbmaster bin]# time gzip /usr/src/file31.iso
real 0m38.915s
user 0m33.966s
sys 0m1.084s
I then decided to get a bit more adventurous and compress 2 files at once the performance of the card was quite heavily affected.
[root@lbmaster bin]# time ./ahazip -f=/usr/src/file32.iso -o=/usr/src/compressed1.gz
real 0m35.888s
user 0m0.011s
sys 0m1.717s
[root@lbmaster bin]# time ./ahazip -f=/usr/src/file31.iso -o=/usr/src/compressed.gz
real 0m28.262s
user 0m0.012s
sys 0m1.611s
and the results for G-Zipping the 2 files
[root@lbmaster bin]# time gzip /usr/src/file31.iso
real 0m58.823s
user 0m43.235s
sys 0m1.480s
[root@lbmaster bin]# time gzip /usr/src/file32.iso
real 1m7.898s
user 0m45.205s
sys 0m1.325s
What impressed me the most was the incredibly low overhead when using the card compared with using the system to do the compression. It barely touched the CPU, leaving the system to get on with other important jobs.
Back to the main event.........
Here is how I did it - you need your G-Zip card to use the standard Zlib compression library, and obviously you need a proxy that can support it. In this instance we are using Nginx as in its current guise HAProxy (both HAProxy and Nginx are available on our appliances) does not support G-Zipping the datastream, so we had to proxy the proxy in order to get it to work. Ill assume your happy in installing it and getting everything running. Below is the procedure for getting the card we were supplied with to work with Nginx.
I had to use a slightly modified configure file as well as a modified makefile which were kindly given to me by their excellent support team. You just need to replace them with the existing ones in the Zlib directory that came with the drivers that you downloaded from their site.
To configure Nginx I used the following command – ./configure –prefix=/usr –sbin-path=/usr/sbin/nginx –conf-path=/etc/nginx/nginx.conf –error-log-path=/var/log/nginx/error.log –pid-path=/var/run/nginx/nginx.pid –lock-path=/var/lock/nginx.lock –user=nginx –group=nginx –with-http_ssl_module –with-http_flv_module –with-http_gzip_static_module –http-log-path=/var/log/nginx/access.log –http-client-body-temp-path=/var/tmp/nginx/client/ –http-proxy-temp-path=/var/tmp/nginx/proxy/ –http-fastcgi-temp-path=/var/tmp/nginx/fcgi/ –with-zlib=/usr/src/AHA36x_ver_2_4_0_08312010/zlib/
The bit in red is the important bit, as that the location of the Zlib source files, these are compiled as you run the make for Nginx.
Then its just a simple -
make
make install
and if you have got your ./configure bit right everything should be fine and dandy and exactly where you wanted it.
I Finally managed to get Nginx to function as a proxy, after raiding various websites and hacking about the options suggested here http://www.cyberciti.biz/faq/rhel-linux-install-nginx-as-reverse-proxy-load-balancer/ i'll include the config files for download later.
To do this take the files included in the archive nginxconf.tar and place the files included in it into /etc/nginx or where ever you set the location of the configuration files to be. Take a look at all of them, but for the moment the ones we are interested in are haproxyredirect.conf and options.conf.
options.conf - This is where you can set the G-Zip options, a full definition of these options is available http://wiki.nginx.org/NginxHttpGzipModule
gzip on; <----- Enables G-Zip
gzip_http_version 1.1; <-------- Compression version
gzip_comp_level 9; <---------- Set the Compression level 0 is no compression 9 is the highest that it will allow, the higher the value the more work that the device has to do.
gzip_min_length 0; <------ it will only G-Zip files larger than this size
gzip_types text/plain text/css text/xml text/javascript application/x-javascript image/gif image/jpeg image/jpg image/png image/bmp image/x-icon; <-- types of file to compress
#gzip_disable "MSIE [1-6].(?!.*SV1)"; <---as G-Zip is not compatable with IE 1-6 it checks for it and disables it.
gzip_vary on;
you can adjust the above settings to your requirements.
Next is the haproxyredirect.conf There are 2 lines that you are interested in - these are
upstream nixcraft {
server 127.0.0.1:8081; <-------- which should be the VIP of haproxy this is where the query made on the address below will be passed to.
}
server {
access_log /var/log/access.log main;
error_log /var/log/error.log;
index index.html;
root /usr/local/nginx/html;
listen 192.168.17.45:8080; <---- this is the ip address and port that your users will connect too
and dont forget to restart Nginx after changing the configuration.
I started with a web page that contained an enormous jpeg file, after some discussion over the already compressed file format and how any differences would be negligible. I settled on a 7.3MB bitmap file. Which using gzip compression was able to be reduced to a paltry 182.3KB which impressed the heck out of me!
An additional note is that the way I compiled Nginx means that it had reverted to software compression, handily the tech support guys (as I have already mentioned) for the card know their stuff and were able to help me out. but it does mean that I was able to see the different load times under the different methods of compression. Software the load time was - 1.65 seconds and with the card enabled it was 1.02s it managed to knock half a second off....not bad.
Now the only question remaining is - haproxy then nginx or nginx then haproxy??
Nginx then haproxy - it must be this way to allow persistence to work, thankfully Nginx does not strip the cookies or any of the header information on the return path its just happy to G-Zip everything up on the way through.
My setup was:
- 192.168.17.45:8080 --> Njinx --> 127.0.0.1:8081 --> HAProxy --> 192.168.17.2
Below is the HAProxy configuration, which can be done from the user interface to save you a bit of typing, but here it is if you need it.
global
#uid 99
#gid 99
daemon
stats socket /var/run/haproxy.stat mode 600 level admin
maxconn 40000
ulimit-n 81000
pidfile /var/run/haproxy.pid
defaults
mode http
contimeout 4000
clitimeout 42000
srvtimeout 43000
balance roundrobin
listen VIP_Name 127.0.0.1:8081
mode http
option httpclose
option forwardfor
cookie SERVERID insert nocache indirect
balance leastconn
server RIP_Name 192.168.17.2:80 weight 1 cookie RIP_Name check inter 2000 rise 2 fall 3
server backup 127.0.0.1:9081 backup
option redispatch
option abortonclose
maxconn 40000
If you have done all that correctly you should have a G-Zip compressing proxy!
So do load balancers really need hardware acceleration for G-Zip compression?
Well I hate saying yes and no, So I'm going to go with NO. Yes, this hardware accelerator clearly works. But given the lightning fast speed of current processors its a bit of an overkill. And if you really need to scale your WAF, then you should just load balance a bunch of them.
After all that is what a load balancer is for!