G-Zip Compression and Load Balancing

Updated on 5 mins

WHERE - Blog Sector

A couple of our customers have asked if our appliances would do G-Zip compression in the past we haven't given it much thought. Then out of the blue a company offered us a card to test with aha.com and some of us in the office welcoming the opportunity to meddle with anything new jumped at the chance.

They provided us with a AHA360-PCIe 2.5 Gbits/sec GZIP Compression/Decompression Accelerator


  • Open standard compression algorithm (GZIP/Deflate)
  • Performs both compression and decompression
  • Compresses and decompresses at a throughput up to 2.5 Gbits/sec
  • Uses the AHA3610
  • x4 PCI-Express interface
  • ASCII text compresses greater than 3.0:1
  • 3.6:1 Canterbury corpus compression ratio
  • Scatter Gather DMA
  • 64-bit OS support
  • Linux and Windows reference drivers with source code
  • SMP and Hypterthreading support
  • 32-bit and 64-bit Linux kernel support

As we all like stats here are a few numbers from the initial performance of the card.

Firstly I compared the G-Ziping (im sure thats not a real word) of a 533MB iso file with the same process running through the card. The numbers are quite good, as you can see it almost entirely offloaded the process to the card and ran the compression though in half the time it took the SW version of G-Zip

time ./ahazip -f=/usr/src/file31.iso -o=/usr/src/compressed.gz
real    0m15.566s
user    0m0.015s
sys    0m1.674s
[root@lbmaster bin]# time gzip /usr/src/file31.iso
real    0m38.915s
user    0m33.966s
sys    0m1.084s

I then decided to get a bit more adventurous and compress 2 files at once the performance of the card was quite heavily affected.

[root@lbmaster bin]# time ./ahazip -f=/usr/src/file32.iso -o=/usr/src/compressed1.gz
real    0m35.888s
user    0m0.011s
sys    0m1.717s
[root@lbmaster bin]# time ./ahazip -f=/usr/src/file31.iso -o=/usr/src/compressed.gz
real    0m28.262s
user    0m0.012s
sys    0m1.611s

and the results for G-Zipping the 2 files

[root@lbmaster bin]# time gzip /usr/src/file31.iso

real    0m58.823s
user    0m43.235s
sys    0m1.480s

[root@lbmaster bin]# time gzip /usr/src/file32.iso
real    1m7.898s
user    0m45.205s
sys    0m1.325s

What impressed me the most was the incredibly low overhead when using the card compared with using the system to do the compression. It barely touched the CPU, leaving the system to get on with other important jobs.

Back to the main event.........

Here is how I did it - you need your G-Zip card to use the standard Zlib compression library, and obviously you need a proxy that can support it. In this instance we are using Nginx as in its current guise HAProxy (both HAProxy and Nginx are available on our appliances) does not support G-Zipping the datastream, so we had to proxy the proxy in order to get it to work. Ill assume your happy in installing it and getting everything running. Below is the procedure for getting the card we were supplied with to work with Nginx.

I had to use a slightly modified configure file as well as a modified makefile which were kindly given to me by their excellent support team.

The 2 files are here

and you just need to replace them with the existing ones in the Zlib directory that came with the drivers that you downloaded from their site.

To configure Nginx I used the following command – ./configure –prefix=/usr –sbin-path=/usr/sbin/nginx –conf-path=/etc/nginx/nginx.conf –error-log-path=/var/log/nginx/error.log –pid-path=/var/run/nginx/nginx.pid –lock-path=/var/lock/nginx.lock –user=nginx –group=nginx –with-http_ssl_module –with-http_flv_module –with-http_gzip_static_module –http-log-path=/var/log/nginx/access.log –http-client-body-temp-path=/var/tmp/nginx/client/ –http-proxy-temp-path=/var/tmp/nginx/proxy/ –http-fastcgi-temp-path=/var/tmp/nginx/fcgi/ –with-zlib=/usr/src/AHA36x_ver_2_4_0_08312010/zlib/

The bit in red is the important bit, as that the location of the Zlib source files, these are compiled as you run the make for Nginx.

Then its just a simple -

make install

and if you have got your ./configure bit right everything should be fine and dandy and exactly where you wanted it.

I Finally managed to get Nginx to function as a proxy, after raiding various websites and hacking about the options suggested here http://www.cyberciti.biz/faq/rhel-linux-install-nginx-as-reverse-proxy-load-balancer/ i'll include the config files for download later.

To do this take the files included in the archive nginxconf.tar and place the files included in it into /etc/nginx or where ever you set the location of the configuration files to be. Take a look at all of them, but for the moment the ones we are interested in are haproxyredirect.conf and options.conf.

options.conf - This is where you can set the G-Zip options, a full definition of these options is available http://wiki.nginx.org/NginxHttpGzipModule

gzip    on; <----- Enables G-Zip
gzip_http_version       1.1; <-------- Compression version
gzip_comp_level         9; <---------- Set the Compression level 0 is no compression 9 is the highest that it will allow, the higher the value the more work that the device has to do.
gzip_min_length 0; <------ it will only G-Zip files larger than this size
gzip_types text/plain text/css text/xml text/javascript application/x-javascript image/gif image/jpeg image/jpg image/png image/bmp image/x-icon; <-- types of file to compress
#gzip_disable "MSIE [1-6].(?!.*SV1)"; <---as G-Zip is not compatable with IE 1-6 it checks for it and disables it.
gzip_vary on;

you can adjust the above settings to your requirements.

Next is the haproxyredirect.conf There are 2 lines that you are interested in - these are

upstream nixcraft  {
server; <-------- which should be the VIP of haproxy this is where the query made on the address below will be passed to.
server {
access_log  /var/log/access.log main;
error_log   /var/log/error.log;
index       index.html;
root        /usr/local/nginx/html;
listen; <---- this is the ip address and port that your users will connect too

and dont forget to restart Nginx after changing the configuration.

I started with a web page that contained an enormous jpeg file, after some discussion over the already compressed file format and how any differences would be negligible. I settled on a 7.3MB bitmap file. Which using gzip compression was able to be reduced to a paltry 182.3KB which impressed the heck out of me!

An additional note is that the way I compiled Nginx means that it had reverted to software compression, handily the tech support guys (as I have already mentioned) for the card know their stuff and were able to help me out. but it does mean that I was able to see the different load times under the different methods of compression. Software the load time was - 1.65 seconds and with the card enabled it was 1.02s it managed to knock half a second off....not bad.

Now the only question remaining is - haproxy then nginx or nginx then haproxy??

Nginx then haproxy - it must be this way to allow persistence to work, thankfully Nginx does not strip the cookies or any of the header information on the return path its just happy to G-Zip everything up on the way through.

My setup was:

  • --> Njinx --> --> HAProxy -->

Below is the HAProxy configuration, which can be done from the user interface to save you a bit of typing, but here it is if you need it.

#uid 99
#gid 99
stats socket /var/run/haproxy.stat mode 600 level admin
maxconn 40000
ulimit-n 81000
pidfile /var/run/haproxy.pid
mode    http
contimeout      4000
clitimeout      42000
srvtimeout      43000
balance roundrobin
listen  VIP_Name
mode    http
option  httpclose
option  forwardfor
cookie  SERVERID insert nocache indirect
balance leastconn
server RIP_Name weight 1 cookie RIP_Name check  inter 2000 rise 2 fall 3
server  backup backup
option redispatch
option abortonclose
maxconn 40000

The config files are rather lengthy so Ive Tarred them up so you can download them at your leisure -

If you have done all that correctly you should have a G-Zip compressing proxy!