Throughout the last year, we’ve had multiple enquiries from our customers asking us to help them configure the load balancer for VMware’s AirWatch Mobile Access Gateway (MAG) and Secure Email Gateway (SEG).

Introduction

AirWatch is a suite of applications that ensures secure connections between the enterprise user’s mobiles to the application servers, protecting information and preventing leaks by securing the mobile devices.

This blog post will be primarily focused on installing and configuring HAProxy on a CentOS box to provide secure, highly available service for AirWatch Mobile Access Gateway and Secure Email Gateway.

Deploying HAProxy on Centos

To install HAProxy, I have used the following command: yum install haproxy at the time of writing this blog. The command will install HAProxy version 1.8.15

On the load balancer that is running on CentOS Linux release 8.1.1911 (Core), by default the built-in firewall daemon (Firewalld) is blocking all the ports. So, I have opened the ports used by Airwatch MAG and SEG services as follows:

firewall-cmd --permanent --add-port=443/tcp
firewall-cmd --permanent --add-port=2010/tcp
firewall-cmd --permanent --add-port=2020/tcp
firewall-cmd --reload
firewall-cmd --list-ports

In RHEL/CentOS variants, the SELinux feature is enabled by default and because we don't want it to block particular services/applications needed for this lab, we make temporary permissive by running setenforce 0

Configuration

The configuration file for HAProxy located under /etc/haproxy/haproxy.cfg is:

global
        daemon
        log         127.0.0.1 local2     #Log configuration
        chroot      /var/lib/haproxy
        pidfile     /var/run/haproxy.pid
        maxconn     4000
        user        haproxy
        group       haproxy
        stats socket /var/lib/haproxy/stats

defaults
        mode                    http
        log                     global
        option                  tcplog
        option              dontlognull
        retries             3
        maxconn                 10000
        option              redispatch
        timeout connect 4s
        timeout client 5m
        timeout server 5m

listen stats
	bind *:8080
        mode http
        option forwardfor
        option httpclose
        stats enable
        stats show-legends
        stats refresh 5s
        stats uri /stats
        stats realm Haproxy\ Statistics
        stats auth loadbalancer:loadbalancer
        stats admin if TRUE

listen AirWatchMAG443
	bind 192.168.1.100:443 transparent
	mode tcp
	balance leastconn
	stick on src
	stick-table type ip size 10240k expire 60m
	server backup 127.0.0.1:9081 backup  non-stick 
	option httpchk HEAD / HTTP/1.0\r\nHost:\ airwatchmag.domain.com\r\n
	option redispatch
	option abortonclose
	server RIP1MAG443 192.168.1.101:443  weight 100  check port 443 check-ssl verify none inter 4000  rise 2  fall 2  slowstart 8000 minconn 0  maxconn 0  on-marked-down shutdown-sessions 
	server RIP2MAG443 192.168.1.102:443  weight 100  check port 443 check-ssl verify none inter 4000  rise 2  fall 2  slowstart 8000 minconn 0  maxconn 0  on-marked-down shutdown-sessions 

listen AirWatchMAG2010
	bind 192.168.1.100:2010 transparent
	mode tcp
	balance leastconn
	stick on src
	stick-table type ip size 10240k expire 60m
	server backup 127.0.0.1:9081 backup  non-stick 
	option redispatch
	option abortonclose
	server RIP1MAG2010 192.168.1.101:2010  weight 100  check port 2010 inter 4000  rise 2  fall 2  slowstart 8000 minconn 0  maxconn 0  on-marked-down shutdown-sessions 
	server RIP2MAG2010 192.168.1.102:2010  weight 100  check port 2010 inter 4000  rise 2  fall 2  slowstart 8000 minconn 0  maxconn 0  on-marked-down shutdown-sessions 

listen AirWatchMAG2020
	bind 192.168.1.100:2020 transparent
	mode tcp
	balance leastconn
	stick on src
	stick-table type ip size 10240k expire 60m 
	server backup 127.0.0.1:9081 backup  non-stick 
	option redispatch
	option abortonclose
	server RIP1MAG2020 192.168.1.101:2020  weight 100  check port 2020 inter 4000  rise 2  fall 2  slowstart 8000 minconn 0  maxconn 0  on-marked-down shutdown-sessions 
	server RIP2MAG2020 192.168.1.102:2020  weight 100  check port 2020 inter 4000  rise 2  fall 2  slowstart 8000 minconn 0  maxconn 0  on-marked-down shutdown-sessions 

listen AirWatchSEG443
	bind 192.168.1.200:443 transparent
	mode tcp
	balance leastconn
	stick on src
	stick-table type ip size 10240k expire 60m
	server backup 127.0.0.1:9081 backup  non-stick 
	option httpchk HEAD / HTTP/1.0\r\nHost:\ airwatchseg.domain.com\r\n
	timeout client 42000
	timeout server 43000
	option redispatch
	option abortonclose
	server RIP1SEG443 192.168.1.101:443  weight 100  check port 443 check-ssl verify none inter 4000  rise 2  fall 2  slowstart 8000 minconn 0  maxconn 0  on-marked-down shutdown-sessions 
	server RIP2SEG443 192.168.1.102:443  weight 100  check port 443 check-ssl verify none inter 4000  rise 2  fall 2  slowstart 8000 minconn 0  maxconn 0  on-marked-down shutdown-sessions 

The persistence method used across all the service is source IP persistence stick on src. Essentially, a device will keep it's connection in a stick table stick-table to a particular real server for 60 minutes from the moment it times out.

Note that for the virtual services running on port 443, the negotiate HTTPS HEAD health check method has been used against the FQDN. Yes, I said HTTPS. Although it says explicitly HTTP option httpchk, the connection is re-encrypted to backend check-ssl verify none in order to get this health check working. Otherwise, the load balancer would send a plain HTTP request which would cause a failed health check because the real servers are expecting encrypted traffic (HTTPS).

Of course, this health check type is not mandatory, and it can be set to something simple as configuring the load balancer to establish just a connection to the real server's port to ensure that the service is running check port 2010.

An interesting pitfall that I came across when writing the configuration file is that HAProxy doesn't like indentation created using space and tabs, and throws an error regarding an 'unknown keyword' when it attempts to start; instead indentation created using only tabs works perfectly fine. I had to learn the hard way that the devs of HAProxy are tabs advocates in the Spaces vs Tabs debate.

Recommendation

It would be most advisable to use a deployment type that will have source IP transparency so that in case a mobile device is compromised, it can be blocked and located. This is easy on our appliance, just use Layer 4 Direct Routing mode.

In order to do achieve source IP transparency with HAProxy, you would need to use a two-arm transparent TPROXY configuration - which can get painful :-).
So follow this guide first if you need to get it working as non-transparent and then read this blog on how to use TPROXY with HAProxy (obviously, that's also supported easily on our appliance but we prefer Layer 4 DR mode where possible.)

Reference links:
https://docs.vmware.com/en/VMware-Workspace-ONE-UEM/1810/WS1-Recommend-Arch.pdf