There's a saying you've probably heard: "Give a man a fish, and you feed him for a day. Teach a man to fish, and you feed him for a lifetime."
Health checks are an important part of load balancing your application, and in many other circumstances too. We are often asked to write custom checks, and of course we always go above and beyond to provide the most simple, most complete check we can come up with — no matter the application.
But if you can write your own custom health check for one of our appliances, that's an invaluable tool you can use time and time again. This blog is here to 'teach you to fish'.
Loadbalancer.org were the original sponsors of the external health check mechanism in HAProxy. We think it's an invaluable tool when you need something a bit special.
We also wanted to make sure that the external health check in HAProxy was compatible with Ldirectord used by Loadbalancer.org for layer 4 load balancing with Linux Virtual Server (LVS).
The actual commands to use in your HAProxy configuration file are pretty simple:
option external-check
external-check command /var/lib/loadbalancer.org/check/examplecheck
HAProxy will then execute this as a shell command and automatically present the variables the script requires for each backend server to be checked.
examplecheck 192.168.100.100 80 192.168.100.0 80
The content of the variables passed to the script are as follows:
$1 = Virtual Service IP (VIP)
$2 = Virtual Service Port (VPT)
$3 = Real Server IP (RIP)
$4 = Real Server Port (RPT)
$5 = Check Source IP
Next we need to know what language we will be using for the health check. In this example I shall keep it simple and use bash.
You should use #!/bin/bash for portability. This is because different *nixes put bash in different places. You can use any scripting language that you are comfortable with.
What about the exit codes from the health check?
These are important, and there is a little bit of a difference between layer 4 and layer 7. Layer 7 expects a silent exit, and an exitcode of 0 as well as a PATH varible defined so it knows where the commands are. Layer 4 does not have this requirement and allows having text output to the console; however it does expect an exitcode of 0 for a passed healthcheck.
If you're using a Loadbalancer.org appliance then put the script in the correct location:
/var/lib/loadbalancer.org/check/
You can see a selection of the other scripts I have here:
Any scripts in this folder are automatically available via the web interface for both layer 4 and layer 7. Layer 4 has an extra check port option to allow for firewall marks and multi-port VIPs.
The layer 4 external health checks are available here:
Layer 7 does not require the check port as it automatically checks the first port listed within the VIP.
If you use a multi-port layer 7 VIP, the check string is different to the single port way.
Single port would start the check as below:
/var/lib/loadbalancer.org/check/examplecheck 172.31.1.99 80 172.31.1.100 80
However, the multi-port VIP check does not know the real server port and uses the first port in the VIP and is seen as below:
/var/lib/loadbalancer.org/check/examplecheck 172.31.1.99 80 172.31.1.100 0
Now we have all the information we need, let's start with an example health check that will work for both layer 4 and layer 7. This example will check something that we do not already provide as a built-in script.
#!/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
VIP=$1
VPT=$2
RIP=$3
if [ "$4" eq "" ]; then # check if $4 empty (Ie. a Multport VIP)
RPT=$VPT # We are multiport - use the check port or VIP first port
else
RPT=$4
fi
Now we have the basis to start the check with we need to decide what check we should actually make.
CHECK_HOST="example.com"
CHECK_STRING="text to find"
# Build curl options variable
CURL_OPTS="--resolve ${CHECK_HOST}:${RPT}:${RIP}"
# Run curl with appropriate options
curl ${CURL_OPTS} -H 'Host: '${CHECK_HOST}'' -m 2 -k https://${CHECK_HOST}/${CHECK_PATH} 2>/dev/null | grep -q "${CHECK_STRING}"
exit $?
This will perform a curl SNI check against example.com.
Now when we put the entire check together it looks like this:
#!/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
VIP=$1
VPT=$2
RIP=$3
if [ "$4" eq "" ]; then # check if $4 empty (Ie. a Multport VIP)
RPT=$VPT # We are multiport - use the check port or VIP first port
else
RPT=$4
fi
CHECK_HOST="example.com"
CHECK_STRING="text to find"
# Build curl options variable
CURL_OPTS="--resolve ${CHECK_HOST}:${RPT}:${RIP}"
# Run curl with appropriate options
curl ${CURL_OPTS} -H 'Host: '${CHECK_HOST}'' -m 2 -k https://${RIP}/${CHECK_PATH} 2>/dev/null | grep -q "${CHECK_STRING}"
exit $?
Don't forget the exit code!
You will see the last line is exit $?
This should present the exit code of 0 for a healthy server.
Any other number will cause a health check failure.
If you have any questions about the above, don't hesitate to get in touch.
References:
https://docs.haproxy.org/2.8/configuration.html#external-check%20command
https://www.loadbalancer.org/blog/load-balancing-dicom-pacs-health-check/
https://www.loadbalancer.org/blog/ntlm-authenticating-proxy-check-script/
https://www.loadbalancer.org/blog/microsoft-windows-print-spooler-and-any-other-windows-service-health-check/