Extending the ModSecurity WAF, with Denial Of Service protection — using HAProxy

The Core Rule Set in ModSecurity has always done a great job of protecting applications from generic attacks such as the OWASP Top 10 security threats, and the CRS has only needed a few minor tweaks over the years to carry out its critical job of protecting you from careless application developers.

But what do you do if you have application-specific firewall demands?

Here we explain why and how we helped one of our customers with a custom set of WAF rules for additional Denial of Service protection with our load balancer, using HAProxy.

Extending ModSecurity WAF: The use case

Our long time partner Metaswitch, desperately needed to stop brute force login attacks on the enterprise phone system, that they supply to large telecom companies. Some sites were also struggling with large scale DOS attacks. And last but not least they wanted to provide WAF protection of the entire aplication.

We helped them solve this problem a long time ago by using custom WAF rules for rate limiting. We also modified a few of the CRS rules to avoid false positives with the Metaswitch application stack. This was very successful and we later extended the functionality for other security issues with version two.

However, for larger customers and/or larger DOS attacks, we found that just using ModSecurity was very inefficient. Why? The state database is not designed for scale and can potentialy make a DOS attack even worse i.e. DOS yourself.

So with the third version that we developed for Metaswitch, we split the DOS functionality out to HAProxy, but kept the logging and application protection functionality in the WAF. Because, as we all know HAProxy is incredibly scaleable and designed for this exact kind of problem.

In this blog, I'll delve into the world of custom WAF rule sets, before explaining how to configure a Custom Rule Set on a Loadbalancer appliance, using the real world example of the Metaswitch rule set my talented colleague Nicholas Turnbull co-developed with them.

The limitations of standard Core Rule Sets

When using a standard core rule set with a Web Application Firewall, it's important to understand that while they provide valuable baseline protection, they can also come with certain limitations.

For example:

• False positives: One of the most common issues with standard Core Rule Sets it that they can sometimes generate false positives. In other words, they can block legitimate traffic. This requires careful tuning and exception handling, explained in detail here: How to train your WAF.
• Common protection: Core rule sets, like the fantastic OWASP Core Rule Set (CRS), are designed to address common web application vulnerabilities, such as SQL injection and cross-site scripting. However, they're designed to be a catch all so may not fully address the unique security needs of every application.
• Evolving security threats: Core rule sets are designed to deal with known knowns. And they are cleverly designed to also deal with new threats, so that they rarely require updates. But this does mean that they are very generic, any specific requirements need to be maualy created.
• The security/performance trade-off: When it comes to core rule sets, more isn't necessarily better. Processing too many rules can create potential latency, meaning the applications themselves that you're trying to protect may not function as intended.

A complex real world example: The Loadbalancer Metaswitch EAS WAF v3

Note, the estimated maintenance window time required for this activity is around 2 hours for a typical deployment.

The previous versions of the Metaswitch EAS WAF Custom Rule set for protecting
CommPortal consist only of a custom set of rules for ModSecurity itself.

Version 3 uses a different approach where rate limiting detection takes place within HAProxy, which has substantial advantages in blocking performance and provides a significant reduction in required system resources. There are three components to this solution:

1. An HAProxy ACL component that is added to each WAF Frontend Layer 7 Virtual IP (VIP) that performs rate limiting and threshold functionality;
2. A shared set of HAProxy backends that are added to the Layer 7 Manual
   Configuration on the appliance; and
3. The ModSecurity Custom Ruleset.

The improved Version 3 rule set has been successfully validated across multiple customers.

How to upgrade to the Metaswitch EAS WAF Custom Rule Set

The following steps must be performed on the Primary load balancer appliance in a maintenance window.

• Navigate to Cluster Configuration > Layer 7 - Manual Configuration.
• Underneath any existing configuration items contained in the Manual Configuration text box, copy and paste the entire content within the HAProxy Manual Configuration Rules section of this document into the editor, which are custom backends used by the HAProxy component of the WAF ruleset.
• Click the Update button.
• When prompted, click the Reload HAProxy button and verify that the configuration has been loaded successfully.
• Next, navigate to Cluster Configuration > Layer 7 - Virtual Services.
• For each of the WAF frontend Virtual IPs protecting a Metaswitch EAS CommPortal service, verify and change the configuration where required as follows:

1. Click Modify next to the WAF frontend VIP.
2. Navigate to the Protocol section, and click Advanced to expand the menu.
3. Ensure that the following options are set:
   Accept Invalid HTTP Requests: Enabled (checked).
   HTTP Request Timeout (DoS Protection): Disabled (unchecked).
4. Navigate to the Persistence section, and click Advanced to expand the menu.
5. Verify that the Persistence Mode is set to X-Forwarded-For and Source IP
   and that the Persistence Timeout to is set to 2100.
6. If the WAF frontend has an associated SSL termination, ensure that Force to
   HTTPS is checked.
7. In the Other section click Advanced to expand the menu. Ensure that the
   options are set as follows:
   Timeout: Enabled (checked)
   Client Timeout: 900000
   Real Server Timeout: 901000
8. Under the ACL Rules section, click the Add Rule button.
9. In the Type dropdown box, select Free Type.
10. Copy and paste the entire HAProxy VIP Rules (ACLs) component of the v3
    rule set into the text box as a single Free Type ACL, replacing the # symbol,
    and click OK. Ensure that the entire set of rules, including the header and
    footer comments at the start and finish, are copied when performing this task.
11. Click Update at the bottom of the page.
12. When prompted, click Reload HAProxy and validate that the configuration
    has been applied successfully. To assist in troubleshooting, please perform
    this for each VIP individually.

• Navigate to Cluster Configuration > WAF - Manual Configuration.
• For each WAF Gateway configured on the appliance that is protecting a CommPortal VIP:

1. Select the VIP name from the drop-down list, which initially shows "None
   Selected".
2. Replace the contents of the text box with the ModSecurity Custom
   Rules component, preserving any additional geolocation or rule exclusion
   rules that may have been added for the specific customer or site deployment.
3. Click Update and then when prompted, click Reload WAF. Please perform
   this for each WAF instance to assist with troubleshooting.

• Perform validation with the customer that the CommPortal works as expected via the load balancer and that SIP client devices can provision successfully.
• Once the new configuration has been validated, repeat the backup process for both appliances and ensure that the customer has this saved for future safekeeping, along with the password set when downloading the backup.

Rollback procedure

If it is determined that a rollback is required (e.g. the process fails and services cannot be brought back online) restore the original appliance configuration from the backup taken in Step 1 of the Update Procedure as follows:

• On the Primary appliance, navigate to Maintenance > Backup & Restore.
• Click on the Restore tab.
• Select the file previously downloaded for the Primary appliance in the file picker.
• Click Restore.
• Once restoration has completed, navigate to Maintenance > Backup & Restore > Synchronisation and click Synchronise configuration with peer.

Alternatively, if an appliance snapshot is available (for Virtual Appliances only), please restore this via the hypervisor.

We need a separate HAProxy stick table for each type of attack!

HAProxy only supports one stick table per backend, and we need a lot more than that! So we work around the problem by creating a bunch of extra backends with the sole purpose of hosting our stick tables.

These are added only once to the Loadbalancer.org Layer 7 > Manual Configuration, and are shared between multiple EAS WAF instances.

# -- Start of Metaswitch EAS Custom Ruleset v3 Manual Configuration Rules

backend rule_1
stick-table type ipv6 size 100k expire 10m store gpt0,http_req_rate(1m) peers
loadbalancer_replication

backend rule_2
stick-table type ipv6 size 100k expire 10m store gpt0,gpc0,gpc0_rate(10m) peers
loadbalancer_replication

backend rule_3
stick-table type ipv6 size 100k expire 10m store gpt0,http_req_rate(1m) peers
loadbalancer_replication

backend rule_4
stick-table type string len 96 size 100k expire 10m store gpt0,http_req_rate(10m)
peers loadbalancer_replication

backend rule_5
stick-table type integer size 100k expire 10m store gpt0,http_req_rate(10m) peers
loadbalancer_replication

backend ps_all_requests
stick-table type ipv6 size 100k expire 24h store gpt0,http_req_rate(30m) peers
loadbalancer_replication

backend ps_config_retrievals
stick-table type ipv6 size 100k expire 24h store gpt0,http_req_rate(3m) peers
loadbalancer_replication

# -- End of Metaswitch EAS Custom Ruleset v3 Manual Configuration Rules

HAProxy DOS rules for each service you're protecting

These HAProxy rules must be added separately to every WAF Frontend VIP that is protecting a CommPortal service. You can simply copy and paste the whole section into a a single Free Type ACL. But we recommend breaking it into chunks for easier editing later.

Now, they might look a little complicated, but they are really quite simple. The only bit that you may wish to configure is the default rate limit settings.

# -- Metaswitch EAS Custom Ruleset v3 HAProxy Rules
acl always_allow src
acl always_deny src
http-request set-var(txn.src) src,ipmask(32,128)
http-request set-var(txn.path) path,url_dec

use-server backup if always_allow
http-request allow if always_allow
http-request deny if always_deny
http-request deny deny_status 429 if { var(txn.src),table_gpt0(ps_all_requests)
eq 1 } || { var(txn.src),table_gpt0(ps_config_retrievals) eq 1 }
http-request deny if { var(txn.src),table_gpt0(rule_1) eq 1 } || {
var(txn.src),table_gpt0(rule_2) eq 1 } || { var(txn.src),table_gpt0(rule_3) eq 1
} || { req.body_param(DirectoryNumber),table_gpt0(rule_4) eq 1 } || {
req.body_param(Password),xxh32,table_gpt0(rule_5) eq 1 }

acl is_login_html var(txn.path) -m end -i login.html
acl is_login var(txn.path) -m end -i /login

acl err_returned res.hdr(Location) -m sub -i error=authenticationfailed
acl err_returned res.hdr(Location) -m sub -i error=retrylimitexceeded

acl is_ps_req var(txn.path) -m beg -i /sip-ps/
acl is_ps_req var(txn.path) -m beg -i /pps/

acl is_config_retrieval var(txn.path) -m reg -i
(?:[[:xdigit:]]{12}(?:\.cfg|(-phone)?\.xml)|(?:config|aastra|tgp.*)\.cfg)$

http-request track-sc0 var(txn.src) table rule_1 if is_login_html
http-request track-sc1 var(txn.src) table rule_2 if is_login
http-request track-sc2 var(txn.src) table rule_3 if is_login METH_POST
http-request track-sc3 req.body_param(DirectoryNumber) table rule_4 if is_login

METH_POST { var(txn.src),table_gpc0_rate(rule_2) gt 2 }
http-request track-sc4 req.body_param(Password),xxh32 table rule_5 if is_login

METH_POST { var(txn.src),table_gpc0_rate(rule_2) gt 2 }
http-request track-sc5 var(txn.src) table ps_all_requests if is_ps_req
http-request track-sc6 var(txn.src) table ps_config_retrievals if
is_config_retrieval

# Configure the default rate limit settings, tune for your environment if required 

http-request sc-set-gpt0(0) 1 if { sc_http_req_rate(0) gt 100 }
http-request sc-set-gpt0(1) 1 if { sc_gpc0_rate(1) ge 20 }
http-request sc-set-gpt0(2) 1 if { sc_http_req_rate(2) gt 500 }
http-request sc-set-gpt0(3) 1 if { sc_http_req_rate(3) gt 20 }
http-request sc-set-gpt0(4) 1 if { sc_http_req_rate(4) gt 20 }
http-request sc-set-gpt0(5) 1 if { sc_http_req_rate(5) gt 7000 }
http-request sc-set-gpt0(6) 1 if { sc_http_req_rate(6) gt 500 }

# Insert a header so ModSecurity can log the attack we've detected

http-request set-header WAF-Block-Rule-1 1 if { sc_get_gpt0(0) eq 1 }
http-request set-header WAF-Block-Rule-2 1 if { sc_get_gpt0(1) eq 1 }
http-request set-header WAF-Block-Rule-3 1 if { sc_get_gpt0(2) eq 1 }
http-request set-header WAF-Block-Rule-4 1 if { sc_get_gpt0(3) eq 1 }
http-request set-header WAF-Block-Rule-5 1 if { sc_get_gpt0(4) eq 1 }

http-request set-header WAF-Block-PS-All-Requests 1 if { sc_get_gpt0(5) eq 1 }

http-request set-header WAF-Block-PS-Config-Retrievals 1 if { sc_get_gpt0(6) eq 1}

http-response sc-inc-gpc0(1) if is_login err_returned

# -- End of EAS Custom Ruleset v3 HAProxy Rules

The ModSecurity custom Rules are now much simpler/more efficient than previous versions

The existing Metaswitch EAS Custom Rule Set rules for each WAF (Cluster
Configuration > WAF > Manual Configuration) should be replaced with the following ruleset,  preserving any custom rules defined for the deployment i.e. for geolocation blocking or rule exclusions etc.

# ------------------------------------------------------------------------------
# Metaswitch Custom WAF Gateway Rule Set
# v3.0.0 (July 2022)
#
# Developer: Andrew Howe ([email protected])
# Original author: Aaron West ([email protected])
# ------------------------------------------------------------------------------
#
# -- Blocking by geographic location -------------------------------------------
#
# Please contact support if you need a licence and example configuration for the proprietary Geolocation database.
#
# Rules must have unique IDs. To avoid ID collisions, use the following range:
# Rule IDs: 1000-1999
#

# ------------------------------------------------------------------------------
# * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
# * * * * * * * * * * * * END OF CONFIGURATION SECTION * * * * * * * * * * * *
# * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
# * * * * * * * * * THE SECTIONS BELOW ARE NOT USER EDITABLE * * * * * * * * *
# * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
# ------------------------------------------------------------------------------
 
#
# -- Define rule set scope -----------------------------------------------------
#
# The WAF rule set restricts the rules of the OWASP Core Rule Set to only
# inspecting traffic related to the CommPortal login page. No other traffic
# should be inspected in this way. This prevents false positives from causing
# the WAF to accidentally block requests that aren't related to what it's
# protecting. These rules almost never require modifying and should not be
# changed unless explicitly instructed to do so.
#
# Rule IDs: 2000-2999
#
SecRule REQUEST_FILENAME "!@rx ^/login" \
"id:2000,\
phase:1,\
pass,\
nolog,\
t:none,t:lowercase,t:normalizePath,\
ctl:ruleRemoveByTag=OWASP_CRS"

#
# -- Logging set up ------------------------------------------------------------
#
# Set up logging for request rate limiting. This is achieved by looking for the
# presence of "WAF-Block-*" HTTP request header fields. The presence of one of
# these indicates that one of the request rate limiting thresholds in HAProxy
# has been exceeded and a new block has been recorded. The following rules write
# human readable messages to the WAF log explaining this, keeping all logging in
# one place (the WAF log).
#
# Rule IDs: 3000-3999
#
SecRule &REQUEST_HEADERS:WAF-Block-Rule-1 "@eq 1" \
"id:3000,\
phase:2,\
pass,\
log,\
msg:'New block applied: IP address blocked from CommPortal login page due to
exceeding the allowed HTTP request rate for /login.html.',\
logdata:'Directory Number: %{ARGS.DirectoryNumber}',\
tag:'METASWITCH/COMMPORTAL_LOGIN/RULE_1/NEW_BLOCK'"
SecRule &REQUEST_HEADERS:WAF-Block-Rule-2 "@eq 1" \
"id:3010,\
phase:2,\
pass,\
log,\
msg:'New block applied: IP address blocked from CommPortal login page due to
exceeding the allowed number of failed log in attempts.',\
logdata:'Directory Number: %{ARGS.DirectoryNumber}',\
tag:'METASWITCH/COMMPORTAL_LOGIN/RULE_2/NEW_BLOCK'"
SecRule &REQUEST_HEADERS:WAF-Block-Rule-3 "@eq 1" \
"id:3020,\
phase:2,\
pass,\
log,\
msg:'New block applied: IP address blocked from CommPortal login page due to
exceeding the allowed number of log in attempts.',\
logdata:'Directory Number: %{ARGS.DirectoryNumber}',\
tag:'METASWITCH/COMMPORTAL_LOGIN/RULE_3/NEW_BLOCK'"
SecRule &REQUEST_HEADERS:WAF-Block-Rule-4 "@eq 1" \
"id:3030,\
phase:2,\
pass,\
log,\
msg:'New block applied: directory number blocked from logging in to
CommPortal due to exceeding the allowed number of log in attempts using the same
directory number.',\
logdata:'Directory Number: %{ARGS.DirectoryNumber}',\
tag:'METASWITCH/COMMPORTAL_LOGIN/RULE_4/NEW_BLOCK'"
SecRule &REQUEST_HEADERS:WAF-Block-Rule-5 "@eq 1" \
"id:3040,\
phase:2,\
pass,\
log,\
msg:'New block applied: password blocked from logging in to CommPortal due to
exceeding the allowed number of log in attempts using the same password.',\
logdata:'Directory Number: %{ARGS.DirectoryNumber}',\
tag:'METASWITCH/COMMPORTAL_LOGIN/RULE_5/NEW_BLOCK'"
SecRule &REQUEST_HEADERS:WAF-Block-PS-All-Requests "@eq 1" \
"id:3050,\
phase:2,\
pass,\
log,\
msg:'New block applied: IP address blocked from provisioning services due to
exceeding the allowed HTTP request rate.',\
tag:'METASWITCH/PROVISIONING_SERVICES/ALL_REQUESTS/NEW_BLOCK'"
SecRule &REQUEST_HEADERS:WAF-Block-PS-Config-Retrievals "@eq 1" \
"id:3060,\
phase:2,\
pass,\
log,\
msg:'New block applied: IP address blocked from provisioning services due to
exceeding the configuration retrieval request rate that an IP address is allowed
to make.',\
tag:'METASWITCH/PROVISIONING_SERVICES/CONFIG_RETRIEVALS/NEW_BLOCK'"

#
# -- Log4j/Log4Shell defense --------------------------------------------------
#
# This addresses exploits against the Log4j library described in several CVEs:
# CVE-2021-44228, CVE-2021-44832, CVE-2021-45046, and CVE-2021-45105.
#
# See https://coreruleset.org/20211213/crs-and-log4j-log4shell-cve-2021-44228/
# for details.
#
# ALL traffic passes through these rules, as agreed with Metaswitch.
#
# Rule IDs: 4000-4999
#
#
# -- Log4Shell rule 1
#
# CRS 4.0.0-rc1
# 932130: Unix shell expressions (detects $(foo), ${foo}, >(foo), etc.)
# Modified: disruptive action (block -> deny), log message, removed OWASP_CRS
# tag, removed PL tag, removed anomaly scoring actions
SecRule
REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|ARGS_NAMES|ARGS|XM
L:/* "@rx (?:\$(?:\((?:\(.*\)|.*)\)|\{.*})|[<>]\(.*\))" \
"id:4000,\
phase:2,\
deny,\
capture,\
t:none,t:cmdLine,\
msg:'Log4Shell defense - Remote Command Execution: Unix Shell Expression
Found',\
logdata:'Matched Data: %{TX.0} found within %{MATCHED_VAR_NAME}:
%{MATCHED_VAR}',\
tag:'application-multi',\
tag:'language-shell',\
tag:'platform-unix',\
tag:'attack-rce',\
tag:'capec/1000/152/248/88',\
tag:'PCI/6.5.2',\
ctl:auditLogParts=+E,\
ver:'OWASP_CRS/4.0.0-rc1',\
severity:'CRITICAL'"
#
# -- Log4Shell rule 2
#
# CRS 4.0.0-rc1
# 932131: Unix shell expressions (adds User-Agent and Referer header inspection)
# Modified: disruptive action (block -> deny), log message, removed OWASP_CRS
# tag, removed PL tag, removed anomaly scoring actions
SecRule REQUEST_HEADERS:User-Agent|REQUEST_HEADERS:Referer "@rx
(?:\$(?:\((?:\(.*\)|.*)\)|\{.*})|[<>]\(.*\))" \
"id:4010,\
phase:1,\
deny,\
capture,\
t:none,t:cmdLine,\
msg:'Log4Shell defense - Remote Command Execution: Unix Shell Expression
Found',\
logdata:'Matched Data: %{TX.0} found within %{MATCHED_VAR_NAME}:
%{MATCHED_VAR}',\
tag:'application-multi',\
tag:'language-shell',\
tag:'platform-unix',\
tag:'attack-rce',\
tag:'capec/1000/152/248/88',\
tag:'PCI/6.5.2',\
ctl:auditLogParts=+E,\
ver:'OWASP_CRS/4.0.0-rc1',\
severity:'CRITICAL'"
#
# -- Log4Shell rule 3
#
# CRS 4.0.0-rc1
# 944151: Log4j/Log4Shell detection (detects ${jndi:, nested use of ${, etc.)
# Modified: disruptive action (block -> deny), log message, removed OWASP_CRS
# tag, removed PL tag, removed anomaly scoring actions
SecRule
REQUEST_LINE|ARGS|ARGS_NAMES|REQUEST_COOKIES|REQUEST_COOKIES_NAMES|REQUEST_HEADER
S|XML:/*|XML://@* "@rx
(?i)(?:\$|&dollar;?)(?:\{|&(?:lbrace|lcub);?)(?:[^}]*(?:\$|&dollar;?)(?:\{|&(?:lb
race|lcub);?)|(?:jndi|ctx))" \
"id:4020,\
phase:2,\
deny,\
t:none,t:urlDecodeUni,t:jsDecode,t:htmlEntityDecode,\
log,\
msg:'Log4Shell defense - Potential Remote Command Execution: Log4j /
Log4shell',\
tag:'application-multi',\
tag:'language-java',\
tag:'platform-multi',\
tag:'attack-rce',\
tag:'capec/1000/152/137/6',\
tag:'PCI/6.5.2',\
ver:'OWASP_CRS/4.0.0-rc1',\
severity:'CRITICAL'"
# END OF RULES

And that's it!

Conclusion: You should always use the right tool for the job.

With this somewhat complex example I've demonstrated that you can effectivley combine HAProxy and ModSecurity to get the best of both tools. Hopefully you won't be dealing with something quite this tricky. However, most of the real world examples we encounter are just as complex as this, if not more!

Regardless of whether you choose a standard or customized CRS, it’s important to regularly tune and monitor your WAF logs to minimize false positives and ensure strong protection.

Because a false positive, or a performance degradation is the same as doing a DOS attack on your own application!

Supplementing standard core rule sets with custom rules, tailored to the needs of your specific application has many benefits. But they can be hard to set up correctly, so we recommend speaking to an expert first.

For help with any of the above, feel free to speak to us, or drop a question in the comments below.