The "ARP problem" for Layer 4 DR (Direct Routing) mode is something that needs to be solved for each of your Real Servers in the virtual service.
What is the "ARP problem"?
Layer 4's DR mode works by changing the MAC address of the inbound packets to match the Real Server selected by the load balancing algorithm.
To enable DR mode to operate, each Real Server must be configured to accept packets destined for both the VIP address and the Real Server’s IP address (RIP). This is because in DR mode the destination address of load-balanced packets is the VIP address, whilst for other traffic such as health checks (administration traffic etc) it’s the Real Server’s own IP address (the RIP). The service/process (e.g. Apache, IIS, etc.) must also respond to both addresses.
Each Real Server must be configured so that it does not respond to ARP requests for the VIP address - only the load balancer should do this.
Configuring the Real Servers in this way is referred to as "solving the ARP problem". The steps required depend on the OS used and the chosen method selected. More information on the ARP problem, and the solutions available can be found in our Appliance Administration guide here.
For the Linux operating system, there are several methods that can be used to solve the ARP problem, including with iptables and using sysctl.
This post provides a couple of example Ansible playbooks that can be used to automate the resolution of the ARP problem, and aid in the automated deployment of new Real Servers as required.
Solving the ARP problem with Ansible using iptables.
The below example uses iptables to solve the problem, and we use one of Ansible's built-in modules (iptables) to make the changes required. An example of a single playbook can be seen below:
---
- hosts:
- webservers
vars:
VIPS: # List of all IPv4 VIP Addresses.
- 192.168.0.40
- 192.168.0.45
tasks:
- name: Solve ARP Problem for all VIP Addresses with IPTables
ansible.builtin.iptables:
table: nat
chain: PREROUTING
destination: '{{ item }}'
jump: REDIRECT
comment: Redirect any incoming packets destined for the Virtual Service '{{ item }}' locally.
become: yes
with_items:
- "{{ VIPS }}"
In this playbook, we are telling Ansible to make these changes to all servers defined in the "webservers" host group (grouping hosts together is a great benefit of using Ansible).
This playbook makes use of variables - in this case, a single variable is called "VIPS". This variable should contain a list of all the IPv4 virtual IP addresses that this Real Server should handle. This variable is required, and can't be omitted or left empty!
The playbook will modify the running iptables, to redirect any incoming traffic (packets) that are destined for the VIP Address, to the local server. When the playbook completes, you should then be able to route traffic to your server correctly from the load balancer while using Layer 4 DR Mode.
As with running the iptables command on the server directly, this method is not persistent and would be lost after a reboot. Additionally, this method is only suitable for IPv4 VIPs.
Solving the ARP problem with Ansible using sysctl.
A more persistent method than "iptables", would be the "sysctl" method.
This makes changes to the sysctl.conf file on the Real Server that is applied on every boot. Additionally, the IP Address of each of the VIPs, is then added to the loopback adapter. Unlike the "iptables" method, this works for both IPv4 and IPv6 traffic.
Note: While the changes to the sysctl.conf configuration values are persistent, the addition of the VIP addresses to the loopback adapter is not persistent, and will not remain after a reboot. The example Ansible playbook below can be expanded on to include the automated creation of this script. However, this process can vary from distribution to distribution, as such it's beyond the scope of this playbook.
An example Ansible playbook that utilizes this method would look like this:
---
- hosts:
- webservers
vars:
VIPS_IPv4: # (Optional) List of all IPv4 VIP Addresses.
- 192.168.0.40
- 192.168.0.45
VIPS_IPv6: # (Optional) List of all IPv6 VIP Addresses.
- 2001:0db8:85a3:0000:0000:8a2e:0370:7334
INTERFACES: # List of all Interfaces on the Real Servers.
- eth0
- eth1
LOOPBACK_NAME: lo
tasks:
- name: Add arp_ignore (IPv4) to sysctl.conf for All Interfaces
lineinfile:
path: /etc/sysctl.conf
regexp: '^net.ipv4.conf.all.arp_ignore='
line: net.ipv4.conf.all.arp_ignore=1
- name: Add arp_ignore (IPv4) to sysctl.conf for Interfaces
lineinfile:
path: /etc/sysctl.conf
regexp: '^net.ipv4.conf.{{ item }}.arp_ignore='
line: net.ipv4.conf.{{ item }}.arp_ignore=1
with_items: "{{ INTERFACES }}"
when: INTERFACES is defined and INTERFACES != None
- name: Add arp_announce (IPv4) to sysctl.conf for All Interfaces
lineinfile:
path: /etc/sysctl.conf
regexp: '^net.ipv4.conf.all.arp_announce='
line: net.ipv4.conf.all.arp_announce=2
- name: Add arp_announce (IPv4) to sysctl.conf for Interfaces
lineinfile:
path: /etc/sysctl.conf
regexp: '^net.ipv4.conf.{{ item }}.arp_announce='
line: net.ipv4.conf.{{ item }}.arp_announce=1
with_items: "{{ INTERFACES }}"
when: INTERFACES is defined and INTERFACES != None
- name: Configure dad_transmits (IPv6) in sysctl.conf for Interfaces
lineinfile:
path: /etc/sysctl.conf
regexp: '^net.ipv6.conf.lo.dad_transmits='
line: net.ipv6.conf.lo.dad_transmits=0
- name: Configure accept_dad (IPv6) in sysctl.conf for Interfaces
lineinfile:
path: /etc/sysctl.conf
regexp: '^net.ipv6.conf.lo.accept_dad='
line: net.ipv6.conf.lo.accept_dad=0
- name: Ensure correct file ownership, group and permissions are set
file:
path: /etc/sysctl.conf
owner: root
group: root
mode: '0644'
- name: Apply SysCtl Settings
shell: /sbin/sysctl -p
- name: Add IPv4 VIP Addresses to the Loopback Adapter
shell: ip addr add dev {{ LOOPBACK_NAME }} {{ item }}/32
with_items: "{{ VIPS_IPv4 }}"
when: VIPS_IPv4 is defined and VIPS_IPv4 != None
- name: Add IPv6 VIP Addresses to the Loopback Adapter
shell: ip addr add dev {{ LOOPBACK_NAME }} {{ item }}/128
with_items: "{{ VIPS_IPv6 }}"
when: VIPS_IPv6 is defined and VIPS_IPv6 != None
Similar to the "iptables" example playbook, this playbook makes use of variables to control the configuration that is applied to the sysctl.conf file, as well as adding the VIP addresses to the loopback adapter.
The "VIPS_IPV4" variable, is an optional list of all the IPv4 virtual IP addresses that this Real Server should handle. If no IPs are provided or the variable is omitted, no IPv4 addresses will be configured in the loopback adapter. However, the IPv4 changes will still be applied to the sysctl.conf file.
The "VIPS_IPV6" variable, is an optional list of all the IPv6 virtual IP addresses that this Real Server should handle. If no IPs are provided or the variable is omitted, no IPv6 addresses will be configured in the loopback adapter. However, the IPv6 changes will still be applied to the sysctl.conf file.
The "INTERFACES" variable, is an optional list, containing the names of all the network interfaces on the Real Servers, to add the IPv4 ARP configuration. For example "eth0", or "eth1" or even "enp0s3" etc.
The "LOOPBACK_NAME" variable, is a string variable, that contains the name of the loopback adapter on the Real Servers (this is typically "lo"). This variable is required, and can't be omitted or left empty.
This playbook will add the IPv4 ARP configuration for "all" interfaces, as well as any provided interfaces to the sysctl.conf file. If the setting already exists in the file, it will be updated to use the value in the playbook (e.g. enabling it, if it was previously disabled). Additionally, it will then configure the DAD (Duplicate Address Detection) values for IPv6.
Once all values have been configured in the sysctl.conf file, it will ensure the file permissions and ownership is correct, and then apply the systctl changes. Finally, the playbook will then add any IPv4 and IPv6 VIP addresses to the loopback adapter.
When the playbook completes, you should then be able to route traffic to your server correctly from the load balancer while using Layer 4 DR mode.
Both playbooks have been tested against a Debian Server (running Debian Buster), with Nginx and a Wordpress website, however, this should also work for most modern Linux distributions.
Tip: The example playbooks above can be split out into an Ansible role as well, allowing for more flexibility and easy integration with any existing roles you may have.