Wow, I just came to update this with the latest version and was shocked to notice the blog is almost 10 years old! — We keep finding new uses for our HAProxy Feedback Agent.. But, please let me know what you use it for in the comments section :-).
So what's a load balancer feedback agent check and why do I need one?
In general when you are load balancing a cluster you can evenly spread the connections through the cluster and you get pretty consistent and even load balancing. However with some applications such as RDS (Microsoft Terminal Servers), you might get very high load from just a few users doing heavy work. This in turn could result in slow performance for other users on the same server.
The solution to this problem is to use some kind of server load feedback agent check.
The Loadbalancer.org appliance has had an adaptive feedback agent for a while now. However with a LOT of help from Simon Horman (of UltraMonkey fame) — we've managed to integrate the functionality into the main open source branch of HAproxy.
We also thought it would be a good idea to open source our previous work on Ldirectord/LVS, make it compatible with HAProxy, and release our Windows service code under the GPL License.
UPDATE: The Loadbalancer.org feedback agent code is now fully supported in HAProxy 1.5-dev21.
And because it's in the mainline branch, all future versions as well.
You can find the latest documentation for the HAProxy agent check here.
How do I install the feedback agent on Windows?
If you're not using Windows Server — You can jump straight to the Linux Agent here.
Otherwise, simply download the latest Windows Feedback Agent installation package here:
http://downloads.loadbalancer.org/agent/loadbalanceragent.msi (v4.6.0) [Updated 01/09/2022]
Upgrading to the new version will install a new generic configuration. If you have made changes to the default thresholds in the configuration file. Please make a copy of these. You can always contact Loadbalancer.org support for assistance before upgrading.
The feedback agent functionality is available in all versions of the Loadbalancer.org appliance software for both layer 4 (Ldirectord) and layer 7 (HAProxy).
Just downloaded and installed the msi file from the link above, accept all the usual security warnings. It's an almost silent install, once the progress bar disapears the agent is installed as a service and starts running with the default setting immediately.
you can find the client application in the start menu it's called "Feedback Agent Service Monitor". Or if you want the specific file location it's here:
C:\ProgramData\LoadBalancer.org\LoadBalancer\monitor.exe
The agent should already be responding to telnet on port 3333 (However, you may need to make an exception for that port in your Windows firewall).
Then make sure you modify your virtual server on the load balancer to poll the feedback agent:
Then on the Windows server client:
You can change the 'mode' setting to drain then 'apply settings and restart' and HAProxy will then set the status to agent drain (green) i.e.:
Or you can set the 'mode' to halt then 'apply settings and restart' and HAProxy will then immediately set the status to DOWN (red) i.e.:
When the agent is running in normal mode it will report back the percentage idle of the system based on the settings in the feedback agent XML file:
C:\ProgramData\LoadBalancer.org\LoadBalancer\config.xml
<xml>
<Cpu>
<ImportanceFactor value="1" />
<ThresholdValue value="100" />
</Cpu>
<Ram>
<ImportanceFactor value="0" />
<ThresholdValue value="100" />
</Ram>
<TCPService>
<Name value="HTTP" />
<IPAddress value="*" />
<Port value="80" />
<MaxConnections value="0" />
<ImportanceFactor value="0" />
</TCPService>
<ReadAgentStatusFromConfig value="False" />
<ReadAgentStatusFromConfigInterval value="5" />
<AgentStatus value="Normal" />
<Interval value="10" />
</xml>
Notice that you can control both the importance of CPU & RAM utilization and also a threshold, so the following logic is used:
- If CPU importance = 0 then ignore, 0.5 means give 50% importance, 1 means 100% importance
- If RAM importance = 0 then ignore etc.
If ThresholdValue is reached on any monitor then immediately go into DRAIN mode (a value of 0 means no threshold is set).
This can be very useful if you have a small number of RDP sessions using a lot of RAM, simply set a ThresholdValue of 85, then as soon as memory crosses that threshold no new users will be sent to that server.
Otherwise to calculate the percentage idle reported by the agent would be to divide the utilization by the number of factors involved i.e.
If you are using two services then:
- utilization = utilization + cpuLoad * cpuImportance%;
- utilization = utilization + ramOccupied * ramImportance%;
- utilization = utilization / 2
So if importance was 1 for both cpu and ram you would only get 0% reported if both CPU and RAM were 100%. (actually it gets a bit weird if one is already reporting 0%, but lets not worry about that....
And if the importance is zero then ignore completely i.e.
- AS: utilization = utilization + cpuLoad * cpuImportance%;
- IF: utilization = utilization + ramOccupied * 0 (importance is zero so ignore)
- THEN: utilization = utilization (one service only so don't divide)
Also the final section TCPService effectively lets you load balance on number of established connections to your server, so you could balance based on the number of RDP connections to port 3389.
For this setting MaxConnections is important to specify as otherwise the agent will have no idea how to calculate the load i.e.
- utilization = MaxConnections / 100 * number of current connections * importance%
In the following screen shot from a Loadbalancer.org appliance you can see that the Windows server is 90% idle, but has been manualy put in forced agent drain mode. Whereas the Linux server has ample capacity at 98% idle.
Does that make sense? Have a play with the config file and let us know what you think....
The agent functionality is obviously not just for Windows users:
Lets look at an example of load balancing Microsoft RDS with HAProxy:
listen RDSTest
bind 192.168.69.22:3389
mode tcp
balance leastconn
persist rdp-cookie
server backup 127.0.0.1:9081 backup non-stick
tcp-request inspect-delay 5s
tcp-request content accept if RDP_COOKIE
timeout client 12h
timeout server 12h
option tcpka
option redispatch
option abortonclose
maxconn 40000
server Win2008R2 192.168.64.50:3389 weight 100 check agent-check agent-port 3333 inter 2000 rise 2 fall 3 minconn 0 maxconn 0 on-marked-down shutdown-sessions
The important bit agent-check agent-port 3333 tells HAProxy to constantly monitor each backend server in the cluster by doing a telnet to port 3333 and grabbing the response which will usually be a percentage idle value i.e.
- 80% - I am not very busy please increase my weight and send me more traffic
- 10% - I'm busy please decrease my weight and stop sending me so much traffic
- drain - Gradually drain the traffic from this server for maintenance
- down - Stop all traffic immediately, kill this backend server
- up - Enable traffic immediately, if no existing state conflicts.
- up ready 20% - Force HAProxy to bring the server up and set weight to 20% (irrespective of how it was taken down, reseting all health states)
UPDATE: We finally have a proper Linux Feedback Agent!
I'm really, really sorry this has taken so long. But our resident genius Nick has done an amazing job with his new Linux feedback agent. It can do a whole load of amazing things, and it's very stable and easy to use.
Oh, and of course its free and completely open source!
The binary is designed to be as cross-platform as possible, with most of the prequisites compiled in for compatibility.
You can download lbfeedback from the github project here, or just grab this binary with wget which is quicker:
wget https://downloads.loadbalancer.org/agent/linux/lbfeedback
Then put the following command in your startup scripts:
lbfeedback run-agent
But I'm lazy so I'm just going to fire it up as a background process:
chmod +x lbfeedback
./lbfeedback run-agent &
Which should look like this:
▄ █ Loadbalancer.org Feedback Agent v5.4.0
█ █ █▄▄ Copyright (C) 2025 Loadbalancer.org Limited
█ █ ▄ █ Licensed under the GNU General Public License v3
This program comes with ABSOLUTELY NO WARRANTY. This is free software, and
you are welcome to redistribute it under certain conditions. For further
information, please read the LICENSE file distributed with this program.
INFO[2025-05-16 14:48:34] *** [Started] Loadbalancer.org Feedback Agent v5.4.0
WARN[2025-05-16 14:48:34] Config file not found; a new file will be created.
WARN[2025-05-16 14:48:34] No configuration loaded; reverting to default services.
WARN[2025-05-16 14:48:34] Metric type 'cpu': no sample time specified; using default of 500ms.
INFO[2025-05-16 14:48:34] Responder 'default' : calculating relative significances, total 1.00.
INFO[2025-05-16 14:48:34] Responder 'default': name 'cpu', type 'cpu': 1.00 -> relative 1.00.
INFO[2025-05-16 14:48:34] Default services successfully configured.
INFO[2025-05-16 14:48:34] Directory not found, created: /opt/lbfeedback
INFO[2025-05-16 14:48:34] File not found, created: /opt/lbfeedback/agent-config.json
INFO[2025-05-16 14:48:34] Configuration file written successfully to '/opt/lbfeedback/agent-config.json'.
INFO[2025-05-16 14:48:34] Directory not found, created: /var/log/lbfeedback
INFO[2025-05-16 14:48:34] Logging to file: /var/log/lbfeedback/agent.log
INFO[2025-05-16 14:48:34] The Feedback Agent is now launching.
INFO[2025-05-16 14:48:34] System Metric Monitor 'cpu' has started (CPU, interval 500ms).
INFO[2025-05-16 14:48:34] Responder 'api': New TLS certificate generated (auto-renewed, expires Sat, 17 May 2025 02:48:34 +0000).
INFO[2025-05-16 14:48:34] Responder 'api' has started (HTTPS-API on 127.0.0.1:3334).
INFO[2025-05-16 14:48:34] Responder 'default' has started (TCP on *:3333).
INFO[2025-05-16 14:48:34] Startup complete; the Feedback Agent has launched.
You can now test this service using telnet and see that by default it is reporting the CPU idle status:
telnet 127.0.0.1 3333
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
95%
Connection closed by foreign host.
You can get information on all the commands with:
lbfeedback help
▄ █ Loadbalancer.org Feedback Agent v5.4.0
█ █ █▄▄ Copyright (C) 2025 Loadbalancer.org Limited
█ █ ▄ █ Licensed under the GNU General Public License v3
This program comes with ABSOLUTELY NO WARRANTY. This is free software, and
you are welcome to redistribute it under certain conditions. For further
information, please read the LICENSE file distributed with this program.
SYNTAX:
lbfeedback [action] [type] [parameters]
ACTIONS:
run-agent: Runs the Agent interactively or from a startup script.
All other Actions are followed by an Action Type, as follows:
add, edit, delete, start, restart, stop:
monitor, responder, source
get:
config, feedback, sources
set:
commands, threshold
force:
halt, drain, online, save-config
send:
online, offline
Note that the running Agent service will automatically save any configuration
changes to its JSON configuration file if they are successful, and no service
restart is required as they are applied immediately.
PARAMETERS:
-name Name identifier of a service.
For the 'force' and 'send' HAProxy command actions,
omitting this parameter will apply the action to all
Feedback Responders for which HAProxy commands are not
disabled; see also '-command-list' below.
-command-list List of HAProxy commands to enable, space-separated.
Example: -command-list up down
These are automatically detected as pertaining to online
or offline states. There are special options as follows:
'none' Disable all HAProxy commands.
'default' Send 'drain' for offline, 'up ready' for online.
-protocol Protocol name for a Responder. Options: 'tcp', 'http'.
-ip Listen IP address for a Responder.
-port Port to listen on for a Responder.
'any' Listen on all ports for the specified IP.
-request-timeout Request timeout (ms).
-response-timeout Response timeout (ms).
-threshold-max Maximum load for an online state (percent).
-threshold-mode Mode for automatic command threshold (default 'none'):
'none' All threshold behaviours are disabled.
'any' Down if any metric or overall relative load
exceeds the configured threshold.
'overall' Down if the overall relative load exceeds the
configured threshold, ignoring individual
metrics.
'metrics' Down if any metric exceeds the configured
threshold, ignoring the overall relative load.
-log-state-changes Log any changes in threshold state (true/false; default
is false). This should usually be disabled in production
to avoid excessively large log files being generated.
-command-interval Time interval to send HAProxy commands for (ms,
default 10000), timed from the first Feedback Request.
-monitor Name identifier of a target Monitor.
-significance Significance value (floating-point; e.g. 1.0). This
is converted into a Relative Significance by summing
the significance of all sources within a Responder
and calculating their ratio.
-smart-shape Enable Z-score (Gaussian) algorithmic load shaping
for a given Monitor (true/false; disabled by default).
This feature aims to prevent sudden excursions in weights
and therefore improves connection distribution between
Real Servers within HAProxy where persistence is enabled.
-max-value Maximum value for a given metric against which to
scale its availability.
-metric-type Type of metric. Options: 'cpu', 'ram', 'disk-usage',
'netconn', 'script'.
-sampling-ms For 'cpu' metrics, the sample window duration (ms).
-script-name For 'script' metrics, the name of the script to run from
the Feedback Agent configuration directory.
-disk-path For 'disk-usage' metrics, the local filesystem path to
monitor for available disk space.
EXAMPLES:
lbfeedback get config
lbfeedback add monitor -name ram -metric-type ram
lbfeedback add source -name default -monitor ram
lbfeedback force halt -name default
Please note that this is an extremely brief outline of the available
CLI configuration commands for controlling the Feedback Agent. For
further information, please consult the accompanying documentation or
contact Loadbalancer.org Support for assistance.
But the most usful ones are:
lbfeedback force halt
lbfeedback force drain
lbfeedback force online
I'll let you experiment with the more complex features, incluing multi-service configs, monitors for ram, disk, network, sampling rates and all sorts of juicy customizations!
Where do I find the source code?
Our Windows feedback agent has recently undergone a complete rewrite in a language called GO. We felt that our previous implementation of the feedback agent was feeling a bit dated, requiring an older version of the .net framework and was becoming a pain to maintain. With this new version, the agent technically allows for cross-platform compatibility, improves performance as GO is a compiled language, and lastly is so much easier to support. A more detailed explanation of the benefits and setbacks of getting thus far will be explained in a future blog.
Our GO based Windows feedback agent is open source and can be found on Github here.
And the Linux version can be found on Github here