I know the world is falling apart with the credit crunch and all that, but please don't call the load balancer vendor when your internet bank falls over. We've had a couple of messages similar to the following:
(*** Asterix to protect the innocent!)
|FROM:E.W TRA |
|PH#: (317)034- EXT:0 |
|EMAIL: E.W.T*****@X***LL.*** |
|MESSAGE:CALLING FROM THE ***.
SHE IS A MEMBER OF THE I SAVE BANK AND
WANTS TO KNOW HOW TO ACCESS HER
ACCOUNT INFORMATION PLS CALL. HER
NUMBER IS 31-70-
What happened here is that 'Ice Save' either overloaded their servers (or just took them offline), but neglected to change the default failure page on our load balancers... hence many angry customers coming to our web site and asking for access to their cash!
The lessons to be learned here are:
- Have enough servers to handle the total load.
- Rate limit at the firewall / router / load balancer to your total rated capacity.
- Change your fall back page to say something sensible like "oh sorry but we've gone bust!"
To stop flames in advance I do feel very sorry for anyone in this situation, i.e. their bank is failing.