It so happens that I’ve recently had the pleasure to deal with a co-lo provider which also happens to offer a managed firewall service. Their infrastructure is based on some fashionable IPS which provides “security” to the clients requesting it.
The concept of an IPS was born when Gartner, in the person of the renowned security expert John Pescatore, declared to the world that IDS was dead because it was too cumbersome to configure, didn’t deliver, etc. etc. etc. The firewall was rapidly provided with “intelligence” directly from Pescatore’s own private stash and promoted to IPS or “Intrusion Prevention System”, an oh-so-more-catchy term.
As the whole world knows IPS has solved the world’s IT security problems with the obvious unfortunate exception of my poor client.
Let us imagine, for sake of argument, that this poor client happens to have a 100% Linux-based infrastructure and that it is a minor variation on the classic LAMP site and that it also uses the highly subversive sendmail MTA.
Let us further imagine that these raving loonies happen to have customers who willingly subscribed to a weekly newsletter and a monthly in-depth variant of the latter and that they would be so bold as to require that weekly newsletters be delivered the same day they are sent and not a month later.
Enter the “managed firewall service” which decides unilaterally that the volume of outbound traffic produced by sendmail couldn’t possibly be legitimate and thereby classes the originating system as an Windows system infected by a trojan or worm1. To protect the Internet it therefore blocks all outbound SMTP traffic until the connection rate drops. This happens when sendmail notices the lack of connections and throttles itself by placing differing retry timeouts.
The co-lo “managed security helpdesk” when queried immediately replies with the terrible Internet infection scenario and the white knight, in the form of their IPS, coming to the rescue.
This is not all, it also detects outbound traffic as Teardrop2 attacks prompting claims that the machines have been compromised with the explanation that the servers are “sending out malformed packets which have the potential to crash certain un-patched servers”3.
The list goes on and on.
The final straw is marking every outbound GIF as an attack against Internet Explorer4 with no actual attempt at determining whether the image in question was an attack or not.
We asked for the packet dumps in pcap format and I started to go through them and here is the outcome of my research on the first of the files sent containing malicious traffic (with some inevitable censorship).
First of all I filtered out all the ARP requests and packets that had nothing to do with the client’s network, that cut the number of packets by 50%, so obviously the IPS alerts on ARP and irrelevant traffic too…
09:25:10.647868 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], length: 76) w.x.y.z.53 > j.k.l.m.53: [udp sum ok] 27809 [1au] AAAA? pool.domainsite.com. ar: . OPT UDPsize=4096 (48) 09:25:25.639925 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], length: 73) w.x.y.z.53 > j.k.l.m.53: [udp sum ok] 29390 [1au] A6? ns1.netscape.com. ar: . OPT UDPsize=4096 (45) 09:26:47.907197 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], length: 76) w.x.y.z.53 > j.k.l.m.53: [udp sum ok] 46858 [1au] MX? pool.domainsite.com. ar: . OPT UDPsize=4096 (48)
09:25:31.763296 IP (tos 0x0, ttl 64, id 28349, offset 0, flags [DF], length: 46) w.x.y.z.55750 > j.k.l.m.25: P [tcp sum ok] 0:6(6) ack 1 win 5840 0x0020: 5018 16d0 ab8e 0000 5155 4954 0d0a P.......QUIT..
09:25:31.763384 IP (tos 0x0, ttl 64, id 28350, offset 0, flags [DF], length: 40) w.x.y.z.55750 > j.k.l.m.25: F [tcp sum ok] 6:6(0) ack 1 win 5840
Note how the above packet is the TCP teardown packet for the connection which was terminated at the protocol level by the packet in 2) above.
Nobody deserves more than this amount of torture: out of 4000+ packets there wasn’t a single malicious packet of any kind save for two connection attempts inbound to the Linux boxes on port 445 (Windows RPC services) which were obviously sent packing. The missing analysis was for the dangerous HTTP packets sending GIFs, “ident”, and sendmail connection setup. I am sure you will thank me for not posting the “analysis” to those too…
Needless to say I didn’t look at the other pcap dumps nor did I ever get a reply to my e-mail, they continue filtering the traffic and the client is going to cancel the contract for the managed firewall service.
There is little to comment about here besides the obvious hype about IPS removing the configuration and maintenance issues of IDS (frankly, if you believed that then I’d be honoured to sell you shares in the Leaning Tower of Pisa and the Fountain of Trevi, Rome).
Sticking a box on a network looking at packets can never be a substitute for training a human on how to respond: if you are incompetent the boxes you manage will be incompetent too.
Not only, these fashionable “auto-reacting” IPS are far from infallible (unlike Mr. Pescatore who is apparently never wrong in his IT security predictions) and, in the specifc case of this co-lo site, will bring them loss of customer revenue (I did recommend asking for damages too).
__
1 There is a subtle difference between the two: a trojan normally gains access to systems by masquerading as something legitimate, e.g. an e-mail, and requires a modicum of end-user interaction whereas a worm spreads from machine to machine making use of holes in the operating system and therefore independently of the end-user’s cooperation.
2 You should follow the link, observe the date of the attack and the vulnerable systems.
3 Do go and read the link provided, please. Then compile a list of systems still running vulnerable software to Teardrop, your answer should comfortably fit in a page of A4. Then highlight any well-known site from the list. Your highlighting pen should remain capped throughout the exercise.
4 Yes, it is possible to attack Internet Explorer with GIF images, see this discussion on SecurityFocus if you have been living under a rock in the past few months.