Problem
Customer calls first thing on a Monday morning saying that something is consuming all 12Mbps of their Internet bandwidth.
Background
Customer is an ASP and hosts several services including financial and billing, email, and document management applications. Clients connect via remote desktop encrypted in a site-to-site VPN tunnel. The tunnel endpoint is a pair of Cisco ASA5510’s in fail-over mode. The internet facing router is a Cisco 2801. All application servers are virtual machines running in VMware vSphere on a cluster of Dell blades.
Troubleshooting
A quick check of the ASA revealed that outbound internet traffic was a constant 8Mbps with bursts as high a 14Mbps. Everyone agreed that the overnight increase in bandwidth use was likely caused by a trojan or virus. But with over 90 virtual servers as potential hosts, isolating the problem server could prove time consuming.
The customer had already spent some time trying to isolate the source by looking through logs on the ASA and analyzing bandwidth consumption on particular virtual machines. Since they hadn’t found anything obvious we decided to take a different approach and look at packets exiting the firewall.
Using Wireshark, we captured traffic on the internet facing interface of the ASA. (Note – I’ve removed the source and destination IP columns)
The first thing that looks a little abnormal is the amount SMTP traffic. The customer hosts a few Exchange servers, but there seems to be an inordinate amount SMTP activity relative to the number of mail servers and clients. Not to mention the strange payload in some of the SMTP packets.
Wireshark can provide a statistical breakdown of the contents of a packet capture. Click on Statistics>Protocol Hierarchy. After processing the capture file you’ll be presented with a chart outlining the protocol statistics.
SMTP packets made up a little over 21% of this capture and accounted for ~ 9Mbps of bandwidth used during the capture window.
From here we can use another Wireshark feature to view the data in the same way the application layer would see it. Locate an SMTP packet in the capture, then click on Analyze>Follow TCP Stream.
Now we have a better view of this particular email transaction and noted the following:
Received from is not one of the customer’s domains.
To: is in the .mx domain.
The email client is Outlook Express 6
Email Subject: hi friend
Content-Disposition: attachment;.filename=”t658657.zip”
The same thing was found in a couple of other streams that were analyzed. The attachment turned out to be a 40Meg zip file. That helped explain what was consuming all of the internet bandwidth.
Using this information, we applied a rule to the ASA that denied all outbound SMTP traffic not originating from one of the mail servers. (a good idea on any network!!) This immediately restored the internet bandwidth consumption to normal.
Logging was enabled on the rule to provide a quick visual indication in the ASDM as to the number of matches on that rule. Within a few minutes, it had logged over 400 hits.
An examination of the logs on the ASA pointed us to the offending terminal server which was subsequently scanned with Trend Micro House Call and the trojan removed.
Conclusions
Wireshark is a phenomenal tool and it’s worth investing time in the lab to understand it’s many features. In this particular instance, it helped identify the problem quickly thanks to it’s embedded packet analysis capability.
Besides the numerous books and on-line tutorials, a good place to start getting familiar with Wireshark is to take a look at the Wireshark Users Guide, capture some packets and experiment.
