ASA Packet Capture Using The CLI

Background
Cisco IOS Embedded Packet Capture (EPC) is an extremely useful troubleshooting tool and can prove particularly helpful when it is not feasible to set up a network tap or when diagnosing a problem on a remote system.

My lab device is a ASA5505, however the capture commands are similar across most platforms.  (ASA, PIX, FWSM and ISR routers running IOS 12.4(20)T or later)

Capturing Traffic on the ASA
The basic syntax of the capture command is as follows:
capture <capture name> interface <interface name>

While the capture is running, you can view the results on the console (not always a good idea and not very easy to read) or in a web browser.
To view the capture in a browser, format the URL as follows:
https://<device ip address>/admin/capture/<capture name>

A better option would be to open the capture in Wireshark.  We can download the capture in libpcap format by specifying the /pcap option at the end of the URL.
https://<device ip address>/admin/capture/<capture name>/pcap

When prompted, select the “Open With” radio button and select Wireshark from the list.  (you’ll need to have Wireshark installed on your host)

To stop the capture, simply negate the capture command, followed by the capture name.  This also removes the capture from the buffer so make sure you export it using the above procedure if you want to save the results:
no capture <capture name>

Options

Besides the many filtering options,  it’s possible to specify multiple simultaneous captures, each with different criteria.  Let’s capture only IP traffic on the outside interface and ARP traffic on the inside interface.

If you lose track of what captures are running on the device, enter show capture at the prompt to display active captures and their options.

Conclusions
EPC is a useful feature and has numerous capabilities beyond what I’ve demonstrated here.  When combined with Wireshark, EPC provides the ability to quickly capture and analyze network traffic without a tap or mirroring a port on an adjacent switch.

Additional Reading
ASA/PIX/FWSM: Packet Capturing using CLI and ASDM Configuration Example

Cisco IOS Embedded Packet Capture

Troubleshooting Bandwidth Consumption Using Wireshark

Problem
Customer calls first thing on a  Monday morning saying that something is consuming all 12Mbps of their Internet bandwidth.

Background
Customer is an ASP and hosts several services including financial and billing, email, and document management applications.  Clients connect via remote desktop encrypted in a site-to-site VPN tunnel.  The tunnel endpoint is a pair of Cisco ASA5510’s in fail-over mode.  The internet facing router is a Cisco 2801.  All application servers are virtual machines running in VMware vSphere on a cluster of Dell blades.

Troubleshooting
A quick check of the ASA revealed that outbound internet traffic was a constant 8Mbps with bursts as high a 14Mbps.  Everyone agreed that the overnight increase in bandwidth use was likely caused by a trojan or virus.  But with over 90 virtual servers as potential hosts, isolating the problem server could prove time consuming.

The customer had already spent some time trying to isolate the source by looking through logs on the ASA and analyzing bandwidth consumption on particular virtual machines.  Since they hadn’t found anything obvious we decided to take a different approach and look at packets exiting the firewall.

Using Wireshark, we captured traffic on the internet facing interface of the ASA.  (Note – I’ve removed the source and destination IP columns)

The first thing that looks a little abnormal is the amount SMTP traffic.  The customer hosts a few Exchange servers, but there seems to be an inordinate amount SMTP activity relative to the number of mail servers and clients.  Not to mention the strange payload in some of the SMTP packets.

Wireshark can provide a statistical breakdown of the contents of a packet capture.  Click on Statistics>Protocol Hierarchy.  After processing the capture file you’ll be presented with a chart outlining the protocol statistics.

SMTP packets made up a little over 21% of this capture and accounted for ~ 9Mbps of bandwidth used during the capture window.

From here we can use another Wireshark feature to view the data in the same way the application layer would see it.  Locate an SMTP packet in the capture, then click on Analyze>Follow TCP Stream.

Now we have a better view of this particular email transaction and noted the following:
Received from is not one of the customer’s domains.
To: is in the .mx domain.
The email client is Outlook Express 6
Email Subject: hi friend
Content-Disposition: attachment;.filename=”t658657.zip”

The same thing was found in a couple of other streams that were analyzed.  The attachment turned out to be a 40Meg zip file.  That helped explain what was consuming all of the internet bandwidth.

Using this information, we applied a rule to the ASA that denied all outbound SMTP traffic not originating from one of the mail servers.  (a good idea on any network!!)  This immediately restored the internet bandwidth consumption to normal.

Logging was enabled on the rule to provide a quick visual indication in the ASDM as to the number of matches on that rule.  Within a few minutes, it had logged over 400 hits.

An examination of the logs on the ASA pointed us to the offending terminal server which was subsequently scanned with Trend Micro House Call and the trojan removed.

Conclusions
Wireshark  is a phenomenal tool and it’s worth investing time in the lab to understand it’s many features.  In this particular instance, it helped identify the problem quickly thanks to it’s embedded packet analysis capability.

Besides the numerous books and on-line tutorials, a good place to start getting familiar with Wireshark is to take a look at the Wireshark Users Guide, capture some packets and experiment.

Detecting DHCP DOS Attacks

Exploiting DHCP vulnerabilities is likely to be within the skills of the novice troublemaker on your LAN.  A scope exhaustion attack is surprisingly simple to execute and potentially difficult to detect and isolate.  Fortunately, preventing one is fairly straightforward if you have switches in your network with the right features.  This post primarily covers symptoms and detection, a later post will discuss prevention.

Let me preface the rest of this post by saying that I’m conducting these experiments in an isolated lab environment.
Do not test exploits on a production network.

Lab Setup
My test environment consists of three virtual machines, Ubuntu (the attacker), a Windows 2003 server running DHCP (the target) and an Xp VM running Wireshark (the observer).

I’ve set up two vSwitches, each with it’s own uplink port (vmnic2 & vmnic3).  Both uplinks are connected to a Cisco Catalyst 3560.  VSwitch2 has two port groups, the Lab Target-Promiscuous port group is set to promiscuous mode to allow packets on vSwitch2 to be captured by Wireshark running on the Xp VM.

Running the Exploit
I’m using Yersinia on the Ubuntu VM to launch the DHCP attack against the Windows 2003 server.

Yersinia is capable of generating DHCPDISCOVER requests at a rapid rate and can quickly exhaust a DHCP scope.

Wireshark provides a better look into just how many DHCPDISCOVER messages are being generated Yersinia, each from a different spoofed MAC addresses.

The DHCP server returns a DHCPOFFER until all of the available addresses in the scope are exhausted.

Forensics
Windows DHCP allocates an address from the appropriate pool when a DHCPDISCOVER message is received.

In the DHCP console, note how none of the addresses are displayed in the Address Leases pane.  A check of the scope statistics reveals that something is a miss.  Take special note of these stats:
Discovers – 69523
Offers – 100 (offers will be made until the available addresses in the scope are exhausted)
In Use – 100% (the DHCP scope is exhausted)

Clicking Refresh will update the stats.  If the attack is still occurring, the Discovers counter will increment significantly with each refresh.

When the scope is exhausted, two entries will be made in the System Event Log, Event ID 1063 and 1020, both stating that the scope is full and no addresses are available.

Restarting the DHCP server service releases the addresses and returns them to the pool.  Otherwise, DHCP will return the addresses to the pool after ten minutes.  If the attack is no longer running, everything will appear normal.

Detection
One possible detection method would be to use Perf Mon to monitor DHCP Discovers/sec.  An alert could be automatically generated if the counter was over a certain threshold.  Yersinia generated over 6000 DHCP Discovers/sec on a 100Mbps link in this test.   On most networks, a high rate of DHCPDISCOVER requests would be considered abnormal.  Below is the Perfmon trace from the DHCP server while the attack was running.

Since Yersinia is capable of spoofing the MAC of each DHCPDISCOVER request, locating the attacker on your network can be a challenge.  If you have managed switches the process is a little easier.  If not, then the threat of this type of attack is a good justification to consider adding managed switches with decent security features to your infrastructure.

Viewing the MAC Address table on the 3560 reveals that the switch has essentially been turned into a hub.  It took Yersinia about ten seconds to fill the MAC table with bogus entries.

We’ll need to look at the interface statistics to isolate the port that the attacker is using.  Granted, only two ports on the switch are active, but even on a busier network Fa0/7 would be suspect based on RXBS.  Note these stats are based on a 5 minute average.

If only it were that easy.  Any good hacker would be unlikely to keep the exploit running for long.  Considering that it took Yersinia about three seconds to deplete a scope of 100 IP addresses, it’s unlikely that amount of traffic would stand out from the background noise on the LAN.  We need a way to monitor a sudden increase in MAC address entries on a port.  I’ll cover that along with port security and DHCP-Snooping in a later post.

If you find this topic interesting, consider picking up the book LAN Switch Security: What Hackers Know About Your Switches. (Cisco Press – ISBN 1587052563)