Symptom
When right-clicking on a file in Windows XP, the context menu takes over four minutes to be displayed.  Right-clicking on a folder exhibited normal behavior and was not affected.

Background
Context Menu Handlers can load from several areas in the registry.  One set of keys that control the menu creation when right-clicking on a file is located in:
HKEY_CLASSES_ROOT\*\shellex\ContextMenuHandlers

Troubleshooting
In this particular case, the end user had been putting up with this for a couple of weeks and was at wits end by the time he finally pulled me aside and asked me to take a look.  I was to happy oblige  as I  was tired of troubleshooting Citrix issues and was ready to look at something different for a few minutes. 

Start by making a backup copy of the entire ContextMenuHandlers subkey by right-clicking on the key and selecting Export.
Remove each key subkey under ContextMenuHandlers until you locate the offending entry.
It’s probably a good idea to export a copy of each subkey before you delete it.  That makes it easier to re-add the non-offending subkey after you’ve isolated the issue.

In my case, it was the “Open With” subkey that was causing the problem.  As soon as it was removed, the menu displayed immediately when right-clicking a file.

After rebooting the PC, I tried importing the “Open With” key back into it’s proper location.  Much to my surprise everything still worked correctly.

The VMware host in my lab is a Dell GX620 running ESX3.51 U2.  It has three NIC’s:
The embedded Broadcom BCM5751 Gigabit Adapter  (vmnic0)
3Com 3C905C-TX 10/100 Adapter  (vmnic1)
Intel 8254N Dual Port Gigabit Adapter  (vmnic2 and vmnic3)

The 3C905 isn’t entirely VMware compatible.  It can be used as a service console connection but will not function as a virtual machine uplink on a vSwitch.  It will always show disconnected.

I want to reassign the service console vswif to the 3Com adapter, keep the same IP address and move the service console port group to a new vSwitch.

VMware doesn’t like having two service console connections with different IP addresses in the same subnet, so I have two options.

1.  Create a service console connection in a different subnet and access the host from that subnet using the VIclient.
2.  Enter the commands directly on the host console.

I recommend option two.  Keep in mind that this process temporarily disrupts network communications to the host via the service console IP.

After obtaining physical access to the host’s console (or network access via a DRAC or ILO), log in and su - to establish root.

Remove vswif0 from vSwitch0
esxcfg-vswif -d vswif0  –ip=192.168.68.35  –netmask=255.255.255.0  –portgroup=SCX

Remove the SCX port group from vSwitch0
esxcfg-vswitch –del-pg=SCX vSwitch0

Create a new vSwitch for the service console;
esxcfg-vswitch -a vSwitch3

Assign the 3Com adapter to the newly created vSwitch
esxcfg-vswitch -L vmnic1 vSwitch3

Add the SCX port group to vSwitch3
esxcfg-vswitch -A SCX vSwitch3

Add vswif0 to the SCX port group and assign the IP address and subnet mask
esxcfg-vswif –add –ip=192.168.68.35 –netmask=255.255.255.0 –Portgroup=SC2 vswif0

Restart the VMware management service
service mgmt-vmware restart

Here’s vSwitch0, still connected to vmnic0 but minus the service console

And here’s the new vSwitch3 with the service console reassigned to vmnic1

Some STP Basics
On most Cisco Catalyst switches, STP is enabled on all ports by default.   Port initialization requires upwards of 30 seconds to complete, and can take as long as 50 seconds.
This thirty second “delay” can be attributed to the time required for the port to transition from Listening to Learning and finally to Forwarding.
The Listening and Learning transitions each require about 15 seconds.
This transition period can be painful for end users waiting to gain access to the network.  To the untrained IT person it can be misdiagnosed as “some sort of network issue”.

PortFast to the Rescue
Portfast shortens the Listening and Learning states allowing the link to transition to the Forwarding state in as little as three seconds.
This translates to quicker access to the network for the end user when they power on their PC, connect a laptop to a wired port, etc.
Enabling PortFast does not disable STP on the port, it simply allows us to get to the Forwarding state much faster.

Let’s enable PortFast on switch ports 1 – 4 using the spanning-tree portfast command.  IOS provides a reminder of the possible consequences.

Looping ports Fa0/2 and Fa0/3 reveals that we still have adequate loop protection as Fa0/3 transitions to a Blocking state within ~2 seconds.  Other hosts on the switch are not affected.
In this scenario, PortFast protects against mistakes made in the wiring closet or on the off chance that two access ports would become looped under a desk.  (don’t laugh, I’ve seen it happen)
Here’s a partial output from the show spanning-tree command.

Note that Fa0/3’s Role has changed to Back.  This is helpful information as it indicates that two or more ports on the same bridge are connect together.
Fa0/3’s Status has changed to BLK, effectively blocking the loop condition .

The Tech Savvy End-User
Let’s say an end user wants to add a couple of extra network ports to their cubicle.  Instead of calling the help desk and being questioned as to what unauthorized device they are trying to connect to your LAN, they pick up an unmanaged switch from local retailer and connect it to their access port, Fa0/2.  (for the sake of this post let’s assume we’re not MAC locking ports on the switch)

What happens when two ports on the parasite switch connected to access port Fa0/2 become looped?  This partial output from show spanning-tree provides some info.

This should generate a call to the help desk as Fa0/2 immediately transitions to a Blocking state, preventing traffic from the looped parasite switch from entering the network.
Role is indicating Designated (DESG) which means that Fa0/2 is not looped with another port on this switch.  The Type field provides additional information.  Self-looped is a good indicator that something interesting is happening on Fa0/2.  No other hosts on the access switch were impacted.

Conclusions
PortFast is a great feature and can be enabled without compromising loop protection.  You should think twice about ever disabling spanning-tree.  I’ve seen a looped parasite switch bring down a 400 node network where spanning-tree had been disabled or wasn’t available on the particular switches the client has deployed.  Yet another argument for purchasing quality switches for your infrastructure.

A Cisco switch generates a small amount of network traffic as part of it’s normal housekeeping functions.
It’s important to be able to recognize normal “background noise” when looking at a packet capture.

Below is a packet capture from a Catalyst 3560.  The only device connected to the switch is  an Xp virtual machine running Wireshark.
Note the four types of packets that appear at regular intervals, STP, LOOP, DTP and CDP.  (click on the image for a larger view)

STP
A Spanning Tree Bridge Protocol Data Unit (BPDU) is sent every two seconds as part of the loop detection process.
This particular packet tells us that the root bridge is 00:22:be:21:3e:80, which also happens to be the switch we are connected to.

It is possible to prevent BPDU’s from being sent out an interface by enabling BPDU Filtering .

BPDU Filtering can be enabled globally for every port that has PortFast enabled by using the spanning-tree portfast bpdufilter default command.

Note this also disables loop detection on all access ports,  probably not a good idea under most circumstances.

LOOP
LOOP Reply is a Layer 2 keepalive packet that is sent every ten seconds by default.
The LOOP Reply verifies to IOS that the link is up.  The switch does not actually listen for a reply, it simply verifies that was able to send the packet out the interface.
Loss of three consecutive Layer 2 keepalives will cause the interface to transition to a down state.
It is possible to configure the interval between packets by using the keepalive interface configuration command.

DTP
Dynamic Trunking Protocol (DTP) is a Cisco proprietary protocol used to negotiate a common trunking mode between two switches.
A trunk link differs from an access port in that a trunk can transport more than one VLAN.
DTP packets are sent every thirty seconds by default.
If the switch port is configured as an access port using the switchport mode access command, DTP packets will not be sent from that interface.

When an access port is reconfigured as a trunk port, transmission of DTP packets will resume 30 seconds after the port is reconfigured.

CDP
Cisco Discovery Protocol (CDP) packets are sent every sixty seconds by default.
CDP provides information about the capabilities of a device to it’s connected neighbor.
CDP can be disabled and it’s not a bad idea to do so under certain circumstances especially if security is a concern.

CDP can be disabled globally with the no cdp run command.

CDP can also be disabled on a particular interface with the no cdp enable command.

vSwitches are capable of advertising and listening for CDP information from an attached Cisco device.
CDP on a vSwitch is set to “Listen” by default.

To enable CDP you’ll need to access the Service Console.
After logging in type su- to establish root.

First let’s take a look at the command options by using esxcfg-vswitch -h

Use esxcfg-vswitch -l to show a list of configured vSwitches on the host.

Now that we know the vSwitch names, let’s check the CDP status on vSwitch1 using esxcfg-vswitch -b vSwitch1

Set CDP to listen and advertise using the “both” option.  esxcfg-vswitch -B both vSwitch1
Verify the settings using esxcfg-vswitch -b vSwitch1

Executing show CDP neighbors from the Catalyst switch shows the ports that the ESX host resides on and the vmnic of the uplink.

In the VCP4 exam blueprint, Objective 1.3 states that you “Understand Service Console firewall operation”.

The command syntax is esxcfg-firewall -<options>

The service console has a small number of services predefined in the firewall.  Use the -e and -d options to enable/disable these services.
To open/close a custom port that is not predefined, use the -o or -c options.

Start by opening an SSH session to the ESX server and entering su- to establish root.

esxcfg-firewall -s
Lists the predefined services known to the VMware host.
Remember “s” for services.

Here’s an example of the command output on an ESX3.51 U2 host showing the predefined services.

esxcfg-firewall -q
Queries the current firewall configuration.
Shows Enabled services and Opened ports.
Remember “q” for query.

esxcfg-firewall -e  <service name>
Enables a predefined service.
Remember “e” for enable service.

esxcfg-firewall -d <service name>
Disables a predefined service.
Remember “d” for disable service.

esxcfg-firewall -o <port, tcp|udp, in|out, name>
Opens a port that is not already defined as a service.
Remember “o” for open port.

esxcfg-firewall -c <port, tcp|udp, in|out, name>
Closes a port that is not already defined as a service.
Remember “c” for close port.

esxcfg-firewall -r
Resets all firewall options to default settings.
Remember “r” for reset.

Exploiting DHCP vulnerabilities is likely to be within the skills of the novice troublemaker on your LAN.  A scope exhaustion attack is surprisingly simple to execute and potentially difficult to detect and isolate.  Fortunately, preventing one is fairly straightforward if you have switches in your network with the right features.  This post primarily covers symptoms and detection, a later post will discuss prevention.

Let me preface the rest of this post by saying that I’m conducting these experiments in an isolated lab environment.
Do not test exploits on a production network.

Lab Setup
My test environment consists of three virtual machines, Ubuntu (the attacker), a Windows 2003 server running DHCP (the target) and an Xp VM running Wireshark (the observer).

I’ve set up two vSwitches, each with it’s own uplink port (vmnic2 & vmnic3).  Both uplinks are connected to a Cisco Catalyst 3560.  VSwitch2 has two port groups, the Lab Target-Promiscuous port group is set to promiscuous mode to allow packets on vSwitch2 to be captured by Wireshark running on the Xp VM.

Running the Exploit
I’m using Yersinia on the Ubuntu VM to launch the DHCP attack against the Windows 2003 server.

Yersinia is capable of generating DHCPDISCOVER requests at a rapid rate and can quickly exhaust a DHCP scope.

Wireshark provides a better look into just how many DHCPDISCOVER messages are being generated Yersinia, each from a different spoofed MAC addresses.

The DHCP server returns a DHCPOFFER until all of the available addresses in the scope are exhausted.

Forensics
Windows DHCP allocates an address from the appropriate pool when a DHCPDISCOVER message is received.

In the DHCP console, note how none of the addresses are displayed in the Address Leases pane.  A check of the scope statistics reveals that something is a miss.  Take special note of these stats:
Discovers – 69523
Offers – 100 (offers will be made until the available addresses in the scope are exhausted)
In Use – 100% (the DHCP scope is exhausted)

Clicking Refresh will update the stats.  If the attack is still occurring, the Discovers counter will increment significantly with each refresh.

When the scope is exhausted, two entries will be made in the System Event Log, Event ID 1063 and 1020, both stating that the scope is full and no addresses are available.

Restarting the DHCP server service releases the addresses and returns them to the pool.  Otherwise, DHCP will return the addresses to the pool after ten minutes.  If the attack is no longer running, everything will appear normal.

Detection
One possible detection method would be to use Perf Mon to monitor DHCP Discovers/sec.  An alert could be automatically generated if the counter was over a certain threshold.  Yersinia generated over 6000 DHCP Discovers/sec on a 100Mbps link in this test.   On most networks, a high rate of DHCPDISCOVER requests would be considered abnormal.  Below is the Perfmon trace from the DHCP server while the attack was running.

Since Yersinia is capable of spoofing the MAC of each DHCPDISCOVER request, locating the attacker on your network can be a challenge.  If you have managed switches the process is a little easier.  If not, then the threat of this type of attack is a good justification to consider adding managed switches with decent security features to your infrastructure.

Viewing the MAC Address table on the 3560 reveals that the switch has essentially been turned into a hub.  It took Yersinia about ten seconds to fill the MAC table with bogus entries.

We’ll need to look at the interface statistics to isolate the port that the attacker is using.  Granted, only two ports on the switch are active, but even on a busier network Fa0/7 would be suspect based on RXBS.  Note these stats are based on a 5 minute average.

If only it were that easy.  Any good hacker would be unlikely to keep the exploit running for long.  Considering that it took Yersinia about three seconds to deplete a scope of 100 IP addresses, it’s unlikely that amount of traffic would stand out from the background noise on the LAN.  We need a way to monitor a sudden increase in MAC address entries on a port.  I’ll cover that along with port security and DHCP-Snooping in a later post.

If you find this topic interesting, consider picking up the book LAN Switch Security: What Hackers Know About Your Switches. (Cisco Press – ISBN 1587052563)

I’ve received a couple of emails from folks wanting to know how I did on the VCP exam.  Regretfully, I have to report that I failed my first attempt, missing a passing score by 32 points.

In hindsight I’m not really surprised I didn’t do well as I really didn’t know what to expect and knew I was “pushing the envelope” in terms of preparation time.

Here are some things to consider if you are thinking about taking the exam and a couple of things  I’m doing to prepare for my second attempt.

1.  You can’t pass the exam by simply taking the class.  The class is good, but it does not cover the exam topics in enough depth to get you through the exam.
2.  Everything you need to know is outlined in the exam blueprint.
3.  All of the info you need to study is in the VMware documentation.
4.  I’m reviewing all of the free training material offered on VMware’s Partner website.
5.  Consider investing a couple of hours and completing the VMware Technical Sale Professional (VTSP) training.  There’s good material there that helped me fill in some of the gaps in my product knowledge.
6.  Consider some supplementary study material.  I’m reading Scott Lowe’s new book “Mastering VMware vShpere 4″.  It’s not a study guide, but is full of good information on vSpheres features and configuration.
7.  Know your configuration maximums but don’t spend all of your time studying them.

The exam is not difficult, the questions are fair and easy to understand.  I was lacking knowledge in two critical areas and it really showed in my score.

Even after deploying VMware solutions for the last couple years the certification process has taught me a great deal about the product.  I guess that’s the whole idea isn’t it?

In terms of ROI, I’ve already gone back to a few of the clients I work with and suggested some changes/enhancements to their configurations.  Having good instructors for the class and  combing through the documentation studying for the exam has opened my eyes to vSphere’s capabilities.

So far I’d have to say that going through the VCP certification process has had a very positive impact on my ability to consult, engineer, deploy and support vSphere.

VCP4 Exam Blueprint

Objective 3.2 – Configure iSCSI SAN Storage

Knowledge

Page numbers refer to the VMware iSCSI SAN Configuration Guide pdf.

1.  Identify iSCSI SAN hardware components (pg. 9)
-Host Bus Adapters (HBA)
-Network Interface Cards (NIC)
-Switches
-Routers
-Cables
-Storage Processors (SP)
-Storage Disk Subsystems

2.  Determine use cases for hardware vs. software iSCSI initiators
-Hardware Initiator – specialized iSCSI HBA, responsible for all iSCSI and network processing and management
-Hardware initiators provide better performance and throughput by offloading iSCSI and TCP processing overhead.

-Software Initiator – Code built into the VMkernel that allows host to connect to the storage device through a std Ethernet adapter.
-Software initiators provide adequate performance for most applications, are less complex and easier to implement.

3.  Configure the iSCSI Software Initiator (pg. 30)
-Configure a VMkernel port for the physical network adapter.
-Enable the software iSCSI initiator
-Activate multi-pathing using the port binding technique on the ESX host if you use multiple network adapters
-Enable jumbo frames if needed and supported.  Jumbo frames must be enabled for each vSwitch via the CLI.
vicfg-vswitch -m <MTU> <vSwitch>
-For jumbo frames, you must create a VMkernel network interface enabled with jumbo frames.
esxcfg-vmknic -a -I <ip address> -n <netmask> -m <MTU> <port group name>

4.  Configure Dynamic/Static Discovery (pg. 35)
-Dynamic Discovery (Send Targets) – the server sends a list of available targets in response to the Send Targets request.
-Targets discovered via Dynamic Discovery are added to the Static Discovery tab
-Static Discovery – the initiator uses a list of targets (IP addresses and target names)
-Required privilege: Host.Configuration.Storage Partition Configuration
-Configuration>Storage Adapters>Properties>Dynamic Discovery>Add
-You cannot change the IP address, DNS name, or port number of an existing Send Targets server. delete the existing server and add a new one.

5.  Configure CHAP Authentication (pg. 37)
-ESX/ESXi supports CHAP at the adapter level.
-Software iSCSI supports per-target CHAP
-For software iSCSI, the CHAP name should not exceed 511 and the CHAP secret 255 alphanumeric characters.
-For hardware iSCSI, the CHAP name should not exceed 255 and the CHAP secret 100 alphanumeric characters.
-Required privilege: Host.Configuration.Storage Partition Configuration
-Configuration>Storage Adapters>Properties>General Tab>CHAP

6.  Configure VMkernel port binding for iSCSI Software multi-pathing  (pg. 32)
-You must create one VMkernel port for each network adapter before you can set up multipathing
-esxcli swiscsi nic add -n <port_name> -d <vmhba>

7.  Discover LUNs  (pg. 60)
-Perform a rescan each time you create a new LUN on the SAN
-By default, the VMkernel scans for LUN 0 to LUN 255 for every target (a total of 256 LUNs).
-You cannot discover LUNs with a LUN ID number that is greater than 255.
-Modify the Disk.MaxLUN parameter to improve LUN discovery speed.
-Configuration>Advanced Settings>Disk>Disk.MaxLUN

8.  Identify iSCSI addressing in the context of the host (pg. 10)
-IP Address
-iSCSI name (world wide unique name)
-iSCSI alias (friendly name, not unique)

-IQN (iSCSI Qualified Name) format  iqn.yyy-mm.naming-authority:unique name
-the naming authority is listed in reverse syntax

-EUI (Enterprise Unique Identifiers) format  eui.<16 hex digits>

Tools
-iSCSI SAN Configuration Guide
-Product Documentation
-VMware vSphere Client
-esxcli