Duplicate MAC addresses in Hyper-V
We had some strange, seemingly random, things happening in our environment lately, but have had a hard time tracking down what was going on. We run a Hyper-V environment consisting of 3 host servers in a failover cluster with Server 2008R2 and an iSCSI SAN. The other day I noticed we were running out of room on our PRTG server (virtualized). When I initially created that server and setup PRTG I didn’t put a whole lot of thought into the configuration and just installed everything using defaults. The default setting puts all the data on the C drive. To fix this I created a new volume on the SAN, added two new network adapters to the PRTG server, configured everything and moved the data to the new volume. The next day we noticed a bunch of errors with iSCSI connections dropping on a different server.
The first step in troubleshooting is to ask, “What was the last change?” In this case it was adding that volume to the PRTG server which was now randomly showing no status at all. The bottom line is that there were multiple network adapters with the same MAC addresses. The indicator was errors on one of the VM hosts saying, “P ort x was prevented from using MAC address y because it is pinned to port z.”
Each Hyper-V host has its own pool of MAC addresses which can be seen by opening Hyper-V manager and clicking Virtual Network Manager. I’m not quite sure how it happened but two of the host servers had the same MAC address pool! This is a huge problem because the network adapters keep the same MAC address even after being migrated to a different host. The MAC address only gets regenerated after a reboot of the virtual machine. This blog explains the MAC regeneration in more detail: http://blogs.msdn.com/b/virtual_pc_guy/archive/2010/05/14/hyper-v-and-dynamic-mac-address-regeneration.aspx
First I needed to find out which network adapters had the same MAC addresses. I’m sure there is probably a PowerShell command I could have used but I didn’t want to spend a bunch of time trying to find it. I just checked each machine and documented the MAC addresses in a spreadsheet and found the duplicates. Then I assigned static MAC address to the duplicates. That was the first step. Next, I migrated all the virtual machines off one of the servers (vm2) and changed its MAC address pool so each of the 3 servers had its own pool that didn’t overlap with the others. I let that sit for a little while and checked logs again. I had the same error on vm3 showing that one mac address was still pinned. Since I had a list of all the MAC addresses it was easy to find the virtual machine. It was a Windows 7 64 bit machine that runs security scanning software and it was the only machine with that MAC address, so I shouldn’t be getting the errors. I migrated it to vm2, which didn’t have any machines on it, and rebooted it. I verified that it picked up a new MAC address (00-15-5D-07-02-01), so I let it sit for a few and checked the logs again. Now vm2 had the same error message indicating Port x was prevented from using MAC 00-15-5D-07-02-01? But the server assigned that MAC address to the machine and it was the only machine on that host. I assigned a static MAC address to the machine that was outside all the other pools and the errors went away.
At this point I was thinking it was a Windows 7 problem so I started thinking about how I was going to manage tracking static MAC address. That was not an appealing thought, then it donned on me that we had another Windows 7 virtual machine that didn’t have this problem. I couldn’t find anything on the net so as a last ditch I removed the network adapter and added a new one. That fixed the problem and now everything is working the way it should.
In summary, make sure each VM host machine has its own MAC address pool that doesn’t overlap with any other VM host. http://blogs.technet.com/b/tonyso/archive/2009/09/17/hyper-v-how-to-manage-mac-address-conflicts.aspx