Could not power on VM - lock was not free

The other day we experienced an incident on the SAN storage with high latency and even loss of connection to the SAN. This can generate a lot of really unpleasant errors on the ESX hosts. Even after the SAN is brought back to a stable state we've seen hosts that won't boot, VM's that won't vMotion and VMs that won't power on due to file locks. 

If you receive a 'locked file error' (like screendump below) and your VM won't boot there are a couple of ways to go about it. This VMware KB article explains it quite well. Either you can cold migrate the VM to the  other hosts in the cluster (to find the ESX host with the lock) and then try to boot it from there or you can try to locate specifically which host has the lock.

If the vCenter log does not tell you specifically which files are locked, this can be viewed in the vmware.log which is located in the VM folder. If you just tried to power on the VM, then relevant info should be at the end of the log file.

In the example below, it is the swap that is still locked.

This can be verified by running the touch command on the locked file.
With vmkfstools you can get the mac address that has the lock:

# vmkfstools -D /vmfs/volumes///

In the screendump below, the MAC address has been highlighted.

The same info can be found in the /var/log/vmkernel log

Once you have the MAC address you can find a match by, for example, logging in to vCenter or onto the Blade enclosure. When you have a match, cold migrate the VM to the relavant ESX host and boot it.

Read More

Error during upgrade: The system call API checksum doesn’t match

Today, I got an error during upgrade from vSphere 4.0 to 4.1 stating something like:

The system call API checksum doesn’t match

There was a lot of similar lines filling the console. I was a bit worried that the upgrade had gone wrong even though I had done three similar upgrades before this one with no errors - and that I would have to reinstall in stead.

Luckily, I found this error description in the 4.1 release notes stating that a reboot will fix the issue. So I waited for a while to be sure that the upgrade finished, rebooted, and everything looks fine:

Link to release notes:

"ESX service console displays error messages when upgrading from ESX 4.0 or ESX 4.1 to ESX 4.1 Update 1

When you upgrade from ESX 4.0 or ESX 4.1 release to ESX 4.1 Update 1, the service console might display error messages similar to the following:
On the ESX 4.0 host: Error during version check: The system call API checksum doesn’t match"
On the ESX 4.1 host: Vmkctl & VMkernel Mismatch,Signature mismatch between Vmkctl & Vmkernel

You can ignore the messages.

Workaround: Reboot the ESX 4.1 Update 1 host. "

Read More

ESX 4.1 install error on BL460c G7 - NIC driver fails to load

The HP BL460c G7 is on the VMware HCL list for ESX 4.1. However, when trying to install ESX 4.1 there's an error during install - it fails to load drivers for the network adapter ("No network adapters were detected"). It doesn't help to update all firmware to latest version (even though this should be done in any case...) (Update 2011.02.18: This problem persists on ESX 4.1 U1)

This is a known error and there's a fix for it. However, it seems strange that the G7 blade has made it to the HCL list...

The problem is that the Integrated NC553i Dual Port FlexFabric 10Gb NIC driver is not included in VMware's installation ISO for ESX 4.1. There are two ways to solve the issue. One is to load a custom set of drivers for the NIC and the other is to use an HP VMware install image. If you're using ESXi or scripted installation of ESX classic, then you have to use the HP image.

(Update 2011.07.19: Custom HBA driver should also be loaded during installation - simply load both ISOs)

Custom NIC driver from VMware can be downloaded here.

Custom HBA driver from VMware can be downloaded here.

(Update 2011.02.18: Apparently, the NIC drivers are updated quite frequently at the moment. Go to this main link and then 'plus out' Driver CDs to find the most recent one.)

HP image for ESX(i) can be downloaded from here.

The custom driver, when downloaded, is in an ISO format. To load it during installation, do the following:

• Upload the ISO file to where you have the ESX installation image
• On the Custom Drivers page in the wizard, choose Yes and click on Add. It will tell you to load the driver CD (see picture below)
• Unmount the ESX installation CD and mount the driver ISO in stead. This can be done without interrupting the installation. You will be prompted to verify the custom driver package, click OK (see picture below)
• That's it. at a later stage in the wizard, you will be prompted to reinsert the installation ISO. Do that when prompted.

Read More