Search This Site

myTemplate

A Real World Template Design for Joomla

Login to VirtualVCP

In order to leave comments, please log in.





Lost Password?
No account yet? Register

Vi-Pedia Virtualization Wiki

Virtualization Wiki

RSS Feeds

Support VirtualVCP

Information and Resources on this site are 100% free, however it costs money to run this site. Please help support this site by giving a donation.

Home
Which ESX host is locking my files?
User Rating: / 1
Troubleshooting Tips - VI3/vSphere: Virtual Machine Operations
Written by Rynardt Spies   
Monday, 22 June 2009

I’ve found myself asking this very annoying question just last week again. Which one of the servers is holding a lock on a virtual machine log file that was last modified 3 months ago?

Last week I came across a problem where VCB failed a job while trying to perform a full backup of one of the VMs. This was because one of the log files for the Virtual Machine was locked on the SAN. VCB was therefore unable to copy the log file to the backup server and therefore failed the entire job.

Normally, a simple VMotion of the Virtual Machine to another host will solve this issue, but I wasn’t as lucky this time. So I thought powering off the VM will do it... Didn’t work! No matter what I did, I just couldn’t get the lock released on that file. One of the ESX hosts in the cluster was holding on to the log file, but how do I go about finding out which one of the 20 ESX hosts is was? To me, this sounded like a job for vmkfstools, and indeed it was. Well, sort off. Using vmksftools, I was able to retrieve the MAC address of the ESX host in the cluster that was holding on to the 3 month old log file.

The command is:

vmkfstools –D <path-to-vm>/filename

In my case this was;

vmkfstools –D /vmfs/volumes/iscsi-002-vmfs/WKSTN01/vmware.log

The output is then written to /var/log/vmkernel.

To get the output, simply do:

tail /var/log/vmkernel

This returned:

Jun 20 15:35:33 esx1 vmkernel: 23:02:22:35.020 cpu0:4174)FS3: 142: <START vmware.log>
Jun 20 15:35:33 esx1 vmkernel: 23:02:22:35.020 cpu0:4174)Lock [type 10c00001 offset 29190144 v 7, hb offset 4083712
Jun 20 15:35:33 esx1 vmkernel: gen 1881, mode 1, owner 4a2128d2-86a81c3a-ce30-000e0cc41e98 mtime 893]
Jun 20 15:35:33 esx1 vmkernel: 23:02:22:35.020 cpu0:4174)Addr <4, 53, 5>, gen 6, links 1, type reg, flags 0x0, uid 0, gid 0, mode 644
Jun 20 15:35:33 esx1 vmkernel: 23:02:22:35.021 cpu0:4174)len 312433, nb 1 tbz 0, cow 0, zla 1, bs 1048576
Jun 20 15:35:33 esx1 vmkernel: 23:02:22:35.021 cpu0:4174)FS3: 144: <END vmware.log> 

The MAC address of the host locking the file is reported in line 3:

000e0cc41e98

Now, this is the bit where I can’t make it any easier for you. Unless you write a script, (and I don’t have that much time at the moment) the only way to find the host with that MAC is to log onto each host via SSH and run: 

esxcfg-info |grep –i  ‘system uuid’

This will then return the UUID for the host you are on. If it matches the MAC retrieved using vmkfstools, then you know the process that’s keeping the lock is on that server.

So what process is locking the file? That I can’t tell you. I can only give you some tips as to how to find it.
1.       Power off the VM in vCenter;
2.       Log onto the service console of the host that’s locking the file;
3.       Try to move or delete the lock file from the service console of the locking host. This worked me. If it works for you, then good. If not, go to step 4;
4.       Try and see if there’s a process running with the filename that is locked;

ps –auxwww |grep <locked filename>

If it returns a line(other than the grep line) kill the process with “kill -9 <PID>"

5.       If it doesn’t return any processes under that filename, then try and search for a PID with the VM name that has a locked file:

ps –auxwww|grep <vm name>

If it returns a PID, kill the PID, as your VM was already powered off in step one and should therefore not have a PID on any host;

6.       If it still doesn’t work, leave a comment and we'll have a look at it ;-) 

 
Last chance to win a trip to VMworld 2009!
User Rating: / 0
The News - Latest News
Written by Rynardt Spies   
Wednesday, 10 June 2009

Here's a last chace to win a trip to VMworld 2009:

http://www.veeam.com/go/video/

 

 
New Release: Veeam Monitor 4.0
User Rating: / 0
The News - Latest News
Written by Rynardt Spies   
Tuesday, 02 June 2009

Image
Image: Veeam
Veeam Software has released Veeam Monitor 4.0 and Veeam Monitor Free Edition 4.0. The new version of Veeam Monitor Includes:

  • Support for VMware vSphere 4;
  • Storage monitoring;
  • Hardware monitoring;
  • More new features requested by customers.

 

I've not had time to install and play with Veeam Monitor 4.0, but if Veeam Monitor 3.0 is anything to go by, I'm sure Veeam Monitor 4.0 will be good. I'll install it and do a review sometime this week.

 

You can download Veeam Monitor 4.0 from: http://www.veeam.com/vmware-esx-monitoring.html 

You can download Veeam Monitor Free Edition 4.0 from: http://www.veeam.com/esxi-monitoring-free.html

 
Using Custom Attributes to Manage VMs
User Rating: / 0
Troubleshooting Tips - VI3/vSphere: vCenter
Written by Rynardt Spies   
Monday, 01 June 2009

This may not be the most technical post, but it should hopefully give VM administrators some ideas on managing their VMs.

Despite having tools like VirtualCenter, keeping track of your VMs can still be a mission. Today I look after thousands of virtual machines running on hundreds of ESX hosts in several data centres. Most of these VMs are production systems, some are clones of production systems, some are test and some are dev. Creating and managing machines for new services is not always an issue. We have processes in place to control VM sprawl. We know which VMs belongs to which customers. We also know who to contact in regards to which VM. This is all documented in change records and CMDBs. However, having to go back to CMDBs and change records every time you need to know who owns a VM is a bit of a slog. Sure we’ve tried adding relevant information into the “Notes” Attribute, but it gets messy and some administrators “forget” to add all the information we need into the notes.

To try and keep track of who owns what, I use a simple but very effective tool inside vCenter to manage VMs. It’s the “Custom Attribute” function of vCenter that allows administrators to specify custom attributes for all the VMs and hosts in vCenter. Custom Attributes are by no means a new feature in Virtual Infrastructure or vSphere, yet a lot of administrators don’t use them as they simply don’t realise that custom attributes functions exists or what custom attributes are for. I’ve seen many virtual Infrastructures built on VMware VI 3 (small environments to large enterprise environments) and I simply can’t recall ever seeing custom attributes being used.

What are custom attributes?

Custom Attributes are attributes that a administrator can define for all Virtual Machines and Hosts in vCenter. The attributes are then displayed in the “Annotations” box for each VM. The custom attributes are also displayed alongside the VM Name, Status, State, CPU Usage, etc, when listing virtual machines in the Datacenter, cluster, host, and resource pool views.

Why use them?

Daily, we receive lots of clone requests for VMs. Clones are very good for application development and application troubleshooting. When an application on a VM goes pear shaped, the application vendor may request a clone for a system. They can then work on the clone system to try and fix some issues while the production system (if not broken completely) may in some cases still remain partially available to the end users. Once they are confident that they have resolved the issue, they can then implement the fix in the production environment.

This is all good and well, but here’s the problem. Clones are a drain on your resources. As administrator, you should be very careful to simply hand out clones as the requests come in. Virtual Machine clones can create a nightmare for administrators as they use huge amounts of additional disk space. Some of the clones we have today are in excess of 420GB. When creating clones, we need to make sure that the following information is recorded somewhere in vCenter:

·         Who requested the clone;

·         The requestor’s contact number/email address

·         The date that the clone was made;

·         The size of the clone on disk (all vDisk sizes combined + the amount of RAM assigned to the VM);

·         Does it need to be backed up (Yes / No);

·         Reason for the clone;

 

Specifying your own Custom Attributes

Creating custom attributes is very simple. In the VI (or vSphere) client on the menu bar, click “Administration” >> “Custom Attributes...”

You’ll now be presented with the Custom Attributes dialog:

Image
Create your attributes by using the Add button. I created Virtual Machine attributes for this post.

Once you’ve created your attributes, you’ll see them in each VM’s annotation’s box on the summary page.

Image
To edit the custom attributes, click

When you click on the "edit" label in the annotations box, you get:

Image
Edit Attributes Dialog
 

Fill out the information for each attribute:

Image
Attributes filled out

 When looking at the annotations box on the summary page of the VM, the custom attributes will be recorded and displayed for that VM:

Image
Annotations is displaying custom attribute information

The cluster view will also now display all the custom attributes. They can be filtered like any other VM attribute:

Image
Cluster View of custom attributes
 

 
<< Start < Prev 1 2 3 4 5 6 7 8 Next > End >>

Results 25 - 36 of 93