Strange issue with “UPTIME” for the host

I  found the strange issue with server update. When I looked at the service console it has uptime for 4 days but VC shows up for 147 days. I am sure service console is correct but why VC showing wired thing.

Describing Memory field on VC2.5

With VC2.5 console we see lot of field about memory and we all start wondering what exactly those field suppose to mean. Kit Colbert has consolidated all those field and posted a nice PDF

Vstor2 Converter Virtual Storage Driver service failed to start due to the following error

One of the VM’s after the P2V was throwing error message


Event Type: Error
Event Source: Service Control Manager
Event Category: None
Event ID: 7000
Date: 10/11/2008
Time: 12:41:43 PM
User: N/A
Computer: xxxx
Description:
The Vstor2 Converter Virtual Storage Driver service failed to start due to the following error:
The system cannot find the file specified.

It was really strange to find out why I am getting driver error when I had uninstalled all the drivers. After little bit of research I found that same physical machine was used for P2V using VMware converter and this version has some known issue
http://www.vmware.com/support/converter/doc/releasenotes_conv3.html

Manual cleanup required of some Converter Agent files from remote source physical machineIn some cases, the automatic cleanup of Converter Agent files from the system on which it ran is incomplete. If there are vmware-ufad-p2v-XXXXX subdirectories remaining under %SystemRoot% on the remote source physical machine after completion of the import (either successful or failed), automatic cleanup was not totally successful, and might cause future import attempts to fail. To clean up manually (recommended), follow these steps:
1. Launch regedit and navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services
2. Remove the stcp2v30 key
3. Remove the vstor2-p2v30 key
4. Remove all vmware-ufad-p2v-XXXXX keys
5. Remove all courier-XXXXX keys
6. Reboot
Under %SystemRoot%\system32, remove all vmware-ufad-p2v-XXXXX subdirectories


So when did P2V for the same physical machine everything got virtualized with the same settings and that was causing the problem. After looking at the registry of virtual machine I found the key for “Vstor2” HKLM\SYSTEM\ConterSet001\Services . Delete the key and rebooted the VM and error went off.

Strange Issue with ILO

I setup my esx host and for some reason “vmfs” partition was not created during the installation. I started creating “vmfs ” partition and left for the day. When I came next day and tried to ILO into the host I found that it is not allowing me to do so and throwing error message

I rebooted the host twice and check the ILO cable to make sure that no one messed it and found everything is OK. What the heck happened because all other host on the same switch was working fine. I called HP support and it took them 2 hours to find solution (NO call was not routed to third world country) . They asked me to upgrade the BIOS as well as ILO and directed me to http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareIndex.jsp?lang=en&cc=us&prodNameId=3288134&prodTypeId=15351&prodSeriesId=1121516&swLang=8&taskId=135&swEnvOID=1005#2913 .

Well I just feel like getting into the phone and hitting the tech support guy for frustrating me ☺ . I gave up on phone and thought of doing at my own. I went to ILO site and downloaded the firmware and thought of updating it

After that update I still had the same issue when I tried to lunch it from desktop. I used some other H/W box and lunch the ILO from it. To my surprise I was able to get into ILO. Was there anything wrong my desktop? Well I don’t think so because I was able to ILO into other ESX host. It is hard to digest the fact that it is happening only with one DL380 box.

Trying To Be Photographer Part16

One of my Telgu friend invited me to cover Telgu Annual Atlanta chapter meet to celebrate "Bathkama". I accepted it with full generosity without charging him back since this being my first event to show up my talent.

Troubleshooting P2V Citrix box on VMware

One of the VM’s was having some performance issue and I found that it was a citrix box which has been P2Ved . It was windows 2000 box and VMware converter was still installed. Who ever done the P2V did not uninstall the converter. Normally Citrix on VMware best practices suggest that you should never P2V. I started troubleshooting this box to find out what went wrong. I thought may be HAL is corrupted because it was showing 4 PROC. I tried running command systeminfo and it throws out the error that it does not recognize the command. Of course it does will not since it was win2K box. So here is the way.
From your desktop XP/WIN2K3 run the command

c:\systeminfo /s /U
e.g. c:\systeminfo /s 192.168.1.101 /U administrator
Type the password for administrator:********

This shows that it has following network card and some one forget to remove these as a post P2V option.

NetWork Card(s): 14 NIC(s) Installed.
[01]: Broadcom NetXtreme Gigabit Ethernet
[02]: Broadcom NetXtreme Gigabit Ethernet
[03]: RAS Async Adapter
[04]: WAN Miniport (L2TP)
[05]: WAN Miniport (PPTP)
[06]: Direct Parallel
[07]: WAN Miniport (IP)
[08]: WAN Miniport (Network Monitor)
[09]: Broadcom NetXtreme Gigabit Ethernet
I opened my own blog for best practices for Citrix on VMware and started following all the steps

1. Uninstall the VMWare converter tool.
2. Delete all the unwanted NIC driver including vmware driver (Make a note of IP)
3. Uninstall the VMware tool.
4. Shut down the machine and then reduce the number of processor to two rather than 4 and that’s what best practices suggest.
5. Install the driver and unselect driver for memory.

Dang and now the performance of this box was excellent.

Start Virtualization Initiative

One with the name Satyaprakash Nivarthi started discussion on linked group “Official VMware Virtualization Group” that
I have been given a mandate to kick start our Virtualization initiative by piloting either DEV or QA environments. Which one do I start? How do I plan for the initiative? Any documentation available?


Then he also provide more inputs like :
Thanks. Yes, the first plan is to choose ESX and download the free version of ESXi. Criteria is a bit hazy now but contains the following: - Consolidating 15-20 servers to 2 or 3
- 99.99% availability
- Creating new Test environments quickly
- Disaster Recovery
- Cost saving
- Backup and Restore


Is there any documentation available regarding the planning aspects of Virtualization?

To start with follow this:
Procurement phase:
Hardware :


What kind of hardware you are going to use for testing phase?
This will depends upon various factor like memory/CPU/NIC/Storage. Now if you say you want to migrate X virtual machine with Y amount of memory then you have to go with X*Y +30% extra memory
Similarly you have to go for CPU and Storage.
CPU if you are going to have total X machine with Y Hz of CPU =X*Y* No. of Core. If you are placing 2vCPU machine that does not mean it needs 2 Physical CPU. CPU scheduling in VMware world happen in different way.
For Storage start with local storage. So if you say you are going to place XGB of Virtual Machine then go for X+40% of extra space because of Swap files and future expansion. VMware hypervisior does not consume much space. Say if you have local 1TB of space then you would get around 950GB of space to play around for VM’s.


NIC is the crucial part of the Hardware planning. But to began with use DUAL gig bit NIC. One dedicated for service console and one for Virtual Machine Network. Here you also have to consider if you want machine with multiple Network which means you need to perform VLAN trunking on your Virtual Machine Network.

So if you have decided on above configuration then research for Hardware on internet. Contact multiple vendor and ask for Quotes. I can suggest you to go for HP DL385 which will have 8CPU (4*4) with 32 GB of memory with 1TB of local HDD. Add dual 1GB NIC into it. In US such configuration will cost you around 8K to 10K based on vendor you choose.

Assembling Hardware:
Now once you hardware with you, it has to be racked and cable it. Make sure you have connected you NIC to two different SWICH port. Now one NIC will be assigned as a service console and one will be your VM network. Work with your N/W engineer to assign the port to appropriate VLAN. Make sure you have physical access to the machine using ILO or whatever you are using. This will help you with installation. Once you are done with cabling now you need to make sure your machine is set to correct RAID level or else you will not see all the 1TB or whatever you have space you want. I would usually set for RAID 5 and select all the space. Also make sure you have enabled your BIOS for Virtualization or else you will have difficulties with 64 bit machine.

Installing ESX:
Before you start installing you should be ready with following information:
Host name : FQDN name
DNS: Primary and secondary
IP address: This IP address will be assigned to Service console or management network. All subnet and Gateway as well.
Once you have all the information in hand start your installation (Assuming you have already downloaded ESX version. Remember that trial version last for 45 days after that you cannot power on ANY VM’s on it. But if any VM’s is powered on then it will remain powered on) with all default setting (I am trying to make things simple for you instead complicated). Once you have completed the installation you need a management interface to manage it.

Management of Physical Host:
You need to have Virtual Center client to manage the host. Download the Virtual Center client on your machine and connect to the host. You need to use name as DNS name of IP address. User name should be “root”. If everything is good you will see beautiful interface through which you can manage the host.
Post Configuration:
Once If you are into VC then you have to setup VM network. By default it will have all default settings. You need to configure VM network with appropriate VLAN ID. And if you have decided to have VLAN trunking then you need to have portgroup created for each individual network.

Migrating Physical machine to ESX : Assuming that your host is ready then use free convertor to convert your physical machine to Virtual Machine. Once you have migrated to VM power of your physical machine.

Expand C: drive for Virtual Server

Before you begin, make sure that you do not have an active snapshot on the VM, extending a virtual disk with a snapshot will cause corruption

Extend the boot volume of Windows Server 2003 Virtual Machine

To start, I have a Windows Server 2003 Virtual Machine that has a 5.3G disk allocated to it, I need to expand this disk to 10G.

Step 1: Power off the virtual machine that holds the boot volume that you want to extend.

Step 2: Make a backup copy of your virtual disk, this is optional but if you mess up don't call me unless you're willing to pay.

Step 3: From the service console, increase the size of the .dsk or .vmdk virtual disk file. This can also be accomplished through the Virtual Infrastructure Client if you are using VirtualCenter 2.x+.

[root@esx-test local]# ls -lah test.vmdk

-rw------- 1 root root 5.4G Jul 18 13:57 test.vmdk

Extend the virtual disk with vmkfstools. The input to the -X switch is the size that you want the disk file to be not the size you want to extend the disk file by.

[root@esx-test local]# vmkfstools -X 10G test.vmdk

View the new size of test.vmdk

[root@esx-test local]# ls -lah test.vmdk

-rw------- 1 root root 10G Jul 18 13:57 test.vmdk

Step 4: For this step you will need an additional Virtual Machine running Windows Server 2003. Power off the second Virtual Machine, and add the disk from the first Virtual Machine to it through the mui. Power up the second Virtual Machine and verify that the imported disk has unallocated space on it.

From the run menu type "diskpart.exe" to enter the command line utility to resize disk partitions in Windows Server 2003.

The command list volume will show you all the available volumes. Select your volume as shown below. select volume 1 corresponds to the "D" volume that I want to extend. Finally extend the volume with the extend command.

If all goes well, the partition will be immediately extended under the Disk Management snap in.

Step 5: Shut down the second Virtual Machine and remove the disk from the second Virtual Machine. Power on the first Virtual Machine and check out your new space.

ESX network configuration automation with esxcfg-vswitch

ESX network configuration automation with esxcfg-vswitch

For VI3 environments with multiple VLANs, configuration accuracy is an important step in providing virtual machines with the correct connectivity. Using the esxcfg-vswitch ESX host commands can make life as a VMware admin easier, but there’s a lot to learn. Lucky for you, I have taken the time to familiarize myself with the esxcfg-vswitch ESX host series of commands and have documented them for quick network configuration recreation. A word of caution, however; esxcfg-vswitch is an interactive command and if you are experimenting, you should work with a host in maintenance mode to avoid causing an interruption to your live workload.

In describing how to use the esxcfg-vswitch command, I want to first explain some practice points that I use in my VI3 environment. Most importantly, I create a virtual switch exclusively for the service console and VMkernel (VMotion) roles, and a separate virtual switch for all of the guest virtual machine port groups. The port groups on this separate virtual switch will have 802.1Q VLAN tags applied. With that, let’s jump into the command.

One of the first things you can do is inspect the current virtual switch configuration. This is done with the following command:

esxcfg-vswitch -l

The result will be something similar to what is shown below:

To recreate networks, we can use the esxcfg-vswitch to do this task. My preference is to remove the existing virtual switch, and re-create it to expected configuration. Here is a sample script that will first remove and then re-create the entire virtual switch and five port groups with their corresponding VLAN identifiers:

esxcfg-vswitch -d MainGuestVirtualSwitch

esxcfg-vswitch -a MainGuestVirtualSwitch

esxcfg-vswitch –link=vmnic3 MainGuestVirtualSwitch

esxcfg-vswitch –link=vmnic4 MainGuestVirtualSwitch

esxcfg-vswitch –add-pg=PrivateNetwork MainGuestVirtualSwitch

esxcfg-vswitch –add-pg=ShopFloor MainGuestVirtualSwitch

esxcfg-vswitch –add-pg=BackBuildingUpper MainGuestVirtualSwitch

esxcfg-vswitch –add-pg=BackBuildingLower MainGuestVirtualSwitch

esxcfg-vswitch –add-pg=CoreCorporate MainGuestVirtualSwitch

esxcfg-vswitch –vlan=334 –pg=PrivateNetwork MainGuestVirtualSwitch

esxcfg-vswitch –vlan=332 –pg=ShopFloor MainGuestVirtualSwitch

esxcfg-vswitch –vlan=236 –pg=BackBuildingUpper MainGuestVirtualSwitch

esxcfg-vswitch –vlan=237 –pg=BackBuildingLower MainGuestVirtualSwitch

esxcfg-vswitch –vlan=100 –pg=CoreCorporate MainGuestVirtualSwitch

A couple of notes on this: MainGuestVirtualSwitch is the name for the virtual switch instead off the default vSwitch1 (remember the first one - vSwitch0 - is for service console and VMkernel). There are two interfaces available to this virtual switch, vmnic3 and vmnic4. The port group names are the friendly names of the networks that are used in VI3. After running this script, the following will immediately appear in the networking configuration section of the host in the VMware Infrastructure Client:

I usually go into the virtual switch after it is created through script to make both interfaces active, instead of the default scripted result of one as active and subsequent vmnic interfaces as standby. Good information on esxcfg-vswitch is a little light, but VMworld 2006 had a couple of mentions of it and virtualization expert Mike Laverick makes some mentions of esxcfg-vswitch on the RTFM site.