Not able to add ESX host into Virtual Center

I was trying to resolve following error message ,this look straight forward?

No it is not as simple as it looks. So I started troubleshooting this

1. I restarted vpx/hostd using following command “service mgmt-vmware restart”. This had no effect at all. Tried couple of time without any luck.
2. We then disconnect the host and tried to reconnect but it was not responding at all.
3. I was able to ping host by netbios name and IP address as well. Host was up and running.
4. I was not able to login to the host bypassing VC. This seems to be bigger problem then it has been anticipated.
5. Surprising part was when I restart mgmt-vmware service all seems to be OK but even though I decided to reinstall the VPX agent

service mgmt-vmware stop

service vmware-vpxa stop

Get the currently installed vpx version:

rpm -qa grep vpxa

This should return something like "VMware-vpxa-2.5.0-84767"

Now remove the vpxa agent:

rpm -e VMware-vpxa-2.5.0-84767

This also can be done using following way :

• On the vCenter Server, look for the following files in C:\Program Files\VMware\Infrastructure\VirtualCenter Server\upgrade

vpx-upgrade-esx-7-linux-104215

vpx-upgrade-esx-7-linux-104215.sig

• copy the files to the ESX host and run the following commands:

sh vpx-upgrade-esx-7-linux-104215

service mgmt-vmware restart

6. Now I thought let me reconnect the host back because that is the time it does push the VC agent back into ESX host. But it did not respond back to reconnect command .
7. Though this host was in cluster but I will not be able to put host into maintenance mode since it was in disconnected from VC.Now it looks like I really have to reboot the host but I did have live production VM.
8. I checked the vm status and all were up and running. We decided to check logs under vmfs partition and guess what we were not able to cd into it. This is the problem because vmfs partition does response to “hostd ” and if there is something wrong with vmfs then we can anticipate such kind of problem.

9. We tried rescanning HBA and it was getting into hung state. At this time we decided to check the health of filer and as soon as I logged in I found following comment

Can this be the problem ? No this was not the problem. Storage admin added some more space and this message went off. But I was not still able to get my HBA response to rescan.

10. I then called my onsite engineer to have physical look at a HBA and guess what he told me. He told light at HBA is not blinking. Is my HBA dead (We have 4 port but still using one port ,reason not to be reveled ) ? I asked him to put in next card on the port and light started blinking. I still can not do anything by rescanning. I have to reboot the host. I killed all the VM’s and then rebooted the host.

This should have been straight forward issue but guess where it turn out to be. Nice troubleshooting experience.

What to do if “ifcfg-vswif0” file is deleted?

I usually change host name and IP address by following my earlier blog. What happened this time was not sure but I lost "ifcfg-vswif0" file and host were out of network. I still can see ""Service Console " and vSwitch0 had all the port group. Now this is very confusing situation , if I create new service console ,will I lost all the portgroup . The answer is NO. So I went ahead and created new Service Console port using following command

[root@server root]# esxcfg-vswif –a vswif0 –i 192.168.1.10 –n 255.255.255.0 –p "Service Console"
['Vnic' warning] Generated New Mac address, 00:50:xx:xx:xx:xx for vswif0

To my surprise I found that all the port group were intact and I got back my "ifcfg-vswif0" file with new MAC address.

Experience Installing VM tools on Citrix P2V server

One of the Admin did P2V for Citrix Server and was not able to install VM tools onto it. While I wrote couple of blogs for Citrix Virtualization

1. Best Practices for deploying Citrix on VMware ESX

2. Troubleshooting P2V Citrix Box on Vmware

3. Clone/Copy PS4.5

But my dear college fail to refer these and complain that he is not able to power on the VM after P2V.

1. I  tried and status of VM remain same . Red and Green button both were  ON

2. So I decided that I will create a new VM and attach the HDD and see if I am able to power on VM. This is make the thing clear if P2V was an issue OS .  When I attach this P2V HDD to the newly created VM , I got a strange message that  file is locked.

How come file is locked . I have shutdown and unregister the VM . I logged into service console to check the status of VM.

‘Ps –ef ‘  did not help. The one which help was

‘ps axuw | grep vmx’

‘ps axuw | grep vmware-vmx’

‘ps auxfww | grep  <vm name>’

To my surprise this shows that VM is still running. So it was clear why it was not allowing me to attach HDD to new VM.

Even ‘esxtop ’ shows that vm was running. Killed all this PID and I was successfully able to attach the HDD to newly created VM and was able to power on. This clear my doubts that it was not the VM neither the OS.

This also raises doubts how this file did got locked after P2V. May be he was following wrong method? Since this happed second time I have decided to investigate entire P2V process which our folks were following.

3. HDD was reverted back to original P2V machine and powered back ON. VM tool was not installed so I decided to install VM tool. I got following error message .      

           

Why not since this is running Citrix and it is suppose to run in Application mode. Message which comes is really confusing and need not to be followed.

You can change terminal service mode from the command line itself

To determine the current mode on the Terminal Server server, run the change user /query command at a command prompt.

Click Start, and then click Run.

In the Open box, type cmd, and then click OK.

At the command prompt, type change user /install, and then press ENTER. The following message appears:

User session is ready to install applications.

Type exit, and then press ENTER.

Add or remove the programs that you want.

To switch a Terminal Services server into Execute mode, follow these steps:

Click Start, and then click Run.

In the Open box, type cmd, and then click OK.

At the command prompt, type change user /iexecute, and then press ENTER. The following message appears:

User session is ready to install applications.

Type exit, and then press ENTER.

When you install programs in Install mode, Terminal Services tracks all registry entries, and the HKEY_CURRENT_USER information is primarily written to the following registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Terminal Server\Install

When you finish the program installation, by clicking Finish or by typing change user /execute, the system returns to Execute mode. The registry information that was written to the HKEY_CURRENT_USER registry hive during installation is written to the HKEY_CURRENT_USER registry hive for each user when they log on to the Terminal Server

SOURCE : KB252330

After this I was able to install VM tools

Upgrading VC2.5 to vSphare2.5 U4

Our environment was running virtual center 2.5 base edition. SQL was running one a shared database and virtual center server on one of the virtual machine. When we planned to upgrade to vShpare2.5 U4 we decided to move everything to physical server with database running on the same server. We shut down the VC service and disabled it so that no updates are happing on the database. After that we planned to perform database from SQL2000 to SQL2005. Once backup completed then we install SQL2005 on windows2003 server.

We then restored the old database on new SQL2005 server. Created new ODBC for SQL server.

With this we are ready to install vShare2.5 U4. It is quite simple and straight forward. It will detect the version of database and prompt for upgrade. We also have to insure that we have shut down VC license server and copy the license file to the new VC server.

After the upgrade is over we have to setup license server pointing to license file. Update the DNS and alias pointing to new vSphare 2.5 U4 server. We are ready to fire our VC client and point to the same "alias" . We may prompt to upgrade the client or it may work with same client.

Here are some surprise:

You will find all the host in disconnect state. DO NOT PANIC. This is normal because all the agent which runs on host is still pointing to old VC server. We have to perform reconnect with all the esx host. Yeah it is pain in the a#$%& and there is no work around. But good part was no permission change, no change in folder hierarchy . This gave me little bit of relief. I had pain with my ESX 2.X host connecting back to new VC. But everything went very smoothly .

It was cool experience to share.

Now new feature of vSphare 2.5 U4 which I did notice was :

You can perform hot cloning , yes hot cloning . You can clone running virtual machine without any packets drop. When it is at 3% then it think a lot J but after that it is very fast and user does not realize that machine is getting cloned . What I want to figure out is till what time that changes are captured? This will help me understand more about hot cloning.

How present NetAPP ISCSI LUN to The Windows Host

I was give assignment to present ISCSI LUN to windows box. I have never done this before. We have to present storage from NetAPP filer FAS3050.This windows box was suppose to run SQL database. Earlier to present storage from filer ,MS ISCSI initiator along with SNAP mirroring were used. It was costing more than HBA’s .So we connected Qlogic HBA “QLE4062C ” and tested it from BIOS (Ctrl + S) to the filer.

1. Once windows is installed then make sure that we have driver for Qlogic card is installed. Because driver don’t get installed by default. Once driver is installed then you can find Qlogic card under device manager

2. After Qlogic card is installed then we have can also test the connectivity to the filer from card using device

3. But this is not the end of the story . My real pain started how to enter target IP .After searching Qlogic website I found that  “SANSurger HBA Manger “ is my friend

http://driverdownloads.qlogic.com/QLogicDriverDownloads_UI/SearchByProduct.aspx?ProductCategory=82&Product=1037&Os=64

4. Download and Install SANSurfer iSCSI HBA Manager in client mode

5. Make sure you choose “iSCSI  GUI and Agent” so that you can have agent and we can also do some additional troubleshooting in case there is problem

6. It will ask for destination to install

7. Select for all the Users Profile

8. Once it is installed lunch it using “local host”

9. Once it is connected it will show all the physical card as well as port

You have to expand the card to see all the ports. Before we configure the lun , we have to ensure that SAN admin has created Lun for WINDOWS. We also have to share IQN with SAN admin

10. You can find the link status and IQN number from this, This needs to be shared with Storage person

11. Select wizard from the top and select “General Configuration Wizard”. Select HBA port and then next

\

12. It will give driver version and MAC address

13. Provide IP address for the HBA

14. Don’t choose anything if you are not using IPV6

15. Don’t select iSNS if you are not uins Storage Network Server

16. This part is very crucial . We have to add the target IP address of the filer. Click on the green + tab. It will ask for target IP address for

17. Select next after adding. It will give summary

18. Once we hit “Next” then it will give warning message

19. Now it is very important steps. It will ask for password and the password is “config”. This information you can find in admin guide. This is not root password not anything but default password for SAN surfer manager

. Once it finished you can find space under “Disk Management”