Step by Step guide for Installing HP Insight Management Agents on ESX server

Why there is not a definitive guide on the Internet I do not know. HP’s site for the install is horribly vague, and the VMware site is similarly vague, referencing the HP site. Oh yeah that helps. Since I am not a linux geek by any measure, it was a bit frustrating. So I had to write my own. For anyone that happens to trip here from google, enjoy. These are written for the uninitiated.

Installing HP Insight Management Agents on ESX Server

In this case, I am using HP Insight Management Agents for VMware 8.1.0 (available here). I am also using ML370 G5 and DL380 G5 Proliant servers with VMware versions 3.0.1, 3.0.2, and 3.5 and a pre-existing HP SIM server running in my Windows Server environment.

(Click through for the full instructions)

Note: do not include the quotes (”) when entering commands.

PREP:

1 Download the correct Insight Manager agents for VMware.

2 Unzip the tgz file using WinRAR or something similar, and peruse the readme for up-to-date info.

3 Using the ILO utility or the physical console, hit Alt-F1, login as root at the command line.

4 To allow root ssh access in ESX as well as enable sshd client access (SFTP, SSH), we have to change the local sshd_config file.

5 At the command line, type: “vi /etc/ssh/sshd_config”

6 Go to the line that says PermitRootLogin, change from “no” to “yes”

7 If you are unfamiliar with vi, use arrow keys to navigate, put cursor on the “n” of “no”, hit the i key to insert, then type in “yes” and delete/backspace “no”. Then hit ESC twice to exit Insert mode. Type colon (:) to get the in-editor command line, type “wq!” and hit enter. This will save the doc and exit vi.

8 Back at the command line type: “service sshd restart”, hit enter.

9 After the service restarts, type “logout” to exit the console, hit alt-f11 to go back to the esx screen.

REMOTE, FROM PC:

1 Download WinSCP or a similar SCP utility (PuttySCP, etc). You will also need PuTTY for this process as well.

2 Open WinSCP, connect to your server with root, and upload the .tgz file you just downloaded to the /home folder on the remote system. You can close WinSCP after the transfer is done.

OR

Select the host and browse the datastore where we have to upload the file

Browse to the folder and then select and upload the file

3 Log in to the remote system using PuTTY, login with root. This should look and feel like the console.

4 Switch to /home directory: “cd /home”, where you copied the .tgz file.

5 Issue the following command to unzip the contents: “tar -zxvf hpmgmt-8.1.0-vmware3x.tgz”

6 This will create two new directories (/hpmgmt/810) under your current directory.

7 Change to the new directory by typing: “cd ./hpmgmt/810″

8 Run the command “./installvm810.sh -–install”, it will start the install.

9 Follow the defaults of the wizard (in depth script answers are below), when asked for the public string enter “public” 2 times (it will not be visible), and be sure to have the HP SIM server’s IP or FQDN and the unique private trap string. Pretty much always answer yes when enabling ports and the snmpd daemon. The only things that you will have to change from default is the snmp string info (see script section below).

10 Once the wizard is done and it restarts the services, you should see: “HP Insight Manager agents have been installed successfully!”

11 To check if the configuration has succeeded, log in to the HP System Homepage https://(esx server ip or dns):2381/. You should see the servername on the right side. Use the root login to access the homepage.

12 To log out of the console and close PuTTY type: “logout”, then hit Alt-F11 to go back to the status screen.

13 You’re done! You can log into the web interface and send a test trap to your SIM host.

SCRIPT ANSWERS:

(answers from script can be edited from web interface of insight mgmt agents)

1 This script will attempt to install the hp insight manager agents, do you wish to continue (y)

2 This script will attempt to shut down pegasus CIM in order to manipulate the HP insight manager agents, do you wish to continue (y)

3 For accessing the system management homepage, the port for hpim service (2381) should be enabled in the firewall, do you want to enable this port? (y)

4 For allowing discovery byt HP system management manager, the port for hpim service (2301) should be enabled in the firewall, do you want to enable this port? (y)

5 Do you want to enable the snmpd service? (y)

6 For adding the HP systems insight manager certificate in the SMH, the port (280) should be enabled in the firewall, do you want to enable this port? (y)

7 Welcome to the hp system health application and … hpasm package installation. … Do you wish to continue? (y)

8 Note: Your SNMP stack can load the 32bit hp proliant management extension… Press enter to continue (enter)

9 Do you want to load the hp modules even though they may “taint” your kernel? (y)

10 Do you require SNMP agents? (y)

11 Do you require storage agent support? (y)

12 Do you require NIC agent support? (y)

13 Do you wish to use an existing snmpd.conf? (n)

14 Enter the localhost SNMP read/write community string: public

15 Reenter the same input to confirm: public

16 Localhost snmp read only community strings are already known. Do you wish to change it? (n)

17 Enter read/write authorized management station IP or DNS name: (10.100.227.135)

18 Enter SNMP read/write community string for management station (private ip): public

19 Reenter the same input to confirm: (your private string) public

20 Read only community strings are already set to rocommunity public, do you wish to change them? (n)

21 The SNMP trap community strings are setup like this: trapcommunity public, do you wish to change this? (y)

22 Enter default SNMP trap community string: (your private string) public

23 Trap sinks are set up like this: trapsink localhost, do you wish to change this? (y)

24 Enter SNMP trap destination IP or DNS name: (your private IP of SIM) 10.100.227.135

25 Enter trap community string: (your private string) public

26 System contact is set to: syscontact root@localhost, do you wish to change it? (n) (can change it later)

27 System location is set to: syslocation room1, do you wish to change it? (n) (can change it later)

28 Do you wish to disable hpsmh support? (n)

Are you ready for Virtualization?

Disclaimer: All views expressed here is mine and my current and past employer not endorse my view.  

Now the buzz word across IT world is Go Green  / Cloud computing /Go virtual /Green Computing. Everyone likes to use this word for various reasons but the million dollar questions 

1. Is your organization ready for it?
2. Is this being forced by your management?
3. Is your support function / Infrastructure ready to take up new challenges?
4. Corporate users also want to use same slogan (Go Green… etc.) ?
5. Does solution provider understand your requirement?

We will try to acknowledge  and outline some key points which is required to implement “Go Virtual”. 

Corporate would like to implement “Go Green” mantra. They want to do it because their competitors have implemented it. They implement it because they want to establish capability proposition for their clients/customer “We are at par with latest IT trend”. There are very few who really understand that it is not just the mantra but also a social obligation. By saving electricity and reducing e-waste, IT world will be repaying to society.

To answer 1^st question: Are you really ready for this change? This question should be asked to each individual in an IT organization. Go Green bring huge change inside the organization. It does not only bring changes to your support model but as an end user it is required a mind set to accept change.

Answer for 2^nd question; 

When management pass the bucks to their colleagues to sing “Go Green ” mantra they should take everyone into confidence. Management should also educate themselves about “Go Green” mantra before they expect their colleagues to follow. Such initiative comes from top or from middle tier. So how does this tier react with such initiative? 

For  example if these initiative comes from middle tier, most of the time it has influence from lower middle technical mindset who want to test few product and showcase their capabilities to their manager. Such sections of people do it 

a) To learn and upgrade their skill for the know reason.

b) Influence by some vested interest to test certain feature. Major drawback with this approach is business direction, as these sections mostly lack such skill. Resulting in the  bad investment. 

If the initiative is business driven then it requires involvement of every layer of the organization, starting from the individual who will be conducting POC to the individual  who would perform TCO . Most of the time such initiative are driven by business goal. Hence this goes through cycle of choosing the right product to right hardware and deploying right solution so that you can see expected TCO. 

In usual world Organization   send out the requirement to multiple vendors offering VDI solution. Vendor perform POC, showcase the capabilities and get management confirmation. Management then compares the available solutions along with the costing. Costing is the other deciding factor for choosing product and in my view it is a correct approach. In my experience, middle tier or lower middle tier always like to go with selective approach for various reason. These reasons can be either their personal interest or personal comfort level. Say for example if I am asked to work on Dell or IBM blade, I may not be   comfortable as I am on HP. Having said I should not tell management as well as management should not take this suggestion for granted. 

To conclude this post:

While choosing VDI solution for your organization we should follow these:

1. Choose correct VDI solution which suits your requirement. 
2. Choose correct hardware to implement these VDI solutions.
3. Training support folks on these technologies are key to make VDI implementation successful.  
4. Handling end user expectation is other key to success. 
5. Organization goal above individual goals. 

Hope this read is been useful and help to answer and understand few of your query

P2V using VMWare convert version 4.2

I was working with XenDesktop where I was suppose to import physical desktop and convert that as a gold image. I install VMWare converter on my laptop and connected to the Virtual Center Server . Once it is connected I can find that under plugin  manager the version shows as 4.2

Right click on the host and select

 

It will pop up and ask for all the relevant information

Make sure that if you are importing XP machine you should install Converter agent on it or else it will not detect it.

Coming to real cool feature which was not there with earlier version of VMWare convertor . You can edit only certain settings with earlier version but here you have lot many option .

The one which is new is “Synchronize Changes that occur to the source during cloning” . This would be very helpful when we are doing conversion for openfiles.

Now it also have feature where you can stop and start service at source and destination

 

This is other new feature which ahs been added.

Once it says finish , the jobs are submitted and rest all you can follow my previous blog for post P2V conversion

Virtual Center 4.1 on a dual homed server

When you have dual NIC and your virtual center server is dual homed . You need to make sure that IP which will be assigned to virtual center should be updated under Managed IP address

 

SRM Error :Failed to recover datastore

We had setup srm with 4.0 U1 and VM’s were on ESX3.5 U4. We setup replication across location and then decided to simulate DR using  “Test RUN” option.  It goes and mount the lun on ESX host fine but when  it try to recover VM’s it was failing with error “Error: Failed to recover datastore: ” .

We then tried to run this with console to NetAPP filer open and we found that

Filer 01> Fri Apr 23 05:04:36 EST [XYZNAP005: wafl.volume.clone.fractional_rsrv.changed:info]: Fractional reservation for    clone 'testfailoverClone_nss_v10745371_S_XYZESX013_14_15_16PP' was changed to 100 percent because guarantee is set to'file' or 'none'.

Fri Apr 23 05:04:37 EST [XYZNAP005: wafl.volume.clone.created:info]: Volume clone  testfailoverClone_nss_v10745371_S_XYZESX013_14_15_16PP of volume S_XYZESX013_14_15_16PP was created successfully.Creation of clone volume 'testfailoverClone_nss_v10745371_S_XYZESX013_14_15_16PP' has completed.

Fri Apr 23 05:04:37 EST [XYZNAP005: lun.newLocation.offline:warning]: LUN /vol/testfailoverClone_nss_v10745371_S_XYZESX013_14_15_16PP/lun12 has been taken offline to prevent map conflicts after a copy or move operation.

Fri Apr 23 05:04:37 EST [XYZNAP005: lun.newLocation.offline:warning]: LUN /vol/testfailoverClone_nss_v10745371_S_XYZESX013_14_15_16PP/lun11 has been taken offline to prevent map conflicts after a copy or move operation.

Fri Apr 23 05:04:37 EST [XYZNAP005: lun.newLocation.offline:warning]: LUN /vol/testfailoverClone_nss_v10745371_S_XYZESX013_14_15_16PP/lun9 has been taken offline to prevent map conflicts after a copy or move operation.

Fri Apr 23 05:04:37 EST [XYZNAP005: lun.newLocation.offline:warning]: LUN /vol/testfailoverClone_nss_v10745371_S_XYZESX013_14_15_16PP/lun10 has been taken offline to prevent map conflicts after a copy or move operation.

Fri Apr 23 05:04:37 EST [XYZNAP005: wafl.inode.fill.disable:info]: fill reservation disabled for inode 33411686 (vol testfailoverClone_nss_v10745371_S_XYZESX013_14_15_16PP).

Fri Apr 23 05:04:37 EST [XYZNAP005: wafl.inode.overwrite.disable:info]: overwrite reservation disabled for inode 33411686 (vol testfailoverClone_nss_v10745371_S_XYZESX013_14_15_16PP).

Fri Apr 23 05:04:38 EST [XYZNAP005: lun.map:info]: LUN /vol/testfailoverClone_nss_v10745371_S_XYZESX013_14_15_16PP/lun12 was mapped to initiator group srm_esx_host=0

Fri Apr 23 05:04:38 EST [XYZNAP005: app.log.info:info]: AMSVCS001PP: Disaster Recovery SAN Adapter Storage Replication Adapter 1.4: (2) Test-Failover-start Event: Disaster Recovery SAN Adapter executed Test-Failover-start operation with errors from OS major version = 5 ,minor version = 2 ,package = Service Pack 2 and build = 3790

Fri Apr 23 05:04:42 EST [XYZNAP005: iscsi.notice:notice]: ISCSI: New session from initiator iqn.2000-04.com.qlogic:qle4062c.lfc0852h55321.2 at IP addr 10.X.X.X

Fri Apr 23 05:04:48 EST [XYZNAP005: wafl.vol.full:notice]: file system on volume testfailoverClone_nss_v10745371_S_XYZESX013_14_15_16PP is full

Fri Apr 23 05:04:48 EST [XYZNAP005: scsitarget.write.failureNoSpace:error]: Write to LUN /vol/testfailoverClone_nss_v10745371_S_XYZESX013_14_15_16PP/lun12 failed due to lack of space.

NetAPP look at the error and told me that it is getting timed out during retry process and not really looks like space issue because “aggr” on which this lun was mounted had enough space.

I decided to test it myself and created two lun of size 100GB and 90GB . These lun’s had few VM with around 75% of free space.  I ran SRM in test and DR mode and both worked great. This gives me enough reason to believe that this is caused by space and not due to some bug.

I called NetAPP and shown him what actually I am doing. At this point they ran following command

Filer > df -r testfailoverClone_nss_v10745371_S_XYZESX001PP  (This is actually the cloned  volume which SRM were trying to mount). It found that that fractional space is filled and because of which cloned lun were not able to mount

During these test I understand that if  protected lun is totally filled and then you try to run test SRM (FlexClone mechanism is used) against it . You have to make sure that at recovery site the volume should double the size as it try to mount the cloned lun on the same volume

SRM with NFS: Important consideration while creating NFS LUN

While implementing SRM with NFS , we have to follow certain guidelines for creating NFS volume on filer or else while configuring array we wont be able to see those volume.
a. Exports must be in the /etc/exports file. Manual exports made with the CLI without the –p option
will not be discovered by SRM.

b. Exports must use values in the RW (read/write) security field in the /etc/exports file. Exports using
the default setting of rw to all hosts will not be discovered by SRM.

Example of a /etc/export line that would be discovered by SRM:

/vol/srm5 -rw=192.168.2.0/24,root=192.168.2.0/24

Example of a /etc/export line that would not be discovered by SRM:
/vol/srm5 -rw,anon=0
So on filer if you run following
Filer101> exportfs -q /vol/nfstest
/vol/nfstest -sec=sys,(ruleid=564),rw=192.168.2.0,root=192.168.2.0
Where as 192.168.2.0 if the vmkernal IP created on Protected ESX host.

This is from TR3671 (page 14) from NetAPP

SRM: Replicated devices could not be matched with data stores

When I was configuring SRM  SRA then  I was getting following error when array where rescanned

Someone suggested that

For ESX host 3.5 you would be enabling “LVM.EnableReSignature ” under  advance option for esx host. ESX4 performs this function automatically whereas in version 3.5 you had to enable it in the advanced settings section of the host configuration.

This did not fixed my problem yet and then I followed KB 1016862 and its resolution says

“Virtual machine components can be on an array which is replicated to another array.

However, VMware does not support virtual machine components on multiple arrays which replicate to a single array as the VMX configurations do not match in terms of UUID of the datastores.

Ensure that virtual machine components are not on multiple arrays that replicate to a single array.”

Well in our case it was different.  Then I start looking at config again. I found that VM’s were mounted using different export IP then what I have added under array manger.

What I meant is NFS IP were different for ESX host and Array manager. I changed the IP and now I was able to see the datastore.

Existing IDE disk is not supported at the moment- Migrating VM from 2.X to 4.X

We had ESX V2.X running in our environment with couple of VM’s. It has highly impossible to have this host into our VC. So we disconnected the host and then copy the VM’s over to 4.0 host. Now we cannot add it into inventories since it has older hardware. We tried creating new VM and then attaching the vmdk which fails . We tried enterprise convertor and then standalone to export the VMDK but it did not work. Finally we decided to clone the disk vmkfstools -i source.vmdk dest.vmdk We then created a new VM and then tried to attach the VMDK and we started getting error message "Adding existing IDE disk is not supported at the moment. IDE disks cannot be hot added or there are no free available IDE Controller slots.". Solution 1. Unregister this VM from Virtual Center 2. Remove all scsi0:* lines from the VMs config file using a text editor. 3. Open the disk desciptor file in a text editor and replace "legacyESX" with "lsilogic". The descriptor file is the few hundred byte vmdk file. 4. Repeat this step for all the virtual disk files for this VM . 5. Register the VM again 6. Add disk(s) to the VM If you do not edit the descriptor file and add the disk to a vSphere VM (with virtual hw v7) it will come up as a working IDE disk. Source

ILO Error 9005/9009/9008 with HP SIM agent on ESX4.0 U1

One of the reader of blog drop me an email with following message

I've installed the HP SIM (v8.3.1) onto my ESX v4.0 update 1 host and configured it using a config file. The config file has the appropriate information to send to my SIM server. Every now and then my host will generate 3 alerts

1: Event Name: (SNMP) Remote Insight/ Integrated LightsOut Self Test Error (9005)

URL: https://xxx:2381/

Event originator: xxx

Event Severity: Critical

Event received: 22-Apr-2010, 18:11:52

Event description: Remote Insight/ Integrated Lights-Out Self Test Error. The Remote Insight/ Integrated Lights-Out firmware has detected a Remote Insight self test error.

2: Event Name: (SNMP) Remote Insight mouse cable disconnected (9009)

URL: https://xxx:2381/

Event originator: xxx

Event Severity: Major

Event received: 22-Apr-2010, 18:11:53

Event description: Mouse Cable Disconnected. The Remote Insight mouse cable has been disconnected.

3: Event Name: (SNMP) Remote Insight keyboard cable disconnected (9008)

URL: https://xxx:2381/

Event originator: xxx

Event Severity: Major

Event received: 22-Apr-2010, 18:11:53

Event description: Keyboard Cable Disconnected. The Remote Insight keyboard cable has been disconnected.

I've tried searching HP and VMware, but as you have said, information is very sparse. The closest I've come up with is a mention about the HPSIM certificate in the iLO2 (http://forums11.itrc.hp.com/service/forums/questionanswer.do?admit=109447626+1271902147498+28353475&threadId=1375014 )

I suggested him to

1. Try reinstalling SIM agent again with host reboot.

2. I would also engage HP since SIM is free product from them.

3. Check ILO (Since I am not sure which DL version you are using) if there are any option or Insight web GUI where you can disable such events

But this the actual solution available on following blog found by Dave. Thanks Dave for finding solution

To disable the alerts add : exclude cmasm2d to the file: - /opt/hp/hp-snmp-agents/cma.conf and restarted the agents: /etc/init.d/hp-snmp-agents restart

How to setup Link mode for Virtual Center 4.0

Today I have done VC upgrade and here is blog for the same. During the VC upgrade it did not ask me for setting VC in link mode which becomes very much necessary when you have multiple VC across environment.

Few things you should know

1. Link mode feature is not available to you if you are holding license for “foundation or essential edition ”. This required stander edition of VC license.

2. It uses ADAM (Active Directory Application Mode ) database to replicate it’s configuration between multiple VC.

3. ADAM stores :

* Connection information (IP addresses and ports)

* Certificates and thumbprints

* License Information

* User Roles

4. Permission can be configured on VC basis.

5. Single VC can manage 300 ESX host with 3000 VM but linked mode can manage 1000 ESX host with 10000 VM’s

More features are available here.

You have to re-run the setup for VC server installation. It will walk you through wizard

It will then detect the VC instance and prompt you for VC link mode setting

By default the link mode configuration will be selected

It will then prompt you to provide server name for which you are connecting

Make sure you have selection as below.

And that is about it.

Why Virtual machine swapfile location is grayed out?

I was trying to figure out why Virtual machine swapfile location is grayed out as shown in the pics below

What I understood is that it inherits properties from cluster. If the cluster has been set to store the swap file on VM then the pic above will be like that.

So you need to choose second option if you have to store swap as per your convenient . Such configuration is done incase where is the requirement to have swap at separate location

Upgrading Virtual Center from 2.5 to 4.0

One of the best upgrade I have done from Virtual Center 2.5 to Virtual Center 4.0 U1. I have done many upgrades where you have to through many pain of redoing the work post upgrade but with 4.0 U1 upgrade it was smoothest and cleanest one. Just pop-in the CD and next –next period.

Here is how I performed it

1. Make sure you have exported all the relevant information from VC so that if any issue comes during the upgrade , you should be able to handle it. Also make sure you have done latest backup of SQL server used by VC. This is how the VC ISO menu looks like and choose

2. It will then detect the VC already running

3. Agree to license agreement and then proceed

4. Fill the correct information

5. t may give message like this and let SQL admin knows about it

6. Type user name with which it can authenticate. Remember this can be your user name since it is just for the authentication purpose

7. Choose the option below as shown and check the box which is mention or else it will not allow to move forwards

8. Here is the option which allow to run the account as a service . Best practice is to run with system account

9. Let this be at default

10. Relax and sit tight till it is done

11. This will do numerous thing during that process

12. Finally it will show like this

Congratulation you have completed the VC upgrade successfully . With my 100+ host environment , I only came across on host which was in disconnect state . It also understand about VC2.5 licensing .

Adding additional vmfs volume on ESX host

We may often come across situation that local VMFS volume is exhausted and you have to add additional VMFS. I have written two different blog regarding the same .

One was how to use ACU from web and other was to create additional vmfs if you not able to see the logical volume

In this one we are adding additional drives in the available DL380 and then creating additional vmfs partition.

1 Reboot the ESX host and get into ACU (Array Configuration Utility ) BIOS by pressing F8 on key board. This will bring screen where you can see menu for configuring Array . We need to select first one as shown below

Once we select that ,next screen will show all the available disk . We make sure that you select RAID 5 and then press enter to create this logical drive

It will then provide summary for the total logical space and asking to press F8 to save the configuration

Press continue to see the total logical volume created

It will then show the summary along with newly created logical volume . Here you can also see the previous logical volume

Reboot the host and go to virtual center . Select the configuration from the host and then choose add storage

It will then run the wizard for creating new VMFS partition . Make sure that you select Disk/Lun

It will then show the newly created logical disk

It will show the warning but continue with space creation

Give the name as _storage

Go with default

End it will provide summary

Now you will have two VMFS volume, one old one and other one newly crated