Friday, October 23, 2009

Storage VMotion myth and deep dive

I had been involved so heavily in storage Vmotion that I decided to write about it. Nice source is here

First of all storage vmotion is

  1. Takes an ESX snapshot
  2. Then file copies the closed VDMK (which is the large majority of the VM) to the target datastore
  3. Then "reparents" the open (and small) VDMK to the target (synchronizing)
  4. Then "deletes" the snapshot (in effect merging the large VDMK with the small reparented VMDK)

Myth and deepdive:

  1. In a cluster if you have 5 host and have 10 luns. If you want to svmotion to 11th lun, the 11th lun should be visible to all the host in the cluster. It does not work on individual host
  2. You can do 1 svmotion per ESX host and maximum of 32 per cluster (Pheew)

  1. You can have 4 svmotion per lun . Which means you have many source lun but at a time only 4 svmotion will done at any target lun

  1. Storage vmotion can happen be only between on datacenter and cannot happen between two different datacenter (phewee).
  2. Storage VMotion cannot be used on a virtual machine with NPIV enabled

Batch Storage Vmotion

I had to do batch svmotoion and I decided to follow this guy nice article

Place postie.exe (Freeware), AutoSVmotion.vbs (Contents at then ) in the Remote CLI (Download and install it from VMWare site) bin directory (default location is "C:\Program Files\VMWare\VMWare VI Remote CLI\Bin"). Also create a batch file in this directory, this will be used to call AutoSVMotion.vbs. In the batch file put all the VM's that you require migrating in the below format:

cscript.exe AutoSVmotion.vbs %VC% %Username% %Password% %Datacentre% %SourceDatastore% %VMXFilePath% %DestDatastore% %SMTPServ% %ToEmailAddress% %FromEmailAddress%

Example of the batch file:

cscript.exe AutoSVmotion.vbs VC01 Admin P@ssw0rd London OLDLUN1 VM1/VM1.vmx NewLUN1 SMTP01 virtuallysi@example.com vikash.roy@test.com
cscript.exe AutoSVmotion.vbs VC01 Admin P@ssw0rd London OLDLUN1 VM2/VM2.vmx NewLUN1 SMTP01 virtuallysi@example.com vikash.roy@test.com

Example of the email output:

From: vikash.roy@test.com
Date: 13 September 2009 08:45:43 GMT+01:00
To: virtuallysi@example.com
Subject: SVMotion Progress Report

Successfully migrated VM1/VM1.vmx

Contents of AutoSVmotion.vbs

Dim WshShell, oArgs, sVCServer, sUsername, sPassword, sDatacenter, sSourceStore, sVMXLocation,_

sDestStore, sSMTPServer, sEmailAddress, sSVMotioncmd, iRetVal, sEMailText, sEmailCmd

On Error Resume Next

Set WshShell = WScript.CreateObject("WScript.Shell")

Set oArgs = WScript.Arguments

sVCServer = oArgs(0)

sUsername = oArgs(1)

sPassword = oArgs(2)

sDatacenter = oArgs(3)

sSourceStore = oArgs(4)

sVMXLocation = oArgs(5)

sDestStore = oArgs(6)

sSMTPServer = oArgs(7)

sToEmailAddress = oArgs(8)

sFromEmailAddress = oArgs(9)

sSVMotioncmd = "cmd.exe /c svmotion.pl --url=https://" & sVCServer & "/sdk --username=" & sUsername & _

" --password=" & sPassword & " --datacenter=" & sDatacenter & " --vm=" & Chr(34) & "[" & sSourceStore & "] " & sVMXLocation &_

":" & sDestStore & Chr(34)

WScript.Echo sSVMotioncmd

iRetVal = WshShell.Run(sSVMotioncmd, 1, True)

If iRetVal = 0 Then

sEMailText = "Successfully migrated " & sVMXLocation

sEmailCmd = "cmd.exe /c postie.exe -host:" & sSMTPServer & " -to:" & sToEmailAddress & " -from:" & sFromEmailAddress & " -s:" & Chr(34) & "SVMotion Progress Report" & Chr(34) & " -msg:" & Chr(34) & sEMailText & Chr(34)

Else

sEMailText = "SVmotion failed for " & sVMXLocation & " with error number " & iRetVal

sEmailCmd = "cmd.exe /c postie.exe -host:" & sSMTPServer & " -to:" & sToEmailAddress & " -from:" & sFromEmailAddress & " -s:" & Chr(34) & "SVMotion Progress Report" & Chr(34) & " -msg:" & Chr(34) & sEMailText & Chr(34)

End If

WshShell.Run sEmailCmd, 1, True

Tuesday, October 13, 2009

How to validate VMWare license ?

If you want to validate license file please refer to the URL
http://www.vmware.com/checklicense/ . Click validate and this your license will be validated.
This will bring the page with following message
Welcome to the license file checking utility. This tool will take a pasted license file and parse, reformat, and attempt to repair it. It will also give you statistics regarding the total number of licenses found and highlight inconsistencies that could potentially cause issues.
Currently, this utility only handles "server-based" license files (used with VirtualCenter). Host-based licenses that stand-alone on a single server are not supported at this time.

Saturday, October 10, 2009

Working with BMC and vif on FAS2020

We had a situation with FAS2020 where we had to connect two interface e0a/e0b  has to be assigned two separate IP but when NetAPP engineer commissioned FAS2020 ,they created vif and assigned both the physical interface e0a and e0b to it.

This is how back panel of FAS2020 looks like

clip_image002

The circled one is basically for BMC which is similar to ILO

FAS2020 have two Ethernet interface  .

So I had a challenge to  delete vif for FAS2020. It’s like virtual interface binded to two physical NIC and filer view can be accessed using vif . So we cannot make changes to vif using filer view . This needs to be accomplish either using serial interface  or BMC(Similar to  ILO which provide console access)

Couple of facts about BMC which I learned (remember I am learning NetAPP)

1. BMC has IP address and you need to do SSH to the IP not telnet.

2. For user ID you need to use “naroot” and password will be that of root.

3. Once you login to BMC console then you can login to system console using root and then root password

This is how to execute the command

login as: naroot

naroot@xx.0.86's password:

=== OEMCLP v1.0.0 BMC v1.2 ===

bmc shell ->

bmc shell ->

bmc shell -> system console

Press ^G to enter BMC command shell

Data ONTAP (xxxfas001.xxx.net)

login: root

Password:

xxxfas001> Fri Oct 9 11:08:06 EST [console_login_mgr:info]: root logged in from console

xxxfas001> ifconfig /all

ifconfig: /all: no such interface

xxxfas001> ifconfig

usage: ifconfig [ -a | [ <interface>

[ [ alias | -alias ] [no_ddns] <address> ] [ up | down ]

[ netmask <mask> ] [ broadcast <address> ]

[ mtusize <size> ]

[ mediatype { tp | tp-fd | 100tx | 100tx-fd | 1000fx | 10g-sr | auto } ]

[ flowcontrol { none | receive | send | full } ]

[ trusted | untrusted ]

[ wins | -wins ]

[ [ partner { <address> | <interface> } ] | [ -partner ] ]

[ nfo | -nfo ] ]

xxxfas001> ifconfig -a

e0a: flags=948043<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

ether 02:a0:98:11:64:24 (auto-1000t-fd-up) flowcontrol full

trunked svif01

e0b: flags=948043<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

ether 02:a0:98:11:64:24 (auto-1000t-fd-up) flowcontrol full

trunked svif01

lo: flags=1948049<UP,LOOPBACK,RUNNING,MULTICAST,TCPCKSUM> mtu 9188

inet 127.0.0.1 netmask 0xff000000 broadcast 127.0.0.1

svif01: flags=948043<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

inet xx.xxx.xx.xx netmask 0xffffff00 broadcast xx.xx.xx.255

ether 02:a0:98:11:64:24 (Enabled virtual interface)

xxxfas001> ifconfig svif01 down

xxxfas001> Fri Oct 9 11:10:23 EST [pvif.vifConfigDown:info]: svif01: Configured down

Fri Oct 9 11:10:23 EST [netif.linkInfo:info]: Ethernet e0b: Link configured down.

Fri Oct 9 11:10:23 EST [netif.linkInfo:info]: Ethernet e0a: Link configured down.

xxxfas001> Fri Oct 9 11:10:37 EST [nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.

xxxfas001> vif destory svif01

vif: Did not recognize option "destory".

Usage:

vif create [single|multi|lacp] <vif_name> -b [rr|mac|ip] [<interface_list>]

vif add <vif_name> <interface_list>

vif delete <vif_name> <interface_name>

vif destroy <vif_name>

vif {favor|nofavor} <interface>

vif status [<vif_name>]

vif stat <vif_name> [interval]

xxxfas001> vif desto4~3~ry svif01

xxxfas001> vif destroy svif01

xxxfas001> if config -a

if not found. Type '?' for a list of commands

xxxfas001> ifconfig -a

e0a: flags=108042<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

ether 00:a0:98:11:64:24 (auto-1000t-fd-cfg_down) flowcontrol full

e0b: flags=108042<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

ether 00:a0:98:11:64:25 (auto-1000t-fd-cfg_down) flowcontrol full

lo: flags=1948049<UP,LOOPBACK,RUNNING,MULTICAST,TCPCKSUM> mtu 9188

inet 127.0.0.1 netmask 0xff000000 broadcast 127.0.0.1

xxxfas001> ifconfig

usage: ifconfig [ -a | [ <interface>

[ [ alias | -alias ] [no_ddns] <address> ] [ up | down ]

[ netmask <mask> ] [ broadcast <address> ]

[ mtusize <size> ]

[ mediatype { tp | tp-fd | 100tx | 100tx-fd | 1000fx | 10g-sr | auto } ]

[ flowcontrol { none | receive | send | full } ]

[ trusted | untrusted ]

[ wins | -wins ]

[ [ partner { <address> | <interface> } ] | [ -partner ] ]

[ nfo | -nfo ] ]

xxxfas001> ifconfig e0a xx.xx.xx netmask 255.255.255.0

xxxfas001> Fri Oct 9 11:17:24 EST [netif.linkUp:info]: Ethernet e0a: Link up.

xxxfas001> ifconfig e0a xx.xx.xx.xx netmask 255.255.25Fri Oct 9 11:17:50 EST [nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.

5.0

xxxfas001> ifconfig e0a xx.xx.xx.xx netmask 255.255.2

xxxfas001> ifconfig -a

e0a: flags=948043<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

inet xx.xx.xx.xx netmask 0xffffff00 broadcast xx.xx.xx.255

ether 00:a0:98:11:64:24 (auto-1000t-fd-up) flowcontrol full

e0b: flags=108042<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

ether 00:a0:98:11:64:25 (auto-1000t-fd-cfg_down) flowcontrol full

lo: flags=1948049<UP,LOOPBACK,RUNNING,MULTICAST,TCPCKSUM> mtu 9188

inet 127.0.0.1 netmask 0xff000000 broadcast 127.0.0.1

xxxfas001> Fri Oct 9 11:20:03 EST [nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.

Fri Oct 9 11:21:16 EST [netif.linkInfo:info]: Ethernet e0b: Link configured down.

Fri Oct 9 11:21:29 EST [netif.linkUp:info]: Ethernet e0b: Link up.

Here you can find after changes how the ISCSI connection has been established.

Fri Oct 9 11:21:38 EST [iscsi.notice:notice]: ISCSI: New session from initiator iqn.2000-04.com.qlogic:qle4062c.lfc0908h84979.2 at IP addr 192.168.0.2

Fri Oct 9 11:21:38 EST [iscsi.notice:notice]: ISCSI: New session from initiator iqn.2000-04.com.qlogic:qle4062c.lfc0908h84979.2 at IP addr 192.168.0.2

Fri Oct 9 11:22:16 EST [nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.

FAS2020: The size must be a simple number like '3' or '512'

I was trying to create 1.3 TB of ISCSI lun on FAS2020 NetApp filer. It was throwing below error message. Looks like it has different version of DataONTAP running.

So what I did is I created 1TB of LUN without .0 so like 1TB. I then got below mention successful message.

I then supply the size as 1099529453568*1.3=1429388289639

When I recheck the lun size 1.3 TB.

Wednesday, September 30, 2009

Few Question about ballooning and config max.

One of the guy asked me following question . Just wanted to share with you all

1) what is ballooning in VMWARE

Ballooning

When the ESX host’s machine memory is scarce or when a VM hits a Limit, The kernel needs to reclaim memory and prefers ballooning over swapping. The balloon driver is installed inside the guest OS as part of the VMware Tools installation and is also known as the vmmemctl driver.

When the ESX kernel wants to reclaim memory, it instructs the balloon driver to inflate. The balloon driver then requests memory from the guest OS. When there is enough memory available, the guest OS will return memory from its “free” list. When there isn’t enough memory, the guest OS will have to use its own memory management techniques to decide which particular pages to reclaim and if necessary page them out to its swap- or page-file.

In the background, the ESX kernel frees up the machine memory page that corresponds to the physical machine memory page allocated to the balloon driver. When there is enough memory reclaimed, the balloon driver will deflate after some time returning physical memory pages to the guest OS again.

This process will also decrease the Host Memory Usage parameter

Ballooning is only effective it the guest has available space in its swap- or page-file, because used memory pages need to be swapped out in order to allocated the page to the balloon driver. Ballooning can lead to high guest memory swapping. This is guest OS swapping inside the VM and is not to be confused with ESX host swapping, which I will discuss later on.

To view balloon activity we use the esxtop uitility again from the COS (see below). From the COS, issue the command “esxtop” en then press “m” to display the memory statistics page. Now press “f” and then “i” to show the vmmemctl (ballooning) columns.

clip_image002

On the top we see the “MEMCTL” counter which shows us the overall ballooning activity. The “curr” and “target” values are the accumulated values of the “MCTLSZ” and “MCTLTGT” as described below. We have to look for the “MCTL” columns to view ballooning activity on a per VM basis:

● “MCTL?”: indicates if the balloon driver is active “Y” or not “N”

● “MCTLSZ”: the amount (in MB) of guest physical memory that is actually reclaimed by the balloon driver

● “MCTLTGT”: the amount (in MB) of guest physical memory that is going to be reclaimed (targetted memory). If this counter is greater than “MCTLSZ”, the balloon driver inflates causing more memory to be reclaimed. If “MCTLTGT” is less than “MCTLSZ”, then the balloon will deflate. This deflating process runs slowly unless the guest requests memory.

● “MCTLMAX”: the maximum amount of guest physical memory that the balloon driver can reclaim. Default is 65% of assigned memory.

You can limit the maximum balloon size by specifying the “sched.mem.maxmemctl” parameter in the .vmx file of the VM. This value must be in MB.

2) what is the limitation of Physical Memory of VMWARE / ESX Server

3) what is the limitation of Physical CPU of VMWARE / ESX server

Hardware Processors

Table 2-1 displays the number of physical processors supported per ESX Server host.

Table 2-1. Supported Processor Configurations

       

 

Maximum Sockets

Maximum Cores

Maximum Threads

Single core

With hyperthreading

16

16

32

Without hyperthreading

16

16

16

Dual core

With hyperthreading

8

16

32

Without hyperthreading

16

32

32

Virtual Processors

A total of 128 virtual processors in all virtual machines per ESX Server host

Memory

64GB of RAM per ESX Server system

4) What is the Limitation of number of VMs in VMware per ESX server: 128