Frantic calls to and from on Saturday night lead myself and a friend to the question, “What the hell is up with iSCSI?”

It seems that the initiator works only with a service console. In fact, the observed results showed that iSCSI was being run with the Service Console’s IP address DESPITE have a vmkernel port configured in the same networks as the iSCSI target.

Well, this is not entirely true. Vmware needs a Service Console to do the initial CHAP authentication and establishment and only then does it switch over to the configured vmkernel interface.

Why even bother with an iSCSI “interface” (vmkernel port0 if you’re going to initiate with the Service Console? In fact, more to the point, why does it split the authentication and actual iSCSI traffic across them.

Based on various forum posts, it sounds like iSCSI was a last minute throw in by VMware, and just not done completely right.

Hence, Vmware Vista.

REF: my mobile phone log with the lingfish and http://communities.vmware.com/thread/209361

Specifically, in the environment I am in at the moment, there is a requirement for the operating system to be deployed to one LUN and the any application partitons to be deployed to another LUN.

Templates only allow me to deploy the machine’s virtual hard disk to one LUN and not split across two. Hence the problem.

There is no quick fix that I could find; so it was off the scripting land I went.

First things first. The customer required the deployment of twenty seven virtual machines all based off one. The process is prety simple; open my Virtual Infrastructure Client, point it at a Virtual Center (ESX servers by themselves don’t have template functionality), and convert the baseline virtual machine to a template.

I could have chosen to deploy twenty seven virtual machines manually, but, well, I’m lazy. So, the lazy man’s solution? Scripting!

The process was as follows:

1) Remove the 40GB hard drive (cloning and templating doesn’t allow hard drives to be placed in multiple datastores eg. osLUN and appLUN)
2) Create a template of the virtual machine.
3) Install powershell and the Vmware VI Toolkit giving us the powershell for Vmware functionality.
4) Save the following to a .ps1 (Powershell script file):

$array = “Machine1″,”Machine2″,”Machine3″ (and so forth)
$TARGET = “FQDN_OF_TARGET_ESX_SERVER”

Connect-viserver FQDN_OF_VIRTUAL_CENTER

foreach ($vm in $array)
{
$vm=New-VM -Name $vm -Template BASELINE_TEMPLATE -Host $TARGET -Datastore osLUN
}

The above script will connect to the VI Center (it will prompt for username and password) and then create a new machine called “Machine1″ etc. for each machine in the list. It will place the virtual machine on the esx server in $TARGET and build it from the template called BASELINE_TEMPLATE.

Now comes the hard part. There are 27 42GB drives that need to be created for the virtual machines. There is no powershell that will allow this to happen. Therefore, a script is required.

In appLUN is a vmdk called BASELINE_TEMPLATE_VMDK and this is the virtual hard drive (pre partitioned and formatted) that will be used for the virtual machines.

for DESTINATION in machine1 machine2 machine3 machine4 ETC.
do
# Copy
echo “Copying $SOURCE to $DESTINATION”
mkdir $DESTINATION
cp -a $SOURCE/* $DESTINATION/
# Rename vmdk files
cd $DESTINATION
for file in $(ls -1 *.vmdk)
do
echo “Renaming $SOURCE vmdks to $DESTINATION”
DEST=$(echo $file | sed s/$SOURCE/$DESTINATION/g)
mv $file $DEST
done
# Edit .vmdk
for file in $(ls -1 *.vmdk | grep -v flat)
do
echo “Replacing references to $SOURCE in .vmdk file $file”
sed -i s/$SOURCE/$DESTINATION/g $file
done
cd ..
done
echo “Finished”

This will copy the BASELINE_TEMPLATE_VMDK virtual hard drive one time for each virtual machine and make sure it is useable for that machine. It will also go through the .vmdk file and change the reference from BASELINE_TEMPLATE_VMDK to the destination virtual machine name.

Once it is run, we have to manually add the newly created vmdk to the corresponding virtual machine.

Then it is a simple matter of logging into each virtual machine and changing their host names and IP addresses.

There is still a fair bit of manual intervention involved, but at least the boring, tedious bits (deploying the templates and copying the hard drives) has been automated.

If you can offer any other scripts that can complete the picture, let me know!

Everyone loves Calvin and Hobbes, don’t they? If you’re not familiar with it:

Calvin and Hobbes was a syndicated comic strip written and illustrated by Bill Watterson, following the humorous antics of Calvin, an imaginative six-year old boy, and Hobbes, his energetic and sardonic stuffed tiger. The pair are named after John Calvin, a 16th-century French Reformation theologian, and Thomas Hobbes, a 17th-century English political philosopher. The strip was syndicated daily from November 18, 1985 to December 31, 1995. At its height, Calvin and Hobbes was featured in over 2,400 newspapers worldwide. As of the publication of The Complete Calvin and Hobbes, more than 30 million copies of the 17 Calvin and Hobbes books had been sold.

Set in the contemporary Midwestern United States in an unspecified suburban community, the broad themes of the strip deal with Calvin’s flights of fantasy and his friendship with Hobbes, his misadventures, his unique views on a diverse range of political and cultural issues and his relationships with the people in his life, especially his parents. The dual nature of Hobbes is also a recurring motif: Calvin sees Hobbes as a live anthropomorphic tiger, while other characters see him as a stuffed animal. Though the series does not mention specific political figures or current events, it does mention broad issues like environmentalism, public education, and the flaws of opinion polls.

That’s a LOT of comic books!

I have taken it upon myself to get all Calvin And Hobbes strips without duplicates. After only a few minutes research, here is the list I have discovered:

•The Essential Calvin and Hobbes
•The Authoritative Calvin and Hobbes
•The Indispensable Calvin and Hobbes
•Attack of the Deranged Mutant Killer Monster Snow Goons
•The Days are Just Packed
•Homicidal Psycho Jungle Cat
•There’s Treasure Everywhere
•It’s A Magical World

The other option, of course it to buy the Collected Works!

1) How often ESX checks for path failure?
==> As soon as an I/O request to a path fails, ESX will initiate a path failover. If there is no I/O outstanding to a path. ESX will probe each physical path every 5 minutes, by default, to proactively detect path failure.

2) How long will ESX wait before trying a different storage path?
==> A different storage path is tried immediately.

3) What happens between the failure detection and the connection recovery?
==> I/O requests will be queued.

4) Under which circumstances will a host initiated target reset or LUN reset occur?
==> Resets are not typically initiated by ESX. Two exceptions to this rule are:
- if a path failure occurs while there is a SCSI-2 reservation outstanding on the failed path
- if the VM or userworld that initiated the I/O request sends a request to abort the outstanding I/O request.

5) Will ESX ever force a LUN trespass in the array?
==> Only in the case of an A/P array when there is no working path on the same SP as the failing path. For example, when using an EMC Clariion (which uses the “trespass” command).

Technically point two needs some work.

ESX SCSI layer has a latency between when the failure occurs and when it will try a different path. This is usually determined by when the device driver returns a failure ie. I/O errors. Supported fibre channel devices will do this inside of thirty seconds. The ESX SCSI layer then has thirty seconds to use another working path. On an active/passive array the activate/trespass/etc command can take a few seconds to complete.

Within a virtual machine, the registry ([HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Disk]
“TimeoutValue”=dword:000000be) can be configured to wait before issuing an abort for an outstanding I/O. If this key is enabled

ESX states that within this 60 seconds it will:
- detect that a path has failed
- select a new path
- activate the new path
- re-issue the cmd from the guest to the new path
- have the newly issued cmd complete successfully and return to the guest.

This is the sort of problem that can happen, especially if you’re trying to download a file from a running Virtual Machine.

The error that appears is a:
Expected put message. Got: ERROR

The problem here is file locks. Old log files (no longer in use) will have no locks on them, so can be downloaded. Files like the vswp (Virtual Machine Swap) will be in constant use by the virtual machine and so cannot have the locks brokent to allow them to be downloaded.

The simple solution is to turn off a vritual machine before downloading the files relating to it.

All bragging aside, I have been working on a Storage Craft disaster recovery (DR) solution combining Vmware and Microsoft Distributed File System. The basic structure is this:

Storagecraft takes snapshots/incremental backups of virtual machines in the existing infrastructure and places them into a folder on it’s local storage that is then replicated across the WAN to the DR site. The DR site, also a Storagecraft server and also having a standalone ESX 3.5 server, can restore images of these machines with a fifteen minute tolerance of when the machine failed. This is all reliant on the most recent incremental backups being transmitted across the wire successfully.

Of course, as a DR site, the client is extremely interested in as much reporting as possible and this includes DFS. There is a builtin reporting function in DFS that requires manual intervention to trigger. The client requires automated reporting, as you would expect.

After trawling the internet, I found a workable solution combining a script and task scheduling in Windows 2003 R2.

As an aside, there was this handy little reference that I stumbled across as well. Nothing too surprising, but definitely worth thinking about.

The solution for the reporting was a script ripped from:

http://blogs.technet.com/filecab/articles/437214.aspx

and it looks like this:

#######
## REM This script needs to run as a daily scheduled task by someone who has
## REM local administrator rights for all machines in reported RG. This script
## REM will generate a health report for all members of the replication group,
## REM with backlog based on a specified reference member. When reports are
## REM complete they will be saved to a specified share. When ALL reports are
## REM done, mail will be sent to the specified e-mail alias with links to the
## REM share.
#######

cl_mail.exe smtp server:my_smtphost to:some_Alias@domain.com from:your_alias@domain.com subject:"DFS Replication Health Reports %DDMMYYYY%" message:@%TEMP%\healthMessageBody.txt auth:sspi


The only potential problems are:

• The requirement for a command line email utility
• Only notification of the report’s location, not the report itself
• Size over time. Each report generated takes about 60KB of disk space.
• The format of the report ie. XML/HTML

Still, it’s better than nothing!!

Sometimes when you set the DAS Isolation addresses, it fails to work, ie. when you take vswif0 offline, the virtual machines get shutdown anyway. Given this is emulating an “isolation respone”, where the service console ONLY fails to respond to the network, the virtual machines should only shutdown if the other cards in the server fails to respond.

Ideally, I want my DAS Isolation addresses in a totally seperate subnet from my service console. When I set up another Service Console, it’s always a good idea to remove HA from the cluster, wait for everything to settle down and then re-enable it on the cluster (a couple of minutes work). This makes sure both service consoles are sending heartbeats.

We can view the heartbeats via tcpdump and monitoring port 8044 by using the following commands:

# tcpdump -i vswifX port 8044
# tcpdump –i vswifY port 8044


As a final check, have a look at /var/log/vmware/aam/aam_config_util.def for the isolation addresses. It should look something like:

Start Object esx5
nodefd esx5 {
nodeAddrs = {
{
sourceType = isolation
source.addr =
destination.addr = 172.16.0.36
}
{
sourceType = isolation
source.addr =
destination.addr = 10.0.0.52
}
{
sourceType = domain
source.addr = 172.16.0.105
destination.addr = 224.0.6.127
}
{
sourceType = domain
source.addr = 10.0.0.5
destination.addr = 224.0.6.127
}
}
}

End Object esx5

Also, if das.usedefaultisolationaddress is not set to false the service console’s default gateway appears as an isolation address in the /var/log/vmware/aam/aam_config_util.def file in addition to any ip address configured with das.isolationaddress and das.isolationaddress2.

[email_link]

So, whilst vSphere is the new kid on the block, Vmware have also released an updated version to this genuinely brilliant utility!

What’s New
The VMware vCenter Converter Standalone 4.0.1 release is an update to Converter Standalone 4.0, and includes the following new features:

Support for vSphere 4.0 as source and destination targets:
• Support for configuring target disks as thin provisioned disks
• Support creation of IDE disks on vSphere 4.0
• Support for backup products to restore vSphere 4.0 virtual machines backed up using VCB
• Support for creation of virtual hardware version 7.0 virtual machines on vSphere 4.0. targets as well as migration of hardware version 7.0 virtual machines from Workstation and Server platforms to vSphere 4.0
• Support for importing OVF 1.0 single virtual machine images.
• Support for customization of Windows Server 2008 guests.

[email_link]

Has that changed in vSphere (ESX 4)?

In vSphere, the mother board remains a 400BX based board, however, it has been modified. One thing that has been added is 32 PCIe root ports, therefore I can leverage up to ten virtual network cards in a virtual machines.

Regrettably, any guest that can’t use PCIe will see the slots and address them as PCI-PCI bridges.

So, let’s all say, “Thanks KRudd!!!” Email me your pictures (krudd@invurted.com) of what you spent your KRudd money on!

Yay, KRudd!!!