Sam.McGeown.co.uk

I rebuilt an ESX host in my HA/DRS cluster today, following my build procedure to configure as per VMware best practices and internal guidelines. When the host was fully configured and up-to-date, I added it to the cluster and enabled HA and DRS. Then I went to generate some DRS recommendations to balance the load an ease off my overstretched host, but no recommendations were made.

I couldn’t manually migrate any VMs either – it was odd, because both hosts were added into the cluster, and could ping and vmkping each other from the console.

I also received email alerts -

[VMware vCenter - Alarm Host error] Error detected on [HOST] in [Data Center]: Agent can't send heartbeats.msg size: 1266, sendto() returned: Operation not permitted

It turns out that there were slight naming differences between the default VMKernels on each host, which stops communication. Since one VMKernel was named “VMKernel” and the other “VMKernel 2” it stops the migrations, and hence DRS. The hosts would add into the cluster OK, DRS actually showed as “imbalanced” on the Cluster summary screen - it was just DRS and vMotion which wouldn’t work.

With the VMKernels renamed to exactly the same thing, DRS kicked off no problem, as did a manual migration.

So the moral of the story is this; name ALL networks in the same cluster identically. It makes sense when you think that the VM needs to see it’s Virtual Machine Network on each host – why should the Service Console and VMKernel be any different?

VMware ESX 3.5, VMWare, VMware DRS/HA vmware, DRS, HA, cluster, vmotion, vmkernel, heartbeats.msg

So you’ve upgraded your ESX 3.x servers to 4.0 and you’ve upgraded your vCenter server, now you want to access the shiny new hot-add feature to upgrade some running server’s memory. Except you can’t, the feature is no-where to be seen. Something to bear in mind though, your OS needs to support hot-add, so you’ll need a Windows Enterprise or Datacenter edition.

Here’s how to enable it:

Upgrade the VM’s tools, if you haven’t already. This requires a re-boot, but don’t reboot, shut the server down down, otherwise you’ll require another reboot to add the feature. The tools can be upgraded from the system tray icon.

 

 

Once the VM is shut down, upgrade the Virtual Hardware (again, assuming you haven’t already!)

 

 

 

 

 

 

 

Once your virtual machine is fully up to date with VMTools and Hardware, you can edit the Advanced options (again, while the machine is shut down) and enable the feature.

 

 

 

 

 

 

 

 

Now with the server booted, you can add Memory and CPU without the need for a reboot. In my tests changing the Memory from 1GB to 2GB caused a single dropped ping, and changing from 1 vCPU to 2 vCPUs caused a ping to drop.

VMware vSphere 4.0, VMware vCenter 4.0, VMWare, VMware ESX 4.0 vmware, esx 4.0, vsphere 4, hot add, memory, cpu, vcpu

Configuring the Virtual Environment and Virtual Machines

 

Note – this configuration will work for ESXi 4 upwards due to the server 2008 MSCS requirement for persistent SCSI-3 reservations.

The first step is to create a new vSwitch for the host-only cluster heartbeat network, don’t assign any network adaptors to the switch as it’s going to be local only.

Create a new virtual machine with a single hard disk. For the purposes of this test, I’ve assigned 2 vProcessors, 1GB RAM, 30GB drive for the OS, 1 vNIC in the default vSwitch0.

Add a second vNIC and assign it to the cluster network vSwitch created in step 1.

Install Windows Server 2008 R2 Enterprise and all the Windows Updates, for the example I’ve named it SQLCluster01.

Clone the server and rename the new one to SQLCluster02. In ESXi you can’t clone, so shut down the first server, copy the files to a new folder and right click the VMX file to add it to the inventory. When you boot it the first time VMware will ask if it’s been moved or copied – select copied.

Create a disk for use as the Quorom, this needs to be shared and since I’m using ESXi with local storage only it must be “eagerzeroedthick”. To do this I have to use the unsupported mode in ESXi (Alt+F1, type unsupported and then your root password) and use the vmkfstools command to create it (vmkfstools –c <size> –d eagerzeroedthick –a lsilogic /vmfs/volumes/<datastore>/<folder>/<disk>.vmdk)

Add the new disk to SQLCluster01 using a new SCSI virtual controller (different from the current controller, e.g. my first HD is on SCSI 0:1, the Quorum is on SCSI 1:0)

Check that the new SCSI controller is set to LSILogic (it is for Server 2008 by default) and set the SCSI Bus Sharing to Virtual.

Add the Quorum disk to the second virtual machine, using the same settings.

Edit the .vmx file for both servers, adding in the following lines (edit for your SCSI controller):

scsi1:0.mode = "independent-persistent"
scsi1:0.shared = "TRUE"

Create a disk for some shared storage for the cluster too, it will be needed for the DTC application as well as the SQL server – in a production environment you may want to separate logs and data, but for my test, I’m just adding another two 10GB disks. Use the same process as for creating the Quorum disk.

[more]

Configuring SQLCluster01 and SQLCluster02 for Microsoft Cluster Services

Once the two VMs are booted, we need to do a number of things:

Activate and format the Quorum disk. In SQLCluster01, open the disk management snap-in and activate the new Quorum disk, create a new partition and format it to NTFS. I mounted it at Q. In SQLCluster02, open the snap-in and activate the Quorum disk – it should pick up the partition and formatting automatically. I re-assigned the disk letter to Q.

Configure the Access Network. I’ve assigned a static IP within the servers range, 192.168.8.0/24 for both servers. They have full network connectivity.

Configure the Heartbeat Network. I’ve assigned the IPs 10.0.0.1 and 10.0.0.2 on a /29 subnet to the Cluster Heartbeat Network Adaptors, and renamed it to Cluster Heartbeat just to keep it tidy.

Join the domain. You can no longer use a workgroup for MSCS, so join the VMs to your domain.

Create DNS records for the Cluster, Virtual-SQL and MSDTC. An A record is required for the cluster itself, the virtual SQL instance and the Distributed Transaction Coordinator. I created vm-cluster, vm-virtualsql and vm-msdtc each with their own IPs.

Install Failover Clustering on both nodes. Launch the “Add Features” wizard and tick to install the Failover Clustering feature for each server (the server may require a restart).

Configure the Cluster. Open the Failover Cluster Management console from Administrative Tools and run the “Validate a Configuration” wizard. Add both servers into the wizard and run all the defaults.

Create the cluster. Run the “Create a Cluster…” wizard, select the servers as you did with the validation wizard, enter the cluster name and IP address that you created earlier (vm-cluster for me). Review the summary and create the cluster. The cluster service should automatically select the best disk for the quorum.

Installing the Distributed Transaction Coordinator on the MSCS

Open the Failover Cluster Management console and right click Services and Applications and select “Configure a service or application”. Select the DTC and click next.

Enter the DNS name you configured earlier for the DTC and the IP address you assigned and click next.

Select the shared storage you’ve assigned to the cluster, and click next.

Complete the wizard and then verify that the DTC is running by selecting it in the Services and Applications window – the Server Name, MSDTC and Drive should all be online.

Installing SQL Server 2008 on a Windows Server 2008 cluster

Assuming you updated the server earlier, you should already have the .Net Framework 3.5 SP1 which is a pre-requisite for SQL Server 2008. I’ll move on to the installation at this point, so open the installed and select a “New SQL Server failover cluster installation”.

Click through the wizard, enter key and license info and install the Setup Support Files. Check and resolve any issues in the setup support rules.

Select the features that you require

Configure the SQL Server Network Name as you created earlier in the DNS records. I’m installing the default instance of SQL.

Check the disk space requirements are OK and click next. Click next to create a new Cluster Resource Group for SQL.

Select the available cluster disk and click next.

Configure the network address you configured earlier for the SQL Cluster

In the Cluster Security Policy dialog box, accept the default value of Use service SIDs (Microsoft’s recommended option). Configure your service accounts and collation, DB Engine Authentication and Data Directories. Again, for production you wouldn’t use the same disk for Logs, Data, TempDB etc, but this is just a test setup.

After that, click through to your installation and complete.

Verify all the resources required for the cluster application are available in the Failover Cluster Management console.

Once that’s all set up, it’s time to install the second cluster node. Run the SQL 2008 installer and select the “Add node to a SQL Server failover cluster” option. As before, go through the setup support rules, check the Cluster Node Configuration, configure the Service Accounts, step through the verifications and install the node.

If it all completed successfully, you now have a working Windows Server 2008 Cluster with an active/passive SQL 2008 database cluster, running on top of a single ESXi 4 Server. I tested connecting with SQL Manager to vm-virtualsql which works as expected. I also tested failing over between hosts with the Cluster console, again it worked as expected.

Thanks for reading and I hope this helps!

Windows Server 2008, Microsoft, SQL Server 2008, Microsoft Cluster Services sql, sql server 2008, windows server 2008, cluster, mscs, microsoft cluster services, active/passive, vmware, esxi 4.0

I'm currently in the process of migrating a 2-host High Availability cluster of ESX 3.5u4 servers to vSphere 4. This is going to come in 3 distinct stages: Stage 1 is to upgrade VirtualCenter Server 2.5 to vCenter 4, which I am going to cover today. Stage 2 is to upgrade each host, and will be covered as I do it. Stage 3 is the upgrade of the Virtual Machines to the latest VMware Tools and then the new VM hardware.

So to start, I'll outline the process:

• Download the vSphere vCenter 4 installer from VMware (~1.8GB).
• Download your updated licensing for vSphere.
• Back up your VirtualCenter server.
• Run the installation.

I'm not going to run through the download of the installer or licensing, if you're not sure how to do that, probably best not to do the rest.

Backing up VirtualCenter Server

My VirtualCenter server is installed on a Virtual Machine, so this makes things a lot simpler – I'll just take a snapshot to start. Being a belt-and-braces kind of situation (live HA cluster), I'm also going to do the database and configuration backup too.

Databases - I'm using SQL Server 2005 express which is supported for vSphere vCenter, so there will be no database upgrades, however the schema will be changed. First off, I've connected to SQL with SQL Management Studio and run a full backup. As I have VMware Update Manager installed too, I'm backing up that database as well.

Configuration file – Make a copy of your vpxd.cfg file, which is stored in the C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter folder.

SSL Certificates – In the same folder as the vpxd.cfg file there's a folder called SSL, which you'll want to backup too.

If you're not using integrated authentication for the database access, you need to ensure you have the user name and password for the DB access.

Once all that's gathered together and safely backed up, you can move on to the installation.

Installing vSphere vCenter

Open services.msc and stop the VMware VirtualCenter Server service.

Insert your vCenter installation CD, the installer pops up:

Click vCenter Server.

Notice it's detected the earlier version of vCenter server and is going to upgrade.

  Enter license details.

Enter DB user details, or leave blank if you're using integrated authentication like me.

If you have any plug-ins installed (e.g. VirtualCenter Update Manager, or Converter) it will let you know that they need to be up to date too.

Select to upgrade the vCenter Server database, and tick that you've backed up the database and SSL folders.

  Select the account that you want to use to run the vCenter Server service.

Configure some ports, I've left them as defaults.

  Finally, install. It will run a DB upgrade, and various other uninstall/upgrades.

At this point I sat and waited…and waited…and waited. SQL server was chewing 70-80% processor, it was progressing, just slowly.

Eventually, it finished and the server settled down. I ran through the upgrade of Update Manager and Converter Enterprise, all click and go.

Stage 1 complete!

VMWare, VMware Converter, VMware Update Manager, VMware vCenter 4.0, VMware vSphere 4.0 vmware, vsphere, vcenter, upgrade, virtualcenter, vsphere 4, vi3

I'm migrating some hosts off of an older storage LUN, but when I drag the disk to the new Datastore with the SVMotion plug-in the job fails with the following error:

The error occurs because the virtual disk cannot be moved without moving the source files, the .vmx, .vswap etc. Simply drag the entire VM, rather than the virtual disk to the new Datastore.

If you're trying to move a 2nd, 3rd or nth disk and you get this error, drag the entire VM as per above over to the new Datastore, once that's completed, go back in to SVMotion and drag the whole VM across again, only this time before you apply, drag the nth disk back to the new Datastore.

VMWare, VMware ESX 3.5, VMware Storage vMotion svmotion, vmware, storage vmotion