«

»

Sep 28 2010

Your HP blades may be underperforming… Why?

This week we have been presented with an interesting scenario to solve. A medium sized VMware View deployment across HP Blades (BL460) and C7000 enclosures was presenting dissimilar performance on virtual desktops with exact same workload. After a little analysis we found out that some hosts were performing below the desired performance numbers. Measuring the environment was possible to identify that CPU, Memory and disk IO were not an issue.

The symptoms:

  • Virtual machine application performance on HP hardware is poor when compared to another host.
  • Applications run slower when they are located on an HP host. When they are moved to another host, performance improves.
  • CPU READY shows a high value.

After a little while one of my colleagues onsite (Thanks Simon) found out that the HP blades had a different Power Regulator setting. After changing the setting all performance issues were gone. Interesting enough this is a Default setting on HP blades.

Let’s see what Power Regulator is responsible for:

In Dynamic Power Savings mode, Power Regulator determines the amount of time each processor in the system is spending in the operating system’s idle loop. When the ratio of time spent in the idle loop to the time performing useful work is high, the algorithm instructs the processor to set its power state to Pmin (the lowest power and performance mode of the processor).

When in HP Dynamic Power Savings mode, the Power Regulator algorithm continuously monitors application and processor loading up to eight times a second and adjusts the P-state accordingly. This continuous monitoring results in optimized P-state transitions.

ESX only sees CPU cycles & MHZ presented to the hypervisor. If the power regulator decides to change the power consumption, therefore changing CPU core frequency, the hypervisor will not be aware of such changes. The Power Regulator algorithm should match processor power consumption at any point in time, but for a unknown reason it does not. (yes, firmware was up-to-date)

If the BL460 were fully loaded and CPU utilisation was high this problem would have never been noticed because the Power Regulator algorithm would never kick in to reduce the CPU core frequency. However, because the environment was under-utilised the Power Regulator was kicking in. Despite HP in their document “Power Regulator for ProLiant servers” mention that dynamic power algorithm executes on a per-core basis I can assume that the process is averaging CPU utilisation or something similar.

Luckily the guys at VMware had already published a KB1018206 for the issue.

There is an alternative option to disabling the feature. Change it to Custom and set it to OS Control Mode to have it managed by ESX 4.0 and later.

Just thought would be nice to share since this is a Default setting and many HP setups could be performing a whole lot better. Double check your settings.

8 comments

2 pings

Skip to comment form

  1. chouse

    Additionally, HP has a document “Using Integrated Lights-Out 2 in a VMware ESX environment” where they recommend setting the iLO power regulator to “OS Control” mode.

  2. @dboftlp

    & here’s the article from VMware showing how to “enable” it. There’s a note at the bottom as a cya…
    http://blogs.vmware.com/performance/2011/09/host-power-management-in-vsphere-5.html
    Am in the middle of doing a larger design on this same hardware, so thanks for pointing this out.

  3. Ed Grigson

    Andre, can you quantify the %CPU ready you were seeing in this case?

    We’re also experiencing this issue but I’m struggling to quantify the impact. The %CPU ready is only around 4% (which reduces to less than 1% after altering the power saving) which I would have thought wasn’t at a level that the users would really notice. Their feedback is that this has made a signficant improvement however. I’d expected high %CPU Ready to be more in the 10-20% bracket.

  4. Andre Leibovici

    @Ed Grigson
    This post and analisys was from October, 2010. I no longer have the system or data avaiable to provide a better analisys for you.

    Andre

  5. Ed Grigson

    @Andre Leibovici
    No problem, I figured as much but thought it was worth a try! Thanks for the article.

  1. WoodITWork.com » Your HP blades may be underperforming… Why? From http://myvirtualcloud.net

    […] Leibovici, a fellow vExpert who works for EMC as a Sr. vSpecialist has blogged today about an issue with HP Blades and a power regulation setting where the default on HP Blades can […]

  2. Is the HP power setting impacting your performance? | www.vExperienced.co.uk

    […] a great blogpost by Andre Leibovici he highlighted a default HP BIOS setting which could be impacting the performance of […]

Leave a Reply