Essential Guide

Managing storage for virtual environments: A complete guide

A comprehensive collection of articles, videos and more, hand-picked by our editors

VMware storage performance analysis

In this podcast interview, Mike Laverick discusses how storage performance analysis is handled in a VMware environment, the pros and cons of its implementation in VMware, and more.

To run either a physical server environment or virtual server environment efficiently, storage performance analysis and storage performance monitoring tools are essential to IT administrators. They’re used to track I/O resources, alert administrators of boot storms and potential performance thresholds, and possibly improve storage capacity and utilization.

In this podcast interview, VMware expert Mike Laverick explores VMware storage performance analysis. Find out about the pros and cons of implementing storage performance analysis in a VMware environment, whether storage performance monitoring tools for a physical server can be used in a virtual server environment, and what storage performance analysis tools are offered by VMware and third-party vendors.

Read the transcript or listen to the podcast on VMware storage performance analysis. Click on the corresponding podcast player to listen to Laverick's answers to each question.

SearchVirtualStorage.com: How is VMware storage performance analysis handled?

Play now:
Download for later:
  • Internet Explorer: Right Click > Save Target As
  • Firefox: Right Click > Save Link As

Laverick: Well, the first thing to remember is that … the I/O is being created … inside the virtual machine. Therefore … the analysis and the data collection [have] to start … with the ESX host—the hypervisor, if you like—because that’s where the I/O is being generated. But it doesn’t stay on the ESX host. It then reports its data to Virtual Center, assuming that the customer has that. And Virtual Center then starts to manipulate that data in SQL over a period of time. So it will take the data that is recorded almost on a second-by-second basis but roll it up into five minutes or an hour, daily, weekly, monthly, even yearly [view]. So you’ve got a view of your performance data over a longer period of time.

The important thing is if you want that information, [the] SQL agent, on the SQL server, which is the back-end database to Virtual Center, must be running; otherwise, it can’t run the job that would then roll up the statistics. And generally, the information that you get is very detailed. The only thing I would say is that historically there’s always been a kind of lack of data around NFS and the performance that we get from NFS data stores. And to a degree that persists, even to this day, even though NFS has become increasingly popular.

SearchVirtualStorage.com: What are the advantages of implementing storage performance analysis in a VMware environment?

Play now:
Download for later:
  • Internet Explorer: Right Click > Save Target As
  • Firefox: Right Click > Save Link As

Laverick: I think you’ve got to remember that the resource that most people run out of first is memory. But I don’t think we can underestimate the impact of storage in a virtualized environment. [That’s] especially [true] if you’ve got very read- or write-intensive virtual machines like SQL or Exchange, or if you experience random I/O issues or even peak activity … or if you work in the retail sector, spikes in demand caused by Black Friday, Thanksgiving, Christmas and that kind of thing. On a more specific level, if you’re doing any VDI work, you can get boot storms once the virtual desktops power up. And also if you’re running virus analysis software inside a VM, that can be the cause of this activity.

The bottom line is if you’re not monitoring, you don’t know what resource, where, is actually creating a problem. But I think just generally [there’s an advantage in] validating your best practices and making sure those are being followed, because in my experience most problems are caused by people not following the best practices.


SearchVirtualStorage.com: Can storage performance monitoring tools for a physical server be used in a virtual server environment?

Play now:
Download for later:
  • Internet Explorer: Right Click > Save Target As
  • Firefox: Right Click > Save Link As

Laverick: I would never really use tools inside a virtual machine if I was monitoring CPU or memory. But there is more temptation, I think, to actually look at disk activity in a virtual machine using the guest operating system tools. After all, a write to a disk is a write to a disk; whatever message it gets to the physical storage, you can’t avoid the physical world. But what makes it different is [that] the VM [just] sees a SCSI adapter; it has no idea if you’ve got Fibre Channel HBAs or whether you’ve got 10 [Gigabit Ethernet] storage. It just sees a virtual SCSI adapter. And any read/write [activity] that takes place inside the virtual machine gets translated [down] through some sort of storage driver or controller on the host.

Now if that’s NFS or iSCSI, it’s going to be … an Ethernet driver. Obviously if it’s Fibre Channel, it’ll be a driver for Emulex or QLogic. So although most in-guest tools will give you reliable information, you have to treat them with great caution, because the big thing is that one virtual machine is totally unaware of all the other virtual machines that are on the same physical host. And all those virtual machines together are sharing the same connection to the storage back end, whether that’s Ethernet or whether that’s Fibre Channel. So if you really want to know what’s going on it has to be … done outside the scope of the virtual machine, outside of the guest operating system, using either tools from the hypervisor vendor, whether that’s VMware or Microsoft. And really what you’re looking for when you’re doing that kind of analysis outside of the guest is you’re looking for contention. You’re looking for volumes or LUNs that are oversubscribed, or you have too many VMs on a single LUN or a volume and they’re just generating far too much I/O. Or [in] most cases, you’re looking at the array; has the LUN that’s being presented to the host got insufficient spindles backing it to deal with the I/O, or has the wrong RAID level been used for that particular profile of storage?

I think the key thing to get across from doing any analysis of storage is that you’re going to probably need a bit of information from both environments—from your virtualization environment and probably from your array. And, in most cases people find that the problem isn’t with the virtual machine but with the storage array. It’s a classic example of mistaking the symptom—poor performance in the virtual machine—for the cause—which is often something external, something to do with the way the array is being configured. Perhaps the most common mistake that people make is when they’re going to create a new virtual machine they sort the data stores that they’ve got available by free space, and they just plunk their VMs on the data stores that have the most free space. What I always suggest to people is sort your data stores by their I/O capabilities—fast or slow storage, is it SATA or is it SAS or even [SSD] storage—and use that as your guide for where you put your virtual machines, not just which ones have got the most free space on.

SearchVirtualStorage.com: What tools does VMware offer to properly implement storage performance analysis?

Play now:
Download for later:
  • Internet Explorer: Right Click > Save Target As
  • Firefox: Right Click > Save Link As

Laverick: There [are] lots of different tools. At the physical ESX host there’s something called ESXTop, which if you’re from a Linux background, it’s kind of VMware’s version of Top. And it can show not just CPU activity, but also memory, disk and network activity. So from the disk view of ESXTop, you can see read/writes per second, and any latency you might have, that’s significant for networking—Ethernet-based storage—and any excessive queue depth you’ve got. One of the things I’ve often found with ESXTop is that it does generate an awful lot of data very rapidly. So I’ve often used a free tool from VMware … called ESXplot, which can take a .csv file and then import that and give you a better way of monitoring and viewing the data. If you’re looking for tools to control the storage, since, I think, vSphere 4.1, Storage I/O Control has been possible. So it’s possible to kind of control on a virtual machine basis, VM by VM, which ones have what access to the storage layer. And as I said before, there’s Virtual Center . If you’re looking to spend more money on top of what you might already have, VMware [has] the vCenter Operations Manager, and that will produce some very interesting statistics and give a thumbnail sketch of what your environment’s currently like, and it learns over a period of time what your expected behavior [is]. And you can have [a dashboard], so if something unusual happens, it’s [flagged] straightaway.


SearchVirtualStorage.com: What third-party vendors are tackling VMware storage performance analysis in a VMware environment, and what tools do they offer?

Play now:
Download for later:
  • Internet Explorer: Right Click > Save Target As
  • Firefox: Right Click > Save Link As

Laverick: That’s a very good question. I remember when I was at [a VMUG meeting] in San Diego a couple of months ago. Somebody was explaining that they had a storage problem, and another guy around the table said immediately, “You need to look at these products.” And I took a step back from that and said, “Before you start looking at third-party vendors and start parting with your hard-earned cash, use what tools you have available,” because a lot of the time I find customers are a little bit too willing to go out and spend money on a third-party solution when they haven’t really exhausted all of the “free” tools they have already. But putting that caveat to one side, when I was in Boston recently at the Tech Field Day organized by Stephen Foskett, I was very impressed by Akorri. [Akorri’s tools] can actually track the full path of the virtual machines through the connections into the array, even into different controllers and to the volumes. You see a full window of where the IOPS are being generated, which I hadn’t seen before. I think vKernel and vMonitor have quite good tools. And I guess there’s always the possibility of taking the spindle out of the equation. In the last 18 months to two years, we’ve seen increasing use of cache or SSD at the front of an array to absorb the IOPS requests of things that are very disk-intensive. And if we can take the spindle out of the equation, which is the most constraining physical resource, there’s no reason why a virtual machine would be affected by storage performance problems, but I guess that’s very much dependent on what generation of array you’re on and whether you’re prepared to spend that money to get those kinds of performance levels.

This was first published in October 2011

Pro+

Features

Enjoy the benefits of Pro+ membership, learn more and join.

Essential Guide

Managing storage for virtual environments: A complete guide

0 comments

Oldest 

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:

SearchSolidStateStorage

SearchCloudStorage

SearchDisasterRecovery

SearchDataBackup

SearchStorage

SearchServerVirtualization

SearchVirtualDesktop

Close