There's been considerable buzz about iSCSI storage-area networks (SANs) for virtual server environments, but Fibre Channel (FC) remains the technology of choice for virtual server environments.
Table of contents
Getting ready for 10 GbE
Spreading out storage provisioning duties
The downside of Fibre Channel
When IT departments began virtualizing servers, many followed the path of least resistance with storage. Small companies initially opted for direct-attached storage (DAS) as they weighed the merits of shared storage. Midsized to larger organizations often turned to their existing FC arrays.
The Apartment Investment and Management Company (AIMCO) in Denver was no different. The real estate investment trust's foray into server virtualization consisted of a two-processor license of VMware Inc. ESX Server with DAS plus its existing Fibre Channel SAN. When AIMCO added eight ESX servers last year, it stuck with its FC Hewlett-Packard (HP) Co. StorageWorks 6100 Enterprise Virtual Array (EVA).
"We already had it. There was no sense in buying anything new. All we had to do was add disks," said Chris Bell, a systems administrator in AIMCO's Greenville, S.C., data center.
But Bell is now intrigued with10 Gigabit Ethernet (GbE) and iSCSI SANs, as AIMCO considers a possible expansion of its virtual server environment. He expressed particular interest in NetApp's storage virtualization and multiprotocol storage devices that support iSCSI out of the box as opposed to the EVA 6100, which AIMCO hasn't licensed for iSCSI.
The NetApp option holds appeal because "it would eventually allow us to eliminate the FC SAN fabric switches and transition the disk-based traffic to ISCSI network traffic," Bell said. "It's all about lowering costs."
Reduced cost is an important consideration when purchasing iSCSI because a company can use its existing IP infrastructure instead of buying expensive FC gear. The software initiators that send the SCSI commands over the IP network are built into all of the major operating systems. Still, IT shops need to do a careful assessment of their virtual environment to make sure that cost savings match their expectations.
"As you add virtual machines onto a physical server, you're increasing the amount of I/O traffic across the physical links," said Stanley Zaffos, a research vice president at Stamford, Conn.-based Gartner Inc. Overhead will rise proportionately from a software initiator perspective.
"At some point you may discover that while something works from a proof-of-concept [perspective], it didn't scale as anticipated," Zaffos said.
He said that might necessitate the purchase of higher performance network interface cards or host adapters that could make the cost approach the level of Fibre Channel.
"For performance, you're going to pay a price. When you pay that price, you may lose the value proposition edge to FC" depending on whether the comparison is to entry-level or high-cost FC, said Greg Schulz, founder and senior analyst at StorageIO Group in Stillwater, Minn.
Whether application workloads will be virtualized on any given physical server, as well as their related bandwidth requirements are other factors to consider. AIMCO's Bell, for instance, said he would only consider 10 GbE.
While the cost of 10 GbE cards is dropping, many organizations may find 1 GbE adequate, especially if they're virtualizing lighter application workloads.
Saddle Creek Corp., for instance, hasn't done anything special to pool multiple GbE connections to boost throughput. Kathy Fulton, senior manager of technical services at the third-party logistics provider, said the company hasn't run into any performance challenges, even with its high-volume warehouse management system.
"At some point we'll look at [10 GbE] because anytime you can do something to improve performance, it's worth investigating," she said.
The Lakeland, Fla.-based company consolidated 70 to 80 physical servers to approximately 10 using VMware, and all applications are now virtualized, including Microsoft Corp.'s Exchange Server, which has more than 600 mailboxes. Saddle Creek shifted from direct-attached storage to an iSCSI SAN from LeftHand Networks Inc. (now owned by Hewlett-Packard) in conjunction with its server virtualization initiative.
Fulton knew that DAS was no longer an option for taking advantage of advanced VMware features such as high availability and VMotion, which allows users to move virtual machines from one server to another. The decision came down to iSCSI vs. FC SAN, Fulton said, but price, flexibility and ease of administration tipped the scales in favor of iSCSI.
"The amount of time my engineers would have to spend administering the SAN seemed to be significantly less from the iSCSI perspective," Fulton said. "They really liked the interface because everything was presented logically."
A server team rather than storage specialists can do much of the storage provisioning for iSCSI, noted Jeff Boles, senior analyst and director, validation services at Taneja Group in Hopkinton, Mass. Many users tend to entertain iSCSI for new projects or when their Fibre Channel storage systems offer the iSCSI option, he added.
In a December 2007 survey conducted by Milford, Mass.-based Enterprise Strategy Group (ESG), 86% of the 365 IT organizations polled used networked storage with their virtual server environments. ESG also found that most respondent firms didn't purchase new storage resources to support server virtualization.
Bob Laliberte, an analyst at ESG, said the biggest area of growth is iSCSI, although he acknowledged that could be a function of the many small- to medium-sized businesses (SMBs) that are taking advantage of server virtualization. SMBs tend to like iSCSI because it's a technology they're familiar with, he added.
Yet Fibre Channel is still the most pervasive storage technology in virtual server environments. For example, Storage magazine's Spring 2009 Purchasing Intentions Survey shows that of the 470 respondents who virtualize servers, 51% use FC SANs as their main type of storage. This is followed by iSCSI SANs (12%), direct-attached storage (10%) and network-attached storage (9%).
VMware didn't natively support iSCSI or NAS storage until the release of ESX 3.0 in June 2006. To use iSCSI and NAS storage before then, a user had to connect it from within the virtual machine, rather than directly connecting it to the ESX Server, according to Jon Bock, senior product marketing manager for business continuity solutions at VMware.
Bock acknowledged that VMware first introduced features such as Storage VMotion and VMware Consolidated Backup for Fibre Channel because of its larger installed base. But he noted that the company now aims, where technically possible, to deliver the same feature set regardless of what type of shared storage a customer might use.
The main knocks against Fibre Channel for virtualized environments are the same as they are for non-virtual environments -- cost and complexity. But performance, reliability and high availability for mission-critical applications have stopped many enterprises from switching to Ethernet-based iSCSI SANs, especially if they already have significant investments in FC.
An Ethernet-based network carries the overhead of TCP/IP to provide for retries, acknowledgements and flow control in delivering information. But even then there's no guarantee a packet won't be dropped. Converged Enhanced Ethernet (CEE) aims to address the shortcomings, but it remains a work in progress.
When CEE-supported products mature, Fibre Channel over Ethernet (FCoE) may become another viable option for virtual server environments. For companies in need of high bandwidth, 10 GbE would mark a step up over the current 8 Gbps FC.
Until then, large corporations tend to favor what they know. RiskMetrics Group Inc., for instance, has years of experience with Fibre Channel and knows how to deal with potential problems. The New York City-based financial services firm has had only four failures -- one involving a host bus adapter (HBA) and three concerning Fibre Channel cables -- during a span of eight years and none were critical, according to Ed Delgado, the company's storage architect.
"That really gave us the confidence that it was the right choice" for the virtual server environment, Delgado wrote in an email.
An additional factor was the company's jump to 4 Gbps Fibre Channel. RiskMetrics has 30 VMware ESX Servers spread across six locations and each one typically runs 10 to 15 virtual machines.
"The biggest issue we had with iSCSI had to do with relying on the network infrastructure, which already is heavily used by our servers for general communication and data transfer," Delgado wrote. "Fibre gave us a dedicated environment that could easily be diagnosed and escalated to the vendor if need be."
Although Delgado views iSCSI as "too much of a wild card for us to run a full VMware implementation," he said the company may try it at one of its smaller branch offices. Three of RiskMetrics' new disk arrays support iSCSI, although the storage team hasn't used it.
"We have been back and forth on iSCSI in our environment period, not just for virtual servers," Delgado wrote. "While it would be nice to leverage the existing Cisco [Systems Inc.] network equipment, we would experience a drop in bandwidth while adding some overhead for the network team of having to create and management multiple separate VLANs [virtual LANs] dedicated to storage."
Dan Iacono, a senior SAN systems engineer in HP's multivendor systems engineering group, considers Fibre Channel to be preferable over iSCSI for virtualized server environments simply for the overhead associated with iSCSI and NAS.
"The storage-to-IP translation uses CPU cycles of the server," he said. "What's always made FC more attractive is the encapsulation of storage protocols on an application-specific integrated circuit [ASIC]. There's no overhead that's put on the CPU."
But for those cases where a user wants to present storage from one virtual machine to another virtual machine, iSCSI and NAS are better, Iacono said.
NAS is infrequently used in virtual environments, although it may hold sway with IT organizations that favor file-based storage and ease of management. With increased network speeds, performance is no longer the issue it once was for higher-end applications.
VMware's Bock said he finds that most companies make their decisions based on "which is better given what [they're] doing today" and "does this storage platform implement the features they need." He sees no inherent technical advantages of one approach over another.
"Is a red car better or a blue car better?" Bock asked. "There are good red cars and bad red cars, and good blue cars and bad blue cars."