SAN FRANCISCO — Storage administrators at VMworld 2009 this week say maintaining enterprise data storage efficiency and manageability while scaling growing VMware server virtualization environments remains a delicate task.
Looking for help, administrators turned up at breakout sessions where VMware Inc. officials and data storage consultants offered tips for managing storage in virtualized environments.
"The biggest challenge in our environment remains picking the right VMFS [Virtual Machine File System] store size" as new virtual machines (VMs) are deployed, said Jules Thomas, system engineer at Pitt Ohio Express in Pittsburgh.
Pitt Ohio Express has approximately 25 TB capacity on a Hewlett-Packard (HP) Co. StorageWorks 8100 enterprise virtual array (EVA), supporting mostly Windows servers. Thomas said the server environment is currently about 70% virtualized with a goal to get that number closer to 100%.
"Even if it's one VM on a box, we want all servers in virtual encapsulation for easier backup, recovery and DR," Thomas said.
The company uses Data Domain Inc. DD565 data deduplication arrays to retain data on disk, and Thomas said server virtualization and disk-based backup have made for easier recoveries by eliminating the need to configure an operating system and patches when restoring a system.
For high-priority applications like SQL Server, however, "it's difficult to figure out how to size your storage going from physical to virtual," Thomas said.
The standard template for VMs in his environment includes a 250 GB data store, but Thomas said he's unsure if that will offer the space needed for snapshots of an active database. Performance is also an issue for a tier one database app, and Thomas said he's considering using solid-state drives (SSDs) to support a virtualized SQL rather than the HP storage-area network (SAN).
Other customers are still moving VMware server virtualization from test and development to production because of difficulty coordinating among different segments of a large IT environment.
"We have VMs running in production, but it's a minority," said Keenan Greer, storage administrator at Memphis-based FedEx, which has five storage administrators to manage 6 PB of data storage on EMC Corp. gear spread among three data centers.
FedEx is looking to consolidate to two data centers and -- like most IT organizations in the current economic downturn -- improve storage efficiency.
"The only way we're going to reduce our workload is to virtualize," Greer said. "[But ] we're broken up into subgroups for operating systems, databases, storage and server administrators—it's difficult trying to get everyone to see the big picture at the same time." Stringent performance requirements amplify the logistical challenges. "We have to be sure it stays the same," he said. "A slowdown is the one thing that's not acceptable."
Administrators in smaller environments find themselves with a choice between simplified management and optimizing VMware performance according to best practices. "We have chosen to keep storage provisioning simple, but we sacrifice some performance to ease of management," said a systems administrator from a West Coast digital printing firm who asked that he not be identified because of company policy preventing him from representing it in the press.
The company, which has approximately 50 TB running on Hitachi Data Systems Adaptable Modular Storage (AMS) arrays, puts virtual machines into single large logical unit numbers (LUNs) encompassing each disk shelf of the Hitachi array.
"Over time, things balance themselves," he said. "Our applications are more CPU than I/O bound, and we don't spend time worrying about each particular LUN."
The administrator added that he attended a breakout session to find out VMware's recommendations for storage management to prepare for the future. "If my workloads change, what am I up against?" he asked.
VMware engineers warn of I/O contention pitfalls
A group of VMware engineers led by Lucas Nguyen, technical alliance manager for storage partners, said users in more performance-constrained environments shouldn't let too many virtual machines share a LUN because of contention created by VMware's VMFS file locking mechanisms.
When operations are conducted in a VMFS environment that require an update to the metadata of the VMDK file, the clustered file system uses SCSI reservations to protect metadata from corruption, according to Bob Slovick, VMware's senior information systems specialist. This reservation gives the VM momentary exclusive access to the file system. VMFS 3.5 added a new file locking scheme to reduce the frequency of that contention, but it can still happen when too many virtual machines are stacked together in a LUN.
"Users should also schedule virtual machine reboots so only one LUN is impacted at any given time. Power on and power off are separate operations, and both will create reservations," Slovick added. The impact of reservations on the environment can be significant enough that Slovick suggested users restrict permissions for any operation that will trigger a reservation to "administrators who understand their effects."
Saving storage space with snapshots: Pros and cons
Users looking to balance storage efficiency with server virtualization best practices were offered further suggestions in a later session on Tuesday by Forsythe storage architect and consultant Brian Peterson titled "Controlling the Storage Impact of Virtual Server Sprawl." Peterson suggested that admins looking to store virtual machine images efficiently create a "golden copy" of an operating system, and then use storage arrays' space-efficient snapshots to make new virtual machines without using twice the space.
However, Peterson acknowledged this technique also has its tradeoffs, depending on the storage array vendor used. For VMware to use the snapshot correctly, it would have to be created using Raw Device Mapping (RDM), he said. This method should be used for operating systems only, and not application data.
Peterson acknowledged that a patch to the golden copy won't automatically proliferate among snapshots. If a patch has to be deployed across all images, users could choose to make each snapshot copy bigger to accommodate the size of the patch, or redeploy the snapshot-based virtual machines. "It's not pretty, but it can be done," Peterson said.
Click here for more information on VMworld 2009 from TechTarget.