The popularity of server virtualization in the data center has helped convince many organizations of the benefits of networked storage, but has also added a layer to the infrastructure that makes SAN management and performance diagnosis more difficult. As they put more production applications on virtual machines, data storage administrators are calling for more end-to-end management tools that provide visibility from storage devices...
all the way to the server virtualization layer.
"It's become a hot issue. Enough enterprise applications have been moved to virtual machines and it has become a problem," said Stephen Foskett, an independent storage consultant and author. "People didn't realize until they started putting production applications on virtual machines. Once it reached critical mass, it became more important for [end users] to have visibility. They found they needed more visibility and end-to-end management tools."
Generally, storage has become the roadblock to further virtualizing their server environments for many customers, said Jon William Toigo, CEO at Toigo Partners International LLC. Toigo said companies are reporting their server virtualization plans generally stall out at 20% because they run up against problems with I/O and mapping data to guest machines. The inability to map, diagnose and troubleshoot from the underlying storage devices up to the virtualization layer is becoming a major headache.
Detecting early SAN trouble signs
For now, administrators are focusing on the problem mostly with point products. Dan Roesler, senior systems administrator at Houston-based Boardwalk Pipeline Partners, said most of his company's servers are virtualized and running some production-level applications. Roesler said the visibility tool in VMware's vSphere 4.0 has improved but it's still lacking depth, so he's relying on NetApp's Akorri BalancePoint agentless software that collects statistics from the entire infrastructure and then analyzes the data to understand how an application workload's server, SAN and storage resources interact so pain points can be detected and addressed.
Boardwalk Pipeline Partners uses Hitachi Data Systems (HDS) storage, and Roesler said he's planning to deploy the full suite of HDS midrange arrays – the AMS 2100, 2300 and 2500. He said the Hitachi Storage Capacity Reporter Software's advanced reporting is helpful and plugs into the Hitachi Virtual Server Reporter. HDS gets both reporting tools through an OEM deal with Aptare Inc. "Hitachi now gives a view of how virtual machines are using storage resources," Roesler said.
Connecting the SAN pieces
Christopher Carlton, storage team lead and senior SAN engineer/administrator at Fort Worth, Texas-based JPS Health Network Solutions uses Virtual Instruments' SANInsight hardware and VirtualWisdom software to gain visibility into his SAN. JPS virtualized 100% of its storage and one-third of its servers on HDS's Universal Storage Platform (USP). "Through virtualization, we lost visibility into our hardware," Carlton said. "You could detect problems, but it was harder to find them. We had to look at a new technology."
Besides its HDS storage, JPS has Brocade Fibre Channel switches, Hewlett-Packard servers and Cisco networking switches. Carlton said he uses thin provisioning and tiered storage to help manage storage, and Hitachi Command management software to determine device performance. But his vendor tools leave gaps, so he uses SANInsight and VirtualWisdom to monitor inter-switch links (ISLs), VMware read/write latency, and vSphere server and memory utilization, and to measure cache and LUN performance, as well as LUN utilization.
VirtualWisdom software extrapolates information to warn administrators of potential problems and anomalies in the systems. For instance, Carlton said VirtualWisdom can show a potential issue with an ISL before it turns into a bigger problem that can take a server down. Carlton said he was able to gain significant cost savings by monitoring how memory cache was being used to determine whether he truly needed more cache. To measure LUN performance, VirtualWisdom uses an Exchange Completion Time (ECT) metric to show SAN-caused application response time latency. ECTs record transactions for each LUN-server pair and can pinpoint an application that has become a bottleneck.
VirtualWisdom's monitor is connected to a ProbeVM that gathers data from VMware's vCenter from the servers, and a ProbeV that collects SNMP data from Fibre Channel switches. On the storage side, a Traffic Access Point (TAP) Patch Panel device sits between the Fibre Channel switches and storage arrays, and connects to a SANInsight ProbeFCX that gets real-time latency data via Fibre Channel frame headers. The TAP provides an out-of-band copy of Fibre Channel traffic.
Looking for better view from the host
Redmond, Wash.-based Concur Technologies Inc. has virtualized 60% of its servers -- including email and Domain Name Servers -- and has set a goal of 80% virtualization, according to lead network engineer Peter Baughn. Concur has HDS and NetApp storage, and turned to ExtraHop Networks Inc. to help with host virtualization-to-storage visibility. Baughn said ExtraHop showed that a badly configured archiving server was using a remote partition to write large amounts of data to the array backing the virtual server cluster. That negatively affected the virtual cluster performance.
Sean Graver, storage architect at Concur, said his team also uses VMware vCenter Server Virtualization Management software to help provision, monitor and manage a virtualized data center. It offers dashboard-level visibility into the memory utilization, processor and disk capacity. The one thing it lacks, Graver said, is a view for disk performance and latency.
"We're evaluating tools to get a better view from the host side so we can see the slow response times," Graver said.
Greater visibility coming?
Storage vendors are paying more attention to visibility tools. Companies like Virtual Instruments and ExtraHop concentrate on this issue, and array vendors are taking steps to provide more visibility – NetApp's recent acquisition of Akorri is an example of that.
"A lot of people look at storage as a black box. They don't have visibility," said Lisa Crewe, senior product marketing manager at NetApp. "It's something you definitely have to consider when deploying virtualization. You have to consider the storage architecture. Is it the right platform? How is this going to be connected to my virtual environment? How will it work from a performance perspective?"
"I liken visibility into a SAN to landing a passenger jet in the fog," said Jesse Rothstein, ExtraHop Networks' CEO and co-founder. "You have to have flight instruments or you can't see anything."
Do SAN management tools help?
Make sense of SAN management software
Eleven SAN management tweaks for optimal storage area network performance