Storage is a key component in a Citrix Systems Inc. XenServer environment. The virtual machine disk image (VMDK) files reside on it, and if something goes wrong with it, the virtual machines can’t be started. Therefore, if your data center is running XenServer and you have some role in its administration, you need to understand how storage is organized.
In a XenServer environment, the physical storage devices are available from a repository, on top of which a database is created that allows the XenServer hosts to connect to the storage. If problems arise in recognizing the storage, it is often due to a mismatch in the IDs of the physical storage with the IDs in the XenServer database. But before we explain how to troubleshoot such a problem, let’s talk about the relationship between XenServer and storage.
In XenServer, storage is organized as Storage Repositories, which contain Virtual Disk Images, Physical Block Devices and Virtual Block Devices. And the virtual machine can use the storage in a few different ways: as a virtual disk file (created in the virtual hard disk, or VHD, format), a logical volume manager (LVM) or a direct connection to a SAN through Citrix StorageLink.
Drilling down a little deeper into XenServer storage, a Storage Repository is an abstraction of the physical disk device, which can be either a local device or a device on a SAN. In the XenServer Storage Repository, Virtual Disk Images are created as a storage abstraction of objects that can be presented to a VM. To do this, the Storage Repository connects to the block-based devices that are located on a local machine, a SAN or anywhere else, using XenServer’s Physical Block Devices connector object. Based on the Virtual Disk Image, storage can be presented to the VM. This storage is offered as a Virtual Block Device connector object, which is seen by the VM as its virtual disk.
As mentioned above, there are three ways for a VM to access storage. The classical way is by using VHD files. These are files stored in the Storage Repository in the standard format defined by Microsoft in 2005. Since the release of XenServer 5.5 in 2009, Citrix has also offered access via LVHD, or LVM-based Virtual Hard Disk. The benefit of this approach is that the underlying LVM layer makes it possible to apply some advanced storage management solutions, such as fast cloning and snapshots. A third option is to map a VM directly to a LUN on the storage array. This method works only if your storage array has a plug-in to support it.
A common problem that can occur on the storage is a mismatch in the identification of the storage. If this happens, access to all storage is lost. On the XenServer platform, the disk devices can be addressed in different ways by different components of the system. In XenCenter, the storage is referred to by a SCSI-ID that matches the UUID that you can see on the XenServer console. If your storage is not accessible from XenCenter, check to see whether the UUIDs that are used in XenCenter match the UUIDs as they are visible on the XenServer console in the /dev/disk/by-uuid directory.
If the storage is LVM-based, you can find the storage ID of the disk device using the pvs command on the XenServer console. The individual VMs are connected to individual logical volumes. To get an overview of these, you can use the lvs command, which again shows an ID that matches the ID that is used in XenCenter.
If there is a misconfiguration in the way that storage is handled, using the xe command on the host might be useful. This command allows you to query the host directly and see which storage devices it sees. The basic command to use is xe sr-list. This command shows the UUID currently in use, the type and all other parameters that help you identify the storage type.
Caption: You can use the xe command to find out more details about connected storage.
Using xe sr-list, you can query the storage repository to get even more information, using additional parameters. If, for instance, you use xe sr-list params=name-label,uuid,VDIs,PBDs you can find the different UUIDs that are assigned to the storage device. The goal is to find the UUIDs of the actual devices as they are seen in the Storage Repository and compare them with the UUIDs as seen in XenCenter. If there is a mismatch, you have to reimport the storage devices into the XenCenter management environment to rebuild the database.
The xe sr-list command offers advanced query options to determine the IDs of the storage devices.
And now, a real-world example of how this misidentification can occur: One IT organization I work with lost connection to all storage devices after migrating its XenServer hosts to a new data center. After analyzing the configuration, it appeared that the problems were caused by a mismatch of the actual IDs on the storage and the IDs as they were known in the database that was used by XenServer. Once this became obvious, the solution wasn’t hard: Use xe sr-rescan to rescan the IDs on the physical devices and rebuild the database.
Storage properties can be monitored from XenCenter.
Sander van Vugt is an independent trainer and consultant based in the Netherlands.