What is NVMesh?
NVMesh is a new generation of software defined storage, specifically designed to make the high IOPS and low latency of NVMe drives available as shared storage across the network. It essentially turns the network into your RAID controller by enabling you to create resilient logical RAID volumes across multiple commodity servers, which are then available for shared access anywhere on the network.
In the old days, you would have bought expensive SAN hardware, probably based on FibreChannel, to implement similar shared storage at small scale and at a performance level that is not appropriate for NVMe drives. Nowadays, the smart software-defined approach of NVMesh gives you much more flexibility and all-flash performance anywhere on your network.
NVMesh can be used as the ideal basis for a shared file system like IBM Spectrum Scale, BeeGFS, NFS or a local file system like xfs or ext4. Alternatively, it can be the basis to vastly speed up databases like Oracle RAC. Also, it can be the basis for a very dynamic environment like your Kubernetes (K8s) cloud.
Who should use NVMesh?
Everyone with an interest in fast, flexible and easy to manage NVMe storage. NVMesh users typically have high throughput, high IOPS or low latency demands for their applications, which classic approaches to shared storage fail to meet by an order of magnitude. This is often the case in Artificial Intelligence / Deep Learning, HPC, finance, life sciences, media & entertainment and the oil & gas industry.
Other drivers for the NVMesh user community are typically cloud or cloud-like environments, where storage volumes need to be dynamically created/deleted (e.g. through Kubernetes), need to be protected from server failures or need to be accessible anywhere on the network.
Do I need NVMe drives to use NVMesh?
NVMesh was specifically designed to deliver the high IOPS and low latency of NVMe drives and thus only manages NVMe drives. With the recent price drops for NVMe drives, the motivation to even consider other storage options has become very low. However, our customers with a need for different types of storage media either run NVMesh side-by-side to their classic spinning disk storage or run file systems like IBM SpectrumScale or BeeGFS on top, which add tiering for different types of storage media.
If you don’t have NVMe drives and want to try NVMesh, you might want to look at the bare metal NVMe storage offerings of Oracle Cloud Infrastructure (OCI), which provide direct access to NVMe drives and RDMA interconnects.
If you are interested in professional consulting for your infrastructure, firstname.lastname@example.org will be glad to assist you.
Do I need an NVMe RAID controller for NVMesh?
No, you will use NVMesh to avoid a RAID controller. A RAID controller is no match for the speed of NVMe drives and limits your flexibility. NVMesh was designed as a software-defined replacement for the classic concept of RAID controllers and enables you to stripe and protect logical volumes across multiple servers and access them anywhere on the network.
How is NVMesh different from a RAID controller?
A RAID controller is no match for the high IOPS and low latency of NVMe multiple drives and limits your flexibility. NVMesh was designed as a software-defined replacement for the classic concept of RAID controllers and enables you to stripe and protect logical volumes across multiple servers and access them anywhere on the network.
How is NVMesh different from mdadm/mdraid?
Linux mdraid was developed back in 2001 with the purpose of providing RAID volumes inside an individual host based on spinning disks with corresponding high access latency, long before NVMe drives with their hundreds of thousands of IOPS and microseconds latency were on the horizon. NVMesh does not only replace mdadm to deliver the full performance of NVMe drives inside a server, it also can span and protect RAID volumes across multiple servers and make these volumes accessible anywhere on the network. NVMesh is scalable from a single host up to a full data center of NVMe storage and comes with various options to manage your storage: An intuitive graphical user interface, a powerful command line interface and a REST API.
How is NVMesh different from NVMe-oF?
NVMesh development started in 2014, years before the first NVMe-oF standard was released. While the NVMe-oF standard protocol was designed as a very simple point-to-point connection between hosts (much like the old iSCSI), NVMesh was designed as a complete storage cluster management solution for NVMe storage. The NVMe-oF standard has no built-in functionality for RAID, management of logical volumes, error notifications or scale-out storage. Thus, NVMesh is not only a superset of NVMe-oF (based on its own highly optimized network protocol), it also is a full software defined scale out storage solution instead of a simple point-to-point connection.
However, if access to NVMesh via the NVMe-oF standard protocol is relevant for you, NVMesh volumes can be made accessible via NVMe-oF through a simple check box in the graphical management interface.
Which network types does NVMesh support?
NVMesh was primarily designed for RDMA-enabled networks such as InfiniBand or RoCE (RDMA over Converged Ethernet) to enable maximum performance and lowest latency. A TCP option to run NVMesh also over non-RDMA networks will be available soon.
How can I benchmark the performance of NVMesh?
Elbencho is a new open source benchmark tool that works for everything from block devices up to shared file systems, to get an understanding of throughput, IOPS, latency, be it for lots of small files or streaming/random access in large files, even including GPUs.
You can find it here: Elbencho on github
How is NVMesh delivered to me?
NVMesh comes as rpm packages for RHEL/CentOS or deb packages for Ubuntu. You will usually want to add the NVMesh yum or apt repository to your system for most convenient installation and updates.
Can I use NVMesh as the basis for IBM Spectrum Scale (formerly GPFS)?
Absolutely. Many of our customers use NVMesh as the basis for IBM Spectrum Scale. This does not only provide the data protection across servers and the full NVMe performance to the Spectrum Scale NSD servers, it also provides your applications much faster access to your Spectrum Scale file system, because NVMesh can make the block volumes directly available on the Spectrum Scale clients, which increases the Spectrum Scale IOPS by more than 10x.
Can I use NVMesh as the basis for BeeGFS?
Absolutely. Our customers combine NVMesh with BeeGFS to have a resilient block layer under BeeGFS, which makes the full NVMe performance available to the BeeGFS services. This combination is especially popular among our Deep Learning customers to ensure that the GPUs are not stalling on storage access.
To add the high availability and failover to BeeGFS services based on NVMesh protected volumes across servers, Excelero provides NVKeep as a very easy to install and manage solution and as an alternative to more complex options like pacemaker.
NVKeep is available here: NVKeep on github
Can I use NVMesh as the basis for NFS?
Absolutely. NVMesh can protect your data across servers and you can run the standard Linux NFS server on top of this, ideally with NFS RDMA transport enabled for best performance. Excelero provides NVKeep to make failover of your NFS service to another host easy.
NVKeep is available here: NVKeep on github
Can I use NVMesh as the basis for StorNext
Absolutely. The Quantum StorNext file system benefits from the fact that NVMesh enables concurrent access to its protected NVMe RAID volumes, looking to StorNext like a (very fast) SAN, while actually running on commodity servers thanks to the software-defined approach of NVMesh.
Can I use NVMesh for an Oracle RAC database cluster?
Absolutely. Thanks to the fact that NVMesh makes its protected volumes available to multiple hosts at the full speed of NVMe, our customers are reporting very impressive performance improvements and cost savings by using NVMesh for their Oracle databases.
Can I use NVMesh for Kubernetes (K8s)?
Absolutely. NVMesh comes with a CSI (Container Storage Interface) driver for Kubernetes, which enables you to dynamically create NVMesh logical volumes when needed and make them available anywhere on the network. This can be the basis for a containerized network file system or for a local file system like xfs/ext4 for the best performance.