NVMe (non-volatile memory express)

What is NVMe?

NVMe (non-volatile memory express) is a host controller interface and storage protocol created to accelerate the transfer speed of data between enterprise and client systems and solid-state drives over a computer's high-speed Peripheral Component Interconnect Express bus.

NVMe is commonly used for solid-state storage, main memory, cache memory or backup memory. It provides an alternative to the Small Computer System Interface (SCSI) standard and the Advanced Technology Attachment (ATA) standard for connecting and transmitting data between a host system and a target storage device. NVMe was designed for use with faster media. The main advantages of NVMe-based PCIe SSDs over other storage types are the reduced latency and higher input/output operations per second (IOPS). 

As solid-state technology became the preferred medium in the storage market, it quickly became clear that existing interfaces and protocols -- such as Serial Advanced Technology Attachment (SATA) and Serial-Attached SCSI (SAS) -- were no longer suitable in data center environments. In early 2011, the initial NVMe spec was released. Nearly 100 tech companies were involved in its development.

NVMe is also a key enabler of evolving technologies and applications such as the internet of things, artificial intelligence and machine learning, which can all benefit from the low latency and high-performance improvements provided by NVMe-attached storage.

How does NVMe work?

The NVMe standard does not state NVMe's main use; however, it does define a register interface, command set and collection of features for PCIe-based SSDs with the goals of high performance and interoperability across a broad range of NVM systems.

The NVMe protocol can support any form of non-volatile memory, such as SSDs that use different types of non-volatile memory, including NAND flash. NVMe reference drivers are available for a variety of operating systems, including Windows and Linux.

An NVMe SSD connects through a PCIe bus or M.2 and U.2 connectors. The NVMe protocol, attached with these connections, enables lower latency and higher IOPS along with a reduction in power use.

NVMe commands map input/output (I/O) and responses to shared memory in a host computer over a PCIe interface. The NVMe interface supports parallel I/O with multicore processors to facilitate high throughput and address central processing unit (CPU) bottlenecks.

Why is NVMe important?

Designed for high-performance non-volatile storage media such as SSDs, NVMe is well-suited for highly demanding, compute-intensive settings. For example, NVMe can handle enterprise workloads while leaving a smaller infrastructure footprint and consuming less power.

By comparison, there is a large performance difference between NVMe and SATA. For example, NVMe experiences much less latency than SAS and SATA protocols. This increase in performance and reduced latency means that NVMe can be used with workload-intensive applications that need real-time processing while avoiding bottlenecks.

Enterprise environments and data centers can take advantage of the high-performance granted by NVMe-based storage.

What are NVMe's use cases?

Because of the blistering performance for flash storage NVMe provides, along with its ability to handle a high number of queues and commands, NVMe is suitable for:

• Professional and prosumer use, handling tasks such as graphics editing.
• Applications with large queue depths for storage I/O, including databases and some web operations.
• High-performance computing. Specifically, in applications where low latency is critical.
• Areas that require the storage of large amounts of data, such as in AI, machine learning, advanced analytics and big data.
• Relational databases. The better performance of NVMe flash memory systems lowers the number of physical servers needed.
• Applications that need to retrieve or store data in real time, such as finance and e-commerce apps.

What are the benefits of NVMe?

Benefits of NVMe include:

• NVMe drives can send commands twice as fast compared to AHCI SATA drives.
• NVMe SSDs have a latency of only a few microseconds, while SATA SSDs have latency between 30 and 100 microseconds.
• Efficient storage, management and data access.
• Has a much higher bandwidth compared to SATA and SAS.
• Supports multiple form factors, including M.2, U.2 and connections.
• Data can be divided and streamlined.
• Supports tunneling protocols, which address privacy.

What are the drawbacks of NVMe?

Potential drawbacks include:

• A lack of support for NVMe on legacy systems.
• Not cost-effective at storing large volumes of data. It is more expensive compared to spinning drives based on storage capacity.
• It is typically used with the M.2 format, which may further limit drive selection.

What are the differences between SATA, NVMe and SAS?

SATA. SATA is a communications protocol developed for computers to interact with hard disk drive (HDD) storage systems. Introduced in 2000, SATA superseded parallel ATA and quickly became the ubiquitous storage system protocol for computers. Over the years, revisions to the spec have been revved up to run at 6 Gbps with an effective throughput of up to 600 MB per second.

Although developed for hard disk technology with mechanical spinning platters and actuator-controlled read/write heads, early SSDs were marketed with SATA interfaces to take advantage of the existing SATA ecosystem. It was a convenient design and helped accelerate SSD adoption, but it was not the ideal interface for NAND flash storage devices and increasingly became a system bottleneck.

NVMe. Designed for flash, NVMe's speed and low latency leave SATA in the dust, and NVMe enables much higher storage capacities in smaller form factors such as M.2. Generally, NVMe performance parameters outdistance those of SATA by five times or higher.

SATA may be more established with a long history and lower implementation costs than NVMe, but it is hard disk technology that has been retrofitted to more modern storage media.

SAS. NVMe supports 64,000 commands in a single message queue and a maximum of 65,535 I/O queues. By contrast, a SAS device's queue depth typically supports up to 256 commands, and a SATA drive supports up to 32 commands in one queue.

However, NVMe-based PCIe SSDs are currently more expensive than SAS SSDs of equivalent capacity, although that delta is narrowing. Also, high-end enterprise NVMe SSDs may consume more power than SAS or SATA SSDs. The SCSI Trade Association claims the more mature SAS SSDs offer additional advantages over NVMe PCIe SSDs, such as greater scalability, hot pluggability and time-tested failover capabilities. NVMe PCIe SSDs also may provide a level of performance that many applications do not require.

History and evolution of NVM Express

The Non-Volatile Memory Host Controller Interface (NVMHCI) Workgroup began developing the NVMe specification in 2009 and published the 1.0 version on March 1, 2011. This specification included the queuing interface, the NVM command set, administration command set and security features. Other noteworthy dates include:

• On Oct. 11, 2012, the NVMHCI Workgroup, commonly known as the NVM Express Workgroup, released NVMe 1.1, which added support for SSDs with multiple PCIe ports to enable multipath I/O and namespace Other new capabilities included autonomous power state transitions during idle time to reduce energy needs and reservations, meaning two or more hosts could coordinate access to a shared namespace to improve fault tolerance.
• In May 2013, the NVM Express Workgroup held its first Plugfest to enable companies to test their products' compliance to the NVMe specification and to check interoperability with other NVMe products.
• In March 2014, the NVM Express Workgroup was incorporated under the NVM Express organization name. The group later became known simply as NVM Express Inc. The nonprofit organization has more than 100 technology member companies.
• On Nov. 3, 2014, the NVMe 1.2 specification emerged, with enhancements such as support for live firmware updates, improved power management and the option for end-to-end data protection.
• On Nov. 17, 2015, the NVM Express organization ratified the 1.0 version of the NVM Express Management Interface (NVMe-MI) to provide an architecture and command set to manage a non-volatile memory subsystem out of band. NVMe-MI enables a management controller to perform tasks such as SSD device and capability discovery, health and temperature monitoring, and non-disruptive firmware updates. Without NVMe-MI, IT managers generally relied on proprietary, vendor-specific management interfaces to enable the administration of PCIe SSDs.
• In June 2017, NVM Express released NVMe 1.3. Highlights centered on sanitize operations, a new framework known as Directives, and virtualization
• In July 2019, NVMe 1.4 was introduced along with enhancements and new features, including a Rebuild assist, Persistent Event Log, Asymmetric Namespace Access, Host Memory Buffer and Persistent Memory Region.
• In 2020, the NVMe Zoned Namespace Command Set Specification was ratified. This specification enables NVMe to isolate and evolve command sets for emerging technologies, including Zoned Namespaces, Key Value and computational storage.
• In June 2021, NVMe 2.0 was released. The specification was restructured to enable faster and easier development of NVMe and to support a more diverse NVMe environment.

NVMe form factors and standards

The need for a storage interface and protocol to better exploit NAND flash's performance potential in enterprise environments was the driving force behind the development of NVMe. But reimagining the connection standard opened the doors to several different types of interface implementations that could stay within the bounds of the new spec while offering a variety of implementation options.

In short order, a number of flash form factors conforming to NVMe specifications emerged, including the new form factors for SSDs dubbed M.2 and U.2.

• AIC. The AIC form factor enables manufacturers to create their own cards that fit into the PCIe bus without worrying about storage bay designs or similar limitations. The cards are often designed for special use cases and may include additional processors and other chips to enhance the performance of the solid-state storage.
• M.2. The M.2 form factor was developed to take advantage of NAND flash's compact size and low heat discharge. As such, M.2 NVMe devices are not intended to fit into traditional drive bay compartments, but rather to be deployed in much smaller spaces. Often described as about the size of a stick of gum, M.2 SSDs measure 22 mm wide and about 80 mm long, although some products may be longer or shorter.
• U.2. Unlike the M.2 form factor, U.2 SSDs were designed to fit into existing storage bays originally intended for standard SATA or SAS devices. U.2 SSDs look similar to older media, as they typically use the 2.5-inch or 3.5-inch enclosures that are familiar housings for HDDs. The idea, of course, was to make it as easy as possible to implement NVMe technology with as little reengineering as possible.
• EDSFF. Another, less widely deployed NVMe form factor is enterprise and data center SSD form factor. The goal of EDSFF is to bring higher performance and capacities to enterprise-class storage systems. Perhaps the best-known example of EDSFF flash is Intel's E1.L long and E1.S short flash devices, which are provided in what was originally referred to as the "ruler" form factor.

NVMe over Fabrics

NVM Express Inc. published version 1.0 of the NVMe over Fabrics (NVMe-oF) specification on June 5, 2016. NVMe-oF is designed to extend the high-performance and low-latency benefits of NVMe across network fabrics that connect servers and storage systems. The NVMe-oF 1.1 specification, which was released in 2019, offers improved fabric communication, finer grain I/O resource management, and end-to-end flow control and support for NVMe/TCP. Both specifications also offer lower latency, improved management and provisioning of flash and remote storage access.

Fabric transports include NVMe-oF using remote direct memory access (RDMA) and NVMe-oF mapped to FC. A technical subgroup of NVM Express Inc. worked on NVMe-oF with RDMA, while the T11 committee of the International Committee for Information Technology Standards (INCITS) developed NVMe over FC (FC-NVMe).

The NVMe-oF specification is largely the same as the NVMe specification. One of the main differences between them is the methodology for transmitting and receiving commands and responses. NVMe is designed for local use and maps commands and responses to a computer's shared memory via PCIe. By contrast, NVMe over Fabrics employs a message-based system to communicate between the host computer and target storage device.

The stated design goal for NVMe-oF was to add no more than 10 microseconds of latency for communication between an NVMe host computer and a network-connected NVMe storage device, in comparison to the latency associated with an NVMe storage device using a local computer's PCIe bus.

Uncertain where to begin with NVMe? Get answers to your most pressing questions.