The NVMe family of specifications published by the NVM Express consortium includes support for the TCP transport. Implementation details are defined in the NVMe over TCP Transport Specification (NVMe/TCP), which describes how to deploy end-to-end NVMe across a standard TCP/IP network without needing to reconfigure the network or implement special equipment. The consortium published the latest release of the specification, Revision 2.1, on July 30th, 2025.
NVMe over TCP enables organizations to support NVMe communications over their TCP/IP networks. When the NVM Express consortium first introduced NVMe in 2011, the protocol was limited to direct-attached SSDs that adhered to the PCI Express (PCIe) connection standard. In 2016, the consortium published the first NVMe over Fabrics (NVMe-oF) specification, which extended NVMe so it could be used over network fabrics.
The initial implementation of NVMe-oF supported only Fibre Channel and remote direct memory access (RDMA) fabrics such as InfiniBand, RDMA over Converged Ethernet (RoCE) and the Internet Wide Area RDMA Protocol (iWARP). Although these are well-known network technologies, they can be complex to implement or require special equipment and configurations.
In October 2019, the NVM Express consortium published NVMe-oF 1.1, which added support for the TCP transport binding, making it possible to use NVMe over any Ethernet network, as well as the internet. With NVMe/TCP, organizations can avoid many of the challenges that come with Fibre Channel and RDMA networks, helping to simplify network storage solutions.
NVMe-oF 1.1 was the last published NVMe-oF specification. Since then, the consortium has rolled much of NVMe-oF into the NVMe Base Specification, while adding separate specifications for the specific transports, including PCIe, RDMA and TCP.
Although a relatively young technology, NVMe/TCP is steadily gaining traction in modern data centers, including cloud providers and hyperscalers. The TCP transport provides a number of key features:
NVMe/TCP uses standard Ethernet networks. It does not require special network equipment such as switches, routers or network interface cards (NICs).
NVMe/TCP defines a methodology for encapsulating NVMe commands directly into TCP packets, making it possible to use NVMe on any TCP/IP network.
NVMe/TCP supports the optional use of Transport Layer Security (TLS), which can be layered on top of TCP.
NVMe/TCP supports software-only implementations that use existing TCP network transport software application interfaces. There is nothing in the specification to prevent NVMe/TCP from also being used for hardware-only or hardware-accelerated implementations.
NVMe/TCP can take full advantage of the scalability and availability inherent in Ethernet networks.
NVMe/TCP offers organizations a state-of-the-art storage protocol that can fully benefit from today's SSDs, unlike older protocols such as iSCSI. With NVMe/TCP, organizations can more effectively support workloads that require high throughputs and low latencies, including AI, machine learning and real time analytics.
The specification describes how a host system and an NVM subsystem controller can communicate directly over TCP by exchanging Protocol Data Units (PDUs) that contain capsule, data or control/status information.
NVMe/TCP binds the PDUs directly into TCP packets to facilitate end-to-end communications over any standard TCP/IP network. In this way, data centers can use their existing network infrastructures to support their distributed workloads, while maximizing performance and minimizing latency.
NVMe over TCP offers a number of important benefits that can make it a good fit for many organizations:
NVMe/TCP can take advantage of TCP's ubiquitous nature. Not only does TCP help drive the internet, but it's also implemented extensively across networks around the world, making it one of the most common transports in use. The protocol is well known, well understood and actively developed by key players that contribute to maintaining and enhancing its capabilities.
NVMe over TCP is designed to work with existing TCP-based systems without requiring changes to the network infrastructure or client devices. IT teams can deploy NVMe/TCP on existing Ethernet routers, switches, adapters and other standard equipment, helping to simplify implementation and minimize downtime and costs. Organizations do not need to invest in new or specialized equipment or the resources necessary to implement and maintain that equipment.
Because TCP is fully routable, it supports large-scale deployments that can span great distances, while maintaining relatively high levels of performance and low levels of latency.
NVMe/TCP can be integrated into cloud-native environments and software-defined storage and can be used with automation and orchestration services. Because TCP has been so widely adopted, organizations can also avoid vendor lock-in.
NVMe over TCP offers a good balance between cost and performance. It does not require a special network infrastructure like Fibre Channel or RDMA, making it easier and cheaper to implement. It can also deliver exceptional performance, much better than iSCSI. It's not as fast as specialized storage networks, but it is more than adequate for many workloads.
Organizations of all sizes can potentially benefit from NVMe/TCP. Data centers can support large-scale workloads while balancing performance and costs. Enterprises can more effectively support their distributed workloads, and smaller organizations can avoid the costs that come with more specialized storage networks.
NVMe over TCP is designed to work with existing TCP-based systems without requiring changes to the network infrastructure or client devices.
Despite the advantages of NVMe over TCP, the technology also comes with its own challenges:
NVMe/TCP can increase system processor loads because TCP requires processing power to manage certain operations, such as calculating checksums.
NVMe/TCP can result in higher latency rates, in part because of the additional copies of data that must be maintained in the TCP stack. This increased latency might not be acceptable for certain mission-critical workloads.
Networks that support NVMe/TCP operations must be carefully tuned and optimized to handle the additional load and traffic type to avoid performance degradation for all workloads.
Other types of storage networks, such as iSCSI, Fibre Channel and RDMA, are well established, well understood and highly implemented. Although NVMe/TCP is gaining traction, it is not nearly as mature and ubiquitous as the other network storage technologies.
Many organizations have already invested in specialized storage networks such as Fibre Channel and RDMA, and switching to NVMe/TCP might not be practical given their current investments and circumstances.
Although NVMe over TCP is not a good fit in all situations, it could prove well-suited to many data-driven applications, especially those that currently depend on iSCSI. In addition, NVMe/TCP performance is likely to improve as more vendors incorporate the technology into their products.
The NVMe/TCP transport binding defines the methodology used to encapsulate and deliver data between a host and a non-volatile memory subsystem. The binding defines how queues, capsules and data should be mapped to support TCP-based communications between a host and controller across a standard IP network.
The host and controller communicate by exchanging PDUs, which provide a structure for transferring data, capsules or control and status information. The exact length and configuration of a PDU depends on its specific purpose during communications.
The following steps provide an overview of the NVMe/TCP communication process:
The host and controller establish a TCP connection using a TCP handshake. The host initiates the connection by sending a request message to the controller.
The controller is a passive component that listens for connection requests. Upon receiving a request, the controller sends a response to the host, acknowledging the request and establishing communications.
After communications have been established, the host sends a PDU to the controller, requesting that the connection be initialized.
The controller responds by sending its own PDU, confirming the connection has been initialized. The host and controller also share connection configuration parameters.
After the connection has been initialized, the host and controller can carry out the data exchange.
If TLS protection is used when establishing a connection, a TLS handshake will follow the initial TCP handshake. The normal PDU exchange process will then follow.
NVMe/TCP communications incorporate a messaging and queuing model that establishes the communication sequence. The TCP binding uses a transport-specific mechanism and, optionally, an in-capsule data component to support data transfers. Each connection incorporates a single queuing pair made up of an admin or I/O submission queue, along with a completion queue.
What is TCP?
TCP is a widely accepted standard that defines how to establish and maintain network communications when exchanging application data across a network. The protocol determines how messages are assembled into smaller packets before transmitting them and how to reassemble them at their destination, ensuring that all data is transmitted and received correctly.
TCP works in conjunction with the Internet Protocol, which determines how to address and route each packet so it reaches the correct destination. Both TCP and IP are part of the TCP/IP suite of communication protocols used to facilitate communications across the internet and private networks.
TCP/IP is divided into four layers: application, transport, network and physical. The transport layer includes TCP, the network layer includes IP and the physical layer includes Ethernet, which defines a standard for interconnecting TCP/IP nodes in a LAN.
Like NVMe/TCP, iSCI is a storage protocol that operates over Ethernet networks. The protocol determines how Small Computer System Interface (SCSI) packets are transported across a TCP/IP network. It enables block-level data to be safely transported between an iSCSI initiator on a server and an iSCSI target on a storage device.
The iSCI protocol was designed to support SCSI-based storage devices. Over the years, it has worked well for networked HDDs. Although iSCSI can also be used with SSDs, it cannot deliver the high performance and low latency necessary to take full advantage of the SSD's capabilities. In contrast, NVMe is optimized for flash storage, supporting higher throughput and lower latency.
In 2023, Dell Technologies conducted performance testing on several transport protocols, including NVMe/TCP and iSCSI. The testing included the following results:
NVMe/TCP delivered up to 75% greater read IOPS than iSCSI and up to 51% greater write IOPS. For operations that were half read and half write, NVMe/TCP delivered up to 75% greater IOPS.
NVMe/TCP delivered up to 42% greater CPU utilization for read operations than iSCSI and up to 29% greater CPU utilization for write operations. For operations that were half read and half write, NVMe/TCP delivered up to 24% greater CPU utilization.
NVMe over TCP uses a streamlined command set that's optimized for flash storage. It also queues commands more efficiently and has lower protocol overhead, resulting in greater throughput and lower latency rates than iSCSI.
NVMe/TCP in action
Storage vendors have been steadily adding support for NVMe over TCP to their products. For example, Lightbits Labs offers a hyperscale software-defined storage platform that incorporates NVMe/TCP. The platform provides disaggregated block storage that can be implemented without affecting the network infrastructure or data center clients.
Dell Technologies is also embracing NVMe over TCP in its storage products. For instance, its PowerFlex software-defined infrastructure platform now supports NVMe/TCP, as does its PowerStore software-driven storage platform.
Pure Storage has also added NVMe/TCP support to its Purity//FA storage software, and VMware now supports NVMe/TCP in its virtualization platform. In addition, Linux and Windows Server have incorporated NVMe/TCP into their systems.
Given the extent to which TCP is used across the internet and within enterprise networks, it's no surprise that NVMe over TCP is making significant inroads into the storage industry. Organizations can use existing TCP-based network infrastructures and state-of-the-art SSDs, helping to ease the transition from legacy storage systems to ones that meet the demands of today's dynamic workloads and massive data sets.
NVMe/TCP might come with its own latency issues compared to NVMe over RDMA or Fibre Channel, but for many organizations, the tradeoff is well worth it, given how easy and cost-effective it is to implement NVMe/TCP.
Robert Sheldon is a freelance technology writer. He has written numerous books, articles and training materials on a wide range of topics, including big data, generative AI, 5D memory crystals, the dark web and the 11th dimension.
