Comparing 7 storage network protocols

What are storage network protocols?

A storage network protocol provides a standard set of rules that define how data is transmitted between devices. Systems such as network-attached storage (NAS) and storage area networks (SANs) rely on storage protocols to facilitate data communications. Cloud storage platforms also use protocols to provide access to their data repositories.

Seven storage network protocols

Although there are numerous niche protocols, there are seven enterprise storage protocols that tend to be far more widely used than the others. Let's take a look at these protocols:

Internet Small Computer System Interface (iSCSI)

ISCSI is a transport layer protocol that provides block-level access to storage devices over a TCP/IP network. The protocol works on top of TCP and describes how to transmit SCSI packets across LANs, WANs or the Internet. ISCSI enables IT to set up a shared storage network, such as a SAN.

Organizations often turn to iSCSI because it uses standard Ethernet technologies, making it cheaper and easier to adopt than Fibre Channel (FC). ISCSI can deliver high speeds across long distances, taking advantage of multipathing, jumbo framing, data center bridging (DCB) and other technologies. SAN implementations based on iSCSI now support data rates as high as 25 Gigabit Ethernet, with 50 GbE and 100 GbE not far behind.

Fibre Channel

Fibre Channel is a high-speed networking technology that delivers lossless, in-order raw block data. The technology defines multiple communication layers for transporting SCSI commands and information units using the Fibre Channel Protocol (FCP). In addition to SCSI, Fibre Channel can also interoperate with IP and other protocols. It offers point-to-point, switched and loop interfaces and can deliver data rates up to 128 Gbps.

Fibre Channel was created to support SANs and address the shortcomings in SCSI and High-Performance Parallel Interface (HIPPI). It offers a reliable and scalable protocol and interface with high throughput and low latency, making it well-suited for shared network storage. When used with optical fiber, Fibre Channel can support devices as far as 10 km apart. However, FC networks can be complex and require specialized equipment such as switches, adapters and ports.

Fibre Channel over Ethernet (FCoE)

The FCoE protocol enables Fibre Channel communications to run directly over Ethernet. The protocol encapsulates the FC frames in Ethernet frames, using a lossless Ethernet fabric and its own frame format. FCoE makes it possible for LAN and SAN traffic to share the same physical network but remain isolated from each other. It works with standard Ethernet cards, switches and cables, along with FCoE-enabled components. FCoE can support the same data rates as high-speed Ethernet.

With FCoE, an organization can use a single cabling method throughout the data center, helping to simplify management and reduce costs compared to regular Fibre Channel. FCoE also retains some of the latency and traffic management benefits of regular Fibre Channel, and it can use DCB to eliminate loss during queue overflow. However, FCoE will not work across routed networks such as Fibre Channel.

Network File System (NFS)

NFS is both a distributed file system and network protocol for accessing and sharing files between devices on the same LAN. The system and its protocol are commonly used to support NAS. NFS is a low-cost option for network file sharing that makes it possible for users and applications to access, store and update files on a remote computer, much like they would with DAS.

NFS uses the Remote Procedure Call (RPC) protocol to route requests between clients and servers. Although participating devices must support NFS, they don't need to understand the network's details. However, RPCs can be insecure, so NFS should be deployed only on trusted networks behind firewalls. The protocol is used primarily in Linux environments, although it is supported by Windows.

Server Message Block/Common Internet File System (SMB/CIFS)

SMB is a client-server communication protocol that enables users and applications to access storage and other network resources on a remote server. Because it's a request-response protocol, it transmits multiple messages between the client and server to establish a connection. SMB operates at the application layer and can run on TCP/IP networks. Like NFS, the protocol is commonly used for NAS.

Since it was first released, numerous SMB dialects (implementations) have been released. One of the earliest was CIFS. Introduced by Microsoft, it was known as a chatty protocol that was buggy and prone to latency issues. Even so, it was embraced by OSes such as Windows, Linux and Unix. Subsequent SMB dialects have made CIFS all but obsolete. Even so, the terms SMB and CIFS are often used interchangeably or referred to as SMB/CIFS, although CIFS is only a single SMB implementation.

Hypertext Transfer Protocol (HTTP)

HTTP isn't typically thought of as a storage protocol, but it supports access to cloud storage services such as Amazon S3, Google Cloud Storage and Microsoft Azure, usually through RESTful APIs and standard HTTP/HTTPS requests. Amazon S3 has become the de facto standard for cloud object storage and is now supported by on-premises storage systems, including NAS, cementing HTTP's role as a storage protocol.

HTTP is a World Wide Web application protocol that runs on top of TCP/IP. It provides a set of rules for transferring data between HTTP endpoints, which send requests and receive responses. The protocol is based on a client-server model and is widely supported and implemented. Most programming languages include HTTP-request capabilities, which makes it possible for almost any application to access storage using standards-based technologies.

Non-volatile memory over fabrics (NVMe-oF / NVMe/TCP)

Built on the NVMe specification, NVMe over Fabrics (NVMe-oF / NVMe/TCP) is a high-speed storage protocol for accessing solid-state storage across network fabrics such as Ethernet, Fibre Channel and InfiniBand. NVMe-oF defines a common architecture for interfacing with storage systems using NVMe message-based commands. The protocol can support many NVMe devices while extending the distances between NVMe devices and their subsystems.

According to NVM Express Inc., 90% of the NVMe-oF protocol is the same as basic NVMe, which was designed for SSDs that connect directly to a computer through a PCIe bus. Like NVMe, NVMe-oF can take better advantage of a flash drive's inherent speeds, which are often limited by more traditional protocols and interfaces. Storage vendors offering all-flash arrays are quickly adopting NVMe-oF to support data-intensive workloads and high-performance computing. Many believe that NVMe-oF will eventually become the de facto protocol for enterprise storage.

How protocols differ in SAN vs. NAS

At one time, there was a clear distinction between NAS hardware and SAN hardware. However, modern storage appliances often support both NAS and SAN functionality. Whether such an appliance is functioning as a NAS or a SAN device comes down to how the storage is being presented to clients and the protocols that are being used.

As an example, a portion of the storage that is available within an appliance might be provisioned for NAS use. This means that the storage contains its own file system, access control list and that clients access the storage at the file level. As an example, an organization might opt to make a few terabytes of space available to users as NAS storage. The appliance would format the storage using something like NTFS, ZFS, or ext4. The organization might then make the storage available to users by creating an SMB share and then mapping a network drive to that share. NAS appliances commonly support the SMB and NFS protocols.

That same appliance might also be configured to act as SAN storage. In doing so, a server would connect to the raw storage using a protocol such as iSCSI, Fibre Channel or NVMe-oF. The server interacts with the SAN storage at the block level. While the server may ultimately format the storage using the same file system as is used on a NAS device, the server continues to access the storage at the block level, not the file level.

Simply put, NAS devices provide file-level access while SAN devices are accessible at the block-level. This is an important distinction, because SAN protocols prioritize performance, low latency and predictable throughput. These characteristics are what make SAN suitable for enterprise use cases such as hosting databases, hypervisors, and enterprise applications.

NAS protocols, on the other hand, prioritize simplicity, multi-user file sharing, centralized management, and cross-platform access. While NAS is used in the enterprise, common use cases include things like collaborative file storage, hosting user's home directories or as a backup repository.

Choosing the right protocol

Unfortunately, there is no enterprise storage protocol that is the best option in every conceivable situation. Storage protocols are designed for specific use cases and often have trade-offs.

When it comes to protocol selection criteria, the first step is to consider the access model. After all, your options will vary depending on whether you need access to block, file or object storage.

Access to block storage can be achieved by using protocols such as iSCSI, Fibre Channel and NVMe-oF (or NVMe/TCP). iSCSI is often the best choice for organizations that need a relatively simple and cost-effective option using existing infrastructure. This is especially true if the organization wants to provide access to storage over an existing Ethernet network. In the real world, it may be useful to perform a storage protocol performance comparison before deciding on a protocol.

Fibre Channel has long been a popular option for enterprise datacenters and is known for delivering predictable performance and extreme reliability. However, enterprise datacenters are increasingly transitioning to newer standards such as NVMe-oF or NVMe/TCP because of their reduced latency and better overall support for NVMe devices.

The primary protocol options for file-level storage are SMB and NFS. SMB is the protocol of choice in Windows environments, while NFS is a better fit for Linux and UNIX environments.

When it comes to object storage, the storage is generally accessed through cloud storage protocols (HTTP/S3). Object storage providers will generally specify the protocol that they require, although many object storage platforms are designed to be S3 compatible.

Brien Posey is a former 22-time Microsoft MVP and a commercial astronaut candidate. In his more than 30 years in IT, he has served as a lead network engineer for the U.S. Department of Defense and a network administrator for some of the largest insurance companies in America.