Alternatives to RESTful API for accessing object storage

This tip examines other options for connecting to object storage when an application isn't compatible with the RESTful API.

The RESTful API is the native access to object storage and the most efficient way to on- and off-ramp object storage. It provides the lowest latency and the highest level of management control. There are three current de facto and published standards: Amazon Simple Storage Services, SNIA CDMI and OpenStack Swift, as well as the proprietary variations, which I have detailed in a previous tip on the topic. However, there are times when it's not practical -- or possible -- to modify an application. This tip examines the alternative software options to modifying an application using object storage RESTful API.

There are many applications and file systems that just cannot be modified to utilize a RESTful API because of license reasons, warranties, dearth of programming skills, etc. For those applications to take advantage of object storage without modification means the object storage has to look like traditional target storage to the application file system. In other words, the object storage has to look like a mountable POSIX-compliant file system such as NFS, CIFS and/or the Hadoop Distributed File System (HDFS), or a SAN storage target via the iSCSI protocol. Fortunately, there is a considerable amount of software available that can do that today.

The most common is conversion, or gateway, software that's packaged as a feature of the object storage software. In this manner, the object storage looks and feels like a mountable POSIX-compliant file system or an iSCSI (block) target storage system. There are upsides and downsides to this approach. The advantages come from the object storage system responding or acting the same as a traditional storage system. No changes or alterations are required to the application or file system. It also allows files stored via NFS to be accessed via other file protocols such as CIFS or HDFS and vice versa. In addition, it allows objects stored via the RESTful API to be accessed as files and files to be accessed as objects.

There are downsides or tradeoffs with this approach as well. First and foremost, there is greater latency, leading to longer response times. Anytime one interface must be converted to another, there will be more latency. In addition, the NAS experience is not identical. This is because the software conversion function is an interface convenience, not a replica of a NAS system with all of its features. The iSCSI response times also tend to be slower than native iSCSI storage (latency again) and are not accessible any other way than as iSCSI blocks.

Many of these conversion features (in some cases, all) are available on object storage from Amplidata Amplistor, Caringo CAStor, CEPH open source, Coho DataStream, EMC Atmos, NetApp StorageGRID, Quantum Lattus and Scality RING.

A significant variation of the bundled gateway features comes from Cleversafe dsNet. Cleversafe delivers Hadoop integration as an optional bundle with its dsNet object storage. Rather than providing a HDFS conversion feature, Cleversafe integrated Hadoop to work natively with its RESTful object storage API, eliminating the additional latency.

Another significant variant is from Joyent Manta. Joyent delivers combined compute and object storage in each node. Applications developed and running on a Manta node automatically take advantage of the Joyent object storage. This is similar to the way Amazon Web Services (AWS) automatically takes advantage of Amazon Simple Storage Service in the Amazon cloud.

Third-party ISV software, data movers

A bit less common is third-party ISV conversion or gateway software available on physical machines or (more frequently) as virtual machines (VMs) on virtual servers. Third-party software operates similarly to software features bundled with object storage, with notable differences. The most common are deduplication and compression as part of the software before the files or objects are stored on the object storage system. Most of these software gateways were designed for object storage provided by cloud service providers. The deduplication and compression is a necessity to reduce WAN traffic and service provider monthly charges. These products do reduce the amount of capacity required on the object storage; however, they add significant latency above and beyond the integrated or feature software, and the files or objects themselves can be accessed only via the gateway software. This can create a significant performance bottleneck. This type of gateway software is frequently limited to a single protocol such as NFS, CIFS or iSCSI.

Third-party ISV gateway software is offered by BridgeSTOR Coronado NAS Gateway, Panzura NAS Quicksilver VM, Solid Instance Geo Shadow Cloud Filing System, South River Technologies Web Drive (Windows and Mac) and TwinStrata NAS/SAN Cloud Array VM.

There are also third-party ISV data movers. This type of software moves data directly to object storage in the form of an archive via CommVault Simpana 10, NTP Software Precision Tiering and Symantec Enterprise Vault; as a backup storage target via Asigra 12.2, CommVault Simpana 10 and Symantec NBU 7.5; or as data migration or tiering via NTP Software Precision Tiering.

Whichever route you decide is best, there are a lot of software options, whether they're object storage features or third-party ISV gateways, tools or applications that enable data on-ramping and off-ramping to object storage.

Next Steps

Everything you need to know about REST APIs

Dig Deeper on Storage architecture and strategy

Disaster Recovery
Data Backup
Data Center
and ESG