SoftNAS Cloud NAS software paces itself when moving petabytes
SoftNAS engineers encountered a problem early this year with Amazon Web Services and Microsoft Azure clouds that caused a delay in the SoftNAS Cloud NAS 3.5 release.
The engineers hit a snag while doing quality assurance testing on pushing petabyte-scale migrations into the cloud using the SoftNAS primary storage filer. They needed to get the testing done in one month’s time, but the public cloud’s algorithms had other ideas.
“When you are moving massive amounts of data in the cloud, you have to have parallel I/O streams,” said Rick Braddy, CEO of SoftNAS. “When you do that, (the algorithms) start stiff-arming you and telling you to slow down. When you start sending data too fast, they start penalizing you. You get error messages and you have to do a retry.”
SoftNAS engineers hit a wall with the ingest rates. This problem forced them to innovate out of necessity because at the pace the cloud algorithms were dictating, the quality assurance testing would have taken six months to put a few petabytes into the cloud.
So the engineers developed patent-pending ObjFast technology, and integrated it into the company’s latest version of its SoftNAS Cloud NAS. The company claims the software-defined-storage product writes to the cloud more than twice as fast as previous SoftNAS Cloud NAS versions, effectively giving customers block-storage performance at the price of object storage.
“This basically delayed our whole product release by a quarter, so we ended up finishing the testing in the second quarter,” Braddy said.
Braddy said his team discovered in testing that each public cloud has its own pace for ingesting and the pacing varies depending on demand.
“It’s a shared service,” he said. “So the amount of throttling is based on everyone’s usage. What happens is if you don’t pace them correctly, they send you error messages and have to go into the retry loop.”
The new ObjFast technology uses massive parallel I/Os of object data streams coupled with algorithms to pace the data ingest rate for each I/O stream. That keeps the ingest rate from going over the clouds’ maximum allowed per stream data rate.
“We are not pacing their algorithm,” Braddy said. “We invented a proprietary pacing that adapts. Based on our testing, we found that Microsoft Azure’s overall ingest rate is faster than Amazon S3.”
SoftNAS Cloud NAS customers migrate PB of data
Braddy said a year ago customers typically moved approximately 50 TB to public clouds, but the vendor began seeing petabyte (PB) scale migrations to the cloud over the past three quarters.
The SoftNAS Cloud NAS marketplace capacities have gone from 1 TB and 20 TB instances to include 50 TB, 100 TB, 250 TB and 1 PB instances with annual licenses that can grow to 16 PB.
“We are seeing customers who are tired of being on the hardware treadmill,” Braddy said. “They have aging hardware now and, increasingly, they are being told by bosses to move to the cloud. A lot of organizations have aging Isilon scale-out file storage and they are moving away from it and into the cloud. We are also seeing a lot of file server consolidation.”
SoftNAS started selling its cloud software in 2012 for AWS, and added support for Azure and VMware vCloud Air in 2014.