CommVault wins best enterprise storage array, Acronis takes midrange honor

@exb @pb

Users tough on software

This is the only software product group in our Quality Awards program, and over the many surveys we've conducted, software seems to be judged more critically than hardware products, with ratings that are typically (and sometimes significantly) lower. But if there's any solace for the software vendors, it's that the scores have been steadily, if modestly, climbing since our first survey. In fact, HP's second-place score of 6.09 would have been good enough to win three of the four previous enterprise surveys.

The story's the same for midrange backup programs. Symantec's second-place overall score of 5.47 for Backup Exec would have bettered the winners of the first three Quality Awards surveys, including its own win in the second survey.

Other promising signs for backup software vendors are the improvements in the average scores for product features. Over the first three midrange surveys, those numbers languished in the 4.18 to 4.74 range, which, on our 1.00 to 8.00 rating scale, indicated the functionality of the rated apps was middling at best. The enterprise-class apps fared slightly better. Satisfaction with product feature sets, however, seems to be on the rise; for enterprise apps, the average feature scores on the last two surveys have been impressive, just barely exceeding 6.00 both times. Midrange products have also shown improvement, although less impressively so, with feature ratings finally climbing over the 5.00 mark.

Buying backup

Any IT purchase can be a gut-wrenching experience, not just because the buy might involve some big bucks, but because each purchase is a commitment. For backup software, the commitment goes beyond time, effort and other resources as you're also committing your company's data to the product. So the sales process is particularly critical for a data storage manager who's about to lock into a particular backup technology.

CommVault scored highest among enterprise backup applications in the sales-force competence judging category with a 6.20; HP, which proved to be a game competitor for CommVault, was second with a solid 6.10, well ahead of third-place finisher EMC NetWorker (5.85).

CommVault scored highest on six of the seven statements in this category, with its best ratings for "The vendor's sales support team is knowledgeable" (6.64) and "My sales rep is knowledgeable about my industry" (6.33). HP also did well on those statements (6.06 and 6.13, respectively), but topped CommVault -- 6.63 to 5.91 -- for when the sales process gets closest to signing on the dotted line ("The vendor's licensing formula offers good value"). All of the vendors fared well for having a knowledgeable sales support team, with EMC (6.37), IBM (5.77) and Symantec (5.65) getting their highest scores in this category for that statement.

The midrange products didn't fare quite as well in the sales-force competence category. Acronis picked up a couple of 6.00-plus ratings on its way to an category winning score of 5.65, but neither Symantec (5.36) nor CA (4.77) could muster one. Smaller businesses may be tougher customers for backup software vendors, but Acronis seems to be following the lead of its enterprise-class brethren, with its highest scores coming for those statements related to having a knowledgeable sales support team (6.11) and offering favorable licensing terms (6.11).

Click here to get a PDF of the Quality Awards V: Enterprise Backup and Recovery standings.

@pb

Click on "Start"

In four of the five evaluation categories for enterprise applications, HP nipped at CommVault's heels but ended up in second place. But in the initial product quality category, HP turned the tables to outdistance CommVault, 6.13 to 5.93, coming out on top for four of the seven statements in the process.

With software products, the key to long-term satisfaction is getting out of the gate fast with a quick and relatively pain-free installation. HP is apparently doing a good job of setting the tone early with its users by having a product that installs easily without customers requiring a lot of help from HP's professional services team. HP received its highest score in this category for the statement "This product was easy to install," and was the only vendor in the group to rise above a score of 5.00 for "This product did not require professional services."

Although it finished second to HP, CommVault was no slouch in the initial product quality category, with especially high ratings for "This product uses tape efficiently" (6.55) and "This product delivers good value for the money" (6.48). Its overall score was dragged down by a 4.52 for the statement about requiring professional services.

Arguably, midrange backup products should be easy to install and get up and running simply because the users of these products are likely to have fewer resources available to get through the initial stages of implementation. All three products showed their mettle in the initial product quality category, with Acronis (5.99) coming out on top, followed by Symantec (5.82) and CA (5.76); in addition, all three had 6.00-plus scores for the critical "easy to install" statement. Acronis scored very well for all seven statements in the category with one glaring exception -- a 4.00 for "This product uses tape efficiently," which, ironically, was where Symantec racked up one of its highest marks on the way to a very respectable performance.

Function follows form

A no-sweat implementation may indeed set the tone for a user's overall experience, but a backup application still has to deliver the goods with the required features and functionality. With the highest score recorded for the product features category (6.70), CommVault proves that there's plenty of substance behind the style of its backup suite. Its victory in this category is a tour de force, as it scored the highest for every statement, highlighted by the only over-7.00 score in the survey, a 7.02 for "This product's file system backup features meet my needs." CommVault also scored impressively for database backups (6.88), for being a "complete solution" (6.81) and for its disk backup functionality (6.80).

Second place HP also did well for features, with its overall 6.14 score bolstered by strong showings for database backup (6.45) and file system backup (6.38). IBM's Tivoli Storage Manager was hard on HP's heels with a category average rating of 6.12, highlighted by a 6.52 for its disk-based backup features. Symantec's NetBackup made a strong showing with an overall 5.89 built on three 6.00-plus statement scores.

Once again, survey respondents were a little tougher with their ratings for midrange backup apps, with the products earning scores lower than their enterprise counterparts. Symantec's Backup Exec showed why it's one of the most widely installed midtier backup applications by topping its competition on seven of the eight statements in the product features category for an average winning score of 5.68. Backup Exec did particularly well for file system backup features (6.05), backup to disk (5.99) and database backup (5.85); its lowest score -- a 5.27 for archiving features -- was just a hair shy of Acronis' score of 5.31. Acronis' Achilles' heels in this category were database backup (5.17) and scalability (5.17).

Click here to get a PDF of the Quality Awards V: Midrange Backup and Recovery standings.

@pb

In for the long run

The true test of a backup application comes with the daily grind of protecting a company's data assets; first impressions and feature sets are all well and good, but long-term reliability and stability are paramount. CommVault was the only enterprise-class backup app with a 6.00-plus average for the product reliability rating category, picking up a 6.24 while leading on six of eight statements. Its strongest showings were for meeting service-level requirements (6.86) and operating system/platform support (6.84).

Second place HP (5.94 overall) also showed strength for those two reliability criteria (a 6.18 and 6.59, respectively) among its consistent marks in this category. For the two statements that CommVault didn't have the leading score, EMC's NetWorker led the pack with a 6.07 for "Requires very few unplanned patches/updates," while IBM Tivoli Storage Manager garnered a top mark of 5.87 for "Very few bugs."

Within the midrange product set, Acronis bested CA (5.38) and Symantec (5.34) for five of the eight reliability statements on its way to winning the category with a 5.68 average. This was CA's sole second-place finish; its ARCserve Backup product came out on top for meeting service-level requirements (5.94) and for requiring few unplanned patches or updates (5.83). Backup Exec finished a close third with a winning score of 5.17 for "Vendor provides comprehensive upgrade guidance" -- an area where all three vendors' ratings were relatively low, indicating that some improvement in the upgrade process would be welcomed by users.

Software support

Backup software vendors have labored diligently to make their products easier to use with improved GUIs, wizards and other interface aids. But with an application as complex as backup, users will invariably need some help from time to time, so how well a vendor supports its product will figure significantly in a user's perception of ease of use. After backup vendors as a group registered a disappointing overall average of 4.88 in the first Quality Awards survey for backup software, they have shown steady progress in the quality of support they provide, with the best numbers yet recorded this time around (a very healthy 5.89).

CommVault's 6.20 score for the technical support category was enough to nudge out HP's 6.12. IBM (5.82), EMC (5.73) and Symantec (5.58) turned up pretty good ratings as well. CommVault is apparently meeting users' support expectations, as its highest score in this category was a 6.86 for "Vendor supplies support as contractually specified." HP topped CommVault on the statements dealing with documentation (a 6.15 vs. CommVault's 6.00) and "Support issues rarely require escalation" (6.00 and 5.69, respectively). EMC was rewarded with a statement-high score of 6.27 for training its third-party partners well.

Again, we detect some grumbling among midrange users as they aren't nearly as satisfied with their vendors' support efforts. Although the overall support score average is higher than it's ever been, it's still a rather unimpressive 5.23.

Acronis topped the category with a modest 5.39 overall support score, but did manage to get the only 6.00-point rating in the category with an even 6.00 for providing support as contracted. All three vendors failed to reach the 5.00 plateau for providing adequate training, with Acronis earning a 4.80, followed by Symantec (4.74) and CA (4.59). And the three didn't fare much better for providing adequate documentation, with scores ranging from Acronis' 5.05 to Symantec's 5.08, with CA sandwiched in between at 5.07. The message to midrange backup software vendors seems clear: Provide more support and you're likely to see better all-around scores on future surveys.

Brand loyalty

As on all Quality Awards surveys, we ask respondents if, given what they now know, they would make the same purchase again. Given its overall performance on this survey, it's not too surprising that CommVault had the highest percentage of users who said they'd do it all over again (83.7%). Only 62.9% of HP's users said they would buy Data Protector again, a surprisingly low number considering how well the product was rated overall.

For the midrange applications, 69.2% of Acronis' users said they're pleased enough to buy the product again, followed by Symantec Backup Exec (66.7%) and CA ARCserve Backup (57.1%).

Although not necessarily related to user satisfaction, it's interesting to see how heavily users are taxing these backup products. Among enterprise programs, IBM's Tivoli Storage Manager is the workhorse, backing up an average of 67.1 TB weekly. For the midrange apps, Symantec's venerable Backup Exec is the heaviest lifter with an average of 13.3 TB of data backed up each week.

BIO: Rich Castagna ([email protected]) is editorial director of the Storage Media Group.