This blog post represents my personal views and not necessarily those of Nimble Storage.

 

Competition is to be welcomed: without it vendors become lazy and customers find it harder to get (a) reasonable prices and (b) appropriate solutions. To work that competition needs to be carried out in a manner where each vendor is honest about what they are proposing and what they say about other vendors. Storage is not the only industry where people often fall short of these goals. At Nimble we pride ourselves on our open and honest approach to our customers in what we say about ourselves and our competition. This note aims to follow those principles and the author asks that if a reader believes it contains any inaccuracies you contact him to clarify and ensure this note is corrected if necessary.


‘Breaking the Law’ – Against 3PAR’s Own Principles

 

Post-acquisition many of the first wave of HP 3PAR customers got a very fair presentation of 3PAR capabilities. Some of the information 3PAR stated about alternative vendors was inaccurate, but it is harder to be right about someone else’s product than your own. The vast majority of 3PAR quotes were for 4 node configurations. This follows 3PAR design principles. At the time of writing (May 2016) it appears the majority of quotes HPE are offering to prospects who are also considering Nimble are for 2 node configurations. I have a history with 3PAR and feel moved to share why I think this is not a good change.


‘Good News First’ - 4 Nodes Good 2 Nodes Bad


One of the distinctive things about the 3PAR architecture is its ability to ‘mesh’ its controllers (nodes). This is similar in some ways to the ‘big iron’ products from EMC and Hitachi. HPE positions 3PAR as being in this ‘Tier 1’ group and they suggest no dual node architecture can match the availability offered by multi-node architectures. Some commentators have referred to 3PAR as being Tier 1.5. My perspective is that the needs of storage buyers are better served by metrics like measured up time across whole install base than somewhat academic positioning of different complex designs … but IT folk love a good architecture spat. 

The diagram below is taken from the HP paper ‘Eliminate compromise’.

TM.png

It’s pretty clear they are not fans of dual controllers! Their hostility is based on a range of suggested weaknesses they claim, in this paper and elsewhere, that other vendors’ dual-controller systems suffer from:

  • Non-active controllers being a waste of money
  • Performance decreases >50% if a controller is lost
  • It is hard or impossible to scale a dual controller
  • Availability is necessarily compromised

I will return to examine how these do not apply to Nimble later in the paper. HPE position 3PAR as having an ‘active-matrix’ and that needs a minimum of 4 nodes. To borrow from Orwell’s brilliant dystopia not all 3PAR are equal and two nodes is in 3PAR’s own terms not good.


‘Can’t Have Your Cake’ – 3PAR’s Animal Farm


‘Some animals are more equal than others’: not all dual controller systems behave the same way. Back in 1999, when the 3PAR design was outlined, many dual controller arrays would respond to a controller being off-line by going to ‘write-through’ mode and performance would collapse. The reason behind this was the only protection mechanism for writes not yet committed to disk was they were mirrored to the other controller. The 3PAR nodes (controllers) are in pairs, but when there are 2 or more pairs the partner of a node that goes down mirrors its writes to other nodes in the cluster. This is one answer to protecting writes – to be fair they did find an answer to the problem. Since then many vendors have used batteries or UPSs to avoid write through-mode. The modern answer is Super Capacitors - their reliability is so high it exceeds both battery or multi-node solutions for protecting data while a controller is off line. HPE 3PAR continue with their 1999 answer to the problem. They do use batteries to de-stage data in the event of power failure (an issue solved by Super Capacitors) and indeed write at length about the problems with batteries in their 3PAR StoreServ Architecture paper.

 

3PAR’s elephant in the room is their dual node (controllers) configurations still go to a write through mode when a node is off-line. Back in the day of the F200 (an old dual node only 3PAR model) the pre-sales folk at 3PAR would only sell an F200 for test labs and never for production use. I know of no case when sales bypassed their pre-sales to slip one in for production use. The pre-sales would also have argued very strongly that 4 nodes was the minimum for production on any 3PAR model. If your storage is HDD based write through mode is almost always crippling. It is not necessarily always as impactful if it is an all flash configuration, but it still represents a degrade mode of operations that is likely to only be tolerable in environments with very modest performance needs.


If dual controllers are so bad how come HP are proposing so many dual controller configs? The hard economics of a storage array are the controller isn’t a low cost item. If it involves custom ASICs then costs may be higher than ones using just standard Intel boards. No vendor wants to be ruled out because of their price. In addition, for many environments one modern controller has more than enough performance capability. The cost of a second can be justified – shared storage needs to be available and performant. Explaining how four nodes are needed when one on its own can do the job must be a tough task. The people pushing 2 node 3PARs may just be trying to get their prospects the lowest price. Even if their motives are good and they don’t understand the flaw in what they propose customers should be warned: a 2 node 3PAR is very much not the active-mesh ‘enterprise’ design that 3PAR put together in 1999, in fact it looks very similar to the dual controller arrays from that time (just with a wide stripe and thin provisioning added).


‘Too Late Too Late’ – 3PAR 8200, A Compromise

 

Capacity planning is part science, part art and a lot of blind luck. At some point the unexpected happens. Even in an all flash configuration the 3PAR 8200 is a compromised 2-node only design that can’t be upgraded to four controllers. It might be OK in a test lab. Anyone considering it for production use should take careful note of its limitation - after purchase it may be too late. This might seem pretty strong wording. This is what HPE say in their Optimized for Flash paper:

PC.png

I’ve heard differing feedback on the performance hit of turning on dedupe on a 3PAR and don’t want to debate this here. 3PAR generates it’s dedupe hashes in cache. It’s unclear from the public documentation if a 3PAR with one node can still generate it’s hashes. It may be that it has to turn off dedupe.

 

‘But I’m Different Now’ – Innovation


3PAR’s was a fresh approach and IT is full of pivot points where change somewhere powers innovation elsewhere. Sometimes a clean sheet is needed to unlock the full potential. The folklore is that the core 3PAR engineers weren’t storage people at all, but rather had compute cluster backgrounds. I have not been able to verify this. Nimble was founded 9 years after 3PAR and we shipped our first product 8 years after the first 3PAR shipped. That amount of time allows a fresh perspective. Nimble’s founders were earlier employees #10 and #11 at two of the storage industries most innovative and successful companies. The world in 2008 had three things that couldn’t be built into a design back in 1999. These are powerful multi-core CPUs (with Moore’s Law holding strong), flash storage and very large capacity drives. Part of the challenge for IT consumers is keeping up with the pace of change. When I started in storage it was a DAS world. At the time 3PAR was interesting. The IT world including storage is reinventing itself with a cycle of something like 10 years. In this context we have to expect very different answers compared to 17 years ago.

 

‘Two Hearts’ – How Nimble Do It


Earlier I gave four common points 3PAR throw at dual controller arrays. These are my responses to them.

  • Non-active controllers are a waste of money
    I can understand this perspective. I even have a degree of sympathy for it. People don’t like to have kit just sitting there – it feels inefficient. Active-active on dual controllers is often as bad an answer too. I have worked with customers of several active-active vendors who refuse to run them at more than 40% as when a node goes off-line (and they do go offline) anything above 40% risks production applications being on their knees. Even the 4 node 3PAR customers I knew only ran at <65% as a node of line is 25% of nodes gone (and everything needs some system headroom).
    The active-standby architecture was one of the things that led me to Nimble for the following reasons. (1) It’s simple. Have you seen the wiring diagrams of everything that isn’t dual node?? Complexity brings cost and multiplies the failure scenarios that have to be designed for. Simple, done right, wins for me on cost and reliability. (2) You can have a high degree of confidence your production systems will have enough performance in a failure situation.
    I am still left feeling I would like to be able to use the standby controller for something useful, so long as the failover to it remains transparent to production hosts. That would be an even better active-standby: but until someone does that, simple active-standby is the winner for me.   
  • Performance decreases >50% if a controller is lost
    This isn’t the case on Nimble, but it is on 3PAR!
    It isn’t on a lot of dual controller systems (it’s no longer the 20th century).

 

  • It is hard or impossible to scale a dual controller
    This is a problem for many dual controller arrays in terms of controllers. Even for those who say a data in place upgrade is possible a lot of these will run a country mile before doing it (because if you don’t design for it then it’s a nightmare). Other than the 8200 you can upgrade a 2 node 3PAR to 4 nodes (but not the controllers themselves), but … if the first two nodes have 128 drives then the recommendation is the second pair also have 128 drives. That’s a big upgrade step, particularly if you only need a little more power from the controllers and no more capacity.
    Nimble is different. Our performance is entirely CPU based and we actively encourage our customers to go with a sufficient model and only upgrade if/when necessary. Controller upgrades are non-disruptive and many customers do them during normal business hours. Our Timeless Storage program even offers customers a 25%+ more powerful controller after 3 years as part of Timeless Support.
    Nimble is also more than a dual controller. We offer scale out cluster so customers can have up to four arrays clustered. Arrays can be added and removed from clusters transparent to the hosts. No more fork lift upgrades! Most of my conversations these days are about all flash but as our clusters can mix all flash and hybrid arrays it doesn’t matter where you start - with Nimble you have the flexibility to scale in any manner that is appropriate.

 

  • Availability is necessarily compromised
    The truth on 3PAR is specific disks sit behind a pair of nodes. Data is available if one node fails, but is not available if the other fails. A 3PAR array is, from an availability perspective, just a collection of dual controllers. They say they are designed for six 9s availability but I have been unable to find any public references to their actual customer measured experience. Nimble has proven availability of 99.9997%. This is not a theoretical number – it is the actual number measured across our entire installed base and for applications that need more than that there are a range of solutions we can discuss.

 

‘Last Chance’ - Summary


Not all dual controller systems are the same. If performance and/or scalability matter to you take particular care with a 2 node 3PAR. If you buy a 2 node 3PAR you are not getting the active-matrix which is at the heart of their architecture. There are many better answers than a 2 node 3PAR: my ask is if you have a 2 node 3PAR quote you consider Nimble Storage, we might surprise you ;-)


HPE Reference Material
Eliminate compromise : http://www8.hp.com/h20195/v2/GetPDF.aspx%2F4AA3-2542ENW.pdf

HPE 3PAR StoreServ Architecture : http://h20195.www2.hp.com/v2/getpdf.aspx/4aa3-3516enw.pdf

HPE 3PAR StoreServ: Optimized for Flash : http://h20195.www2.hp.com/v2/GetPDF.aspx%2F4AA4-7264ENW.pdf


(PS No musicians were hurt in the writing of this piece. I like all but one of the songs whose titles are used in sub-headings but I refuse to say which I don’t.)