First, take a look at this graphic depicting a “typical” server storage array consisting of 100 enterprise-class HDDs.

Each short-stroked, enterprise-class HDD has a capacity of 300 Gbytes, for a total array capacity of 30 Tbytes. This enterprise-class HDD array delivers 30K IOPS, costs $55,000, and consumes 1.75 kilowatts of electricity (not to mention an equivalent amount of electricity required for cooling). That’s the baseline.

Now look at this graphic, which compares the previously discussed array of enterprise-class HDDs with a hybrid array consisting of one SSD and 30 high-capacity, 1-Tbyte HDDs.

The high-capacity drives are not short-stroked, so they can provide a total storage capacity of 30T bytes with only 30 drives instead of 100 enterprise-class HDDs. However, the one SSD inserted into the drive array provides the same IOPS read performance as the 100 enterprise-class HDDs, so the use of the slower, less expensive, high-capacity HDDs in the second array is not a detriment to the second array’s IOPS performance, as long as the server software is written to make use of the hybrid array’s abilities.

The SSD-enhanced drive array costs $6040 or about 90% less than the array of 100 enterprise-class HDDs. The SSD-enhanced array consumes 0.392 kilowatts, which is nearly 80% less than the enterprise-class array of 100 short-stroked HDDs. Consequently, the second drive array generates substantially less waste heat (that must be cooled) than the full array of enterprise-class HDDs.

As a result, the SSD-enhanced drive array saves the enterprise customer a substantial amount of money when viewed from a systemic perspective. Relative to the enterprise-class HDD array, the SSD-enhanced hybrid drive array costs less to purchase; costs less to provision because fewer drives require less rack space and fewer racks; consume less electricity for operation; need less electricity for cooling because fewer, slower drives generate less heat; and reduce maintenance costs because the high-capacity drives run cooler (increasing MTBF), because there are fewer drives to maintain, and because high-capacity drives are much less expensive than enterprise-class drives. Overall, the TCO calculations favor the SSD-enhanced, hybrid drive array.

Handy took Sun’s numbers a step further by calculating the crossover point where TCO considerations favor an SSD-enhanced drive array over an array of enterprise-class HDDs when the IOPS performance is the main consideration rather than capacity. Here are Handy’s graphs:

The left graph shows price curves for an array of enterprise-class HDDs versus an array of SSDs. The array of SSDs initially costs more than the enterprise-class HDDs, so the crossover point is 1200 IOPS due to the high initial SSD cost. As the IOPS requirement rises, you need to add enterprise-class HDDs to the array to meet the higher IOPS requirements but one SSD gets you a lot of IOPS so there’s no need to add one until the IOPS requirement exceeds around 3000 IOPS. For the enterprise-class hybrid array, which mates one SSD with several high-capacity HDDs, the purchase cost of the SSD-enhanced array is much lower for a given capacity so the crossover point is also lower—just 400 IOPS.

TCO computations such as these are required for storage and for memory subsystems. It’s easy to be myopic and compare component cost to component cost, but system architects are creating systems and should always try to view component costs through a TCO lens.

• Solid-state disk drives (SSDs). With SATA and SAS interfaces, SSDs can plug directly into most existing systems and will provide an immediate performance boost.
• Flash caches. Located very near the DRAM, Flash caches provide fast ways to back up DRAM data over wide memory buses with high bandwidth and low latency. AgigA Tech’s AGIGARAM and Denali’s FlashPoint controller are both aimed at this NAND subniche.
• Specialty storage devices based on non-disk interface standards. Disk interfaces including SATA and SAS were developed with built-in assumptions about the drives they support. Those assumptions include some temporal assumptions based on having rotating mechanical memory. Those assumptions don’t apply to NAND-based storage devices so it’s possible to use interfaces with more bandwidth, PCIe and Hypertransport for example, to connect such storage and get better performance. This is the sort of product available from Fusion-io.

Which finally brings us to the trigger for this blog entry. The MIT/Stanford Venture Lab (VLAB) held a panel discussion at Stanford University on Tuesday, November 17 and the evening’s topic was “SSDs: Game-Changing Technology for Better, Bigger, Faster Applications and Application Development” and the first speaker was David Flynn, President and CTO of Fusion-io. Flynn’s talk contained many interesting and worthwhile things for followers of NAND-related topics as they relate to computer system design.

Early in his presentation, Flynn projected a photo of Charlie Chaplin playing one of the last great roles of his life, “The Great Dictator.” However, Chaplin’s roundish face had been replaced with an inset photo of a hard-disk platter and the caption was: “Getting rid of nasty Disc-tators.” Flynn emphasized that Fusion-io’s PCIe-connected products are not solid-state disks and they deliver more performance than solid-state disks because they are connected to a data pipeline that delivers more performance than existing disk interfaces. They are I/O-memory devices that provide 10x the capacity of DRAM per dollar, 50x the capacity of DRAM per “module,” and 100x the capacity of DRAM per Watt. Using these metrics, Flynn is making it clear that he understands the figures of merit valued by his company’s prospects.

Flynn then compared NAND Flash memory to aircraft aluminum. When metallurgists developed aluminum alloys suitable for aircraft, the entire airframe had to be re-engineered because aeronautical engineers couldn’t use aluminum as a direct replacement for wooden struts and dope-covered fabric. Aluminum ushered in new types of airframes that rapidly evolved. Aircraft performance soared as a result.

The same is true of computer systems (and software) developed before and after the Flash Zone is filled with something, whether it’s SSDs, Flash caches, or I/O-attached storage. Assumptions must be rethought and systems and software need to be redesigned to fully exploit the advantages of a populated Flash Zone.