The Future of NAND Flash Memory – Predictions for 2010

This is the last AgigA Tech blog entry for 2009. We’ve covered a lot of ground in a bit less than three short months of blogging and it’s been a blast. Now it’s time for some prognostication. What will happen to NAND Flash memory next year? There are tons of clues. Here are some educated guesses:

2x nm will happen – Currently, vendors are shipping 34nm NAND Flash memory in volume. That’s a tremendous engineering feat in itself. As reported in an earlier blog entry (“Intel, Micron striving to regain lead in NAND tech”), Intel and Micron Technology are apparently planning to ship NAND Flash devices based on 2x nm lithography (called 2x because it’s not yet clear if it’s 26nm, 25nm, 24nm, or 22nm). Lithography shrinks are a true race to the bottom. Features on chips have become so small that one or two atoms of difference from one part of a chip to another cause real changes in device characteristics. This phenomenon is called on-chip variation or on-chip variability (OCV) and it’s a tough problem to tackle, requiring the use of smarter on-chip circuitry to deal with the variation. (See “My Head Hurts, My Timing Stinks, and I Don’t Love On-Chip Variation” by Matt Weber.) The problems do not appear to be insurmountable and NAND Flash vendors currently leading the lithography derby appear intent on keeping that lead until it’s no longer possible.

SDDs will get bigger and faster – This prediction needs to be written, but it’s really a no-brainer. The “bigger” part of the prediction is based on the ever-increasing capacity of the NAND Flash chips used to make SSDs. They will get bigger, driven by economic forces far beyond SSD usage. USB Flash-memory drives are the big volume driver in this market and there’s always demand for more capacity there while the form factor emphasizes small and slim. Nowhere to go but more on-chip capacity. Number two driver is SD cards for cameras and camcorders, with the same forces at work. As for faster, it’s clear that SATA 6G is in the immediate future for SSDs. Marvel’s SATA 6G controller (see “Early Results Show SATA 6G Performance All Over the Map”) and Micron’s introduction this month of an SSD that exploits SATA 6G to good effect (see “New SSD Introductions from Seagate and Micron”) clearly point the way to faster SSD operation, once the internal architectural designs are aligned with the faster interface.

ONFi 2.1 gets big – ONFi stands for the “Open NAND Flash interface” and the ONFi organization ( bills it as the fastest Flash interface on the planet. Hyperbole aside, ONFi supports transfer rates to 200 Mbytes/sec. and that’s speedy in anyone’s book. The ONFi Working Group was formed in May 2006 and currently has over 80 member companies including Hynix Semiconductor, Intel Corporation, Micron Technology, Inc., Numonyx, Phison Electronics Corporation, Sony Corporation and Spansion. Wanna fight those guys as a group?

MLC and TLC get big – It’s already happening. Several NAND Flash vendors already offer MLC (multi-level cell) and TLC (three-level cell) NAND Flash devices. They are working to improve the reliability of these devices while SSD and other subsystem manufacturers are working to develop system-level techniques to mask the reliability of these devices. That’s not a patch job. HDD vendors have spent the last 50 years paving over the data-reliability problems of rotating magnetic storage and no one pays much attention any more except the engineers tasked with keeping those problems at bay. Much the same will happen for NAND Flash devices and for products based on those devices.

NAND Flash prices firm up – NAND Flash prices have recently risen and a lack of capital investment in new fabs and processing equipment foretells the usual period of spot shortages and price peaks associated with scarcity. See MemoTrek’s extensive analysis: NAND Flash Prices: 4Q Trends & 2010 Forecast.

Joker’s wild – If we told you, then it wouldn’t be a surprise, would it?

Please have a happy and safe New Year’s celebration and we’ll see you next year.

Wednesday, December 30th, 2009 at 15:10

New SSD Introductions from Seagate and Micron

December has been a big month for new entrants in the SSD (solid-state disk) market. Two big-brand contenders that announced products this month are Seagate and Micron. SSDs represent new product lines for both of these companies. Seagate Technologies, started initially as Shugart Technologies by the godfather of storage Alan Shugart, essentially launched the small hard-disk drive (HDD) revolution in 1980 with the 5.25-inch ST506, which had a whopping storage capacity of 5 Mbytes. Micron is a leading semiconductor memory vendor and its NAND Flash chips are going straight into its SSDs.

Seagate announced its Pulsar SSD line on December 7 or 8 (depending on which version of the press release Google finds for you), allowing a show to drop that people had expected for more than a year. Pulsar drives use the familiar 2.5-inch HDD form factor and a SATA interface, making it easy to drop the drives into existing computer and server systems. Seagate’s Pulsar SSDs employ SLC (single-level cell) NAND Flash devices, which cost more per bit than MLC (multi-level cell) and TLC (three-level cell) NAND Flash devices. In exchange for the higher cost, you get more reliable memory, as was discussed in this blog a while back. (Check out “More than Moore: SLC, MLC, and TLC NAND Flash.”)

Seagate Pulsar SSD

The use of SLC NAND Flash underscores Seagate’s focus on enterprise-class storage for the SSD. There are at least two good reasons for Seagate’s enterprise focus. First, enterprise customers are more able to translate an SSD’s speed advantage over HDDs into dollars (as previously discussed in the blog entry “SSD TCO (Total Cost of Ownership”). Second, SSDs are a premium product with a premium price. Enterprise customers more easily accept the higher cost/Gbyte price tag attached to SSDs. Seagate’s Pulsar SSDs are available in storage capacities to 200 Gbytes and the SSDs achieve “a peak performance of up to 30,000 read IOPS and 25,000 write IOPS, 240MB/s sequential read and 200 MB/s sequential write” according to Seagate’s press release. The Pulsar drives have a 5-year limited warranty.

Micron Technology rolled out its RealSSD C300 less than a week before Seagate’s SSD announcement. The first glaringly obvious difference in Micron’s C300 SSD is that it sports a 6-Gbyte/sec SATA 6.0 interface. However, the faster interface alone will not boost performance (discussed earlier in this blog here) if the drive internals aren’t designed to sustain high transfer rates supported by SATA 6.0. To that end, Micron’s RealSSD C300 press release discloses the fact that the new Micron SSD “leverages a finely tuned architecture and high-speed ONFI 2.1 NAND Flash  to provide a whole new level of performance.” (ONFi, the Open NAND Flash interface, is discussed in this previous blog entry.) The result: a read throughput speed of up to 355MB/s and a write throughput speed of up to 215MB/s.

Compare those numbers to Seagate’s Pulsar and you’ll see that the Micron drive’s read throughput is nearly 50% faster but the write throughput is only 7.5% faster. Write throughput is one of the Achilles’ heels of SSDs. NAND Flash devices had an erase/write cycle that simply takes time.

Micron RealSSD C300

Micron’s C300 SSDs will be offered in 1.8-inch and 2.5-inch form factors, with both form factors supporting 128- and 256-Gbyte capacities. Micron is currently sampling the C300 SSD in limited quantities and expects to enter production in the first quarter of calendar 2010.

Both companies are making smart moves into the SSD market. Seagate, like Western Digital and its acquisition of SSD vendor SiliconSystems in March of this year, recognizes that it’s not in the HDD business—it’s in the storage business and SSD storage is hot right now. Micron, like Intel, sees SSDs as a value-added way to package and market it’s NAND Flash devices. Both companies have made very smart moves into the SSD market.

Wednesday, December 30th, 2009 at 02:34

Micron 1Q 2010 Financial Report and Other’s Forecasts Show Health in Semiconductor Memory Markets

Just before Christmas, Micron Technology (a major DRAM and NAND Flash vendor) announced some pretty positive financial results for the first quarter of the 2010 fiscal year. Micron had net income attributable to Micron shareholders of $204 million (23 cents per diluted share) on net sales of $1.74 billion compared to a loss of $100 million (a loss of 12 cents per diluted share) on net sales of $1.3 billion for the fourth quarter of fiscal 2009 and a loss of $718 million (a loss of 93 cents per diluted share) on net sales of $1.4 billion for the first quarter of fiscal 2009.

Profit can be a great indicator of industry health and Micron achieved these numbers with a mix of cost cutting and improved sales. In fact, DRAM and NAND Flash memory sales improved both in sales volume and in average sale price. Both are good news if these numbers can be extrapolated to the rest of the semiconductor industry. Quoting from Micron’s quarterly report:

“Revenue from sales of DRAM products increased 50 percent in the first quarter compared to the fourth quarter due to a 25 percent increase in sales volume and a 21 percent increase in average selling prices. Revenue from sales of NAND Flash products increased 21 percent in the first quarter compared to the fourth quarter due to a 16 percent increase in sales volume and a five percent increase in average selling prices. The company’s gross margin on sales of memory products improved from 12 percent in the fourth quarter of fiscal 2009 to 27 percent in the first quarter of fiscal 2010 due primarily to the increases in average selling prices.”

These numbers do seem to correlate well with the DRAM pricing graphs we published last week from DRAM Exchange (see “DRAM Exchange Calls for DRAM Shortage, Puts Lump of Coal in Your Stocking, Makes DDR3 Predictions”) and they track semiconductor consultant, gadabout, and blogger Daniel Nenni’s 2010 Semiconductor Forecast. (Nenni is a consultant and expert in strategic semiconductor foundry relationships and he’s a very active blogger for the semiconductor industry. His blog self-describes him as “intelligent, clever, charming, humble, and a pleasure to work with.” Gotta love a guy like that.)

Here’s what Nenni’s forecast had to say about the memory sector of the larger semiconductor industry:

“DRAM supply was already lagging demand when Windows 7 came out and [Windows 7] put even more pressure on PC and laptop users to upgrade. Micron Technology, the DRAM bellwether, is now profitable for the first time in three years. Expect increased pricing, long lead times, and continued DRAM allocation in 2010.”

AgigA Tech designs and sells bulletproof server memory DIMMs based on DRAM and NAND Flash devices. In a sense, we don’t like to see memory prices going up because there are implications with respect to component costs for our own products. But clearly, semiconductor memory companies need to stay in business, which means they must ultimately make a profit, so Micron’s latest results are clearly encouraging for the industry as a whole.

Tuesday, December 29th, 2009 at 17:51

DDR3 Clock Rate and Performance – Exhaustive Testing Results from Xbit Labs

Memory vendors offer four speed grades for DDR3 memory: DDR3-800, DDR3-1067, DDR3-1333, and DDR3-1600. Ever wonder how much affect SDRAM clock rate has on performance? Well, so did Ilya Gavrichenkov, the Hardware Editor at and co-founder of Xbit Laboratories. He’s just published an article titled “Choosing DDR3 SDRAM for LGA1156 Platform” with exhaustive tests on various speed grades of DDR3 SDRAM on an Intel Core i7-based platform. Now Xbit Labs has an overclocker orientation and is not really focused on server design but the results are instructive nevertheless.

Here’s the testbed that Gavrichenkov used for his tests:

  • CPU: Intel Core i7-860 (Lynnfield, 2.80 GHz, 4 x 256 KBL2, 8 MB L3)
  • Mainboard: ASUS P7P55D Premium (LGA1156, Intel P55 Express)
  • System memory: 2 x 2 GB, DDR3-1600 SDRAM (Kingston HyperX KHX1600C8D3K2/4GX, Corsair Dominator CMD4GX3M2A1600C8)
  • Graphics card: ATI Radeon HD 5870
  • HDD: Western Digital  VelociRaptor WD3000HLFS
  • PSU: Tagan TG880-U33II (880 W)
  • OS: Microsoft Windows 7 Ultimate x64

Using memory-intensive synthetic benchmarks, Gavrichenkov did observe some performance differences between DDR3-1067 and DDR3-1600 SDRAM. Although the DDR3-1067 clock and transfer rates are 37% slower than for DDR3-1600, Gavrichenkov observed a maximum of 18% performance difference between the two DDR3 SDRAM speed grades. Results from a multi-threaded synthetic benchmark called MaxMEM2 showed that DDR3-1600 SDRAM gave a maximum of 40% more performance than DDR3-1067 SDRAM, suggesting that multithreaded processor operations get more benefit from faster SDRAM transfer rates. Published results for non-synthetic video-transcoding, X.264 video-decoding, and file-compression benchmarks seem to verify the synthetic benchmark results, at least qualitatively. The Intel Core i7 processor does get some benefit from the faster SDRAM in the benchmarks based on real-world applications.

To some extent, these results should not surprise anyone familiar with the Intel Core i7 multicore processor architecture. The chip carries four multithreaded processors cores. Each processor core has private L1 and L2 caches and all four cores share a large, 8-Mbyte, 16-way associative L3 cache called the “last-level cache” or LLC. There are three SDRAM channels run by an on-chip Integrated Memory Controller (IMC), which manages the traffic between the LLC and the attached SDRAM.

The LLC serves as a huge buffer between the Core i7 processor’s multiple processor cores and the SDRAM channels and it makes sense that the LLC can damp down the performance differences among DDR3 SDRAM speed grades for single-threaded environments. That’s what a good cache does. It also makes sense that the buffering job gets harder when multithreading is involved because the memory accesses become less correlated and therefore too messy to cleanly cache.

Do the Xbit Labs’ results hold true in a server environment? Good question. The results suggest that server designers ought to be running tests of their own. At least for servers based on the Nehalem architecture (Intel’s Core i7, Core i5, Core i3, and Xeon processors), that big on-chip LLC could represent significant savings with respect to memory costs.

Monday, December 28th, 2009 at 22:53

Intel, Micron striving to regain lead in NAND tech

Here’s an odd little story about NAND Flash from Intel and Micron taken from Cyber India Online Limited, which credits the story to India’s CyberMedia News wire service. According to this story, Micron has announced that it’s about to start sampling NAND Flash parts based on a 2x nm process technology. Currently the most advanced processing node for Micron is 34nm. Micron and Intel have a joint NAND Flash venture called IM Flash Technologies LLC.

Both Intel and Micron are selling solid-state drives (SSDs) based on their most advanced Flash parts and so both companies have internal incentives to cut per-bit Flash costs as quickly as they can both to make their NAND Flash ICs competitive and to drive down the cost of their SSDs. Cost is now the biggest obstacle for SSDs in their quest to become mainstream storage devices.

Also according to this article, Micron Technology posted its first profit in 3 years during the period ending December 3, 2009 with sales worth $1.74 billion. The company’s NAND business rose 21% quarter over quarter and its DRAM business rose by as much as 50% during the same period. Good tidings indeed for Micron and the Flash business.

Thursday, December 24th, 2009 at 19:26

DRAM Exchange Calls for DRAM Shortage, Puts Lump of Coal in Your Stocking, Makes DDR3 Predictions

Just in time for the holiday, DRAM Exchange called a DRAM shortage today. Citing accelerating PC shipments, a trend since August, the DRAM Exchange has noted climbing prices for DDR2 and DDR3 DIMMs over the past several months. The following graphs from the DRAM Exchange tell the story of the pricing:

The reason behind the rising DRAM chip and module pricing was predictable by anyone who has followed the semiconductor industry for a decade or two. The last few years have been rocky for semiconductor memory vendors and whenever times are tough, these vendors know what to do to drive prices up: reduce capital expenditures, stop building memory fabs, and stop making so many memory chips. And that’s exactly what’s happened. It helps that in tight economic times, it’s relatively easy to forego the big capital expenditures needed to build new memory fabs or refit older fabs with new chip-making equipment.

DRAM Exchange provided a little fire to burn that coal in your stocking with the following heat-up-the-market predictions:

  1. Capital expenditures for DRAM vendors will increase 80% year over year to US$7.85B from US$4.30B in 2009
  2. DRAM aggregate demand will be slightly below aggregate supply in Q1 2010
  3. DRAM pricing will fall appropriately 10% to 20% quarter over quarter in Q1 2010

DRAM Exchange also made the following predictions for DDR3 DRAM in the coming year:

  1. DDR3 DRAM goes mainstream in Q1 2010
  2. DDR3 DRAM market share will account for 60% and will likely reach 80% of commodity DRAM by 2H 2010
  3. DDR3 DRAM prices will decline less than DDR2 DRAM prices given the strong platform migration momentum

Like Dickens’ ghosts of Christmas past, present, and future in “A Christmas Carol,” (go see the new movie!), these predictions from DRAM Exchange are merely shadows. Reality may or may not prove the predictions correct. Make your own decisions. TG Daily picked up DRAM Exchange’s predictions of a DRAM shortage for 2010 and one reader commented: “Prfft, they said the same thing last year.”

Thursday, December 24th, 2009 at 18:14

DDR3 and Server Memory Evolution

Semiconductor memory is always in a state of flux. New semiconductor memory technologies emerge, grow in popularity, and take over the lion’s share of the market. Older memory technologies hang around for a while and then slowly vanish as they’re supplanted by the new. How can you predict which technologies will succeed? Well, Marc Greenberg, Denali Software’s Director of Technical Marketing (whose tutorial on DRAM provided the graphic below), has a saying about the memory market: “Never bet against the market.” By that, he means that semiconductor vendors are always placing their bets on four major factors:

  • More density
  • More speed
  • Less cost
  • Less power

Depending on the specific application, one or two of these major factors may be more important than the other, but they’re all important factors—all of the time.

Currently, we have divided the use space for semiconductor memory into four big regions:

  • SRAM serves applications that require frequent, fast data access—more speed (usually cache)
  • DRAM serves applications that need large storage space—more density—for frequently changing data at a low price—less cost
  • NOR Flash currently serves the spot for holding code and data that must be accessed quickly—more speed—but doesn’t change often. You see NOR Flash mostly in smaller embedded applications because larger embedded applications, computers, and servers combine hard disk drives (HDDs) or solid-state disks (SSDs) with DRAM to serve the same function instead of NOR Flash.
  • NAND Flash is a story all by itself. As an industry, we use NAND Flash in a wide variety of ways. We use it to hold data and code in bulk because it’s the cheapest semiconductor memory available—less cost. At the same time, NAND Flash is non-volatile, so it’s useful for retaining information through power outages. That’s why SSDs are packed with NAND Flash chips and it’s also why AgigA Tech uses NAND Flash to back up DRAM in its AGIGARAM bulletproof Non Volatile System (NVS) memory modules. Even better, NAND Flash power consumption is fairly low—less power—if the system uses the NAND Flash memories infrequently, which is exactly how they’re applied in AGIGARAM NVS memory modules.

Memory Technology Inflection Points

The immense importance of memory in a processor-centric, multicore world results in tremendous technology R&D efforts to develop semiconductor memories that improve on one or more of the four major factors listed above. Memory storage technology and memory cell design get a lot of attention. One aspect of memory design that sporadically pops up in importance is the memory interface.

For some, the memory interface isn’t nearly as glamorous as a new kind of memory cell (think phase-change memory or PCM, which has held the limelight lately) or lithographic shrinks (think 32nm heading for 2x nm). However, the memory’s interface performance plays a major role in determining how a memory performs and even how much power it consumes.

In the world of NAND Flash, ONFi (the Open NAND Flash interface) and the Toggle-Mode NAND interface are coming to the fore. We’ll leave the discussion of these competing, high-speed NAND Flash interfaces for another blog post. Today’s topic is DRAMs. For DRAMs, the hot “new” interface is DDR3, which is the third major iteration of the JEDEC interface standard for synchronous, double-date-rate (DDR) DRAM.

The original DDR (double data rate) specification appeared in June, 2000 after a four-year gestation. The DDR memory interface replaced the original JEDEC SDRAM interface, which appeared in 1993. Before that, DRAM used the baroque RAS/CAS asynchronous control structure and multiplexed row/column address lines that Mostek developed for the MK4096 4-kbit DRAM in 1973 to reduce package pin count. That old RAS/CAS stuff is still there, deep inside of today’s advanced DRAMs, but it’s now buried inside of the DDR parts where you can’t see it unless you’re a DRAM chip designer.

DDR3 Memory’s Advantages

What are the advantages of DDR3 over DDR2? They go straight back to the four major factors listed at the beginning of this blog post. Compared to DDR2, DDR3 memory provides more speed, more density, operates at lower voltage (and therefore consumes less power), and it will be less expensive than DDR2 memory at some point in the coming year. In short, DDR3 memory improves on all four of the major factors relative to DDR2 memory. Bets don’t get much safer than that.

For enterprise-class systems (servers), DDR3 memory provides many specific advantages. First, it promises denser memory modules by accommodating DRAM chips as large as 16 Gbits, permitting the development of 16-Gbyte registered DIMMs. Enterprise-class server architects love denser memory modules because they’re always strapped for room inside of their server boxes. DIMMs take up space and, worse, they block air flow and make cooling more difficult inside of the enclosures. Fewer DIMMs is definitely better for air flow.

Second, DDR3 memory transfers twice as much data per clock as DDR2 memory. Enterprise-class server architects can use this speed in one of two ways. They can run their processors faster with faster memory or they can run at the same speed but cut the clock rate to the memory modules and thus cut power consumption.

Real Power Savings

But the real power savings comes from DDR3’s lower operating voltage. DDR2 memory is specified for a 1.8V operating voltage while DDR3 memory operates at 1.35V (and maybe 1.2V in future low-power DDR3L devices). Because operating power is proportional to the square of the operating voltage, that small 150mV drop between DDR2 and DDR3 operating voltages translates into an appreciable drop in operating power—about 30% less!

Enterprise-class server designers like lower operating power, therefore less waste heat. In fact, they like it a lot. That’s because data centers pay double for every excess Watt of server power. Roughly speaking, each Watt consumed by a server takes one Watt of electricity to run and another Watt to cool the server. By at least one estimate, DRAM power usage accounts for 25% to 40% of a data center’s energy costs (and can be more than 50% according to this Denali memory blog post). By another estimate, Google’s power costs were $50 million in 2006. So power reduction is very high on the server designers’ wish lists because data-center operators can easily translate reduced power consumption into monetary savings and they are quite aware of that sort of calculation with respect to total cost of ownership (TCO) when evaluating competing servers.

Perhaps the biggest force driving the adoption of DDR3 memory is the support of Intel and AMD. Intel’s Core i7 and AMD’s Phenom II multicore processors and chipsets presume DDR3 memory. It won’t take long before this presumption filters down to the lesser PC processors and PC processors are the big dogs in the memory kennel. They largely drive what happens with mainstream DRAM parts. So DDR3 memory’s success is likely assured, just as DDR2’s was before that and just as DDR memory supplanted SDRAM. The cycle repeats, and often.

Currently, AgigA Tech offers AGIGARAM NVS modules with SDRAM and DDR2 interfaces. It doesn’t yet offer an off-the-shelf DDR3 module, but given the industry’s track, you can safely bet that there’s a DDR3 AGIGARAM module on the road map.

Tuesday, December 22nd, 2009 at 21:33

SSD Performance Secrets

Compared to hard disk drives (HDDs), solid-state disks (SSDs) are fast. They’re an order of magnitude faster than the fastest enterprise-class HDDS in write IOPS and two orders of magnitude faster in read IOPS. What’s not to like? Well, just as HDDs deliver variable read/write performance depending on where the read/write arm is currently positioned relative to where it needs to be for the next read/write operation, SSD IOPS performance also varies—but in very complex ways. It’s nothing so simple as having the read/write head be in the wrong place at the wrong time. Although in a sense, that’s exactly what’s happening with SSDs.

Find those last two sentences confusing or contradictory? Here’s the explanation.

SSDs have no read/write heads or positioning arms. Instead, they consist of several NAND Flash chips and a controller chip. There’s an array of memory blocks on each NAND Flash chip. The size of the Flash memory block is the smallest amount of memory a NAND Flash chip can write in one operation because NAND Flash memory blocks are atomic with respect to erasure. You can’t write just one byte or word because you must erase the entire block before writing to the block. That means an SSD can only write an entire NAND block at a time.

Here’s a graphic from Jim Handy’s SSD keynote at the Bell Micro SSD seminar held in early December that helps to explain the situation:

SSD Block Organization

Each SSD consists of a stack of visible NAND memory blocks that the SSD controller uses to store written data. There’s also a shorter stack of spare NAND memory blocks that hold data in temporary storage. These spare blocks are also used to replace a visible block when it wears out from repeated write/erase cycles. All NAND blocks are equally accessible, so there’s no time penalty for writing NAND blocks out of sequence as there is when writing on non-adjacent or non-contiguous tracks with HDD storage.

However, most virtual operating systems don’t write in blocks, they write in 4-Kbyte pages that are much smaller than NAND Flash blocks. For example, Numonyx’ 1-to-16-Gbit NAND Flash devices have 128-Kbyte blocks. As a result, modifying one 4-Kbyte page in a NAND Flash block requires a relatively complex sequence:

  1. Read the data for the entire block from NAND Flash into a RAM buffer
  2. Modify the appropriate page in the block image now stored in RAM
  3. Write the block back to an erased NAND Flash block
  4. Fix pointers to the new memory block
  5. Erase the old memory block as a background task

Consequently, SSD performance varies over time and the performance varies depending on how many erased and spare memory blocks are available across all of the NAND Flash chips in the SSD. SSD performance also depends on the ratio of reads versus writes—because reads occur ten times faster than writes for SSDS—and they vary over time as the NAND Flash chips fill up.

The following figure from Handy’s keynote shows a 3D data surface plot representing the IOPS performance of one SSD. (The figure is from a presentation at the August 2009 Flash Memory Summit made by Esther Spanjer, Director of SSD Marketing at Smart Modular Technologies.)

SSD 3D Performance Surface

The X axis of the surface shows the ratio of reads to writes and varies from 100% writes on the left to 100% reads on the right. The Y axis shows SSD performance in IOPS. The Z axis plots “block” size, from the SSD-level perspective (which is page size from the NAND Flash chip’s perspective, yes that’s confusing).

The first thing to note from this surface plot is that performance is a lot better on the right-hand side, which is dominated by reads. You’d expect that because SSD read performance is 10x better than SSD write performance. It’s the nature of NAND Flash memory. Note how fast the performance falls off as the percentage of write transactions increases. Then note that there’s a sort of saddle effect along the Z axis. The saddle peaks at 4-Kbyte blocks. Most SSD designs are optimized for 4-Kbyte blocks because most virtual operating systems employ 4-Kbyte blocks (and have for decades, in spite of the radical, orders-of-magnitude increase in memory use by both operating systems and application software).

So, clearly, when an SSD vendor gives an IOPS rating for an SSD, you need to take that one number with a grain of salt. SSD performance varies significantly depending on the read/write mix and on block size. Consequently, SSD performance can’t be captured in one or two numbers.

Next, Handy presented this graphic from SandForce (which makes SSD controller chips):


This graph shows an initial conditioning period during which the test preconditions (fills up) the SSD using sequential 128-Kbyte writes. The initial transfer performance (about 80 Mbytes/sec for the particular drive being tested) drops slightly as the drive fills and the internal SSD controller starts shuffling full NAND Flash blocks off to spare memory. The falloff isn’t big because the sequential writes place a predictable load on the SSD controller. However, when the test switches to random 4-Kbyte writes about 4000 seconds into the test, performance drops significantly because the SSD controller suddenly needs to make small changes to memory stored in the NAND Flash blocks but the drive’s full and there are no empty blocks. Blocks must be erased to make room for the new data and block erasure takes time. Consequently, there’s a big performance falloff as the controller starts to shuffle data around inside of the drive to make room for new data.

Perhaps more interesting is what happens when the test switches back to large sequential writes about 11,000 seconds into the test. Initially, the sequential writes cause the drive performance to vary wildly because the preceding random writes have scattered the spare blocks and left them distributed throughout the SSD’s internal NAND Flash memory space. Eventually, the SSD’s internal controller gets things sorted out and the performance for large sequential writes returns to the initial steady-state level.

(Note: This graph is not supposed to typify the performance of all SSDs. The graph shows the results of a test on one particular SSD.)

So what’s to be learned from all of this data? SSD performance measurement isn’t simple. Creating controllers and firmware that deliver optimum SSD performance isn’t simple either. As drive and chip vendors learn more about the use of NAND Flash for storage, they develop better algorithms for extracting more performance from the NAND Flash chips.

NAND Flash chips are complicated, whether used in SSDs or for server memory backup as with AgigA Tech’s AGIGARAM modules. It takes experience to get the most performance from these memory devices.

My thanks to Jim Handy for all of the great information in his Bell Micro keynote, and for generously letting me use the information in this series of blog entries.

Friday, December 18th, 2009 at 18:52

SSD TCO (Total Cost of Ownership)

Back in November, this blog covered TCO (total cost of ownership) for blackout-proof server memory using NAND-Flash-backed AGIGARAM versus battery-backed DRAM. (See “Bulletproof Memory for RAID Servers, Part 2.”) Those numbers showed that NAND Flash provided a clear advantage over batteries in terms of TCO. Unsurprisingly, there’s a similar sort of situation with solid-state drives (SSDs) built from NAND Flash memory versus hard disk drives (HDDs). That NAND Flash memory is just really handy stuff. And speaking of handy, Jim Handy covered the topic of SSD TCO for servers in his Bell Micro seminar on solid-state disks (SSDs). He used an interesting calculation presented by Sun Microsystems at an Open Storage Breakfast that computed the crossover point where it made economic sense to use SSDs mixed with high-capacity HDDs rather than enterprise-class HDDs based on the IOPS required by the server design.

First, take a look at this graphic depicting a “typical” server storage array consisting of 100 enterprise-class HDDs.

Handy HDD Server Array Image

Each short-stroked, enterprise-class HDD has a capacity of 300 Gbytes, for a total array capacity of 30 Tbytes. This enterprise-class HDD array delivers 30K IOPS, costs $55,000, and consumes 1.75 kilowatts of electricity (not to mention an equivalent amount of electricity required for cooling). That’s the baseline.

Now look at this graphic, which compares the previously discussed array of enterprise-class HDDs with a hybrid array consisting of one SSD and 30 high-capacity, 1-Tbyte HDDs.

Handy HDD  Plus SSD Server Array Image

The high-capacity drives are not short-stroked, so they can provide a total storage capacity of 30T bytes with only 30 drives instead of 100 enterprise-class HDDs. However, the one SSD inserted into the drive array provides the same IOPS read performance as the 100 enterprise-class HDDs, so the use of the slower, less expensive, high-capacity HDDs in the second array is not a detriment to the second array’s IOPS performance, as long as the server software is written to make use of the hybrid array’s abilities.

The SSD-enhanced drive array costs $6040 or about 90% less than the array of 100 enterprise-class HDDs. The SSD-enhanced array consumes 0.392 kilowatts, which is nearly 80% less than the enterprise-class array of 100 short-stroked HDDs. Consequently, the second drive array generates substantially less waste heat (that must be cooled) than the full array of enterprise-class HDDs.

As a result, the SSD-enhanced drive array saves the enterprise customer a substantial amount of money when viewed from a systemic perspective. Relative to the enterprise-class HDD array, the SSD-enhanced hybrid drive array costs less to purchase; costs less to provision because fewer drives require less rack space and fewer racks; consume less electricity for operation; need less electricity for cooling because fewer, slower drives generate less heat; and reduce maintenance costs because the high-capacity drives run cooler (increasing MTBF), because there are fewer drives to maintain, and because high-capacity drives are much less expensive than enterprise-class drives. Overall, the TCO calculations favor the SSD-enhanced, hybrid drive array.

Handy took Sun’s numbers a step further by calculating the crossover point where TCO considerations favor an SSD-enhanced drive array over an array of enterprise-class HDDs when the IOPS performance is the main consideration rather than capacity. Here are Handy’s graphs:

Handy HDD and SSD TCO Graphs

The left graph shows price curves for an array of enterprise-class HDDs versus an array of SSDs. The array of SSDs initially costs more than the enterprise-class HDDs, so the crossover point is 1200 IOPS due to the high initial SSD cost. As the IOPS requirement rises, you need to add enterprise-class HDDs to the array to meet the higher IOPS requirements but one SSD gets you a lot of IOPS so there’s no need to add one until the IOPS requirement exceeds around 3000 IOPS. For the enterprise-class hybrid array, which mates one SSD with several high-capacity HDDs, the purchase cost of the SSD-enhanced array is much lower for a given capacity so the crossover point is also lower—just 400 IOPS.

TCO computations such as these are required for storage and for memory subsystems. It’s easy to be myopic and compare component cost to component cost, but system architects are creating systems and should always try to view component costs through a TCO lens.

Wednesday, December 16th, 2009 at 23:33

SSDs: Some Cold, Hard Numbers to Flavor Your Opinions

There’s nothing like cold, hard numbers to shore up an opinion. One of the slides that Jim Handy used in his Bell Micro seminar on solid-state disks (SSDs) compared what he labeled a “capacity HDD” (that’s a consumer-grade hard disk drive), an enterprise HDD (that’s a short-stroked speed demon), and an enterprise SSD. The one graph tells you a lot about how SSDs stack up against “rotating-rust” storage.

Handy HDD vs SSD Graph

As you can see from the graph, an enterprise-class SSD has more than twice the read bandwidth of an enterprise-class HDD and more than three times the read bandwidth of a “capacity” HDD. The enterprise-class SSD bests the enterprise-class HDD’s write bandwith by only 10%, reflecting the slow write/erase cycles of NAND Flash compared to the NAND Flash read latency.

The IOPS line in the graph is slightly misleading. While SSDs may perform 35,000 IOPS for reads, they’re about 90% slower for write IOPS. In reality, there’s a mix of reads and writes in any system, so real-life IOPS performance will fall somewhere between the extremes.

However, focus on the last line of the graph for a moment. This line describes latency and it’s here that SSDs truly shine due to their lack of mechanical components. A “capacity” HDD has a 15-msec latency, mostly due to the average amount of time required to move the read/write head from one disk track to the next. The short-stroked enterprise-class HDD has been restricted so that head seeks are radically reduced. The benefit is a nearly 10:1 reduction in latency, from 15 msec to 2 msec. The cost is a huge loss in disk capacity. Yet many enterprise system designers are willing to pay this cost to get the huge latency cut.

But look at that last number—the latency number for an enterprise SSD. There’s no mechanical head to move. There’s no disk platter to rotate into the right position. There’s just a large array of addressable NAND Flash blocks, all accessible in essentially the same amount of time. Latency is one of the biggest SSD benefits and it is one reason why server architects are beginning to intermix SSDs and HDDs to balance capacity with speed.

Wednesday, December 16th, 2009 at 20:09