Bulletproof Memory for RAID Servers, Part 1

Envision a data center with row upon row of rack-mounted RAID servers. All of these servers have battery-backup units for their RAM caches but buried somewhere deep inside this maze of racks, there’s a battery years past its prime. Perhaps there are several such batteries. These batteries are supposed to be changed out annually, but you know how things go. Sometimes, preventative maintenance just doesn’t happen on time. Or at all.

In fact, one of those batteries has failed. The RAM cache it protects is at risk when the next power outage occurs. When that happens, one or more of the data center’s customers will lose data. Critical data. After all, what data isn’t critical?

Worse, the failed battery is leaking. Acid is oozing out of the battery. It’s quite possible that the acid is leaking onto critical circuitry inside of the RAID enclosure. Drip. Drip. Drip. The acid starts to etch into the circuitry. The disaster is perhaps moments away…

Customers buy one thing from RAID vendors: a safe haven for their bits. The bulletproof aspect of a RAID array’s disk storage resides in the redundancy of the disk drives themselves. A RAID 5 array protects against data loss should one disk drive fail and a RAID 6 array protects against faults should two drives fail. Both types employ disk striping with parity (double parity for RAID 6). Because data has value—and some data has tremendous value—the use of RAID systems based on hardware RAID controllers is skyrocketing. However, power loss can negate the efficacy of a RAID system and puts the data at risk.

One critical point of failure in RAID systems with respect to power outages is the write cache. RAID systems employ write caches to speed disk transactions—to boost the IOPS (I/O operations per second) rating. Once a computer system squirts a chunk of data into a RAM-based cache, the RAID system can immediately acknowledge the transaction before actually writing the data to disk. So there’s a critical period of time when the data is at risk from a power failure, after the acknowledgement but before the data is on the disk. If power is lost while the data is in RAM cache, then it’s lost forever.

One way to avoid this problem entirely is to disable the RAID system’s RAM cache. This approach preserves the data but with a huge performance hit. No RAM cache, no performance.

Another way to avoid the problem is to protect the data in a write cache from power failures using a battery-backup unit (BBU). That way, the RAID controller can recognize an impending power failure, can halt transactions, and the BBU will maintain any data yet to be written to disk and thus ride through the power failure.

Sounds great in theory, but in practice there are many problems with BBUs:

  • Batteries have short, finite lifetimes compared to other electronic components and heat further shortens their electrochemical lives. There’s heat aplenty inside most server enclosures. Consequently, battery health should be closely monitored but it’s often not monitored at all. In fact, some data-center operations teams are surprised to discover that there’s a high-maintenance battery inside of many RAID systems. Of course, by the time they realize that there’s a battery to be maintained, it’s often too late because the event that brought this fact to light was a data failure induced by power loss.
  • Batteries need to be replaced every one to two years. First, that’s not going to happen if no one knows there’s a battery to be replaced. Second, battery maintenance often falls pretty low on the priority list of tasks to be performed and the replacement may be dangerously deferred when it’s done at all. Third, there’s no standardization in BBUs so the correct battery pack may not be on hand. Worse, the required BBU may be discontinued, no longer be available. If you can’t order a new one, then what? Fourth, battery packs cost money and so does the time it takes to install new ones.
  • When replacing the BBU, the RAID server must be taken offline, or at least the RAM cache needs to be taken off line and it must stay off line until the BBU charges up. RAID performance suffers during the downtime. Consumer-level products such as PCs and PVRs (personal video recorders) may not benefit much from faster disk drives. Enterprise systems do. Enterprise computing clients know precisely what a second’s worth of delay costs in their business. Sometimes a microsecond’s delay costs big money. For example, Google and Amazon know to the penny what each additional second of response delay costs them in terms of lost customer purchases. High-frequency securities traders and arbitrage houses employ trading strategies that are highly dependent on ultra-low latency networks. In fact, they co-locate their trading servers with the trading floor to minimize communications latency with the computers at the market exchange. These traders profit only by feeding information on competing bids and offers to their trading algorithms microseconds faster than their competitors. Loss of write-cache performance in a RAID system could literally cost such traders millions of dollars per microsecond of delay.
  • Batteries are not environmentally friendly so it’s a bad idea to just toss them in the trash. Batteries should be properly recycled and proper recycling is expensive, beyond the cost of the replacement BBU. Even when recycled properly, batteries just aren’t that great for the environment.

So what’s the right answer to the need for bulletproof RAID write cache? AgigA Tech believes that the answer can be found in a fusion of NAND Flash and ultra-capacitor technologies. Ultra-capacitors are essentially made of benign carbon and have many superior qualities compared to batteries. In particular, they charge faster (less downtime) and they have longer lives (when properly applied). NAND Flash can save a RAM cache’s contents indefinitely and without power. So AgigA Tech’s AGIGARAM modules can be used as RAID RAM-cache modules, providing all of the benefits of battery-backed write caches but without the many liabilities batteries incur.

What about the cost of such an approach? Stay tuned. We’ll address that in the next blog entry.

Thursday, November 12th, 2009 at 23:26
No comments yet.

Leave a comment

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>