Well, we may have recently avoided the ‘fiscal cliff’ in Washington DC, but some cliffs cannot be avoided. Just ask Norm of the TV show “Cheers” how many times he wished he could avoid Cliff Clavin. In the world of SSDs, the cliff that we want to avoid is the SSD “write cliff.”
Why the Write Cliff Matters
The SSD write cliff is the effect where SSD write performance drops off after all the free flash memory pages in an SSD have been initially written to and the device cannot provide enough free pages to keep up with subsequent write requests. Each new write request then requires the SSD locate a block that can be erased for the new data. If a block that needs to be erased contains active data, the active data must be written to a new location to free up the block to be erased. This process of copying valid data from one block to a new block, called ‘write amplification’ increases SSD wear and is the primary cause of the write cliff.
The SSD write cliff and ways to avoid it are discussed in depth in the VeloBit White Paper: SSD Performance Tips: Avoid The Write Cliff. Figure 1 shows an example of the write cliff observed in laboratory testing of a high performance, 320 GB PCI-E SSD at VeloBit. What this shows is that the SSD achieves maximum bandwidth (500 MB/s) until the SSD is filled with data (320 GB on the x-axis). However, once the device is full, the IO bandwidth rapidly drops off for additional write requests, hence the term ‘cliff’. In this test, the “__% size” lines represent the allowable address space for data being written to the device. We can see that bandwidth drops rapidly once the SSD is full and sustaining maximum IO is not possible.
Figure 1. SSD write cliff observed in lab testing
The TRIM Command Can Help
When a file is stored by an operating system and the OS “deletes” the file, the operating system file structure simply removes the file name from its file list and records that the disk space associated with the file is now available for writing new data into. The delete command does not, however physically erase/remove the data from the storage media. For SSDs, this means that a deleted file is still active in the SSD file mapping system and the memory cells used by the ‘deleted’ file are unavailable for use. As SSDs became more widely used, the problem of having ‘deleted’ files retained in valuable SSD storage became a big issue. The TRIM command was recently developed specifically to address this problem. TRIM is used to inform the SSD which file has been deleted and hence which blocks of data are no longer considered in use. The SSD can then make these blocks available for erasure/reuse, reducing write amplification and SSD wear.
TRIM Does Not Create World Peace
While the TRIM command is a valuable tool to be used to extend SSD life and improve performance, it really can only help postpone the write cliff somewhat. The SSD hits the write cliff when it runs out of free SSD data pages and starts encountering write amplification. When TRIM commands mark entire pages of data as ‘deleted’ before the SSD runs out of free data pages, the deleted page can be erased and written with new data without having to copy any valid data thereby reducing write amplification, increasing performance and prolonging SSD life. TRIM can help delay, but can’t eliminate the write cliff.
TRIM Requires New Software
So, good programming practices would include using the TRIM to help manage SSD life and performance. However, many existing applications and installations were written before TRIM was available. Using SSD caching software like VeloBit is an easier to deploy SSDs and properly manage them in legacy systems without having to rewrite applications.
File System Alignment for SSDs
There are many techniques and methods that can be used to tune an IT system to improve overall system performance. SSD caching is widely being used as a low cost, easy to implement method to quickly increase performance with little or no impact on existing operations. One thing that needs to be considered when choosing SSD caching is how to configure a file system to maximize the performance increase SSD caching provides.
Operating Systems and Block Size
In the past few years, the sector size of HDDs has increased from 512 bytes to 4k bytes. Relatively new operating systems (e.g. Windows Server 2008 R2 to present) have evolved to work with this not-so-new sector size to take advantage of higher density drives. Problems, however, can occur for older operating systems that were not designed to work with the larger block size devices. This becomes a critical issue when SSDs are introduced into the mix. SSDs are all designed to operate with a 4k block size (block and sector are used interchangeably). As a result, IT managers buying SSDs to improve system performance may not be getting the most out of their new, relatively expensive investment. Proper alignment of the file system may help to get better performance in this case.
Some Points to Consider Regarding Block Size and Alignment
Here are some things you should consider when using SSDs and modern HDDs in your system:
1. Block size – always use 4k block size
2. Block alignment – needs to be set to 4k. If the blocks are not 4K aligned, there is additional disk I/O overhead because 2x blocks need to be fetched for every access.
3. Partition alignment – Ensure that the starting partition is aligned to 4k. If your system was upgraded from pre-Windows 2008 R2 OS, make sure you re-align the partitions to 4k.
4. Memory allocation (Linux only - no control on Windows)
5. VMware environments – align all guest data disks on 4k boundaries
6. Hyper-V – align all guest data disks on 4k boundaries
Best Practices Results
Optimizing your system according to the above recommendations can improve performance up to 50%. Most modern OS file systems use these methods by default so there is no need to do anything. XFS and Windows versions earlier than Windows 2008R2 do not do this alignment automatically.
Velobit HyperCache caching software operates based on 4k blocks. If your existing system is not set up to work this way, Velobit will still work with the different block size making Velobit an ideal way to get performance improvement for older, legacy systems. However, the performance increase provided by Velobit will be limited due to additional overhead associated with managing non-4k block sizes.
At Velobit, we are lucky to be in the position to talk to many experienced SSD users to discuss real world issues with SSD integration and operation. We have just as many conversations with some not so experienced folks who are just beginning the investigation/search/selection process for SSD solutions. A topic that comes up frequently has to do with the cost difference between enterprise and consumer SSDs which we typically answer by discussing the differences between the two products. We will start with the basic cost question and work our way into the details from there.
Enterprise SSD and Consumer SSD Price Point
When I first started writing this blog, I thought it was going to be easy to differentiate these two categories by just going to a favorite vendor like newegg and get a few simple examples to show that enterprise SSDs and consumer SSDs can be defined simply by price. Well I was wrong about that. newegg.com actually has a category called ‘Enterprise SSD’ with 21 items in it. The list includes SAS, SATA and PCI-E SSDs ranging from $11.50/GB to $2/GB. newegg also has a category called ‘Internal SSD’ with 377 items (377 items! No wonder people are confused). This list also has different interface options and prices ranging from $4/GB down to less than $.60/GB.
So the categories are not easily defined just by price. With all the product options available, it is no wonder why we are frequently asked what the difference is between enterprise and consumer SSDs and why should you have to the spend extra money to get an enterprise SSD.
Thinking Inside The Box
We will have to go inside an SSD to discuss what the differences actually are in SSD products. Figure 1 shows a simplified block diagram of an SSD showing the internal components we need to discuss:
· CPU and SSD controller firmware
· Flash memory type and over-provisioning
· RAM
· Power supply (and backup)
SSD Controller Firmware
Every SSD has a processor inside which manages the device. The program the CPU runs is contained in the firmware shipped with the SSD. The right firmware can make all the difference in SSD performance. There are a limited number of flash memory vendors and lots of SSD vendors. One way to differentiate product performance is in the firmware algorithms that control the write access to the flash. For example, we tested an OCZ Vertex 4 SSD with firmware version 1.3 and 1.5 to see what difference upgrading to the newer firmware version would make. We found that (see OCZ Vertex 4 low performance with firmware 1.3) the newer firmware actually boosted performance up to 500% for some tests.
How do you see the impact of firmware when buying enterprise vs. consumer SSDs? Consumer grade SSD firmware tends to be optimized for a ‘read’ heavy workload. Enterprise grade SSD firmware is optimized for the mixed read/write environment of a typical IT application. Thus the consumer firmware performance is not optimal for enterprise and therefore, consumer SSDs are not really intended for enterprise use.
Figure 1. Simplified SSD Block Diagram
Source: http://www.csee.umbc.edu/~squire/images/ssd1.pdf
Flash Memory Type and Over-provisioning
There are 2 different types of flash memory available from chip vendors: Single-Level Cell (SLC) and Multi-Level Cell (MLC). Tons of stuff has been written about these topics so we will just keep it at a high level here. Basically, SLC is faster, uses less power and can handle more write cycles than MLC. MLC is cheaper and has higher data density. Does SLC or MLC define an enterprise SSD vs. consumer SSD? No, again, it is not that simple, although all consumer SSDs are MLC. Many enterprise SSDs use MLC technology and through their firmware, they manage the write cycle problem.
The amount of flash installed in an SSD does differ between enterprise and consumer devices. This is referred to as “over-provisioning”. Simply stated, this means is that the SSD contains more physical flash memory than is stated in the SSD specification. For example, an SSD with 128 GB of physical flash can be specified as either a 100 or 120 GB SSD (28% and 7% over-provisioned, respectively). This is done to help reduce wear on the flash memory chips themselves making the SSD last longer. So, an enterprise SSD may be specified as smaller size resulting in higher $/GB.
There is a second level of over-provisioning that is typically a user setting when formatting the SSD. Users can decide to set aside a percentage (e.g. 20%) of the specified space for over-provisioning as a trade-off between reduced storage capacity and increased endurance and performance. That can typically be done for either enterprise or consumer SSD.
On Board RAM
SSDs typically use RAM cache to manage the data flow to/from the flash devices. The size of the RAM on the SSD varies from 16 MB on the low end consumer SSDs to up to 1GB on the newest enterprise SSD. More RAM costs more but produces higher performance SSDs.
Power Supply
Because RAM is used for SSD data management data loss can occur in the event of a system-wide power loss. Robust SSD power supply design is important for data preservation. Enterprise SSDs can have large capacitors and battery backup built into their power supplies to preserve data. Consumer-grade SSDs do not have the same level of data loss prevention as enterprise devices.
Conclusion
It is understandable why there is so much confusion regarding consumer and enterprise SSDs. Basically, enterprise SSDs are designed to perform better, last longer and be more reliable for critical data operations. These added benefits results in enterprise SSDs costing more than consumer grade devices. Is it worth the extra money to buy an enterprise SSD? That is a difficult question to generalize upon, but I would have to say that extra benefits of using an enterprise SSD have significant value and if you can afford one, get one. While a consumer grade SSD will still provide you with significant performance benefits, its lower durability make it unsuitable for the data center. Deploying the SSD as a cache is most efficient and effective way to use this valuable asset and will almost always guarantee you the best price/performance.
This blog begins a series of blogs which present test results of testing performed at Velobit to investigate the effect of block sizes and outstanding IO requests on IO bandwidth when SSDs are operated in a simulated enterprise environment. This series of blogs is similar to the series we previously presented documenting IO bandwidth vs. read/write ratio. This first blog will document the test set up and procedures and present the results from one tested device. In following weeks, I will refer the reader back to this blog for test set up and procedure information and simply present the results for another device. After presenting results of a particular week, I will make observations as appropriate. Test Motivation and ProcedureFor this group of tests, we wanted to focus on the effect of the number of outstanding IOs and request size on SSD IO bandwidth. We are particularly interested in observing the impact of increasing the number of concurrent IOs and request size. We want to see which of the factors has the biggest effect on IO bandwidth and which workload type. Because of the internal parallel structure of SSD we anticipate that there should be performance benefits from sending multiple requests to the SSD. SSDs should be able to serve multiple small random or sequential requests simultaneously. SSDs also can break big sequential requests into small requests to take advantage of their internal RAID-like structure. We tested 8 different SSDs from 6 different manufactures. Some are SATA devices and some are PCI-E devices. We used a Linux based system running with the Intel Open Storage Toolkit, a tool similar to Iometer that runs on Linux. The Open Storage Toolkit generates synthesized workloads with various sizes and queue lengths and also provides monitoring and IO tracing capabilities. We also used another set of tools available in Linux, blktrace and blkparse that intercept all block-level requests. The idea was to observe IO bandwidth under operating conditions. So for this test, we ran an identical test suite for each SSD and observed IO bandwidth. We used 6 different IO request sizes: 1 KB, 4 KB, 16 KB, 64 KB, 128 KB and 256 KB and varied the outstanding IO requests from 0 to 64. We ran 4 different sets of these tests based on IO type: random read, sequential read, random write and sequential write.General ObservationsIn general, we observed that the largest IO request sizes generally provided the maximum IO bandwidth. Increasing the number of outstanding IO requests did increase IO bandwidth. In many cases a relatively small amount of outstanding IO requests would allow the SSD reach the maximum bandwidth for any particular request size. Test SSD: 250 GB OCZ Z Drive PCI-E based SSD
Figure 1: IO Bandwidth for OCZ Z Drive: a) random read b) sequential read
Figure 2: IO Bandwidth for OCZ Z Drive: a) random write b) sequential write Figures 1 and 2 shows the results for the OCZ Z Drive. For all the four workload types, the bandwidth increases as the number of outstanding IO increases, which indicates that there is a significant amount of parallelism within the device. However, the benefit of increasing the number of outstanding IO is highly dependent on the request size. When the request size is 1 KB or 4 KB, the benefit of increasing the number of outstanding IO is very limited. When the request size is 16 KB, 64 KB, 128 KB and 256 KB, the performance increases very quickly with just a small increase in the number of outstanding IOs. It appears that the OCZ Z drive is quite sensitive to the number of outstanding IO for some test cases. For random read workloads, the SSD performance continues to increase even when the number of outstanding IOs is more than 32. This indicates that the high end PCI-E based OCZ Z drive can process as many concurrent requests before reaching its performance limit. Come back next week for results and observations on another SSD.
Posted by
IT Dog on Mon, May 28, 2012 @ 08:06 AM
Posted by
Qing Yang on Mon, May 21, 2012 @ 08:00 AM
- Application layer: Applications can be developed to specifically address the SSD write concern by simply minimizing write to the SSDs. In this case, an application can consolidate writes (sometimes called write coalescing) and reduce write amplification.
- Data Deduplication: Data deduplication is a data reduction technique that simply removes duplicate data from the data stream. This means that it only removes data that is exactly like a block (or “chunk”) of data previously stored. This reduces the amount of data written into an SSD.
- Data Compression: Data compression is a technique where the original data is encoded into another representation of the data. The subsequent encoded version of the data uses less bits to represent the original data (e.g. music mp3 files). There are many forms of data compression. Some retain all of the original data exactly (lossless compression) and some (like mp3) do not retain all of the original data but exhibit some loss in the data representation (lossy compression). In either case, the goal is to reduce the size of the data set being written into SSD.
- Collect statistics: Algorithms can be run to monitor data blocks to classify blocks by groups based on write frequency. Once information is collected on the write characteristics of the data, the algorithm can combine blocks with similar write frequency together. When the time comes to erase a block, chances are that entire block is obsolete at the same time so no valid data is required to be moved to a new block. In this case, garbage collection and hence write amplification are decreased.
- Use SSD as cache: Using SSD as cache allows caching algorithms to be used to analyze and select data to be written into cache. Cold data can be identified before it is ever written into the SSD, thereby eliminating unnecessary SSD writes.
ConclusionNo matter how you use SSDs in your IT system, be sure to understand the implications of excessive writing to the SSD and design or use software that effectively manages the SSD to reduce write amplification and premature wearing of the device.
Posted by
IT Dog on Mon, Apr 16, 2012 @ 05:22 PM
The IT Dog is waggin’ his tail today with this one. I love progress and the SSD revolution is certainly pushing the storage industry forward on many fronts. New products with SSD in every segment of the IT data chain from the server side SSD to SSD raid storage. SSD capabilities has so radically changed the performance parameters of data IO that long standing data interface standards need to be updated to take advantage of the true benefits SSDs offer. Today’s standard for discussion is SATA.
SATA Yesterday and Today
Sounds like the name of great conference somewhere in the mid-west in January that you would like to attend, right? Well, I just wanted to give a little snapshot of where SATA is before I talk about what is next. SATA stands for - Serial Advanced Technology Attachment (must have been a fun meeting in Santa Clara when they came up with name). SATA is a computer bus interface design specification for connecting bus controllers to hard drives. It has been around since 2002 and has been incrementally changed over the years. Version 1.0 was specified to run at 1.5 Gb/s. The current version (from 2009) is 3.0 and it is specified to run at 6.0 Gb/s. Now you can guess what the problem is.
SSDs Too Fast for SATA
I can hear you saying, “but Dog, 6 Gb/s sounds pretty fast”, and it is. But, it does not go fast enough. Current SSDs already approach this speed and the SATA interface is becoming a bottleneck. Also SATA need to go faster to match the performance PCI-E based SSDs are already running at. So, not to be left behind, the good folks at SATA International Org. have been working on the new, souped up version of SATA. And because SATA did not want to let PCI be the only ones with an “Express” specification (PCI Express), the SATA Org guys called theirs – wait for it, – SATA Express!!! Calm down now. I know I used three exclamation points but the name really is not that exciting.
SATA-E Is Coming To A Product Near You
So, the SATA Express specification is in the works. It was announced fall 2011 and was going to be ready by “end of 2011” but as far as I can tell, 2011 is ended and I have not seen any info about the spec being released. But the release date really is not that important here. What is important is the new SATA Express is being developed to take advantage of high speed SSD devices by supporting both 8 Gb/s and 16 Gb/s transfer rates. A new generation of SSDs will be coming soon to take advantage of these higher transfer rates and further expand the performance lead that SSD based hardware and software products have in the marketplace.
Cool Or Hot?
So, is the new SATA standard cool or hot? What would you have called the new SATA standard? (Hint: don’t suggest “Express”)
Posted by
Jack Huang on Wed, Mar 14, 2012 @ 09:40 AM
Posted by
IT Dog on Mon, Feb 27, 2012 @ 08:17 AM
Posted by
IT Dog on Mon, Feb 13, 2012 @ 08:00 AM
SSD optimization thoughts from The Dog today. I was dreaming about SSD optimization and SSD optimization software yesterday while laying in the warm sun and thought I would get more out of the dream if I wrote some of this stuff down.
What You Should Know When You Buy SSDs
“What’s all the talk about warm sun Dog, it is the middle of winter where I am!” I did not say I was outside in the warm sun. You just need to find a good window to hang out near and be patient. The sun will come to you!
The idea behind SSDs is pretty simple. Fast performance, low power, great form factors and dropping prices. What is not to like? Well, there is some baggage that comes with buying SSD that may intimidate some managers. Things like understanding how an SSD may change your implementation architecture or will your existing software plug and play nicely with your new investment. Here are some SSD optimization points to focus on when evaluating SSD solutions:
- Performance - You are spending bug bucks on SSD to improve your system performance. True, SSDs are fast, but the software/application using it must be optimized to work with the physics of SSD to keep them fast or you will be just throwing that money away.
- Ease of deployment - You have a lot of time and money invested in your existing applications, backup and data protection systems. You don’t want the installation and use of SSDs to disrupt your money making business operation.
- Cost - We all know SSDs are still expensive relative to HDD so you must use SSD efficiently. This means deploy them as cache and use SSD cache software to selectively cache only a portion of data: the data that will maximize performance improvement. Use standard storage for other data.
- SSD management - Optimize how data is managed on the SSD. This means be very aware that SSDs are great for reading but writing is slow and inefficient. Writing to SSDs can trigger erase commands which shorten the life of an SSD. Manage the SSD to avoid the write cliff.
- Application awareness - Most applications are built upon data pattern assumptions of traditional storage systems. The application developers aren’t aware of special considerations when using SSD. As a result the applications the don’t use SSDs optimally. This impacts performance and SSD durability. SSD optimization software like VeloBit serves as an application translation layer so applications don’t have to be care about the storage type used.
Ready To Go Deeper?Well, I feel better now, getting all this SSD optimization stuff down. You can get more information on SSD optimization software by reading Velobit’s Ebook “SSD Caching: Boost Storage I/O Speed and Application Performance by 10X Through Content Locality Caching.”