ZFS – Downgrade Commands for Centos:

To do the downgrade/re-install dance:

ZFS tuning for SAS3 Storage Tank

ZFS tuning for SAS3 ALL SSD + nVME Storage Tank

Some more good advice:



ZFS – SNMP Extend Commands:


SSDs (Solid State Drives) use NAND flash chips. Each of these chips contain millions of cells with limited number of write cycles. There are different types of NAND flash chips in use today with different characteristics as follows:

SLC (Single Level Cell)

– highest performance, at a very high cost, enterprise grade NAND
~ 50-100k P/E (Program/Erase) cycles per cell, highest endurance
– lowest density (1 bit per cell, lower is better for endurance)
– lower power consumption
– faster write speeds
– much higher cost (3+ times higher than MLC)
– good fit for industrial grade devices, embedded systems, critical applications.

eMLC (Enterprise Multi Level Cell)

– good performance, aimed at enterprise use
~ 20-30k P/E cycles per cell, great endurance
– high density (2 bits per cell)
– lower endurance limit than SLC, higher than MLC
– lower cost
– good fit for light enterprise use and high-end consumer products with more disk writes than MLC.

MLC (Multi Level Cell)

– average performance, consumer grade NAND
~ 5-10k P/E cycles per cell
– higher density (2 or more bits per cell)
– lower endurance limit than SLC/eMLC
– lower cost (3 times lower than SLC)
– good fit for consumer products. Not suggested for critical applications which require frequent updates of data

TLC (Three Level Cell)

– lower performance, lowest cost NAND
~ 1-5k P/E cycles per cell
– highest density (3 bits per cell)
– lower endurance limit than MLC and SLC
– best price point (30% lower than MLC)
– somewhat slower read and write speed than MLC
– good fit for lower-end consumer products. Not recommended for critical applications which require frequent updating of data

3D Vertical NAND

– newer TLC NAND with different architecture, larger geometry and much higher endurance than TLC
~ 1-10k P/E cycles per cell
– faster and more reliable than TLC
– uses less power than TLC
– performance and P/E cycles comparable to MLC

Generally, SLC drives are traditionally the fastest, most reliable and most expensive drives available, usually used in the enterprise because of their considerably higher cost. Both MLC and TLC are widely used consumer grade memory, with MLC being better in terms of endurance. Newer 3D NAND TLC is comparable in performance to MLC drives, with even better price point.

The move from 34nm to 25nm (and then to 19nm) has generally reduced the NAND PE lifespan – on 34nm each cell was rated for 5-10k PE cycles, on 25nm this drops down to 3-5k. On 19nm TLC NAND it varies a lot, but it can drop down to as low as ~1k estimated PE cycles.

NAND Endurance ratings for common/older SSDs:

  • ADATA (S511) – 5k PE cycles, 25nm
  • Corsair (Force 3/GT) – 3k PE cycles, 25nm
  • Corsair (Performance 3) – 5k PE cycles, 32nm
  • Crucial (M4) – 3k PE cycles, 25nm
  • Intel (SSD 320) – 5k PE cycles, 25nm
  • Intel (SSD 510) – 5k PE cycles, 34nm
  • Kingston (HyperX) – 5k PE cycles, 25nm
  • Mushkin (Chronos) – 5k PE cycles, 32nm
  • OCZ (Agility 3, Vertex 3, Solid 3) – 3k PE cycles, 25nm
  • Patriot (Pyro) 3k PE cycles, 25nm
  • Plextor (PX-128M2S/P) 5k PE cycles, 32nm
  • Samsung (SSD 830) 5k PE cycles

Larger TLC drives may yield similar longevity as smaller MLC drives, considering you can average out the wear over higher number of cells.
Over-provisioning (OP) is sometimes used to increase drive endurance by setting aside free space that is inaccessible by the user for controller swap space.
To improve SSD endurance, one can leave at least 10-20% of free space to simulate over-provisioning. This can also be achieved with formatting to a lower capacity.
Even with low PE cycles NAND, endurance of most modern SSDs is over 10 years of typical use.

My current SSD favourites for a ZFS tank:

  1. HGST Ultrastar SS300 SAS SSD
    Be aware there is a MLC NAND (HUSM*, higher IOPs, higher write throughput) and a TLC NAND (HUST*, lower IOPs, lower write throughput) version
    Product ID: 0B34897 (HUSTR7696ASS200)
  2. Toshiba PX05 960GB
    Product ID: PX05SRB096
  3. HGST Ultrastar SS200 SAS SSD
    Product ID: 0TS1396


ZFS – Single Path

ZFS – Multi Path