Installation ZFS RAIDZ settings defaults

April 17 2013

Fresh SmartOS installs automatically choose default RAIDZ settings for ZFS filesystems vary based on the number of disks you add to the ZPOOL when you run the installer. The installer sets up the system so fast you may not realize what it did, so here I show a few different systems and what RAIDZ defaults were chosen. 

But first, a reading from the the book of man zpool:

       
     raidz
     raidz1
     raidz2
     raidz3
A variation on RAID-5 that allows for better distribution of parity and eliminates the "RAID-5 write hole" (in which data and parity become inconsistent after a power loss). 

Data and parity is striped across all disks within a raidz group. A raidz group can have single-, double-, or triple parity, meaning that the raidz group can sustain one, two, or three failures, respectively, without  losing any data. 

The raidz1 vdev type specifies a single-parity raidz group; the raidz2 vdev type specifies a double-parity raidz group; and the raidz3 vdev type specifies a triple-parity raidz group. The raidz vdev type is an alias for raidz1. 

A raidz group with N disks of size X with P parity disks can hold approximately (N-P)*X bytes and can withstand P device(s) failing before data integrity is compromised. The minimum number of devices in a raidz group is one more than the number of parity disks. The recommended number is between 3 and 9 to help increase performance.

The choices the SmartOS installer makes behind the scenes are reasonable, but can be altered if you wish to reconfigure after install, as the operating system is not on the physical drive set.

OK, here are the defaults as installed:

Tiny - Single SATA 250Mb HDD - 
ZFS no RAID

[root@e0-cb-4e-0a-84-c1 ~]# zpool status
  pool: zones
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        zones       ONLINE       0     0     0
          c1t0d0    ONLINE       0     0     0

errors: No known data errors
[root@e0-cb-4e-0a-84-c1 ~]# zpool list
NAME    SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
zones   232G  4.32G   228G         -     1%  1.00x  ONLINE  -


Small - 2x Matched SATA 1Tb or 2Tb HDD - MIRRORED (like RAID1)

1 Tb Drives:
[root@2c-27-d7-2e-93-55 ~]# zpool status
  pool: zones
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        zones       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            c1d0    ONLINE       0     0     0
            c2d0    ONLINE       0     0     0

errors: No known data errors
[root@2c-27-d7-2e-93-55 ~]# zpool list
NAME    SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
zones   928G  4.32G   924G     8.44E     0%  1.00x  ONLINE  -

2 Tb Drives:
root@3c-d9-2b-50-d3-d9 ~]# zpool status
  pool: zones
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        zones       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            c1d0    ONLINE       0     0     0
            c2d0    ONLINE       0     0     0

errors: No known data errors
[root@3c-d9-2b-50-d3-d9 ~]# zpool list
NAME    SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
zones  1.81T  4.32G  1.81T     1.16E     0%  1.00x  ONLINE  -
 
Large - 16x Matched SAS 3Tb - 
RAIDZ1 (Like RAID5)
[root@00-25-90-93-a3-2a ~]# zpool status
  pool: zones
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        zones       ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            c0t1d0  ONLINE       0     0     0
            c0t1d1  ONLINE       0     0     0
            c0t1d2  ONLINE       0     0     0
            c0t1d3  ONLINE       0     0     0
            c0t1d4  ONLINE       0     0     0
            c0t1d5  ONLINE       0     0     0
            c0t1d6  ONLINE       0     0     0
            c0t1d7  ONLINE       0     0     0
            c0t2d0  ONLINE       0     0     0
            c0t2d1  ONLINE       0     0     0
            c0t2d2  ONLINE       0     0     0
            c0t2d3  ONLINE       0     0     0
            c0t2d4  ONLINE       0     0     0
            c0t2d5  ONLINE       0     0     0
            c0t2d6  ONLINE       0     0     0
            c0t2d7  ONLINE       0     0     0

errors: No known data errors
[root@00-25-90-93-a3-2a ~]# zpool list
NAME    SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
zones  43.5T  5.68G  43.5T         -     0%  1.00x  ONLINE  -



Un-Mirror: Simple zpool Example
For an example of reconfiguring SmartOS I un-mirror-ed the default zone in my system with 2x 1Tb disks, removing one disk from the mirror for more space, and adding it back without mirroring, so it presents 2Tb without redundancy.

Starting state:
[root@2c-27-d7-2e-93-55 /]# zpool status
  pool: zones
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        zones       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            c1d0    ONLINE       0     0     0
            c2d0    ONLINE       0     0     0

errors: No known data errors
[root@2c-27-d7-2e-93-55 ~]# zpool list
NAME    SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
zones   928G  4.32G   924G     8.44E     0%  1.00x  ONLINE  -


Detach the second mirrored drive (vdev c2d0):
[root@2c-27-d7-2e-93-55 /]# zpool detach zones c2d0
[root@2c-27-d7-2e-93-55 /]# zpool status
  pool: zones
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        zones       ONLINE       0     0     0
          c1d0      ONLINE       0     0     0

errors: No known data errors
[root@2c-27-d7-2e-93-55 /]# zpool list
NAME    SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
zones   928G  4.32G   924G     8.44E     0%  1.00x  ONLINE  -


Add the drive back to the zones zpool:
[root@2c-27-d7-2e-93-55 /]# zpool add zones c2d0
[root@2c-27-d7-2e-93-55 /]# zpool status
  pool: zones
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        zones       ONLINE       0     0     0
          c1d0      ONLINE       0     0     0
          c2d0      ONLINE       0     0     0

errors: No known data errors
[root@2c-27-d7-2e-93-55 /]# zpool list
NAME    SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
zones  1.81T  4.32G  1.81T     8.44E     0%  1.00x  ONLINE  -

So - two commands to change the default to un-mirrored and full use of the available disk space without redundancy.


Comments