I’ve been using FreeNAS for over a year now and it is an awesome piece of software to turn an old PC into a Network Attached Storage device.  I’ve been running it on two old PCs, one full of old TV Shows and Movies I ripped from DVD’s to the hard drive.  This is for the boys, I was getting sick of them leaving piles of discs on the floor in front of the DVD player.  They were getting more scratched all the time so I backed them up to the FreeNAS server and they use XBMC (xbox media Centre) on another PC hooked up to the TV.

The other FreeNAS box has all my important files I’ve collected over the years.  But my most important files are backed up on my Drobo, which is awesome!

The Drobo is another NAS that is hooked up to my Mac Mini and is all set to share upto 16 Terabytes of data (whenever my Drobo is expanded to that size).  The Drobo is cool because it can be expanded by replacing the smaller drives in it with larger ones and it will keep growing.  Currently I have four 1 Terabyte drives in my Drobo which gives my about 2.7 Terabytes of protected storage.  If one of the drives in the drobo dies my data is still ok and all I have to do is replace the bad drive and away I go…

Now the problem is when I upgrade the drives in the drobo I’m left with spares that need to be used!  So I’ve decided to get into the Nightly build FreeNAS server which supports ZFS!

ZFS is an amazing file system that can be used as a wrapper around many pools of vdevs of drives.  The vdevs can be configured in single or RAID combinations and in Raid 5 & 6 can even be expanded similarly to a Drobo!  But unlike the Drobo Raid 5 & 6’s like all the drives to be the same size otherwise you waste a lot of storage space.

So I thought I might as well use my spare drives configuraed as JBOD (just a bunch of drives) but FreeNAS (official release 0.69) doesn’t support the size of the JBOD properly and you don’t get the use out of them.  So I decided the best choice is to use the nightly build and stripe the drives together within a vdev under ZFS.  Another bonus of ZFS is it does tests for bitrot and ensures your data is maintained.  The only real problem I see if that ZFS isn’t very mature yet.  It was written for Solaris and is ported over to FreeBSD which FreeNAS is based on.  But it has come a long way now and I’m having a lot of success so far running this version of FreeNAS so far.

My plan for now is to use the FreeNAS box I use for the boys TVShows & Movies.  I’m using a bunch (7 total) of old drives striped toggether in a ZFS vdev and added to a ZFS pool called backuppool0.  This pool will be rsynced nightly from my main drive which (for now) is a 1 terabyte drive that is on it’s own as a UFS/(EFI GPT) formatted drive.  It used to be rsynced nightly onto two 500Gig drives, a movie section on one 500Gig drive and the TV Shows on the other 500 Gig drive.  But since I’ve decided to rip the my TV shows that I want to watch and put those on my drives so that I can access them with a click of a button the 500 Gig drive is getting too full…

ZFS to the rescue!

I’ve been recording most of my experiments so far and I’m currently rsyncing my 1 Terabyte drive to my ZFS backuppool0:  Here’s the data I’ve collected so far.  Hopefully other FreeNAS experimenters would like the info (if they can find this page…)

Important things I’ve found out recently with FreeNAS and ZFS.  Make sure that you do the format part with each drive you are going to use in your ZFS vdev.  If you don’t I’ve found that they come up with some GPT errors which I couldn’t fix without doing the ZFS format.  Disks|Format (choose the Disk) then in the File System drop down box be sure to select ZFS storage pool device.  Then you will end up with a nice clean ZFS system…

FreeNAS 0.7 Sardaukar (revision 4519) built on Sun Apr 5 10:59:48 UTC 2009

Original

pool0
[8.70GB used of 586GB] [577GB available of 586GB] 1% of 586GB
Total: 586G | Used: 8.70G | Free: 577G | State: ONLINE

Name Size Used Free Capacity Health AltRoot
pool0 586G 8.70G 577G 1% ONLINE –
ad0 194481MB Maxtor 6B200P0/BAH41B70 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad1 117800MB IC35L120AVV207-1/V24OA66A 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad16 194481MB Maxtor 6Y200P0/YAR41BW0 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad17 238476MB HDS722525VLAT80/V36OA6EA 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad2 78534MB HDS728080PLAT20/PF2OA21B 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad3 6150MB Maxtor 90640D4/PAS24F15 5 ZFS storage pool device ONLINE
pool: pool0
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
pool0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad1 ONLINE 0 0 0
ad2 ONLINE 0 0 0
ad3 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0 ONLINE 0 0 0
ad16 ONLINE 0 0 0
ad17 ONLINE 0 0 0

errors: No known data errors

capacity operations bandwidth
pool used avail read write read write
———- —– —– —– —– —– —–
pool0 8.70G 577G 0 0 80 715
raidz1 4.08G 13.9G 0 0 13 239
ad1 – – 0 0 2.16K 140
ad2 – – 0 0 2.26K 137
ad3 – – 0 0 2.05K 137
raidz1 4.61G 563G 0 0 67 476
ad0 – – 0 0 1.50K 254
ad16 – – 0 0 1.72K 240
ad17 – – 0 0 1.50K 247
———- —– —– —– —– —– —–
Name Type Comment
vdev0 raidz1 vdev0 raid5 single parity – smallest drives [Edit device] [Delete device]
vdev1 raidz1 vdev1 raid5 single parity – largest drives

#zfs list
NAME USED AVAIL REFER MOUNTPOINT
pool0 5.79G 378G 5.79G /mnt/pool0
#zpool scrub pool0

pool: pool0
state: ONLINE
scrub: scrub completed with 0 errors on Fri Apr 10 16:12:01 2009
config:

NAME STATE READ WRITE CKSUM
pool0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad1 ONLINE 0 0 0
ad2 ONLINE 0 0 0
ad3 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0 ONLINE 0 0 0
ad16 ONLINE 0 0 0
ad17 ONLINE 0 0 0

errors: No known data errors
#zpool offline pool0 ad3

pool: pool0
state: DEGRADED
status: One or more devices has been taken offline by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using ‘zpool online’ or replace the device with
‘zpool replace’.
scrub: scrub completed with 0 errors on Fri Apr 10 16:12:01 2009
config:

NAME STATE READ WRITE CKSUM
pool0 DEGRADED 0 0 0
raidz1 DEGRADED 0 0 0
ad1 ONLINE 0 0 0
ad2 ONLINE 0 0 0
ad3 OFFLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0 ONLINE 0 0 0
ad16 ONLINE 0 0 0
ad17 ONLINE 0 0 0

errors: No known data errors

Shutdown the computer change out ad3 (6 G drive with 200 Gig drive)
Reboot

In GUI I went to disk management and removed ad3 drive then added the new 200gig ad3 preformatted file system ZFS storage pool device
Apply Changes
Do replace command:

#zpool replace pool0 ad3

pool: pool0
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scrub: resilver in progress, 88.41% done, 0h0m to go
config:

NAME STATE READ WRITE CKSUM
pool0 DEGRADED 0 0 0
raidz1 DEGRADED 0 0 0
ad1 ONLINE 0 0 0
ad2 ONLINE 0 0 0
replacing DEGRADED 0 0 0
ad3/old OFFLINE 0 0 0
ad3 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0 ONLINE 0 0 0
ad16 ONLINE 0 0 0
ad17 ONLINE 0 0 0

errors: No known data errors

After a while, the resivering has finished

pool: pool0
state: ONLINE
scrub: resilver completed with 0 errors on Fri Apr 10 16:30:49 2009
config:

NAME STATE READ WRITE CKSUM
pool0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad1 ONLINE 0 0 0
ad2 ONLINE 0 0 0
ad3 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0 ONLINE 0 0 0
ad16 ONLINE 0 0 0
ad17 ONLINE 0 0 0

errors: No known data errors

#zfs list
NAME USED AVAIL REFER MOUNTPOINT
pool0 5.79G 378G 5.79G /mnt/pool0

Reboot

pool0
[8.70GB used of 798GB] [789GB available of 798GB] 1% of 798GB
Total: 798G | Used: 8.70G | Free: 789G | State: ONLINE
———————————————————–Try adding SATA drive to IDE vdevs in pool———-

pool0
[8.70GB used of 798GB] [789GB available of 798GB] 1% of 798GB
Total: 798G | Used: 8.70G | Free: 789G | State: ONLINE
#zpool scrub pool0

After Scrubbing is complete:

pool: pool0
state: ONLINE
scrub: scrub completed with 0 errors on Fri Apr 10 17:05:29 2009
config:

NAME STATE READ WRITE CKSUM
pool0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad1 ONLINE 0 0 0
ad2 ONLINE 0 0 0
ad3 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0 ONLINE 0 0 0
ad16 ONLINE 0 0 0
ad17 ONLINE 0 0 0

errors: No known data errors
Disk Size Description Standby time File system Status
ad0 194481MB Maxtor 6B200P0/BAH41B70 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad1 117800MB IC35L120AVV207-1/V24OA66A 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad16 194481MB Maxtor 6Y200P0/YAR41BW0 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad17 238476MB HDS722525VLAT80/V36OA6EA 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad2 78534MB HDS728080PLAT20/PF2OA21B 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad3 194481MB Maxtor 6B200P0/BAH41B70 5 ZFS storage pool device ONLINE
This time I’ll try to replace ad2 80 Gig IDE drive with a 300 Gig SATA drive

# zpool offline pool0 ad2

pool: pool0
state: DEGRADED
status: One or more devices has been taken offline by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using ‘zpool online’ or replace the device with
‘zpool replace’.
scrub: scrub completed with 0 errors on Fri Apr 10 17:05:29 2009
config:

NAME STATE READ WRITE CKSUM
pool0 DEGRADED 0 0 0
raidz1 DEGRADED 0 0 0
ad1 ONLINE 0 0 0
ad2 OFFLINE 0 0 0
ad3 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0 ONLINE 0 0 0
ad16 ONLINE 0 0 0
ad17 ONLINE 0 0 0

errors: No known data errors

Shutdown and add the 300 Gig SATA drive

300 Gig drive showed up as device ad8

Add as new ZFS storage device (Apply Changes)

ad2 (80 Gig) still exist on the system at this point, as I haven’t removed it physically or from disk management

#zpool replace pool0 ad2 ad8

pool: pool0
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scrub: resilver in progress, 51.26% done, 0h0m to go
config:

NAME STATE READ WRITE CKSUM
pool0 DEGRADED 0 0 0
raidz1 DEGRADED 0 0 0
ad1 ONLINE 0 0 0
replacing DEGRADED 0 0 0
ad2 OFFLINE 0 0 0
ad8 ONLINE 0 0 0
ad3 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0 ONLINE 0 0 0
ad16 ONLINE 0 0 0
ad17 ONLINE 0 0 0

errors: No known data errors

pool: pool0
state: ONLINE
scrub: resilver completed with 0 errors on Fri Apr 10 17:22:40 2009
config:

NAME STATE READ WRITE CKSUM
pool0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad1 ONLINE 0 0 0
ad8 ONLINE 0 0 0
ad3 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0 ONLINE 0 0 0
ad16 ONLINE 0 0 0
ad17 ONLINE 0 0 0

errors: No known data errors
pool0
[8.70GB used of 798GB] [789GB available of 798GB] 1% of 798GB
Total: 798G | Used: 8.70G | Free: 789G | State: ONLINE

Reboot

pool0
[8.70GB used of 913GB] [904GB available of 913GB] 0% of 913GB
Total: 913G | Used: 8.70G | Free: 904G | State: ONLINE

pool: pool0
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
pool0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad1 ONLINE 0 0 0
ad8 ONLINE 0 0 0
ad3 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad0 ONLINE 0 0 0
ad16 ONLINE 0 0 0
ad17 ONLINE 0 0 0

errors: No known data errors

Disk Size Description Standby time File system Status
ad0 194481MB Maxtor 6B200P0/BAH41B70 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad1 117800MB IC35L120AVV207-1/V24OA66A 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad16 194481MB Maxtor 6Y200P0/YAR41BW0 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad17 238476MB HDS722525VLAT80/V36OA6EA 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad2 78534MB HDS728080PLAT20/PF2OA21B 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad3 194481MB Maxtor 6B200P0/BAH41B70 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad8 286188MB 7V300F0/VA111630 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]

*******************************************************************************************************************************************

Test Raid5 single parity growing 3 small drives, adding a larger drive and filling it up. As df -h doesn’t show the proper size so I don’t know if the

operating system will handle it…

ad1 117800MB IC35L120AVV207-1/V24OA66A 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad2 78534MB HDS728080PLAT20/PF2OA21B 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
da1 3820MB 4gig usb drive Always on ZFS storage pool device ONLINE [Edit disk] [Delete disk]

vdev0 raid5 single parity into pool0

pool: pool0
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
pool0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad1 ONLINE 0 0 0
ad2 ONLINE 0 0 0
da1 ONLINE 0 0 0

errors: No known data errors

$ df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/da0s1a 494M 87M 367M 19% /
devfs 1.0K 1.0K 0B 100% /dev
pool0 7.3G 595M 6.7G 8% /mnt/pool0

At this point df -h works fine. I’ll fill up the drive and then grow the zpool and add more data to it to see if freeNAS can handle it.

Filled pool0:
pool0
[10.9GB used of 11.1GB] [258MB available of 11.1GB] 97% of 11.1GB
Total: 11.1G | Used: 10.9G | Free: 258M | State: ONLINE

pool: pool0
state: ONLINE
scrub: scrub in progress, 2.37% done, 0h13m to go
config:

NAME STATE READ WRITE CKSUM
pool0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad1 ONLINE 0 0 0
ad2 ONLINE 0 0 0
da1 ONLINE 0 0 0

errors: No known data errors

offline da1 (click on ONLINE in GUI beside da1)

Command output:
Bringing device da1 offline

pool: pool0
state: DEGRADED
status: One or more devices has been taken offline by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using ‘zpool online’ or replace the device with
‘zpool replace’.
scrub: resilver completed with 0 errors on Fri Apr 10 19:41:38 2009
config:

NAME STATE READ WRITE CKSUM
pool0 DEGRADED 0 0 0
raidz1 DEGRADED 0 0 0
ad1 ONLINE 0 0 0
ad2 ONLINE 0 0 0
da1 OFFLINE 0 0 0

errors: No known data errors

#zpool replace pool0 da1 ad8

pool: pool0
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scrub: resilver in progress, 2.58% done, 0h1m to go
config:

NAME STATE READ WRITE CKSUM
pool0 DEGRADED 0 0 1
raidz1 DEGRADED 0 0 1
ad1 ONLINE 0 0 0
ad2 ONLINE 0 0 0
replacing DEGRADED 0 0 0
da1 OFFLINE 0 0 0
ad8 ONLINE 0 0 0

errors: No known data errors

pool: pool0
state: ONLINE
scrub: resilver completed with 0 errors on Fri Apr 10 19:49:02 2009
config:

NAME STATE READ WRITE CKSUM
pool0 ONLINE 0 0 1
raidz1 ONLINE 0 0 1
ad1 ONLINE 0 0 0
ad2 ONLINE 0 0 0
ad8 ONLINE 0 0 0

errors: No known data errors

scrub pool0

pool: pool0
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using ‘zpool clear’ or replace the device with ‘zpool replace’.
see: http://www.sun.com/msg/ZFS-8000-9P
scrub: scrub in progress, 3.87% done, 0h2m to go
config:

NAME STATE READ WRITE CKSUM
pool0 ONLINE 0 0 1
raidz1 ONLINE 0 0 1
ad1 ONLINE 0 0 2
ad2 ONLINE 0 0 0
ad8 ONLINE 0 0 0

errors: No known data errors

pool: pool0
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using ‘zpool clear’ or replace the device with ‘zpool replace’.
see: http://www.sun.com/msg/ZFS-8000-9P
scrub: scrub completed with 0 errors on Fri Apr 10 19:51:56 2009
config:

NAME STATE READ WRITE CKSUM
pool0 ONLINE 0 0 1
raidz1 ONLINE 0 0 1
ad1 ONLINE 0 0 2
ad2 ONLINE 0 0 0
ad8 ONLINE 0 0 0

errors: No known data errors

Tested Big RAR files on the pool0 and all data is still perfect (ZFS doing it’s thing!)

Before reboot:

pool0
[10.9GB used of 11.1GB] [258MB available of 11.1GB] 97% of 11.1GB
Total: 11.1G | Used: 10.9G | Free: 258M | State: ONLINE

Reboot to get new ZFS pool0 size updated

pool0
[10.9GB used of 230GB] [219GB available of 230GB] 4% of 230GB
Total: 230G | Used: 10.9G | Free: 219G | State: ONLINE

$ df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/da0s1a 494M 87M 367M 19% /
devfs 1.0K 1.0K 0B 100% /dev
pool0 151G 7.2G 144G 5% /mnt/pool0

pool: pool0
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
pool0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad1 ONLINE 0 0 0
ad2 ONLINE 0 0 0
ad8 ONLINE 0 0 0

errors: No known data errors

hmmmm…. df -h shows it grew but now the same numbers as zpool command

capacity operations bandwidth
pool used avail read write read write
———- —– —– —– —– —– —–
pool0 10.9G 219G 122 1 14.8M 3.24K
raidz1 10.9G 219G 122 1 14.8M 3.24K
ad1 – – 64 0 7.45M 3.50K
ad2 – – 65 0 7.45M 3.48K
ad8 – – 65 0 7.44M 3.49K
———- —– —– —– —– —– —–
Added a 2.7 Gig file

capacity operations bandwidth
pool used avail read write read write
———- —– —– —– —– —– —–
pool0 14.9G 215G 27 13 3.26M 1.26M
raidz1 14.9G 215G 27 13 3.26M 1.26M
ad1 – – 14 7 1.64M 646K
ad2 – – 14 7 1.64M 646K
ad8 – – 14 8 1.64M 646K
———- —– —– —– —– —– —–

$ df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/da0s1a 494M 87M 367M 19% /
devfs 1.0K 1.0K 0B 100% /dev
pool0 151G 9.9G 141G 7% /mnt/pool0
Looks like df -h are the real numbers to use!
*******************************************************************************************************************************************************

Replacing the 80 gig drive with a 500Gig:

ad0 194481MB Maxtor 6B200P0/BAH41B70 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad1 117800MB IC35L120AVV207-1/V24OA66A 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad16 194481MB Maxtor 6Y200P0/YAR41BW0 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad17 238476MB HDS722525VLAT80/V36OA6EA 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad2 78534MB HDS728080PLAT20/PF2OA21B 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad3 194481MB Maxtor 6B200P0/BAH41B70 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad8 286188MB 7V300F0/VA111630 5 ZFS storage pool device ONLINE

So ad2 will now be big!

Depower swap out drives.

Disk Size Description Standby time File system Status
ad0 194481MB Maxtor 6B200P0/BAH41B70 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad1 117800MB IC35L120AVV207-1/V24OA66A 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad16 194481MB Maxtor 6Y200P0/YAR41BW0 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad17 238476MB HDS722525VLAT80/V36OA6EA 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad2 476941MB ST3500630A/3.AAE 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad3 194481MB Maxtor 6B200P0/BAH41B70 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad8 286188MB 7V300F0/VA111630 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
Add 1.0 TB drive to SATA controller as motherboard didn’t want to boot with drive attached directly to it. (Maybe it doesn’t like big drives? not good for

the future…)

Disk Size Description Standby time File system Status
ad0 194481MB Maxtor 6B200P0/BAH41B70 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad1 117800MB IC35L120AVV207-1/V24OA66A 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad14 953870MB WDC WD10EAVS-00D7B1/01.01A01 5 UFS ONLINE [Edit disk] [Delete disk]
ad16 194481MB Maxtor 6Y200P0/YAR41BW0 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad17 238476MB HDS722525VLAT80/V36OA6EA 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad2 476941MB ST3500630A/3.AAE 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad3 194481MB Maxtor 6B200P0/BAH41B70 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad8 286188MB 7V300F0/VA111630 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]

We now have ad14 as a 1TB drive. This has data on it as a stand alone drive which must be copied to BackupPool first then can be added to the new pool0

which will be a stripe of this 1TB drive and a 500 G drive. They will be Rsynced with the backup-pool nightly for redundancy.

Rsynch transfer rate is about 150 Mb/s

Disk Size Description Standby time File system Status
*ad0 194481MB Maxtor 6B200P0/BAH41B70 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
*ad1 117800MB IC35L120AVV207-1/V24OA66A 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad14 953870MB WDC WD10EAVS-00D7B1/01.01A01 5 UFS ONLINE [Edit disk] [Delete disk]
ad16 194481MB Maxtor 6Y200P0/YAR41BW0 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
ad17 238476MB HDS722525VLAT80/V36OA6EA 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
*ad2 476941MB ST3500630A/3.AAE 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
*ad3 194481MB Maxtor 6B200P0/BAH41B70 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
*ad8 286188MB 7V300F0/VA111630 5 ZFS storage pool device ONLINE [Edit disk] [Delete disk]
11:37AM
Disk Capacity Description I/O statistics Temperature Status
ad0 194481MB Maxtor 6B200P0/BAH41B70 56.33 KiB/t, 9 tps, 0.50 MiB/s 41 °C ONLINE
ad1 117800MB IC35L120AVV207-1/V24OA66A 56.78 KiB/t, 9 tps, 0.50 MiB/s 41 °C ONLINE
ad14 953870MB WDC WD10EAVS-00D7B1/01.01A01 63.37 KiB/t, 58 tps, 3.57 MiB/s 37 °C ONLINE
ad16 194481MB Maxtor 6Y200P0/YAR41BW0 56.30 KiB/t, 9 tps, 0.50 MiB/s 56 °C ONLINE
ad17 238476MB HDS722525VLAT80/V36OA6EA 56.35 KiB/t, 9 tps, 0.50 MiB/s 52 °C ONLINE
ad2 476941MB ST3500630A/3.AAE 57.79 KiB/t, 10 tps, 0.57 MiB/s 51 °C ONLINE
ad3 194481MB Maxtor 6B200P0/BAH41B70 56.49 KiB/t, 9 tps, 0.50 MiB/s 50 °C ONLINE
ad8 286188MB 7V300F0/VA111630 56.90 KiB/t, 10 tps, 0.53 MiB/s 46 °C ONLINE

After putting more old spare drives between them to insulate:

11:40AM
Disk Capacity Description I/O statistics Temperature Status
ad0 194481MB Maxtor 6B200P0/BAH41B70 50.24 KiB/t, 14 tps, 0.68 MiB/s 41 °C ONLINE
ad1 117800MB IC35L120AVV207-1/V24OA66A 50.73 KiB/t, 14 tps, 0.68 MiB/s 41 °C ONLINE
ad14 953870MB WDC WD10EAVS-00D7B1/01.01A01 63.43 KiB/t, 78 tps, 4.83 MiB/s 37 °C ONLINE
ad16 194481MB Maxtor 6Y200P0/YAR41BW0 50.20 KiB/t, 14 tps, 0.68 MiB/s 50 °C ONLINE
ad17 238476MB HDS722525VLAT80/V36OA6EA 50.40 KiB/t, 14 tps, 0.68 MiB/s 51 °C ONLINE
ad2 476941MB ST3500630A/3.AAE 52.58 KiB/t, 15 tps, 0.79 MiB/s 51 °C ONLINE
ad3 194481MB Maxtor 6B200P0/BAH41B70 50.97 KiB/t, 14 tps, 0.68 MiB/s 50 °C ONLINE
ad8 286188MB 7V300F0/VA111630 51.22 KiB/t, 14 tps, 0.72 MiB/s 47 °C ONLINE

11:42AM
Disk Capacity Description I/O statistics Temperature Status
ad0 194481MB Maxtor 6B200P0/BAH41B70 46.93 KiB/t, 18 tps, 0.83 MiB/s 41 °C ONLINE
ad1 117800MB IC35L120AVV207-1/V24OA66A 47.10 KiB/t, 18 tps, 0.83 MiB/s 41 °C ONLINE
ad14 953870MB WDC WD10EAVS-00D7B1/01.01A01 63.39 KiB/t, 95 tps, 5.88 MiB/s 37 °C ONLINE
ad16 194481MB Maxtor 6Y200P0/YAR41BW0 46.43 KiB/t, 18 tps, 0.83 MiB/s 48 °C ONLINE
ad17 238476MB HDS722525VLAT80/V36OA6EA 47.48 KiB/t, 18 tps, 0.83 MiB/s 51 °C ONLINE
ad2 476941MB ST3500630A/3.AAE 49.79 KiB/t, 20 tps, 0.98 MiB/s 51 °C ONLINE
ad3 194481MB Maxtor 6B200P0/BAH41B70 48.19 KiB/t, 18 tps, 0.83 MiB/s 50 °C ONLINE
ad8 286188MB 7V300F0/VA111630 48.62 KiB/t, 19 tps, 0.90 MiB/s 47 °C ONLINE

12:25PM
Disk Capacity Description I/O statistics Temperature Status
ad0 194481MB Maxtor 6B200P0/BAH41B70 40.48 KiB/t, 48 tps, 1.92 MiB/s 42 °C ONLINE
ad1 117800MB IC35L120AVV207-1/V24OA66A 40.62 KiB/t, 48 tps, 1.92 MiB/s 45 °C ONLINE
ad14 953870MB WDC WD10EAVS-00D7B1/01.01A01 63.29 KiB/t, 223 tps, 13.77 MiB/s 42 °C ONLINE
ad16 194481MB Maxtor 6Y200P0/YAR41BW0 39.31 KiB/t, 51 tps, 1.95 MiB/s 55 °C ONLINE
ad17 238476MB HDS722525VLAT80/V36OA6EA 40.59 KiB/t, 51 tps, 2.01 MiB/s 51 °C ONLINE
ad2 476941MB ST3500630A/3.AAE 42.39 KiB/t, 56 tps, 2.34 MiB/s 53 °C ONLINE
ad3 194481MB Maxtor 6B200P0/BAH41B70 41.52 KiB/t, 47 tps, 1.92 MiB/s 53 °C ONLINE
ad8 286188MB 7V300F0/VA111630 43.89 KiB/t, 53 tps, 2.29 MiB/s 48 °C ONLINE
Re-arranged spare drives as insulators and seemed to help drive ad16 194481MB Maxtor 6Y200P0/YAR41BW0
12:30PM
Disk Capacity Description I/O statistics Temperature Status
ad0 194481MB Maxtor 6B200P0/BAH41B70 40.43 KiB/t, 49 tps, 1.95 MiB/s 42 °C ONLINE
ad1 117800MB IC35L120AVV207-1/V24OA66A 40.54 KiB/t, 49 tps, 1.95 MiB/s 45 °C ONLINE
ad14 953870MB WDC WD10EAVS-00D7B1/01.01A01 63.29 KiB/t, 227 tps, 14.04 MiB/s 42 °C ONLINE
ad16 194481MB Maxtor 6Y200P0/YAR41BW0 39.19 KiB/t, 52 tps, 1.98 MiB/s 54 °C ONLINE
ad17 238476MB HDS722525VLAT80/V36OA6EA 40.42 KiB/t, 52 tps, 2.05 MiB/s 51 °C ONLINE
ad2 476941MB ST3500630A/3.AAE 42.16 KiB/t, 58 tps, 2.39 MiB/s 53 °C ONLINE
ad3 194481MB Maxtor 6B200P0/BAH41B70 41.39 KiB/t, 48 tps, 1.95 MiB/s 53 °C ONLINE
ad8 286188MB 7V300F0/VA111630 43.82 KiB/t, 55 tps, 2.34 MiB/s 47 °C ONLINE
12:33PM
Disk Capacity Description I/O statistics Temperature Status
ad0 194481MB Maxtor 6B200P0/BAH41B70 40.36 KiB/t, 50 tps, 1.98 MiB/s 43 °C ONLINE
ad1 117800MB IC35L120AVV207-1/V24OA66A 40.47 KiB/t, 50 tps, 1.98 MiB/s 45 °C ONLINE
ad14 953870MB WDC WD10EAVS-00D7B1/01.01A01 63.26 KiB/t, 231 tps, 14.25 MiB/s 42 °C ONLINE
ad16 194481MB Maxtor 6Y200P0/YAR41BW0 39.11 KiB/t, 53 tps, 2.02 MiB/s 52 °C ONLINE
ad17 238476MB HDS722525VLAT80/V36OA6EA 40.27 KiB/t, 53 tps, 2.08 MiB/s 51 °C ONLINE
ad2 476941MB ST3500630A/3.AAE 41.90 KiB/t, 59 tps, 2.42 MiB/s 53 °C ONLINE
ad3 194481MB Maxtor 6B200P0/BAH41B70 41.28 KiB/t, 49 tps, 1.98 MiB/s 53 °C ONLINE
ad8 286188MB 7V300F0/VA111630 43.75 KiB/t, 56 tps, 2.37 MiB/s 47 °C ONLINE
12:41PM
Disk Capacity Description I/O statistics Temperature Status
ad0 194481MB Maxtor 6B200P0/BAH41B70 40.48 KiB/t, 52 tps, 2.04 MiB/s 43 °C ONLINE
ad1 117800MB IC35L120AVV207-1/V24OA66A 40.47 KiB/t, 52 tps, 2.04 MiB/s 45 °C ONLINE
ad14 953870MB WDC WD10EAVS-00D7B1/01.01A01 63.31 KiB/t, 237 tps, 14.66 MiB/s 43 °C ONLINE
ad16 194481MB Maxtor 6Y200P0/YAR41BW0 38.93 KiB/t, 54 tps, 2.07 MiB/s 52 °C ONLINE
ad17 238476MB HDS722525VLAT80/V36OA6EA 39.91 KiB/t, 55 tps, 2.14 MiB/s 52 °C ONLINE
ad2 476941MB ST3500630A/3.AAE 41.54 KiB/t, 61 tps, 2.49 MiB/s 53 °C ONLINE
ad3 194481MB Maxtor 6B200P0/BAH41B70 41.25 KiB/t, 50 tps, 2.03 MiB/s 53 °C ONLINE
ad8 286188MB 7V300F0/VA111630 43.85 KiB/t, 57 tps, 2.45 MiB/s 47 °C ONLINE

12:43PM
Disk Capacity Description I/O statistics Temperature Status
ad0 194481MB Maxtor 6B200P0/BAH41B70 40.68 KiB/t, 52 tps, 2.06 MiB/s 43 °C ONLINE
ad1 117800MB IC35L120AVV207-1/V24OA66A 40.46 KiB/t, 52 tps, 2.06 MiB/s 45 °C ONLINE
ad14 953870MB WDC WD10EAVS-00D7B1/01.01A01 63.34 KiB/t, 240 tps, 14.84 MiB/s 43 °C ONLINE
ad16 194481MB Maxtor 6Y200P0/YAR41BW0 38.83 KiB/t, 55 tps, 2.10 MiB/s 50 °C ONLINE
ad17 238476MB HDS722525VLAT80/V36OA6EA 39.79 KiB/t, 56 tps, 2.17 MiB/s 53 °C ONLINE
ad2 476941MB ST3500630A/3.AAE 41.43 KiB/t, 62 tps, 2.52 MiB/s 52 °C ONLINE
ad3 194481MB Maxtor 6B200P0/BAH41B70 41.36 KiB/t, 51 tps, 2.06 MiB/s 50 °C ONLINE
ad8 286188MB 7V300F0/VA111630 44.01 KiB/t, 58 tps, 2.48 MiB/s 47 °C ONLINE

More re-arrangements of insulator drives

12:48PM
Disk Capacity Description I/O statistics Temperature Status
ad0 194481MB Maxtor 6B200P0/BAH41B70 40.79 KiB/t, 53 tps, 2.09 MiB/s 43 °C ONLINE
ad1 117800MB IC35L120AVV207-1/V24OA66A 40.49 KiB/t, 53 tps, 2.10 MiB/s 45 °C ONLINE
ad14 953870MB WDC WD10EAVS-00D7B1/01.01A01 63.38 KiB/t, 244 tps, 15.09 MiB/s 42 °C ONLINE
ad16 194481MB Maxtor 6Y200P0/YAR41BW0 38.79 KiB/t, 56 tps, 2.13 MiB/s 50 °C ONLINE
ad17 238476MB HDS722525VLAT80/V36OA6EA 39.68 KiB/t, 57 tps, 2.21 MiB/s 52 °C ONLINE
ad2 476941MB ST3500630A/3.AAE 41.25 KiB/t, 64 tps, 2.57 MiB/s 50 °C ONLINE
ad3 194481MB Maxtor 6B200P0/BAH41B70 41.44 KiB/t, 52 tps, 2.09 MiB/s 49 °C ONLINE
ad8 286188MB 7V300F0/VA111630 44.16 KiB/t, 58 tps, 2.52 MiB/s 46 °C ONLINE

12:49PM
Disk Capacity Description I/O statistics Temperature Status
ad0 194481MB Maxtor 6B200P0/BAH41B70 40.85 KiB/t, 53 tps, 2.10 MiB/s 42 °C ONLINE
ad1 117800MB IC35L120AVV207-1/V24OA66A 40.53 KiB/t, 53 tps, 2.10 MiB/s 45 °C ONLINE
ad14 953870MB WDC WD10EAVS-00D7B1/01.01A01 63.39 KiB/t, 245 tps, 15.14 MiB/s 42 °C ONLINE
ad16 194481MB Maxtor 6Y200P0/YAR41BW0 38.76 KiB/t, 56 tps, 2.14 MiB/s 50 °C ONLINE
ad17 238476MB HDS722525VLAT80/V36OA6EA 39.63 KiB/t, 57 tps, 2.21 MiB/s 51 °C ONLINE
ad2 476941MB ST3500630A/3.AAE 41.20 KiB/t, 64 tps, 2.58 MiB/s 50 °C ONLINE
ad3 194481MB Maxtor 6B200P0/BAH41B70 41.48 KiB/t, 52 tps, 2.10 MiB/s 48 °C ONLINE
ad8 286188MB 7V300F0/VA111630 44.23 KiB/t, 59 tps, 2.53 MiB/s 44 °C ONLINE
01:19PM
Disk Capacity Description I/O statistics Temperature Status
ad0 194481MB Maxtor 6B200P0/BAH41B70 41.00 KiB/t, 56 tps, 2.24 MiB/s 42 °C ONLINE
ad1 117800MB IC35L120AVV207-1/V24OA66A 40.11 KiB/t, 57 tps, 2.24 MiB/s 45 °C ONLINE
ad14 953870MB WDC WD10EAVS-00D7B1/01.01A01 63.53 KiB/t, 261 tps, 16.19 MiB/s 40 °C ONLINE
ad16 194481MB Maxtor 6Y200P0/YAR41BW0 37.70 KiB/t, 62 tps, 2.28 MiB/s 50 °C ONLINE
ad17 238476MB HDS722525VLAT80/V36OA6EA 38.08 KiB/t, 64 tps, 2.38 MiB/s 49 °C ONLINE
ad2 476941MB ST3500630A/3.AAE 39.62 KiB/t, 71 tps, 2.76 MiB/s 48 °C ONLINE
ad3 194481MB Maxtor 6B200P0/BAH41B70 41.25 KiB/t, 55 tps, 2.23 MiB/s 46 °C ONLINE
ad8 286188MB 7V300F0/VA111630 44.47 KiB/t, 63 tps, 2.72 MiB/s 47 °C ONLINE

Things are looking cooler now! The Rsync wont be finished for another 8 hours…
02:09PM
Disk Capacity Description I/O statistics Temperature Status
ad0 194481MB Maxtor 6B200P0/BAH41B70 41.16 KiB/t, 59 tps, 2.36 MiB/s 41 °C ONLINE
ad1 117800MB IC35L120AVV207-1/V24OA66A 39.64 KiB/t, 61 tps, 2.37 MiB/s 45 °C ONLINE
ad14 953870MB WDC WD10EAVS-00D7B1/01.01A01 63.65 KiB/t, 275 tps, 17.11 MiB/s 40 °C ONLINE
ad16 194481MB Maxtor 6Y200P0/YAR41BW0 36.85 KiB/t, 67 tps, 2.41 MiB/s 50 °C ONLINE
ad17 238476MB HDS722525VLAT80/V36OA6EA 36.86 KiB/t, 70 tps, 2.52 MiB/s 48 °C ONLINE
ad2 476941MB ST3500630A/3.AAE 38.40 KiB/t, 78 tps, 2.92 MiB/s 48 °C ONLINE
ad3 194481MB Maxtor 6B200P0/BAH41B70 41.12 KiB/t, 58 tps, 2.35 MiB/s 44 °C ONLINE
ad8 286188MB 7V300F0/VA111630 44.76 KiB/t, 66 tps, 2.88 MiB/s 47 °C ONLINE