Hey everyone, here's the plan, migrate all of my VM's to a data store on my NFS server, I've been wanting to do this for a while, to get off of the dedicated local storage, and move to shared. Big benefits of doing so, one of which is multiple hosts can access the same VM's for redundancy. But, I initially set up using ZFS. I went with ZFS for compression and data deduping. Turns out, deduping is a huge memory hog. Compression is a huge CPU hog (not a shocker). So, when I first created them, doing some benchmarks, I ended up turning off compression. But, as more and more data got put on the NFS box, it slowed down, it's now down to around 1MB/s. Yes, that is not a type-o, in fact, getting 1MB is a good day. So, I spent a bunch of time trying to figure it out. My primary thought, optimize the 3ware card. There was some settings I changed.

Turning off the auto-verify/rebuild. Increased the speed to about 3MB/s. But still not anywhere near good enough. It was currently verifying the raid.
Turned off/on the ECC override, no change.
Turned off/on the write cache, no change. Turns out the controller will only use write cache on Raid5.
StorSave was already set to Performance.
Disabled Rapid RAID Recovery, no change.

Now, for the big performance changes, ZFS.

Disabled dedup, increased write performance to 60MB/s on a single drive. Huge increase there. Remind you, this is only on a single drive pool. And my raid card is 100MB/s+. Using ESX, I was still getting slower performance than expected, so, while digging through some ZFS performance blogs, someone mentioned disabling the sync. zfs set sync=disabled <pool>. I tried it with the 3 values, Auto, Always and Disabled. Using the same dd command, when set to always, horrible performance, 4MB/s. When disabled, 150+MB/s. Hmm, I'm disabling that.

I'm also considering dumping ZFS since now the benefits of using ZFS are, well, gone. When I find more performance fixes to increase the speed of the entire disk system, I will add them here. Running iostat, shows it writing to the disks in bursts. The command I was using to get my statistics: dd if=/dev/zero of=testfile bs=16384