> ZFS is the way forward. For cross compatability, for reliability, for stability, for lots of use cases.

I'm not entirely convinced we should settle on ZFS just yet. It's fantastic and quite possibly the best option right now but it has a few limitations:

- The Linux implentation seems to have issues with releasing memory from the ARC back to the system

- The only way to expand a zpool is to add a new vdev. Pools are essentially a RAID0 of vdevs so if a single vdev fails, your entire server fails. You can mirror or RAID within a vdev but this means that the reliability of a vdev is the reliability of your entire pool. The problem here is that you can't just add 1 or 2 new disks since adding a 1 or 2 disk vdev would be data-suicide. For smaller servers, this is silly.

BTRFS looked like it was getting there but it proved buggy and unreliable. Personally, I'm waiting for bcachefs: https://www.patreon.com/bcachefs

There's nothing wrong with a 2-disk vdev. Mirrored instead of RAIDZ is probably the best choice for "smaller servers."

I strongly disagree. If I have an 8 disk server at home with mirroring, the pool could only take a single disk failure. The usable space would be 50% of the total disk. Further, I'd have to buy any extra disks in pairs.

If instead I ran RAID6, I'd have 80% of the disk available and I could add disks in single disk increments.

I think ZFS makes great sense for businesses that can throw money at disks but for smaller businesses or home servers it's kinda bad.

I think it's not a very good idea to design storage solutions around how inexpensive it is to add capacity to them. Your 9+-drive RAID6 is going to take forever to rebuild; 4+ mirrored vdevs (or mirrored RAID of course) will not be a problem at all.

I run a FreeNAS device at home (ZFS underneath).

It was kind of a pain to configure (albeit quite flexible) but it's been pretty nice overall, already survived 1 disk failure and a capacity upgrade (during which I had to resilver after every individual disk upgrade, which was time-consuming, but after the last disk got upgraded, the extra space finally showed up)

You also wouldn't get the same I/O performance a stripped mirror will give you. You can get extra resiliency using a three disk mirror.

Disk is cheap. There's no reason to design like it's not.

Disk is at least ~$0.023/GB. Whether it's "cheap" or not depends on how much of it you need.

Plus, disk may be cheap but servers to house it are not (the kind of servers you'd run in your house, I know you can get cheap SC846 off eBay).

Servers can be inexpensive. It really depends on your use case. You don't have to drop 50K on a box to make a killer ZFS storage array.

My scale starts at 40PB with a "P". Disks aren't cheap.

I use stripped mirrors for most of my ZFS deployments. Disk is cheap. If you want more resilience do a three way mirror.

I'd extend your point on zpool expansion to ZFS's general inflexibility when it comes to reshaping pools or rebalancing ZRAIDs.

I'm sure it's not an issue in enterprise environments, but I've personally been bit by ZFS's inability to shrink or rebalance pools (even offline) several times now through personal use.

Those sort of use cases need to be handled if ZFS is to become a more general use filesystem.

> Personally, I'm waiting for bcachefs

tires++

;-)