Once I find some equipment to use them on I will definitely check out FreeBSD and PFsense.
I personally really like FreeBSD, but generally tell people that unless you have a reason, you're probably better off with Linux, because of the larger user-base for help, and hardware support (which is much better than in the past, but still leans away from supporting weirder consumer hardware). If you go for it, expect a bit of a learning curve - as a user, you'll notice familiar command line utilities have different flags and other relatively minor things, but proper administration requires some learning. I also really prefer pfsense to iptables, entirely for the configuration. Pfsense's config is *much* easier to follow and reason about for nontrivial configurations. (You can write scripts to drive iptables, and write them such that they're easier to follow, but with pfsense, you don't have to.)
With ZFS (or BTRFS, for that matter), also expect to do some learning. It does offer a lot to make administration easier (snapshots, and the ability to trivially send them to another machine are pure gold), but if you treat it like just another file system, you can run in to problems. My suggestion to folks new to it is to resist the urge, upon acquiring hardware, to set up your Dream System(tm) for a little while, and go through the administration commands along with reading material and *try to cause* some of the bad situations that can happen, so you understand them[1]. If you have experience with "enterprise" filesystems that do snapshotting and whatnot, you're ahead of the game, but ZFS of course has its own particular quirks.
As far as comparing ZFS and BTRFS, that's the raw material of flame wars. I haven't run BTRFS in production, so you should probably ignore me anyway. But generally, I can say that ZFS is extremely solid, and (depending on how you set it up) can be very performant. BTRFS is much younger, and as I understand it not considered feature-complete yet, so, from my personal perspective, I won't trust it with important data until it has been boring for a few years.
Comparing ext3 with ZFS is almost apples and oranges. ZFS checksums everything and self-repairs on the fly as much as possible. If you want, you can even run ext3 (or XFS, or...) other partitions inside ZFS datasets - that can cause complications, but allows you to use some ZFS features with other file systems.
One thing a lot of people balk at is that the ZFS folks highly recommend running ECC RAM with it. You don't have to, and running ZFS without it is still safer than, say, ext3. But without ECC, some ZFS guarantees about data integrity don't apply. Again, this is not something specific to ZFS - silent RAM corruption will destroy data on any filesystem. I think the difference is just that the ZFS developers are more vocal about the issue.
Sorry to turn this into a threadjack - it is fresh on my mind, because I also run ZFS at work and just got done adding a bunch of storage and revamping part of our backups.
[1] Most problems people get themselves in to are either at initial configuration (specifying an ashift that works poorly with your disks, picking the wrong type of redundancy for your gear/workload, some other configuration details), or letting the pools fill up. And unless you understand exactly what you're doing, never turn dedup on!
