Thread

Btrfs/ZFS/bcachefs under workloads classic benchmarks skip

179 points · 190 comments · farlight

fenio · 1789847733 · focus · HN ↗
The author of the benchmark here. I went over some comments and I'll try to tackle them here. I'm pretty clear that GH runner based benchmark is far from perfect due to noisy neighbours etc. Thus every test first is running so called calibration... to reject completely unreliable VMs. I'm fully aware that this can't completely fix the issue. Can limit it but not fix. But as of now there are 593 runs recorded so average should still be quite meaningful.
Having that said I'm desperately trying to get REAL hardware to run that benchmark. With some successes ;)
Few months ago I got Hetzner machine from Kent Overstreet and I was able to finish 3 runs before machine died... Results: <a href="https://bartosz.fenski.pl/modern-fs-benchmark/real-hw/" rel="nofollow">https://bartosz.fenski.pl/modern-fs-benchmark/real-hw/
Currently I've got even more interesting machine with tons of disks and I'm running new set of benchmarks but it's really in its initial stage.
<a href="https://bartosz.fenski.pl/modern-fs-benchmark/sas-hdd/" rel="nofollow">https://bartosz.fenski.pl/modern-fs-benchmark/sas-hdd/ 2nd run in progress... one run on REAL hardware takes much more time than on GH runner so it's slow.
But this new hardware has also so many disks that the plan is to try also more complex, tiered cache topologies. I'm working on it.
I'm happy to answer any other questions, sources of every piece of this benchmark are freely available and I'm not saying they are 100% correct. I'm open to improvements.
1. koverstreet · 1789848001 · focus · HN ↗
  I went back and forth with Hetzner a couple times, I think we just got a bad machine :)
  I've been saying it for months, but eventually I'm going to move the automated builds off the 48 core monster and we'll be able to use that for automated perf testing too. The machine we just got has spindles for EC perf testing, but the Hetzner monster has very high end enterprise ssdd.
  Also, just got done with the Rust for Linux conference, still not home but here's slides that still need reformatting: <a href="https://evilpiepirate.org/~kent/Kangrejos-2026-bcachefs.pdf" rel="nofollow">https://evilpiepirate.org/~kent/Kangrejos-2026-bcachefs.pdf
2. Joel_Mckay · 1789849034 · focus · HN ↗
  Should include:
  1. Dual Ext4 + external 32GB journal X4 pcie SSD (the prior winner of benchmark surveys)
  2. Bare F2FS after a trim and SSD vendor software cache flush operation (it should be slower, but knowing how much slower on identical hardware could be interesting.)
  3. DRBD across a 48U 100Gbps host rack (single X4 pcie data drive per host, OS on primary)
  4. CephFS across a 48U 100Gbps host rack (single X4 pcie data drive per host, OS on primary)
  Best regards =3
  1. throwaway270925 · 1789853840 · focus · HN ↗
    Also
    5. a ZFS dRaid configuration. There could be very different characteristics there with it using slabs.
    Speaking of slabs, MS ReFS of you feel adventurous!
  2. delamon · 1789894355 · focus · HN ↗
    6. modern nvme drive, preferrably pcie gen5 that can push >14GiB/sec
3. Grayskull · 1789888803 · focus · HN ↗
  Love to hear that there will be more real hw tests. At the moment I am building NAS and used your benchmark for evaluating the filesystems. I am glad to see that your data roughly matches mine (apart from scrub which on 4x 6tb HDDs took 15 hours for md-raid10 while CoW systems took seconds). Personally I found that array of HDDs behaves very differently than GH runner (my feeling is that since it runs on same disk you are testing theoretical throughput rather than ability to utilize disks). My tests gave an idea for following topologies:
  * 4 HDDs (for example dm-raid has read balancing optimized specifically for HDDs) * 5 HDDs (classical raid should see no improvement but btrfs and bcachefs should balance the load) * 4 SSDs * 3 HDDs + 1 SSD no tiering * 2 HDDs + 2 SSD no tiering * 1 drive 10x larger than others (since how bcachefs and btrfs allocators work) * nocow
  Thanks for awesome work
  1. ciupicri · 1789905050 · focus · HN ↗
    How can a scrub take only seconds when it has to read all the data?
    Polizeiposaune · 1789914139 · focus · HN ↗
    zfs-style scrub only reads allocated blocks and skips unallocated blocks; if the pool is mostly empty it can complete very quickly.
    Layered storage systems with a RAID layer that makes N disks look like one big disk generally don't have visibility into which blocks are free and which are allocated so they must "scrub" all the disks on initialization and repair even if only 1% is used.
4. zenoprax · 1789889559 · focus · HN ↗
  I have three identical Lenovo SFF PCs with a U.2 SSD in each. I'm currently running them in a Ceph cluster but I'll be tearing that down soon. I could run some benchmarks with three in one box and report back? Would be a one-time thing rather than an on-going commitment though.
  Disregard, I just saw you have some RAID10 tests in there so three SSDs won't be enough.
5. craniumjello · 1789910190 · focus · HN ↗
  [dead]
6. BoingBoomTschak · 1789916250 · focus · HN ↗
  Good job and clean website interface! Something as exhaustive as this is clearly needed.
  Some remarks:
  1. Why does the CoW button remove XFS from the list? Even <a href="https://github.com/fenio/modern-fs-benchmark/blob/main/scripts/fs/xfs.sh" rel="nofollow">https://github.com/fenio/modern-fs-benchmark/blob/main/scrip... mentions it has reflink enabled
  2. If you have the time, adding XFS + mdraid + dm-integrity [1] (in bitmap mode) as a comparison point against ZFS RAID-Zx might be an interesting data point. That's what I run, personally.
  3. Did you give some thoughts to the I/O scheduler choice? Might matter a lot in some cases.
  [1] <a href="https://www.kernel.org/doc/html/latest/admin-guide/device-mapper/dm-integrity.html" rel="nofollow">https://www.kernel.org/doc/html/latest/admin-guide/device-ma...
  1. fenio · 1789924588 · focus · HN ↗
    Thanks, all three are fair points.
    1. XFS reflink is enabled and its reflink/CoW-break measurements do run. The dashboard button currently means “native/full-CoW filesystem family”, not “supports reflink”, but that distinction is not clear from the label. I’ll rename it to “Native CoW” and add a separate reflink-capable filter that includes XFS.
    2. The current integrity comparison is XFS on LVM/dm-raid10 with dm-integrity in its default journal mode. It is not mdraid and not bitmap mode, so your suggested stack would be a genuinely different and useful data point. An md RAID5/6 over per-member bitmap-mode dm-integrity comparison against RAID-Z1/Z2 makes sense, with the weaker post-crash bitmap semantics documented.
    3. I did not pin or record the scheduler, which is a reproducibility gap. The dedicated SAS machine currently has mq-deadline active on all HDDs and SSDs. I’ll add queue/scheduler metadata to results before considering separate scheduler variants, since it can strongly affect the mixed and latency-sensitive phases.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.