Thread

Btrfs/ZFS/bcachefs under workloads classic benchmarks skip

179 points · 190 comments · farlight

Farmadupe · 1789844719 · focus · HN ↗
> CI runs use loop devices on shared ephemeral VMs (one VM per filesystem): compare shapes and ratios, not absolute MB/s. Each job records a host-calibration anchor — see the table.
I think if you're not using baremetal for such tests, it's likely that the results are simply not comparable at all? What if another tenant is also using the disk?
1. walrus01 · 1789845079 · focus · HN ↗
  It's a fair point but it's also possible the person running the tests has a dedicated test hypervisor for this , so that different configurations of filesystems and VMs can be created and destroyed quickly in an automated manner.
  If it's something as simple as a KVM hypervisor that only runs 1 test VM at a time (with no other load from anything else other than the basic systemd daemons, ssh daemon etc running on the hypervisor), the results could be very close to bare metal.
  I can see it being very time consuming and annoying to do repeated manual bare metal OS installs and new partitioning/filesystem creation for such a large variety of tests.
  The author does also say that performance isn't really the main thing but rather, data integrity:
  <a href="https://github.com/fenio/modern-fs-benchmark" rel="nofollow">https://github.com/fenio/modern-fs-benchmark
  1. toast0 · 1789845759 · focus · HN ↗
    > I can see it being very time consuming and annoying to do repeated manual bare metal OS installs.
    Well don't do that then. There's lots of other options. Probably the simplest is a single bare metal install on a simple filesystem on one device. run the filesystems under test on other storage dedicated to testing.
    You could also boot into a network install and use local storage exclusively for testing.
    Farmadupe · 1789845933 · focus · HN ↗
    Yes exactly. The issueThe epherrality of the VMs isn't an issue, it's the _shared_ part that's the concern here. Going by the fact that the kernel is listed as "kernel 7.0.0-1012-azure" I feel like it's a fair risk that there may have been noisy neighbours.
    fenio · 1789848617 · focus · HN ↗
    Obviously they could and probably were noisy neighbours. I'm not trying to hide that fact. Initial step for every benchmark is test of underlying device to at least reject completely unlucky cases. Also after almost 600 runs average is probably more or less correct...
    Also take a look at tests on real hardware. There are not many of them but there are some. I pointed to them in my first answer.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.