The author of the benchmark here. I went over some comments and I'll try to tackle them here.
I'm pretty clear that GH runner based benchmark is far from perfect due to noisy neighbours etc. Thus every test first is running so called calibration... to reject completely unreliable VMs.
I'm fully aware that this can't completely fix the issue. Can limit it but not fix.
But as of now there are 593 runs recorded so average should still be quite meaningful.
Having that said I'm desperately trying to get REAL hardware to run that benchmark. With some successes ;)
But this new hardware has also so many disks that the plan is to try also more complex, tiered cache topologies. I'm working on it.
I'm happy to answer any other questions, sources of every piece of this benchmark are freely available and I'm not saying they are 100% correct. I'm open to improvements.
I went back and forth with Hetzner a couple times, I think we just got a bad machine :)
I've been saying it for months, but eventually I'm going to move the automated builds off the 48 core monster and we'll be able to use that for automated perf testing too. The machine we just got has spindles for EC perf testing, but the Hetzner monster has very high end enterprise ssdd.
Love to hear that there will be more real hw tests.
At the moment I am building NAS and used your benchmark for evaluating the filesystems.
I am glad to see that your data roughly matches mine (apart from scrub which on 4x 6tb HDDs took 15 hours for md-raid10 while CoW systems took seconds).
Personally I found that array of HDDs behaves very differently than GH runner (my feeling is that since it runs on same disk you are testing theoretical throughput rather than ability to utilize disks).
My tests gave an idea for following topologies:
* 4 HDDs (for example dm-raid has read balancing optimized specifically for HDDs)
* 5 HDDs (classical raid should see no improvement but btrfs and bcachefs should balance the load)
* 4 SSDs
* 3 HDDs + 1 SSD no tiering
* 2 HDDs + 2 SSD no tiering
* 1 drive 10x larger than others (since how bcachefs and btrfs allocators work)
* nocow
zfs-style scrub only reads allocated blocks and skips unallocated blocks; if the pool is mostly empty it can complete very quickly.
Layered storage systems with a RAID layer that makes N disks look like one big disk generally don't have visibility into which blocks are free and which are allocated so they must "scrub" all the disks on initialization and repair even if only 1% is used.
I have three identical Lenovo SFF PCs with a U.2 SSD in each. I'm currently running them in a Ceph cluster but I'll be tearing that down soon. I could run some benchmarks with three in one box and report back? Would be a one-time thing rather than an on-going commitment though.
Disregard, I just saw you have some RAID10 tests in there so three SSDs won't be enough.
1. Dual Ext4 + external 32GB journal X4 pcie SSD (the prior winner of benchmark surveys)
2. Bare F2FS after a trim and SSD vendor software cache flush operation (it should be slower, but knowing how much slower on identical hardware could be interesting.)
3. DRBD across a 48U 100Gbps host rack (single X4 pcie data drive per host, OS on primary)
4. CephFS across a 48U 100Gbps host rack (single X4 pcie data drive per host, OS on primary)
2. If you have the time, adding XFS + mdraid + dm-integrity [1] (in bitmap mode) as a comparison point against ZFS RAID-Zx might be an interesting data point. That's what I run, personally.
3. Did you give some thoughts to the I/O scheduler choice? Might matter a lot in some cases.
1. XFS reflink is enabled and its reflink/CoW-break measurements do run. The dashboard button currently means “native/full-CoW filesystem family”, not “supports reflink”, but that distinction is not clear from the label. I’ll rename it to “Native CoW” and add a separate reflink-capable filter that includes XFS.
2. The current integrity comparison is XFS on LVM/dm-raid10 with dm-integrity in its default journal mode. It is not mdraid and not bitmap mode, so your suggested stack would be a genuinely different and useful data point. An md RAID5/6 over per-member bitmap-mode dm-integrity comparison against RAID-Z1/Z2 makes sense, with the weaker post-crash bitmap semantics documented.
3. I did not pin or record the scheduler, which is a reproducibility gap. The dedicated SAS machine currently has mq-deadline active on all HDDs and SSDs. I’ll add queue/scheduler metadata to results before considering separate scheduler variants, since it can strongly affect the mixed and latency-sensitive phases.
Interesting although I'd have liked more summaries: there's an awful lot there.
But the reasons I choose filesystems are more about reliability, failure modes, surrounding tooling, and so on.
Btrfs fails in several critical areas:
1. No way to accurately find free space
2. catastrophic failure on write if a volume fills up, the probability of which is greater because of #1
3. repair tools usually do not recover a corrupted volume and in my testing are most likely to render as damaged volume completely unreadable, which makes #2 worse
Put these things together and I can never trust Btrfs again. In the 9 years since I encountered these, I see no effort to fix them, just fooling around witg unimportant side details like performance tweaks.
Fix the critical issues first then make it faster.
This. Btrfs blew up without ANY reason at all in my case. Rebooted and the system won't even recognize any filesystem. All btrfs tools fails to recover a single file.
I used ZFS until it self-corrupted and refused to mount rw ever again. Community support was totally unhelpful and there was no resolution except buy another array and use something else. There's way too much ZFS cult fanboy glazing out there it doesn't deserve.
It's hard to show with any accuracy how likely a filesystem is to not break when the SHTF or something weird happens, or if they've handled all the weird corner cases, with any kind of automated test.
For that you have to dig into the methodology, look at the code, look at user reports, etc.
But you can get a pretty good approximation just from the philosophies and attitudes of the engineers and what they're talking about.
The talk I just gave at the Rust for Linux conference was all about that - how do we make the system debugable, the community aspect of how we respond to bug reports and talk to users, the prep work for the Rust conversion and formal verification and how we're approaching all that.
Reliability doesn't come out of nowhere, "all bugs are shallow with enough eyeballs" really doesn't apply to filesystems. You just have to plan for it, come up with a methodology, and do the work.
It's not materially better now. The devs are in denial about the problems because lots of big users are saying "works fine on my machine."
Sure, if you have lots of backups, if you have huge volumes on huge disks and they never fill up...
But it's the default in Fedora, Spiral Linux, Garuda Linux, siduction and others. Personal distros for people's own PCs and those are not well-supported enterprise kit.
Seeing great results from bcachefs just makes me more sad that Kent and the other kernel devs couldn't come to an understanding to keep bcachefs in-tree. I want to use it for my storage arrays so badly, but I'm still stuck with btrfs as the only available in-tree filesystem with modern features.
Personally I've had 0 issues with the bcachefs dkms packages from distro repos. Unlike zfs it keeps up with upstream kernel releases so it's far less hassle than running zfs dkms.
> Seeing great results from bcachefs just makes me more sad that Kent and the other kernel devs couldn't come to an understanding to keep bcachefs in-tree.
I think you are sugarcoating the shit show that was bcachefs's history of involvement in the linux kernel. I mean, do I need to mention that the person was subjected to a code of conduct enforcement action due to his long history of abuse and unprofessional behavior?
> CI runs use loop devices on shared ephemeral VMs (one VM per filesystem): compare shapes and ratios, not absolute MB/s. Each job records a host-calibration anchor — see the table.
I think if you're not using baremetal for such tests, it's likely that the results are simply not comparable at all? What if another tenant is also using the disk?
It's a fair point but it's also possible the person running the tests has a dedicated test hypervisor for this , so that different configurations of filesystems and VMs can be created and destroyed quickly in an automated manner.
If it's something as simple as a KVM hypervisor that only runs 1 test VM at a time (with no other load from anything else other than the basic systemd daemons, ssh daemon etc running on the hypervisor), the results could be very close to bare metal.
I can see it being very time consuming and annoying to do repeated manual bare metal OS installs and new partitioning/filesystem creation for such a large variety of tests.
The author does also say that performance isn't really the main thing but rather, data integrity:
> I can see it being very time consuming and annoying to do repeated manual bare metal OS installs.
Well don't do that then. There's lots of other options. Probably the simplest is a single bare metal install on a simple filesystem on one device. run the filesystems under test on other storage dedicated to testing.
You could also boot into a network install and use local storage exclusively for testing.
Yes exactly. The issueThe epherrality of the VMs isn't an issue, it's the _shared_ part that's the concern here. Going by the fact that the kernel is listed as "kernel 7.0.0-1012-azure" I feel like it's a fair risk that there may have been noisy neighbours.
Obviously they could and probably were noisy neighbours. I'm not trying to hide that fact.
Initial step for every benchmark is test of underlying device to at least reject completely unlucky cases.
Also after almost 600 runs average is probably more or less correct...
Also take a look at tests on real hardware. There are not many of them but there are some. I pointed to them in my first answer.
In this case, given that the author's own disclaimer (above) already disclaims the numeric readings, I'm not sure how it's possible to make any inference on "shapes and ratios" derived from the numeric readings.
For the main linked benchmark there are no dedicated test hypervisors. It's all based on GH runners with all the limitations and quirks that came with it.
So two filesystems that are essentially shunned from the Linux kernel and permanent second-class citizens, and one that was removed from Red Hat and has a questionable history of reliability. Oh boy which do I choose?
There's also not a lot of money in Canonical for someone (Oracle) to sue for. Red Hat (now IBM/Red Hat) and I guess even SUSE is a different equation. Red Hat just wouldn't touch ZFS because of licensing.
it is big enough that it prevents distributions to include zfs. and that in reality is the problem. i would not care much if all distributions decided to ignore the copyright issue like canonical does and i could therefore get reliable support for zfs. but alone the thought of having to boot a rescue system to fix my broken computer and then not having zfs support because no one built a rescue system with it included is making the risk to high.
Distributions aren't locked from distributing ZFS, the only group that actually is suing is few kernel developers w/ their lawyers who have non-standard understanding of copyright (including one that isn't norm for the linux kernel, including for filesystems).
who is at fault doesn't really matter. the end result is that distributions aren't including it, and that kills it. unlike almost anything else which i can add to a system after booting it, filesystems need to be available from the start. until debian, fedora, suse, etc are convinced that there is no threat this just won't happen.
I learned about VDO today. [0][1] I'd never heard of it before, but I'm using Gentoo Linux and both the dm-vdo kernel module and the 'vdo' software provided by [1] are distributed by Gentoo... so this isn't some weirdo Red Hat thing. It looks like you manage and used these just like any other thin-provisioned LVM volume [2], but -like I said- I've not used this before, and have only just skimmed the docs, so it's possible that I'm missing something important.
I don't see that as odd. It looks like it was released at about the same time that zstd was getting added to filesystems, and a little bit prior to when IBM acquired Red Hat. It's quite possible that IBM decided that "little" things like enhancing LVM aren't worth the money.
BCacheFS is the best Linux filesystem now that storage costs a premium.
You can mix devices of different sizes and types on bcachefs. You can have foreground and background devices to balance performance and also different compression settings for foreground and background transactions.
You can set replicas=N to the individual file or directory on bcachefs. For example files you can just re-download or re-build. Likewise you can set a higher number of copies to important files.
If you have "refine until it's perfect and don't screw with things you don't understand" thoroughly ingrained, along with the dangers of overconfidence, you'll do fine with AI.
Not everyone gets it though, that's for sure.
And, if you want to know if it's mature, I'd trust the user reports over the one liners :)
I've been using ZFS on my Linux servers for years and have pretty good experience with it. I've not been following Linux development too much. Is bcachefs usable/stable/reliable enough to replace it?
It's not as battle-tested like ZFS with 15+ years of field usage, but it's good enough for me and other users.
There's a NAS appliance called NASty and it shares publicly usage stats: https://nasty-telemetry.pages.dev/
Those numbers are NASty alone. There are more users on other systems.
15+ years of ZFS code is also often said as being a nearly unmanageable pile of mess, supposedly to the point that there’s some of the code that maintainers are afraid of touching, and with plenty of unresolved weird bugs (which I experienced myself with data loss).
Sometimes starting fresh with one coherent codebase and all features design baked in from the start might be better.
It takes 10+ years for any storage engine to iron out all the wrinkles, get it to a state where it's both fast enough, and you can trust it to not lose data. It's stupid to throw it away once it reaches that point.
People keep saying it takes 10 years, but that presupposes methods that never improve. Why would we keep doing the same thing over and over again? Wouldn't be much point in that.
> There's always one of two of you rabble going "You dared defy the kernel community!
No. It's very simple. 1) do QA and follow the release cycle, 2) act like a person when interacting with other developers.
If you fail to do both, you should not be surprised that people go through great lengths to not have to deal with you and all your nonsense.
The fact that after all this time you still feel compelled to gaslight everyone with this victim mentality shows that you still have a long way to go to, and a lot of soul searching to do.
Sir Overstreet still seems insistent on having done no wrong on account of fixing user bugs taking priority over literally everything, so clearly there’s a mindset clash that’s gonna prevent a re-merging unless that Torvalds decides to change the same procedure he’s been doing for the past 2 decades or Overstreet knocks his own ego down a peg.
> I can’t remember anyone earning the level of hostile treatment he did and getting kicked out.
You don't seem to be following it very closely, as apparently you failed to come across the history of abuse and hostile behavior of bcachefs's maintainer.
What is md-raid10 doing that is so much worse than lvm-raid10? In terms of "I/O" and "responsiveness." It's not really obvious to me from either the linked page or https://github.com/fenio/modern-fs-benchmark . In principle they should be similar?
I think the reviews should also include the social aspect of these filesystems...
There is and have been many promising and exciting FS to replace the old boring ones, but for storage you not only want to avoid technical issues but also maintainer(s) drama...
drama matters because most users don't want to have their favourite FS randomly removed from the kernel unexpectedly after some OS update :)
That said I certainly hope that one day the technical advantage of bcachefs will be so overwhelming that maybe the decision to remove it will be overturned. And if big vendors make it their default FS the bus factor will disappear (even if unofficially you'd still be the sole maintainer, but no one cares about that in the enterprise world...)
> most users don't want to have their favourite FS randomly removed from the kernel
True. But when you look at this, isn't the deeper problem that Linux remains such a monolith, and there's a stark difference between "included" and "not included" in the kernel. It's now over 35 years old. The fact that we can't have stable APIs and develop more out-of-tree drivers is not a strength, it's a weakness.
Even old Unix systems like SVR4 managed to have stable, public driver interfaces, despite being rather proprietary. FreeBSD manages to have drivers in its ports tree, with a stable API for a given major version. What makes Linux so special that it can't manage this?
I understand all of the arguments about why this has to be so. But... they might have made sense in the early days, but after 35 years it screams of immaturity. Plenty of other systems, including other open source systems, manage to do this, including having versioned interfaces so things aren't set in stone. Linux remains right at the extreme end of guaranteeing nothing. I've long thought this was unnecessary and counterproductive.
the downside of a stable API is more non-free drivers. the API being unstable is a strength in that it forces driver developers to use a GPL compatible license in order to get their drivers into the kernel, or it motivates the FOSS community to develop alternative drivers that are GPL compatible.
and, i believe i read somewhere that this is not an inability to stabilize the API but a conscious decision to not promise a stable API expressly because it creates the effect i described.
To answer the original question, most people who care about their filesystem at all care about its stability. Not just "does it work now" but also "will it work and improve over time". Infighting puts the future at risk.
But you might want to check out the bus factor on btrfs too; when a maintainer says "but we've saved Facebook billions and billions of dollars!", calls for the other filesystem maintainer to be ejected from the community, then quits to join Anthropic a month later - that's not a vote of confidence.
I'd be very happy if people could just stop bringing up drama and us factors. We put it behind us a year ago, but it seems not everyone got the memo.
Other things: since no companies sponsoring its development had any interest in raid 5/6, which is mostly a home enthusiast thing.
Western Digital picked up this work last year and is slowly getting through the new design without the write hole, I think we'll see working raid 5/6 in the next year or two.
> Other things: since no companies sponsoring its development had any interest in raid 5/6, which is mostly a home enthusiast thing.
I've used RAID-Z1/2/3 at a couple of jobs: yes IOps is suck-y, but if it's for backups of other systems, or the central logging server, or network monitoring (Suricata, Snort), sometimes your priority is cheap/bulk.
I'm currently in the HPC space, and Lustre is a thing here, and it has tiered storage via policies: you can (e.g.) put your recent/hot data on NVMe, but older/colder bits on spinning rust on ZFS.
Might end in a new bcachefs fork beeing included back into mainline. If companies sponsor a few fulltime devs working on the new fork, we finally might have a stable enterprise ready CoW filesystem.
That's what I meant, it's always risky to rely on a single person for such an important subsystem as storage... I'm also very sad that bcachefs has been kicked out of the kernel, it is really a very promising FS.
So that leaves btrfs which always seemed complex and brittle to me (compared to zfs at least)
> Every push/2-hourly cron builds each filesystem across 4 loop devices backed by sparse files, runs the suite, and publishes a results table in the job summary plus JSON artifacts.
I get that real hardware costs (author mentions EUR 70 a month for a suitable server), but without at least a baseline snapshot comparison run between real hardware, both SSD and HDD, and the sparse file-backed loop devices, it's hard to take much away from this.
Sadly the AI apocalypse isn't making stuff like this easy to do as a hobby.
That might be the best thing happened to the project since now development can happen at its own pace without the clicky bait influencers.
In fact they delivered the erasure coding for parity raid back in march this year.
The thing is that as soon as you seriously give a chance to Bcachefs you see how good it is. I can only tell you that mixing different device tiers and having a per-file/directory replication setting is a god send specially in these times where storage costs more than gold.
> The thing is that as soon as you seriously give a chance to Bcachefs you see how good it is.
I'm pretty sure Bcachefs is amazing and better than Btrfs. I also think Zfs is amazing and better than Btrfs. Even so, I still use Btrfs because I know it is guaranteed to always be present on any Linux without any effort on my part.
> That might be the best thing happened to the project since now development can happen at its own pace without the clicky bait influencers.
A better approach might have been to just paused mainline merging instead of forcing being kicked out?
Eg "Hey Linus, Bcachefs is still in early development and I need to merge changes in a pace that is not compatible with Linux development process. So I'm going to pause for a while now and once it reaches maintenance status I will focus on submitting patches in a healthy pace that you can digest".
btrfs has one critical issue they don't fix: it blocks access to fs for minutes if you remove large files. I am not sure how this is acceptable for prod grade fs..
> it blocks access to fs for minutes if you remove large files.
How large is large? I've deleted files with sizes of tens to hundreds of GBs and not seen that, and can probably whip up a test with a single-digit TB file if motivated.
Do you perhaps have 'discard=sync' in your mount options, or are using a kernel earlier than 6.2, which is the version -according to the docs- where async discard became the default?
> How large is large? I've deleted files with sizes of tens to hundreds of GBs and not seen that, and can probably whip up a test with a single-digit TB file if motivated.
for 1TB compressed (probably 5tb uncompressed) it is reproducable 100% reliably for me.
I made a ~3TB btrfs FS and mounted it with 'force-compress', put 5TB of zeros on it (which compressed down to like 160GB), and did a delete along with some concurrent operations on that same FS. Based on what I saw, btrfs doesn't "block access for minutes" while a large delete is in progress, but access to the volume that has the delete in progress is dreadfully slow. I used vim to create a new file in the mountpoint and saw that write delays were between ten and twenty seconds. Really bad, but still functional. Not at all blocked.
Someone in that Reddit discussion that you linked to says that all btrfs filesystems hang during an extremely large delete. This is not what happens for me. The only btrfs FS made slow was the one that had the ongoing delete. I have four other btrfs filesystems mounted and they're all just fine, whether or not they're on the same physical disk that has the ongoing delete. space_cache is v2 on all of my btrfs filesystems.
On my system, it looks like an events_unbound kworker was eating 100% of a single CPU while the big delete was in progress. No other kernel threads seemed to be consistently occupied.
For fun, I re-ran the thing I document below when mounted without compression, and then with an uncompressable file when mounted with non-forced compression. I had to reduce the size of the file to 2TB for both scenarios, but omitting compression writes out like 10x the data to disk, so it still seems like a fair test.
I'm not going to the trouble to provide a transcript for those two runs, but both when mounted without compression enabled and when an uncompressable file was written to a compression-not-forced volume, I saw absolutely no delays in filesystem operations while I was deleting that 2TB file. FWIW, putting 2TB of /dev/zero on that compression-not-forced volume and deleting it behaved the same as it did on a 'force-compress' mount.
Whatever is causing the dreadful slowness is directly linked to transparent compression, rather than being something you get when you run btrfs in all configurations. "Why run btrfs if not for transparent compression?" you might ask. I would answer: "Snapshots and reflinks, and yes, I know that XFS has reflinks too.".
A lightly-edited terminal transcript follows for if you want to double-check my work up to the end of the 'force-compress' run.
bcachefs on Arch is a bit better supported, we have the distro package maintainer in the bcachefs IRC channel, and I've never lagged on mainline support like ZFS has.
Actual distro support, and doing it right with people actually communicating with each other, has always been a priority for the project.
> I still use Btrfs because I know it is guaranteed to always be present on any Linux without any effort on my part.
I used to think that about ReiserFS, too. It was in the mainline kernel, development was snappy, and it solved some performance problems. I used it all over the place.
This sounds like FUD, do you have any references? Genuinely asking. I follow LWN reporting religiously, which in turn follows Fedora development (and associated drama) closely, and haven't seen anything said in this direction. Just had a quick look on LWN and Fedora development resources, and nothing came up.
This is what I read here and elsewhere bunch of times, which is why I said "supposedly". A quick search did not reveal any meaningful proof, though. I'll ask next time I hear someone say this.
unlikely. reference please. btrfs was the default filesystem on fedora installs last time i checked. to remove it they would have to first change that, then give it a few years before even considering removing it from the kernel. redhat could remove it because it was never default and never recommended.
This is what I read here and elsewhere bunch of times, which is why I said "supposedly". A quick search did not reveal any meaningful proof, though. I'll ask next time I hear someone say this.
I think in a perfect world they should had put someone in between to mediate and curate patches while providing DKMS for urgent patches.
As for BTRFS I think its also pretty good. Its just that I have the impression its development is guided by the needs of its sponsors and sadly for us META doesn't need RAID5.
A lot of things were tried, people did try to mediate.
The particularly galling thing though was when I finally started looking - post split - comparing bcachefs PRs to other subsystems and especially XFS - I was being more conservative with what I considered a critical bugfix.
There was never a clear statement on what the issue was. What you guys got in public was about as much as I got.
All I can say is - going fast when you're stabilizing and getting bugfixes out the door is what you can and should be doing when you've invested in test coverage, test automation, keeping the codebase clean and asserted, and building up a community that works well together on testing and shaking things out.
bcachefs was already working at its own pace prior to being accepted in the kernel. It could have continued doing so for years until it was really "ready".
Instead it got kicked out because Kent constantly ignored the kernel's contribution rules and is unlikely it will ever be accepted back into the kernel.
I'd really appreciate it if we could drop the FUD over contribution rules. There are no such rules, it is explicitly Linus's way or the highway, and I already replied to that elsewhere.
And it went in when it did because Redhat was pushing for it and claiming to be supportive - but that never materialized. They wanted to get something for free without investing, or putting in the absolute bare minimum.
A _lot_ of people were saying publicly and privately "dear god yes we need something better than btrfs" - but no one from the existing kernel community was interested in stepping up.
Community's still growing, though. A lot of people have gotten active in making sure bcachefs actually works well for people end to end, and there's a hell of a lot more to shipping a filesystem than just writing kernel code.
You can't let Reddit guide your technical decisions.
The FS was marked experimental, so there is no urgency in fixing bugs or providing features in a certain cycle. Everyone using it knows what they got themselves into. You can still provide the DKMS module for faster fixes and features for anyone who wants to use BCacheFS more seriously for the time that the upstreaming process takes, but eventually it would have all been on mainline.
Asahi is taking a similar approach where they have their downstream kernel and push things upstream once they are mature.
That means the upstream kernel is not useful for running on that hardware now, but things are moving there eventually.
Not something I'd want to do with my filesystem though. ZFS is different story it it isn't on the root fs because it might be packaged by the distribution.
also tricks to make it easy to convert a root FS to ZFS now that Ubuntu Server 24.04 added native root-on-zfs support: https://github.com/pirate/zfsify
The numbers are quite interesting, does bcachefs run on Almalinux 10? I only see Fedora supported. All my servers run Almalinux today and I am super happy with it, no plans to use something else.
How is root support for bcachefs? This is the one thing I really miss with ZFS today, its just too much work. I do love my FreeBSD systems with a root ZFS though.
The current “Integrity” label is broader than the test actually proves, and I’m changing it to “Corruption probe.”
FAIL means that file changed or became unreadable. SURVIVED means only that the file remained readable and hash-identical. It does not prove the entire filesystem was healthy, that every overwritten byte was allocated, or that all 2 GiB were repaired.
Depending on your current setup, dRaid can give you quite the performance boost nowadays, since its data slabs are spread out over all disks in the array its using all of the disks bandwith combined.
The script[1] shows it's a ZFS mirror (RAID-1) with 8k max record size, as opposed to the default 128k.
I say max, as ZFS stores multiples of the physical block size up to the record size. So even with 128k record size, if your file fits in two physical blocks, it will write two physical blocks of data.
The `ashift` parameter[2] controls the physical block size, typically 512 bytes or 4k for HDDs. Though higher can be useful on SSDs (but less tested, have seen some bug reports with >4k block sizes).
Typically you'd only want to use such small records if you are tuning it for a specific workload, like if you have a database that writes 8k pages.
Compression acts on records, so by limiting the record size, you limit the effectiveness of the compression: 1.2 and 1.8 blocks worth of data both gets written as 2 physical blocks.
On the upside, record size is a dataset property, so you can have many different datasets with different record sizes on the same pool.
At this point with these results I wonder if there’s big corp backing for bcachefs yet? I’d imagine they’d want to migrate to it at some point for some of their uses.
Great to hear. Keep up the great work and I hope that with time and project maturity, I.e. when the code changes are more of the maintenance than “new feature” nature, it makes its way back to the mainline just fine.
Also hope Proxmox decides to incorporate it somewhat soon in PVE.
It could use some explanation for the exact setups. Like how many drives are there in each raid, what does the ec config actually look like, etc. As far as I can see it's not described.
@farlight assuming that you're the creator do you think you'd be able to rework the HTML/CSS? I'm sure you've got good data but speaking on behalf of my eyeballs, the results page is... hard to read!
> 2G of random garbage is written directly onto one member device (behind the filesystem's back, offset 1G — python injector; uutils dd mis-seeks on dm devices), caches dropped, then a full scrub: btrfs scrub -B, zpool scrub + wait, bcachefs scrub, md/lvm sync-action 'check' (which can only COUNT mismatches — no checksums to know which copy is right).
I'm not sure that nuking 2G of the underlying block device is a recoverable error on any filesystem that I'm aware of? Can you confirm if any ofthe filesystems really came out of the other side in a usable state after scrubbing?
-----
> Trivial-op p99, idle (ms) # A trivial operation — one 4k write + fsync every 200ms (like a shell appending history or an editor updating its swap file) — run alone for 10s. p99 of the fsync completion
In fact, if it's OK for me to ask, are any of the metrics tht you used standard industry metrics? It looks like several of the tests are bypassing the kernel's page cache? -- which I worry may fall into the trap of "I modified the system to be unrepresentative of reality and then tested it".
----
> kernel 7.0.0-1012-azure
Can you confirm if you tested on a bare metal machine? were you the only tenant?
> I'm not sure that nuking 2G of the underlying block device is a recoverable error on any filesystem that I'm aware of?
ZFS and btrfs were designed from the start to handle this, by using checksums on every piece of (meta)data and redundancy to return the same data as was stored to the kernel, and rewrite the bad data.
I've tested my own machines running ZFS by random writes out of band from the filesystem/kernel and it has always found and fixed them.
I answered in other thread how exactly integrity check looks like and why the name might be misleading. I'm also in the middle of updating descriptions to be more adequate. Thanks for pointing that out.
Speaking about standard industry metrics.
fio is an established tool, and throughput, IOPS, fsync latency, and percentiles are standard concepts. However, the exact job recipes and the composite “Overall Core” score are project-specific.
The trivial-operation test is also custom: one 4 KiB write plus fsync every 200 ms. The idle window contains at most about 50 operations, so its p99 is effectively the slowest sample’s fio histogram bucket, not a statistically stable population percentile. It should be treated as a small-write durability-latency probe, not a universal application metric.
But after all all tests are in the repository. If they need tweaks, changes I'm open to do so... I started from scratch and did whatever came to my mind. Some tests are added after my initial link here which went mostly unnoticed several weeks ago but I got some requests for more tests which I implemented.
But to sum up. I want this test to be useful so feel free to open PRs with improvements. It's not like I've got some agenda. In fact I wrote here and there on the page that I'm counting on communities of various filesystems to provide improvements, changes etc to make their filesystem shining.
This is personal project made when I realized that multiple-devices benchmarks were almost completely absent. Since I had not access to real hardware I decided to make at least initially everything based on GH runner with all the limitations that came with this approach. I tried to limit these limitations as far as I could. But feel free to submit bugreports, PRs, propositions for improvements.
I'd actually recommend some excellent books on the "philosophy" of data presentation: The first that comes to mind is "the visual display of quantitative information" by Edward Tufte seems to be freely available online, and the other one on my mind is "how charts lie" by Alberto Cairo (which doesn't seem to be freely accessible)
But if it helps, just some "initial gut feel observations" from me:
* It's definitely not possible to find issue with the the _sheer amount_ of results, but there's just far too much for a human to absorb, all presented at once
* Overall text size is quite small, and difficult to read
* The page doesn't make a strong statement of _what_ is under test: the first words are: "modern-fs-benchmark Multi-device CoW filesystems under workloads classic benchmarks skip" -- which defines the webpage in terms of what it is _not_, without stating what benchmarks are actually present.
* The first line of teh page contains run statistics that probably eithre want to b at the bottom, or just don't need to be in the webpage at all: "latest run 2026-09-18 18:50:45 UTC, kernel 7.0.0-1012-azure, 593 runs recorded · 145 trend points shown"
* A significant proportion of the free text is caveats. There's nothing wrong with being transparent about limitations, but they may be a sign that there might be alternative ways to present the data, or that the data may be flawed (depending on the caveat)
* Theres several categories that I think have been invented for the purpose of collation, but I don't think are defined on the page. I think "Overall Core" and "Core I/O" aren't explained, which means by definition it's impossible for a reader to understand the score table.
* And as we're all aware right now, current Claude models are currently struggling to write coherent English. There's several incoherent sentences on the page. It's a Claude issue.
Speaking as someone who consumes this, I appreciate how it's laid out. As a developer, we can often see at a glance where the bottleneck is if we have enough data laid out - IOW, data overload for you is me feeling like a kid in a candy store.
Sometimes there are ways to make things easier without dumbing them down, but way too many people conflate the two; I get nervous when non engineers say "I've studied this, it should be easy".
ah, yes that's obviously fine; I'm definitely coming fromt he perspective of "bored guy on a Saturday evening who happened to stumble across someone else's wall of charts on HN"
I think the Phoronix test suite does pretty well for the end user who wants something easy to digest; the exact choice of benchmarks is sometimes odd but the harness itself is quite good.
At some point I'd like to get our own automated pts runs going, since Michael is not consistent with what hardware he tests on and he hasn't been consistent with getting them out.
It's not easy to show so many data and make everyone happy about the way it is presented. In fact I'm aiming more at engineers and trying to provide as much info as it is possible to be clear about methodology and everything around it.
But raw data is in JSON files so you can always make PR for creating additional view aimed at philosophers and not engineers ;)
It'll be interesting when people start applying similar effort that is going into LLM search for security vulnerabilities to finding conventional reliability/performance bugs.
In filesystem subject a proper critique is mentioning ssd vs hdd. I did not added value for karma increase, but karma lost? Who haven't I wished a happy birthday?
The author of the benchmark here. I went over some comments and I'll try to tackle them here. I'm pretty clear that GH runner based benchmark is far from perfect due to noisy neighbours etc. Thus every test first is running so called calibration... to reject completely unreliable VMs. I'm fully aware that this can't completely fix the issue. Can limit it but not fix. But as of now there are 593 runs recorded so average should still be quite meaningful.
Having that said I'm desperately trying to get REAL hardware to run that benchmark. With some successes ;)
Few months ago I got Hetzner machine from Kent Overstreet and I was able to finish 3 runs before machine died... Results: https://bartosz.fenski.pl/modern-fs-benchmark/real-hw/
Currently I've got even more interesting machine with tons of disks and I'm running new set of benchmarks but it's really in its initial stage.
https://bartosz.fenski.pl/modern-fs-benchmark/sas-hdd/ 2nd run in progress... one run on REAL hardware takes much more time than on GH runner so it's slow.
But this new hardware has also so many disks that the plan is to try also more complex, tiered cache topologies. I'm working on it.
I'm happy to answer any other questions, sources of every piece of this benchmark are freely available and I'm not saying they are 100% correct. I'm open to improvements.
I went back and forth with Hetzner a couple times, I think we just got a bad machine :)
I've been saying it for months, but eventually I'm going to move the automated builds off the 48 core monster and we'll be able to use that for automated perf testing too. The machine we just got has spindles for EC perf testing, but the Hetzner monster has very high end enterprise ssdd.
Also, just got done with the Rust for Linux conference, still not home but here's slides that still need reformatting: https://evilpiepirate.org/~kent/Kangrejos-2026-bcachefs.pdf
Love to hear that there will be more real hw tests. At the moment I am building NAS and used your benchmark for evaluating the filesystems. I am glad to see that your data roughly matches mine (apart from scrub which on 4x 6tb HDDs took 15 hours for md-raid10 while CoW systems took seconds). Personally I found that array of HDDs behaves very differently than GH runner (my feeling is that since it runs on same disk you are testing theoretical throughput rather than ability to utilize disks). My tests gave an idea for following topologies:
Thanks for awesome workHow can a scrub take only seconds when it has to read all the data?
zfs-style scrub only reads allocated blocks and skips unallocated blocks; if the pool is mostly empty it can complete very quickly.
Layered storage systems with a RAID layer that makes N disks look like one big disk generally don't have visibility into which blocks are free and which are allocated so they must "scrub" all the disks on initialization and repair even if only 1% is used.
I have three identical Lenovo SFF PCs with a U.2 SSD in each. I'm currently running them in a Ceph cluster but I'll be tearing that down soon. I could run some benchmarks with three in one box and report back? Would be a one-time thing rather than an on-going commitment though.
Disregard, I just saw you have some RAID10 tests in there so three SSDs won't be enough.
Should include:
1. Dual Ext4 + external 32GB journal X4 pcie SSD (the prior winner of benchmark surveys)
2. Bare F2FS after a trim and SSD vendor software cache flush operation (it should be slower, but knowing how much slower on identical hardware could be interesting.)
3. DRBD across a 48U 100Gbps host rack (single X4 pcie data drive per host, OS on primary)
4. CephFS across a 48U 100Gbps host rack (single X4 pcie data drive per host, OS on primary)
Best regards =3
Also
5. a ZFS dRaid configuration. There could be very different characteristics there with it using slabs.
Speaking of slabs, MS ReFS of you feel adventurous!
6. modern nvme drive, preferrably pcie gen5 that can push >14GiB/sec
Good job and clean website interface! Something as exhaustive as this is clearly needed.
Some remarks:
1. Why does the CoW button remove XFS from the list? Even https://github.com/fenio/modern-fs-benchmark/blob/main/scrip... mentions it has reflink enabled
2. If you have the time, adding XFS + mdraid + dm-integrity [1] (in bitmap mode) as a comparison point against ZFS RAID-Zx might be an interesting data point. That's what I run, personally.
3. Did you give some thoughts to the I/O scheduler choice? Might matter a lot in some cases.
[1] https://www.kernel.org/doc/html/latest/admin-guide/device-ma...
Thanks, all three are fair points.
1. XFS reflink is enabled and its reflink/CoW-break measurements do run. The dashboard button currently means “native/full-CoW filesystem family”, not “supports reflink”, but that distinction is not clear from the label. I’ll rename it to “Native CoW” and add a separate reflink-capable filter that includes XFS.
2. The current integrity comparison is XFS on LVM/dm-raid10 with dm-integrity in its default journal mode. It is not mdraid and not bitmap mode, so your suggested stack would be a genuinely different and useful data point. An md RAID5/6 over per-member bitmap-mode dm-integrity comparison against RAID-Z1/Z2 makes sense, with the weaker post-crash bitmap semantics documented.
3. I did not pin or record the scheduler, which is a reproducibility gap. The dedicated SAS machine currently has mq-deadline active on all HDDs and SSDs. I’ll add queue/scheduler metadata to results before considering separate scheduler variants, since it can strongly affect the mixed and latency-sensitive phases.
You so smart
Interesting although I'd have liked more summaries: there's an awful lot there.
But the reasons I choose filesystems are more about reliability, failure modes, surrounding tooling, and so on.
Btrfs fails in several critical areas:
1. No way to accurately find free space
2. catastrophic failure on write if a volume fills up, the probability of which is greater because of #1
3. repair tools usually do not recover a corrupted volume and in my testing are most likely to render as damaged volume completely unreadable, which makes #2 worse
Put these things together and I can never trust Btrfs again. In the 9 years since I encountered these, I see no effort to fix them, just fooling around witg unimportant side details like performance tweaks.
Fix the critical issues first then make it faster.
This. Btrfs blew up without ANY reason at all in my case. Rebooted and the system won't even recognize any filesystem. All btrfs tools fails to recover a single file.
Here is my journey: https://forum.cgsecurity.org/phpBB3/viewtopic.php?p=39143
I switched to ZFS and never Bad a Problem again.
I used ZFS until it self-corrupted and refused to mount rw ever again. Community support was totally unhelpful and there was no resolution except buy another array and use something else. There's way too much ZFS cult fanboy glazing out there it doesn't deserve.
Now that's something I've not heard of before. On what OS? What happened, as far as you know or can tell?
It would mount read-only, then? So you could copy your stuff off onto other media? Because if so, it's better than my experiences with Btrfs.
Does the report say any of this? I only see "FAIL" on a few tests with ext4 and one with xfs.
It's hard to show with any accuracy how likely a filesystem is to not break when the SHTF or something weird happens, or if they've handled all the weird corner cases, with any kind of automated test.
For that you have to dig into the methodology, look at the code, look at user reports, etc.
But you can get a pretty good approximation just from the philosophies and attitudes of the engineers and what they're talking about.
The talk I just gave at the Rust for Linux conference was all about that - how do we make the system debugable, the community aspect of how we respond to bug reports and talk to users, the prep work for the Rust conversion and formal verification and how we're approaching all that.
Reliability doesn't come out of nowhere, "all bugs are shallow with enough eyeballs" really doesn't apply to filesystems. You just have to plan for it, come up with a methodology, and do the work.
It's not just me and it's had serious problems for years:
https://arstechnica.com/gadgets/2021/09/examining-btrfs-linu...
<- 5Y ago.
It's not materially better now. The devs are in denial about the problems because lots of big users are saying "works fine on my machine."
Sure, if you have lots of backups, if you have huge volumes on huge disks and they never fill up...
But it's the default in Fedora, Spiral Linux, Garuda Linux, siduction and others. Personal distros for people's own PCs and those are not well-supported enterprise kit.
This is why I have fully switched to Bcachefs, which is just way more stable and performant than btrfs
Seeing great results from bcachefs just makes me more sad that Kent and the other kernel devs couldn't come to an understanding to keep bcachefs in-tree. I want to use it for my storage arrays so badly, but I'm still stuck with btrfs as the only available in-tree filesystem with modern features.
Personally I've had 0 issues with the bcachefs dkms packages from distro repos. Unlike zfs it keeps up with upstream kernel releases so it's far less hassle than running zfs dkms.
> Unlike zfs it keeps up with upstream kernel releases
Most normal users and especially servers have no reason to run latest upstream kernels
You can use NASty as NAS appliance. It's based on NixOS, offers flexibility, atomic upgrades and all bells and whistles of bcachefs.
> Seeing great results from bcachefs just makes me more sad that Kent and the other kernel devs couldn't come to an understanding to keep bcachefs in-tree.
I think you are sugarcoating the shit show that was bcachefs's history of involvement in the linux kernel. I mean, do I need to mention that the person was subjected to a code of conduct enforcement action due to his long history of abuse and unprofessional behavior?
https://lwn.net/Articles/999197/
No, you probably didn't need to mention that.
NixOS. You can't go wrong.
What do u mean NixOS? It's an OS not a file system
NixOS thoroughly solves all the external module fragility. They also do distro level CI testing.
> CI runs use loop devices on shared ephemeral VMs (one VM per filesystem): compare shapes and ratios, not absolute MB/s. Each job records a host-calibration anchor — see the table.
I think if you're not using baremetal for such tests, it's likely that the results are simply not comparable at all? What if another tenant is also using the disk?
It's a fair point but it's also possible the person running the tests has a dedicated test hypervisor for this , so that different configurations of filesystems and VMs can be created and destroyed quickly in an automated manner.
If it's something as simple as a KVM hypervisor that only runs 1 test VM at a time (with no other load from anything else other than the basic systemd daemons, ssh daemon etc running on the hypervisor), the results could be very close to bare metal.
I can see it being very time consuming and annoying to do repeated manual bare metal OS installs and new partitioning/filesystem creation for such a large variety of tests.
The author does also say that performance isn't really the main thing but rather, data integrity:
https://github.com/fenio/modern-fs-benchmark
> I can see it being very time consuming and annoying to do repeated manual bare metal OS installs.
Well don't do that then. There's lots of other options. Probably the simplest is a single bare metal install on a simple filesystem on one device. run the filesystems under test on other storage dedicated to testing.
You could also boot into a network install and use local storage exclusively for testing.
Yes exactly. The issueThe epherrality of the VMs isn't an issue, it's the _shared_ part that's the concern here. Going by the fact that the kernel is listed as "kernel 7.0.0-1012-azure" I feel like it's a fair risk that there may have been noisy neighbours.
Obviously they could and probably were noisy neighbours. I'm not trying to hide that fact. Initial step for every benchmark is test of underlying device to at least reject completely unlucky cases. Also after almost 600 runs average is probably more or less correct...
Also take a look at tests on real hardware. There are not many of them but there are some. I pointed to them in my first answer.
> compare shapes and ratios, not absolute MB/s
In this case, given that the author's own disclaimer (above) already disclaims the numeric readings, I'm not sure how it's possible to make any inference on "shapes and ratios" derived from the numeric readings.
For the main linked benchmark there are no dedicated test hypervisors. It's all based on GH runners with all the limitations and quirks that came with it.
Real hardware is used in: https://bartosz.fenski.pl/modern-fs-benchmark/real-hw/ https://bartosz.fenski.pl/modern-fs-benchmark/sas-hdd/
But unfortunatelly it's much more limited number of actual runs. sas-hdd is still in progress so numbers for it should increase over time.
So two filesystems that are essentially shunned from the Linux kernel and permanent second-class citizens, and one that was removed from Red Hat and has a questionable history of reliability. Oh boy which do I choose?
I'm saying ZFS on another OS.
no thanks to red hat. i switched to debian for servers. fedora fortunately still supports btrfs.
zfs is shunned. can't really get around that copyright issue. bcachefs just hit a setback, which i am hopeful will eventually be resolved.
> zfs is shunned. can't really get around that copyright issue
Well except for Ubuntu, one of the most popular Linux distros supporting it.
There's also not a lot of money in Canonical for someone (Oracle) to sue for. Red Hat (now IBM/Red Hat) and I guess even SUSE is a different equation. Red Hat just wouldn't touch ZFS because of licensing.
Nevertheless, Ubuntu is one the most popular linux distributions. So “can’t be done” is obviously not true since it has been done.
One can say it is “dangerous” and they haven’t been sued “yet” but the deed is done already.
So use Ubuntu. Why do you care if other distros consider it too risky given different business positions/models?
> can't really get around that copyright issue.
Eh. Just wait until Oracle goes bankrupt due to their AI misinvestments & see where ZFS rights end up.
The copyright issue is not really that big, except from some devs who decided to sure Ubuntu :V
But there's a LOT of FUD about it.
it is big enough that it prevents distributions to include zfs. and that in reality is the problem. i would not care much if all distributions decided to ignore the copyright issue like canonical does and i could therefore get reliable support for zfs. but alone the thought of having to boot a rescue system to fix my broken computer and then not having zfs support because no one built a rescue system with it included is making the risk to high.
Distributions aren't locked from distributing ZFS, the only group that actually is suing is few kernel developers w/ their lawyers who have non-standard understanding of copyright (including one that isn't norm for the linux kernel, including for filesystems).
who is at fault doesn't really matter. the end result is that distributions aren't including it, and that kills it. unlike almost anything else which i can add to a system after booting it, filesystems need to be available from the start. until debian, fedora, suse, etc are convinced that there is no threat this just won't happen.
That's fair, yes.
I guess my specific "bubble" simply didn't have that issue for a long time
xfs/lvm-raid10 or ext4/lvm-raid10, obviously.
If I'm forced to use Linux, sure.
I could drop bricks on and cord pull those all day and they would not lose data. Which is a small ask for a filesystem IMO.
would want to have compression..
> would want to have compression..
I learned about VDO today. [0][1] I'd never heard of it before, but I'm using Gentoo Linux and both the dm-vdo kernel module and the 'vdo' software provided by [1] are distributed by Gentoo... so this isn't some weirdo Red Hat thing. It looks like you manage and used these just like any other thin-provisioned LVM volume [2], but -like I said- I've not used this before, and have only just skimmed the docs, so it's possible that I'm missing something important.
[0] <https://docs.redhat.com/en/documentation/red_hat_enterprise_...>
[1] <https://github.com/dm-vdo/vdo>
[2] <https://docs.redhat.com/en/documentation/red_hat_enterprise_...>
Yeah, I know that thing, but oddly they support only lz4 compression which is much weaker to zstd used everywhere else.
I don't see that as odd. It looks like it was released at about the same time that zstd was getting added to filesystems, and a little bit prior to when IBM acquired Red Hat. It's quite possible that IBM decided that "little" things like enhancing LVM aren't worth the money.
Its odd to me because it looks trivial to add another compression algorithm implementation
BCacheFS is the best Linux filesystem now that storage costs a premium.
You can mix devices of different sizes and types on bcachefs. You can have foreground and background devices to balance performance and also different compression settings for foreground and background transactions.
You can set replicas=N to the individual file or directory on bcachefs. For example files you can just re-download or re-build. Likewise you can set a higher number of copies to important files.
Sure, when it's mature...
Yeah, and I think Overstreet might be going hard on AI, so not sure how that might effect the project
If you have "refine until it's perfect and don't screw with things you don't understand" thoroughly ingrained, along with the dangers of overconfidence, you'll do fine with AI.
Not everyone gets it though, that's for sure.
And, if you want to know if it's mature, I'd trust the user reports over the one liners :)
I've been using ZFS on my Linux servers for years and have pretty good experience with it. I've not been following Linux development too much. Is bcachefs usable/stable/reliable enough to replace it?
It's not as battle-tested like ZFS with 15+ years of field usage, but it's good enough for me and other users.
There's a NAS appliance called NASty and it shares publicly usage stats: https://nasty-telemetry.pages.dev/ Those numbers are NASty alone. There are more users on other systems.
15+ years of ZFS code is also often said as being a nearly unmanageable pile of mess, supposedly to the point that there’s some of the code that maintainers are afraid of touching, and with plenty of unresolved weird bugs (which I experienced myself with data loss).
Sometimes starting fresh with one coherent codebase and all features design baked in from the start might be better.
It takes 10+ years for any storage engine to iron out all the wrinkles, get it to a state where it's both fast enough, and you can trust it to not lose data. It's stupid to throw it away once it reaches that point.
It's not stupid if the 10+ years code is unmaintainable. Why did you ignore that part?
People keep saying it takes 10 years, but that presupposes methods that never improve. Why would we keep doing the same thing over and over again? Wouldn't be much point in that.
In case someone wonders why bcachefs is sadly no longer part of the mainline kernel. Start reading here: https://lwn.net/Articles/1027289/
Is there any chance bcachefs will be included again in the future?
> Is there any chance bcachefs will be included again in the future?
If a professional and mature developer who understands a release cycle is willing to step up as a maintainer, yes.
There's always one of two of you rabble going "You dared defy the kernel community! You dared defy Linus! Rabble rabble rabble!"
Meanwhile, I no longer have to stress about whether I'll be about to get bugfixes out, actual users seem happy and my life is far better :)
> There's always one of two of you rabble going "You dared defy the kernel community!
No. It's very simple. 1) do QA and follow the release cycle, 2) act like a person when interacting with other developers.
If you fail to do both, you should not be surprised that people go through great lengths to not have to deal with you and all your nonsense.
The fact that after all this time you still feel compelled to gaslight everyone with this victim mentality shows that you still have a long way to go to, and a lot of soul searching to do.
I'm optimistic.
Sir Overstreet still seems insistent on having done no wrong on account of fixing user bugs taking priority over literally everything, so clearly there’s a mindset clash that’s gonna prevent a re-merging unless that Torvalds decides to change the same procedure he’s been doing for the past 2 decades or Overstreet knocks his own ego down a peg.
Edit: I think this makes for some good reading: https://lkml.org/lkml/2025/8/9/427
The key is forming a team of maintainers.
If you want to be a filesystem maintainer, you start out by doing the work involved.
I'm afraid not all opinions are created equal, this isn't design by committee :)
I’ve been following kernel development for something like three years using LWN and other sources.
I can’t remember anyone earning the level of hostile treatment he did and getting kicked out.
I don’t see how it would ever happen.
> I can’t remember anyone earning the level of hostile treatment he did and getting kicked out.
You don't seem to be following it very closely, as apparently you failed to come across the history of abuse and hostile behavior of bcachefs's maintainer.
What is md-raid10 doing that is so much worse than lvm-raid10? In terms of "I/O" and "responsiveness." It's not really obvious to me from either the linked page or https://github.com/fenio/modern-fs-benchmark . In principle they should be similar?
I think the reviews should also include the social aspect of these filesystems...
There is and have been many promising and exciting FS to replace the old boring ones, but for storage you not only want to avoid technical issues but also maintainer(s) drama...
Why do people keep bringing up drama?
The community infighting has sucked, but that's a thing that matters primarily for maintainers.
I think most users just want something that works.
drama matters because most users don't want to have their favourite FS randomly removed from the kernel unexpectedly after some OS update :)
That said I certainly hope that one day the technical advantage of bcachefs will be so overwhelming that maybe the decision to remove it will be overturned. And if big vendors make it their default FS the bus factor will disappear (even if unofficially you'd still be the sole maintainer, but no one cares about that in the enterprise world...)
> drama matters because most users don't want to have their favourite FS randomly removed from the kernel unexpectedly after some OS update :)
Jokes on you then: my favourite FS can't be "randomly removed from the kernel unexpectedly after some OS update" because it never made it in — ZFS. :)
> most users don't want to have their favourite FS randomly removed from the kernel
True. But when you look at this, isn't the deeper problem that Linux remains such a monolith, and there's a stark difference between "included" and "not included" in the kernel. It's now over 35 years old. The fact that we can't have stable APIs and develop more out-of-tree drivers is not a strength, it's a weakness.
Even old Unix systems like SVR4 managed to have stable, public driver interfaces, despite being rather proprietary. FreeBSD manages to have drivers in its ports tree, with a stable API for a given major version. What makes Linux so special that it can't manage this?
I understand all of the arguments about why this has to be so. But... they might have made sense in the early days, but after 35 years it screams of immaturity. Plenty of other systems, including other open source systems, manage to do this, including having versioned interfaces so things aren't set in stone. Linux remains right at the extreme end of guaranteeing nothing. I've long thought this was unnecessary and counterproductive.
the downside of a stable API is more non-free drivers. the API being unstable is a strength in that it forces driver developers to use a GPL compatible license in order to get their drivers into the kernel, or it motivates the FOSS community to develop alternative drivers that are GPL compatible.
and, i believe i read somewhere that this is not an inability to stabilize the API but a conscious decision to not promise a stable API expressly because it creates the effect i described.
Related username?
To answer the original question, most people who care about their filesystem at all care about its stability. Not just "does it work now" but also "will it work and improve over time". Infighting puts the future at risk.
It really does.
But you might want to check out the bus factor on btrfs too; when a maintainer says "but we've saved Facebook billions and billions of dollars!", calls for the other filesystem maintainer to be ejected from the community, then quits to join Anthropic a month later - that's not a vote of confidence.
I'd be very happy if people could just stop bringing up drama and us factors. We put it behind us a year ago, but it seems not everyone got the memo.
Checking the bus factor:
Btrfs regulars:
- 1 from Meta
- 1 from Oracle
- 4 from SuSe
- 2 from WDC
bcachefs:
- Kent Overstreet
And Btrfs still does not have a RAID-5/6 that they themselves recommend for usage:
* https://btrfs.readthedocs.io/en/latest/btrfs-man5.html#man-b...
What have they been doing for the last decade(+)?
Other things: since no companies sponsoring its development had any interest in raid 5/6, which is mostly a home enthusiast thing.
Western Digital picked up this work last year and is slowly getting through the new design without the write hole, I think we'll see working raid 5/6 in the next year or two.
> Other things: since no companies sponsoring its development had any interest in raid 5/6, which is mostly a home enthusiast thing.
I've used RAID-Z1/2/3 at a couple of jobs: yes IOps is suck-y, but if it's for backups of other systems, or the central logging server, or network monitoring (Suricata, Snort), sometimes your priority is cheap/bulk.
I'm currently in the HPC space, and Lustre is a thing here, and it has tiered storage via policies: you can (e.g.) put your recent/hot data on NVMe, but older/colder bits on spinning rust on ZFS.
Might end in a new bcachefs fork beeing included back into mainline. If companies sponsor a few fulltime devs working on the new fork, we finally might have a stable enterprise ready CoW filesystem.
Uhh, do you know what the upstream stance is on testing and fixing bugs?
I doubt there'd be any real interest in a bastardized fork that only exists so the deep pocketed vendors can get away with code dump and run.
I don't care about the drama, I just want to say thank you for your continued work to advance the state of the art in open file systems.
Appreciate it :)
Reiserfs was the OG drama fs
That's what I meant, it's always risky to rely on a single person for such an important subsystem as storage... I'm also very sad that bcachefs has been kicked out of the kernel, it is really a very promising FS.
So that leaves btrfs which always seemed complex and brittle to me (compared to zfs at least)
> Every push/2-hourly cron builds each filesystem across 4 loop devices backed by sparse files, runs the suite, and publishes a results table in the job summary plus JSON artifacts.
I get that real hardware costs (author mentions EUR 70 a month for a suitable server), but without at least a baseline snapshot comparison run between real hardware, both SSD and HDD, and the sparse file-backed loop devices, it's hard to take much away from this.
Sadly the AI apocalypse isn't making stuff like this easy to do as a hobby.
First hybrid tier run: https://bartosz.fenski.pl/modern-fs-benchmark/sas-hdd/hybrid... It is FIRST run... I will probably start tuning it now. And I'm open for suggestions what and how to tune ;)
Oh, so bcachefs is doing pretty well.
Except for the fact that the developer has sabotaged the project into being removed from mainline?
That might be the best thing happened to the project since now development can happen at its own pace without the clicky bait influencers.
In fact they delivered the erasure coding for parity raid back in march this year.
The thing is that as soon as you seriously give a chance to Bcachefs you see how good it is. I can only tell you that mixing different device tiers and having a per-file/directory replication setting is a god send specially in these times where storage costs more than gold.
> The thing is that as soon as you seriously give a chance to Bcachefs you see how good it is.
I'm pretty sure Bcachefs is amazing and better than Btrfs. I also think Zfs is amazing and better than Btrfs. Even so, I still use Btrfs because I know it is guaranteed to always be present on any Linux without any effort on my part.
> That might be the best thing happened to the project since now development can happen at its own pace without the clicky bait influencers.
A better approach might have been to just paused mainline merging instead of forcing being kicked out?
Eg "Hey Linus, Bcachefs is still in early development and I need to merge changes in a pace that is not compatible with Linux development process. So I'm going to pause for a while now and once it reaches maintenance status I will focus on submitting patches in a healthy pace that you can digest".
> Even so, I still use Btrfs because I know it is guaranteed to always be present on any Linux without any effort on my part.
100%. My system is rock solid and the last thing I need is rolling the dice after every update on whether my system will boot. https://www.reddit.com/r/archlinux/comments/eywcp7/linux_551...
I'm impressed with bcachefs's accomplishments though, and if they ever reconcile with the kernel I'll surely give it a fair shake.
btrfs has one critical issue they don't fix: it blocks access to fs for minutes if you remove large files. I am not sure how this is acceptable for prod grade fs..
> it blocks access to fs for minutes if you remove large files.
How large is large? I've deleted files with sizes of tens to hundreds of GBs and not seen that, and can probably whip up a test with a single-digit TB file if motivated.
Do you perhaps have 'discard=sync' in your mount options, or are using a kernel earlier than 6.2, which is the version -according to the docs- where async discard became the default?
> Do you perhaps have 'discard=sync'
I have 'discard=async'
> How large is large? I've deleted files with sizes of tens to hundreds of GBs and not seen that, and can probably whip up a test with a single-digit TB file if motivated.
for 1TB compressed (probably 5tb uncompressed) it is reproducable 100% reliably for me.
Here is some discussion: https://www.reddit.com/r/btrfs/comments/1mok440/filesystem_l...
Thanks for the information.
I made a ~3TB btrfs FS and mounted it with 'force-compress', put 5TB of zeros on it (which compressed down to like 160GB), and did a delete along with some concurrent operations on that same FS. Based on what I saw, btrfs doesn't "block access for minutes" while a large delete is in progress, but access to the volume that has the delete in progress is dreadfully slow. I used vim to create a new file in the mountpoint and saw that write delays were between ten and twenty seconds. Really bad, but still functional. Not at all blocked.
Someone in that Reddit discussion that you linked to says that all btrfs filesystems hang during an extremely large delete. This is not what happens for me. The only btrfs FS made slow was the one that had the ongoing delete. I have four other btrfs filesystems mounted and they're all just fine, whether or not they're on the same physical disk that has the ongoing delete. space_cache is v2 on all of my btrfs filesystems.
On my system, it looks like an events_unbound kworker was eating 100% of a single CPU while the big delete was in progress. No other kernel threads seemed to be consistently occupied.
For fun, I re-ran the thing I document below when mounted without compression, and then with an uncompressable file when mounted with non-forced compression. I had to reduce the size of the file to 2TB for both scenarios, but omitting compression writes out like 10x the data to disk, so it still seems like a fair test.
I'm not going to the trouble to provide a transcript for those two runs, but both when mounted without compression enabled and when an uncompressable file was written to a compression-not-forced volume, I saw absolutely no delays in filesystem operations while I was deleting that 2TB file. FWIW, putting 2TB of /dev/zero on that compression-not-forced volume and deleting it behaved the same as it did on a 'force-compress' mount.
Whatever is causing the dreadful slowness is directly linked to transparent compression, rather than being something you get when you run btrfs in all configurations. "Why run btrfs if not for transparent compression?" you might ask. I would answer: "Snapshots and reflinks, and yes, I know that XFS has reflinks too.".
A lightly-edited terminal transcript follows for if you want to double-check my work up to the end of the 'force-compress' run.
The lightly-edited transcript, relocated to a second comment to maybe avoid folding up the report:
Only if you have quotas enabled IIRC.
no quota enabled
bcachefs on Arch is a bit better supported, we have the distro package maintainer in the bcachefs IRC channel, and I've never lagged on mainline support like ZFS has.
Actual distro support, and doing it right with people actually communicating with each other, has always been a priority for the project.
> I still use Btrfs because I know it is guaranteed to always be present on any Linux without any effort on my part.
I used to think that about ReiserFS, too. It was in the mainline kernel, development was snappy, and it solved some performance problems. I used it all over the place.
Things then subsequently... changed. :-/
You had plenty of time to move away from reiserfs. Something like 15 years between the conviction and removal.
> I still use Btrfs because I know it is guaranteed to always be present on any Linux without any effort on my part.
Except RHEL. They don’t include it in their kernels.
Alma Linux started including it again though.
It can never be easy.
Came here to say this. Supposedly Fedora also considers giving up on it.
This sounds like FUD, do you have any references? Genuinely asking. I follow LWN reporting religiously, which in turn follows Fedora development (and associated drama) closely, and haven't seen anything said in this direction. Just had a quick look on LWN and Fedora development resources, and nothing came up.
This is what I read here and elsewhere bunch of times, which is why I said "supposedly". A quick search did not reveal any meaningful proof, though. I'll ask next time I hear someone say this.
unlikely. reference please. btrfs was the default filesystem on fedora installs last time i checked. to remove it they would have to first change that, then give it a few years before even considering removing it from the kernel. redhat could remove it because it was never default and never recommended.
This is what I read here and elsewhere bunch of times, which is why I said "supposedly". A quick search did not reveal any meaningful proof, though. I'll ask next time I hear someone say this.
I think in a perfect world they should had put someone in between to mediate and curate patches while providing DKMS for urgent patches.
As for BTRFS I think its also pretty good. Its just that I have the impression its development is guided by the needs of its sponsors and sadly for us META doesn't need RAID5.
Meta doesn't have anyone working on btrfs anymore, it appears to be two guys at SuSE and drive bys.
Not true, Boris Burkov from Meta works on btrfs. And two people from WDC.
Within the last year? I think bcachefs is going to be overtaking btrfs soon on active developers, from the trends I saw in the commit logs
I was very up front about where we were at.
A lot of things were tried, people did try to mediate.
The particularly galling thing though was when I finally started looking - post split - comparing bcachefs PRs to other subsystems and especially XFS - I was being more conservative with what I considered a critical bugfix.
There was never a clear statement on what the issue was. What you guys got in public was about as much as I got.
All I can say is - going fast when you're stabilizing and getting bugfixes out the door is what you can and should be doing when you've invested in test coverage, test automation, keeping the codebase clean and asserted, and building up a community that works well together on testing and shaking things out.
I genuinely do not know what they were thinking.
Just pleases try to get it back into mainline.
> There was never a clear statement on what the issue was. What you guys got in public was about as much as I got.
If you are referring to why bcachefs was removed from the Linux kernel, here's a discussion on bcachefs being removed from the Linux kernel.
https://news.ycombinator.com/item?id=44868868
I'd like to point out that HN user koverstreet was involved in those threads, here.
They already know what was discussed.
(Good? Bad? Indifferent? I don't know and I don't have a dog in this race. I'm just here connecting the dots.)
> I'd like to point out that HN user koverstreet was involved in those threads, here.
Yes, that's why those remarks on how it's a mystery how bcachefs was pulled from the kernel are perplexing. To me they sound like gaslighting.
bcachefs was already working at its own pace prior to being accepted in the kernel. It could have continued doing so for years until it was really "ready".
Instead it got kicked out because Kent constantly ignored the kernel's contribution rules and is unlikely it will ever be accepted back into the kernel.
That would be a shame since there is nothing else over there with the same set of features. Disregarding drama, I'm telling you it’s that good.
I'd really appreciate it if we could drop the FUD over contribution rules. There are no such rules, it is explicitly Linus's way or the highway, and I already replied to that elsewhere.
And it went in when it did because Redhat was pushing for it and claiming to be supportive - but that never materialized. They wanted to get something for free without investing, or putting in the absolute bare minimum.
A _lot_ of people were saying publicly and privately "dear god yes we need something better than btrfs" - but no one from the existing kernel community was interested in stepping up.
Community's still growing, though. A lot of people have gotten active in making sure bcachefs actually works well for people end to end, and there's a hell of a lot more to shipping a filesystem than just writing kernel code.
You can't let Reddit guide your technical decisions.
The FS was marked experimental, so there is no urgency in fixing bugs or providing features in a certain cycle. Everyone using it knows what they got themselves into. You can still provide the DKMS module for faster fixes and features for anyone who wants to use BCacheFS more seriously for the time that the upstreaming process takes, but eventually it would have all been on mainline.
Asahi is taking a similar approach where they have their downstream kernel and push things upstream once they are mature.
That means the upstream kernel is not useful for running on that hardware now, but things are moving there eventually.
No urgency over fixing bugs? What do you think this is, btrfs? :)
All this has been discussed to death, we don't need people armchair quarterbacking a year later. It's over, it's time to move on.
It's relatively easy to get it working as a kernel module at least. I got it set up on a NixOS box without too much trouble.
Not something I'd want to do with my filesystem though. ZFS is different story it it isn't on the root fs because it might be packaged by the distribution.
there are tricks to avoid needing distro support / kernel modules for ZFS: https://github.com/pirate/zfsbox
also tricks to make it easy to convert a root FS to ZFS now that Ubuntu Server 24.04 added native root-on-zfs support: https://github.com/pirate/zfsify
I hadn't heard of zfsify, but I think I'll check it out on yet another old laptop running ext4.
Fair, I still don’t use it on my main machine, just a shitty laptop that I keep around to play with.
that's not necessarily a sabotage.
Sabotage might not be the best word, but it hurt trust and adoption.
It's just been a lot less drama within the project since the split.
I do have a lot more pull requests to merge than I did before. I don't know if you want to count "Kent isn't reviewing PRs fast enough" as drama :)
https://github.com/fenio/modern-fs-benchmark
The numbers are quite interesting, does bcachefs run on Almalinux 10? I only see Fedora supported. All my servers run Almalinux today and I am super happy with it, no plans to use something else.
How is root support for bcachefs? This is the one thing I really miss with ZFS today, its just too much work. I do love my FreeBSD systems with a root ZFS though.
Don’t know Almalinux, but can vouch for bcachefs root being just fine with NixOS.
what does "integrity" fail mean in the first table? that the case didn't recover from the 2G corruption?
The current “Integrity” label is broader than the test actually proves, and I’m changing it to “Corruption probe.” FAIL means that file changed or became unreadable. SURVIVED means only that the file remained readable and hash-identical. It does not prove the entire filesystem was healthy, that every overwritten byte was allocated, or that all 2 GiB were repaired.
Thanks for pointing it.
I'd be interested to see what the upcoming Btrfs performance improvements in linux 7.3 do for these benchmarks.
https://www.phoronix.com/review/linux-73-btrfs
For peace of mind I'm still using zfs (since the last 15+ years) but I'm definitely not impressed by the performance...
Different tools for different jobs; use ZFS for your data store and ext4 for your primary drive.
yes indeed, zfs for my nas basically
Depending on your current setup, dRaid can give you quite the performance boost nowadays, since its data slabs are spread out over all disks in the array its using all of the disks bandwith combined.
zfs-mirror-8k (whatever that is) looks alright.
The script[1] shows it's a ZFS mirror (RAID-1) with 8k max record size, as opposed to the default 128k.
I say max, as ZFS stores multiples of the physical block size up to the record size. So even with 128k record size, if your file fits in two physical blocks, it will write two physical blocks of data.
The `ashift` parameter[2] controls the physical block size, typically 512 bytes or 4k for HDDs. Though higher can be useful on SSDs (but less tested, have seen some bug reports with >4k block sizes).
Typically you'd only want to use such small records if you are tuning it for a specific workload, like if you have a database that writes 8k pages.
Compression acts on records, so by limiting the record size, you limit the effectiveness of the compression: 1.2 and 1.8 blocks worth of data both gets written as 2 physical blocks.
On the upside, record size is a dataset property, so you can have many different datasets with different record sizes on the same pool.
[1]: https://github.com/fenio/modern-fs-benchmark/blob/599ec72fe3...
[2]: https://openzfs.github.io/openzfs-docs/Performance%20and%20T...
At this point with these results I wonder if there’s big corp backing for bcachefs yet? I’d imagine they’d want to migrate to it at some point for some of their uses.
Yes, has been for awhile
Great to hear. Keep up the great work and I hope that with time and project maturity, I.e. when the code changes are more of the maintenance than “new feature” nature, it makes its way back to the mainline just fine.
Also hope Proxmox decides to incorporate it somewhat soon in PVE.
I'm interested in seeing these results on cloud instances. I'd expect rather extreme differences in latency.
It could use some explanation for the exact setups. Like how many drives are there in each raid, what does the ec config actually look like, etc. As far as I can see it's not described.
I extended info about setup. Should be visible in top part of benchmark.
Excuse my ignorance, but why didn't you benchmark on a RAM disk for more accurate results?
xfs is by far the slowest in "ftruncate empty file to 1G" but the fastest in "ftruncate 256M file to 512M" - does anyone know why?
Seems like a bug in benchmark code for that test, opened GH issue https://github.com/fenio/modern-fs-benchmark/issues/35
Should be fixed. Thanks.
@farlight assuming that you're the creator do you think you'd be able to rework the HTML/CSS? I'm sure you've got good data but speaking on behalf of my eyeballs, the results page is... hard to read!
> 2G of random garbage is written directly onto one member device (behind the filesystem's back, offset 1G — python injector; uutils dd mis-seeks on dm devices), caches dropped, then a full scrub: btrfs scrub -B, zpool scrub + wait, bcachefs scrub, md/lvm sync-action 'check' (which can only COUNT mismatches — no checksums to know which copy is right).
I'm not sure that nuking 2G of the underlying block device is a recoverable error on any filesystem that I'm aware of? Can you confirm if any ofthe filesystems really came out of the other side in a usable state after scrubbing?
-----
> Trivial-op p99, idle (ms) # A trivial operation — one 4k write + fsync every 200ms (like a shell appending history or an editor updating its swap file) — run alone for 10s. p99 of the fsync completion
In fact, if it's OK for me to ask, are any of the metrics tht you used standard industry metrics? It looks like several of the tests are bypassing the kernel's page cache? -- which I worry may fall into the trap of "I modified the system to be unrepresentative of reality and then tested it".
----
> kernel 7.0.0-1012-azure
Can you confirm if you tested on a bare metal machine? were you the only tenant?
> I'm not sure that nuking 2G of the underlying block device is a recoverable error on any filesystem that I'm aware of?
ZFS and btrfs were designed from the start to handle this, by using checksums on every piece of (meta)data and redundancy to return the same data as was stored to the kernel, and rewrite the bad data.
I've tested my own machines running ZFS by random writes out of band from the filesystem/kernel and it has always found and fixed them.
I answered in other thread how exactly integrity check looks like and why the name might be misleading. I'm also in the middle of updating descriptions to be more adequate. Thanks for pointing that out.
Speaking about standard industry metrics.
fio is an established tool, and throughput, IOPS, fsync latency, and percentiles are standard concepts. However, the exact job recipes and the composite “Overall Core” score are project-specific.
The trivial-operation test is also custom: one 4 KiB write plus fsync every 200 ms. The idle window contains at most about 50 operations, so its p99 is effectively the slowest sample’s fio histogram bucket, not a statistically stable population percentile. It should be treated as a small-write durability-latency probe, not a universal application metric.
But after all all tests are in the repository. If they need tweaks, changes I'm open to do so... I started from scratch and did whatever came to my mind. Some tests are added after my initial link here which went mostly unnoticed several weeks ago but I got some requests for more tests which I implemented.
But to sum up. I want this test to be useful so feel free to open PRs with improvements. It's not like I've got some agenda. In fact I wrote here and there on the page that I'm counting on communities of various filesystems to provide improvements, changes etc to make their filesystem shining.
This is personal project made when I realized that multiple-devices benchmarks were almost completely absent. Since I had not access to real hardware I decided to make at least initially everything based on GH runner with all the limitations that came with this approach. I tried to limit these limitations as far as I could. But feel free to submit bugreports, PRs, propositions for improvements.
1 device out of the replica set I’m assuming so all of them should recover.
The integrity check is only on the tests which are either RAID or the filesystem equivalent.
what exactly would you like to improve?
I'd actually recommend some excellent books on the "philosophy" of data presentation: The first that comes to mind is "the visual display of quantitative information" by Edward Tufte seems to be freely available online, and the other one on my mind is "how charts lie" by Alberto Cairo (which doesn't seem to be freely accessible)
But if it helps, just some "initial gut feel observations" from me:
* It's definitely not possible to find issue with the the _sheer amount_ of results, but there's just far too much for a human to absorb, all presented at once
* Overall text size is quite small, and difficult to read
* The page doesn't make a strong statement of _what_ is under test: the first words are: "modern-fs-benchmark Multi-device CoW filesystems under workloads classic benchmarks skip" -- which defines the webpage in terms of what it is _not_, without stating what benchmarks are actually present.
* The first line of teh page contains run statistics that probably eithre want to b at the bottom, or just don't need to be in the webpage at all: "latest run 2026-09-18 18:50:45 UTC, kernel 7.0.0-1012-azure, 593 runs recorded · 145 trend points shown"
* A significant proportion of the free text is caveats. There's nothing wrong with being transparent about limitations, but they may be a sign that there might be alternative ways to present the data, or that the data may be flawed (depending on the caveat)
* Theres several categories that I think have been invented for the purpose of collation, but I don't think are defined on the page. I think "Overall Core" and "Core I/O" aren't explained, which means by definition it's impossible for a reader to understand the score table.
* And as we're all aware right now, current Claude models are currently struggling to write coherent English. There's several incoherent sentences on the page. It's a Claude issue.
Speaking as someone who consumes this, I appreciate how it's laid out. As a developer, we can often see at a glance where the bottleneck is if we have enough data laid out - IOW, data overload for you is me feeling like a kid in a candy store.
Sometimes there are ways to make things easier without dumbing them down, but way too many people conflate the two; I get nervous when non engineers say "I've studied this, it should be easy".
ah, yes that's obviously fine; I'm definitely coming fromt he perspective of "bored guy on a Saturday evening who happened to stumble across someone else's wall of charts on HN"
I think the Phoronix test suite does pretty well for the end user who wants something easy to digest; the exact choice of benchmarks is sometimes odd but the harness itself is quite good.
At some point I'd like to get our own automated pts runs going, since Michael is not consistent with what hardware he tests on and he hasn't been consistent with getting them out.
It's not easy to show so many data and make everyone happy about the way it is presented. In fact I'm aiming more at engineers and trying to provide as much info as it is possible to be clear about methodology and everything around it. But raw data is in JSON files so you can always make PR for creating additional view aimed at philosophers and not engineers ;)
It'll be interesting when people start applying similar effort that is going into LLM search for security vulnerabilities to finding conventional reliability/performance bugs.
clicked, ctrl+f ssd, hdd, not found, closed
The new machine - I mentioned in the other thread - has both HDDs and SSDs so tests with mixed topologies are planed.
In filesystem subject a proper critique is mentioning ssd vs hdd. I did not added value for karma increase, but karma lost? Who haven't I wished a happy birthday?