If your algorithm does a ton of small allocations to the point where the allocator is the bottleneck, you're already doing it wrong. The allocator necessarily comes with a lot of overhead because it needs to accommodate diverse use cases, avoid fragmentation, and ideally, implement a variety of security checks. If you're doing something alloc-intensive, you're probably allocating and freeing a lot of identical structures and you'd be better off grabbing some continuous memory and managing that yourself in a task-specific way.
But the reality is that almost no one actually cares about performance because compute is cheaper than expertise and labor, at least in the short haul. Everything is getting more bloated and slower and we just compensate by adding CPU cores, gigabytes and gigahertz.
No, musl's allocator is just bad even in completely normal programs, and it is especially awful if you are using even two threads much less a lot of them. It has no TLABs or arenas. It has a single global mutex over alloc/free paths. It does syscalls underneath that lock (mmap) meaning the few fast paths it has are rarely taken under contention and have to fall back to futex wakes, so even 2 threads with minor contention and allocation rate will have visible wait points in profiles, stuck waiting for the allocator. It returns mapped memory to the OS very eagerly when a size class is empty, so even single allocs followed by a single free can cause thrashing as it mmaps/unmmaps things repeatedly for a size class over and over. Etc. You quite literally have to limit your thread count when using musl, because it will tank the performance of actually highly threaded programs that can scale with core count, even at very modest allocation rates and small working set sizes.
Its string routines and memory copy routines are also similarly bad, as the article alludes to. They are just naive loops with nearly no optimization. These are not small insignificant functions where using them is "doing it wrong", they are the backbone of vast amounts of code and can be made multiple times faster. You can similarly see string routines pop up in profiles all the time in musl builds in my experience. And unlike the memory allocator these cannot be "fixed" systematically across the application at link time, so you are stuck with it.
Real programs have to often do things like allocate memory and use multiple threads and process strings. People have been optimizing these things for decades, there is vast amounts of prior art, the musl developers simply did not do so because they prioritize simplicity over nearly everything else (from what I can tell) including performance.
It has a single global mutex over alloc/free paths. It does syscalls underneath that lock (mmap)
Every default malloc implementation worked this way about 12 years ago. Making lots of small allocations, even from multiple threads then blaming the allocator is a losing strategy. An allocator is only going to be able to mitigate the damage to speed and interactivity.
The solution is and always has been to make larger allocations and use those efficiently.
They are just naive loops with nearly no optimization.
The compiler should be able to take something with good access patterns and make something fast, especially out of the basic C functions.
they are the backbone of vast amounts of code
Performance wise it's unlikely C string functions are actually the bottleneck in a program. Maybe for specific programs a naive memory copy function could benefit from AVX instructions.
Real programs have to often do things like allocate memory
"Have to" and "often" are debatable. Any allocations in a hot loop are the very first things that should be optimized away after profiling.
>Performance wise it's unlikely C string functions are actually the bottleneck in a program. Maybe for specific programs a naive memory copy function could benefit from AVX instructions.
Many programs use lots of strings. It tends to become a bottleneck. It also tends to be very difficult to improve because the strings are everywhere in that kind of program, and refactoring to eliminate them is either impossible or very risky.
Many programs use lots of strings. It tends to become a bottleneck.
I would dispute this in anything that isn't mostly about string processing and in that case you can always easily grab different string functions, which you should probably do anyway if strings are that important.
It also tends to be very difficult to improve because the strings are everywhere in that kind of program,
I don't know what 'that kind of program' is supposed to mean.
and refactoring to eliminate them is either impossible or very risky.
This doesn't sound like a general purpose statement that applies to anything broadly.
All I'm saying is the the title is wrong and musl doesn't do much to prevent speed in a program. If someone was really trying to optimize, blaming the standard library is not going to get them very far and it's easy to work around, but needing to do that is very rare.
>I would dispute this in anything that isn't mostly about string processing and in that case you can always easily grab different string functions, which you should probably do anyway if strings are that important.
I'm telling you that I've seen it, in stuff as diverse as video games and robotics. Lots of things use strings as values. It's easy to say "just change everything in millions of lines of code" when you aren't the one who has to make that change.
By the way most software does copious amounts of string processing... I think that should be common knowledge, but I guess it isn't.
>This doesn't sound like a general purpose statement that applies to anything broadly.
You sound like you have zero experience. If your config is in strings, and hundreds of thousands of lines of code already rely on the string-ness of the data, then you just can't make the change easily.
>All I'm saying is the the title is wrong and musl doesn't do much to prevent speed in a program. If someone was really trying to optimize, blaming the standard library is not going to get them very far and it's easy to work around, but needing to do that is very rare.
I believe the title is accurate. People in performance-sensitive areas gripe about libraries, even standard libraries, quite often. I don't mean to insult you but you're making bold assertions despite clearly lacking the experience to know how things are done in industry generally.
To be clear, you're saying that in video games and robotics people are using strings instead of numbers and when that becomes a performance problem you think simple C string functions are to blame? How about not using strings as values?
By the way most software does copious amounts of string processing... I think that should be common knowledge, but I guess it isn't.
It isn't because it's not true if "copious" is about CPU time. String processing is rarely the bottleneck.
I believe the title is accurate
Well, it isn't. It's unlikely that musl prevents anyone from making a fast program. It doesn't even make sense. In the off chance anything was a real bottleneck you could bring in something faster and you would want to do that anyway.
People in performance-sensitive areas gripe about libraries, even standard libraries, quite often.
I have done a lot of optimization and I've never seen the standard library be a problem for exactly what I just outlined.
A lot of what you're saying is just "it's a problem because it is, trust me". That isn't evidence or an explanation.
As soon as allocation is slow you can avoid allocations (which you should do anyway) or use a different one (which you would do even with a standard libc anyway).
If you really have string problems (and not some fake problem like just parsing strings over and over instead of caching values) then you would use an optimized library. The benefit from a regular libc over musl is minuscule compared to the real solutions to optimizing.
you're making bold assertions despite clearly lacking the experience to know how things are done in industry generally.
Trying for insults doesn't add any sort of technical explanation. To be very clear any program written by someone who says their memory allocator is their bottleneck is something I could speed up by orders of magnitude and the standard library isn't going to matter.
>To be clear, you're saying that in video games and robotics people are using strings instead of numbers and when that becomes a performance problem you think simple C string functions are to blame? How about not using strings as values?
To bring this back to the article, the problem is actually that one library does this worse than others. If you are unfortunate enough to already rely on these functions performing up to a certain standard, then having them become dramatically worse is in fact an issue.
Why don't you just not use slow functions? Well, that goes back to my references to refactoring. Even if you could get approval to refactor the stuff, it's still risky and a lot of work. Compare that to just not using an oddball standard library with worse performance...
>Well, it isn't. It's unlikely that musl prevents anyone from making a fast program. It doesn't even make sense. In the off chance anything was a real bottleneck you could bring in something faster and you would want to do that anyway.
THAT IS THE POINT OF THE ARTICLE TITLE: If you require performance in certain key areas, MUSL may not be acceptable.
>I have done a lot of optimization and I've never seen the standard library be a problem for exactly what I just outlined.
The article here is literally complaining about a standard library's performance, which is not uncommon in the blog-o-sphere. So, you are ignoring evidence right in your face. People like me are telling you it sometimes matters, and people blog about such problems frequently, but you still aren't getting it.
I can only tell you vaguely about codebases I've worked on. I can't tell you where, or show you code, or anything like that. Get used to it.
>A lot of what you're saying is just "it's a problem because it is, trust me". That isn't evidence or an explanation.
Everything you've said is "It's NOT a problem because I'VE never seen it be a problem!" When the evidence is right in front of your face and people are telling you, yes, it is a problem. Do you think I'm getting paid to share this wisdom with you?
If you don't think string processing is a bottleneck, you should consider how many applications are document-based and string-based. Basically, it's a MAJORITY of applications in the world, and I'd put money on that.
>As soon as allocation is slow you can avoid allocations (which you should do anyway) or use a different one (which you would do even with a standard libc anyway).
More "just refactor bro" or "just use a different library" (the point of the article). Only one of these is likely to be practical in any given situation, especially since MUSL is not the default for most stacks.
>The benefit from a regular libc over musl is minuscule compared to the real solutions to optimizing.
Bro, if the stats in the article are right (and I have no reason to doubt) the difference is significant (at least numerically). It's easy to tell other people to go do a ton of work to optimize. The radically easier solution is to just not use MUSL if that's your problem. Again, the entire point of the article.
>Trying for insults doesn't add any sort of technical explanation.
Saying you're inexperienced is not insulting, especially since you're a stranger. You clearly deny having experience with the stuff I'm talking about, which I think is common-knowledge in optimization circles, and then insist on labor-intensive solutions to easily solved problems. Some insulting thoughts have crossed my mind here but I know we've all been inexperienced at some point, so I'm trying to keep it civil to teach you something.
>To be very clear any program written by someone who says their memory allocator is their bottleneck is something I could speed up by orders of magnitude and the standard library isn't going to matter.
This is youthful arrogance (I can only assume you're young; if not, you at least haven't matured in your career). It's not always possible to do such optimization, from either a technical perspective or a pragmatic one. Most people do not have authority to go on an optimization binge across their codebases, assuming the penalty is even paid by their own code (it often comes from upstream libraries!). Even if you did have the authority, expertise, and time to do the optimization, it could be a horrible idea and introduce a LOT of potential bugs.
I like the idea of MUSL, and wish the project well. I may even use it for something one day. But none of this makes their performance better. It may be that getting better performance would compromise their other objectives, such as simplicity.
If you are unfortunate enough to already rely on these functions performing up to a certain standard, then having them become dramatically worse is in fact an issue.
There are a few problems here. The first saying that anything is dramatically worse. The second is thinking that nothing can be changed. The third is thinking that there are lots of programs out there that spend all their time in C string functions yet nothing can be altered except for linking in a different standard library.
I can't tell you where, or show you code, or anything like that. Get used to it.
I didn't expect at any point that you to be able to back up what you are saying with examples.
Everything you've said is "It's NOT a problem because I'VE never seen it be a problem!"
That couldn't be further from the truth. I'm saying it isn't a problem because the problems are easily fixable and musl doesn't prevent them from being fixed.
You hallucinated a quote and made up something completely different in your head.
If you don't think string processing is a bottleneck, you should consider how many applications are document-based and string-based. Basically, it's a MAJORITY of applications in the world, and I'd put money on that.
You think these programs are bottlenecked by the C-string functions in their standard library? That's a bold claim. Why would a program completely dependent on strings even use C string functions in the first place? You have to scan to a newline to find the length, they work with ascii and they are known to be incredibly insecure. What you're saying doesn't make sense.
You clearly deny having experience with the stuff I'm talking about,
I deny that there are programs that can only be sped up by switching to a different C library and nothing else, since that's nonsense.
just not use MUSL if that's your problem.
People can do whatever they want, all I've ever said is that musl doesn't prevent anyone from making a fast program. That's it.
Saying you're inexperienced is not insulting,
I'll take your word for it because you seem extremely inexperienced at optimizing, especially if you think making fast software should include leaning on C strings and having memory allocation show up on your profiler.
This is youthful arrogance (I can only assume you're young; if not, you at least haven't matured in your career).
You can go for the insults and try to be patronizing again, I expect that as the last resort of someone frustrated that repeating their claim isn't taken as evidence. I explained a lot in detail about why musl isn't going to prevent anyone from writing fast software because I've done it over and over.
You seem to be saying that you can speed up legacy programs somewhat that weren't made well in the first place with a faster libc and I'm sure that's true, but it has nothing at all to do with the premise the musl prevents a program from being fast or even is much of a bump in the road.
The speed ups from a more optimized libc are percentage points shaved off the the times where you actually use it. Using better string functions, minimizing allocations and paying attention to memory access patterns are going to be order of magnitude changes. Minimizing allocations is going to be at least 7x on a single core, better memory access patterns are going to be 20x-25x.
You don't have any evidence or explanation that musl prevents someone from writing fast software, which is the title and the title is wrong.
>There are a few problems here. The first saying that anything is dramatically worse. The second is thinking that nothing can be changed. The third is thinking that there are lots of programs out there that spend all their time in C string functions yet nothing can be altered except for linking in a different standard library.
25% slowdown can be dramatic for some applications. Secondly, I didn't say that nothing can be changed. I said that change is expensive. Thirdly, I think linking another library is probably acceptable to get an easy 25% speedup! This problem was probably discovered by somebody saying "Why is this shit so slow when I link with MUSL?"
Regarding "Nothing can be altered except for linking" -- There are many such cases. This especially happens with upstream code. If you use a library that you aren't willing or able to fork, you have to deal with its limitations. This can happen for open-source projects, or for private commercial projects.
>You hallucinated a quote and made up something completely different in your head.
I summarized your whole position in an ironic quote to show you how dumb it is. I'm sorry you don't see how you come off. Calling my rhetoric "hallucination" is laughable. I could swear I'm arguing with a bot.
>I'm saying it isn't a problem because the problems are easily fixable and musl doesn't prevent them from being fixed.
Your fix suggestion amounts to calling for a huge refactoring, as I said. MUSL does not prevent you from doing that, but it's easier to just not link MUSL if it's causing problems for you.
>You think these programs are bottlenecked by the C-string functions in their standard library? That's a bold claim. Why would a program completely dependent on strings even use C string functions in the first place? You have to scan to a newline to find the length, they work with ascii and they are known to be incredibly insecure. What you're saying doesn't make sense.
Performance-sensitive programs and libraries are often written in C. C-string representation is widely used by all programming languages, which are usually written in C or C++ (which uses C).
>I deny that there are programs that can only be sped up by switching to a different C library and nothing else, since that's nonsense.
This statement is the real nonsense. Again with the "I've never seen it, so it can't exist" bullshit.
>People can do whatever they want, all I've ever said is that musl doesn't prevent anyone from making a fast program. That's it.
No, that's not all you've said. You said the title is wrong. You said (roughly speaking) that the choice of standard library is never a decision point for performance. The title of this article may be a bit exaggerated, but there's a clear example of poor MUSL performance in the article. It's also not JUST slow string functions, it's slow memory allocation too. What's next, you gonna say you've never seen a program that needs lots of memory allocation? Or that I should go fork some upstream project to work around MUSL's limitations?
>You can go for the insults and try to be patronizing again, I expect that as the last resort of someone frustrated that repeating their claim isn't taken as evidence. I explained a lot in detail about why musl isn't going to prevent anyone from writing fast software because I've done it over and over.
You can keep saying it over and over and it won't be any more true. It's true that I'm making claims and you're not accepting them. What you should ask yourself is what I have to gain by making these claims. The answer is nothing. I'm beginning to think you're a troll. Your username certainly suggests it.
>You seem to be saying that you can speed up legacy programs somewhat that weren't made well in the first place with a faster libc and I'm sure that's true, but it has nothing at all to do with the premise the musl prevents a program from being fast or even is much of a bump in the road.
I am CLEARLY saying that. Linking MUSL to any program that heavily uses the slow functions will make it slower. Since MUSL is not the default for most software, this will be observed as totally unnecessary and inexcusable performance degradation. If you're trying to build the fastest version of some software, you should use the fastest libraries.
Your position seems to be that the title is wrong because it is theoretically possible to make MUSL-dependent programs fast according to some unstated performance metric, so the title is necessarily wrong. What you don't see is that no matter what performance metric you choose, if I wrote the program to be fast with MUSL, it would be EVEN FASTER with a faster library. It might be "fast enough" for somebody with MUSL alone. But if that somebody cares about performance (like the title says) then they will use the fastest library they can. They won't refactor all their code to make MUSL work faster. Sometimes the objective of caring about performance is to have literally the fastest thing possible, not just "fast enough".
>You don't have any evidence or explanation that musl prevents someone from writing fast software, which is the title and the title is wrong.
The title doesn't say that MUSL will stop you from writing fast software. I never said that either. The title says "Don't use MUSL if you care about performance." But keep burning that straw man bro. I'm done with this bullshit conversation.
25% slowdown can be dramatic for some applications.
What you keep forgetting is that's 25% of the part that you're actually using.
You keep avoiding and denying that for a program where any sort of speed matters you just aren't spending any of your time hitting the C standard library.
If a program is slow because of the C standard library and it needs to be faster, someone messed up a long time ago. That isn't the C libraries fault. You can squeeze a little more out of it if you have find a more optimized library, but it's all a drop in the ocean compared to how much faster it would go by doing basic real optimizations.
Performance-sensitive programs and libraries are often written in C. C-string representation is widely used by all programming languages, which are usually written in C or C++ (which uses C).
You can say that, but really it's almost all C++ and people avoid C strings for exactly why I outlined in detail. You don't have length up front, it's all ascii, you're dealing with pointers to arbitrary runs of bytes, etc.
This is not good for memory access patterns, dealing with lots of characters at one time, dealing with unicode, minimizing memory allocations etc.
It's true that I'm making claims and you're not accepting them.
That is true that you are making lots of claims and that I'm not accepting them, because you don't have any evidence or explanations and they don't make sense.
All I've said is that MUSL doesn't prevent someone from making fast programs and you barely have even confronted that, let alone explained how it isn't true.
Linking MUSL to any program that heavily uses the slow functions will make it slower.
Any program that is hammering the the C standard library can be sped up by orders of magnitude and 25% is nothing. Again, lifting memory allocations will speed something up by 10x, so that 25% isn't going to matter anymore because it's 25% of something that isn't even going to show up on a profiler, let alone be a bottleneck.
Your position seems to be that the title is wrong because it is theoretically possible to make MUSL-dependent programs fast according to some unstated performance metric
I think you mean it's trivially possible according to the detailed explanation I gave from a lot of experience optimizing.
Sometimes the objective of caring about performance is to have literally the fastest thing possible, not just "fast enough".
25% better might be fast enough for you, I like speeding up programs by 100x by changing to C++ instead of C and focusing on the optimizations that matter.
The title says "Don't use MUSL if you care about performance."
I use musl and I care a lot about performance. It doesn't matter because has no bearing on how fast my programs are.
I'm done with this bullshit conversation.
I'm sure it seems that way when someone doesn't accept the same claim over and over without any actual explanation or evidence. Saying the same thing and getting more upset is not an effective way to make your point. You need real information, not insults and fake quotes.
I awoke today with renewed energy to deal with your nonsense, and I think after reading this comment it deserves a (hopefully) short reply.
>I use musl and I care a lot about performance. It doesn't matter because has no bearing on how fast my programs are.
This is the crux of the matter. You clearly have some investment in MUSL, that much has been apparent all along. If you actually USE it, which I'm sure you do, then you are not using the fastest possible library. Regardless of your skill, simply switching to a faster library will make a program faster. Sometimes, that's what is needed. We JUST want to find easy ways to make code as fast as possible. We aren't looking to rewrite the world.
I don't have any investment in MUSL. I wish the project well, but if it's not the fastest library then it will necessarily be unsuitable for some applications.
>What you keep forgetting is that's 25% of the part that you're actually using.
First off, I didn't forget shit. There are surely programs for which the penalty is actually 25% or close to it. The penalty can actually be far worse than that due to algorithmic complexity issues. Secondly, it is quite possible that one could use the standard library almost exclusively. Thirdly, any performance drop could be significant.
>You can say that, but really it's almost all C++ and people avoid C strings for exactly why I outlined in detail. You don't have length up front, it's all ascii, you're dealing with pointers to arbitrary runs of bytes, etc.
C strings are used extensively in C++. Again you prove how inexperienced you are.
>This is not good for memory access patterns, dealing with lots of characters at one time, dealing with unicode, minimizing memory allocations etc.
Keep deflecting with bullshit that is irrelevant to the choice of library, and that can't be changed but for massive refactoring.
>That is true that you are making lots of claims and that I'm not accepting them, because you don't have any evidence or explanations and they don't make sense.
There's only so much evidence and explanation I can provide to strangers on the internet. Then there is also the level of evidence that I can provide, such as other blog posts, that I don't feel like providing. A lot of the things I've said are true a priori (assuming some simplifications), such as the fact that a slower library will always be worse for performance than a faster library. If MUSL is measurably slower than others, then it can be a problem.
>Any program that is hammering the the C standard library can be sped up by orders of magnitude and 25% is nothing. Again, lifting memory allocations will speed something up by 10x, so that 25% isn't going to matter anymore because it's 25% of something that isn't even going to show up on a profiler, let alone be a bottleneck.
As I've said many times, rewriting software is not always on the table. Even minor changes may be forbidden or burdensome for various reasons. What you're talking about is a redesign, that may not be applicable to all cases anyway.
>>Your position seems to be that the title is wrong because it is theoretically possible to make MUSL-dependent programs fast according to some unstated performance metric
>
>I think you mean it's trivially possible according to the detailed explanation I gave from a lot of experience optimizing.
Right, your position is definitely that making these changes is "trivial"... I threw in "theoretically" because I interjected my own knowledge that these optimizations are often only theoretically possible, and can't be done for many practical reasons. My wrong statement of your position made it more defensible than it actually is.
>25% better might be fast enough for you, I like speeding up programs by 100x by changing to C++ instead of C and focusing on the optimizations that matter.
I don't want to get off on another tangent but you are trivializing and exaggerating a lot of stuff in this one statement. Changing languages and altering the code is not an easy win, and might actually trigger a performance setback. Switching to a faster library is a relatively easy win.
>Saying the same thing and getting more upset is not an effective way to make your point. You need real information, not insults and fake quotes.
I've made my point already, you just haven't accepted it. I assume you are either incapable of understanding (perhaps temporarily), or have unstated biases as I pointed out due to being a MUSL enthusiast or contributor. You're not convincing me either way. My position is that people who care about being as fast as possible should use the fastest available library, which clearly isn't MUSL. There may be other valid reasons to use MUSL, and it may be "fast enough", but other libraries are yet faster.
This is how I see your argument so far:
- MUSL is fast enough, trust me bro.
- Ok, maybe someone measured some functions to be slower, but everyone knows that you shouldn't be using the STANDARD LIBRARY heavily. Nobody has ever used strings or memory allocation that heavily, and if they did then they are doing it wrong (even if other libraries provide adequate performance for them).
- If you have a problem with this, you just need to do some trivial optimizations, or change languages to C++. It's SUPER EASY (at least for elite MFers like myself).
- Because rewriting the code to compensate for MUSL performance is always on the table, it's never reasonable to say that MUSL is causing a performance problem.
I only added a slight amount of emphasis. You've been pretty close to that bombastic in the whole exchange. It's ridiculous, and I think you are smart enough to know better but admitting that you are wrong is beyond the pale. Instead of wasting my time, how about applying those elite optimization skills you claim to have to the MUSL code to make it faster. Then you will be able to write a blog like "Don't use glibc if you care about performance" and argue with people on HN if you want. By the way, I didn't write this blog post, and I don't have any other conflicts of interest such as being a glibc developer, so don't start up with that shit either.
C strings are used extensively in C++. Again you prove how inexperienced you are.
You can try to be patronizing if you want, but this doesn't make sense with you not understanding memory access patterns. If someone uses C++ they will use C++ strings. Have you used C++? It works well and it has been around for 40 years, you should give it a try.
Keep deflecting with bullshit that is irrelevant to the choice of library, and that can't be changed but for massive refactoring.
You painting yourself into a corner has nothing to do with a C library being a few percentage points slower. This is not a general scenario, it's something you are dealing with because you can't optimize C strings.
If you think memory access patterns are 'bullshit' you haven't done much optimization. That's everything in modern optimization.
If there is an expert somewhere around you, please talk to them so you can get up to speed on modern optimization. Minimizing allocations is table stakes, memory access is the meat.
If you have any examples on github of things you think "can't" be optimized, go ahead and link it and tell me what a profiler shows, and I will explain what to do.
I've made my point already, you just haven't accepted it.
You made your claim, you haven't had any evidence or explanation, that's how it works. You say the same thing over and over and get more upset, but that isn't evidence. Show me where musl prevents someone from writing a fast program.
Insults and hallucinated quotes aren't a good foundation for proving your point. If you have to pretend that someone said something different, then maybe you aren't really making sense.
how about applying those elite optimization skills you claim to have to the MUSL code to make it faster.
Why would I do that when I don't use the C library for performance sensitive programs? You never seem to be able to confront this. The stuff made 50 years ago isn't the fastest possible stuff. It's still pretty fast and if you want something faster, do something else. It's not that complicated but it seems to really upset you.
I don't have any other conflicts of interest such as being a glibc developer, so don't start up with that shit either.
No one said anything about that, I think you're hallucinating or predicting something that never happened.
> Every default malloc implementation worked this way about 12 years ago.
Perhaps "default" is doing the heavy lifting here. Glibc malloc was quite bad for a long time, true. But TCmalloc / jemalloc are 21-22 years old, respectively, and jemalloc has been the FreeBSD (released) default malloc implementation for the last 18.
> The solution is and always has been to make larger allocations and use those efficiently.
Having a not-dogshit allocator really doesn't hurt. There's no reason to defend shitty allocator + every application doing manual memory pools on top of it to paper over the bad allocator.
I'm not defending anything, I'm saying it's usually trivial to make allocation time marginal.
+ every application doing manual memory pools
Usually it's simple data structures in flat memory.
If allocation is taking all the time, that's a poorly optimized program with lots of low hanging fruit and a different allocator is not the right fix. It's like having a boat with a hole in the bottom and someone says the solution is a smaller hole.
But TCmalloc / jemalloc are 21-22 years old, respectively, and jemalloc has been the FreeBSD (released) default malloc implementation for the last 18.
jemalloc is also possibly bigger than all of musl. If it was a problem after optimization I would use it and I have in the past, it's just nowhere near as important as minimizing allocations in the first place. OpenBSD uses straight mmap.
There's also a reason this stuff started getting adopted around 20 years ago (I'll add macOS to the pile: it added per-CPU malloc sharding in 2008). It's not just due to overall growth in OS complexity. It's because that's when multicore CPUs were taking off. Before that, the cost of a global lock was far lower.
OpenBSD is straight unconcerned about performance. That's their choice.
> Every default malloc implementation worked this way about 12 years ago.
Yes, it is now 12 years later, and memory allocators are better. The memory allocators of that time were also better than the ones 12 years their prior. That's the point.
> Making lots of small allocations, even from multiple threads then blaming the allocator is a losing strategy. An allocator is only going to be able to mitigate the damage to speed and interactivity.
It's just a reality that musl is measurably worse at multiple threads allocating even in very polite conditions, because it causes lots of contention. If your program allocates in multiple threads, it is probably going to get slower with musl. If you don't want that, other memory allocators will do great even at high allocation rates with more threads. You could write many other data structures that had equally poor behavior under multi-threaded contention by just throwing a lock around everything and calling it a day, and those bad data structures would also cause "damage to speed and interactivity" or whatever. This isn't very hard to understand.
> The compiler should be able to take something with good access patterns and make something fast, especially out of the basic C functions.
I agree, modern compilers are good. But these are extremely common specified functions, they are called everywhere all the time in every C codebase (and that's partially why compilers even recognize these patterns specifically so they can insert optimized routines). Mature implementations that are hand optimized still pay off and also tend to be tuned for various edge cases or quirks that aren't going to come for free from the C compiler either, so it's still work even if you aren't writing assembly for everything or whatever (e.g. uarch dependent codepaths, or optimizations for short strings or whatever).
glibc's AVX2 based memcpy functions have a non-negligible performance impact in at least 1 application I maintain on the order of like 8-ish% vs musl (wall clock). It just has to memcpy/memmove a whole lot. Whether or not that's tolerable is up to debate, but a spade is a spade.
> Performance wise it's unlikely C string functions are actually the bottleneck in a program. Maybe for specific programs a naive memory copy function could benefit from AVX instructions.
I said "backbone", not "bottleneck". They are common functions sprinkled in everywhere throughout every application in every codepath on something like a modern Linux desktop. An inverted callstack flamegraph can show you stuff like this. It is basically no different than compiling your application at -O1 and -O2 with GCC. Does the fact your program get 20% faster from -O2 mean that there were "bottlenecks" the compiler solved? No, there was just performance left on the table by emitting better code.
> "Have to" and "often" are debatable.
Not really. I have to spell it out apparently: actual programs written by normal human programmers do those things, all the time, they exist in and are common in the world, they depend on other code that does that and is common in the world, they run on your desktop and phone and all servers, and they benefit quite a lot from optimized components like memory allocators and string routines and -O2 making their programs faster. This is pretty easy to observe and the means of doing so should be quite obvious, so there's no real debate.
Now whether this fact holds -- whether these programs "have to" do these things or not -- in the imaginary fantasy land people have in their heads where they make up arguments to themselves about how, if every program was written how they liked it, it would be better? That I'm not so sure about, I will admit.
The memory allocators of that time were also better than the ones 12 years their prior. That's the point.
The point is that memory allocation shouldn't be a bottleneck either way. If it is the program needs to be optimized or redesigned. Better allocators give you more slack, they don't solve the problem. If the problem is already solved, then a basic allocator isn't going to make a big performance difference because it isn't the bottleneck.
those bad data structures would also cause "damage to speed and interactivity" or whatever. This isn't very hard to understand.
It depends on how much they are used and how much contention there is. Sometimes putting a mutex around things is fine.
But these are extremely common specified functions, they are called everywhere all the time
Not necessarily, especially for C string functions, but they do get linked in so it's a good thing musl makes them small.
I said "backbone", not "bottleneck".
Then the point is lost, because 'backbone' doesn't mean anything if it works. If it isn't a bottleneck in throughput or latency anywhere then the speed doesn't matter.
The other important thing is that better stuff can be included in pieces as it's needed. The reverse isn't true. If you use a big fat C library, you have a dependency that isn't going to get better.
Not really. I have to spell it out apparently: actual programs written by normal human programmers do those things, all the time,
You spelled it out last time, it's just not true in the sense that programs have to have these functions as bottlenecks. Strings, allocators and memory copying can all be dealt with independently, but again it's rare that strings and allocations really need to be the bottleneck and in those circumstances you probably want more than a different standard library anyway.
in the imaginary fantasy land people have in their heads where they make up arguments to themselves about how, if every program was written how they liked it,
I'm not sure what this is supposed to mean, there is nothing I've said that doesn't make perfect sense. If you want something to go faster you can make it go faster. A better allocator pales in comparison to lifting allocations out of hot loops.
My point is the musl is useful and the disadvantages are easy to work around. I'm not really sure what your point is, do you think people are going to force you to use it?
Your point seems to be that if you use a slow standard library and complain, it's not a problem with the slow standard library because you can just reimplement the slow parts independently.
The problem with your argument is that it's a universal argument against performance. And if an argument is universal, then it doesn't have any information value.
Your point seems to be that if you use a slow standard library and complain, it's not a problem with the slow standard library because you can just reimplement the slow parts independently.
No, this is something that nuanced. The title is wrong because musl isn't going to prevent you from writing fast software.
Whatever benefit there is to a different libc, is absolutely miniscule compared to do actual optimizations like avoiding allocations.
I'll give you real numbers: if you put allocations of short vectors of a dozen floats in a hot loop, when you lift the allocations out your program is going to instantly get about 10x faster. The allocation is no longer going to be the bottleneck, it will be marginal and then a faster allocator isn't going to matter at all.
If someone gets an easy speedup from using a different libc that's great, but the vast majority of time it isn't going to matter and isn't going to be where any real speedups come from. The difference is a small percentage speedup vs orders of magnitude.
The amounts to the title being wrong, using musl or a small standard library just doesn't prevent a program from running fast. It is a tiny difference and even that tiny difference can be changed from things like better allocators which you would do anyway with a standard libc.
Think about it this way because the issue isn’t specific to allocators: it’s pretty good in general but can often be beaten if you have special understanding of what you need to do. That’s OK.
You can buy cars and trucks that are optimized for driving on freeways and residential streets carrying stuff people often carry. But then there are special vehicles like fork lifts and such that are kinds of large special cases. And then there are weirdo specialised vehicles that have four wheels but are rare and their users can’t live without them.
Languages like C++ let you plug in special allocators if you want. But most people don’t. Some, like HFT people do crazy headstands to avoid slow allocations. I don’t ever want to do that but if they want to, why not. I don’t think they complain that the default case doesn’t fit their needs!
For a typical program, I bet that the overall impact of the glibc allocator is well under 0.1%. If you can choose between < 0.1% and < 0.12%, I guess it matters in some sense, but not in any practical way. We almost never spend time on other sub-0.1% optimizations. You could probably squeeze a lot more by optimizing CPU branch predictor performance, minimizing CPU cache misses, or fine-tuning the scheduling strategy, but we also don't bother. 'Tis is the era of "native" apps written in Electron.
A decade ago, I worked on a simulation program that involved a C++ core with a Python wrapper and DB interface. End users cared a lot about throughout with a rather limited size, weight and power budget. We spent a lot of time optimizing the core -- but basically hit a bottleneck once we got to about 15% of the time that was spent in malloc-related functions. It turned out that was all in the Python layer. Probably there was some level of bad code in our Python code, but it was impractical to figure out where that was. I was shocked because I assumed the simulation core (which ended up almost allocation-free by the end) would always dominate CPU usage.
Both allocators and Python have probably gotten better since then, but it was a fascinatingly large and stubborn fraction of CPU time.
"But the reality is that almost no one actually cares about performance because compute is cheaper than expertise and labor, at least in the short haul."
Doesn't have to stay that way, with hardware prices soaring and development cost allegedly in free fall.
Yeah, because that's the exact sort of hardware musl is aimed at.
Here's my counter-bogus-statistic: musl is infinitely faster than glibc on a Nios II processor because musl will fit into memory and glibc won't. Therefore, don't use glibc if you care about your code being able to run. QED.
People have such different perspectives. 26% slower does not sound "terrible" to me; it sounds like quite a reasonable price one might choose to pay for the convenience musl offers. If musl's allocator were 2.6x slower, I might call that "not so great"... but in order to qualify as "terrible" I think the difference would have to be an order of magnitude!
The 26% number at the top of the article is from using mimalloc (which is a high performance allocator, at least as fast as the glibc allocator) + musl for some task, and the slowdown is coming from (probably) slow musl implementations of memcpy/memset. The musl allocator is even worse.
Yeah, doing compute-heavy work a couple of jobs ago, we tried small images with musl, and the default allocator was a catastrophe: 75%+ slowdowns for our real life tasks. Even with a better allocator, we were way better off with the larger image.
> the slowdown is coming from (probably) slow musl implementations of memcpy/memset.
It's wild that such a fundamental piece of code (you can't really implement operation on structs without those) is library-supplied. I wish compilers would just have something like __builtin_memcpy and __builtin_memset, and provided some highly optimized, specialist-crafted assembly in those, instead of having to inline the library code and hopefully be able to optimize it.
Clang and GCC do provide these, and automatically use them in many situations (particularly small copies). But c-libraries can actually do it better in many cases, especially for large copies.
Glibc, for example, has perhaps ten different implementations of memcpy just for x86. The compiler certainly could provide all that, but the next step is harder:
glibc automatically dispatches to the proper one at runtime based on the actual microarchitecture that the binary is running on. You pay the extra dispatch cost once, but all of non-inline function call cost every time. This is what allows distros to compile to a nice baseline architecture, but still get near-optimal memcpy performance on many more architectures than a single inline instance could possibly give. These differences matter.
And it does it for not just memcpy, but half-a-dozen other extremely performance sensitive library functions, like strcpy and so on.
Inlining works very much against this strategy. If you can guarantee that the target microarch never changes, then it isn't a good one. But that is somewhat unusual for everyone but those who build their own binaries to run on a single class of machines forever.
Worse, inlining the really high performance versions of these ends up being terrible from a code size perspective, because they are often hundreds of instructions, which can have bad caching effects. And once you amortize the function-call cost over many iterations of the loop, it isn't so expensive to call out to the library.
Anyway, just some additional considerations to think about.
Even if the attempt is inside of a function called memcpy() which contains no code other than your copy loop, and links with priority over the libc implementation! (as all embedded firmware engineers learn at some point in their journey)
Maybe you're being sarcastic, but I'm pretty sure clang + gcc do offer these.
The problems at first glance :
- Not having control over the implementation detail of the interface that your library provides is probably not wise. Sounds like a lot of bad bug reports and edge cases that you have no control over.
As others have alluded, __builtin_memcpy doesn't resolve to a runtime implementation. GCC and clang treat functions like memcpy specially. Because they're defined by the standard and are reserved names, compilers can assume the exact semantics specified by the standard and elide library calls altogether with optimized inline code. But if the compiler can't do the optimization (can't prove alignment, indeterminate length, etc), it just emits a library call, even if your source has some other local function definition named "memcpy". Explicit use of __builtin_memcpy is treated identically to calls to memcpy, unless the compiler is invoked with -ffreestanding, in which case it only optimizes __builtin_memcpy and skips special treatment of calls to memcpy, but __builtin_memcpy could still expand into a call to memcpy. If you're writing a C library you want to use -ffreestanding. (I think. There may be more nuance. More info at https://gcc.gnu.org/bugzilla/show_bug.cgi?id=56888)
The 26% slower appears to be for their whole application, not just the allocator. For some parts of the application to make the whole this much slower it must mean that those parts are quite a lot slower, likely much more than 2x.
Moreover the 26% is with mimalloc, with musl's allocator it's 144%, so there are likely other parts that are slower (likely the memcpy implementation)
Ops here, I think if you NEED that convenience, sure, rock with MUSL BUT I also see a ton of devs crowing about using MUSL on my 128GB x86 Kubernetes hosts. I have plenty of Disk Space, you can ship glibc based container.
26% slower could turn into a huge hardware bill, and could render the library unusable for some purposes. There are many applications for which 26% is negligible, but it ain't nothing...
... you're fine with trading an app running 26% slower, to save a dozen megabytes ? that sounds positively insane to me. That's accepting to go from e.g. 60fps to ~45 fps (e.g. completely unacceptable)
That would depend on the application. The convenience isn't just a smaller binary, but also being independent of the distros glibc version (being on an LTS distro and not updating often, I have experienced incompatible glibc versions often enough). The megabytes shaved off are significant if the final binary size is < 1MB, which is a completely different size class.
And not all applications are performance sensitive. Something like UNIX's bc command would benefit more from having easier compatibility and a faster startup than more optimized allocators and string functions.
In the end, it ~doesnt even matter~ is all trade-offs. The nice thing is that it is up to the developer to make the decision of which libc to use, so everyone gets what they want.
Musl's allocator being awful is pretty well known, though mostly in that it's absolutely awful in multithreaded context. TFA points out that musl has a bunch of other noticeably slower functions, which is less well known (though they're also slower by a smaller factor, and they don't worsen as your parallelism increases).
Yeah... I've recently had a chance to compare how fgets is implemented in both GNU libc and musl, and, well. With glibc, it was a challenge to even find where the fgets's code actually is.
I haven’t dug into why, but for unknown-linux builds on x86, Rust binaries have been substantially smaller on musl than standard dynamic linking to glibc, for me. No idea if I’m doing something wrong or if the handful of cases I tried were all special in some way.
I really find complaints about the minimum viable placeholder malloc in musl confusing. Does anyone seriously try to use malloc-ng for performance critical use cases?
Our entire linux distro is musl based BUT we swap out the default malloc with mimalloc for high performance because why would you not? Best of both worlds.
If this specific use case is of high interest to you and you have some available bandwidth, contributing to it, maybe becoming a maintainer, and eventually organising a tier 2 MCP would definitely be a good idea.
Note that this is no-std no-alloc target, with all the limitations that leads to.
You could add alloc with a custom global allocator, but I don't even know what high perf global allocator you could use that wouldn't need libc. Jemalloc and mimalloc are out. Some embedded allocators would work (but those are rarely high performance, instead being optimised for small code and data footprints).
That said, with enough effort (quite a lot!) it would be possible to add support for alloc and std without libc on Linux specifically (since it has a stable syscall ABI).
What might be more realistic though is looking at relibc (a rust implementation of libc, made for Redox OS but from what I read it also supports Linux). But I haven't tried it and I don't know the state (or goal) of it.
> it would be possible to add support for alloc and std without libc on Linux specifically (since it has a stable syscall ABI).
Well yes that’s a Linux specific target so that’s kinda the point.
Technically you could do libcless on a few other platforms which are not actively hostile to it (yet) like freebsd, but that would have no chance of getting to tier 2 if it was even accepted.
I just remembered that there is also https://github.com/sunfishcode/eyra (but I think it might be a dead project) which is close to that, it had slipped my mind.
All of these are going to mean you can't link any (non-freestanding) C code, load any dylibs, etc. So you will be fairly limited in what sort of applications you can write. Forget most GUI frameworks, even native ones. You won't be able to load GL or Vulkan drivers for example. You are basically stuck with command line or servers.
I would hazard the guess that that’s perfectly fine. Desirable even. People who run scratch or alpine images and link against musl aren’t usually looking to write desktop applications or video games.
I wouldn't mind having a standalone binary that anyone can download and just run on their Linux desktop, regardless if it is Arch, Alpine or Debian stable.
With glibc that is a pain, I need to build in a container with tthe oldest glibc I want to support, and that still doesn't cover Alpine. And I dont know if a static musl build would even work for that either (if I need to be able to load GUI libraries).
You can do FROM scratch, and use still glibc; it’s just that you need to copy more than one file. I don’t really understand if you are already dealing with images why you still need the image to contain a single file.
I feel with Rust I try and avoid re-allocations in most cases anyway, so I'm not sure that musl's allocator being slow would significantly affect performance (though I haven't benchmarked it). I feel like part of the appeal of Rust is that you can do imperatively-style mutation-heavy code comparatively risk-free, so despite me normally being the "Functional Programming Nerd", I generally write Rust in a style that's a bit closer to C.
I use musl for my Rust stuff because I have noticed that for the stuff I write it appears to have a lower memory footprint; since a lot of what I do is IO-bound anyway, I care more about using less memory than raw performance.
I swear "bifrost" has to be the most overused name in computing, possibly only behind "yggdrasil." I'm not sure what's so magnetic about those names but I've seen at least 10 different things called that.
good article points, but what's a good alternative Docker Image with a small footprint and a package manager to Alpine Linux, if you need to install something on it and powered by glibc?
Most of musl's performance issues come from their allocator. Using it with a third party high performance allocator allows you to benefit from static linking with very little performance loss.
> Most of musl's performance issues come from their allocator. Using it with a third party high performance allocator allows you to benefit from static linking with very little performance loss.
This is addressed and disputed very early in the article. The very first benchmark presented shows a 26% regression using musl + mimalloc, a high-performance 3rd party allocator.
It's not really disputed since musl without mimalloc has a 144% overhead, so most of the performance issues do indeed come from the allocator, by a pretty large margin (~85% of it). Not only that, but some of the "other code" performance hit might still come from the allocator: when you set a global allocator on the Rust side, musl still uses its own allocator internally (as demonstrated by https://github.com/BurntSushi/ripgrep/issues/3494).
And the compounding issue is that the allocator issues get significantly worse as parallelism increases, as the allocator is serial, so as concurrency increases so does the impact of the allocator, which is not the case for most of the "regular slow" code (of musl), those have a relatively constant overhead per thread.
"Using it with a third party high performance allocator allows you to benefit from static linking with very little performance loss" is disputed; 26% is not "very little," even if 144% is worse.
If your algorithm does a ton of small allocations to the point where the allocator is the bottleneck, you're already doing it wrong. The allocator necessarily comes with a lot of overhead because it needs to accommodate diverse use cases, avoid fragmentation, and ideally, implement a variety of security checks. If you're doing something alloc-intensive, you're probably allocating and freeing a lot of identical structures and you'd be better off grabbing some continuous memory and managing that yourself in a task-specific way.
But the reality is that almost no one actually cares about performance because compute is cheaper than expertise and labor, at least in the short haul. Everything is getting more bloated and slower and we just compensate by adding CPU cores, gigabytes and gigahertz.
No, musl's allocator is just bad even in completely normal programs, and it is especially awful if you are using even two threads much less a lot of them. It has no TLABs or arenas. It has a single global mutex over alloc/free paths. It does syscalls underneath that lock (mmap) meaning the few fast paths it has are rarely taken under contention and have to fall back to futex wakes, so even 2 threads with minor contention and allocation rate will have visible wait points in profiles, stuck waiting for the allocator. It returns mapped memory to the OS very eagerly when a size class is empty, so even single allocs followed by a single free can cause thrashing as it mmaps/unmmaps things repeatedly for a size class over and over. Etc. You quite literally have to limit your thread count when using musl, because it will tank the performance of actually highly threaded programs that can scale with core count, even at very modest allocation rates and small working set sizes.
Its string routines and memory copy routines are also similarly bad, as the article alludes to. They are just naive loops with nearly no optimization. These are not small insignificant functions where using them is "doing it wrong", they are the backbone of vast amounts of code and can be made multiple times faster. You can similarly see string routines pop up in profiles all the time in musl builds in my experience. And unlike the memory allocator these cannot be "fixed" systematically across the application at link time, so you are stuck with it.
Real programs have to often do things like allocate memory and use multiple threads and process strings. People have been optimizing these things for decades, there is vast amounts of prior art, the musl developers simply did not do so because they prioritize simplicity over nearly everything else (from what I can tell) including performance.
It has a single global mutex over alloc/free paths. It does syscalls underneath that lock (mmap)
Every default malloc implementation worked this way about 12 years ago. Making lots of small allocations, even from multiple threads then blaming the allocator is a losing strategy. An allocator is only going to be able to mitigate the damage to speed and interactivity.
The solution is and always has been to make larger allocations and use those efficiently.
They are just naive loops with nearly no optimization.
The compiler should be able to take something with good access patterns and make something fast, especially out of the basic C functions.
they are the backbone of vast amounts of code
Performance wise it's unlikely C string functions are actually the bottleneck in a program. Maybe for specific programs a naive memory copy function could benefit from AVX instructions.
Real programs have to often do things like allocate memory
"Have to" and "often" are debatable. Any allocations in a hot loop are the very first things that should be optimized away after profiling.
>Performance wise it's unlikely C string functions are actually the bottleneck in a program. Maybe for specific programs a naive memory copy function could benefit from AVX instructions.
Many programs use lots of strings. It tends to become a bottleneck. It also tends to be very difficult to improve because the strings are everywhere in that kind of program, and refactoring to eliminate them is either impossible or very risky.
Many programs use lots of strings. It tends to become a bottleneck.
I would dispute this in anything that isn't mostly about string processing and in that case you can always easily grab different string functions, which you should probably do anyway if strings are that important.
It also tends to be very difficult to improve because the strings are everywhere in that kind of program,
I don't know what 'that kind of program' is supposed to mean.
and refactoring to eliminate them is either impossible or very risky.
This doesn't sound like a general purpose statement that applies to anything broadly.
All I'm saying is the the title is wrong and musl doesn't do much to prevent speed in a program. If someone was really trying to optimize, blaming the standard library is not going to get them very far and it's easy to work around, but needing to do that is very rare.
>I would dispute this in anything that isn't mostly about string processing and in that case you can always easily grab different string functions, which you should probably do anyway if strings are that important.
I'm telling you that I've seen it, in stuff as diverse as video games and robotics. Lots of things use strings as values. It's easy to say "just change everything in millions of lines of code" when you aren't the one who has to make that change.
By the way most software does copious amounts of string processing... I think that should be common knowledge, but I guess it isn't.
>This doesn't sound like a general purpose statement that applies to anything broadly.
You sound like you have zero experience. If your config is in strings, and hundreds of thousands of lines of code already rely on the string-ness of the data, then you just can't make the change easily.
>All I'm saying is the the title is wrong and musl doesn't do much to prevent speed in a program. If someone was really trying to optimize, blaming the standard library is not going to get them very far and it's easy to work around, but needing to do that is very rare.
I believe the title is accurate. People in performance-sensitive areas gripe about libraries, even standard libraries, quite often. I don't mean to insult you but you're making bold assertions despite clearly lacking the experience to know how things are done in industry generally.
Lots of things use strings as values.
To be clear, you're saying that in video games and robotics people are using strings instead of numbers and when that becomes a performance problem you think simple C string functions are to blame? How about not using strings as values?
By the way most software does copious amounts of string processing... I think that should be common knowledge, but I guess it isn't.
It isn't because it's not true if "copious" is about CPU time. String processing is rarely the bottleneck.
I believe the title is accurate
Well, it isn't. It's unlikely that musl prevents anyone from making a fast program. It doesn't even make sense. In the off chance anything was a real bottleneck you could bring in something faster and you would want to do that anyway.
People in performance-sensitive areas gripe about libraries, even standard libraries, quite often.
I have done a lot of optimization and I've never seen the standard library be a problem for exactly what I just outlined.
A lot of what you're saying is just "it's a problem because it is, trust me". That isn't evidence or an explanation.
As soon as allocation is slow you can avoid allocations (which you should do anyway) or use a different one (which you would do even with a standard libc anyway).
If you really have string problems (and not some fake problem like just parsing strings over and over instead of caching values) then you would use an optimized library. The benefit from a regular libc over musl is minuscule compared to the real solutions to optimizing.
you're making bold assertions despite clearly lacking the experience to know how things are done in industry generally.
Trying for insults doesn't add any sort of technical explanation. To be very clear any program written by someone who says their memory allocator is their bottleneck is something I could speed up by orders of magnitude and the standard library isn't going to matter.
>To be clear, you're saying that in video games and robotics people are using strings instead of numbers and when that becomes a performance problem you think simple C string functions are to blame? How about not using strings as values?
To bring this back to the article, the problem is actually that one library does this worse than others. If you are unfortunate enough to already rely on these functions performing up to a certain standard, then having them become dramatically worse is in fact an issue.
Why don't you just not use slow functions? Well, that goes back to my references to refactoring. Even if you could get approval to refactor the stuff, it's still risky and a lot of work. Compare that to just not using an oddball standard library with worse performance...
>Well, it isn't. It's unlikely that musl prevents anyone from making a fast program. It doesn't even make sense. In the off chance anything was a real bottleneck you could bring in something faster and you would want to do that anyway.
THAT IS THE POINT OF THE ARTICLE TITLE: If you require performance in certain key areas, MUSL may not be acceptable.
>I have done a lot of optimization and I've never seen the standard library be a problem for exactly what I just outlined.
The article here is literally complaining about a standard library's performance, which is not uncommon in the blog-o-sphere. So, you are ignoring evidence right in your face. People like me are telling you it sometimes matters, and people blog about such problems frequently, but you still aren't getting it.
I can only tell you vaguely about codebases I've worked on. I can't tell you where, or show you code, or anything like that. Get used to it.
>A lot of what you're saying is just "it's a problem because it is, trust me". That isn't evidence or an explanation.
Everything you've said is "It's NOT a problem because I'VE never seen it be a problem!" When the evidence is right in front of your face and people are telling you, yes, it is a problem. Do you think I'm getting paid to share this wisdom with you?
If you don't think string processing is a bottleneck, you should consider how many applications are document-based and string-based. Basically, it's a MAJORITY of applications in the world, and I'd put money on that.
>As soon as allocation is slow you can avoid allocations (which you should do anyway) or use a different one (which you would do even with a standard libc anyway).
More "just refactor bro" or "just use a different library" (the point of the article). Only one of these is likely to be practical in any given situation, especially since MUSL is not the default for most stacks.
>The benefit from a regular libc over musl is minuscule compared to the real solutions to optimizing.
Bro, if the stats in the article are right (and I have no reason to doubt) the difference is significant (at least numerically). It's easy to tell other people to go do a ton of work to optimize. The radically easier solution is to just not use MUSL if that's your problem. Again, the entire point of the article.
>Trying for insults doesn't add any sort of technical explanation.
Saying you're inexperienced is not insulting, especially since you're a stranger. You clearly deny having experience with the stuff I'm talking about, which I think is common-knowledge in optimization circles, and then insist on labor-intensive solutions to easily solved problems. Some insulting thoughts have crossed my mind here but I know we've all been inexperienced at some point, so I'm trying to keep it civil to teach you something.
>To be very clear any program written by someone who says their memory allocator is their bottleneck is something I could speed up by orders of magnitude and the standard library isn't going to matter.
This is youthful arrogance (I can only assume you're young; if not, you at least haven't matured in your career). It's not always possible to do such optimization, from either a technical perspective or a pragmatic one. Most people do not have authority to go on an optimization binge across their codebases, assuming the penalty is even paid by their own code (it often comes from upstream libraries!). Even if you did have the authority, expertise, and time to do the optimization, it could be a horrible idea and introduce a LOT of potential bugs.
I like the idea of MUSL, and wish the project well. I may even use it for something one day. But none of this makes their performance better. It may be that getting better performance would compromise their other objectives, such as simplicity.
If you are unfortunate enough to already rely on these functions performing up to a certain standard, then having them become dramatically worse is in fact an issue.
There are a few problems here. The first saying that anything is dramatically worse. The second is thinking that nothing can be changed. The third is thinking that there are lots of programs out there that spend all their time in C string functions yet nothing can be altered except for linking in a different standard library.
I can't tell you where, or show you code, or anything like that. Get used to it.
I didn't expect at any point that you to be able to back up what you are saying with examples.
Everything you've said is "It's NOT a problem because I'VE never seen it be a problem!"
That couldn't be further from the truth. I'm saying it isn't a problem because the problems are easily fixable and musl doesn't prevent them from being fixed.
You hallucinated a quote and made up something completely different in your head.
If you don't think string processing is a bottleneck, you should consider how many applications are document-based and string-based. Basically, it's a MAJORITY of applications in the world, and I'd put money on that.
You think these programs are bottlenecked by the C-string functions in their standard library? That's a bold claim. Why would a program completely dependent on strings even use C string functions in the first place? You have to scan to a newline to find the length, they work with ascii and they are known to be incredibly insecure. What you're saying doesn't make sense.
You clearly deny having experience with the stuff I'm talking about,
I deny that there are programs that can only be sped up by switching to a different C library and nothing else, since that's nonsense.
just not use MUSL if that's your problem.
People can do whatever they want, all I've ever said is that musl doesn't prevent anyone from making a fast program. That's it.
Saying you're inexperienced is not insulting,
I'll take your word for it because you seem extremely inexperienced at optimizing, especially if you think making fast software should include leaning on C strings and having memory allocation show up on your profiler.
This is youthful arrogance (I can only assume you're young; if not, you at least haven't matured in your career).
You can go for the insults and try to be patronizing again, I expect that as the last resort of someone frustrated that repeating their claim isn't taken as evidence. I explained a lot in detail about why musl isn't going to prevent anyone from writing fast software because I've done it over and over.
You seem to be saying that you can speed up legacy programs somewhat that weren't made well in the first place with a faster libc and I'm sure that's true, but it has nothing at all to do with the premise the musl prevents a program from being fast or even is much of a bump in the road.
The speed ups from a more optimized libc are percentage points shaved off the the times where you actually use it. Using better string functions, minimizing allocations and paying attention to memory access patterns are going to be order of magnitude changes. Minimizing allocations is going to be at least 7x on a single core, better memory access patterns are going to be 20x-25x.
You don't have any evidence or explanation that musl prevents someone from writing fast software, which is the title and the title is wrong.
>There are a few problems here. The first saying that anything is dramatically worse. The second is thinking that nothing can be changed. The third is thinking that there are lots of programs out there that spend all their time in C string functions yet nothing can be altered except for linking in a different standard library.
25% slowdown can be dramatic for some applications. Secondly, I didn't say that nothing can be changed. I said that change is expensive. Thirdly, I think linking another library is probably acceptable to get an easy 25% speedup! This problem was probably discovered by somebody saying "Why is this shit so slow when I link with MUSL?"
Regarding "Nothing can be altered except for linking" -- There are many such cases. This especially happens with upstream code. If you use a library that you aren't willing or able to fork, you have to deal with its limitations. This can happen for open-source projects, or for private commercial projects.
>You hallucinated a quote and made up something completely different in your head.
I summarized your whole position in an ironic quote to show you how dumb it is. I'm sorry you don't see how you come off. Calling my rhetoric "hallucination" is laughable. I could swear I'm arguing with a bot.
>I'm saying it isn't a problem because the problems are easily fixable and musl doesn't prevent them from being fixed.
Your fix suggestion amounts to calling for a huge refactoring, as I said. MUSL does not prevent you from doing that, but it's easier to just not link MUSL if it's causing problems for you.
>You think these programs are bottlenecked by the C-string functions in their standard library? That's a bold claim. Why would a program completely dependent on strings even use C string functions in the first place? You have to scan to a newline to find the length, they work with ascii and they are known to be incredibly insecure. What you're saying doesn't make sense.
Performance-sensitive programs and libraries are often written in C. C-string representation is widely used by all programming languages, which are usually written in C or C++ (which uses C).
>I deny that there are programs that can only be sped up by switching to a different C library and nothing else, since that's nonsense.
This statement is the real nonsense. Again with the "I've never seen it, so it can't exist" bullshit.
>People can do whatever they want, all I've ever said is that musl doesn't prevent anyone from making a fast program. That's it.
No, that's not all you've said. You said the title is wrong. You said (roughly speaking) that the choice of standard library is never a decision point for performance. The title of this article may be a bit exaggerated, but there's a clear example of poor MUSL performance in the article. It's also not JUST slow string functions, it's slow memory allocation too. What's next, you gonna say you've never seen a program that needs lots of memory allocation? Or that I should go fork some upstream project to work around MUSL's limitations?
>You can go for the insults and try to be patronizing again, I expect that as the last resort of someone frustrated that repeating their claim isn't taken as evidence. I explained a lot in detail about why musl isn't going to prevent anyone from writing fast software because I've done it over and over.
You can keep saying it over and over and it won't be any more true. It's true that I'm making claims and you're not accepting them. What you should ask yourself is what I have to gain by making these claims. The answer is nothing. I'm beginning to think you're a troll. Your username certainly suggests it.
>You seem to be saying that you can speed up legacy programs somewhat that weren't made well in the first place with a faster libc and I'm sure that's true, but it has nothing at all to do with the premise the musl prevents a program from being fast or even is much of a bump in the road.
I am CLEARLY saying that. Linking MUSL to any program that heavily uses the slow functions will make it slower. Since MUSL is not the default for most software, this will be observed as totally unnecessary and inexcusable performance degradation. If you're trying to build the fastest version of some software, you should use the fastest libraries.
Your position seems to be that the title is wrong because it is theoretically possible to make MUSL-dependent programs fast according to some unstated performance metric, so the title is necessarily wrong. What you don't see is that no matter what performance metric you choose, if I wrote the program to be fast with MUSL, it would be EVEN FASTER with a faster library. It might be "fast enough" for somebody with MUSL alone. But if that somebody cares about performance (like the title says) then they will use the fastest library they can. They won't refactor all their code to make MUSL work faster. Sometimes the objective of caring about performance is to have literally the fastest thing possible, not just "fast enough".
>You don't have any evidence or explanation that musl prevents someone from writing fast software, which is the title and the title is wrong.
The title doesn't say that MUSL will stop you from writing fast software. I never said that either. The title says "Don't use MUSL if you care about performance." But keep burning that straw man bro. I'm done with this bullshit conversation.
25% slowdown can be dramatic for some applications.
What you keep forgetting is that's 25% of the part that you're actually using.
You keep avoiding and denying that for a program where any sort of speed matters you just aren't spending any of your time hitting the C standard library.
If a program is slow because of the C standard library and it needs to be faster, someone messed up a long time ago. That isn't the C libraries fault. You can squeeze a little more out of it if you have find a more optimized library, but it's all a drop in the ocean compared to how much faster it would go by doing basic real optimizations.
Performance-sensitive programs and libraries are often written in C. C-string representation is widely used by all programming languages, which are usually written in C or C++ (which uses C).
You can say that, but really it's almost all C++ and people avoid C strings for exactly why I outlined in detail. You don't have length up front, it's all ascii, you're dealing with pointers to arbitrary runs of bytes, etc.
This is not good for memory access patterns, dealing with lots of characters at one time, dealing with unicode, minimizing memory allocations etc.
It's true that I'm making claims and you're not accepting them.
That is true that you are making lots of claims and that I'm not accepting them, because you don't have any evidence or explanations and they don't make sense.
All I've said is that MUSL doesn't prevent someone from making fast programs and you barely have even confronted that, let alone explained how it isn't true.
Linking MUSL to any program that heavily uses the slow functions will make it slower.
Any program that is hammering the the C standard library can be sped up by orders of magnitude and 25% is nothing. Again, lifting memory allocations will speed something up by 10x, so that 25% isn't going to matter anymore because it's 25% of something that isn't even going to show up on a profiler, let alone be a bottleneck.
Your position seems to be that the title is wrong because it is theoretically possible to make MUSL-dependent programs fast according to some unstated performance metric
I think you mean it's trivially possible according to the detailed explanation I gave from a lot of experience optimizing.
Sometimes the objective of caring about performance is to have literally the fastest thing possible, not just "fast enough".
25% better might be fast enough for you, I like speeding up programs by 100x by changing to C++ instead of C and focusing on the optimizations that matter.
The title says "Don't use MUSL if you care about performance."
I use musl and I care a lot about performance. It doesn't matter because has no bearing on how fast my programs are.
I'm done with this bullshit conversation.
I'm sure it seems that way when someone doesn't accept the same claim over and over without any actual explanation or evidence. Saying the same thing and getting more upset is not an effective way to make your point. You need real information, not insults and fake quotes.
I awoke today with renewed energy to deal with your nonsense, and I think after reading this comment it deserves a (hopefully) short reply.
>I use musl and I care a lot about performance. It doesn't matter because has no bearing on how fast my programs are.
This is the crux of the matter. You clearly have some investment in MUSL, that much has been apparent all along. If you actually USE it, which I'm sure you do, then you are not using the fastest possible library. Regardless of your skill, simply switching to a faster library will make a program faster. Sometimes, that's what is needed. We JUST want to find easy ways to make code as fast as possible. We aren't looking to rewrite the world.
I don't have any investment in MUSL. I wish the project well, but if it's not the fastest library then it will necessarily be unsuitable for some applications.
>What you keep forgetting is that's 25% of the part that you're actually using.
First off, I didn't forget shit. There are surely programs for which the penalty is actually 25% or close to it. The penalty can actually be far worse than that due to algorithmic complexity issues. Secondly, it is quite possible that one could use the standard library almost exclusively. Thirdly, any performance drop could be significant.
>You can say that, but really it's almost all C++ and people avoid C strings for exactly why I outlined in detail. You don't have length up front, it's all ascii, you're dealing with pointers to arbitrary runs of bytes, etc.
C strings are used extensively in C++. Again you prove how inexperienced you are.
>This is not good for memory access patterns, dealing with lots of characters at one time, dealing with unicode, minimizing memory allocations etc.
Keep deflecting with bullshit that is irrelevant to the choice of library, and that can't be changed but for massive refactoring.
>That is true that you are making lots of claims and that I'm not accepting them, because you don't have any evidence or explanations and they don't make sense.
There's only so much evidence and explanation I can provide to strangers on the internet. Then there is also the level of evidence that I can provide, such as other blog posts, that I don't feel like providing. A lot of the things I've said are true a priori (assuming some simplifications), such as the fact that a slower library will always be worse for performance than a faster library. If MUSL is measurably slower than others, then it can be a problem.
>Any program that is hammering the the C standard library can be sped up by orders of magnitude and 25% is nothing. Again, lifting memory allocations will speed something up by 10x, so that 25% isn't going to matter anymore because it's 25% of something that isn't even going to show up on a profiler, let alone be a bottleneck.
As I've said many times, rewriting software is not always on the table. Even minor changes may be forbidden or burdensome for various reasons. What you're talking about is a redesign, that may not be applicable to all cases anyway.
>>Your position seems to be that the title is wrong because it is theoretically possible to make MUSL-dependent programs fast according to some unstated performance metric
>
>I think you mean it's trivially possible according to the detailed explanation I gave from a lot of experience optimizing.
Right, your position is definitely that making these changes is "trivial"... I threw in "theoretically" because I interjected my own knowledge that these optimizations are often only theoretically possible, and can't be done for many practical reasons. My wrong statement of your position made it more defensible than it actually is.
>25% better might be fast enough for you, I like speeding up programs by 100x by changing to C++ instead of C and focusing on the optimizations that matter.
I don't want to get off on another tangent but you are trivializing and exaggerating a lot of stuff in this one statement. Changing languages and altering the code is not an easy win, and might actually trigger a performance setback. Switching to a faster library is a relatively easy win.
>Saying the same thing and getting more upset is not an effective way to make your point. You need real information, not insults and fake quotes.
I've made my point already, you just haven't accepted it. I assume you are either incapable of understanding (perhaps temporarily), or have unstated biases as I pointed out due to being a MUSL enthusiast or contributor. You're not convincing me either way. My position is that people who care about being as fast as possible should use the fastest available library, which clearly isn't MUSL. There may be other valid reasons to use MUSL, and it may be "fast enough", but other libraries are yet faster.
This is how I see your argument so far:
- MUSL is fast enough, trust me bro.
- Ok, maybe someone measured some functions to be slower, but everyone knows that you shouldn't be using the STANDARD LIBRARY heavily. Nobody has ever used strings or memory allocation that heavily, and if they did then they are doing it wrong (even if other libraries provide adequate performance for them).
- If you have a problem with this, you just need to do some trivial optimizations, or change languages to C++. It's SUPER EASY (at least for elite MFers like myself).
- Because rewriting the code to compensate for MUSL performance is always on the table, it's never reasonable to say that MUSL is causing a performance problem.
I only added a slight amount of emphasis. You've been pretty close to that bombastic in the whole exchange. It's ridiculous, and I think you are smart enough to know better but admitting that you are wrong is beyond the pale. Instead of wasting my time, how about applying those elite optimization skills you claim to have to the MUSL code to make it faster. Then you will be able to write a blog like "Don't use glibc if you care about performance" and argue with people on HN if you want. By the way, I didn't write this blog post, and I don't have any other conflicts of interest such as being a glibc developer, so don't start up with that shit either.
C strings are used extensively in C++. Again you prove how inexperienced you are.
You can try to be patronizing if you want, but this doesn't make sense with you not understanding memory access patterns. If someone uses C++ they will use C++ strings. Have you used C++? It works well and it has been around for 40 years, you should give it a try.
Keep deflecting with bullshit that is irrelevant to the choice of library, and that can't be changed but for massive refactoring.
You painting yourself into a corner has nothing to do with a C library being a few percentage points slower. This is not a general scenario, it's something you are dealing with because you can't optimize C strings.
If you think memory access patterns are 'bullshit' you haven't done much optimization. That's everything in modern optimization.
If there is an expert somewhere around you, please talk to them so you can get up to speed on modern optimization. Minimizing allocations is table stakes, memory access is the meat.
If you have any examples on github of things you think "can't" be optimized, go ahead and link it and tell me what a profiler shows, and I will explain what to do.
I've made my point already, you just haven't accepted it.
You made your claim, you haven't had any evidence or explanation, that's how it works. You say the same thing over and over and get more upset, but that isn't evidence. Show me where musl prevents someone from writing a fast program.
Insults and hallucinated quotes aren't a good foundation for proving your point. If you have to pretend that someone said something different, then maybe you aren't really making sense.
how about applying those elite optimization skills you claim to have to the MUSL code to make it faster.
Why would I do that when I don't use the C library for performance sensitive programs? You never seem to be able to confront this. The stuff made 50 years ago isn't the fastest possible stuff. It's still pretty fast and if you want something faster, do something else. It's not that complicated but it seems to really upset you.
I don't have any other conflicts of interest such as being a glibc developer, so don't start up with that shit either.
No one said anything about that, I think you're hallucinating or predicting something that never happened.
> Every default malloc implementation worked this way about 12 years ago.
Perhaps "default" is doing the heavy lifting here. Glibc malloc was quite bad for a long time, true. But TCmalloc / jemalloc are 21-22 years old, respectively, and jemalloc has been the FreeBSD (released) default malloc implementation for the last 18.
> The solution is and always has been to make larger allocations and use those efficiently.
Having a not-dogshit allocator really doesn't hurt. There's no reason to defend shitty allocator + every application doing manual memory pools on top of it to paper over the bad allocator.
I'm not defending anything, I'm saying it's usually trivial to make allocation time marginal.
+ every application doing manual memory pools
Usually it's simple data structures in flat memory.
If allocation is taking all the time, that's a poorly optimized program with lots of low hanging fruit and a different allocator is not the right fix. It's like having a boat with a hole in the bottom and someone says the solution is a smaller hole.
But TCmalloc / jemalloc are 21-22 years old, respectively, and jemalloc has been the FreeBSD (released) default malloc implementation for the last 18.
jemalloc is also possibly bigger than all of musl. If it was a problem after optimization I would use it and I have in the past, it's just nowhere near as important as minimizing allocations in the first place. OpenBSD uses straight mmap.
There's also a reason this stuff started getting adopted around 20 years ago (I'll add macOS to the pile: it added per-CPU malloc sharding in 2008). It's not just due to overall growth in OS complexity. It's because that's when multicore CPUs were taking off. Before that, the cost of a global lock was far lower.
OpenBSD is straight unconcerned about performance. That's their choice.
glibc's AVX2 based memcpy functions have a non-negligible performance impact in at least 1 application I maintain on the order of like 8-ish% vs musl (wall clock). It just has to memcpy/memmove a whole lot. Whether or not that's tolerable is up to debate, but a spade is a spade.
I said "backbone", not "bottleneck". They are common functions sprinkled in everywhere throughout every application in every codepath on something like a modern Linux desktop. An inverted callstack flamegraph can show you stuff like this. It is basically no different than compiling your application at -O1 and -O2 with GCC. Does the fact your program get 20% faster from -O2 mean that there were "bottlenecks" the compiler solved? No, there was just performance left on the table by emitting better code. Not really. I have to spell it out apparently: actual programs written by normal human programmers do those things, all the time, they exist in and are common in the world, they depend on other code that does that and is common in the world, they run on your desktop and phone and all servers, and they benefit quite a lot from optimized components like memory allocators and string routines and -O2 making their programs faster. This is pretty easy to observe and the means of doing so should be quite obvious, so there's no real debate.Now whether this fact holds -- whether these programs "have to" do these things or not -- in the imaginary fantasy land people have in their heads where they make up arguments to themselves about how, if every program was written how they liked it, it would be better? That I'm not so sure about, I will admit.
The memory allocators of that time were also better than the ones 12 years their prior. That's the point.
The point is that memory allocation shouldn't be a bottleneck either way. If it is the program needs to be optimized or redesigned. Better allocators give you more slack, they don't solve the problem. If the problem is already solved, then a basic allocator isn't going to make a big performance difference because it isn't the bottleneck.
those bad data structures would also cause "damage to speed and interactivity" or whatever. This isn't very hard to understand.
It depends on how much they are used and how much contention there is. Sometimes putting a mutex around things is fine.
But these are extremely common specified functions, they are called everywhere all the time
Not necessarily, especially for C string functions, but they do get linked in so it's a good thing musl makes them small.
I said "backbone", not "bottleneck".
Then the point is lost, because 'backbone' doesn't mean anything if it works. If it isn't a bottleneck in throughput or latency anywhere then the speed doesn't matter.
The other important thing is that better stuff can be included in pieces as it's needed. The reverse isn't true. If you use a big fat C library, you have a dependency that isn't going to get better.
Not really. I have to spell it out apparently: actual programs written by normal human programmers do those things, all the time,
You spelled it out last time, it's just not true in the sense that programs have to have these functions as bottlenecks. Strings, allocators and memory copying can all be dealt with independently, but again it's rare that strings and allocations really need to be the bottleneck and in those circumstances you probably want more than a different standard library anyway.
in the imaginary fantasy land people have in their heads where they make up arguments to themselves about how, if every program was written how they liked it,
I'm not sure what this is supposed to mean, there is nothing I've said that doesn't make perfect sense. If you want something to go faster you can make it go faster. A better allocator pales in comparison to lifting allocations out of hot loops.
My point is the musl is useful and the disadvantages are easy to work around. I'm not really sure what your point is, do you think people are going to force you to use it?
Your point seems to be that if you use a slow standard library and complain, it's not a problem with the slow standard library because you can just reimplement the slow parts independently.
The problem with your argument is that it's a universal argument against performance. And if an argument is universal, then it doesn't have any information value.
Your point seems to be that if you use a slow standard library and complain, it's not a problem with the slow standard library because you can just reimplement the slow parts independently.
No, this is something that nuanced. The title is wrong because musl isn't going to prevent you from writing fast software.
Whatever benefit there is to a different libc, is absolutely miniscule compared to do actual optimizations like avoiding allocations.
I'll give you real numbers: if you put allocations of short vectors of a dozen floats in a hot loop, when you lift the allocations out your program is going to instantly get about 10x faster. The allocation is no longer going to be the bottleneck, it will be marginal and then a faster allocator isn't going to matter at all.
If someone gets an easy speedup from using a different libc that's great, but the vast majority of time it isn't going to matter and isn't going to be where any real speedups come from. The difference is a small percentage speedup vs orders of magnitude.
The amounts to the title being wrong, using musl or a small standard library just doesn't prevent a program from running fast. It is a tiny difference and even that tiny difference can be changed from things like better allocators which you would do anyway with a standard libc.
That's an interesting viewpoint, but then, will the allocator's performance never matter for any use case that is not "wrong"? It doesn't feel right.
Think about it this way because the issue isn’t specific to allocators: it’s pretty good in general but can often be beaten if you have special understanding of what you need to do. That’s OK.
You can buy cars and trucks that are optimized for driving on freeways and residential streets carrying stuff people often carry. But then there are special vehicles like fork lifts and such that are kinds of large special cases. And then there are weirdo specialised vehicles that have four wheels but are rare and their users can’t live without them.
Languages like C++ let you plug in special allocators if you want. But most people don’t. Some, like HFT people do crazy headstands to avoid slow allocations. I don’t ever want to do that but if they want to, why not. I don’t think they complain that the default case doesn’t fit their needs!
For a typical program, I bet that the overall impact of the glibc allocator is well under 0.1%. If you can choose between < 0.1% and < 0.12%, I guess it matters in some sense, but not in any practical way. We almost never spend time on other sub-0.1% optimizations. You could probably squeeze a lot more by optimizing CPU branch predictor performance, minimizing CPU cache misses, or fine-tuning the scheduling strategy, but we also don't bother. 'Tis is the era of "native" apps written in Electron.
A decade ago, I worked on a simulation program that involved a C++ core with a Python wrapper and DB interface. End users cared a lot about throughout with a rather limited size, weight and power budget. We spent a lot of time optimizing the core -- but basically hit a bottleneck once we got to about 15% of the time that was spent in malloc-related functions. It turned out that was all in the Python layer. Probably there was some level of bad code in our Python code, but it was impractical to figure out where that was. I was shocked because I assumed the simulation core (which ended up almost allocation-free by the end) would always dominate CPU usage.
Both allocators and Python have probably gotten better since then, but it was a fascinatingly large and stubborn fraction of CPU time.
C++ and python are exactly where I would expect this problem to come up
"But the reality is that almost no one actually cares about performance because compute is cheaper than expertise and labor, at least in the short haul."
Doesn't have to stay that way, with hardware prices soaring and development cost allegedly in free fall.
Also:
> these numbers are from 4 core EC2 VMs
Yeah, because that's the exact sort of hardware musl is aimed at.
Here's my counter-bogus-statistic: musl is infinitely faster than glibc on a Nios II processor because musl will fit into memory and glibc won't. Therefore, don't use glibc if you care about your code being able to run. QED.
People have such different perspectives. 26% slower does not sound "terrible" to me; it sounds like quite a reasonable price one might choose to pay for the convenience musl offers. If musl's allocator were 2.6x slower, I might call that "not so great"... but in order to qualify as "terrible" I think the difference would have to be an order of magnitude!
The 26% number at the top of the article is from using mimalloc (which is a high performance allocator, at least as fast as the glibc allocator) + musl for some task, and the slowdown is coming from (probably) slow musl implementations of memcpy/memset. The musl allocator is even worse.
Yeah, doing compute-heavy work a couple of jobs ago, we tried small images with musl, and the default allocator was a catastrophe: 75%+ slowdowns for our real life tasks. Even with a better allocator, we were way better off with the larger image.
> the slowdown is coming from (probably) slow musl implementations of memcpy/memset.
It's wild that such a fundamental piece of code (you can't really implement operation on structs without those) is library-supplied. I wish compilers would just have something like __builtin_memcpy and __builtin_memset, and provided some highly optimized, specialist-crafted assembly in those, instead of having to inline the library code and hopefully be able to optimize it.
Clang and GCC do provide these, and automatically use them in many situations (particularly small copies). But c-libraries can actually do it better in many cases, especially for large copies.
Glibc, for example, has perhaps ten different implementations of memcpy just for x86. The compiler certainly could provide all that, but the next step is harder:
glibc automatically dispatches to the proper one at runtime based on the actual microarchitecture that the binary is running on. You pay the extra dispatch cost once, but all of non-inline function call cost every time. This is what allows distros to compile to a nice baseline architecture, but still get near-optimal memcpy performance on many more architectures than a single inline instance could possibly give. These differences matter.
And it does it for not just memcpy, but half-a-dozen other extremely performance sensitive library functions, like strcpy and so on.
Inlining works very much against this strategy. If you can guarantee that the target microarch never changes, then it isn't a good one. But that is somewhat unusual for everyone but those who build their own binaries to run on a single class of machines forever.
Worse, inlining the really high performance versions of these ends up being terrible from a code size perspective, because they are often hundreds of instructions, which can have bad caching effects. And once you amortize the function-call cost over many iterations of the loop, it isn't so expensive to call out to the library.
Anyway, just some additional considerations to think about.
These builtins of course exist, it's how compilers keep track of the behavior of these functions.
For GCC, there is -minline-all-stringops:
https://gcc.gnu.org/onlinedocs/gcc-16.2.0/gcc/x86-Options.ht...
It does what it says, but the results may not be what you expect.
> I wish compilers would just have something like __builtin_memcpy and __builtin_memset
The ones provided by the compilers are simply the libc ones.
LLVM will even go as far as detect attempts to rewrite memcpy and replace them with a call to the libc one!
Even if the attempt is inside of a function called memcpy() which contains no code other than your copy loop, and links with priority over the libc implementation! (as all embedded firmware engineers learn at some point in their journey)
Maybe you're being sarcastic, but I'm pretty sure clang + gcc do offer these.
The problems at first glance :
- Not having control over the implementation detail of the interface that your library provides is probably not wise. Sounds like a lot of bad bug reports and edge cases that you have no control over.
- Not all compilers may provide these.
As others have alluded, __builtin_memcpy doesn't resolve to a runtime implementation. GCC and clang treat functions like memcpy specially. Because they're defined by the standard and are reserved names, compilers can assume the exact semantics specified by the standard and elide library calls altogether with optimized inline code. But if the compiler can't do the optimization (can't prove alignment, indeterminate length, etc), it just emits a library call, even if your source has some other local function definition named "memcpy". Explicit use of __builtin_memcpy is treated identically to calls to memcpy, unless the compiler is invoked with -ffreestanding, in which case it only optimizes __builtin_memcpy and skips special treatment of calls to memcpy, but __builtin_memcpy could still expand into a call to memcpy. If you're writing a C library you want to use -ffreestanding. (I think. There may be more nuance. More info at https://gcc.gnu.org/bugzilla/show_bug.cgi?id=56888)
The 26% slower appears to be for their whole application, not just the allocator. For some parts of the application to make the whole this much slower it must mean that those parts are quite a lot slower, likely much more than 2x.
Moreover the 26% is with mimalloc, with musl's allocator it's 144%, so there are likely other parts that are slower (likely the memcpy implementation)
Ops here, I think if you NEED that convenience, sure, rock with MUSL BUT I also see a ton of devs crowing about using MUSL on my 128GB x86 Kubernetes hosts. I have plenty of Disk Space, you can ship glibc based container.
26% slower could turn into a huge hardware bill, and could render the library unusable for some purposes. There are many applications for which 26% is negligible, but it ain't nothing...
I'm curious. What convenience, specifically, are people benefiting from by using musl?
If you want to ship a prebuilt binary that'll run on any linux distro you need to statically link libc and musl is by far the easiest way to do that.
except it only can work for CLI apps or anything that doesn't use the GPU as for instance nvidia drivers require glibc
The convenience (?) of not having to comply with GPL terms. Some people really hate copyleft.
Glibc is lgpl, you _definitely_ don't need to comply with GPL to link to it.
Right. I should have said "comply with copyleft licenses".
> for the convenience musl offers
... you're fine with trading an app running 26% slower, to save a dozen megabytes ? that sounds positively insane to me. That's accepting to go from e.g. 60fps to ~45 fps (e.g. completely unacceptable)
That would depend on the application. The convenience isn't just a smaller binary, but also being independent of the distros glibc version (being on an LTS distro and not updating often, I have experienced incompatible glibc versions often enough). The megabytes shaved off are significant if the final binary size is < 1MB, which is a completely different size class.
And not all applications are performance sensitive. Something like UNIX's bc command would benefit more from having easier compatibility and a faster startup than more optimized allocators and string functions.
In the end, it ~doesnt even matter~ is all trade-offs. The nice thing is that it is up to the developer to make the decision of which libc to use, so everyone gets what they want.
Tell your employer a 26% pay decrease for you is acceptable.
This is only one aspect of "performance". glibc's allocator may be faster, but it also uses more memory.
For a much more technical discussion, see https://github.com/sharkdp/fd/issues/710
Posted this the other day but the whole musl allocator thing seems to be well known [0]
[0]: https://news.ycombinator.com/item?id=45143347
Musl's allocator being awful is pretty well known, though mostly in that it's absolutely awful in multithreaded context. TFA points out that musl has a bunch of other noticeably slower functions, which is less well known (though they're also slower by a smaller factor, and they don't worsen as your parallelism increases).
I think the size of linked binaries and simplicity were always the main features?
Yeah... I've recently had a chance to compare how fgets is implemented in both GNU libc and musl, and, well. With glibc, it was a challenge to even find where the fgets's code actually is.
I haven’t dug into why, but for unknown-linux builds on x86, Rust binaries have been substantially smaller on musl than standard dynamic linking to glibc, for me. No idea if I’m doing something wrong or if the handful of cases I tried were all special in some way.
I really find complaints about the minimum viable placeholder malloc in musl confusing. Does anyone seriously try to use malloc-ng for performance critical use cases?
Our entire linux distro is musl based BUT we swap out the default malloc with mimalloc for high performance because why would you not? Best of both worlds.
https://codeberg.org/stagex/stagex/src/branch/main/packages/...
Are there other options if I want to ship a 'FROM scratch' image with just a single Rust executable, and everything compiled in?
That to me is the main driver for MUSL.
The best option would be the x86_64-unknown-linux-none target (https://doc.rust-lang.org/nightly/rustc/platform-support/x86...) however it currently a tier 3 with a single maintainer (so it is technically available but unsupported).
If this specific use case is of high interest to you and you have some available bandwidth, contributing to it, maybe becoming a maintainer, and eventually organising a tier 2 MCP would definitely be a good idea.
Note that this is no-std no-alloc target, with all the limitations that leads to.
You could add alloc with a custom global allocator, but I don't even know what high perf global allocator you could use that wouldn't need libc. Jemalloc and mimalloc are out. Some embedded allocators would work (but those are rarely high performance, instead being optimised for small code and data footprints).
That said, with enough effort (quite a lot!) it would be possible to add support for alloc and std without libc on Linux specifically (since it has a stable syscall ABI).
What might be more realistic though is looking at relibc (a rust implementation of libc, made for Redox OS but from what I read it also supports Linux). But I haven't tried it and I don't know the state (or goal) of it.
> it would be possible to add support for alloc and std without libc on Linux specifically (since it has a stable syscall ABI).
Well yes that’s a Linux specific target so that’s kinda the point.
Technically you could do libcless on a few other platforms which are not actively hostile to it (yet) like freebsd, but that would have no chance of getting to tier 2 if it was even accepted.
I just remembered that there is also https://github.com/sunfishcode/eyra (but I think it might be a dead project) which is close to that, it had slipped my mind.
All of these are going to mean you can't link any (non-freestanding) C code, load any dylibs, etc. So you will be fairly limited in what sort of applications you can write. Forget most GUI frameworks, even native ones. You won't be able to load GL or Vulkan drivers for example. You are basically stuck with command line or servers.
I would hazard the guess that that’s perfectly fine. Desirable even. People who run scratch or alpine images and link against musl aren’t usually looking to write desktop applications or video games.
I wouldn't mind having a standalone binary that anyone can download and just run on their Linux desktop, regardless if it is Arch, Alpine or Debian stable.
With glibc that is a pain, I need to build in a container with tthe oldest glibc I want to support, and that still doesn't cover Alpine. And I dont know if a static musl build would even work for that either (if I need to be able to load GUI libraries).
You can do FROM scratch, and use still glibc; it’s just that you need to copy more than one file. I don’t really understand if you are already dealing with images why you still need the image to contain a single file.
Same usecase here, I use musl for compiling self contained Nim utilities I use on containers and servers without having to deal with glibc hell.
Funny thing is that the major reason most people use musl is because glibc make it (artificially) hard to do completely static linking.
I feel with Rust I try and avoid re-allocations in most cases anyway, so I'm not sure that musl's allocator being slow would significantly affect performance (though I haven't benchmarked it). I feel like part of the appeal of Rust is that you can do imperatively-style mutation-heavy code comparatively risk-free, so despite me normally being the "Functional Programming Nerd", I generally write Rust in a style that's a bit closer to C.
I use musl for my Rust stuff because I have noticed that for the stuff I write it appears to have a lower memory footprint; since a lot of what I do is IO-bound anyway, I care more about using less memory than raw performance.
I swear "bifrost" has to be the most overused name in computing, possibly only behind "yggdrasil." I'm not sure what's so magnetic about those names but I've seen at least 10 different things called that.
Ops here. Don't use musl if you care about disk size or compatibility either. For reference:
docker.io/node:26-alpine - 61MB
docker.io/node:26-trixie-slim: 84MB
gcr.io/distroless/nodejs26-debian13: 55MB
Before this post, we have issues with different, slower DNS resolution in musl.
In 2026, there in no reason other than self-inflicted compatibility and performance pain to use musl in production services.
It has it's place, for binary cross-OS distribution, and generally having an alternate C STL. But avoid for SaaS.
good article points, but what's a good alternative Docker Image with a small footprint and a package manager to Alpine Linux, if you need to install something on it and powered by glibc?
Also musl is not a complete runtime
[dead]
[dead]
Most of musl's performance issues come from their allocator. Using it with a third party high performance allocator allows you to benefit from static linking with very little performance loss.
> Most of musl's performance issues come from their allocator. Using it with a third party high performance allocator allows you to benefit from static linking with very little performance loss.
This is addressed and disputed very early in the article. The very first benchmark presented shows a 26% regression using musl + mimalloc, a high-performance 3rd party allocator.
It's not really disputed since musl without mimalloc has a 144% overhead, so most of the performance issues do indeed come from the allocator, by a pretty large margin (~85% of it). Not only that, but some of the "other code" performance hit might still come from the allocator: when you set a global allocator on the Rust side, musl still uses its own allocator internally (as demonstrated by https://github.com/BurntSushi/ripgrep/issues/3494).
And the compounding issue is that the allocator issues get significantly worse as parallelism increases, as the allocator is serial, so as concurrency increases so does the impact of the allocator, which is not the case for most of the "regular slow" code (of musl), those have a relatively constant overhead per thread.
"Using it with a third party high performance allocator allows you to benefit from static linking with very little performance loss" is disputed; 26% is not "very little," even if 144% is worse.