Also there was a post about why just generating ninja using python can be a good option. I do this in my project and it has been very productive so far. I couldn’t find this post now but it was saying to use ninja_syntax.py from ninja codebase and just doing something minimal for a project
An under noticed ninja feature I adore, which was implemented relatively recently, is the ability to configure how its build progress is printed. In my fish config, I have the `NINJA_STATUS` envvar:
If someone sees this: The ninja package on PyPI [0] currently stays at version 1.13.0 . There is an issue in 1.13.0 preventing it building projects on Windows. The issue is already fixed in 1.13.1 almost a year ago, but the PyPI package hasn't got an update, see [1], and many downstream projects have to stay at 1.11 . I hope it could update soon.
What a messy and frankly, absurd situation to be left in. To fork a project in order to provide a tool through Pypi, only to then stop updating it on a broken version. That's more a disservice than a service for the community... If you're going to stay stuck, better to drop the broken release and stay stuck on the previous working one.
Similar to make, it does mtime chronological comparison of dependencies with target to determinate if dependencies changed. This is just so flawed and simple to fool by operations on filesystem that do not change mtime (move, rename):
1) pick a source file and make a copy of it for for later
2) edit selected source file and rebuild
3) move the copy to it's original location
4) try to rebuild, nothing happens
By not tracking file metadata through an index file, mtime-only incremental build systems trade a lot of reliability for only slightly more simplicity. https://apenwarr.ca/log/20181113
Copy (1) and edit (2) both bump mtime, usually. It's not obvious that in the workflow you describe ninja is problematic, rather than the workflow itself (which is atypical).
Copy and edit do, but move (aka rename) generally does not, and that is the part that is problematic.
I don't think the described sequence of operations is all that unusual. Not the most common case for sure, but hardly unlikely in the grand scheme of things.
ninja fails to detect that file changed from last build - all it's mtime, ctime, inode and size can change, yet it's not detected as long as mtime is not newer than target.
Again, this is just a weird workflow, and you're assuming copy/edit don't bump mtime. That usually isn't the case. If you're doing this weird thing, you can just run `touch` when you move files over existing files like this to explicitly bump mtime.
I run into this issue when building against different environments, each with a
1. A library depends on a system package. To test against the different versions of the system package, the library is compiled within a container.
2. To minimize the incremental rebuild time, the `build` directory is mounted into the build container. Even when using a different version of the system package, this allows re-use of system-independent portions of the build.
3. When switching to a build container with a different version of the system package, the mtime of the system package is that of its compilation, not that of the build container's initialization. Therefore, the library is erroneously considered up-to-date.
Because the mtime is the only field checked to see if the library is up to date, I need to choose between having larger disk footprint (separate `build` directory for each build container), slower builds (touch the system package on entering the container, forcing a rebuild), or less safe incremental builds (shared `build` directory, manually touch files when necessary).
Incorrect, I only assume move/rename of backup to original location doesn't change it's mtime (which it doesn't with default flags or from IDE or file manager). And I don't think this is a weird or obscure workflow, I do it all the time - have two versions of a file or make a backup before some experimental changes, and restore it later.
mtime rebuild logic is half-baked even by 1970s standards.
Bazel and Shake avoid this class of bug with content hashes, so a rename, restore, or tar extract does not leave the build graph in a stale state. Speed matters, but not enough to bet your repo on timestamp luck.
Serious question: how can a build tool be fast or slow? From my understanding all it does is delegate the build steps to other tools, so wouldn't those be the bottleneck? Is it the resolution of order of build steps that takes so much time that a different build system can make a difference?
I'm afraid I still don't understand. One factor is having fewer features and not looking for obsolete files, that I can understand. I guess the other thing is using better rules to figure out when a file truly need to be rebuilt?
I used ninja only a few years ago when contributing to KDE software (Dolphin, Kate, KTextEditor, etc.). I had no prior experience with it and it was easy to apprehend, so a rather good experience.
My teammate has a great time reimplementing Ninja (slop-free) in Go here https://github.com/buildbuddy-io/reninja to make it even faster with Remote Build Execution.
All the main build tools (cmake, meson/ninja and GNU configure) have different benefits. For instance, I expect "--help" to work, but only really GNU configure supports it as-is. I could list more advantages and disadvantages in general here, but by and large I prefer meson/ninja. To me it feels by far the fastest and I also have the fewest issues usually (excluding python breaking its pip stack but that's not the fault of meson as such). ninja can be used via cmake too but most uses I see are from meson.
> ninja can be used via cmake too but most uses I see are from meson
How do you know though when the choice of cmake-generator is entirely up to the user? E.g. you can't look at a cmake file and know what generator the user will select to build the project.
FWIW I usually prefer the Ninja generator over the Makefile generator since ninja better 'auto-parallelises' - e.g. with the Makefile generator the two 'simple' options are either to run the build single-threaded or completely grind the machine to a halt because the default setting for 'parallel build' seems to heavily overcommit hardware resources. Ninja just generally does the right thing (run parallel build, but not enough parallelism to make the computer unusable).
ninja supports separate build groups and different max number of parallel jobs for each. CMake's ninja generator puts compilation and linking steps in their own respective groups. End result is by default `nproc` parallel jobs for compilation but 1 job for linking. This helps because linking can be way more memory intensive or sometimes the linker itself has support for parallelism. Most projects have only a handful of linking steps to run anyway.
I find Meson's --help fairly useful, at least compared to the disaster that is CMake's. (Try to find out, as a user not experienced with either, how you'd make a debug build.) I agree that configure --help is more useful for surfacing project-specific options, though.
I believe the same author has made a ninja successor (n2) that uses hashes instead. Haven't tried personally but hopefully get around to try that in the near future
I have never observed that issue, and I have been using it to build MMLoC repositories. Perhaps the reason being is that I always use it coupled with ccache. Have you tried that?
ccache is a workaround for the mtime problem. You can either hash with ccache or hash directly in the build system, but either way there's no substitute for hashing something. Ccache is hashing the input to the build, but there may be elements of the build that lie outside of ccache's awareness that having a hash-aware build system would take care of. Partial rebuilds devolve to a cache invalidation problem pretty quickly either way.
The absolute best thing about coding agents is not having to waste time on build systems. I had Claude code port my autotools scripts to meson (which uses ninja) and it’s been a huge quality of life improvement.
Getting builds to work was some of the most tedious, least interesting work. It is so satisfying to watch the agent try 4 different things and magically go on its way. No more makefile syntax searches or cmake hell.
For real. I stopped writing Makefiles because of how tedious it was, but now with AI I'm back to throwing Makefiles in everything. It's wonderful to have the same build, test, release commands in all projects instead of mix compile in one, npm build in another, etc. This is my favorite part of AI
Ninja is one of the best tools I have used. It is extremely simple and always works flawlessly.
Some blog posts from the creator of ninja:
https://neugierig.org/software/blog/2018/07/options.html
https://neugierig.org/software/blog/2011/04/complexity.html
Also there was a post about why just generating ninja using python can be a good option. I do this in my project and it has been very productive so far. I couldn’t find this post now but it was saying to use ninja_syntax.py from ninja codebase and just doing something minimal for a project
OMG, so many open source projects need to read this:
https://neugierig.org/software/blog/2018/07/options.html
(hello FreeCAD ;)
Evan Martin (evmar) started Ninja when he was working on Chrome at Google:
https://neugierig.org/software/chromium/notes/2011/02/ninja....
Hence, it's used in a lot of Google projects.
Ninja is possibly the best example of the "Do one thing and do it well" philosophy. All it does is execute commands based on a static build graph.
It's syntax is simple enough that it's trivial to e.g. write a shell script to generate the build items if you need dynamic dependencies.
An under noticed ninja feature I adore, which was implemented relatively recently, is the ability to configure how its build progress is printed. In my fish config, I have the `NINJA_STATUS` envvar:
Which prints the time elapsed and projected in a readable multi-line format.If someone sees this: The ninja package on PyPI [0] currently stays at version 1.13.0 . There is an issue in 1.13.0 preventing it building projects on Windows. The issue is already fixed in 1.13.1 almost a year ago, but the PyPI package hasn't got an update, see [1], and many downstream projects have to stay at 1.11 . I hope it could update soon.
[0] https://pypi.org/project/ninja/
[1] https://github.com/scikit-build/ninja-python-distributions/i...
Why is a C++ project being distributed on PyPi at all?
Probably for the same reason other binaries are distributed by npm: lack of cross platform general package managers and registries
Also for cases where a python project needs to depend on it.
Or lack of a tool like Goreleaser in the language ecosystem that handles that
You may be interested in this discussion: https://discuss.python.org/t/use-of-pypi-as-a-generic-storag...
Because the development world either hasn't heard of nix or has collectively decided to not use nix.
What a messy and frankly, absurd situation to be left in. To fork a project in order to provide a tool through Pypi, only to then stop updating it on a broken version. That's more a disservice than a service for the community... If you're going to stay stuck, better to drop the broken release and stay stuck on the previous working one.
Ninja is great and feels natural coming from Make. What it lacks in features it makes up for with speed, which is what ultimately matters.
Also worth mentioning is samurai[1], a pure C implementation of Ninja that's almost as fast yet easier to bootstrap needing only a C compiler.
[1] https://github.com/michaelforney/samurai
Similar to make, it does mtime chronological comparison of dependencies with target to determinate if dependencies changed. This is just so flawed and simple to fool by operations on filesystem that do not change mtime (move, rename):
1) pick a source file and make a copy of it for for later 2) edit selected source file and rebuild 3) move the copy to it's original location 4) try to rebuild, nothing happens
[ninja author] I did some thinking about this problem and eventually revisited with what I think is a pretty neat solution. I wrote about it here: https://neugierig.org/software/blog/2022/03/n2.html
Imagine if filesystems had exposed the file hash next to its mtime.
By not tracking file metadata through an index file, mtime-only incremental build systems trade a lot of reliability for only slightly more simplicity. https://apenwarr.ca/log/20181113
See also https://gittup.org/tup/build_system_rules_and_algorithms.pdf
Copy (1) and edit (2) both bump mtime, usually. It's not obvious that in the workflow you describe ninja is problematic, rather than the workflow itself (which is atypical).
> 3) move the copy to it's original location
Copy and edit do, but move (aka rename) generally does not, and that is the part that is problematic.
I don't think the described sequence of operations is all that unusual. Not the most common case for sure, but hardly unlikely in the grand scheme of things.
ninja fails to detect that file changed from last build - all it's mtime, ctime, inode and size can change, yet it's not detected as long as mtime is not newer than target.
Again, this is just a weird workflow, and you're assuming copy/edit don't bump mtime. That usually isn't the case. If you're doing this weird thing, you can just run `touch` when you move files over existing files like this to explicitly bump mtime.
I run into this issue when building against different environments, each with a
1. A library depends on a system package. To test against the different versions of the system package, the library is compiled within a container.
2. To minimize the incremental rebuild time, the `build` directory is mounted into the build container. Even when using a different version of the system package, this allows re-use of system-independent portions of the build.
3. When switching to a build container with a different version of the system package, the mtime of the system package is that of its compilation, not that of the build container's initialization. Therefore, the library is erroneously considered up-to-date.
Because the mtime is the only field checked to see if the library is up to date, I need to choose between having larger disk footprint (separate `build` directory for each build container), slower builds (touch the system package on entering the container, forcing a rebuild), or less safe incremental builds (shared `build` directory, manually touch files when necessary).
>you're assuming copy/edit don't bump mtime
Incorrect, I only assume move/rename of backup to original location doesn't change it's mtime (which it doesn't with default flags or from IDE or file manager). And I don't think this is a weird or obscure workflow, I do it all the time - have two versions of a file or make a backup before some experimental changes, and restore it later.
My guess is that it's for drop-in compatibility with make.
There is (at least) one open issue about this - the solution/alternatives are not trivial:
https://github.com/ninja-build/ninja/issues/1459
mtime rebuild logic is half-baked even by 1970s standards.
Bazel and Shake avoid this class of bug with content hashes, so a rename, restore, or tar extract does not leave the build graph in a stale state. Speed matters, but not enough to bet your repo on timestamp luck.
Postgres uses Meson+Ninja in their builds. That seems like a pretty big endorsement.
Most of Gnome is built using that too.
Serious question: how can a build tool be fast or slow? From my understanding all it does is delegate the build steps to other tools, so wouldn't those be the bottleneck? Is it the resolution of order of build steps that takes so much time that a different build system can make a difference?
[ninja author] My first post about Ninja goes into this: https://neugierig.org/software/chromium/notes/2011/02/ninja....
I'm afraid I still don't understand. One factor is having fewer features and not looking for obsolete files, that I can understand. I guess the other thing is using better rules to figure out when a file truly need to be rebuilt?
I used ninja only a few years ago when contributing to KDE software (Dolphin, Kate, KTextEditor, etc.). I had no prior experience with it and it was easy to apprehend, so a rather good experience.
My teammate has a great time reimplementing Ninja (slop-free) in Go here https://github.com/buildbuddy-io/reninja to make it even faster with Remote Build Execution.
This is cool. Going to see if I can use it at work.
All the main build tools (cmake, meson/ninja and GNU configure) have different benefits. For instance, I expect "--help" to work, but only really GNU configure supports it as-is. I could list more advantages and disadvantages in general here, but by and large I prefer meson/ninja. To me it feels by far the fastest and I also have the fewest issues usually (excluding python breaking its pip stack but that's not the fault of meson as such). ninja can be used via cmake too but most uses I see are from meson.
> ninja can be used via cmake too but most uses I see are from meson
How do you know though when the choice of cmake-generator is entirely up to the user? E.g. you can't look at a cmake file and know what generator the user will select to build the project.
FWIW I usually prefer the Ninja generator over the Makefile generator since ninja better 'auto-parallelises' - e.g. with the Makefile generator the two 'simple' options are either to run the build single-threaded or completely grind the machine to a halt because the default setting for 'parallel build' seems to heavily overcommit hardware resources. Ninja just generally does the right thing (run parallel build, but not enough parallelism to make the computer unusable).
ninja supports separate build groups and different max number of parallel jobs for each. CMake's ninja generator puts compilation and linking steps in their own respective groups. End result is by default `nproc` parallel jobs for compilation but 1 job for linking. This helps because linking can be way more memory intensive or sometimes the linker itself has support for parallelism. Most projects have only a handful of linking steps to run anyway.
I find Meson's --help fairly useful, at least compared to the disaster that is CMake's. (Try to find out, as a user not experienced with either, how you'd make a debug build.) I agree that configure --help is more useful for surfacing project-specific options, though.
Ninja religions following of treating timestamps (mtime) as 'modified' marker makes it useless with Git and large projects.
You switched some branches back and forward? Enjoy your 20 minutes rebuild.
* https://github.com/ninja-build/ninja/issues/1459
I believe the same author has made a ninja successor (n2) that uses hashes instead. Haven't tried personally but hopefully get around to try that in the near future
I have never observed that issue, and I have been using it to build MMLoC repositories. Perhaps the reason being is that I always use it coupled with ccache. Have you tried that?
ccache is a workaround for the mtime problem. You can either hash with ccache or hash directly in the build system, but either way there's no substitute for hashing something. Ccache is hashing the input to the build, but there may be elements of the build that lie outside of ccache's awareness that having a hash-aware build system would take care of. Partial rebuilds devolve to a cache invalidation problem pretty quickly either way.
The absolute best thing about coding agents is not having to waste time on build systems. I had Claude code port my autotools scripts to meson (which uses ninja) and it’s been a huge quality of life improvement.
Getting builds to work was some of the most tedious, least interesting work. It is so satisfying to watch the agent try 4 different things and magically go on its way. No more makefile syntax searches or cmake hell.
For real. I stopped writing Makefiles because of how tedious it was, but now with AI I'm back to throwing Makefiles in everything. It's wonderful to have the same build, test, release commands in all projects instead of mix compile in one, npm build in another, etc. This is my favorite part of AI
Soon you won’t even have to waste time on the code part…
It's only a waste of your tine if you don't want to do it.
I can remember having to uninstall ninja temporarily because it messed with building packages. I only use it because other packages need it.