I think this post accurately isolates the single main issue with GitHub Actions, i.e. the lack of a tight feedback loop. Pushing and waiting for completion on what's often a very simple failure mode is frustrating.
Others have pointed out that there are architectural steps you can take to minimize this pain, like keeping all CI operations isolated within scripts that can be run locally (and treating GitHub Actions features purely as progressive enhancements, e.g. only using `GITHUB_STEP_SUMMARY` if actually present).
Another thing that works pretty well to address the feedback loop pain is `workflow_dispatch` + `gh workflow run`: you still need to go through a push cycle, but `gh workflow run` lets you stay in development flow until you actually need to go look at the logs.
(One frustrating limitation with that is that `gh workflow run` doesn't actually spit out the URL of the workflow run it triggers. GitHub claims this is because it's an async dispatch, but I don't see how there can possibly be no context for GitHub to provide here, given that they clearly obtain it later in the web UI.)
I've standardized on getting github actions to create/pull a docker image and run build/test inside that. So if something goes wrong I have a decent live debug environment that's very similar to what github actions is running. For what it's worth.
I do the same with Nix as it works for macOS builds as well
It has the massive benefit of solving the lock-in problem. Your workflow is generally very short so it is easy to move to an alternative CI if (for example) Github were to jack up their prices for self hosted runners...
That said, when using it in this way I personally love Github actions
Nix is so nice that you can put almost your entire workflow into a check or package. Like your code-coverage report step(s) become a package that you build (I'm not brave enough to do this)
I run my own jenkins for personal stuff on top of nixos, all jobs run inside devenv shell, devenv handles whatever background services required (i.e. database), /nix/store is shared between workers + attic cache in local network.
Oh, and there is also nixosModule that is tested in the VM that also smoke tests the service.
First build might take some time, but all future jobs run fast. The same can be done on GHA, but on github-hosted runners you can't get shared /nix/store.
package.lock is JSON only, Nix is for the entire system, similar to a Dockerfile
Nix specifies dependencies declaritively, and more precisely, than Docker (does by default), so the resulting environment is reproducibly the same. It caches really well and doubles as a package manager.
Despite the initial learning curve, I now personally prefer Nix's declarative style to a Dockerfile
same here. though, i think bazel is better for DAGs. i wish i could use it for my personal project (in conjunction with, and bootstrapped with nix), but that's a pretty serious tooling investment that I just feel is just going to be a rabbit hole.
I tend to have most of my workflows setup as scripts that can run locally in a _scripts diorectory, I've also started to lean on Deno if I need anything more complex than I'm comfortable with in bash (even bash in windows) or powershell, since it executes .ts directly and can refer directly to modules/repos without a separate install step.
This may also leverage docker (compose) to build/run different services depending on the stage of action. Sometimes also creating "builder" containers that will have a mount point for src and output to build and output the project in different OSes, etc. Docker + QEMU allows for some nice cross-compile options.
The less I rely on Github Actions environment the happier I am... the main points of use are checkout, deno runtime, release please and uploading assets in a release.
It sucks that the process is less connected and slow, but ensuring as much as reasonable can run locally goes a very long way.
I was doing something similar when moving from Earthly. But I have since moved to Nix to manage the environment. It is a lot better of a developer experience and faster! I would checkout an environment manager like Nix/Mise etc so you can have the same tools etc locally and on CI.
Yeah, images seem to work very well as an abstraction layer for most CI/CD users. It's kind of unfortunate that they don't (can't) fully generalize across Windows and macOS runners as well, though, since in practice that's where a lot of people start to get snagged by needing to do things in GitHub Actions versus using GitHub Actions as an execution layer.
I’ve VNCed into CI to debug selenium tests failing because of platform font and scrollbar rendering. I never really thought about doing that locally in a docker container, but it definitely wouldn’t be convenient to always run those tests locally in a docker container. I guess having an option to would sort of simplify debugging, but I’d still have to VNC into the docker container I think
I tried this five years ago back when I was an engineer on the PyTorch project, and it didn't work well enough to be worth it. Has it improved since then?
It works, but there are fair amount of caveats, especially for someone working on things like Pytorch, the runtime is close but not the same, and its support of certain architectures etc can create annoying bugs.
I tried this recently and it seems like you have to make a lot of decisions to support Act. It in no way "just works", but instead requires writing actions knowing that they'll run on Act.
I tried act, on the surface it seems like a godsend. Not until you try to use it do you realize it's almost impossible to recreate any moderately complex workflow.
You HAVE to run it against a container, so if you're using self hosted runners your environment may not match at all.
I've often thought about this. There are times I would rather have CI run locally, and use my PGP signature to add a git note to the commit. Something like:
Then CI could check git notes and check the dev signature, and skip the workflow/pipeline if correctly signed. With more local CI, the incentive may shift to buying devs fancier machines instead of spending that money on cloud CI. I bet most devs have extra cores to spare and would not mind having a beefier dev machine.
I think this is a sound approach, but I do see one legitimate reason to keep using a third-party CI service: reducing the chance of a software supply chain attack by building in a hardened environment that has (presumably) had attention from security people. I'd say the importance of this is increasing.
Yeah, I'm one of those people who seems to consistently have middling-to-bad experiences with Git hooks (and Git hook managers). I think the bigger issue is that even with consistent developer tooling between both developer machines and CI, you still have the issue where CI needs to do a lot more stuff that local machines just can't/won't do (like matrix builds).
Those things are fundamentally remote and kind of annoying to debug, but GitHub could invest a lot more in reducing the frustration involved in getting a fast remote cycle set up.
But very often the CI operations _are_ the problem. It's just YAML files with unlimited configuration options that have very limited documentation, without any type of LSP.
We need SSH access to the failed instances so we can poke around and iterate from any step in the workflow.
Production runs should be immutable, but we should be able to get in to diagnose, edit, and retry. It'd lead to faster diagnosis, resolution, and fixing.
The logs and everything should be there for us.
And speaking of the logs situation, the GHA logs are really buggy sometimes. They don't load about half of the time I need them to.
This is one of the big problems we solved with the RWX CI platform (RWX.com). You can use ‘rwx run’ and it automatically syncs your local changes, so no need to push — and with our automated caching, steps like setting up the environment cache hit so you don’t have to execute the same stuff over and over again while writing your workflow. Plus with our API or MCP server you can get the results directly in your terminal so no need to open the UI at all unless you want to do some in-depth spelunking.
I’ve contemplated building my own CI tool (with a local runner) and the thing is if you assume “write a pipeline that runs locally but also on push”, then the feature depth is mostly about queuing, analyzing output, and often left off, but IMO important, charting telemetry about the build history.
Most of these are off the shelf, at least in some programming languages. It’s the integrations and the overmanagement where a lot of the weight is.
Jenkins has it's own set of issues. The theory behind GHA is you *should* be able to keep everything in git and not need a another service with it's own abstractions.
But Actions just screws this up. I consider them both to be equally bad.
I'm almost considering n8n to replace both but that will just lead to more problems.
All my Jenkins builds are between 1 and 3 lines depending on readability:
Clone
Cd $clonedFolder
./build.sh
Add an archive step, all set.
I will say even building and running unit testing in Jenkins isn’t bad, but if you aren’t careful it can get messy.
Jenkins can be a super simple build tool. Strong-arming it into doing all the things starts to get messy quickly. Build, run unit tests, archive. I’m not even big on like, building on push. Nothing wrong with it, I’m just big on keeping things as simple as possible. GHA is just a ball of complexity.
I've never used gh workflow run, but I have used the GitHub API to run workflows and wanted to show the URL. I had to have it make another call to get the workflow runs and assume the last run is the one with the correct URL. This would obviously not work correctly if there were multiple run requests at the same time. Maybe some more checking could detect that, but it works for my purposes so far.
I wonder what prevents a GH action from connecting to your VPN (Wireguard is fine) and post tons of diagnostics right onto your screen, and then, when something goes badly wrong, or when a certain point is reached, to just keep polling an HTTP endpoint for shell commands to execute.
I mean. I understand, it would time out eventually. But it may be enough time to interactively check a few things right inside the running task's process.
Of course this should only happen if the PR contains a file that says where to connect and when to stop for interactive input. You would only push such a file when an action is misbehaving, and you want to debug it.
I understand that it's a band-aid, but a band-aid is better than the nothing which is available right now.
1. Don't use bash, use a scripting language that is more CI friendly. I strongly prefer pwsh.
2. Don't have logic in your workflows. Workflows should be dumb and simple (KISS) and they should call your scripts.
3. Having standalone scripts will allow you to develop/modify and test locally without having to get caught in a loop of hell.
4. Design your entire CI pipeline for easier debugging, put that print state in, echo out the version of whatever. You don't need it _now_, but your future self will thank you when you do it need it.
5. Consider using third party runners that have better debugging capabilities
I would disagree with 1. if you need anything more than shell that starts to become a smell to me. The build/testing process etc should be simple enough to not need anything more.
I agree with #2, I meant more if you are calling out to something that is not a task runner(Make, Taskfile, Just etc) or a shell script thats a bit of a smell to me. E.g. I have seen people call out to Python scripts etc and it concerns me.
My software runs on Windows, Linux and MacOS. The same Python testing code runs on all three platforms. I mostly dislike Python but I can't think of anything better for this use case.
You might consider Deno with Typescript... it's a single exe runtime, with a self-update mechanism (deno upgrade) and can run typescript/javascript files that directly reference the repository/http/modules that it needs and doesn't require a separate install step for dependency management.
I've been using it for most of my local and environment scripting since relatively early on.
> The same Python testing code runs on all three platforms.
I have no objections to Python being used for testing, I use it myself for the end to end tests in my projects. I just don't think Python as a build script/task runner is a good idea, see below where I got Claude to convert one of my open source projects for an example.
"pwsh" is often used as the short-hand for modern cross-platform PowerShell to better differentiate it from the old Windows-only PowerShell.
I think pwsh is worth exploring. It is cross-platform. It is post-Python and the Python mantra that "~~code~~ scripts are read more often than they are written". It provides a lot of nice tools out of the box. It's built in an "object-oriented" way, resembling Python and owing much to C#. When done well the "object-oriented" way provides a number of benefits over "dumb text pipes" that shells like bash were built on. It is easy to extend with C# and a few other languages, should you need to extend it.
I would consider not dismissing it off hand without trying it just because Microsoft built it and/or that it was for a while Windows-only.
It's also both a larger download and slower to start than Java, which is not known for being light and nimble. In fact, PowerShell is so slow that you can both compile and run the equivalent C# program before PowerShell finishes launching. Not ideal for a shell or a scripting language.
Also, the newer versions aren't included with Windows, which would have been useful – instead Windows includes an incompatible older version that admonishes you to download the new version. But why would you download several hundred megabytes of pwsh when you can equally well download any other language runtime?
Also, it sends "telemetry" to Microsoft by default.
Also, the error handling is just awful, silencing errors by default, requiring several different incantations to fix.
Also, the documentation is vague and useless. And the syntax is ugly.
It gets faster to boot on subsequent launches and some distros are now packaging pre-baked versions.
The new versions aren't included in Windows and the old versions are still in Windows for the exact same reasons of Windows backwards compatibility requirements. But at this point the bootstrap on Windows is as easy as `winget install --id Microsoft.PowerShell`.
The error handling isn't far from the bash defaults, but the magic incantations actually tell you what they do versus the number of bash scripts littered with `set -euxo pipefail` is the exact same as the number of scripts that need an `$ErrorActionPreference = "Stop"` and/or a `$PSNativeCommandUseErrorActionPreference = true`.
I find the documentation less vague and more useful than the average `man` page and the syntax is fine (and better than bash) to me, but I understand how much of that is personal preference and familiarity.
Huh? Who cares if the script is .sh, .bash, Makefile, Justfile, .py, .js or even .php? If it works it works, as long as you can run it locally, it'll be good enough, and sometimes it's an even better idea to keep it in the same language the rest of the project is. It all depends and what language a script is made in shouldn't be considered a "smell".
Once you get beyond shell, make, docker (and similar), dependencies become relevant. At my current employer, we're mostly in TypeScript, which means you've got NPM dependencies, the NodeJS version, and operating system differences that you're fighting with. Now anyone running your build and tests (including your CI environment) needs to be able to set all those things up and keep them in working shape. For us, that includes different projects requiring different NodeJS versions.
Meanwhile, if you can stick to the very basics, you can do anything more involved inside a container, where you can be confident that you, your CI environment, and even your less tech-savvy coworkers can all be using the exact same dependencies and execution environment. It eliminates entire classes of build and testing errors.
I use to have my Makefile call out and do `docker build ...` and `docker run ...` etc with a volume mount of the source code to manage and maintain tooling versions etc.
It works okay, better than a lot of other workflows I have seen. But it is a bit slow, a bit cumbersome(for langs like Go or Node.js that want to write to HOME) and I had some issues on my ARM Macbook about no ARM images etc.
I would recommend taking a look at Nix, it is what I switched to.
* It is faster.
* Has access to more tools.
* Works on ARM, X86 etc.
I've switched to using Deno for most of my orchestration scripts, especially shell scripts. It's a single portable, self-upgradeable executable and your shell scripts can directly reference the repositories/http(s) modules/versions it needs to run without a separate install step.
I know I've mentioned it a few times in this thread, just a very happy user and have found it a really good option for a lot of usage. I'll mostly just use the Deno.* methods or jsr:std for most things at this point, but there's also npm:zx which can help depending on what you're doing.
It also is a decent option for e2e testing regardless of the project language used.
Shell and bash are easy to write insecurely and open your CI runners or dev machines up for exploitation by shell injection. Non-enthusiasts writing complex CI pipelines pulling and piping remote assets in bash without ShellCheck is a risky business.
You shouldn't be pulling untrusted assets in CI regardless. Hacking your bash runner is the hardest approach anyways, just patch some subroutine in a dependency that you'll call during your build or tests.
> Huh? Who cares if the script is .sh, .bash, Makefile, Justfile, .py, .js or even .php?
Me, typically I have found it to be a sign of over-engineering and found no benefits over just using shell script/task runner, as all it should be is plumbing that should be simple enough that a task runner can handle it.
> If it works it works, as long as you can run it locally, it'll be good enough,
Maybe when it is your own personal project "If it works it works" is fine. But when you come to corporate environment there starts to be issues of readability, maintainability, proprietary tooling, additional dependencies etc I have found when people start to over-engineer and use programming languages(like Python).
E.g.
> never_inline 30 minutes ago | parent | prev | next [–]
> Build a CLI in python or whatever which does the same thing as CI, every CI stage should just call its subcommands.
However,
> and sometimes it's an even better idea to keep it in the same language the rest of the project is
I'll agree. Depending on the project's language etc other options might make sense. But personally so far everytime I have come across something not using a task runner it has just been the wrong decision.
> But personally so far everytime I have come across something not using a task runner it has just been the wrong decision.
Yeah, tends to happen a lot when you hold strong opinions with strong conviction :) Not that it's wrong or anything, but it's highly subjective in the end.
Typically I see larger issues being created from "under-engineering" and just rushing with the first idea people can think of when they implement things, rather than "over-engineering" causing similarly sized future issues. But then I also know everyone's history is vastly different, my views are surely shaped by the specific issues I've witnessed (and sometimes contributed to :| ), than anything else.
> Yeah, tends to happen a lot when you hold strong opinions with strong conviction :) Not that it's wrong or anything, but it's highly subjective in the end.
Strong opinions, loosely held :)
> Typically I see larger issues being created from "under-engineering" and just rushing with the first idea people can think of when they implement things, rather than "over-engineering"
Funnily enough running with the first idea I think is creating a lot of the "over-engineering" I am seeing. Not stopping to consider other simpler solutions or even if the problem needs/is worth solving in the first place.
> Yeah, tends to happen a lot when you hold strong opinions with strong conviction :) Not that it's wrong or anything, but it's highly subjective in the end.
I quickly asked Claude to convert one of my open source repos using Make/Nix/Shell -> Python/Nix to see how it would look. It is actually one of the better Python as a task runners I have seen.
While the Python version is not as bad as I have seen previously, I am still struggling to see why you'd want it over Make/Shell.
It introduces more dependencies(Python which I solved via Nix) but others haven't solved this problem and the Python script has dependencies(such as Click for the CLI).
It is less maintainable as it is more code, roughly x3 the amount of the Makefile.
To me the Python code is more verbose and not as simple compared to the Makefile's target so it is less readable as well.
> It introduces more dependencies(Python which I solved via Nix) but others haven't solved this problem and the Python script has dependencies(such as Click for the CLI).
UV scripts are great for this type of workflow
There are even scripts which will install uv in the same file effectively making it just equivalent to ./run-file.py and it would handle all the dependency management the python version management and everything included and would work everywhere
Personally I end up just downloading uv and so not using the uv download script from this but if I am using something like github action which are more (ephemeral?) I'd just do this.
Something like this can start out simple and can scale much more than the limitations of bash which can be abundant at times
That being said, I still make some shell scripts because executing other applications is first class support in bash but not so much in python but after discovering this I might create some new scripts with python with automated uv because I end up installing uv on many devices anyway (because uv's really good for python)
I am interested in bun-shell as well but that feels way too much bloated and even not used by many so less (AI assistance at times?) and I haven't understood bun shell at the same time too and so bash is superior to it usually
So previously when I have seen Python used as a task runner I think they used UV to call it. Although I don't think they had as a complete solution as your here auto-installing UV etc.
Although the example you've linked is installing UV if missing, the version is not pinned, I also don't think it is handling missing Python which is not pinned even if installed locally. So you could get different versions on CI vs locally.
While yes you are removing some of the dependencies problems created via using Python over Make/Shell I don't think this completely solves it.
> Something like this can start out simple and can scale much more than the limitations of bash which can be abundant at times
I personally haven't witnessed anytime I would consider the scales to have tipped in favour of Python and I would be concerned if they ever do, as really the task runner etc should be plumbing, so it should be simple.
> That being said, I still make some shell scripts because executing other applications is first class support in bash but not so much in python but after discovering this I might create some new scripts with python with automated uv because I end up installing uv on many devices anyway (because uv's really good for python)
Using Python/UV to do anything more complex than my example PR above?
I think UV scripts can/will actually install python and manage it itself as well and you can actually pin a specific version of python itself via Uv scripts
# Use some syntax added in Python 3.12
type Point = tuple[float, float]
print(Point)
> Using Python/UV to do anything more complex than my example PR above?
I can agree that this might be complex but that complexity has a trade off and of course nothing is shoe fits all but there are times when someone has to manage a complex CI environment and I looked at and there are some CI deterministic options too like invoke etc. and when you combine all of these, I feel like the workflow can definitely be interesting to say the least
Once again, I don't know what really ends up in github actions since I have never really used it properly, I am basing its critiques based on what I've read and what solutions (python came quite frequently) and something recently which I discovered (which was the blog)
Does it really matter if its a global install of uv or not especially on Github Actions
Also if this still bothers you, nothing stops you from removing the first x lines of code and having it in another .py file if this feels obnoxious to you
> Oh, and later the author suggests the script modify itself after running. What the fuck. Absolutely unacceptable way to deploy software.
Regarding author suggest its removes itself its because it does still feel clutterish but there is virtually 0 overhead in using/having it still be if you are already using uv or want to use uv
Oh also, (I am not the Author) but I have played extensively with UV and I feel like the script can definitely be changed to install it locally rather than globally.
They themselves mention it as #overkill on their website but even then it is better than whatever github action is
Yes I believe the same too and I think we are on the same goal. I think that I can probably patch this code to install uv, let's say locally instead of globally if that's a major concern. I feel like its not that hard.
It's easy enough to patch. It's the philosophy that bugs me. We already have a huge problem with routine workflows pulling things from the network (often, without even a semblance of hash-locking) and foregoing the traditional separation between environment setup and business logic. There's a lot of value into having discrete steps for downloading/installing stuff and doing development, because then you can pay special attention to the former, look for anything odd, read release notes, and so on. Between explicit, human-solicited upgrades, dev workflows should be using, ideally, vendored dependencies, or, if not that, then at least stuff that's hash-verified end-to-end.
Someday, someone is going to have a really big disaster that comes out of casual getting unauthenticated stuff from somebody else's computer.
I think your reason of worrying is either that A) packages can update and contain malware or B) Uv's installation itself. might have malware if any of A) or B) get hacked
Regarding B) I feel like they provide attestations via GitHub Artifact Attestations and the script could once again be modified to actually verify it via github attestations and they also provide ghcr artifacts (as such immutability) atleast of docker images and I looked further into it and it seems that you can use github artifacts to upload normal binary files as well so I will probably take a look into seeing if I can do something like this for uv's ghcr
Effectively after A) and B) the trust just ends up being reliant on Github's Microsoft infrastructure (usually) and perhaps python infrastructure which is on fastly
But I feel like this is for cases of extremely sensitive workflows But I feel like I might still take a look at it because security still feels very interesting to me and just because of this discussion, I can see at some pointers of following up on curiosity lol
Anyways would love to continue our discussion and probably update you on trying to make a script which could actually be completely pinned (atleast uv binary instead of just running a shell script from the astral servers in such case)
I find that shell scripting has a sharp cliff. I agree with the sentiment that most things are over engineered. However it’s really easy to go from a simple shell script running a few commands to something significantly more complex just to do something seemingly simple, like parse a semantic version, make an api call and check the status code etc, etc.
The other problem with shell scripting on things like GHA is that it’s really easy to introduce security vulnerabilities by e.g forgetting to quote your variables and letting an uncontrolled input through.
There’s no middle ground between bash and python and a lot of functionality lives in that space.
> However it’s really easy to go from a simple shell script running a few commands to something significantly more complex just to do something seemingly simple, like parse a semantic version, make an api call and check the status code etc, etc.
Maybe I keep making the wrong assumption that everyone is using the same tools the same way and thats why my opinions seem very strong. But I wouldn't even think of trying to "parse a semantic version" in shell, I am treating the shell scripts and task runners as plumbing, I would be handing that of a dedicated tool to action.
Let’s say have a folder of tarballs and need to install the latest version. I could reach for an additional “dedicated” tool, get it installed into the CI environment and then incorporate it into the build process, or I could just make a slight modification to my existing shell script and do something like “ls mypkg-*.tar.gz | xargs -n1 | sort -Vr | head -n1” and then move on. But then we start publishing multiple release candidates and now I need to add to that logic to further distinguish between earlier and later rc versions. And so on and so forth…
Now I’m in agreement with you that this is a bad fit for shell scripting, but it is often pragmatic and expedient. And bc there is a cliff between bash and (say) python, some of the time, you’re going to choose the path of least resistance.
Now scale this out to a small team of engineers all facing the same dumb decision of needing to make some tradeoff when they would much rather be writing application logic. The lack of a ubiquitous and robust intermediate language leads to brittle CI fraught with security vulnerabilities.
While the example I provided is a bit contrived, this behavior isn’t hypothetical. I see it everywhere I’ve worked.
Yes I am seeing what your saying, it can start of simple and people will choose the path of least resistance and you'll end up with a ball of mud eventually. Fortunately I have not come across anything that bad yet.
I know it is a made up example, but I am not sure Python would improve the situation. I would be looking at either Docker or SBOMs to try and improve the situation and then hopefully the release process would become a lot simpler.
yea imagine having to maintain a python dependency (which undergoes security constraints) all because some junior cant read/write bash... and then that junior telling you you're the problem lmao
> If your CI can do things that you can't do locally: that is a problem.
IME this is where all the issues lie. Our CI pipeline can push to a remote container registry, but we can't do this locally. CI uses wildly different caching strategies to local builds, which diverges. Breaking up builds into different steps means that you need to "stash" the output of stages somewhere. If all your CI does is `make test && make deploy` then sure, but when you grow beyond that (my current project takes 45 minutes with a _warm_ cache) you need to diverge, and that's where the problems start.
Ironically, at least for a couple recent projects... just installing dependencies fresh is as fast on GH Actions as the GH caching methods, so I removed the caching and simplified the workflows.
> With uv, it turns out that it's often faster to omit pre-built wheels from the cache (and instead re-download them from the registry on each run). On the other hand, caching wheels that are built from source tends to be worthwhile, since the wheel building process can be expensive, especially for extension modules.
> If your CI can do things that you can't do locally: that is a problem.
Probably most of the times when this is an actual problem, is building across many platforms. I'm running Linux x86_64 locally, but some of my deliverables are for macOS and Windows and ARM, and while I could cross-compile for all of them on Linux (macOS was a bitch to get working though), it always felt better to compile on the hardware I'm targeting.
Sometimes there are Windows/macOS-specific failures, and if I couldn't just ssh in and correct/investigate, and instead had to "change > commit > push" in an endless loop, it's possible I'd quite literally would lose my mind.
I literally had to do this push > commit > test loop yesterday because apparently building universal Python wheels on MacOS is a pain in the ass. And I don't have a mac, so if I want to somewhat reliably reproduce how the runner might behave, I have to either test it on GH actions or rent one from something like Scaleway. Mainly because I don't currently knwo how else to do it. It's so, so frustrating and if anyone has ideas on making my life a bit better that would be nice lol.
there is quickemu which can install mac vm on linux (or any other host) rather quickly, what are your thoughts on it (I am an absolute quickemu shill because I love that software)
That's what I intend to use it for, Mac and Windows... I'm starting on an app that I want to work cross platform (tauri/rust w/ react+mui) and want to be able to do manual testing or troubleshooting as needed on mac and windows without needing a separate machine.
My laptop is an M1 MacBook Air, and I do have an N100 I could use for Windows... I'd just assume use my fast desktop which even emulated is likely faster and not have to move seats.
yes, I think just the amount of friction it can reduce might be worth it in the first place.
Oh btw although there are many primitives which help transferring files between VM's and others by having sshfs etc., one of the things which I enjoyed doing in quickemu is using the beloved piping-server
The speeds might be slow but I was using it to build simple shell scripts and you can self host it or deploy on cf workers too most likely which is really simple but I haven't done it
But for quick deployments/transfers of binaries/simple files, its great as well. Tauri is meant to be lightweight/produce small binaries so I suppose one can try it but there are other options as well
Piping Serrvers + quickemu felt like a cheatcode to me atleast for more ephemeral vm's based workflow but of course YMMV
Good luck with your project! I tried building a tauri app once for android just out of mere curiosity on linux and it was hell. I didn't know anything about android development but setting up the developer environment was really hard and I think I forgot everything I learnt from that but wish I had made notes or even video documenting the process
Fortunately/Unfortunately it wouldn't be a good experience for Phone use, maybe table as part of it will be displaying BBS-ANSI art and messages which lends itself to a larger display.
> If your CI can do things that you can't do locally: that is a problem.
Completely agree.
> I'm a huge fan of "train as you fight", whatever build tools you have locally should be what's used in CI.
That is what I am doing, having my GitHub Actions just call the Make targets I am using locally.
> I mean, at some point you are bash calling some other language anyway.
Yes, shell scripts and or task runners(Make, Just, Task etc) are really just plumbing around calling other tools. Which is why it feels like a smell to me when you need something more.
I don't agree with (1), but agree with (2). I recommend just putting a Makefile in the repo and have that have CI targets, which you can then easily call from CI via a simple `make ci-test` or similar. And don't make the Makefiles overcomplicated.
Of course, if you use something else as a task runner, that works as well.
For certain things, makefiles are great options. For others though they are a nightmare. From a security perspective, especially if you are trying to reach SLSA level 2+, you want all the build execution to be isolated and executed in a trusted, attestable and disposable environment, following predefined steps. Having makefiles (or scripts) with logical steps within them, makes it much, much harder to have properly attested outputs.
Using makefiles mixes execution contexts between the CI pipeline and the code within the repository (that ends up containing the logic for the build), instead of using - centrally stored - external workflows that contains all the business logic for the build steps (e.g., compiler options, docker build steps etc.).
For example, how can you attest in the CI that your code is tested if the workflow only contains "make test"? You need to double check at runtime what the makefile did, but the makefile might have been modified by that time, so you need to build a chain of trust etc.
Instead, in a standardized workflow, you just need to establish the ground truth (e.g., tools are installed and are at this path), and the execution cannot be modified by in-repo resources.
That doesn't make any sense. Nothing about SLSA precludes using make instead of some other build tool. Either inputs to a process are hermetic and attested or they're not. Makefiles are all about executing "predefined steps".
It doesn't matter whether you run "make test" or "npm test whatever": you're trusting the code you've checked out to verify its own correctness. It can lie to you either way. You're either verifying changes or you're not.
You haven't engaged with what I wrote, of course it doesn't make sense.
The easiest and most accessible way to attest what has been done is to have all the logic of what needs to be done in a single context, a single place. A reusable workflow that is executed by hash in a trusted environment and will execute exactly those steps, for example.
In this case, step A does x, and step B attests that x has been done, because the logic is immutably in a place that cannot be tampered with by whoever invokes that workflow.
In the case of the makefile, in most cases, the makefile (and therefore the steps to execute) will be in a file in the repository, I.e., under partial control of anybody who can commit and under full control of those who can merge.
If I execute a CI and step A now says "make x", the semantic actually depends on what the makefile in the repo includes, so the contexts are mixed between the GHA workflow and the repository content.
Any step of the workflow now can't attest directly that x happened, because the logic of x is not in its context.
Of course, you can do everything in the makefile, including the attestation steps, bringing them again in the same context, but that makes it so that once again the security relevant steps are in a potentially untrusted environment. My thinking specifically hints at the case of an organization with hundreds of repositories that need to be brought under control.
Even more, what I am saying make sense if you want to use the objectively convenient GH attestation service (probably one of the only good feature they pushed in the last 5 years).
Usually, the people writing the Makefile are the same that could also be writing this stuff out in a YAML (lol) file as the CI instructions, often located in the same repository anyway. The irony in that is striking. And then we have people who can change environment variables for the CI workflows. Usually, also developers, often the same people that can commit changes to the Makefile.
I don't think it changes much, aside from security theater. If changes are not properly reviewed, then all fancy titles will not help. If anything, using Make will allow for a less flaky CI experience, that doesn't break the next time the git hoster changes something about their CI language and doesn't suffer from YAMLitis.
You're correct. It's absolutely security theater. Either you trust the repository contents or you don't. There's no, none, zilch trust improvement arising from the outer orchestration being done in a YAML file checked into the repo and executed by CI instead of a Makefile also executed by CI.
What's the threat model Wilder is using exactly? Look, I'm ordinarily all for nuance and saying reasonable people can disagree when it comes to technical opinions, but here I can't see any merit whatsoever to the claim that orchestrating CI actions with Make is somehow a security risk when the implementations of these actions at some level live in the repo anyway.
> when the implementation of these actions at some level live in the repo anyway
This is the false assumption. You can standardize that not to happen, and you can verify at runtime that's not the case.
You can control admission of container images for example, restricting those that were built by your workflow X (the approved, central, controlled, standard one) and reject anything else. You do this via build provenance attestation.
With makefile I don't know how you can achieve standard and centrally manageable (and verifiable) instructions (which are part of SLSA).
> With makefile I don't know how you can achieve standard and centrally manageable (and verifiable) instructions (which are part of SLSA).
The way I'm thinking about it, we distinguish instructions from policy. Policy, of course, has to be imposed from outside a package somehow, but the instructions taken within policy seem like things that should evolve with it. For example, "no network access" might be an externally enforced policy. Another might be limitations of dependencies, enforced by making exceptions to the previous policy.
But in the end, you have to do something. Are you suggesting a model in which a project's entire test suite lives in a separate repository with an access control policy that differs from that project's so that the project's authors can't cheat on test coverage or pass rate?
Sure, you can do that, but the instructions for running those tests still have to live somewhere, and a makefile in that test repository seems like as good a place as any.
A classic example would be build (container build) images instructions.
Of course, other examples are running a security static analysis, running tests, or whatever other standard practice is desidered for security.
Say that you build 100 images, each with its own repository where there is the code, there is the Dockerfile etc.
Now, if you build images with "make build", it means that ultimately the actual command line flags for the build command are going to be in the Makefile, inside each repository. These flags could include flags like --cache-from, build arguments, --add-host, or many other flags that may not be desirable in an organization.
Imagine what the workflow file would like for such a case: you have your in-repo CI file that clones the repo, and then executes make test, make build, make push, etc.
Do you know if make push sent the PCI registry secret somewhere else on the internet? Do you know what exactly flags were used to build? Not really, you need to trust the code inside the repo to at most collect this information, and how do you know it was not tampered or spoofed?
Compare it with the following scenario:
All 100 repos at some point in that CI invoke an external workflow, which contains instructions to build images, push them to the registry (and run tests before, whatever). This can obviously also be a combination or workflows, I am saying one for simplicity.
This workflow (or workflows) at the end produces an attestation, including provenance data and about what is run.
It can do so reliably, because the instructions of what had to be done are within its own context. There is no input that comes from outside (e.g., the repository) that determined build flags or instructions, outside of what the workflow exposed (e.g., few variables).
In the second case, I can then use a policy that restrict images that have attestation and that were built by my workflow(s) that contain standard and deterministic instructions. This way I know that no image was built using weird build flags, including strange dependencies, maybe tampering with network and DNS (in makefile I can write whatever) or local docker cache, etc.
I know this because a standard workflow was executed in a fresh, "sterile" environment with a standard set of instructions, that can't be changed or tampered by those controlling the repository content.
Note that all of the above is solely concerned with being able to make security statements on the build artifact produced, not on the context of the build (e.g., no network access).
That's a great point. If we keep following the requirement for attestation to its logical conclusion we would end up replicating the entire server running the repository at the source, then the cycle repeats
Not really, that's the point. Reusable workflow, in a tightly controlled repos avoid exactly what you are saying and they are a fairly standard practice (if anything, also to avoid having 200 versions of CI instructions).
You can also verify attestations provenance by enforcing attestation performed via that particular, approved workflow, which is not security theater, it's actual real controls.
Makefile or scripts/do_thing either way this is correct. CI workflows should only do 1 thing each step. That one thing should be a command. What that command does is up to you in the Makefile or scripts. This keeps workflows/actions readable and mostly reusable.
Neither do most people, probably but it's kinda neat how they suggested fix for github actions' ploy to maintain vendor lock-in is to swap it with a language invented by that very same vendor.
I was once hired to manage a build farm. All of the build jobs were huge pipelines of Jenkins plugins that did various things in various orders. It was a freaking nightmare. Never again. Since then, every CI setup I’ve touched is a wrapper around “make build” or similar, with all the smarts living in Git next to the code it was building. I’ll die on this hill.
They could noop on local runs but be reflected in the github/gitlab as separate steps/stages and allow resumes and retries and such. As it stands there's no way to really have CI/CD run the exact same scripts locally and get all the insights and functionality.
I haven't seen anything like that but it would be nice to know.
I typically use make for this and feel like I’m constantly clawing back scripts written in workflows that are hard to debug if they’re even runnable locally.
This isn’t only a problem with GitHub Actions though. I’ve run into it with every CI runner I’ve come across.
How do you handle persistent state in your actions?
For my actions, the part that takes the longest to run is installing all the dependencies from scratch. I'd like to speed that up but I could never figure it out. All the options I could find for caching deps sounded so complicated.
> How do you handle persistent state in your actions?
You shouldn't. Besides caching that is.
> All the options I could find for caching deps sounded so complicated.
In reality, it's fairly simple, as long as you leverage content-hashing. First, take your lock file, compute the sha256sum. Then check if the cache has an artifact with that hash as the ID. If it's found, download and extract, those are your dependencies. If not, you run the installation of the dependencies, then archive the results, with the ID set to the hash.
It really isn't more to it. I'm sure there are helpers/sub-actions/whatever Microsoft calls it, for doing all of this with 1-3 lines or something.
The tricky bit for me was figuring out which cache to use, and how to use and test it locally. Do you use the proprietary github actions stuff? If the installation process inside the actions runner is different from what we use in the developer machines, now we have two sets of scripts and it's harder to test and debug...
> Do you use the proprietary github actions stuff?
If I can avoid it, no. Almost everything I can control is outside of the Microsoft ecosystem. But as a freelancer, I have to deal a bunch with GitHub and Microsoft anyways, so in many of those cases, yes.
Many times, I end up using https://github.com/actions/cache for the clients who already use Actions, and none of that runs in the local machines at all.
Typically I use a single Makefile/Justfile, that sometimes have most of the logic inside of it for running tests and what not, sometimes shell out to "proper" scripts.
But that's disconnected from the required "setup", so Make/Just doesn't actually download dependencies, that's outside of the responsibilities of whatever runs the test.
And also, with a lot of languages, it doesn't matter if you run an extra "npm install" over already existing node_modules/, it'll figure out what's missing/there already, so you could in theory still have "make test" do absolute everything locally, including installing dependencies (if you now wish this), and still do the whole "hash > find cache > extract > continue" thing before running "make test", and it'll skip the dependencies part if it's there already.
Depends on the build toolchain but usually you'd hash the dependency file and that hash is your cache key for a folder in which you keep your dependencies. You can also make a Docker image containing all your dependencies but usually downloading and spinning that up will take as long as installing the dependencies.
For things like installing deps, you can use GitHub Actions or several third party runners have their own caching capabilities that are more mature than what GHA offers.
We use Nuke for this purpose and I really like it. We're a .NET shop, and Nuke is C#, so works pretty well. Almost all our Nuke targets are shared between local and CI except some docker stuff that needs caching in CI.
Minor variance on #1, I've come to use Deno typescripts for anything more complex than what can be easily done in bash or powershell. While I recognize that pwsh can do a LOT in the box, I absolutely hate the ergonomics and a lot of the interactions are awkward for people not used to it, while IMO more developers will be more closely aligned to TypeScript/JavaScript.
Not to mention, Deno can run TS directly and can reference repository/http modules directly without a separate install step, which is useful for shell scripting beyond what pwsh can do. ex: pulling a dbms client and interacting directly for testing, setup or configuration.
For the above reasons, I'll also use Deno for e2e testing over other languages that may be used for the actual project/library/app.
Step 0. Stop using CI services that purposefully waste your time, and use CI services that have "Rebuild with SSH" or similar. From previous discussions (https://news.ycombinator.com/item?id=46592643), seems like Semaphore CI still offers that.
Its not Github Actions' fault but the horrors people create in it, all under the pretense that automation is simply about wrapping a GitHub Action around something. Learn to create a script in Python or similar and put all logic there so you can execute it locally and can port it to the next CI system when a new CTO arrives.
I think in this case they hate the fact that they cannot easily SSH into the failing VM and debug from there. Like "I have to edit my workflow, push it, wait for it to run and fail, and repeat".
Yep, my company moved onto GH Actions a few years ago and this was probably the single biggest pain point. But also the whole system just feels awkward and annoying to work with. It feels like a classic Microsoft product that would never get traction from a standalone company but because it's part of Microsoft/GitHub and is "good enough" lots of people put up with it.
I had the weirdest thing happen a few days ago.. and only seemed to be happening in the GH runner for a PR.... somehow a function was duplicated in the action runner, and not in local or anwhere else... no idea how the corruption happened... it literally took me hours or pushing minor changes to try to correct the issue... I finally cat'd that file contents out and yep, the function was duplicated... no idea how. Had to create a new branch, copy the changes and then it worked.|
Still no idea what happened or how to ever fix/prevent it again.
Unless you've got something sanitizing env prints in your logs, no, they don't get masked. I've leaked secrets accidentally while debugging environment variable issues.
Of course the bigger side of the issue is that anyone who can run code in the CI context can view all data in the CI context. So if an untrusted dev can write a build script that executes with secrets in the environment, they can leak those secrets. CI is RCE-as-a-service, you implicitly give permission for anyone who can edit build scripts to run arbitrary code. Like you say, they can echo the secret to a file & upload it elsewhere, and they can also use it directly from the edited build script.
Yes. I've done it. That's why there is an "approve and run" button when the PR comes from another repo. So that the maintainers can check if the PR is trying to exfiltrate secrets... or do other bad things of course.
The whole point of the github action secrets is that they are accessible by the CI when needed. Which means they are accessible. It's just a convenient way to store the secret in such a way that it is not completely public. But you still have to trust github and your maintainers.
No, it is github's fault. They encourage the horrors because they lead to vendor lock in. This is the source of most of Microsoft's real profit margins.
This is probably why they invented a whole programming language and then neglected to build any debugging tools for it.
> Learn to create a script in Python or similar and put all logic there so you can execute it locally and can port it to the next CI system when a new CTO arrives.
That's a perfectly good several weeks of time un-estimateable busywork I could be getting paid for!
All discussions so far seem to be around building artifacts which is the easy part. How to make the full CI/CD pipeline work with GHA is a pain. GHA has no state mechanism easily used, you have to create it all yourself.
systemd units that are small, simple, and call into a single script are usually fantastic. There's no reason for these scripts to be part of another init system; but making as much of your code completely agnostic to the env it runs in sounds good regardless. I think that's the feeling you're feeling.
The way I deal with all these terrible CI platforms (there is no good one, merely lesser evils) is to do my entire CI process in a container and the CI tool just pulls and runs that. You can trivially run this locally when needed.
Of course, the platforms would rather have you not do that since it nullifies their vendor lock-in.
1. When the build fails, you can SSH into the machine and debug it from there.
2. You can super easily edit & run the manifest without having to push to a branch at all. That makes it super easy to even try a minimum reproducible example on the remote machine.
Other than that, self-hosting (with Github or preferrably Forgejo) makes it easy to debug on the machine, but then you have to self-host.
Self-hosted runners with Github is a whole world of pain because it literally just runs commands on the host and does not handle provisioning/cleanup, meaning you need to make sure your `docker run` commands don't leave any leftover state that can mess up future/concurrent builds. It doesn't even handle concurrency by itself, so you have to run multiple instances of the runner.
Github supports ephemeral runners which are limited to a single job.
You can use `workflow_job` webhook to be notified of a new job, after that you need to call `generate-jitconfig` API to get a just-in-time configuration token, and then you can start a Github runner in ephemeral mode with the JIT token.
This allows you to orchestrate Docker containers, KVM instances, etc., which are used for a single time, and then destroyed.
There are some open source projects, such as using ephemeral Kubernetes pods with the ephemeral runners.
Thats what i always did for our gitlab CI pipeline, just deploy dedicated images for different purposes. We had general terraform images for terraform code, this made it easy to standardize versions etc. Then we made specific ones for projects with a lot of dependencies so we could run the deployment pipeline in seconds instead of minutes. But now you need to maintain the docker images too. All about trade-offs.
It's kind of their whole thing, when you think about it. They didn't get to where they are by playing nice with others. If you're supporting anything in the Apple ecosystem, the fix is in.
Github delivers an exceptionally detailed webhook. Ironically, there is zero actual lock in, the CI/CD cargo cult is that strong they didn't even need it.
I like Github Actions and it is better than what I used before (Travis) and I think it solves an important problem. For OSS projects it's a super valuable free resource.
For me what worked wonders was adopting Nix. Make sure you have a reproducible dev environment and wrap your commands in `nix-shell --run`, or even better `nix develop --command`, or even better your most of your CI tasks derivations that run with `nix build` or `nix flake check`.
Not only does this make it super easy to work with Github Actions, also with your colleagues or other contributors.
Second the Nix approach. One can even build a github actions-compatible container out of a flake and have actions run in it. I have done so for my personal projects https://github.com/anttiharju/compare-changes
Wrote a comment to someone here but I thought of deciding to make a main comment here as well
Note that I don't really use github actions much but have heard about its architecture
From my understanding, I feel like Github actions should just be a call to some bash or python file. Bash has its issues so I prefer python
I recommend people to take a look at https://paulw.tokyo/standalone-python-script-with-uv/ and please tell me if something like this might be perfect for python scripts in github actions as this script would automatically install uv, get all the dependencies of python and even the runtime I think and then execute the python code all while being very managable usually and it can run locally as well
The only Issue I feel like I might have with this is say why go something with this complex when bash exists or the performance concerns of installing uv but considering its github actions, I feel like the latter is ruled out.
Bash is good as well but bash has some severe limitations. and I feel like Python can be good case for something like this plus its ecosystem is a bit mature and you could even create web servers or have some logs be reported to your custom server or automate just basically everything
To me this script feels like the best of both worlds and something genuinely sane to build upon.
> I feel like Github actions should just be a call to some bash or python file. Bash has its issues so I prefer python
...and that's how it is supposed to be used. The YAML file is mainly there for defining the runtime environment (e.g. the operating system or docker image to run in), and to describe the job dependency tree. The actual actions to execute should be delegated to scripts outside the GH YAML.
May also be worth adding mise to the README install options given its growing popularity. I installed globally with `mise use -g ubi:frankwiles/gg@latest`. I've invoked gg so many times over the last day, it's exactly what I needed - thanks again.
As a founder of Depot [0], where we offer our own faster and cheaper GitHub Actions runners, I can assure everyone that this feeling is the majority and not the minority.
Sounds strange to say as someone who has a product that is built around making GitHub Actions exponentially faster to close these feedback loops faster.
But I can honestly say it's only really possible because the overall system with GitHub Actions is so poor. We discover new bottlenecks in the runner and control plane weekly. Things that you'd think would be simple are either non-existent, don't work, or have terrible performance.
I'm convinced there are better ways of doing things, and we are actively building ideas in that realm. So if anybody wants to learn more or just have a therapy session about what could be better with GitHub Actions, my email is in my bio.
Would a tool like act help here? (https://github.com/nektos/act) I suppose orchestration that is hiding things from different processor architectures could also very well run differently online than offline, but still.
That's correct and it's linux-only (as of the last time I looked), you can run it on macOS but you can't run macOS runners (which is where I need the most help debugging normally, for building iOS apps).
It's generally very helpful - someone else mentioned here the fundamental problem is lack of a tight feedback loop. It doesn't perfectly replicate the GH environment, but for my use case that doesn't matter and it's super nice to have.
The main problem with actions is the way they advertise its usage "just put workflows together" is a horrible and non-debuggable way to do things. But even in the tech itself, caching is pretty stingy which can slow dev builds for fairly simple projects because every run will repeat some common work unless you have the cache perfectly configured (did u cover npm, docker, etc. with cache keys correctly?)
Looking at these flaws, running workflows from a persistent VM of ur own becomes pretty tempting because you don't need to copy caches around and can easily SSH in.
GH Actions isn't great compared to other CI systems, but it's also not particularly worse until you get into the nitty gritty details.
The most important advice is probably to put as much code as possible into locally runnable scripts written in a cross-platform scripting language (e.g. Python or Node.js) to avoid 'commit-push-ci-failure' roundtrips.
Only use the GH Actions YAML for defining the runtime environment and job dependency tree.
The best CI platforms let you "Rebuild with SSH" or something similar, and instead of having the cycle of "change > commit > push > wait > see results" (when you're testing CI specific stuff, not iterating on Makefiles or whatever, assuming most of it is scripts you can run both locally and in CI), you get a URL to connect to while the job is running, so you can effectively ensure manually it works, then just copy-paste whatever you did to your local sources.
I use that a lot with SourceHut: after a build fails, you have 10 minutes to SSH into the machine and debug from there. Also they have a very nice "edit manifest and run" feature that makes it easy to quickly test a change while debugging.
Are there other platforms allowing that? Genuinely interested.
> after a build fails, you have 10 minutes to SSH into the machine and debug from there.
Ah, that's like 90% of the way there, just need to enable so the SSH endpoint is created at the beginning, rather than the end, so you could for example watch memory usage and stuff while the "real" job is running in the same instance.
But great to hear they let you have access to the runner at all, only that fact makes it a lot better than most CI services out there, creds to SourceHut.
No. Act is for running actions locally. What was mentioned is a way to insert an SSH step at some well-chosen point of a workflow so you can login at the runner and understand what is wrong and why it's not working. I have written one such thing, it relies on cloudflare free tunnels. https://github.com/efrecon/sshd-cloudflared. There are other solutions around to achieve more or less the same goal.
I designed a fairly complex test matrix with a lot of logic offloading to the control mechanisms Gitlab offers. You create job templates or base jobs that control the overall logic and extend them for each particular use case. I had varying degrees of success, and it's not a job for a Devs side quest, that means I think you need someone dedicated to explore, build and debug these pipelines, but for a CI tool it's very good.
Because you can extend and override jobs, you can create seams so that each piece of the pipeline is isolated and testable. This way there is very little that can go wrong in production that's the CI fault. And I think that's only possible because of the way that Gitlab models their jobs and stages.
Of all the valid complaints about Github Actions or CI in general, this seems to be an odd one. No details about what was tried or not tried, but hard to see a `-run: go install cuelang.org/go/cmd/cue@latest` step not working?
> For the love of all that is holy, don’t let GitHub Actions
> manage your logic. Keep your scripts under your own damn
> control and just make the Actions call them!
I mean your problem was not `build.rs` here and Makefiles did not solve it, was your logic not already in `build.rs` which was called by Cargo via GitHub Actions?
The problem was the environment setup? You couldn't get CUE on Linux ARM and I am assuming when you moved to Makefiles you removed the need for CUE or something? So really the solution was something like Nix or Mise to install the tooling, so you have the same tooling/version locally & on CI?
"GitHub actions bad" is a valid take - you should reduce your use to a minimum.
"My build failed because of GitHub actions couldn't install a dependency of my build" is a skill issue. Don't use GitHub actions to install a program your build depends on.
So many engineers could put the hours spent debugging GH actions to use developing expertise to run their own CI. But people either don’t believe they can, can’t convince decision makers to let them try, or just want to fix their own problem and move on.
I was convinced GH actions was best practice and it was normal to waste hours on try-and-pray build debugging, until one day GH actions went down and I ran deploys from my laptop and remembered how much better life can be without it..
(Solo dev here - but opensource CI on an EC2 instance can be just as nice)
> So many engineers could put the hours spent debugging GH actions to use developing expertise to run their own CI.
If I run my own CI, then the compliance team has to get involved to run various endpoint security and update management tools on whatever system I'm running the CI on.
Then the costs of staying with GH actions need to be made known so they can be balanced against the cost of doing things differently. Of course there’s a cost involved in just getting those numbers too.
lol, this post makes me feel like crying at my job in BigTech co. He’s complaining about a 30 min feedback loop maybe, mine is 2+ hours.
I write a piece of code, however small the change. Then run the proc, first it takes ~20 mins to compile. Then since it is ML, it can only run on a remote server. It takes an easy 20 more minutes to start running in the remote server. Then after another 40 minutes I can confirm the job failed. The only way to debug is to read through a massive log file, for which we have an in built log reader lol. The log file will have thousands of errors that literally don’t matter, and you’ll never have enough context to know which errors don’t matter. You can simply ping someone and ask, does this error matter, could this be the reason my entire proc failed, oh no this is just a useless log that fails all the time, I shouldn’t have been wasting a day digging into this, ok thanks bye.
But this isn’t it, not even close lol, we in fact have a custom DSL to define computational graphs, which of course does not have any linter, or even any compiler and a very broken visualizer but our entire org runs on this. Syntax errors, logic errors, actual race conditions are all caught the exact same way —- as the process dying after trying to run on a remote server for 2+ hours with no useful error log. So our workflow is to just get a cup of coffee and stare at the graph which can get thousands and thousands of lines big to find at times completely trivial bugs that any half decent language would have caught as a syntax error.
My exp in BigTechCo makes me completely understand GitHub actions, my guess is many big companies have equally janky tools that harm dev productivity but still somehow take absolutely massive workloads. GitHub just thought of sharing it to the public.
If you wanted a better version of GitHub Actions/CI (the orchestrator, the job definition interface, or the programming of scripts to execute inside those jobs), it would presumably need to be more opinionated and have more constraints?
Who here has been thinking about this problem? Have you come up with any interesting ideas? What's the state of the art in this space?
GHA was designed in ~2018. What would it look like if you designed it today, with all we know now?
I've been working for the last 5 years on an alternative called Dagger. Ended up building a company around it.
We started from the ideal state of CI, and set out to build the platform that would support that.
For us this ideal state of CI boils down to 4 things:
- local-first (local execution should be a first-class citizen, with no exception)
- repeatable (the same inputs should yield the same output, with affordances for handling inevitable side effects in an explicit and pragmatic way)
- programmable. my workflows are software. I want all the convenience of a modern software development experience: types, IDE support, a rich system API
, debugging tools, an ecosystem of reusable components, etc.
- observable. I want all the information in one place about everything that happened in my workflow, with good tooling to get the information I need quickly, and interop with the existing observability ecosystem (eg. open telemetry)
So Dagger is our best effort at a CI platform focused on those 4 things.
Sorry if this comes across as a sales pitch. When
you're building solves a problem you're obsessed with, it's hard to discuss the problem without also mentioning the solution that seems the most obvious to you :)
I've been watching Dagger with great interest, although have not moved production workloads to it (nor, admittedly, even committed an afternoon to setting up any workflows/graphs).
Passive comment readers should be aware that ^shykes here cofounded Docker (my gratitude), so it's really worth a look.
Can anyone comment on the ergonomics of Dagger after using it for a while?
I was just looking at the docs earlier this week to consider a migration but got confused by the AI sections...
You're not the only one... At some point last year, we discovered that CI/CD workflows and so-called "AI agent workflows" have a lot in common, and Dagger can in theory be used as an execution engine for both. We attempted to explain this - "great for CI/CD and for Agents!". But the feedback was mostly negative - it came across as confusing and lacking focus. So, we are rolling back this messaging and refocusing on CI/CD again. If you haven't checked our docs in the latest 12 hours, it's worth checking again: you'll see clear signs of this refocusing (although we are not done).
In doubt, I recommend joining our public discord server (https://discord.com/invite/dagger-io) it is basically a support group for CI/CD nerds who believe that a better way is possible, and want to discuss it with like-minded people.
What always surprises me in these discussions is how few people have used GitLab CI. What gives? It's far, far superior to GitHub and even existed first.
Not sure what you mean by "smallest". With GitLab you create docker images. That means you can easily run them locally and share them. In the example you gave the GitHub actions one looks like more lines of YAML but only works because they made an "action". If they had provided a Docker image then GitLab would be just as easy (and trivial to test locally).
mise already is trivial to test locally though, and I'm not sure I agree that maintaining a CI config + Docker image is just as easy as maintaining a CI config.
At work we have a bunch of GitHub actions for integration testing, building models, publishing, reporting, and whatnot. It was horrible to maintain and look into whenever something went wrong, so I rewrote all the individual parts in Perl and hooked them together with pipes inside GHA, and it works wonders.
Also, GitHub actions itself just breaks sometimes, which is super annoying. A couple of weeks ago, half of all the macOS runner images broke when using GitHub's caching system. Seems like that would have been caught on GitHub's side before that happened, but what do I know!
I've gotten to a point where my workflow YAML files are mostly `mise` tool calls (because it handles versioning of all tooling and has cache support) and webhooks, and still it is a pain. Also their concurrency and matrix strategies are just not working well, and sometimes you end up having to use a REST API endpoint to force cancel a job because their normal cancel functionality simply does not take.
There was a time I wanted our GH actions to be more capable, but now I just want them to do as little as possible. I've got a Cloudflare worker receiving the GitHub webhooks firehose, storing metadata about each push and each run so I don't have to pass variables between workflows (which somehow is a horrible experience), and any long-running task that should run in parallel (like evaluations) happens on a Hetzner machine instead.
I'm very open to hear of nice alternatives that integrate well with GitHub, but are more fun to configure.
Honestly part of the reason why I left my last job was because I had to extensively work with GitHub workflows and actions. Debugging was absolutely hell, especially with long running tasks (~1hr) that would fail with next to no debug or traceability. I offered many times to overhaul the system and make it easier to maintain, but we had no time budget. I now work for 30% less but am substantially happier in life without github. It sounds crazy, but I’m not exaggerating.
Run your own server for GitHub Actions. There is a simple library they have you install so your runner gets registered with your repo. Then you can SSH in whenever a job fails. This lets you fully inspect the state, and to execute one-off commands to test theories. It’s a much faster way to iterate.
I've always found things like AWS codebuild or even just a self hosted bare bones jenkins server far easier to work with. What is the advantage that github actions provide that people put up with it? The feedback seems almost universally negative.
It is overwhelmingly less resource-intensive to maintain over time than any Jenkins server. And the config is in the repo (not the Jenkins server's config).
Using jdx/mise in GitHub Actions has been very nice, I recommend it. It's not just a nice task runner, it also manages the dependencies of your tasks (stuff like Python, Node.js, even random tools from GitHub) so that "mise run build" does the exact same thing both locally and in CI. Then the ideal GHA workflow is just "install mise, run this mise task". And not only for building, we have it set up for commit checks too.
> For the love of all that is holy, don’t let GitHub Actions
> manage your logic. Keep your scripts under your own damn
> control and just make the Actions call them!
The pain is real. I think everyone that's ever used GitHub actions has come to this conclusion. An ideal action has 2 steps: (1) check out the code, (2) invoke a sane script that you can test locally.
Honestly, I wonder if a better workflow definition would just have a single input: a single command to run. Remove the temptation to actually put logic in the actions workflow.
I assume you're using the currently recommended docker-in-docker method. The legacy Gitlab way is horrible and it makes it basically impossible to run pipelines locally.
GitHub introduced all their fancy GHA apps or callables or whatever they're called for specific tasks, and the community went wild. Then people built test and build workflows entirely in GHA instead of independent of it. And added tons of complexity to the point they have a whole build and test application written in GHA YAML.
This is basically how most other CI systems work. GitLab CI, Jenkins, Buildbot, Cirrus CI, etc. are all other systems I've used and they work this way.
I find GitHub Actions abhorrent in a way that I never found a CI/CD system before...
It seems more of a cultural issue that -- I'm pretty sure -- predates Microsoft's acquisition of GitHub. I assume crappy proprietary yaml can be blamed on use of Ruby. And there seems to be an odd and pervasive "80% is good enough" feel to pretty much everything in GitHub, which is definitely cultural, and I'm pretty sure, also predates Microsoft's acquisition.
GHA is based on Azure Actions. This is evident in how bad its security stance is, since Azure Actions was designed to be used in a more closed/controlled environment.
> I think everyone that's ever used GitHub actions has come to this conclusion
This is not even specific to GitHub Actions. The logic goes into the scripts, and the CI handles CI specific stuff (checkout, setup tooling, artifacts, cache...). No matter which CI you use, you're in for a bad time if you don't do this.
That is easy to say on the surface for a simple build script.
How do you orchrestate a full CI/CD pipeline where you need state? You just move that complex logic to another monolithic script with its own problems?
> I think everyone that's ever used GitHub actions has come to this conclusion.
I agree that that should be reasonable but unfortunately I can tell you that not all developers (including seniors) naturally arrive at such conclusion no.
I am not having fun with GitHub Actions right now! Why does everything have to be so hard?
I like being able to run self-hosted runners, that is a very cool part of GitHub Actions/Workflow.
I appreciate all the other advice about limit my yamls to: 1) checkout, 2) call a script to do the entire task. I am already half-way there, just need to knuckle-down and do the work.
I was dismayed that parallel tasks aren't really a thing in the yaml, I wanted to fanout a bunch of parallel tasks and I found I couldn't do it. Now that I'm going to consolidate my build process into a single script I own, I can do the fanout myself.
Jobs run in parallel, so if you used Make you could have one job called 'formatting' calling 'make check-formatting', one called linting calling ''make check-linting', one called 'compiling' calling 'make compile' etc.
As much as I hate GitHub Actions, I prefer it over Jenkins and others, because it is right there. I don't need to go and maintain my own servers, or even talk to a 3rd party to host the services.
Like many things, the tool people reach for is the one that's in the box... As opposed to a trip to the hardware store, getting distracted for a few hours, coming back home and no longer in the mood to work on your project/choore.
I think the root problem here is scope creep on using build tools, configuration management, and ci tools: don't make them do things they are not designed for / weak at. The issue isn't necessarily github ci, which is fine if you keep it simple. But things just getting to the stage where they are not simple anymore.
I've seen this over and over again over the years in projects where things like ant, maven, gradle, puppet, ansible, etc. Invariably somebody tries to do something really complicated/clever in a convoluted way. They'll add plugins, more plugins, all sorts of complex configuration, etc. Your script complexity explodes. And then it doesn't work and you spend hours/days trying to make it do the right thing and fighting a tool that just wasn't designed to do what you are doing well. The problem is using the wrong tool for the job. All these tools have the tendency to evolve into everything tools. And they just aren't good at everything. Just because it has some feature doesn't mean it's a good idea to use it.
The author actually calls this out. Just write a bash script and run that instead. If that gets too complicated pick something else more appropriate to the job. Python, whatever you like. The point here is to pick something that's easy for you to run, test, and debug locally. Obviously people have different preferences here. If you are shoe horning complex fork/join behavior, conditional logic, etc. into a Yaml / CI build, maybe simplify your build. Yaml just isn't suitable as a general purpose programming language.
Externalizing the complex stuff to some script also has the benefit of that stuff still working if you ever decide to switch CI provider. Maybe you want to move to Gitlab. Or somebody decides Jenkins wasn't so bad after all. The same scripts will probably be easy to adapt to those.
One of my current github actions basically starts a vm, runs a build.sh script, stops the vm. I don't need a custom runner for that. I get to pick the vm type. I can do whatever I need to in my build.sh. I have one project with an actual matrix build. But that's about the most complex thing I do with github actions. A lot of my build steps are just inline shell commands.
And obligatory AI comment here, if you are doing all this manually, having scripts that run locally also means you can put claude code/codex/whatever to work fixing them for you. I've been working on some ansible scripts with codex today. Works great. It produces better ansible scripts than me. These tools work better if they can test/run what they are working on.
GHA’s componentized architecture is appealing, but it’s astonishing to me that there’s still seemingly no way to actually debug workflows, run them locally, or rapidly iterate on them in any way. Alas.
fyi i maintain a repo that accidentally tracks github actions cron reliability (https://www.swyx.io/github-scraping) - just runs a small script every hour.
In general, I've never really experienced the issues mentioned, but I also use Gitea with Actions rather than GitHub. I also avoid using any complex logic within an Action.
For the script getting run, there's one other thing. I build my containers locally, test the scripts thoroughly, and those scripts and container are what are then used in the build and deploy via Action. As the entire environment is the same, I haven't encountered many issues at all.
As soon as I need more than two tries to get some workflow working, I set up a tmate session and debug things using a proper remote shell. It doesn't solve all the pain points, but it makes things a lot better.
Until I resd this blog I was under the impression that everyone wrote Python/ other files and used Github Actions to just call the scripts!
This way we can test it on local machine before deployment.
Also as other commenters have said - bash is not a good option - Use Python or some other language and write reusabe scripts. If not for this then for the off chance that it'll be migrated to some other cicd platform
Before that, most people would avoid Jenkins and probably never try Buildbot (because devs typically don't want to spend any time learning tools). Devs would require "devops" to do the CI stuff. Again, mostly because they couldn't be arsed to make it themselves, but also because it required setting up a machine (do you self-host, do you use a VPS?).
Then came tools like Travis or CircleCI, which made it more accessible. "Just write some kind of script and we run it on our machines". Many devs started using that.
And then came GitHub Actions, which were a lot better than Travis and CircleCI: faster, more machines, and free (for open source projects at least). I was happy to move everything there.
But as soon as something becomes more accessible, you get people who had never done it before. They can't say "it enables me to do it, so it's better than me relying on a devops team before" or "well it's better than my experience with Travis". They will just complain because it's not perfect.
And for the OP's defense, I do agree that not being able to SSH into a machine after the build fails is very frustrating.
I think it's possible to both think GitHub Actions is an incredible piece of technology (and an incredible de facto public resource), while also thinking it has significant architectural and experiential flaws. The latter can be fixed; the former is difficult for competitors to replicate.
(In general, I think a lot of criticisms of GitHub Actions don't consider the fully loaded cost of an alternative -- there are lots of great alternative CI/CD services out there, but very few of them will give you the OS/architecture matrix and resource caps that GitHub Actions gives every single OSS project for free.)
For the last decade I've been doing my CI/CD as simple .NET console apps that run wherever. I don't see why we switch to these wildly different technologies when the tools we are already using can do the job.
Being able to run your entire "pipeline" locally with breakpoints is much more productive than whatever the hell goes on in GH Actions these days.
I can do that with github actions too? For tests, I can either run them locally (with a debugger if I want), or in github actions. Smaller checks go in a pre-commit config that github action also runs.
Setting up my github actions (or gitlab) checks in a way that can easily run locally can be a bit of extra work, but it's not difficult.
Standard msft absurdity. 8 years later there is still no local gh action runner to test your script before you commit, push, and churn through logs, and without some 3rd party hack, no way to ssh in and debug. It doesn't matter how simple the build command you write is, because the workflow itself is totally foreign technology to most, and no one wants to be a gh action dev.
Like most of the glaring nonsense that costs people time when using msft, this is financially beneficial to msft in that each failed run counts against paid minutes. It's a racket from disgusting sleaze scum who literally hold meetings dedicated to increasing user pain because otherwise the bottom line will slip fractionally and no one in redmond has a single clue how to make money without ripping off the userbase.
I am surprised that these sort of declarative specs are so popular in certain domains. Essentially you always seem to be putting settings into the ether and hoping they interact with each other in the way you expect.
I prefer an api with documented contracts between its abstractions
Issue is, op is trying to use matrix strategy when with cross-compiling they could avoid it. I have done it for https://github.com/anttiharju/compare-changes (which has nontrivial CI pipelines but they could be a lot simpler for op's needs)
Main issue is Rust. Writing catchy headlines about hating something may feel good, but a lot of people could avoid these pains if
You should try RWX! You can trigger runs from the CLI for way faster feedback loops. The automatic caching is surprisingly good too. https://www.rwx.com/
The main downside of bitbucket pipelines is bitbucket. And the only significant feature I recall over GitHub Actions is that Pipelines support cron jobs.
I think this is a specific example of a generalized mistake, one that various bits of our infrastructure and architecture all but beg us to make, over and over, and which must be resisted, which is: Your development feedback loop must be as tight as possible.
Granted, if you are working on "Windows 12", you won't be building, installing, testing, and deploying that locally. I understand and acknowledge that "as tight as possible" will still sometimes push you into remote services or heavyweight processes that can't be pushed towards you locally. This is an ideal to strive for, but not one that can always be accomplished.
However, I see people surrender the ability to work locally much sooner than they should, and implement massively heavyweight processes without any thought for whether you could have gotten 90% of the result of that process with a bit more thought and kept it local and fast.
And even once you pass the event horizon where the system as a whole can't be feasibly built/tested/whatever on anything but a CI system, I see them surrendering the ability to at least run the part of the thing you're working on locally.
I know it's a bit more work, building sufficient mocks and stubs for expensive remote services that you can feasibly run things locally, but the payoff for putting a bit of work into having it run locally for testing and development purposes is just huge, really huge, the sort of huge you should not be ignoring.
"Locally" here does not mean "on your local machine" per se, though that is a pretty good case, but more like, in an environment that you have sole access to, where you're not constantly fighting with latency, and where you have full control. Where if you're debugging even a complex orchestration between internal microservices, you have enough power to crank them all up to "don't ever timeout" and attach debuggers to all of them simultaneously, if you want to. Where you can afford to log every message in the system, interrupt any process, run any test, and change any component in the system in any manner necessary for debugging or development without having to coordinate with anyone. The more only the CI system can do by basically mailing it a PR, and the harder it is to convince it to do just the thing you need right now rather than the other 45 minutes of testing it's going to run before running the 10 second test you actually need, the worse your development speed is going to be.
Fortunately, and I don't even how exactly the ratio between sarcasm and seriousness here (but I'm definitely non-zero serious), this is probably going to fix itself in the next decade or so... because while paying humans to sit there and wait for CI and get sidetracked and distracted is just Humans Doing Work and after all what else are we paying them for, all of this stuff is going to be murder on AI-centric workflows, which need tight testing cycles to work at their best. Can't afford to have AI waiting for 30 minutes to find out that its PR is syntactically invalid, and can't afford for the invalid syntax to come back with bad error messages that leave it baffled as to what the actual problem is. If we won't do it for the humans, we'll do it for the AIs. This is definitely not something AI fixes, despite the fact they are way more patient than us and much less prone to distraction in the meantime since from their "lived experience" they don't experience the time taken for things to build and test, it is made much worse and more obvious that this is a real problem and not just humans being whiny and refusing to tough it through.
So the article is about the frustrating experience of fixing GitHub Actions when something goes wrong, especially when a workflow only fails on one platform, potentially due to how GitHub runner is set up (inconsistently across platforms).
Took me a while to figure that out. While I appreciate occasional banters in blog articles, this one seems to diverge into rant a bit too much, and could have made its point much clearer, with, for example, meaningful section headers.
I wouldn't say that, but I would say there's no "should" here; it's often much more hassle than people expect and everyone has to decide for themselves whether the number of users is worth it.
As a PM trying to understand what was happening, it took me a minute. Turning this into an analogy so that it might benefit other non-too-techies:
Original plan: You have an oven in another building that automatically bakes your cake. But the oven needs a mixer, and every time you bake, it has to wait 2-3 minutes for someone to bring the mixer from storage.
Problem: For one specific oven (Linux ARM), the mixer never arrives. So your cake fails. You keep trying different ways to get the mixer delivered. Each attempt: 2-3 minutes wait.
What you finally do: Stop waiting for the mixer to be delivered. Just mix the batter at home where you already have a mixer. Send the pre-mixed batter to the other building. Now the oven just bakes it - no waiting for the mixer.
Translation: Stop trying to generate files in GitHub Actions (where it takes 2-3 minutes each time). Generate them locally on your computer where you already have the tools. Upload the finished files. GitHub Actions just uses them.
Sometimes "pre-mix the batter at home" beats "wait for the mixer every single time."
Prefacing this with the fact that act is great, however, it has many shortcomings. Too often I've run into roadblocks, and when looking up the issue for it, it seems they are hard to address. Simpler workflows work fine with it, but more complex workflows will be much harder.
Don't put your logic in proprietary tooling. I have started writing all logic into mise tasks since I already manage the tool dependencies with mise. I tend to write them in a way where it can easily take advantage of GHA features such as concurrency, matrixes, etc. But beyond that, it is all running within mise tasks.
act is often mentioned as a drop-in replacement but I never got it to replicate GitHub actions environment. I didn't try it for this particular case, though.
“ For the love of all that is holy, don’t let GitHub Actions manage your logic. Keep your scripts under your own damn control and just make the Actions call them!”
> Sure, I still make fun of the PHP I remember from the days of PHP 4.1, but even then I didn’t hate it. (ref. And “PHP: Training Wheels Without a Bike” is still in the Top 10 of my favorite memes.)
I still use it to build my latest projects, and for me it’s like breathing. Simple and functional. It’s not perfect, but no language is.
Yeah, no reason for PHP to catch strays here, especially if his knowledge is still based on 4.1. It's really a whole new language compared to back then.
To some extent I do agree that is sounds like "my build was failing on a platform I had never tested, and I was pissed because I had to debug it".
But it is true that GitHub Actions don't make it easy to debug (e.g. you can't just SSH into the machine after the build fails). Not sure if it justifies hating with Passion, though.
is it your contention that once anybody becomes sufficiently skillful with a technology they will come to love it? And thus stating that one does not love the specific technology demonstrates the lack of skill?
I think this post accurately isolates the single main issue with GitHub Actions, i.e. the lack of a tight feedback loop. Pushing and waiting for completion on what's often a very simple failure mode is frustrating.
Others have pointed out that there are architectural steps you can take to minimize this pain, like keeping all CI operations isolated within scripts that can be run locally (and treating GitHub Actions features purely as progressive enhancements, e.g. only using `GITHUB_STEP_SUMMARY` if actually present).
Another thing that works pretty well to address the feedback loop pain is `workflow_dispatch` + `gh workflow run`: you still need to go through a push cycle, but `gh workflow run` lets you stay in development flow until you actually need to go look at the logs.
(One frustrating limitation with that is that `gh workflow run` doesn't actually spit out the URL of the workflow run it triggers. GitHub claims this is because it's an async dispatch, but I don't see how there can possibly be no context for GitHub to provide here, given that they clearly obtain it later in the web UI.)
I've standardized on getting github actions to create/pull a docker image and run build/test inside that. So if something goes wrong I have a decent live debug environment that's very similar to what github actions is running. For what it's worth.
I do the same with Nix as it works for macOS builds as well
It has the massive benefit of solving the lock-in problem. Your workflow is generally very short so it is easy to move to an alternative CI if (for example) Github were to jack up their prices for self hosted runners...
That said, when using it in this way I personally love Github actions
Nix is so nice that you can put almost your entire workflow into a check or package. Like your code-coverage report step(s) become a package that you build (I'm not brave enough to do this)
I run my own jenkins for personal stuff on top of nixos, all jobs run inside devenv shell, devenv handles whatever background services required (i.e. database), /nix/store is shared between workers + attic cache in local network.
Oh, and there is also nixosModule that is tested in the VM that also smoke tests the service.
First build might take some time, but all future jobs run fast. The same can be done on GHA, but on github-hosted runners you can't get shared /nix/store.
I'm scared by all these references to nix in the replies here. Sounds like I'm going to have learn nix. Sounds hard.
Gemini/ChatGPT help (a lot) when getting going. They make up for the poor documentation
LLMs are awful at nix in my experience. Just learn the fundamentals of the language and build something with it.
Calling nix documentation poor is an insult to actually poor documentation.
Remember the old meme image about Vim and Emacs learning curves? Nix is both of those combined.
It's like custom made for me on the idea level, declarative everything? Sign me up!
But holy crap I have wasted so much time and messed up a few laptops completely trying to make sense of it :D
Whata the killer benefit of nix over, like, a docker file or a package.lock or whatever?
package.lock is JSON only, Nix is for the entire system, similar to a Dockerfile
Nix specifies dependencies declaritively, and more precisely, than Docker (does by default), so the resulting environment is reproducibly the same. It caches really well and doubles as a package manager.
Despite the initial learning curve, I now personally prefer Nix's declarative style to a Dockerfile
same here. though, i think bazel is better for DAGs. i wish i could use it for my personal project (in conjunction with, and bootstrapped with nix), but that's a pretty serious tooling investment that I just feel is just going to be a rabbit hole.
I tend to have most of my workflows setup as scripts that can run locally in a _scripts diorectory, I've also started to lean on Deno if I need anything more complex than I'm comfortable with in bash (even bash in windows) or powershell, since it executes .ts directly and can refer directly to modules/repos without a separate install step.
This may also leverage docker (compose) to build/run different services depending on the stage of action. Sometimes also creating "builder" containers that will have a mount point for src and output to build and output the project in different OSes, etc. Docker + QEMU allows for some nice cross-compile options.
The less I rely on Github Actions environment the happier I am... the main points of use are checkout, deno runtime, release please and uploading assets in a release.
It sucks that the process is less connected and slow, but ensuring as much as reasonable can run locally goes a very long way.
I just use the fact that any action run can trigger a webhook.
The action does nothing other than trigger the hook.
Then my server catches the hook and can do whatever I want.
I wish I had the courage to run my own CI server. But yes, I think your approach is the best for serious teams that can manage more infrastructure.
I am embarrassed that I didn't think to do this. Thank you :)
I was doing something similar when moving from Earthly. But I have since moved to Nix to manage the environment. It is a lot better of a developer experience and faster! I would checkout an environment manager like Nix/Mise etc so you can have the same tools etc locally and on CI.
Yeah, images seem to work very well as an abstraction layer for most CI/CD users. It's kind of unfortunate that they don't (can't) fully generalize across Windows and macOS runners as well, though, since in practice that's where a lot of people start to get snagged by needing to do things in GitHub Actions versus using GitHub Actions as an execution layer.
I’ve VNCed into CI to debug selenium tests failing because of platform font and scrollbar rendering. I never really thought about doing that locally in a docker container, but it definitely wouldn’t be convenient to always run those tests locally in a docker container. I guess having an option to would sort of simplify debugging, but I’d still have to VNC into the docker container I think
Me, too, though getting the trusted publisher NPM settings working didn't help with this. But it does help with most other CI issues.
Most of the npm modules I've built are fortunately pretty much just feature complete... I haven't had to deal with that in a while...
I do have plans to create a couple libraries in the near future so will have to work through the pain(s)... also wanting to publish to jsr for it.
So you've implemented GitLab CI in GitHub... We used to do this in Jenkins like 7 years ago.
https://github.com/nektos/act
Lets you run your actions locally. I've had significant success with it for fast local feedback.
I tried this five years ago back when I was an engineer on the PyTorch project, and it didn't work well enough to be worth it. Has it improved since then?
It works well enough that I didn’t realize this wasn’t first party till right now.
It works, but there are fair amount of caveats, especially for someone working on things like Pytorch, the runtime is close but not the same, and its support of certain architectures etc can create annoying bugs.
For me, no. Spend days trying to get it to recreate a production environment workflow. It is too different than production.
it has. it's improved to work with ~ 75% of steps . fast enough to worth trying before push
I tried this recently and it seems like you have to make a lot of decisions to support Act. It in no way "just works", but instead requires writing actions knowing that they'll run on Act.
I tried act, on the surface it seems like a godsend. Not until you try to use it do you realize it's almost impossible to recreate any moderately complex workflow.
You HAVE to run it against a container, so if you're using self hosted runners your environment may not match at all.
It's insane to me that being able to run CI steps locally is not the first priority of every CI system. It ought to be a basic requirement.
I've often thought about this. There are times I would rather have CI run locally, and use my PGP signature to add a git note to the commit. Something like:
``` echo "CI passed" | gpg2 --clearsign --output=- | git notes add -F- ```
Then CI could check git notes and check the dev signature, and skip the workflow/pipeline if correctly signed. With more local CI, the incentive may shift to buying devs fancier machines instead of spending that money on cloud CI. I bet most devs have extra cores to spare and would not mind having a beefier dev machine.
I think this is a sound approach, but I do see one legitimate reason to keep using a third-party CI service: reducing the chance of a software supply chain attack by building in a hardened environment that has (presumably) had attention from security people. I'd say the importance of this is increasing.
"Works on my machine!"
This goes against every incentive for the CI service provider
Not necessarily. For example, Buildkite lets you host your own runners.
https://github.com/nektos/act
> i.e. the lack of a tight feedback loop.
Lefthook helps a lot https://anttiharju.dev/a/1#pre-commit-hooks-are-useful
Thing is that people are not willing to invest in it due to bad experiences with various git hooks, but there are ways to have it be excellent
Yeah, I'm one of those people who seems to consistently have middling-to-bad experiences with Git hooks (and Git hook managers). I think the bigger issue is that even with consistent developer tooling between both developer machines and CI, you still have the issue where CI needs to do a lot more stuff that local machines just can't/won't do (like matrix builds).
Those things are fundamentally remote and kind of annoying to debug, but GitHub could invest a lot more in reducing the frustration involved in getting a fast remote cycle set up.
GitHub could invest a lot more in actions for sure. Even just in basic stuff like actions/checkout@v6 being broken for self-hosted runners.
But very often the CI operations _are_ the problem. It's just YAML files with unlimited configuration options that have very limited documentation, without any type of LSP.
We need SSH access to the failed instances so we can poke around and iterate from any step in the workflow.
Production runs should be immutable, but we should be able to get in to diagnose, edit, and retry. It'd lead to faster diagnosis, resolution, and fixing.
The logs and everything should be there for us.
And speaking of the logs situation, the GHA logs are really buggy sometimes. They don't load about half of the time I need them to.
I wrote something recently with webrtc to get terminal on failure: https://blog.gripdev.xyz/2026/01/10/actions-terminal-on-fail...
Are there solutions to this like https://github.com/marketplace/actions/ssh-to-github-action-... ?
This is one of the big problems we solved with the RWX CI platform (RWX.com). You can use ‘rwx run’ and it automatically syncs your local changes, so no need to push — and with our automated caching, steps like setting up the environment cache hit so you don’t have to execute the same stuff over and over again while writing your workflow. Plus with our API or MCP server you can get the results directly in your terminal so no need to open the UI at all unless you want to do some in-depth spelunking.
I’ve contemplated building my own CI tool (with a local runner) and the thing is if you assume “write a pipeline that runs locally but also on push”, then the feature depth is mostly about queuing, analyzing output, and often left off, but IMO important, charting telemetry about the build history.
Most of these are off the shelf, at least in some programming languages. It’s the integrations and the overmanagement where a lot of the weight is.
I think you described Jenkins, which is infinitely better than GitHub runners.
Jenkins has it's own set of issues. The theory behind GHA is you *should* be able to keep everything in git and not need a another service with it's own abstractions.
But Actions just screws this up. I consider them both to be equally bad.
I'm almost considering n8n to replace both but that will just lead to more problems.
All my Jenkins builds are between 1 and 3 lines depending on readability:
Clone
Cd $clonedFolder
./build.sh
Add an archive step, all set.
I will say even building and running unit testing in Jenkins isn’t bad, but if you aren’t careful it can get messy.
Jenkins can be a super simple build tool. Strong-arming it into doing all the things starts to get messy quickly. Build, run unit tests, archive. I’m not even big on like, building on push. Nothing wrong with it, I’m just big on keeping things as simple as possible. GHA is just a ball of complexity.
I've never used gh workflow run, but I have used the GitHub API to run workflows and wanted to show the URL. I had to have it make another call to get the workflow runs and assume the last run is the one with the correct URL. This would obviously not work correctly if there were multiple run requests at the same time. Maybe some more checking could detect that, but it works for my purposes so far.
Does the metadata in the further call not identify the branch/start time/some other useful info that could help disambiguate this? (honest question)
I wonder what prevents a GH action from connecting to your VPN (Wireguard is fine) and post tons of diagnostics right onto your screen, and then, when something goes badly wrong, or when a certain point is reached, to just keep polling an HTTP endpoint for shell commands to execute.
I mean. I understand, it would time out eventually. But it may be enough time to interactively check a few things right inside the running task's process.
Of course this should only happen if the PR contains a file that says where to connect and when to stop for interactive input. You would only push such a file when an action is misbehaving, and you want to debug it.
I understand that it's a band-aid, but a band-aid is better than the nothing which is available right now.
I’ve never used Nix and frankly am a sceptic, but can it solve this problem by efficiently caching steps?
1. Don't use bash, use a scripting language that is more CI friendly. I strongly prefer pwsh.
2. Don't have logic in your workflows. Workflows should be dumb and simple (KISS) and they should call your scripts.
3. Having standalone scripts will allow you to develop/modify and test locally without having to get caught in a loop of hell.
4. Design your entire CI pipeline for easier debugging, put that print state in, echo out the version of whatever. You don't need it _now_, but your future self will thank you when you do it need it.
5. Consider using third party runners that have better debugging capabilities
I would disagree with 1. if you need anything more than shell that starts to become a smell to me. The build/testing process etc should be simple enough to not need anything more.
That's literally point #2, but I had the same reaction as you when I first read point #1 :)
I agree with #2, I meant more if you are calling out to something that is not a task runner(Make, Taskfile, Just etc) or a shell script thats a bit of a smell to me. E.g. I have seen people call out to Python scripts etc and it concerns me.
My software runs on Windows, Linux and MacOS. The same Python testing code runs on all three platforms. I mostly dislike Python but I can't think of anything better for this use case.
You might consider Deno with Typescript... it's a single exe runtime, with a self-update mechanism (deno upgrade) and can run typescript/javascript files that directly reference the repository/http/modules that it needs and doesn't require a separate install step for dependency management.
I've been using it for most of my local and environment scripting since relatively early on.
I don't touch Windows so I would not know.
> The same Python testing code runs on all three platforms.
I have no objections to Python being used for testing, I use it myself for the end to end tests in my projects. I just don't think Python as a build script/task runner is a good idea, see below where I got Claude to convert one of my open source projects for an example.
It's interesting because #1 is still suggesting a shell script, it's just suggesting a better shell to script.
I had no idea 'pwsh' was PowerShell. Personally not interested, maybe if your a Microsoft shop or something then yeah.
"pwsh" is often used as the short-hand for modern cross-platform PowerShell to better differentiate it from the old Windows-only PowerShell.
I think pwsh is worth exploring. It is cross-platform. It is post-Python and the Python mantra that "~~code~~ scripts are read more often than they are written". It provides a lot of nice tools out of the box. It's built in an "object-oriented" way, resembling Python and owing much to C#. When done well the "object-oriented" way provides a number of benefits over "dumb text pipes" that shells like bash were built on. It is easy to extend with C# and a few other languages, should you need to extend it.
I would consider not dismissing it off hand without trying it just because Microsoft built it and/or that it was for a while Windows-only.
It's also both a larger download and slower to start than Java, which is not known for being light and nimble. In fact, PowerShell is so slow that you can both compile and run the equivalent C# program before PowerShell finishes launching. Not ideal for a shell or a scripting language.
Also, the newer versions aren't included with Windows, which would have been useful – instead Windows includes an incompatible older version that admonishes you to download the new version. But why would you download several hundred megabytes of pwsh when you can equally well download any other language runtime?
Also, it sends "telemetry" to Microsoft by default.
Also, the error handling is just awful, silencing errors by default, requiring several different incantations to fix.
Also, the documentation is vague and useless. And the syntax is ugly.
It gets faster to boot on subsequent launches and some distros are now packaging pre-baked versions.
The new versions aren't included in Windows and the old versions are still in Windows for the exact same reasons of Windows backwards compatibility requirements. But at this point the bootstrap on Windows is as easy as `winget install --id Microsoft.PowerShell`.
The error handling isn't far from the bash defaults, but the magic incantations actually tell you what they do versus the number of bash scripts littered with `set -euxo pipefail` is the exact same as the number of scripts that need an `$ErrorActionPreference = "Stop"` and/or a `$PSNativeCommandUseErrorActionPreference = true`.
I find the documentation less vague and more useful than the average `man` page and the syntax is fine (and better than bash) to me, but I understand how much of that is personal preference and familiarity.
It's actually a pretty good shell! FOSS and cross-platform, too.
Huh? Who cares if the script is .sh, .bash, Makefile, Justfile, .py, .js or even .php? If it works it works, as long as you can run it locally, it'll be good enough, and sometimes it's an even better idea to keep it in the same language the rest of the project is. It all depends and what language a script is made in shouldn't be considered a "smell".
Once you get beyond shell, make, docker (and similar), dependencies become relevant. At my current employer, we're mostly in TypeScript, which means you've got NPM dependencies, the NodeJS version, and operating system differences that you're fighting with. Now anyone running your build and tests (including your CI environment) needs to be able to set all those things up and keep them in working shape. For us, that includes different projects requiring different NodeJS versions.
Meanwhile, if you can stick to the very basics, you can do anything more involved inside a container, where you can be confident that you, your CI environment, and even your less tech-savvy coworkers can all be using the exact same dependencies and execution environment. It eliminates entire classes of build and testing errors.
I use to have my Makefile call out and do `docker build ...` and `docker run ...` etc with a volume mount of the source code to manage and maintain tooling versions etc.
It works okay, better than a lot of other workflows I have seen. But it is a bit slow, a bit cumbersome(for langs like Go or Node.js that want to write to HOME) and I had some issues on my ARM Macbook about no ARM images etc.
I would recommend taking a look at Nix, it is what I switched to.
* It is faster. * Has access to more tools. * Works on ARM, X86 etc.
You mean nix inside a container? Or what exactly?
I replaced building/testing etc inside Docker containers to just using Nix.
* https://github.com/DeveloperC286/clean_git_history/commit/f8...
I've switched to using Deno for most of my orchestration scripts, especially shell scripts. It's a single portable, self-upgradeable executable and your shell scripts can directly reference the repositories/http(s) modules/versions it needs to run without a separate install step.
I know I've mentioned it a few times in this thread, just a very happy user and have found it a really good option for a lot of usage. I'll mostly just use the Deno.* methods or jsr:std for most things at this point, but there's also npm:zx which can help depending on what you're doing.
It also is a decent option for e2e testing regardless of the project language used.
Shell and bash are easy to write insecurely and open your CI runners or dev machines up for exploitation by shell injection. Non-enthusiasts writing complex CI pipelines pulling and piping remote assets in bash without ShellCheck is a risky business.
Python is a lot easier to write safely.
You shouldn't be pulling untrusted assets in CI regardless. Hacking your bash runner is the hardest approach anyways, just patch some subroutine in a dependency that you'll call during your build or tests.
> Huh? Who cares if the script is .sh, .bash, Makefile, Justfile, .py, .js or even .php?
Me, typically I have found it to be a sign of over-engineering and found no benefits over just using shell script/task runner, as all it should be is plumbing that should be simple enough that a task runner can handle it.
> If it works it works, as long as you can run it locally, it'll be good enough,
Maybe when it is your own personal project "If it works it works" is fine. But when you come to corporate environment there starts to be issues of readability, maintainability, proprietary tooling, additional dependencies etc I have found when people start to over-engineer and use programming languages(like Python).
E.g.
> never_inline 30 minutes ago | parent | prev | next [–]
> Build a CLI in python or whatever which does the same thing as CI, every CI stage should just call its subcommands.
However,
> and sometimes it's an even better idea to keep it in the same language the rest of the project is
I'll agree. Depending on the project's language etc other options might make sense. But personally so far everytime I have come across something not using a task runner it has just been the wrong decision.
> But personally so far everytime I have come across something not using a task runner it has just been the wrong decision.
Yeah, tends to happen a lot when you hold strong opinions with strong conviction :) Not that it's wrong or anything, but it's highly subjective in the end.
Typically I see larger issues being created from "under-engineering" and just rushing with the first idea people can think of when they implement things, rather than "over-engineering" causing similarly sized future issues. But then I also know everyone's history is vastly different, my views are surely shaped by the specific issues I've witnessed (and sometimes contributed to :| ), than anything else.
> Yeah, tends to happen a lot when you hold strong opinions with strong conviction :) Not that it's wrong or anything, but it's highly subjective in the end.
Strong opinions, loosely held :)
> Typically I see larger issues being created from "under-engineering" and just rushing with the first idea people can think of when they implement things, rather than "over-engineering"
Funnily enough running with the first idea I think is creating a lot of the "over-engineering" I am seeing. Not stopping to consider other simpler solutions or even if the problem needs/is worth solving in the first place.
> Yeah, tends to happen a lot when you hold strong opinions with strong conviction :) Not that it's wrong or anything, but it's highly subjective in the end.
I quickly asked Claude to convert one of my open source repos using Make/Nix/Shell -> Python/Nix to see how it would look. It is actually one of the better Python as a task runners I have seen.
* https://github.com/DeveloperC286/clean_git_history/pull/431
While the Python version is not as bad as I have seen previously, I am still struggling to see why you'd want it over Make/Shell.
It introduces more dependencies(Python which I solved via Nix) but others haven't solved this problem and the Python script has dependencies(such as Click for the CLI).
It is less maintainable as it is more code, roughly x3 the amount of the Makefile.
To me the Python code is more verbose and not as simple compared to the Makefile's target so it is less readable as well.
> It introduces more dependencies(Python which I solved via Nix) but others haven't solved this problem and the Python script has dependencies(such as Click for the CLI).
UV scripts are great for this type of workflow
There are even scripts which will install uv in the same file effectively making it just equivalent to ./run-file.py and it would handle all the dependency management the python version management and everything included and would work everywhere
https://paulw.tokyo/standalone-python-script-with-uv/
Personally I end up just downloading uv and so not using the uv download script from this but if I am using something like github action which are more (ephemeral?) I'd just do this.
Something like this can start out simple and can scale much more than the limitations of bash which can be abundant at times
That being said, I still make some shell scripts because executing other applications is first class support in bash but not so much in python but after discovering this I might create some new scripts with python with automated uv because I end up installing uv on many devices anyway (because uv's really good for python)
I am interested in bun-shell as well but that feels way too much bloated and even not used by many so less (AI assistance at times?) and I haven't understood bun shell at the same time too and so bash is superior to it usually
> UV scripts are great for this type of workflow
So previously when I have seen Python used as a task runner I think they used UV to call it. Although I don't think they had as a complete solution as your here auto-installing UV etc.
Although the example you've linked is installing UV if missing, the version is not pinned, I also don't think it is handling missing Python which is not pinned even if installed locally. So you could get different versions on CI vs locally.
While yes you are removing some of the dependencies problems created via using Python over Make/Shell I don't think this completely solves it.
> Something like this can start out simple and can scale much more than the limitations of bash which can be abundant at times
I personally haven't witnessed anytime I would consider the scales to have tipped in favour of Python and I would be concerned if they ever do, as really the task runner etc should be plumbing, so it should be simple.
> That being said, I still make some shell scripts because executing other applications is first class support in bash but not so much in python but after discovering this I might create some new scripts with python with automated uv because I end up installing uv on many devices anyway (because uv's really good for python)
Using Python/UV to do anything more complex than my example PR above?
I think UV scripts can/will actually install python and manage it itself as well and you can actually pin a specific version of python itself via Uv scripts
I copied this from their website (https://docs.astral.sh/uv/guides/scripts/#declaring-script-d...)
uv also respects Python version requirements: example.py
# /// script # requires-python = ">=3.12" # dependencies = [] # ///
# Use some syntax added in Python 3.12 type Point = tuple[float, float] print(Point)
> Using Python/UV to do anything more complex than my example PR above?
I can agree that this might be complex but that complexity has a trade off and of course nothing is shoe fits all but there are times when someone has to manage a complex CI environment and I looked at and there are some CI deterministic options too like invoke etc. and when you combine all of these, I feel like the workflow can definitely be interesting to say the least
Once again, I don't know what really ends up in github actions since I have never really used it properly, I am basing its critiques based on what I've read and what solutions (python came quite frequently) and something recently which I discovered (which was the blog)
This thing does a global uv install when run? That's obnoxious! Never running stuff from whoever wrote this.
Oh, and later the author suggests the script modify itself after running. What the fuck. Absolutely unacceptable way to deploy software.
Does it really matter if its a global install of uv or not especially on Github Actions
Also if this still bothers you, nothing stops you from removing the first x lines of code and having it in another .py file if this feels obnoxious to you
> Oh, and later the author suggests the script modify itself after running. What the fuck. Absolutely unacceptable way to deploy software.
Regarding author suggest its removes itself its because it does still feel clutterish but there is virtually 0 overhead in using/having it still be if you are already using uv or want to use uv
Oh also, (I am not the Author) but I have played extensively with UV and I feel like the script can definitely be changed to install it locally rather than globally.
They themselves mention it as #overkill on their website but even then it is better than whatever github action is
I'm a huge believer in the rule that everything GH actions does should be a script you can also run locally.
Yes I believe the same too and I think we are on the same goal. I think that I can probably patch this code to install uv, let's say locally instead of globally if that's a major concern. I feel like its not that hard.
It's easy enough to patch. It's the philosophy that bugs me. We already have a huge problem with routine workflows pulling things from the network (often, without even a semblance of hash-locking) and foregoing the traditional separation between environment setup and business logic. There's a lot of value into having discrete steps for downloading/installing stuff and doing development, because then you can pay special attention to the former, look for anything odd, read release notes, and so on. Between explicit, human-solicited upgrades, dev workflows should be using, ideally, vendored dependencies, or, if not that, then at least stuff that's hash-verified end-to-end.
Someday, someone is going to have a really big disaster that comes out of casual getting unauthenticated stuff from somebody else's computer.
I agree with you and you raise some good points
I think your reason of worrying is either that A) packages can update and contain malware or B) Uv's installation itself. might have malware if any of A) or B) get hacked
Regarding A) I feel like uv's dependencies can be pinned to a certain date to make them reproducible and this can come of help (https://docs.astral.sh/uv/guides/scripts/#improving-reproduc...)
Regarding B) I feel like they provide attestations via GitHub Artifact Attestations and the script could once again be modified to actually verify it via github attestations and they also provide ghcr artifacts (as such immutability) atleast of docker images and I looked further into it and it seems that you can use github artifacts to upload normal binary files as well so I will probably take a look into seeing if I can do something like this for uv's ghcr
Effectively after A) and B) the trust just ends up being reliant on Github's Microsoft infrastructure (usually) and perhaps python infrastructure which is on fastly
But I feel like this is for cases of extremely sensitive workflows But I feel like I might still take a look at it because security still feels very interesting to me and just because of this discussion, I can see at some pointers of following up on curiosity lol
Anyways would love to continue our discussion and probably update you on trying to make a script which could actually be completely pinned (atleast uv binary instead of just running a shell script from the astral servers in such case)
Using shell becomes deeply miserable as soon as you encounter its kryptonite, the space character. Especially but not limited to filenames.
I find that shell scripting has a sharp cliff. I agree with the sentiment that most things are over engineered. However it’s really easy to go from a simple shell script running a few commands to something significantly more complex just to do something seemingly simple, like parse a semantic version, make an api call and check the status code etc, etc.
The other problem with shell scripting on things like GHA is that it’s really easy to introduce security vulnerabilities by e.g forgetting to quote your variables and letting an uncontrolled input through.
There’s no middle ground between bash and python and a lot of functionality lives in that space.
> However it’s really easy to go from a simple shell script running a few commands to something significantly more complex just to do something seemingly simple, like parse a semantic version, make an api call and check the status code etc, etc.
Maybe I keep making the wrong assumption that everyone is using the same tools the same way and thats why my opinions seem very strong. But I wouldn't even think of trying to "parse a semantic version" in shell, I am treating the shell scripts and task runners as plumbing, I would be handing that of a dedicated tool to action.
Let’s say have a folder of tarballs and need to install the latest version. I could reach for an additional “dedicated” tool, get it installed into the CI environment and then incorporate it into the build process, or I could just make a slight modification to my existing shell script and do something like “ls mypkg-*.tar.gz | xargs -n1 | sort -Vr | head -n1” and then move on. But then we start publishing multiple release candidates and now I need to add to that logic to further distinguish between earlier and later rc versions. And so on and so forth…
Now I’m in agreement with you that this is a bad fit for shell scripting, but it is often pragmatic and expedient. And bc there is a cliff between bash and (say) python, some of the time, you’re going to choose the path of least resistance.
Now scale this out to a small team of engineers all facing the same dumb decision of needing to make some tradeoff when they would much rather be writing application logic. The lack of a ubiquitous and robust intermediate language leads to brittle CI fraught with security vulnerabilities.
While the example I provided is a bit contrived, this behavior isn’t hypothetical. I see it everywhere I’ve worked.
Yes I am seeing what your saying, it can start of simple and people will choose the path of least resistance and you'll end up with a ball of mud eventually. Fortunately I have not come across anything that bad yet.
I know it is a made up example, but I am not sure Python would improve the situation. I would be looking at either Docker or SBOMs to try and improve the situation and then hopefully the release process would become a lot simpler.
yea imagine having to maintain a python dependency (which undergoes security constraints) all because some junior cant read/write bash... and then that junior telling you you're the problem lmao
I mean, at some point you are bash calling some other language anyway.
I'm a huge fan of "train as you fight", whatever build tools you have locally should be what's used in CI.
If your CI can do things that you can't do locally: that is a problem.
> If your CI can do things that you can't do locally: that is a problem.
IME this is where all the issues lie. Our CI pipeline can push to a remote container registry, but we can't do this locally. CI uses wildly different caching strategies to local builds, which diverges. Breaking up builds into different steps means that you need to "stash" the output of stages somewhere. If all your CI does is `make test && make deploy` then sure, but when you grow beyond that (my current project takes 45 minutes with a _warm_ cache) you need to diverge, and that's where the problems start.
Ironically, at least for a couple recent projects... just installing dependencies fresh is as fast on GH Actions as the GH caching methods, so I removed the caching and simplified the workflows.
In fact, uv's docs half-suggest this:
> With uv, it turns out that it's often faster to omit pre-built wheels from the cache (and instead re-download them from the registry on each run). On the other hand, caching wheels that are built from source tends to be worthwhile, since the wheel building process can be expensive, especially for extension modules.
https://docs.astral.sh/uv/concepts/cache/#caching-in-continu...
On the projects we work on, we don’t use ephemeral workers and we rely on tool chains for incremental builds. It’s flakier but 100x faster…
> If your CI can do things that you can't do locally: that is a problem.
Probably most of the times when this is an actual problem, is building across many platforms. I'm running Linux x86_64 locally, but some of my deliverables are for macOS and Windows and ARM, and while I could cross-compile for all of them on Linux (macOS was a bitch to get working though), it always felt better to compile on the hardware I'm targeting.
Sometimes there are Windows/macOS-specific failures, and if I couldn't just ssh in and correct/investigate, and instead had to "change > commit > push" in an endless loop, it's possible I'd quite literally would lose my mind.
I literally had to do this push > commit > test loop yesterday because apparently building universal Python wheels on MacOS is a pain in the ass. And I don't have a mac, so if I want to somewhat reliably reproduce how the runner might behave, I have to either test it on GH actions or rent one from something like Scaleway. Mainly because I don't currently knwo how else to do it. It's so, so frustrating and if anyone has ideas on making my life a bit better that would be nice lol.
there is quickemu which can install mac vm on linux (or any other host) rather quickly, what are your thoughts on it (I am an absolute quickemu shill because I love that software)
https://github.com/quickemu-project/quickemu [ Quickly create and run optimised Windows, macOS and Linux virtual machines ]
Thank you so much for this... If I could +1 a dozen times I would.
Thanks! Glad I could help. If I may ask, what specific use case are you using quickemu for? Is it also for running mac machines on say linux?
That's what I intend to use it for, Mac and Windows... I'm starting on an app that I want to work cross platform (tauri/rust w/ react+mui) and want to be able to do manual testing or troubleshooting as needed on mac and windows without needing a separate machine.
My laptop is an M1 MacBook Air, and I do have an N100 I could use for Windows... I'd just assume use my fast desktop which even emulated is likely faster and not have to move seats.
yes, I think just the amount of friction it can reduce might be worth it in the first place.
Oh btw although there are many primitives which help transferring files between VM's and others by having sshfs etc., one of the things which I enjoyed doing in quickemu is using the beloved piping-server
https://github.com/nwtgck/piping-server Infinitely transfer between every device over pure HTTP with pipes or browsers
The speeds might be slow but I was using it to build simple shell scripts and you can self host it or deploy on cf workers too most likely which is really simple but I haven't done it
But for quick deployments/transfers of binaries/simple files, its great as well. Tauri is meant to be lightweight/produce small binaries so I suppose one can try it but there are other options as well
Piping Serrvers + quickemu felt like a cheatcode to me atleast for more ephemeral vm's based workflow but of course YMMV
Good luck with your project! I tried building a tauri app once for android just out of mere curiosity on linux and it was hell. I didn't know anything about android development but setting up the developer environment was really hard and I think I forgot everything I learnt from that but wish I had made notes or even video documenting the process
Fortunately/Unfortunately it wouldn't be a good experience for Phone use, maybe table as part of it will be displaying BBS-ANSI art and messages which lends itself to a larger display.
> If your CI can do things that you can't do locally: that is a problem.
Completely agree.
> I'm a huge fan of "train as you fight", whatever build tools you have locally should be what's used in CI.
That is what I am doing, having my GitHub Actions just call the Make targets I am using locally.
> I mean, at some point you are bash calling some other language anyway.
Yes, shell scripts and or task runners(Make, Just, Task etc) are really just plumbing around calling other tools. Which is why it feels like a smell to me when you need something more.
I don't agree with (1), but agree with (2). I recommend just putting a Makefile in the repo and have that have CI targets, which you can then easily call from CI via a simple `make ci-test` or similar. And don't make the Makefiles overcomplicated.
Of course, if you use something else as a task runner, that works as well.
For certain things, makefiles are great options. For others though they are a nightmare. From a security perspective, especially if you are trying to reach SLSA level 2+, you want all the build execution to be isolated and executed in a trusted, attestable and disposable environment, following predefined steps. Having makefiles (or scripts) with logical steps within them, makes it much, much harder to have properly attested outputs.
Using makefiles mixes execution contexts between the CI pipeline and the code within the repository (that ends up containing the logic for the build), instead of using - centrally stored - external workflows that contains all the business logic for the build steps (e.g., compiler options, docker build steps etc.).
For example, how can you attest in the CI that your code is tested if the workflow only contains "make test"? You need to double check at runtime what the makefile did, but the makefile might have been modified by that time, so you need to build a chain of trust etc. Instead, in a standardized workflow, you just need to establish the ground truth (e.g., tools are installed and are at this path), and the execution cannot be modified by in-repo resources.
That doesn't make any sense. Nothing about SLSA precludes using make instead of some other build tool. Either inputs to a process are hermetic and attested or they're not. Makefiles are all about executing "predefined steps".
It doesn't matter whether you run "make test" or "npm test whatever": you're trusting the code you've checked out to verify its own correctness. It can lie to you either way. You're either verifying changes or you're not.
You haven't engaged with what I wrote, of course it doesn't make sense.
The easiest and most accessible way to attest what has been done is to have all the logic of what needs to be done in a single context, a single place. A reusable workflow that is executed by hash in a trusted environment and will execute exactly those steps, for example. In this case, step A does x, and step B attests that x has been done, because the logic is immutably in a place that cannot be tampered with by whoever invokes that workflow.
In the case of the makefile, in most cases, the makefile (and therefore the steps to execute) will be in a file in the repository, I.e., under partial control of anybody who can commit and under full control of those who can merge. If I execute a CI and step A now says "make x", the semantic actually depends on what the makefile in the repo includes, so the contexts are mixed between the GHA workflow and the repository content. Any step of the workflow now can't attest directly that x happened, because the logic of x is not in its context.
Of course, you can do everything in the makefile, including the attestation steps, bringing them again in the same context, but that makes it so that once again the security relevant steps are in a potentially untrusted environment. My thinking specifically hints at the case of an organization with hundreds of repositories that need to be brought under control. Even more, what I am saying make sense if you want to use the objectively convenient GH attestation service (probably one of the only good feature they pushed in the last 5 years).
Usually, the people writing the Makefile are the same that could also be writing this stuff out in a YAML (lol) file as the CI instructions, often located in the same repository anyway. The irony in that is striking. And then we have people who can change environment variables for the CI workflows. Usually, also developers, often the same people that can commit changes to the Makefile.
I don't think it changes much, aside from security theater. If changes are not properly reviewed, then all fancy titles will not help. If anything, using Make will allow for a less flaky CI experience, that doesn't break the next time the git hoster changes something about their CI language and doesn't suffer from YAMLitis.
You're correct. It's absolutely security theater. Either you trust the repository contents or you don't. There's no, none, zilch trust improvement arising from the outer orchestration being done in a YAML file checked into the repo and executed by CI instead of a Makefile also executed by CI.
What's the threat model Wilder is using exactly? Look, I'm ordinarily all for nuance and saying reasonable people can disagree when it comes to technical opinions, but here I can't see any merit whatsoever to the claim that orchestrating CI actions with Make is somehow a security risk when the implementations of these actions at some level live in the repo anyway.
See my other comment.
> when the implementation of these actions at some level live in the repo anyway
This is the false assumption. You can standardize that not to happen, and you can verify at runtime that's not the case. You can control admission of container images for example, restricting those that were built by your workflow X (the approved, central, controlled, standard one) and reject anything else. You do this via build provenance attestation.
With makefile I don't know how you can achieve standard and centrally manageable (and verifiable) instructions (which are part of SLSA).
Maybe a concrete example would help.
> With makefile I don't know how you can achieve standard and centrally manageable (and verifiable) instructions (which are part of SLSA).
The way I'm thinking about it, we distinguish instructions from policy. Policy, of course, has to be imposed from outside a package somehow, but the instructions taken within policy seem like things that should evolve with it. For example, "no network access" might be an externally enforced policy. Another might be limitations of dependencies, enforced by making exceptions to the previous policy.
But in the end, you have to do something. Are you suggesting a model in which a project's entire test suite lives in a separate repository with an access control policy that differs from that project's so that the project's authors can't cheat on test coverage or pass rate?
Sure, you can do that, but the instructions for running those tests still have to live somewhere, and a makefile in that test repository seems like as good a place as any.
A classic example would be build (container build) images instructions. Of course, other examples are running a security static analysis, running tests, or whatever other standard practice is desidered for security.
Say that you build 100 images, each with its own repository where there is the code, there is the Dockerfile etc. Now, if you build images with "make build", it means that ultimately the actual command line flags for the build command are going to be in the Makefile, inside each repository. These flags could include flags like --cache-from, build arguments, --add-host, or many other flags that may not be desirable in an organization. Imagine what the workflow file would like for such a case: you have your in-repo CI file that clones the repo, and then executes make test, make build, make push, etc. Do you know if make push sent the PCI registry secret somewhere else on the internet? Do you know what exactly flags were used to build? Not really, you need to trust the code inside the repo to at most collect this information, and how do you know it was not tampered or spoofed?
Compare it with the following scenario: All 100 repos at some point in that CI invoke an external workflow, which contains instructions to build images, push them to the registry (and run tests before, whatever). This can obviously also be a combination or workflows, I am saying one for simplicity. This workflow (or workflows) at the end produces an attestation, including provenance data and about what is run. It can do so reliably, because the instructions of what had to be done are within its own context. There is no input that comes from outside (e.g., the repository) that determined build flags or instructions, outside of what the workflow exposed (e.g., few variables).
In the second case, I can then use a policy that restrict images that have attestation and that were built by my workflow(s) that contain standard and deterministic instructions. This way I know that no image was built using weird build flags, including strange dependencies, maybe tampering with network and DNS (in makefile I can write whatever) or local docker cache, etc. I know this because a standard workflow was executed in a fresh, "sterile" environment with a standard set of instructions, that can't be changed or tampered by those controlling the repository content.
Note that all of the above is solely concerned with being able to make security statements on the build artifact produced, not on the context of the build (e.g., no network access).
That's a great point. If we keep following the requirement for attestation to its logical conclusion we would end up replicating the entire server running the repository at the source, then the cycle repeats
Not really, that's the point. Reusable workflow, in a tightly controlled repos avoid exactly what you are saying and they are a fairly standard practice (if anything, also to avoid having 200 versions of CI instructions). You can also verify attestations provenance by enforcing attestation performed via that particular, approved workflow, which is not security theater, it's actual real controls.
Makefile or scripts/do_thing either way this is correct. CI workflows should only do 1 thing each step. That one thing should be a command. What that command does is up to you in the Makefile or scripts. This keeps workflows/actions readable and mostly reusable.
>I don't agree with (1)
Neither do most people, probably but it's kinda neat how they suggested fix for github actions' ploy to maintain vendor lock-in is to swap it with a language invented by that very same vendor.
makefile commands are the way
I was once hired to manage a build farm. All of the build jobs were huge pipelines of Jenkins plugins that did various things in various orders. It was a freaking nightmare. Never again. Since then, every CI setup I’ve touched is a wrapper around “make build” or similar, with all the smarts living in Git next to the code it was building. I’ll die on this hill.
Pathological organizations for some reason all prefer to never version the CI in the same repository it is testing.
#2 is not a slam dunk because the CI system loses insight into your build process if you just use one big script.
Does anyone have a way to mark script sections as separate build steps with defined artifacts? Would be nice to just have scripts with something like.
They could noop on local runs but be reflected in the github/gitlab as separate steps/stages and allow resumes and retries and such. As it stands there's no way to really have CI/CD run the exact same scripts locally and get all the insights and functionality.I haven't seen anything like that but it would be nice to know.
Do you (or does anyone) see possible value in a CI tool that just launches your script directly?
It seems like if you
> 2. Don't have logic in your workflows. Workflows should be dumb and simple (KISS) and they should call your scripts.
then you’re basically working against or despite the CI tool, and at that point maybe someone should build a better or more suitable CI tool.
Can we have a CI tool, that simply takes a Makefile as input? Perhaps takes all targets, that start with "ci" or something.
Build a CLI in python or whatever which does the same thing as CI, every CI stage should just call its subcommands.
Just use a task runner(Make, Just, Taskfile) this is what they were designed for.
I typically use make for this and feel like I’m constantly clawing back scripts written in workflows that are hard to debug if they’re even runnable locally.
This isn’t only a problem with GitHub Actions though. I’ve run into it with every CI runner I’ve come across.
In many enterprise environments, deployment logic would be quite large for bash.
Personally, I have never found the Python as a task runners to be less code, more readable or maintainable.
How do you handle persistent state in your actions?
For my actions, the part that takes the longest to run is installing all the dependencies from scratch. I'd like to speed that up but I could never figure it out. All the options I could find for caching deps sounded so complicated.
> How do you handle persistent state in your actions?
You shouldn't. Besides caching that is.
> All the options I could find for caching deps sounded so complicated.
In reality, it's fairly simple, as long as you leverage content-hashing. First, take your lock file, compute the sha256sum. Then check if the cache has an artifact with that hash as the ID. If it's found, download and extract, those are your dependencies. If not, you run the installation of the dependencies, then archive the results, with the ID set to the hash.
It really isn't more to it. I'm sure there are helpers/sub-actions/whatever Microsoft calls it, for doing all of this with 1-3 lines or something.
The tricky bit for me was figuring out which cache to use, and how to use and test it locally. Do you use the proprietary github actions stuff? If the installation process inside the actions runner is different from what we use in the developer machines, now we have two sets of scripts and it's harder to test and debug...
> Do you use the proprietary github actions stuff?
If I can avoid it, no. Almost everything I can control is outside of the Microsoft ecosystem. But as a freelancer, I have to deal a bunch with GitHub and Microsoft anyways, so in many of those cases, yes.
Many times, I end up using https://github.com/actions/cache for the clients who already use Actions, and none of that runs in the local machines at all.
Typically I use a single Makefile/Justfile, that sometimes have most of the logic inside of it for running tests and what not, sometimes shell out to "proper" scripts.
But that's disconnected from the required "setup", so Make/Just doesn't actually download dependencies, that's outside of the responsibilities of whatever runs the test.
And also, with a lot of languages, it doesn't matter if you run an extra "npm install" over already existing node_modules/, it'll figure out what's missing/there already, so you could in theory still have "make test" do absolute everything locally, including installing dependencies (if you now wish this), and still do the whole "hash > find cache > extract > continue" thing before running "make test", and it'll skip the dependencies part if it's there already.
If you are able to use the large runners, custom images are a recent addition to what Github offers.
https://docs.github.com/en/actions/how-tos/manage-runners/la...
Depends on the build toolchain but usually you'd hash the dependency file and that hash is your cache key for a folder in which you keep your dependencies. You can also make a Docker image containing all your dependencies but usually downloading and spinning that up will take as long as installing the dependencies.
For caching you use GitHubs own cache action.
You don't.
For things like installing deps, you can use GitHub Actions or several third party runners have their own caching capabilities that are more mature than what GHA offers.
We use Nuke for this purpose and I really like it. We're a .NET shop, and Nuke is C#, so works pretty well. Almost all our Nuke targets are shared between local and CI except some docker stuff that needs caching in CI.
Minor variance on #1, I've come to use Deno typescripts for anything more complex than what can be easily done in bash or powershell. While I recognize that pwsh can do a LOT in the box, I absolutely hate the ergonomics and a lot of the interactions are awkward for people not used to it, while IMO more developers will be more closely aligned to TypeScript/JavaScript.
Not to mention, Deno can run TS directly and can reference repository/http modules directly without a separate install step, which is useful for shell scripting beyond what pwsh can do. ex: pulling a dbms client and interacting directly for testing, setup or configuration.
For the above reasons, I'll also use Deno for e2e testing over other languages that may be used for the actual project/library/app.
Step 0. Stop using CI services that purposefully waste your time, and use CI services that have "Rebuild with SSH" or similar. From previous discussions (https://news.ycombinator.com/item?id=46592643), seems like Semaphore CI still offers that.
> Don't use bash
What? Bash is the best scripting language available for CI flows.
1. Just no. Unless you are some sort of Windows shop.
Pwsh scripts are portable across mac, linux and windows with arguably less headache than bash. Its actually really nice. You should try it.
If you don't like it, you can get bash to work on windows anyway.
If you're building for Windows, then bash is "just no", so it's either cmd/.bat, or pwsh/.ps. <shrugs>
All my windows work / ci runs still use bash.
I develop on Windows. And I use bash and (gnu) make - combination that cannot be beat, in my experience.
That’s the only reason for sure.
I mean, if you're a Windows shop you really should be using powershell.
I've never been a fan of PowerShell. I work across Windows and Linux on .NET and Nuke has become my go to tool for this.
you should try it, powershell isnt just 'type insanely long phrases', there are aliases for it
Its not Github Actions' fault but the horrors people create in it, all under the pretense that automation is simply about wrapping a GitHub Action around something. Learn to create a script in Python or similar and put all logic there so you can execute it locally and can port it to the next CI system when a new CTO arrives.
I think in this case they hate the fact that they cannot easily SSH into the failing VM and debug from there. Like "I have to edit my workflow, push it, wait for it to run and fail, and repeat".
Yep, my company moved onto GH Actions a few years ago and this was probably the single biggest pain point. But also the whole system just feels awkward and annoying to work with. It feels like a classic Microsoft product that would never get traction from a standalone company but because it's part of Microsoft/GitHub and is "good enough" lots of people put up with it.
I had the weirdest thing happen a few days ago.. and only seemed to be happening in the GH runner for a PR.... somehow a function was duplicated in the action runner, and not in local or anwhere else... no idea how the corruption happened... it literally took me hours or pushing minor changes to try to correct the issue... I finally cat'd that file contents out and yep, the function was duplicated... no idea how. Had to create a new branch, copy the changes and then it worked.|
Still no idea what happened or how to ever fix/prevent it again.
Obviously, by doing everything inside a container, where you control things. The GH action should only start the container, and pass it some env vars.
CircleCI had this exact feature
One can get the ssh access with self-hosted runners but it is problematic because uncovering secrets becomes trivial.
Uncovering secrets is usually trivial. `printenv` in a build script does that pretty reliably.
What do you mean? Simple env prints get masked as *** in logs
I guess one can always just echo the secret to a file and upload-artifact it
Unless you've got something sanitizing env prints in your logs, no, they don't get masked. I've leaked secrets accidentally while debugging environment variable issues.
Of course the bigger side of the issue is that anyone who can run code in the CI context can view all data in the CI context. So if an untrusted dev can write a build script that executes with secrets in the environment, they can leak those secrets. CI is RCE-as-a-service, you implicitly give permission for anyone who can edit build scripts to run arbitrary code. Like you say, they can echo the secret to a file & upload it elsewhere, and they can also use it directly from the edited build script.
Secrets that you store in github actions secrets?
Yes. I've done it. That's why there is an "approve and run" button when the PR comes from another repo. So that the maintainers can check if the PR is trying to exfiltrate secrets... or do other bad things of course.
The whole point of the github action secrets is that they are accessible by the CI when needed. Which means they are accessible. It's just a convenient way to store the secret in such a way that it is not completely public. But you still have to trust github and your maintainers.
Like masking would work against anything but accidents.
No, it is github's fault. They encourage the horrors because they lead to vendor lock in. This is the source of most of Microsoft's real profit margins.
This is probably why they invented a whole programming language and then neglected to build any debugging tools for it.
[dead]
> Learn to create a script in Python or similar and put all logic there so you can execute it locally and can port it to the next CI system when a new CTO arrives.
That's a perfectly good several weeks of time un-estimateable busywork I could be getting paid for!
The hard parts are things like deploying artifacts, publishing releases, caching, etc. that are CI-specific and can't be run locally.
This!
All discussions so far seem to be around building artifacts which is the easy part. How to make the full CI/CD pipeline work with GHA is a pain. GHA has no state mechanism easily used, you have to create it all yourself.
This is giving "Debian systemd units call their old init.d scripts" energy but I kind of like it
systemd units that are small, simple, and call into a single script are usually fantastic. There's no reason for these scripts to be part of another init system; but making as much of your code completely agnostic to the env it runs in sounds good regardless. I think that's the feeling you're feeling.
It's 100% GA's fault and they do it on purpose.
if you can't run it locally, it's shit
The way I deal with all these terrible CI platforms (there is no good one, merely lesser evils) is to do my entire CI process in a container and the CI tool just pulls and runs that. You can trivially run this locally when needed.
Of course, the platforms would rather have you not do that since it nullifies their vendor lock-in.
I really like the SourceHut CI, because:
1. When the build fails, you can SSH into the machine and debug it from there.
2. You can super easily edit & run the manifest without having to push to a branch at all. That makes it super easy to even try a minimum reproducible example on the remote machine.
Other than that, self-hosting (with Github or preferrably Forgejo) makes it easy to debug on the machine, but then you have to self-host.
Self-hosted runners with Github is a whole world of pain because it literally just runs commands on the host and does not handle provisioning/cleanup, meaning you need to make sure your `docker run` commands don't leave any leftover state that can mess up future/concurrent builds. It doesn't even handle concurrency by itself, so you have to run multiple instances of the runner.
Github supports ephemeral runners which are limited to a single job.
You can use `workflow_job` webhook to be notified of a new job, after that you need to call `generate-jitconfig` API to get a just-in-time configuration token, and then you can start a Github runner in ephemeral mode with the JIT token.
This allows you to orchestrate Docker containers, KVM instances, etc., which are used for a single time, and then destroyed.
There are some open source projects, such as using ephemeral Kubernetes pods with the ephemeral runners.
That was included in my "but then you have to self-host" :-).
As I said, I really like the SourceHut CI.
Do you have any experience self-hosting SourceHut? I’d really like to do so, but I get weak knees every time I look at the docs for it.
I don't, but I would be curious.
But I'm happy to contribute (money) to SourceHut, they're doing a good job.
Forgejo handles all of this quite well and is, in my experience, muuuch faster than GHA
Thats what i always did for our gitlab CI pipeline, just deploy dedicated images for different purposes. We had general terraform images for terraform code, this made it easy to standardize versions etc. Then we made specific ones for projects with a lot of dependencies so we could run the deployment pipeline in seconds instead of minutes. But now you need to maintain the docker images too. All about trade-offs.
The one issue with that is there isn’t a good way to containerise macOS builds.
I mean this has been an issue building for iOS forever. The MacOS lock-in sucks really really badly.
It's kind of their whole thing, when you think about it. They didn't get to where they are by playing nice with others. If you're supporting anything in the Apple ecosystem, the fix is in.
Github delivers an exceptionally detailed webhook. Ironically, there is zero actual lock in, the CI/CD cargo cult is that strong they didn't even need it.
Your newsletter. I needs it.
I like Github Actions and it is better than what I used before (Travis) and I think it solves an important problem. For OSS projects it's a super valuable free resource.
For me what worked wonders was adopting Nix. Make sure you have a reproducible dev environment and wrap your commands in `nix-shell --run`, or even better `nix develop --command`, or even better your most of your CI tasks derivations that run with `nix build` or `nix flake check`.
Not only does this make it super easy to work with Github Actions, also with your colleagues or other contributors.
Second the Nix approach. One can even build a github actions-compatible container out of a flake and have actions run in it. I have done so for my personal projects https://github.com/anttiharju/compare-changes
isn't this better served with something like Bazel (behind Nix of course?)
Could be, not familiar with Bazel. Thanks for mentioning it now can check it out.
Wrote a comment to someone here but I thought of deciding to make a main comment here as well
Note that I don't really use github actions much but have heard about its architecture
From my understanding, I feel like Github actions should just be a call to some bash or python file. Bash has its issues so I prefer python
I recommend people to take a look at https://paulw.tokyo/standalone-python-script-with-uv/ and please tell me if something like this might be perfect for python scripts in github actions as this script would automatically install uv, get all the dependencies of python and even the runtime I think and then execute the python code all while being very managable usually and it can run locally as well
The only Issue I feel like I might have with this is say why go something with this complex when bash exists or the performance concerns of installing uv but considering its github actions, I feel like the latter is ruled out.
Bash is good as well but bash has some severe limitations. and I feel like Python can be good case for something like this plus its ecosystem is a bit mature and you could even create web servers or have some logs be reported to your custom server or automate just basically everything
To me this script feels like the best of both worlds and something genuinely sane to build upon.
> I feel like Github actions should just be a call to some bash or python file. Bash has its issues so I prefer python
...and that's how it is supposed to be used. The YAML file is mainly there for defining the runtime environment (e.g. the operating system or docker image to run in), and to describe the job dependency tree. The actual actions to execute should be delegated to scripts outside the GH YAML.
I actually built the last thing last weekend weirdly enough.
gg watch action
Finds the most recent or currently running action for the branch you have checked out. Among other things.
https://github.com/frankwiles/gg
Oh this is excellent. This is everything I wanted the `gh` cli to be, thanks.
edit: Just a quick note, the `gg` and `gg tui` commands for me don't show any repos at all, the current context stuff all works perfectly though.
Ah sorry need to make the docs more clear. You need to run ‘gg data refresh’ to populate the local DB first.
Ah, magnificent! Thanks!
Sure thing. Also just made it easier to install with Homebrew since people seem to be liking it.
May also be worth adding mise to the README install options given its growing popularity. I installed globally with `mise use -g ubi:frankwiles/gg@latest`. I've invoked gg so many times over the last day, it's exactly what I needed - thanks again.
As a founder of Depot [0], where we offer our own faster and cheaper GitHub Actions runners, I can assure everyone that this feeling is the majority and not the minority.
Sounds strange to say as someone who has a product that is built around making GitHub Actions exponentially faster to close these feedback loops faster.
But I can honestly say it's only really possible because the overall system with GitHub Actions is so poor. We discover new bottlenecks in the runner and control plane weekly. Things that you'd think would be simple are either non-existent, don't work, or have terrible performance.
I'm convinced there are better ways of doing things, and we are actively building ideas in that realm. So if anybody wants to learn more or just have a therapy session about what could be better with GitHub Actions, my email is in my bio.
[0] https://depot.dev/
Would a tool like act help here? (https://github.com/nektos/act) I suppose orchestration that is hiding things from different processor architectures could also very well run differently online than offline, but still.
I haven’t looked into act for some time but I remember it NOT being a direct stand in locally. Like it covered 80% of use cases.
Maybe that has changed.
That's correct and it's linux-only (as of the last time I looked), you can run it on macOS but you can't run macOS runners (which is where I need the most help debugging normally, for building iOS apps).
it still isn't an 100% drop-in replacement
It's generally very helpful - someone else mentioned here the fundamental problem is lack of a tight feedback loop. It doesn't perfectly replicate the GH environment, but for my use case that doesn't matter and it's super nice to have.
I think really what would help is a way to SSH into the machine after it fails. SourceHut allows that, and I find it great.
The main problem with actions is the way they advertise its usage "just put workflows together" is a horrible and non-debuggable way to do things. But even in the tech itself, caching is pretty stingy which can slow dev builds for fairly simple projects because every run will repeat some common work unless you have the cache perfectly configured (did u cover npm, docker, etc. with cache keys correctly?)
Looking at these flaws, running workflows from a persistent VM of ur own becomes pretty tempting because you don't need to copy caches around and can easily SSH in.
GH Actions isn't great compared to other CI systems, but it's also not particularly worse until you get into the nitty gritty details.
The most important advice is probably to put as much code as possible into locally runnable scripts written in a cross-platform scripting language (e.g. Python or Node.js) to avoid 'commit-push-ci-failure' roundtrips.
Only use the GH Actions YAML for defining the runtime environment and job dependency tree.
s/Node/Deno/ - direct repo/module references, no install step required.
Is any of this unique to GitHub Actions that does not happen on other cloud CI platforms?
The best CI platforms let you "Rebuild with SSH" or something similar, and instead of having the cycle of "change > commit > push > wait > see results" (when you're testing CI specific stuff, not iterating on Makefiles or whatever, assuming most of it is scripts you can run both locally and in CI), you get a URL to connect to while the job is running, so you can effectively ensure manually it works, then just copy-paste whatever you did to your local sources.
I use that a lot with SourceHut: after a build fails, you have 10 minutes to SSH into the machine and debug from there. Also they have a very nice "edit manifest and run" feature that makes it easy to quickly test a change while debugging.
Are there other platforms allowing that? Genuinely interested.
> after a build fails, you have 10 minutes to SSH into the machine and debug from there.
Ah, that's like 90% of the way there, just need to enable so the SSH endpoint is created at the beginning, rather than the end, so you could for example watch memory usage and stuff while the "real" job is running in the same instance.
But great to hear they let you have access to the runner at all, only that fact makes it a lot better than most CI services out there, creds to SourceHut.
> just need to enable so the SSH endpoint is created at the beginning
Maybe it is, I've never tried :-). I don't see a reason why not, probably it is.
There are a couple of GitHub actions that let you do this.
do you mean https://github.com/nektos/act or there is something else ?
No. Act is for running actions locally. What was mentioned is a way to insert an SSH step at some well-chosen point of a workflow so you can login at the runner and understand what is wrong and why it's not working. I have written one such thing, it relies on cloudflare free tunnels. https://github.com/efrecon/sshd-cloudflared. There are other solutions around to achieve more or less the same goal.
I designed a fairly complex test matrix with a lot of logic offloading to the control mechanisms Gitlab offers. You create job templates or base jobs that control the overall logic and extend them for each particular use case. I had varying degrees of success, and it's not a job for a Devs side quest, that means I think you need someone dedicated to explore, build and debug these pipelines, but for a CI tool it's very good.
Because you can extend and override jobs, you can create seams so that each piece of the pipeline is isolated and testable. This way there is very little that can go wrong in production that's the CI fault. And I think that's only possible because of the way that Gitlab models their jobs and stages.
Guys,
GitHub action is a totally broken piece of s !! I know about that broken loops cause I had to deal with it an incredible number of times.
I very often mention OneDev in my comments, and you know what ? Robin solved this issue 3 years ago : https://docs.onedev.io/tutorials/cicd/diagnose-with-web-term...
You can pause your action, connect through a web terminal, and debug/fix things live until it works. Then, you just patch your action easily.
And that’s just one of the many features that make OneDev superior to pretty much every other CI/CD product out there.
Of all the valid complaints about Github Actions or CI in general, this seems to be an odd one. No details about what was tried or not tried, but hard to see a `-run: go install cuelang.org/go/cmd/cue@latest` step not working?
The problem was the environment setup? You couldn't get CUE on Linux ARM and I am assuming when you moved to Makefiles you removed the need for CUE or something? So really the solution was something like Nix or Mise to install the tooling, so you have the same tooling/version locally & on CI?
Exactly.
"GitHub actions bad" is a valid take - you should reduce your use to a minimum.
"My build failed because of GitHub actions couldn't install a dependency of my build" is a skill issue. Don't use GitHub actions to install a program your build depends on.
[dead]
So many engineers could put the hours spent debugging GH actions to use developing expertise to run their own CI. But people either don’t believe they can, can’t convince decision makers to let them try, or just want to fix their own problem and move on.
I was convinced GH actions was best practice and it was normal to waste hours on try-and-pray build debugging, until one day GH actions went down and I ran deploys from my laptop and remembered how much better life can be without it..
(Solo dev here - but opensource CI on an EC2 instance can be just as nice)
> So many engineers could put the hours spent debugging GH actions to use developing expertise to run their own CI.
If I run my own CI, then the compliance team has to get involved to run various endpoint security and update management tools on whatever system I'm running the CI on.
Then the costs of staying with GH actions need to be made known so they can be balanced against the cost of doing things differently. Of course there’s a cost involved in just getting those numbers too.
It’s all trade offs.
lol, this post makes me feel like crying at my job in BigTech co. He’s complaining about a 30 min feedback loop maybe, mine is 2+ hours.
I write a piece of code, however small the change. Then run the proc, first it takes ~20 mins to compile. Then since it is ML, it can only run on a remote server. It takes an easy 20 more minutes to start running in the remote server. Then after another 40 minutes I can confirm the job failed. The only way to debug is to read through a massive log file, for which we have an in built log reader lol. The log file will have thousands of errors that literally don’t matter, and you’ll never have enough context to know which errors don’t matter. You can simply ping someone and ask, does this error matter, could this be the reason my entire proc failed, oh no this is just a useless log that fails all the time, I shouldn’t have been wasting a day digging into this, ok thanks bye.
But this isn’t it, not even close lol, we in fact have a custom DSL to define computational graphs, which of course does not have any linter, or even any compiler and a very broken visualizer but our entire org runs on this. Syntax errors, logic errors, actual race conditions are all caught the exact same way —- as the process dying after trying to run on a remote server for 2+ hours with no useful error log. So our workflow is to just get a cup of coffee and stare at the graph which can get thousands and thousands of lines big to find at times completely trivial bugs that any half decent language would have caught as a syntax error.
My exp in BigTechCo makes me completely understand GitHub actions, my guess is many big companies have equally janky tools that harm dev productivity but still somehow take absolutely massive workloads. GitHub just thought of sharing it to the public.
If you wanted a better version of GitHub Actions/CI (the orchestrator, the job definition interface, or the programming of scripts to execute inside those jobs), it would presumably need to be more opinionated and have more constraints?
Who here has been thinking about this problem? Have you come up with any interesting ideas? What's the state of the art in this space?
GHA was designed in ~2018. What would it look like if you designed it today, with all we know now?
I've been working for the last 5 years on an alternative called Dagger. Ended up building a company around it.
We started from the ideal state of CI, and set out to build the platform that would support that.
For us this ideal state of CI boils down to 4 things:
- local-first (local execution should be a first-class citizen, with no exception)
- repeatable (the same inputs should yield the same output, with affordances for handling inevitable side effects in an explicit and pragmatic way)
- programmable. my workflows are software. I want all the convenience of a modern software development experience: types, IDE support, a rich system API , debugging tools, an ecosystem of reusable components, etc.
- observable. I want all the information in one place about everything that happened in my workflow, with good tooling to get the information I need quickly, and interop with the existing observability ecosystem (eg. open telemetry)
So Dagger is our best effort at a CI platform focused on those 4 things.
Sorry if this comes across as a sales pitch. When you're building solves a problem you're obsessed with, it's hard to discuss the problem without also mentioning the solution that seems the most obvious to you :)
I've been watching Dagger with great interest, although have not moved production workloads to it (nor, admittedly, even committed an afternoon to setting up any workflows/graphs).
Passive comment readers should be aware that ^shykes here cofounded Docker (my gratitude), so it's really worth a look.
Can anyone comment on the ergonomics of Dagger after using it for a while?
I was just looking at the docs earlier this week to consider a migration but got confused by the AI sections...
> got confused by the AI sections...
You're not the only one... At some point last year, we discovered that CI/CD workflows and so-called "AI agent workflows" have a lot in common, and Dagger can in theory be used as an execution engine for both. We attempted to explain this - "great for CI/CD and for Agents!". But the feedback was mostly negative - it came across as confusing and lacking focus. So, we are rolling back this messaging and refocusing on CI/CD again. If you haven't checked our docs in the latest 12 hours, it's worth checking again: you'll see clear signs of this refocusing (although we are not done).
In doubt, I recommend joining our public discord server (https://discord.com/invite/dagger-io) it is basically a support group for CI/CD nerds who believe that a better way is possible, and want to discuss it with like-minded people.
Thanks for the kind words!
Do you have to use discord? All that information is locked away in a vendors system. Why not choose an open source chat app?
What alternatives would you recommend?
I've been working on this problem for the past couple of years. State of the art:
- local CLI instead of git push to run
- graph-based task definitions with automatic distributed execution, instead of the job/step abstraction
- automatic content-based caching to skip unnecessary executions (happens a lot in CI pipelines)
- container-based runtime (instead of proprietary base images) without using docker directly (too slow)
There are a lot of other ways to improve the developer experience. Happy to chat with anybody interested, I'm dan@rwx.com
What always surprises me in these discussions is how few people have used GitLab CI. What gives? It's far, far superior to GitHub and even existed first.
I found gitlab ci's yaml the smallest of the 3 I've used (gitlab, GitHub actions, CircleCI).
But does gitlab ci have anything for sharing? GitHub actions are built around it, and CircleCI has orbs and contexts.
For example, muse's guide for gitlab involves making your own container and managing the cache yourself (ref: https://mise.jdx.dev/continuous-integration.html#gitlab-ci)
GitHub actions is a couple of lines (ref: https://mise.jdx.dev/continuous-integration.html#github-acti...)
Not sure what you mean by "smallest". With GitLab you create docker images. That means you can easily run them locally and share them. In the example you gave the GitHub actions one looks like more lines of YAML but only works because they made an "action". If they had provided a Docker image then GitLab would be just as easy (and trivial to test locally).
mise already is trivial to test locally though, and I'm not sure I agree that maintaining a CI config + Docker image is just as easy as maintaining a CI config.
Missed my edit window, but the just-docker solution also doesn't do anything for caching, that's still left up to every job, and the `mise install`.
Again, I don't mine gitlab ci, but they have no solution for sharing repeated tasks.
At work we have a bunch of GitHub actions for integration testing, building models, publishing, reporting, and whatnot. It was horrible to maintain and look into whenever something went wrong, so I rewrote all the individual parts in Perl and hooked them together with pipes inside GHA, and it works wonders.
Also, GitHub actions itself just breaks sometimes, which is super annoying. A couple of weeks ago, half of all the macOS runner images broke when using GitHub's caching system. Seems like that would have been caught on GitHub's side before that happened, but what do I know!
> https://github.com/actions/runner/issues/449
> https://github.com/orgs/community/discussions/180160
I've gotten to a point where my workflow YAML files are mostly `mise` tool calls (because it handles versioning of all tooling and has cache support) and webhooks, and still it is a pain. Also their concurrency and matrix strategies are just not working well, and sometimes you end up having to use a REST API endpoint to force cancel a job because their normal cancel functionality simply does not take.
There was a time I wanted our GH actions to be more capable, but now I just want them to do as little as possible. I've got a Cloudflare worker receiving the GitHub webhooks firehose, storing metadata about each push and each run so I don't have to pass variables between workflows (which somehow is a horrible experience), and any long-running task that should run in parallel (like evaluations) happens on a Hetzner machine instead.
I'm very open to hear of nice alternatives that integrate well with GitHub, but are more fun to configure.
Honestly part of the reason why I left my last job was because I had to extensively work with GitHub workflows and actions. Debugging was absolutely hell, especially with long running tasks (~1hr) that would fail with next to no debug or traceability. I offered many times to overhaul the system and make it easier to maintain, but we had no time budget. I now work for 30% less but am substantially happier in life without github. It sounds crazy, but I’m not exaggerating.
Run your own server for GitHub Actions. There is a simple library they have you install so your runner gets registered with your repo. Then you can SSH in whenever a job fails. This lets you fully inspect the state, and to execute one-off commands to test theories. It’s a much faster way to iterate.
I've always found things like AWS codebuild or even just a self hosted bare bones jenkins server far easier to work with. What is the advantage that github actions provide that people put up with it? The feedback seems almost universally negative.
> What is the advantage that github actions provide that people put up with it?
It's the default CI system on github and you get relatively free compute.
It is overwhelmingly less resource-intensive to maintain over time than any Jenkins server. And the config is in the repo (not the Jenkins server's config).
Being free or practically free for many use cases. Especially if you consider multi-CPU builds.
Using jdx/mise in GitHub Actions has been very nice, I recommend it. It's not just a nice task runner, it also manages the dependencies of your tasks (stuff like Python, Node.js, even random tools from GitHub) so that "mise run build" does the exact same thing both locally and in CI. Then the ideal GHA workflow is just "install mise, run this mise task". And not only for building, we have it set up for commit checks too.
Honestly, I wonder if a better workflow definition would just have a single input: a single command to run. Remove the temptation to actually put logic in the actions workflow.
This is how we did things with Jenkins and gitlab runners before, idk why folks would do it differently for GHA.
If you can't run the same scripts locally (minus external hosted service/API) then how do you debug them w/o running the whole pipeline?
I assume you're using the currently recommended docker-in-docker method. The legacy Gitlab way is horrible and it makes it basically impossible to run pipelines locally.
Containers all the way down
GitHub introduced all their fancy GHA apps or callables or whatever they're called for specific tasks, and the community went wild. Then people built test and build workflows entirely in GHA instead of independent of it. And added tons of complexity to the point they have a whole build and test application written in GHA YAML.
This is basically how most other CI systems work. GitLab CI, Jenkins, Buildbot, Cirrus CI, etc. are all other systems I've used and they work this way.
I find GitHub Actions abhorrent in a way that I never found a CI/CD system before...
as usual for Microslop products: it's designed for maximum lock-in
everything is including some crappy proprietary yaml rather than using standard tooling
so instead of being a collection of easily composable and testable bits it's a mess that only works on their platform
It seems more of a cultural issue that -- I'm pretty sure -- predates Microsoft's acquisition of GitHub. I assume crappy proprietary yaml can be blamed on use of Ruby. And there seems to be an odd and pervasive "80% is good enough" feel to pretty much everything in GitHub, which is definitely cultural, and I'm pretty sure, also predates Microsoft's acquisition.
GHA is based on Azure Actions. This is evident in how bad its security stance is, since Azure Actions was designed to be used in a more closed/controlled environment.
> I find GitHub Actions abhorrent in a way that I never found a CI/CD system before...
That's just the good old Microsoft effect, they have a reverse-midas-touch when it comes to actually delivering good UX experiences.
> I think everyone that's ever used GitHub actions has come to this conclusion
This is not even specific to GitHub Actions. The logic goes into the scripts, and the CI handles CI specific stuff (checkout, setup tooling, artifacts, cache...). No matter which CI you use, you're in for a bad time if you don't do this.
That is easy to say on the surface for a simple build script.
How do you orchrestate a full CI/CD pipeline where you need state? You just move that complex logic to another monolithic script with its own problems?
> I think everyone that's ever used GitHub actions has come to this conclusion.
I agree that that should be reasonable but unfortunately I can tell you that not all developers (including seniors) naturally arrive at such conclusion no.
I thought that's how actions are supposed to work. Python is king. Just use the Actions script to feed your variables.
run your ci in docker containers so github actions is just a orchestrator
or try something like https://docs.dagger.io/use-cases#portable-ci
I am not having fun with GitHub Actions right now! Why does everything have to be so hard?
I like being able to run self-hosted runners, that is a very cool part of GitHub Actions/Workflow.
I appreciate all the other advice about limit my yamls to: 1) checkout, 2) call a script to do the entire task. I am already half-way there, just need to knuckle-down and do the work.
I was dismayed that parallel tasks aren't really a thing in the yaml, I wanted to fanout a bunch of parallel tasks and I found I couldn't do it. Now that I'm going to consolidate my build process into a single script I own, I can do the fanout myself.
Jobs run in parallel, so if you used Make you could have one job called 'formatting' calling 'make check-formatting', one called linting calling ''make check-linting', one called 'compiling' calling 'make compile' etc.
As much as I hate GitHub Actions, I prefer it over Jenkins and others, because it is right there. I don't need to go and maintain my own servers, or even talk to a 3rd party to host the services.
Like many things, the tool people reach for is the one that's in the box... As opposed to a trip to the hardware store, getting distracted for a few hours, coming back home and no longer in the mood to work on your project/choore.
I think the root problem here is scope creep on using build tools, configuration management, and ci tools: don't make them do things they are not designed for / weak at. The issue isn't necessarily github ci, which is fine if you keep it simple. But things just getting to the stage where they are not simple anymore.
I've seen this over and over again over the years in projects where things like ant, maven, gradle, puppet, ansible, etc. Invariably somebody tries to do something really complicated/clever in a convoluted way. They'll add plugins, more plugins, all sorts of complex configuration, etc. Your script complexity explodes. And then it doesn't work and you spend hours/days trying to make it do the right thing and fighting a tool that just wasn't designed to do what you are doing well. The problem is using the wrong tool for the job. All these tools have the tendency to evolve into everything tools. And they just aren't good at everything. Just because it has some feature doesn't mean it's a good idea to use it.
The author actually calls this out. Just write a bash script and run that instead. If that gets too complicated pick something else more appropriate to the job. Python, whatever you like. The point here is to pick something that's easy for you to run, test, and debug locally. Obviously people have different preferences here. If you are shoe horning complex fork/join behavior, conditional logic, etc. into a Yaml / CI build, maybe simplify your build. Yaml just isn't suitable as a general purpose programming language.
Externalizing the complex stuff to some script also has the benefit of that stuff still working if you ever decide to switch CI provider. Maybe you want to move to Gitlab. Or somebody decides Jenkins wasn't so bad after all. The same scripts will probably be easy to adapt to those.
One of my current github actions basically starts a vm, runs a build.sh script, stops the vm. I don't need a custom runner for that. I get to pick the vm type. I can do whatever I need to in my build.sh. I have one project with an actual matrix build. But that's about the most complex thing I do with github actions. A lot of my build steps are just inline shell commands.
And obligatory AI comment here, if you are doing all this manually, having scripts that run locally also means you can put claude code/codex/whatever to work fixing them for you. I've been working on some ansible scripts with codex today. Works great. It produces better ansible scripts than me. These tools work better if they can test/run what they are working on.
GHA’s componentized architecture is appealing, but it’s astonishing to me that there’s still seemingly no way to actually debug workflows, run them locally, or rapidly iterate on them in any way. Alas.
fyi i maintain a repo that accidentally tracks github actions cron reliability (https://www.swyx.io/github-scraping) - just runs a small script every hour.
i just checked and in 2025 there was at least 2 outages a month every month https://x.com/swyx/status/2011463717683118449?s=20 . not quite 3 nines.
Would something like this help with the feed back loop?
https://github.com/nektos/act
> i.e. a way that I could testbed different runs without polluting history of both Git and Action runs.
How about writing a separate repo and testing it separately
Keywords: reusable workflow/actions
I tried this but then I need to mock the scripts or other external dependencies to truly test the workflows, became too much pain.
In general, I've never really experienced the issues mentioned, but I also use Gitea with Actions rather than GitHub. I also avoid using any complex logic within an Action.
For the script getting run, there's one other thing. I build my containers locally, test the scripts thoroughly, and those scripts and container are what are then used in the build and deploy via Action. As the entire environment is the same, I haven't encountered many issues at all.
That sounds sane and also completely different from GH Actions.
A lot of the pain of GitHub Actions gets much better using tools like action-tmate: https://github.com/mxschmitt/action-tmate
As soon as I need more than two tries to get some workflow working, I set up a tmate session and debug things using a proper remote shell. It doesn't solve all the pain points, but it makes things a lot better.
Tmate is not available anymore, and will be fully decommissioned[0]. Use upterm[1] and action-upterm[2] instead.
Honestly, this should be built into GitHub Actions.
[0] https://github.com/tmate-io/tmate/issues/322
[1] https://upterm.dev/
[2] https://github.com/marketplace/actions/debug-with-ssh
Until I resd this blog I was under the impression that everyone wrote Python/ other files and used Github Actions to just call the scripts!
This way we can test it on local machine before deployment.
Also as other commenters have said - bash is not a good option - Use Python or some other language and write reusabe scripts. If not for this then for the off chance that it'll be migrated to some other cicd platform
The love for Github Actions dissipated fast, it wasn't that long ago we had to read about how amazing Github Actions where. What changed?
I think it made CI management more accessible.
Before that, most people would avoid Jenkins and probably never try Buildbot (because devs typically don't want to spend any time learning tools). Devs would require "devops" to do the CI stuff. Again, mostly because they couldn't be arsed to make it themselves, but also because it required setting up a machine (do you self-host, do you use a VPS?).
Then came tools like Travis or CircleCI, which made it more accessible. "Just write some kind of script and we run it on our machines". Many devs started using that.
And then came GitHub Actions, which were a lot better than Travis and CircleCI: faster, more machines, and free (for open source projects at least). I was happy to move everything there.
But as soon as something becomes more accessible, you get people who had never done it before. They can't say "it enables me to do it, so it's better than me relying on a devops team before" or "well it's better than my experience with Travis". They will just complain because it's not perfect.
And for the OP's defense, I do agree that not being able to SSH into a machine after the build fails is very frustrating.
We used it.
And we realized that the bs sales/marketing material was just as bs as always.
I think it's possible to both think GitHub Actions is an incredible piece of technology (and an incredible de facto public resource), while also thinking it has significant architectural and experiential flaws. The latter can be fixed; the former is difficult for competitors to replicate.
(In general, I think a lot of criticisms of GitHub Actions don't consider the fully loaded cost of an alternative -- there are lots of great alternative CI/CD services out there, but very few of them will give you the OS/architecture matrix and resource caps that GitHub Actions gives every single OSS project for free.)
The typical cycle of all technologies:
1) New technology comes out, people get excited
2) People start recognising the drawbacks of the technology
3) Someone else makes an improved version that claims to fix all of the issues. GOTO 1
For the last decade I've been doing my CI/CD as simple .NET console apps that run wherever. I don't see why we switch to these wildly different technologies when the tools we are already using can do the job.
Being able to run your entire "pipeline" locally with breakpoints is much more productive than whatever the hell goes on in GH Actions these days.
I can do that with github actions too? For tests, I can either run them locally (with a debugger if I want), or in github actions. Smaller checks go in a pre-commit config that github action also runs.
Setting up my github actions (or gitlab) checks in a way that can easily run locally can be a bit of extra work, but it's not difficult.
Standard msft absurdity. 8 years later there is still no local gh action runner to test your script before you commit, push, and churn through logs, and without some 3rd party hack, no way to ssh in and debug. It doesn't matter how simple the build command you write is, because the workflow itself is totally foreign technology to most, and no one wants to be a gh action dev.
Like most of the glaring nonsense that costs people time when using msft, this is financially beneficial to msft in that each failed run counts against paid minutes. It's a racket from disgusting sleaze scum who literally hold meetings dedicated to increasing user pain because otherwise the bottom line will slip fractionally and no one in redmond has a single clue how to make money without ripping off the userbase.
This authors problem isn't GitHub it's the fact that they used Rust when they should have used Go. Never would have had this issue
I recently, for Rust targets reasons, have decided to punt GitHub Actions AND GitHub. Opted for Radicle. Had to figure out my own CI.
Doc'd it here: https://revolveteam.com/blog/goa-radicle-ci/
I am surprised that these sort of declarative specs are so popular in certain domains. Essentially you always seem to be putting settings into the ether and hoping they interact with each other in the way you expect.
I prefer an api with documented contracts between its abstractions
Issue is, op is trying to use matrix strategy when with cross-compiling they could avoid it. I have done it for https://github.com/anttiharju/compare-changes (which has nontrivial CI pipelines but they could be a lot simpler for op's needs)
Main issue is Rust. Writing catchy headlines about hating something may feel good, but a lot of people could avoid these pains if
- zig cc gets support for new linker flag that Rust requires https://codeberg.org/ziglang/zig/pulls/30628 - rust-lang/libc gets to 1.0 which removes iconv issues for macos https://github.com/rust-lang/libc/issues/3248
Ah a classic stack overflow flavored answer ala: do it differently so you don't encounter these issues
Does this discredit the answer?
This also helps one to avoid the expensive macos runners
Who doesn’t? I use it with Mise to have a very simple locally tested way or running tasks.
You should try RWX! You can trigger runs from the CLI for way faster feedback loops. The automatic caching is surprisingly good too. https://www.rwx.com/
Thanks for the shoutout!
If anyone has any questions or wants to give it a shot, I work at RWX and would be happy to chat more about it!
First time I see jj being mentioned in a post not about jj. Made me very happy.
I know GitHub Actions won the war, but I think Bitbucket Pipelines are much nicer to work with. They just seem simpler and less fragile.
But almost every company uses GitHub, and changing to Bitbucket isn't usually viable.
The main downside of bitbucket pipelines is bitbucket. And the only significant feature I recall over GitHub Actions is that Pipelines support cron jobs.
Reading the rest of the thread, it looks like Actions also has cron jobs.
Correct.
I avoid actions for these exact reasons unless I can run the exact same build on another host.
And that’s where there’s a Mac Studio that sits sadly in the corner, waiting for a new check in so it has something to do.
I think this is a specific example of a generalized mistake, one that various bits of our infrastructure and architecture all but beg us to make, over and over, and which must be resisted, which is: Your development feedback loop must be as tight as possible.
Granted, if you are working on "Windows 12", you won't be building, installing, testing, and deploying that locally. I understand and acknowledge that "as tight as possible" will still sometimes push you into remote services or heavyweight processes that can't be pushed towards you locally. This is an ideal to strive for, but not one that can always be accomplished.
However, I see people surrender the ability to work locally much sooner than they should, and implement massively heavyweight processes without any thought for whether you could have gotten 90% of the result of that process with a bit more thought and kept it local and fast.
And even once you pass the event horizon where the system as a whole can't be feasibly built/tested/whatever on anything but a CI system, I see them surrendering the ability to at least run the part of the thing you're working on locally.
I know it's a bit more work, building sufficient mocks and stubs for expensive remote services that you can feasibly run things locally, but the payoff for putting a bit of work into having it run locally for testing and development purposes is just huge, really huge, the sort of huge you should not be ignoring.
"Locally" here does not mean "on your local machine" per se, though that is a pretty good case, but more like, in an environment that you have sole access to, where you're not constantly fighting with latency, and where you have full control. Where if you're debugging even a complex orchestration between internal microservices, you have enough power to crank them all up to "don't ever timeout" and attach debuggers to all of them simultaneously, if you want to. Where you can afford to log every message in the system, interrupt any process, run any test, and change any component in the system in any manner necessary for debugging or development without having to coordinate with anyone. The more only the CI system can do by basically mailing it a PR, and the harder it is to convince it to do just the thing you need right now rather than the other 45 minutes of testing it's going to run before running the 10 second test you actually need, the worse your development speed is going to be.
Fortunately, and I don't even how exactly the ratio between sarcasm and seriousness here (but I'm definitely non-zero serious), this is probably going to fix itself in the next decade or so... because while paying humans to sit there and wait for CI and get sidetracked and distracted is just Humans Doing Work and after all what else are we paying them for, all of this stuff is going to be murder on AI-centric workflows, which need tight testing cycles to work at their best. Can't afford to have AI waiting for 30 minutes to find out that its PR is syntactically invalid, and can't afford for the invalid syntax to come back with bad error messages that leave it baffled as to what the actual problem is. If we won't do it for the humans, we'll do it for the AIs. This is definitely not something AI fixes, despite the fact they are way more patient than us and much less prone to distraction in the meantime since from their "lived experience" they don't experience the time taken for things to build and test, it is made much worse and more obvious that this is a real problem and not just humans being whiny and refusing to tough it through.
Yes exactly! I am using Nix/Make and I can prompt Claude to use Nix/Make, it uses these local feedback loops and corrects itself sometimes etc.
There's no excuse to complain about bad software these days. Fork Gitea, and make your own GitHub actions, it should take a day with coding agents.
Can't you just have a good ole git repo in a server, use hooks to sync with github, and more hooks when you need 'actions' TM (C) (R)
> I can’t overstate how much I hate GitHub Actions. I don’t even remember hating any other piece of technology I used.
Ansible. Same reasons but 100x worse.
I still do not understand how anybody can set up real-life CI on Github? Have they ever looked into real CI solutions?
Of course avoid Actions if you expect tasks to complete promptly or ever. Have we forgotten safe_sleep.sh? I don't think it was unique.
> I don’t even remember hating any other piece of technology I used
Jira, Microsoft Teams, HP Printers, Ticketmaster, …
This is very validating as someone just trying out Actions and getting frustrated with the less than ideal UX.
Is there still no way to ssh into an instance? If not, then github actions needs to work on that.
For those who have experience, how does github actions compare to azure devops pipelines?
I have spent the majority of my professional career waiting for ci to complete. It is hell.
This feels like a rage bait.
So the article is about the frustrating experience of fixing GitHub Actions when something goes wrong, especially when a workflow only fails on one platform, potentially due to how GitHub runner is set up (inconsistently across platforms).
Took me a while to figure that out. While I appreciate occasional banters in blog articles, this one seems to diverge into rant a bit too much, and could have made its point much clearer, with, for example, meaningful section headers.
> Even though my user base can be counted on a fingers of one-arm-less and second-arm-hook-equipped pirate, it’s still a thing “One Should Do”.
No. It's cargo cult science.
I wouldn't say that, but I would say there's no "should" here; it's often much more hassle than people expect and everyone has to decide for themselves whether the number of users is worth it.
I agree. Why are you building for platforms you don't even use?
As a PM trying to understand what was happening, it took me a minute. Turning this into an analogy so that it might benefit other non-too-techies:
Original plan: You have an oven in another building that automatically bakes your cake. But the oven needs a mixer, and every time you bake, it has to wait 2-3 minutes for someone to bring the mixer from storage.
Problem: For one specific oven (Linux ARM), the mixer never arrives. So your cake fails. You keep trying different ways to get the mixer delivered. Each attempt: 2-3 minutes wait.
What you finally do: Stop waiting for the mixer to be delivered. Just mix the batter at home where you already have a mixer. Send the pre-mixed batter to the other building. Now the oven just bakes it - no waiting for the mixer.
Translation: Stop trying to generate files in GitHub Actions (where it takes 2-3 minutes each time). Generate them locally on your computer where you already have the tools. Upload the finished files. GitHub Actions just uses them.
Sometimes "pre-mix the batter at home" beats "wait for the mixer every single time."
This happens when there’s too much logic in the build server.
I take GitHub Actions over any of the other CI platforms.
Slap your problem in an agentic loop, this becomes the following single step:
1. Here's your goal "...", fix it, jj squash and git push, run gh pr checks --watch --fail-fast pr-link
Anyone use Argo Workflows as an alternative?
>Now of course, in some Perfect World, GitHub could have a local runner with all the bells and whistles.
Not by GitHub, but isn't act supposed to be that?
https://github.com/nektos/act
Prefacing this with the fact that act is great, however, it has many shortcomings. Too often I've run into roadblocks, and when looking up the issue for it, it seems they are hard to address. Simpler workflows work fine with it, but more complex workflows will be much harder.
Don't put your logic in proprietary tooling. I have started writing all logic into mise tasks since I already manage the tool dependencies with mise. I tend to write them in a way where it can easily take advantage of GHA features such as concurrency, matrixes, etc. But beyond that, it is all running within mise tasks.
act is often mentioned as a drop-in replacement but I never got it to replicate GitHub actions environment. I didn't try it for this particular case, though.
“ For the love of all that is holy, don’t let GitHub Actions manage your logic. Keep your scripts under your own damn control and just make the Actions call them!”
This is why I started DSCI - https://github.com/melezhik/DSCI
One can replace GitHub actions yaml pipeline runner by general purpose languages/scripts but still using all other standard features of GitHub server
Interesting, but the opening line threw me off:
> Sure, I still make fun of the PHP I remember from the days of PHP 4.1, but even then I didn’t hate it. (ref. And “PHP: Training Wheels Without a Bike” is still in the Top 10 of my favorite memes.)
I still use it to build my latest projects, and for me it’s like breathing. Simple and functional. It’s not perfect, but no language is.
Yeah, no reason for PHP to catch strays here, especially if his knowledge is still based on 4.1. It's really a whole new language compared to back then.
> A word of explanation. I’m building tmplr for 4 platforms:
Oh, here we go again. Java was invented to solve that, "Write once run everywhere" :] I.e. `int` means `i32` on all platforms, no `usize`.Java has numerous problems, it's not as simple as use Java not Rust. To be honest I actively avoid any tooling written in Java.
jAvA BaD!!1
Skill issue.
To some extent I do agree that is sounds like "my build was failing on a platform I had never tested, and I was pissed because I had to debug it".
But it is true that GitHub Actions don't make it easy to debug (e.g. you can't just SSH into the machine after the build fails). Not sure if it justifies hating with Passion, though.
Care to elaborate?
Totally agree.
Agreed
I was so lonely in this opinion for so long, and it is great to see it becoming mainstream. GHA is terrible.
Skill issue bro: https://github.com/nektos/act
just came here to say same, they are the absolute worst
skill issue bro https://github.com/nektos/act
[flagged]
is it your contention that once anybody becomes sufficiently skillful with a technology they will come to love it? And thus stating that one does not love the specific technology demonstrates the lack of skill?