> How to run shell commands in parallel while limiting their concurrency.
Depending upon the task, generate a Makefile and run it with:
make -j <concurrency>
> My original use case had to do with making a large number of web requests
This is in fact the use case that caused me to think about using a Makefile for concurrency probably 20 years ago. A rule for downloading each URL to a file. Then a simple `make -j <jobs>` with whatever concurrency I desired and bonus: automatically skipping already downloaded files. e.g.:
Yeah, and all those concurrency controls of the shell suck. Here's a snippet to run N commands in parallel, wait for all of them, while supporting interrupting them with simple Ctrl+C:
(
trap 'kill 0' SIGINT;
for i in {1..$N}
do
whatever_command $i &
done
wait
)
The parentheses are absolutely crucial there for some reason, the { } won't give you pass-through of SIGINT. No wonder people keep re-inventing PID files, manually tracking $! and all kinds of other crazy bullshit that work 90% of the time, but sometimes fail to properly terminate the runaway children.
I shamelessly use concurrency in Bash shells but typically I do it because I already have a Bash shell and it's now outgrowing its purpose and becoming to slow (say because I begin to use on too much data).
So instead of rewriting everything to another language, I add concurrency to that already existing shell scripts.
xargs / max args / forking to "bash -c" etc.
It's tricky. There are gotchas (with piping for example but also how to behave when one subshell fails, how to cleanly interrupt everything if one fails, etc.).
But it works and it's a cheap way to greatly speed things up.
Also that's the kind of stuff that became not just a bit but much easier now: just give a shell script to a LLM and ask it to add concurrency where needed.
> just give a shell script to a LLM and ask it to add concurrency where needed.
You know, I personally consider the whole LLM revolution a sort of indictment of our whole industry up to this point. We had more than thirty years to figure out and make robust, composable, reusable building blocks, that would not require fiddling and filing away the sharp edges and writing megabytes of gluing boilerplate code — and we failed. Miserably. So now LLMs come in and finally give us the reusability in a very brute-forced kind of way: by being able to regurgitate those pieces of software, slightly adjusted to the context as needed.
So. In a sane world, the shell would already have a bloody "par" builtin that would deal "with piping for example but also how to behave when one subshell fails, how to cleanly interrupt everything if one fails, etc." out of the box, without additional fuss — but we don't. Well, we do have GNU parallel, but frankly, almost nobody knows about, or gives a damn, about anything that's beyond the GNU coreutils. Like, have you heard of e.g. GNU stow? Apparently it's been a thing for decades.
Anyway. We're not living in the sane world, so instead of a simple, robust, widely available built-in tool for parallel execution, we have dozens of little bespoke homegrown systems, cobbled together from xargs, jobs, pid- and lock-files lying around, and goodness knows what else, all of them mostly working 95% of the time but subtly broken in different subtle ways 5% of the time... and so the advent of LLMs which you could just ask "hey, here's the shell script, just parallelize it", and then get the working output in a minute, was a godsend. Of course it was! But that's because most of our software is barely fit for human use and consumption.
> How to run shell commands in parallel while limiting their concurrency.
Depending upon the task, generate a Makefile and run it with:
> My original use case had to do with making a large number of web requestsThis is in fact the use case that caused me to think about using a Makefile for concurrency probably 20 years ago. A rule for downloading each URL to a file. Then a simple `make -j <jobs>` with whatever concurrency I desired and bonus: automatically skipping already downloaded files. e.g.:
Or, to emulate one of the examples in the blog post: Run with: > You do not need to know all the commands that you want executed up-front, you can submit commands to it over time.Ah, yeah, that does put Make in the same bucket as `xargs -P` / GNU `parallel` and requires something more clever.
There is also GNU Parallel's sem: https://www.gnu.org/software/parallel/sem.html
What about using GNU Parallel ?
I guess that works.
I've always been fond of using make for things to get free parallelism / ability to interrupt / restart / incremental updates.
> language with poor concurrency controls
Few other languages have as many builtins/keywords for concurrency control: &, wait, jobs ...
Yeah, and all those concurrency controls of the shell suck. Here's a snippet to run N commands in parallel, wait for all of them, while supporting interrupting them with simple Ctrl+C:
The parentheses are absolutely crucial there for some reason, the { } won't give you pass-through of SIGINT. No wonder people keep re-inventing PID files, manually tracking $! and all kinds of other crazy bullshit that work 90% of the time, but sometimes fail to properly terminate the runaway children.perhaps that's more an indictment of most programming languages
I shamelessly use concurrency in Bash shells but typically I do it because I already have a Bash shell and it's now outgrowing its purpose and becoming to slow (say because I begin to use on too much data).
So instead of rewriting everything to another language, I add concurrency to that already existing shell scripts.
xargs / max args / forking to "bash -c" etc.
It's tricky. There are gotchas (with piping for example but also how to behave when one subshell fails, how to cleanly interrupt everything if one fails, etc.).
But it works and it's a cheap way to greatly speed things up.
Also that's the kind of stuff that became not just a bit but much easier now: just give a shell script to a LLM and ask it to add concurrency where needed.
> just give a shell script to a LLM and ask it to add concurrency where needed.
You know, I personally consider the whole LLM revolution a sort of indictment of our whole industry up to this point. We had more than thirty years to figure out and make robust, composable, reusable building blocks, that would not require fiddling and filing away the sharp edges and writing megabytes of gluing boilerplate code — and we failed. Miserably. So now LLMs come in and finally give us the reusability in a very brute-forced kind of way: by being able to regurgitate those pieces of software, slightly adjusted to the context as needed.
So. In a sane world, the shell would already have a bloody "par" builtin that would deal "with piping for example but also how to behave when one subshell fails, how to cleanly interrupt everything if one fails, etc." out of the box, without additional fuss — but we don't. Well, we do have GNU parallel, but frankly, almost nobody knows about, or gives a damn, about anything that's beyond the GNU coreutils. Like, have you heard of e.g. GNU stow? Apparently it's been a thing for decades.
Anyway. We're not living in the sane world, so instead of a simple, robust, widely available built-in tool for parallel execution, we have dozens of little bespoke homegrown systems, cobbled together from xargs, jobs, pid- and lock-files lying around, and goodness knows what else, all of them mostly working 95% of the time but subtly broken in different subtle ways 5% of the time... and so the advent of LLMs which you could just ask "hey, here's the shell script, just parallelize it", and then get the working output in a minute, was a godsend. Of course it was! But that's because most of our software is barely fit for human use and consumption.