IMO, a scripting language is the lesser contributor to the power of shell scripting. My ranked-order is:
1. Universal Inter-Process Interface: Arbitrary program composition via the Standard Input / Output / Error interface, and "everything is a file" abstraction... None of the rest is possible without some such core design decision ("everything is an object" is the fraternal twin).
2. Widespread standard utilities: The broad availability of core util tools designed to cooperate via the same abstraction is a close second. Along with some default scripting language, that is also built for the same computing model.
3. Language of choice: Important for ease of access, but a distant third in terms of raw power, be it bash, perl, awk, node, clojure (babashka) etc. That's icing on the cake, because if one had no choice in the matter, the shell program would still be awesomely powerful. Proof is in the pudding---"scripts that matter" are always /bin/sh.
Generally speaking, the pain of learning to avoid sh and bash footguns is primarily a matter of some basic research reading: manuals, lore, and warnings (I love BashPitfalls). Well, either that, or learning the hard way by copy-pasting code, and if luck runs out, rm -rf ing something you shouldn't have.
There is no substitute to taking the day or so to read the manual, so you know it's a lot, but it ain't magic, and you can eat that elephant one bite at a time, at your own pace. Seriously... the wall-clock time of trial and error with BashPitfalls (about a day or so), is nothing compared to learning to use---and then having to re-learn every so often---the python or node or whatever ecosystem without losing it. Besides, the former knowledge is stable and so are the programs --- my Bash code will keep working the same way, forever, practically speaking.
I’m going to say something … and it might trigger folks
Learning and getting good at shell scripting is something that takes a while usually. I would recommend that people new at shell scripting actually try to use Python instead writing shell scripts where possible.
I’ve seen people not use traps, not know when to use set properly, xargs if you don’t know what you’re doing with it is just a giant foot cannon, and folks new at shell scripting don’t know how to debug running shell processes.
They may have learned some or all of these things for python?(please? )
But you gotta learn about processes, signals, job control, traps, exit statuses, pipelines, file descriptors and all those things though. Shell is still a great glue language that helps with that. Learning Python does not really replace knowing how the shell and Unix process model work. It could, but try translating some very short shell commands into Python and see how much more machinery you need!
> They may have learned some or all of these things for python?(please? )
On other languages those things are either obvious because you have to write the behavior yourself, or fail loudly in a way that requires you to learn them before your code will run.
You don't have to hope. The same people that fall to the foot-guns in sh don't make the same mistakes when they write it in python.
I was pretty close in guessing what they believe was "wrong" (or really, superfluous) with their initial shell script example.
I'd even drop the "if" from their footnote version, and go with simply
npm install || (echo "boom" && exit)
(Not exactly equivalent because there won't be an "exit" if echo fails, but it's my preferred instict over ";") Also, "(" starts a subshell, if we are being pedantic.
I wonder if there’s some obscure syntax for "do this if the preceding command fails and propagate its exit code". For just returning some error you can of course have something like `die("boom")` (though not as a builtin?) but sometimes it’s useful to know the exact code.
test 1 == 2 && echo "ok" || (error_code="$?"; echo "failed with '$error_code'"; exit "$error_code")
When command (test 1 == 2) fails, it goes to the "or" section, sets the error_code variable, does something (echo command, but it could be anything), and then re-exits with the same exit code.
It's not so obscure, there is just no native "re-raise" function so you have to record the exit code yourself (actually, bash's built-in exit will return the last command's exit code, but in the example above, that's the "echo" which actually succeeds).
Edit: since the above runs it in a subshell, $error_code is not available outside the parentheses but if you need this, you can try:
test 1 == 2 && echo "ok" || error_code="$?" && echo "failed with '$error_code'"
exit $error_code
if table --warble farble; then :; else table_ec=$?; echo "Garble warble farble! :(" >&2; exit $table_ec; fi
Works in dash/bash/zsh at least; not 100% sure whether the behavior of $? on the command in an if test is mandated by POSIX and/or portable. Note that you can't replace the empty then-branch with a negated condition because the negation also eats the return code.
Also, absolutely correct propagation is essentially impossible (but usually not useful anyway), because some errors might be unexpected signal exits or inability to find the command in the first place, and those will get punned onto other nonzero exit codes. I forget the exact conventions OTTOMH but generally they're in the high half (≥128).
Signals get a 128+n, with n being the signal code.
Where texinfo is available, "info bash" is a great reference; "man bash" is everything in a single page — so less manageable — but you can look for BUILTIN section, which documents "exit" built-in function (among other things).
On the higher level question in the article of why do people not use their main languages for basic file and command management?
I believe the answer is the same as to why Python remains hugely popular despite all the well-documented shortcomings: batteries included.
Shell languages include so many neat and concise ways to execute commands, deal with their input, output and exit codes, allow simple traversal of deep directory structures with globs and advanced tools like 'find' or 'xargs', that everything else is very unergonomic.
And niche by default (until it gets used widely enough), so even your LANG-OF-CHOICE-expert you recruit, they will not know a thing about that shell-focused library in LANG-OF-CHOICE.
Yes, I would say exactly that. Other languages are of course capable of doing all the things that shell does, but in my experience they really suck at doing it in anything resembling an ergonomic and concise way.
Right, and I would also argue that scripting goes way beyond the terminal.
Scripting in the terminal can manipulate windows too (e.g. xdg-open with KWin), or send notification locally (e.g. notify-send) to any devices (again or ntfy), or open dialogs (e.g. kdialog), or controlling other devices (e.g. curl to HomeAssistant API) so indeed it opens up an entire new world of possibilities... and it's combinatorial.
Also for another bit of context, scripting works ... on the weirdest newest devices, e.g. a Meta Quest on Android thanks to Termux. If there is an OS there is surely a terminal in there somewhere, and if there is, you can script on it. So... scripting isn't "just" for your desktop or a remote server, it's everywhere.
100%!
I use ntfy to send commands from within scripts to my android devices. Tasker watches for new ntfy notifications and then executes actions based on them to do things like volume changes, Don't disturb, sending OTP codes, triggering a syncthing resync, etc.
I was sure this article would be about how we tell computers to do things now (writing prompts to an AI in plain English). I was wrong. Interesting nonetheless.
LLMs solved this exact problem. Not perfectly (they can sneak in some malicious commands) but good enough for everyone. This is also mostly what non-developers mean by "vibe-coding". They can "tell the computer to do the things" they want. Previously they were limited to existing GUI applications. Now they can automate and customize their workflows as they wish and think they're a 'coder'.
That's a huge part of the appeal. If you're really lazy you can tell an LLM to add a specific column to a specific database, and it will handroll some SQL just for you and run it.
It's slow and very inefficient, but it works.
Why do this? It would be insane for production, but for development it frees you up to improvise and experiment. You can iterate strategies and solutions far more quickly, test as you go, and bake a final production version into static code.
TIL that the duckdb REPL supports reading json files (https://duckdb.org/docs/current/data/json/overview). This might actually be useful the next time I need to investigate a massive json file (`jq` can only go so far).
That philosophical point about what it even means to "tell a computer to do things" is a good one. Sure, clicking around in a GUI is also telling the computer what to do. But there's a huge difference between using someone else's abstraction and building your own from scratch. The article is about that second part - the ability to throw stuff together however you want. With a GUI, you're stuck with whatever the designer thought of. With a shell, you can build just about anything
But you’re still limited by the designer of that shell? Or userspace? Or kernel? Or cpu architecture? I mean, how deep should you go before «it’s however YOU want»?
There are some really powerful UIs that are extremely flexible, like spreadsheets, or ComfyUI, or certain issue trackers. On the whole however I would say they all still are mostly their own interaction silos and integrations with other programs have to be explicitly built for each app.
The great thing about a shell is that you can glue together programs that don't know about each other. This is not a silver bullet, with shell programs being text oriented etc., but it does give rise to some powerful possibilities that don't really work with UIs.
Totally agree, and also this was one of the most powerful aspects of using nix development environments: when everyone is guaranteed to be on all the same (recent) versions of bash, make, awk, sed, and any other system deps, you can confidentially write your scripts knowing the whole team can use them, without having to deal with Mac users’ scripts breaking because their bash is old, or because somebody doesn’t have jq installed, or whatever.
I also assumed this was going to be an intro to using AI as an interface for your local device. I love terminal use and scripting, but after experiencing what it's like to tell an AI to do something in plain language and it does it I'm also excited about that future. There won't be a need to memorize obscure "commands" or syntax when an AI can directly do what you ask or even provide a script for you. We might might occasionally miss the old days when we had a unique skill set, but progress marches on.
I've learned a surprising amount by watching AI do stuff in the shell, or running an agent in a sandbox and having it tell me what commands to run on the machine I'm debugging.
I disagree on the premise that the author is "using" shell. Often when reading about why the command line / shell is better than GUI, the salient advantage comes down to one thing, scripting. Yet scripting in an IDE, which is a GUI, is a ton easier than in vim, simply because the surface for discovery of actions is much more transparent.
If scripting were to be excluded, the belittling of GUIs falls a bit flat. I'd like to see graphic design work be done via CLI.
Your next homework assignment is to write your own scripting language that emits asm. Then you'll be one step closer to understanding what it really means to tell your computer to do things!
Why are we assuming using the terminal is the only way of "telling a computer to do things"? Every way you interact with a machine, you are telling it to do things in theory.
For example
> At a fundamental level, I wasn’t able to tell my computer to accomplish anything that involved logic or stitching together multiple programs
How can that be true if you are telling it to boot on into a os? It doesn't involve logic? Or stiching together multiple programs?
Using a shell =\= telling a machine what to do on a broad level. You are just using the abstractions provided by the os. If you mean get better at shell so you can use it more efficiently than okay cool but that doesn't someone who uses a GUI doesn't "tell" the machine what to do as much as you.
IMO, a scripting language is the lesser contributor to the power of shell scripting. My ranked-order is:
1. Universal Inter-Process Interface: Arbitrary program composition via the Standard Input / Output / Error interface, and "everything is a file" abstraction... None of the rest is possible without some such core design decision ("everything is an object" is the fraternal twin).
2. Widespread standard utilities: The broad availability of core util tools designed to cooperate via the same abstraction is a close second. Along with some default scripting language, that is also built for the same computing model.
3. Language of choice: Important for ease of access, but a distant third in terms of raw power, be it bash, perl, awk, node, clojure (babashka) etc. That's icing on the cake, because if one had no choice in the matter, the shell program would still be awesomely powerful. Proof is in the pudding---"scripts that matter" are always /bin/sh.
Generally speaking, the pain of learning to avoid sh and bash footguns is primarily a matter of some basic research reading: manuals, lore, and warnings (I love BashPitfalls). Well, either that, or learning the hard way by copy-pasting code, and if luck runs out, rm -rf ing something you shouldn't have.
There is no substitute to taking the day or so to read the manual, so you know it's a lot, but it ain't magic, and you can eat that elephant one bite at a time, at your own pace. Seriously... the wall-clock time of trial and error with BashPitfalls (about a day or so), is nothing compared to learning to use---and then having to re-learn every so often---the python or node or whatever ecosystem without losing it. Besides, the former knowledge is stable and so are the programs --- my Bash code will keep working the same way, forever, practically speaking.
I’m going to say something … and it might trigger folks
Learning and getting good at shell scripting is something that takes a while usually. I would recommend that people new at shell scripting actually try to use Python instead writing shell scripts where possible.
I’ve seen people not use traps, not know when to use set properly, xargs if you don’t know what you’re doing with it is just a giant foot cannon, and folks new at shell scripting don’t know how to debug running shell processes.
They may have learned some or all of these things for python?(please? )
But you gotta learn about processes, signals, job control, traps, exit statuses, pipelines, file descriptors and all those things though. Shell is still a great glue language that helps with that. Learning Python does not really replace knowing how the shell and Unix process model work. It could, but try translating some very short shell commands into Python and see how much more machinery you need!
My 2 cents. :)
> They may have learned some or all of these things for python?(please? )
On other languages those things are either obvious because you have to write the behavior yourself, or fail loudly in a way that requires you to learn them before your code will run.
You don't have to hope. The same people that fall to the foot-guns in sh don't make the same mistakes when they write it in python.
I was pretty close in guessing what they believe was "wrong" (or really, superfluous) with their initial shell script example.
I'd even drop the "if" from their footnote version, and go with simply
(Not exactly equivalent because there won't be an "exit" if echo fails, but it's my preferred instict over ";") Also, "(" starts a subshell, if we are being pedantic.I wonder if there’s some obscure syntax for "do this if the preceding command fails and propagate its exit code". For just returning some error you can of course have something like `die("boom")` (though not as a builtin?) but sometimes it’s useful to know the exact code.
It's not so obscure, there is just no native "re-raise" function so you have to record the exit code yourself (actually, bash's built-in exit will return the last command's exit code, but in the example above, that's the "echo" which actually succeeds).
Edit: since the above runs it in a subshell, $error_code is not available outside the parentheses but if you need this, you can try:
if table --warble farble; then :; else table_ec=$?; echo "Garble warble farble! :(" >&2; exit $table_ec; fi
Works in dash/bash/zsh at least; not 100% sure whether the behavior of $? on the command in an if test is mandated by POSIX and/or portable. Note that you can't replace the empty then-branch with a negated condition because the negation also eats the return code.
Also, absolutely correct propagation is essentially impossible (but usually not useful anyway), because some errors might be unexpected signal exits or inability to find the command in the first place, and those will get punned onto other nonzero exit codes. I forget the exact conventions OTTOMH but generally they're in the high half (≥128).
Signals get a 128+n, with n being the signal code.
Where texinfo is available, "info bash" is a great reference; "man bash" is everything in a single page — so less manageable — but you can look for BUILTIN section, which documents "exit" built-in function (among other things).
On the higher level question in the article of why do people not use their main languages for basic file and command management?
I believe the answer is the same as to why Python remains hugely popular despite all the well-documented shortcomings: batteries included.
Shell languages include so many neat and concise ways to execute commands, deal with their input, output and exit codes, allow simple traversal of deep directory structures with globs and advanced tools like 'find' or 'xargs', that everything else is very unergonomic.
And niche by default (until it gets used widely enough), so even your LANG-OF-CHOICE-expert you recruit, they will not know a thing about that shell-focused library in LANG-OF-CHOICE.
Yes, I would say exactly that. Other languages are of course capable of doing all the things that shell does, but in my experience they really suck at doing it in anything resembling an ergonomic and concise way.
Right, and I would also argue that scripting goes way beyond the terminal.
Scripting in the terminal can manipulate windows too (e.g. xdg-open with KWin), or send notification locally (e.g. notify-send) to any devices (again or ntfy), or open dialogs (e.g. kdialog), or controlling other devices (e.g. curl to HomeAssistant API) so indeed it opens up an entire new world of possibilities... and it's combinatorial.
Also for another bit of context, scripting works ... on the weirdest newest devices, e.g. a Meta Quest on Android thanks to Termux. If there is an OS there is surely a terminal in there somewhere, and if there is, you can script on it. So... scripting isn't "just" for your desktop or a remote server, it's everywhere.
100%! I use ntfy to send commands from within scripts to my android devices. Tasker watches for new ntfy notifications and then executes actions based on them to do things like volume changes, Don't disturb, sending OTP codes, triggering a syncthing resync, etc.
I was sure this article would be about how we tell computers to do things now (writing prompts to an AI in plain English). I was wrong. Interesting nonetheless.
LLMs solved this exact problem. Not perfectly (they can sneak in some malicious commands) but good enough for everyone. This is also mostly what non-developers mean by "vibe-coding". They can "tell the computer to do the things" they want. Previously they were limited to existing GUI applications. Now they can automate and customize their workflows as they wish and think they're a 'coder'.
That's a huge part of the appeal. If you're really lazy you can tell an LLM to add a specific column to a specific database, and it will handroll some SQL just for you and run it.
It's slow and very inefficient, but it works.
Why do this? It would be insane for production, but for development it frees you up to improvise and experiment. You can iterate strategies and solutions far more quickly, test as you go, and bake a final production version into static code.
TIL that the duckdb REPL supports reading json files (https://duckdb.org/docs/current/data/json/overview). This might actually be useful the next time I need to investigate a massive json file (`jq` can only go so far).
in what case jq, a specialized json tool, is more limiting than sql?
There is, of course, a programming language designed to do shelly things as succinctly as a shell script: Perl.
Also Scsh[0].
[0]: https://en.wikipedia.org/wiki/Scsh
I was amused to see DeepSeek using it recently, as I didn't realize it was installed on my machine (macos with brew).
Ruby is much, much prettier for similar purposes though, and so that's what I'll learn if I ever need to do this more frequently.
My problem with shell scripting is I've learned it many times, and forgotten it an equal number of times.
Thank God for LLMs.
That philosophical point about what it even means to "tell a computer to do things" is a good one. Sure, clicking around in a GUI is also telling the computer what to do. But there's a huge difference between using someone else's abstraction and building your own from scratch. The article is about that second part - the ability to throw stuff together however you want. With a GUI, you're stuck with whatever the designer thought of. With a shell, you can build just about anything
But you’re still limited by the designer of that shell? Or userspace? Or kernel? Or cpu architecture? I mean, how deep should you go before «it’s however YOU want»?
There are some really powerful UIs that are extremely flexible, like spreadsheets, or ComfyUI, or certain issue trackers. On the whole however I would say they all still are mostly their own interaction silos and integrations with other programs have to be explicitly built for each app.
The great thing about a shell is that you can glue together programs that don't know about each other. This is not a silver bullet, with shell programs being text oriented etc., but it does give rise to some powerful possibilities that don't really work with UIs.
This is the exact problem with the Unix greybeards smug nature: no matter how low level you go you are always on top of some abstractions.
Criticising people just because they don't use your choice of tools is just elitism and leaves a bad taste.
If someone uses a GUI to accomplish their goals that should be complimented not criticised.
You're limited by the time, knowledge, and complexity costs of hand-rolling and testing a solution to your problem.
As soon as commands become even a little non-trivial, those costs mount up.
Totally agree, and also this was one of the most powerful aspects of using nix development environments: when everyone is guaranteed to be on all the same (recent) versions of bash, make, awk, sed, and any other system deps, you can confidentially write your scripts knowing the whole team can use them, without having to deal with Mac users’ scripts breaking because their bash is old, or because somebody doesn’t have jq installed, or whatever.
I also assumed this was going to be an intro to using AI as an interface for your local device. I love terminal use and scripting, but after experiencing what it's like to tell an AI to do something in plain language and it does it I'm also excited about that future. There won't be a need to memorize obscure "commands" or syntax when an AI can directly do what you ask or even provide a script for you. We might might occasionally miss the old days when we had a unique skill set, but progress marches on.
I've learned a surprising amount by watching AI do stuff in the shell, or running an agent in a sandbox and having it tell me what commands to run on the machine I'm debugging.
Good advice. I've gotten a ton of utility out of my shell without learning very many tools particularly well.
If you take nothing else away from this, memorize the syntax of one-liner for loops, it has saved me so many times.
Nice article about actually learning stuff and putting this knowledge to use together. Not with AI but with one’s own brain.
I disagree on the premise that the author is "using" shell. Often when reading about why the command line / shell is better than GUI, the salient advantage comes down to one thing, scripting. Yet scripting in an IDE, which is a GUI, is a ton easier than in vim, simply because the surface for discovery of actions is much more transparent.
If scripting were to be excluded, the belittling of GUIs falls a bit flat. I'd like to see graphic design work be done via CLI.
Your next homework assignment is to write your own scripting language that emits asm. Then you'll be one step closer to understanding what it really means to tell your computer to do things!
Why are we assuming using the terminal is the only way of "telling a computer to do things"? Every way you interact with a machine, you are telling it to do things in theory.
For example
> At a fundamental level, I wasn’t able to tell my computer to accomplish anything that involved logic or stitching together multiple programs
How can that be true if you are telling it to boot on into a os? It doesn't involve logic? Or stiching together multiple programs?
Using a shell =\= telling a machine what to do on a broad level. You are just using the abstractions provided by the os. If you mean get better at shell so you can use it more efficiently than okay cool but that doesn't someone who uses a GUI doesn't "tell" the machine what to do as much as you.
[flagged]