I love this! I have done a lot of CS research lately, and some of these I haven’t come across yet.
Let me share some of my favourites not listed here, off the top of my head:
- Ian Piumarta’s “Open, Extensible Object Models” (https://www.piumarta.com/software/id-objmodel/objmodel2.pdf) is about creating the most minimal object-oriented metaobject system that allows the maximum amount of freedom for the programmer. It basically only defines a message send operation, everything else can be changed at runtime. The practical counterpart to the dense “Art of the Metaobject Protocol” book.
- John Ousterhout “Scripting: Higher-Level Programming for the 21st Century” (https://web.stanford.edu/~ouster/cgi-bin/papers/scripting.pd...) - not really a paper, but an article from the creator of Tcl about the dichotomy between systems programming languages and scripting languages. Obvious at first sight, the lessons therein have wide ramifications IMO. We always seek the perfect multi-paradigm language that can do anything at high performance with the most productivity, while perhaps it is best to have compiled, fast, clunky systems languages paired with ergonomic, flexible interpreted frontend. Often all you need is C+Tcl in the same app. A must-read for anyone writing yet another programming language.
- Niklaus Wirth's Project Oberon (https://people.inf.ethz.ch/wirth/ProjectOberon/) is the implementation of an entire computer system, from the high-level UI down to kernel, compiler, and a RISC-like CPU architecture. He wrote the seminal "plea for lean software" and actually walked the walk. A long lost art in the era of dependency hell and towering abstractions from mediocre coders.
Hmm, I disagree with Ousterhout's dichotomy and conclusions.
First, my understanding of his points - a language is either a systems language like C or a scripting language like TCL or python. Systems language have "strong" types and are for data structures/algorithms, scripting languages are "typeless" and are for "gluing things together".
The main claim is that scripting languages are more concise and allow for faster development when gluing due to their 'typeless' nature.
With C++
and Microsoft Foundation Classes (MFC), it requires
about 25 lines of code in three procedures. 1 Just set-
ting the font requires several lines of code in MFC:
CFont *fontPtr = new CFont();
fontPtr->CreateFont(16, 0, 0, 0, 700,
0, 0, 0, ANSI_CHARSET,
OUT_DEFAULT_PRECIS,
CLIP_DEFAULT_PRECIS,
DEFAULT_QUALITY,
DEFAULT_PITCH|FF_DONTCARE,
“Times New Roman”);
buttonPtr->SetFont(fontPtr);
Much of this code is a consequence of the strong typ-
ing[...] In Tcl, the essential characteristics of the
font (typeface Times, size 16 points) can be used
immediately with no declarations or conversions.
Furthermore, Tcl allows the button’s behavior
to be included directly in the command that cre-
ates the button, while C++ and Java require it to
be placed in a separately declared method.
End quote.
We've come a long way, and examples have made clear that this dichotomy is a false one. Ousterhout's view was colored by his limited experience, resulting in him misunderstanding what he actually likes about Tcl.
Let's talk about syntax. His exact Tcl code as presented could be parsed and compiled by a language that does static type analysis. It's not, because he's running it in an interpreter that only checks types at runtime. But the point is that whether code is compiled or interpreted is an implementation detail that has very little to do with the syntax. He likes his syntax because his syntax is obviously better than C++, nothing more.
And types: he claims that 'typeless' languages allow for faster development because of fewer restrictions. This is, ofc, nonsense. The amount of restrictions you have is a function of your problem, not your language. If it feels like dynamic typing is letting you develop faster, that's because you're putting off encountering those restrictions til later.
It's better to guarantee we encounter all type errors before the program even runs. But since you can do static type analysis for any language, why don't all languages do that?
Because it's hard. Type theory is complicated. Not all type systems are decidable, we need to pick one that is for practical reasons. Ones that are may require annotations or complex algorithms/semantics restrictions like Hindley-Milner.
As a PL designer, it's a whole lot easier to just give up and only check types at runtime. And that makes a whole lot of sense if your priority is just embedding a usable language ASAP. But let's not pretend that it's because it's actually better.
I opine that typeless languages are best for short programs, because one can understand 100% of the program. Typed languages are a way to manage the complexity of a larger program.
For a related example, the BASIC programming language is great for programs that fit in 24 lines (which is how many lines a glass tty displayed).
There's clearly a cost-benefit tradeoff on the spectrum of static typing. We aren't all using Coq. OTOH, the burden of "basic" static typing is very low at this point given the successes we've seen with type inference. [0]
It seems like all of the dynamically typed programming languages are pursuing (optional/gradual) static typing systems, and we've mostly moved on past "pure" dynamically typed PLs at this point. It's more about how-much and what-kind of static typing. And perhaps this depends upon your domain (e.g. Rust and its focus on memory safety while eschewing GC, which necessitates lifetime annotations).
0) I don't think type inference is a panacea. For example, it's not super helpful when you have a complex type system inferring crazy types for you, and you're stuck debugging them when the inferrer has failed.
Having functions that fit on one screen is still a good idea I think.
My point was that printing, teletype-style terminals don't have an intrinsic vertical space limit like video display terminals (and eventually the printouts can get quite long).
But now that I think about it, perhaps the designers of BASIC might have been thinking of making it easy for a beginner to spend an afternoon or two learning how to write something comparable to a short FORTRAN program that might fit on a small deck of 80 column (72 with sequence number) punched cards.
I agree, my rule of thumb is > 1k LoC is best built with statically typed languages. For 20 line programs that I don't plan to maintain for a long time, I'd hardly ever reach for Rust over Python.
Static typing lets you build very large rigid structures where types ensure constraints. However, these constraints get in the way of moving quickly. If you change a signature of a function, you have to first change every piece of code that calls the function. You then have to compile, restart your program, rebuild the whole rigid structure, and confirm the change works as expected. This whole process can take many minutes. The claim is that catching these errors at compile time results in more reliable code.
Dynamic typing has no such constraints to help you catch errors at compile time. Dynamic typing allows you to build organic structures that more fluidly change. If you change a function signature, you don't have to change every piece of code that calls the function. You can change the function and test it in a running application without restarting the application. Doing this takes seconds. Dynamic typing allows you to iterate 100x faster. Being able to rapidly iterate often means better tested code and more reliable code.
Both are valuable. Dynamic typing is valuable during development when you need to iterate rapidly and build a prototype when requirements are often changing. Static typing is valuable when code need is more mature, changing less frequently, and code needs to be put in production.
Ideally you have both. More mature parts of the program are statically typed and more fluid parts of the program are dynamically typed.
> Static typing lets you build very large rigid structures where types ensure constraints. However, these constraints get in the way of moving quickly. If you change a signature of a function, you have to first change every piece of code that calls the function.
I strongly disagree that this sort of refactoring is "quicker" with dynamically typed languages, particularly with non-trivial (i.e. large) code bases.
You have zero feedback and no indication of how disruptive such a change is when working with a dynamically typed language. In fact, you have no feedback on whether you've actually finished the change at all.
While with static type checking, you have immediate and valuable feedback guiding you through such a change. Each line of code which requires updating is identified for you and communicated to you.
Spending my time doing work the compiler should have done for me (including tracking down heisenbugs) has wasted far more of my time than I’ve ever gained from the lax enforcement of dynamic typing.
> You then have to compile, restart your program, rebuild the whole rigid structure, and confirm the change works as expected. This whole process can take many minutes.
This is not my experience at all. Virgil is a fully statically-typed language and compiles fast. When working on the Virgil compiler (approximately 50KLOC), my typical workflow is to run the compiler in the interpreter in my debug, edit, run loop. The interpreter can load the entire compiler, typecheck it, and run it with my changes in 90 milliseconds. If we throw a complete optimized compile in the loop, it's 400 milliseconds.
It takes me longer to switch between my two terminal tabs.
It's not static typing versus dynamic typing. It's ridiculously inefficient languages/build systems versus efficient ones.
Inefficient languages/build systems can make the problem worse.
There are many statically typed languages that can compile in under a second. But when you change any function signature, you still have to ensure every caller is fixed. You then have to recompile, restart the application and rebuild all the data structures. Some static languages allow for hot reloading in interpretive mode, but the types of changes allowed are limited.
In dynamic languages like Common Lisp and Smalltalk you can update the program while it is running. The speed of iteration is 100x faster, because you don't have to restart the application and rebuild all the data structures to get to the point where you can test your change.
> It's better to guarantee we encounter all type errors before the program even runs.
This is only valid if you either are writing mission-critical software, or have infinite time.
Your argument doesn’t consider the case that you have a deadline and need to ship, so optimising for productivity, rather than asymptotic typing perfection, is paramount. There is a reason even performance-critical environments, where scripting languages are not very well suited, somehow still lean on them for productivity.
Case in point, game dev (Unreal Engine with its blueprint system, Godot with GDScript, the myriad of game engines in C++ paired with Lua.) Of course in a vacuum game devs would like to write ideal data structures with strong typing so that the game doesn’t crash, but their goal is to ship a game within the decade, so limit the strong typing to performance critical engine and can focus on building and iterating on the core product without consulting type theory books.
The point of Mr. Ousterhout’s argument is that there are only two choices: either we invent the panacea, the mythical productive strong-typed language that gives 100% safety yet enables experimentation, optional typing and compiles to C++ speed native code, or we accept that this is an impossible pipe dream, and we need to use the correct tool for the problem. Again, obvious on the surface, but still a contentious point to this day.
>The point of Mr. Ousterhout’s argument is that there are only two choices:
Ultimately, it all compiles down to assembly (which is executed), or an interpreter (which is executed). Assembly all the way. The real choice here is: turn into assembly ahead of time (programmer's code becomes assembly), or do so at runtime (assembly interprets programmer's code). Or some in-between like JIT compilation. And: what guardrails you put in place.
So you could say that higher-level languages are just a way to make tedious/nasty assembly coding more palatable. Eg. by producing error reports vs. just crashing a machine. Or provide a sandbox. Or provide a model of computation that fits well with programmer's thinking.
Between those, you simply need some useful abstractions. Preferably ones that are simple yet versatile/universal.
Eg. a (bitmap) "screen" could just be a flat memory space (framebuffer). Or a 2-D array of pixels, potentially with "pixel" defined elsewhere.
Programmer doesn't care how GPU is coerced into displaying those pixels, low-level code doesn't care what higher-level software does to produce them.
And then have as few as those abstractions as possible (but no less!). Tcl: "everything is a string". Lisp: "everything is a list". Unix(-like): "everything is a file". Etc.
Personally, I have a soft spot for languages that punch above their weight for expressiveness / few but useful abstractions vs. their implementation size. Things like Forth, Tcl or Lua come to mind.
But hey that's just me. Developers & organisations they're in make their own choices.
> The real choice here is: turn into assembly ahead of time (programmer's code becomes assembly), or do so at runtime (assembly interprets programmer's code). Or some in-between like JIT compilation. And: what guardrails you put in place.
A language's preferred compilation/interpretation model is, roughly speaking, orthogonal to its type system and to whether it emphasises speed of development, correctness, or performance. Also, many languages can practically be implemented using various compilation/interpretation models. There are interpreters for C, for instance, and ahead-of-time machine-code compilers for Java.
> So you could say that higher-level languages are just a way to make tedious/nasty assembly coding more palatable.
Well, the point of scripting languages is mostly dynamic typing and dynamic binding, which means resolving bindings at runtime, which means slower in execution. There is no silver bullet: you want speed, you need a clunky language that compiles to fast machine code. If you want ease of use and less ceremony, you’ll get a slower language.
What I've never understood about this argument is this:
How often are you passing around data that you don't fully understand?
Also, people use types and then end up reaching for reflection to perform pattern matching on types at which point you've just moved the typing from the user level to a type system. Not much gained imo.
Most people don't reach for reflection to pattern match on types, because when a type _changes_ you get very nice errors at all places where you've just introduced a bug.
But some languages DO have less restrictions. There's a lot less I have to worry about with Python than the quagmire of Java. You may say I'm putting something off (and maybe I am), but that is perfectly reasonable in a lot of scripting use cases.
Mmmh yes and no. Some languages truly do have more/less restrictions.
For example, a garbage collector is the compiler/runtime taking on more work to avoid the restrictions of proper memory management.
This is in some sense analogous to the C++ vs Rust debate. Borrow checking allows the compiler to statically verify whether the program is memory safe. Like statically verified types, it turns a class of undesirable runtime behavior into a compilation failure. Ofc, this is a restriction that may be undesirable, which is why you can turn it off with 'unsafe'.
Python and Java though... do you really have that much more to worry about in Java?
In my eyes, you've got your problem. Regardless of your language choice, you can write a pseudocode algorithm for it. Terms in your algorithm have types already.
When we translate this algorithm into python/Java, the types of the program need to match the types of our pseudocode. Regardless of whether we get feedback on errors at runtime or compile time, we just need to get the types right. I'd prefer my error at compile time. The benefit is tiny for short scripts, but becomes massive when working on a large code base.
What about Java (being a "systems" language) makes you worry about so much more than Python (the "scripting" language)? There are definitely differences, but I don't think they stem from this systems/scripting dichotomy that's intrinsically linked to whether you check your types at compile or run time.
With Python I can often do some task in like half a page of code with zero boiler plate. Just some loops and a couple of objects and a library call. I don't even write classes most of the time and the state is all global. In Java, there is just sooo much more for the equivalent Python code. Maybe asking an LLM will save some time, but yeah ...there really is a lot more ceremony with Java.
Of course Python has some restrictions due to the garbage collector. Good catch on that as it is if course true and limits you in other areas I don't care about. I was mainly referring to scripting needs where some languages really do shine above others. There's a reason Python is ubiquitous in that area and Java is not. Just like there's a reason Java and C# are ubiquitous in large enterprise software and Python is not.
To summarize though, I think I understand your point, but we're getting lost in semantics haha.
> And types: he claims that 'typeless' languages allow for faster
> development because of fewer restrictions. This is, ofc, nonsense.
> The amount of restrictions you have is a function of your problem,
> not your language.
Unless you're implementing something that's already known, you're effectively carving out your problem when you're in the process of writing code. There are more unknowns than there are knowns. A strongly typed language forces you to focus on satisfying the types before you can do anything, at a point where you don't exactly yet know what you want to do. The right time to cast your data rigidly into types is gradually when your data structures settle in their own place one by one; in other words when you're pretty sure things won't change too easily anymore. This allows writing code that works first and then, depending on your needs, gets next correct and then fast, or next fast and then correct. You use the weakly typed or "typeless" style as a prototyping tool that generates snippets of solid code that you can then typefy and write down for good.
Solutions to any given problem are necessarily described with types. You can check those types at runtime or you can check them at compile time, but meaningful programs are typed. If you've got a function that takes a string and your type system lets you run a program where you pass an int to it... did it just allow you to 'quickly iterate'? Sure, you're running a program sooner, but you're going to run into the same type error (function expects string, not int).
Maybe in a larger code base you could argue that the effort involved in satisfying typechecks meaningfully slows down iteration, but even then it's clear that being able to compute the list of type errors you introduced elsewhere (via static typing semantics/analysis) is valuable.
Languages introduce restrictions of their own all the time. In Java I can't have a function that works on any type of number, I have to specify the type. In Haskell I can. In Haskell I can't put random stuff in a list, in Java I can as long as they are all objects.
Those language restrictions slow down your development as you have to design and code around them. People see the extra design\code as a good trade off because it improves safety, but it slows you down if you are just messing around.
>We've come a long way, and examples have made clear that this dichotomy is a false one.
I wouldn't be so sure. Even Javascript, when is added types in the form of Typescript, looks more like the convoluted Windows API example and less like the TCL example.
>Let's talk about syntax. His exact Tcl code as presented could be parsed and compiled by a language that does static type analysis. It's not, because he's running it in an interpreter that only checks types at runtime. But the point is that whether code is compiled or interpreted is an implementation detail that has very little to do with the syntax.
Pedantically true. You can of course compile anything, including BASIC and TCL, or intepret C++ if you wished so.
But to really take advantage of compilation, languages tend to add types and other annotations or semantic constructs (like Haskell does). And so compiled languages tend to a certain syntax/semantics, and interpreted languages to another.
>And types: he claims that 'typeless' languages allow for faster development because of fewer restrictions. This is, ofc, nonsense. The amount of restrictions you have is a function of your problem, not your language. If it feels like dynamic typing is letting you develop faster, that's because you're putting off encountering those restrictions til later.
So it's not "nonsense", but rather a tradeoff, just like Ousterhout puts it.
For many, it is much faster to "putting off encountering those restrictions til later", as that's a large part of exploring the problem space.
See, while "the amount of restrictions you have is a function of your problem", you don't always know your exact problem, its boundaries, and the form you'll use to solve it, until you've experimented and played with early prototype solutions and changed them, etc. While doing so, having to tackle hard restrictions based on assumptions that can and will change or be discarded, slows you down.
>It's better to guarantee we encounter all type errors before the program even runs.
Only if it doesn't entail other issues, like it making our exploratory programming to end up with a design for our program slower and more tedious.
Love this post. Writing on programming languages has changed how I think about _programming_ in general.
I often think about this quote from TAPL. This framing of “safety” changed how I design systems.
> Informally, though, safe languages can be defined as ones that make it impossible to shoot yourself in the foot while programming.
> Refining this intuition a little, we could say that a safe language _is one that protects its own abstractions_.
> Safety refers to the language's ability to guarantee the integrity of these abstractions and of higher-level abstractions introduced by the programmer using the definitional facilities of the language. For example, a language may provide arrays, with access and update operations, as an abstraction of the underlying memory. A programmer using this language then expects that an array can be changed only by using the update operation on it explicitly—and not, for example, by writing past the end of some other data structure.
Regarding weird development methods of interest...Aaron Hsu of APL fame writes a lot of code in calligraphy with fountain pens when trying to organize his thoughts. I do something kind of similar, but in print with a crummy bic pen and a flow chart of Python objects (kind of like poor man's UML).
I also turn to a fountain pen for the hardest problems. It puts me in a completely different head space. Something about the limited editing ability forces more coherent, linear thought, but also the freedom to seamlessly switch between English, code, math, and diagrams opens up the creativity.
I would also highly recommend watching one of Rich Hickey's talks (especially the earlier ones). Watching those certainly changed how I thought about programming in general.
Skip “Simple made easy” because I cannot stand hearing that talk quoted by basically every single conference speaker in the last decade. It’s become its own cliché.
(Joking of course. I much prefer “Hammock driven development” but it’s not very corporate friendly)
I hacked up a toy brainfuck interpreter using that technique and it was pretty fast. Not sure I'd get the chance to use it elsewhere, but experimenting with it was useful regardless:
I am very curious about the difference between an array of closures and an array of alternating function pointers and data. The latter doesn't come very naturally in most languages, you need something very C like for it to make sense. But my gut feeling says static functions and having access to the data right there in the same array might play better with cache.
It's a shame that Abdulaziz went quiet after moving back to Kuwait. He was our intern on Maxine VM back in 2009. A super nice guy and that paper is a gem!
I like this guy, so nothing against him, but none of these are about PL and all about these are about compilers (except for the one about GC). Which is fine (I like compilers) but they're just not in any way about PL.
I wish someone would write this for higher-level languages: JavaScript or .NET. I'm sure this person is brilliant, but they're operating at a much lower (higher?) level than most of us.
I learnt a lot about bytecode interpreters from working on it, and it helped me understand the cpython source code a lot more easily from having played with a python translation of it first.
I love this! I have done a lot of CS research lately, and some of these I haven’t come across yet.
Let me share some of my favourites not listed here, off the top of my head:
- Ian Piumarta’s “Open, Extensible Object Models” (https://www.piumarta.com/software/id-objmodel/objmodel2.pdf) is about creating the most minimal object-oriented metaobject system that allows the maximum amount of freedom for the programmer. It basically only defines a message send operation, everything else can be changed at runtime. The practical counterpart to the dense “Art of the Metaobject Protocol” book.
- John Ousterhout “Scripting: Higher-Level Programming for the 21st Century” (https://web.stanford.edu/~ouster/cgi-bin/papers/scripting.pd...) - not really a paper, but an article from the creator of Tcl about the dichotomy between systems programming languages and scripting languages. Obvious at first sight, the lessons therein have wide ramifications IMO. We always seek the perfect multi-paradigm language that can do anything at high performance with the most productivity, while perhaps it is best to have compiled, fast, clunky systems languages paired with ergonomic, flexible interpreted frontend. Often all you need is C+Tcl in the same app. A must-read for anyone writing yet another programming language.
- Niklaus Wirth's Project Oberon (https://people.inf.ethz.ch/wirth/ProjectOberon/) is the implementation of an entire computer system, from the high-level UI down to kernel, compiler, and a RISC-like CPU architecture. He wrote the seminal "plea for lean software" and actually walked the walk. A long lost art in the era of dependency hell and towering abstractions from mediocre coders.
Hmm, I disagree with Ousterhout's dichotomy and conclusions.
First, my understanding of his points - a language is either a systems language like C or a scripting language like TCL or python. Systems language have "strong" types and are for data structures/algorithms, scripting languages are "typeless" and are for "gluing things together".
The main claim is that scripting languages are more concise and allow for faster development when gluing due to their 'typeless' nature.
In his example, he creates a button in Tcl.
button .b -text Hello! -font {Times 16} -command {puts hello}
He goes on to say:
With C++ and Microsoft Foundation Classes (MFC), it requires about 25 lines of code in three procedures. 1 Just set- ting the font requires several lines of code in MFC: CFont *fontPtr = new CFont(); fontPtr->CreateFont(16, 0, 0, 0, 700, 0, 0, 0, ANSI_CHARSET, OUT_DEFAULT_PRECIS, CLIP_DEFAULT_PRECIS, DEFAULT_QUALITY, DEFAULT_PITCH|FF_DONTCARE, “Times New Roman”); buttonPtr->SetFont(fontPtr);
Much of this code is a consequence of the strong typ- ing[...] In Tcl, the essential characteristics of the font (typeface Times, size 16 points) can be used immediately with no declarations or conversions. Furthermore, Tcl allows the button’s behavior to be included directly in the command that cre- ates the button, while C++ and Java require it to be placed in a separately declared method.
End quote.
We've come a long way, and examples have made clear that this dichotomy is a false one. Ousterhout's view was colored by his limited experience, resulting in him misunderstanding what he actually likes about Tcl.
Let's talk about syntax. His exact Tcl code as presented could be parsed and compiled by a language that does static type analysis. It's not, because he's running it in an interpreter that only checks types at runtime. But the point is that whether code is compiled or interpreted is an implementation detail that has very little to do with the syntax. He likes his syntax because his syntax is obviously better than C++, nothing more.
And types: he claims that 'typeless' languages allow for faster development because of fewer restrictions. This is, ofc, nonsense. The amount of restrictions you have is a function of your problem, not your language. If it feels like dynamic typing is letting you develop faster, that's because you're putting off encountering those restrictions til later.
It's better to guarantee we encounter all type errors before the program even runs. But since you can do static type analysis for any language, why don't all languages do that?
Because it's hard. Type theory is complicated. Not all type systems are decidable, we need to pick one that is for practical reasons. Ones that are may require annotations or complex algorithms/semantics restrictions like Hindley-Milner.
As a PL designer, it's a whole lot easier to just give up and only check types at runtime. And that makes a whole lot of sense if your priority is just embedding a usable language ASAP. But let's not pretend that it's because it's actually better.
I opine that typeless languages are best for short programs, because one can understand 100% of the program. Typed languages are a way to manage the complexity of a larger program.
For a related example, the BASIC programming language is great for programs that fit in 24 lines (which is how many lines a glass tty displayed).
There's clearly a cost-benefit tradeoff on the spectrum of static typing. We aren't all using Coq. OTOH, the burden of "basic" static typing is very low at this point given the successes we've seen with type inference. [0]
It seems like all of the dynamically typed programming languages are pursuing (optional/gradual) static typing systems, and we've mostly moved on past "pure" dynamically typed PLs at this point. It's more about how-much and what-kind of static typing. And perhaps this depends upon your domain (e.g. Rust and its focus on memory safety while eschewing GC, which necessitates lifetime annotations).
0) I don't think type inference is a panacea. For example, it's not super helpful when you have a complex type system inferring crazy types for you, and you're stuck debugging them when the inferrer has failed.
BASIC variables are typed. BASIC was also originally designed for printing terminals, so there was no 24 line limit.
Modern BASICs are suitable for writing larger programs.
Also seen today on HN:
"I Miss Visual Basic" https://news.ycombinator.com/item?id=43988853
"The history and legacy of Visual Basic" https://news.ycombinator.com/item?id=43949855
The BASIC I used was BASIC-10 on the PDP-10 in the 70's. It wasn't much improved over the original.
The 24 line thing was a limitation of the glass tty, not BASIC.
Having functions that fit on one screen is still a good idea I think.
My point was that printing, teletype-style terminals don't have an intrinsic vertical space limit like video display terminals (and eventually the printouts can get quite long).
But now that I think about it, perhaps the designers of BASIC might have been thinking of making it easy for a beginner to spend an afternoon or two learning how to write something comparable to a short FORTRAN program that might fit on a small deck of 80 column (72 with sequence number) punched cards.
I agree, my rule of thumb is > 1k LoC is best built with statically typed languages. For 20 line programs that I don't plan to maintain for a long time, I'd hardly ever reach for Rust over Python.
Yeah, there's no reason at all for a borrow checker for 20 lines of code, unless one is experimenting with the borrow checker.
Static typing lets you build very large rigid structures where types ensure constraints. However, these constraints get in the way of moving quickly. If you change a signature of a function, you have to first change every piece of code that calls the function. You then have to compile, restart your program, rebuild the whole rigid structure, and confirm the change works as expected. This whole process can take many minutes. The claim is that catching these errors at compile time results in more reliable code.
Dynamic typing has no such constraints to help you catch errors at compile time. Dynamic typing allows you to build organic structures that more fluidly change. If you change a function signature, you don't have to change every piece of code that calls the function. You can change the function and test it in a running application without restarting the application. Doing this takes seconds. Dynamic typing allows you to iterate 100x faster. Being able to rapidly iterate often means better tested code and more reliable code.
Both are valuable. Dynamic typing is valuable during development when you need to iterate rapidly and build a prototype when requirements are often changing. Static typing is valuable when code need is more mature, changing less frequently, and code needs to be put in production.
Ideally you have both. More mature parts of the program are statically typed and more fluid parts of the program are dynamically typed.
> Static typing lets you build very large rigid structures where types ensure constraints. However, these constraints get in the way of moving quickly. If you change a signature of a function, you have to first change every piece of code that calls the function.
I strongly disagree that this sort of refactoring is "quicker" with dynamically typed languages, particularly with non-trivial (i.e. large) code bases.
You have zero feedback and no indication of how disruptive such a change is when working with a dynamically typed language. In fact, you have no feedback on whether you've actually finished the change at all.
While with static type checking, you have immediate and valuable feedback guiding you through such a change. Each line of code which requires updating is identified for you and communicated to you.
> The claim is that catching these errors at compile time results in more reliable code.
For me, the most important reason is that static typing makes the code much more self-documenting.
I can see what that function is returning, and if it's something unknown I can go to the type declaration and see what I can do with that type.
This is entirely unlike dynamic typing where you have to pray the docs are very good and up to date, which frequentlyis not the case.
Spending my time doing work the compiler should have done for me (including tracking down heisenbugs) has wasted far more of my time than I’ve ever gained from the lax enforcement of dynamic typing.
> You then have to compile, restart your program, rebuild the whole rigid structure, and confirm the change works as expected. This whole process can take many minutes.
This is not my experience at all. Virgil is a fully statically-typed language and compiles fast. When working on the Virgil compiler (approximately 50KLOC), my typical workflow is to run the compiler in the interpreter in my debug, edit, run loop. The interpreter can load the entire compiler, typecheck it, and run it with my changes in 90 milliseconds. If we throw a complete optimized compile in the loop, it's 400 milliseconds.
It takes me longer to switch between my two terminal tabs.
It's not static typing versus dynamic typing. It's ridiculously inefficient languages/build systems versus efficient ones.
Inefficient languages/build systems can make the problem worse.
There are many statically typed languages that can compile in under a second. But when you change any function signature, you still have to ensure every caller is fixed. You then have to recompile, restart the application and rebuild all the data structures. Some static languages allow for hot reloading in interpretive mode, but the types of changes allowed are limited.
In dynamic languages like Common Lisp and Smalltalk you can update the program while it is running. The speed of iteration is 100x faster, because you don't have to restart the application and rebuild all the data structures to get to the point where you can test your change.
> It's better to guarantee we encounter all type errors before the program even runs.
This is only valid if you either are writing mission-critical software, or have infinite time.
Your argument doesn’t consider the case that you have a deadline and need to ship, so optimising for productivity, rather than asymptotic typing perfection, is paramount. There is a reason even performance-critical environments, where scripting languages are not very well suited, somehow still lean on them for productivity.
Case in point, game dev (Unreal Engine with its blueprint system, Godot with GDScript, the myriad of game engines in C++ paired with Lua.) Of course in a vacuum game devs would like to write ideal data structures with strong typing so that the game doesn’t crash, but their goal is to ship a game within the decade, so limit the strong typing to performance critical engine and can focus on building and iterating on the core product without consulting type theory books.
The point of Mr. Ousterhout’s argument is that there are only two choices: either we invent the panacea, the mythical productive strong-typed language that gives 100% safety yet enables experimentation, optional typing and compiles to C++ speed native code, or we accept that this is an impossible pipe dream, and we need to use the correct tool for the problem. Again, obvious on the surface, but still a contentious point to this day.
>The point of Mr. Ousterhout’s argument is that there are only two choices:
Ultimately, it all compiles down to assembly (which is executed), or an interpreter (which is executed). Assembly all the way. The real choice here is: turn into assembly ahead of time (programmer's code becomes assembly), or do so at runtime (assembly interprets programmer's code). Or some in-between like JIT compilation. And: what guardrails you put in place.
So you could say that higher-level languages are just a way to make tedious/nasty assembly coding more palatable. Eg. by producing error reports vs. just crashing a machine. Or provide a sandbox. Or provide a model of computation that fits well with programmer's thinking.
Between those, you simply need some useful abstractions. Preferably ones that are simple yet versatile/universal.
Eg. a (bitmap) "screen" could just be a flat memory space (framebuffer). Or a 2-D array of pixels, potentially with "pixel" defined elsewhere.
Programmer doesn't care how GPU is coerced into displaying those pixels, low-level code doesn't care what higher-level software does to produce them.
And then have as few as those abstractions as possible (but no less!). Tcl: "everything is a string". Lisp: "everything is a list". Unix(-like): "everything is a file". Etc.
Personally, I have a soft spot for languages that punch above their weight for expressiveness / few but useful abstractions vs. their implementation size. Things like Forth, Tcl or Lua come to mind.
But hey that's just me. Developers & organisations they're in make their own choices.
> The real choice here is: turn into assembly ahead of time (programmer's code becomes assembly), or do so at runtime (assembly interprets programmer's code). Or some in-between like JIT compilation. And: what guardrails you put in place.
A language's preferred compilation/interpretation model is, roughly speaking, orthogonal to its type system and to whether it emphasises speed of development, correctness, or performance. Also, many languages can practically be implemented using various compilation/interpretation models. There are interpreters for C, for instance, and ahead-of-time machine-code compilers for Java.
> So you could say that higher-level languages are just a way to make tedious/nasty assembly coding more palatable.
Well, the point of scripting languages is mostly dynamic typing and dynamic binding, which means resolving bindings at runtime, which means slower in execution. There is no silver bullet: you want speed, you need a clunky language that compiles to fast machine code. If you want ease of use and less ceremony, you’ll get a slower language.
Dynamic languages are definitely easier to use for highly dynamic problems.
They are not necessarily easier to use for statically-knowable problems.
Static languages get benefits in correctness and speed.
Key word is dynamic.
> Ousterhout's view was colored by his limited experience
Wow… If he has limited experience, I better go doing something else…
Limited experience may have been the wrong phrase. What I meant was limited perspective, given that he's writing this in 1998.
What I've never understood about this argument is this:
How often are you passing around data that you don't fully understand?
Also, people use types and then end up reaching for reflection to perform pattern matching on types at which point you've just moved the typing from the user level to a type system. Not much gained imo.
>How often are you passing around data that you don't fully understand?
Like all the time. As a program grows, the full extend of what is passed where, explodes.
>Also, people use types and then end up reaching for reflection to perform pattern matching
As a tool in their disposal, with the default being the opposite.
> How often are you passing around data that you don't fully understand?
When you're using the new code your colleague just committed after a week of intense work, for example.
Most people don't reach for reflection to pattern match on types, because when a type _changes_ you get very nice errors at all places where you've just introduced a bug.
But some languages DO have less restrictions. There's a lot less I have to worry about with Python than the quagmire of Java. You may say I'm putting something off (and maybe I am), but that is perfectly reasonable in a lot of scripting use cases.
Mmmh yes and no. Some languages truly do have more/less restrictions.
For example, a garbage collector is the compiler/runtime taking on more work to avoid the restrictions of proper memory management.
This is in some sense analogous to the C++ vs Rust debate. Borrow checking allows the compiler to statically verify whether the program is memory safe. Like statically verified types, it turns a class of undesirable runtime behavior into a compilation failure. Ofc, this is a restriction that may be undesirable, which is why you can turn it off with 'unsafe'.
Python and Java though... do you really have that much more to worry about in Java?
In my eyes, you've got your problem. Regardless of your language choice, you can write a pseudocode algorithm for it. Terms in your algorithm have types already.
When we translate this algorithm into python/Java, the types of the program need to match the types of our pseudocode. Regardless of whether we get feedback on errors at runtime or compile time, we just need to get the types right. I'd prefer my error at compile time. The benefit is tiny for short scripts, but becomes massive when working on a large code base.
What about Java (being a "systems" language) makes you worry about so much more than Python (the "scripting" language)? There are definitely differences, but I don't think they stem from this systems/scripting dichotomy that's intrinsically linked to whether you check your types at compile or run time.
With Python I can often do some task in like half a page of code with zero boiler plate. Just some loops and a couple of objects and a library call. I don't even write classes most of the time and the state is all global. In Java, there is just sooo much more for the equivalent Python code. Maybe asking an LLM will save some time, but yeah ...there really is a lot more ceremony with Java.
Of course Python has some restrictions due to the garbage collector. Good catch on that as it is if course true and limits you in other areas I don't care about. I was mainly referring to scripting needs where some languages really do shine above others. There's a reason Python is ubiquitous in that area and Java is not. Just like there's a reason Java and C# are ubiquitous in large enterprise software and Python is not.
To summarize though, I think I understand your point, but we're getting lost in semantics haha.
> And types: he claims that 'typeless' languages allow for faster > development because of fewer restrictions. This is, ofc, nonsense. > The amount of restrictions you have is a function of your problem, > not your language.
Unless you're implementing something that's already known, you're effectively carving out your problem when you're in the process of writing code. There are more unknowns than there are knowns. A strongly typed language forces you to focus on satisfying the types before you can do anything, at a point where you don't exactly yet know what you want to do. The right time to cast your data rigidly into types is gradually when your data structures settle in their own place one by one; in other words when you're pretty sure things won't change too easily anymore. This allows writing code that works first and then, depending on your needs, gets next correct and then fast, or next fast and then correct. You use the weakly typed or "typeless" style as a prototyping tool that generates snippets of solid code that you can then typefy and write down for good.
Solutions to any given problem are necessarily described with types. You can check those types at runtime or you can check them at compile time, but meaningful programs are typed. If you've got a function that takes a string and your type system lets you run a program where you pass an int to it... did it just allow you to 'quickly iterate'? Sure, you're running a program sooner, but you're going to run into the same type error (function expects string, not int).
Maybe in a larger code base you could argue that the effort involved in satisfying typechecks meaningfully slows down iteration, but even then it's clear that being able to compute the list of type errors you introduced elsewhere (via static typing semantics/analysis) is valuable.
Languages introduce restrictions of their own all the time. In Java I can't have a function that works on any type of number, I have to specify the type. In Haskell I can. In Haskell I can't put random stuff in a list, in Java I can as long as they are all objects.
Those language restrictions slow down your development as you have to design and code around them. People see the extra design\code as a good trade off because it improves safety, but it slows you down if you are just messing around.
Actually there's really not that much difference. In Java you have to cast to Object, in Haskell you can wrap in Dynamic:
https://www.stackage.org/haddock/lts-23.22/base-4.19.2.0/Dat...
>We've come a long way, and examples have made clear that this dichotomy is a false one.
I wouldn't be so sure. Even Javascript, when is added types in the form of Typescript, looks more like the convoluted Windows API example and less like the TCL example.
>Let's talk about syntax. His exact Tcl code as presented could be parsed and compiled by a language that does static type analysis. It's not, because he's running it in an interpreter that only checks types at runtime. But the point is that whether code is compiled or interpreted is an implementation detail that has very little to do with the syntax.
Pedantically true. You can of course compile anything, including BASIC and TCL, or intepret C++ if you wished so.
But to really take advantage of compilation, languages tend to add types and other annotations or semantic constructs (like Haskell does). And so compiled languages tend to a certain syntax/semantics, and interpreted languages to another.
>And types: he claims that 'typeless' languages allow for faster development because of fewer restrictions. This is, ofc, nonsense. The amount of restrictions you have is a function of your problem, not your language. If it feels like dynamic typing is letting you develop faster, that's because you're putting off encountering those restrictions til later.
So it's not "nonsense", but rather a tradeoff, just like Ousterhout puts it.
For many, it is much faster to "putting off encountering those restrictions til later", as that's a large part of exploring the problem space.
See, while "the amount of restrictions you have is a function of your problem", you don't always know your exact problem, its boundaries, and the form you'll use to solve it, until you've experimented and played with early prototype solutions and changed them, etc. While doing so, having to tackle hard restrictions based on assumptions that can and will change or be discarded, slows you down.
>It's better to guarantee we encounter all type errors before the program even runs.
Only if it doesn't entail other issues, like it making our exploratory programming to end up with a design for our program slower and more tedious.
Commenting just because you can't favorite comments
Click the comment timestamp and use the favourite link to add it to your favourites, accessible from your profile page.
Click the time (next to the author) to go to a page for the comment, where you will be able to favorite it.
Love this post. Writing on programming languages has changed how I think about _programming_ in general.
I often think about this quote from TAPL. This framing of “safety” changed how I design systems.
> Informally, though, safe languages can be defined as ones that make it impossible to shoot yourself in the foot while programming.
> Refining this intuition a little, we could say that a safe language _is one that protects its own abstractions_.
> Safety refers to the language's ability to guarantee the integrity of these abstractions and of higher-level abstractions introduced by the programmer using the definitional facilities of the language. For example, a language may provide arrays, with access and update operations, as an abstraction of the underlying memory. A programmer using this language then expects that an array can be changed only by using the update operation on it explicitly—and not, for example, by writing past the end of some other data structure.
https://www.cis.upenn.edu/~bcpierce/tapl/
Regarding weird development methods of interest...Aaron Hsu of APL fame writes a lot of code in calligraphy with fountain pens when trying to organize his thoughts. I do something kind of similar, but in print with a crummy bic pen and a flow chart of Python objects (kind of like poor man's UML).
I also turn to a fountain pen for the hardest problems. It puts me in a completely different head space. Something about the limited editing ability forces more coherent, linear thought, but also the freedom to seamlessly switch between English, code, math, and diagrams opens up the creativity.
There is a proven association between hand-writing and memory retention. Taking notes on a computer is like leaving fingerprints on a handrail.
I wish OCR technology got so good that I could only write by hand and have perfect storage and searchability.
I would also highly recommend watching one of Rich Hickey's talks (especially the earlier ones). Watching those certainly changed how I thought about programming in general.
Skip “Simple made easy” because I cannot stand hearing that talk quoted by basically every single conference speaker in the last decade. It’s become its own cliché.
(Joking of course. I much prefer “Hammock driven development” but it’s not very corporate friendly)
You say that, but if there's one thing I wish more mid-level engineers understood better about good code it's the distinction between simple and easy.
Definitely the most influential for me along with Larry Wall's "Programming Perl".
There was a nice post here recently about speeding up interpreters via closure-based interpreter:
https://news.ycombinator.com/item?id=43595283
I hacked up a toy brainfuck interpreter using that technique and it was pretty fast. Not sure I'd get the chance to use it elsewhere, but experimenting with it was useful regardless:
https://github.com/skx/closure-based-brainfuck-vm
I am very curious about the difference between an array of closures and an array of alternating function pointers and data. The latter doesn't come very naturally in most languages, you need something very C like for it to make sense. But my gut feeling says static functions and having access to the data right there in the same array might play better with cache.
It's a shame that Abdulaziz went quiet after moving back to Kuwait. He was our intern on Maxine VM back in 2009. A super nice guy and that paper is a gem!
I know :( but it looks like he opened a bakery and that business is going well. So in some sense he's living the dream
I like this guy, so nothing against him, but none of these are about PL and all about these are about compilers (except for the one about GC). Which is fine (I like compilers) but they're just not in any way about PL.
Programming language (implementation)? :)
I wish someone would write this for higher-level languages: JavaScript or .NET. I'm sure this person is brilliant, but they're operating at a much lower (higher?) level than most of us.
pytype is based in part upon byterun https://github.com/google/pytype/blob/main/docs/developers/i...
I learnt a lot about bytecode interpreters from working on it, and it helped me understand the cpython source code a lot more easily from having played with a python translation of it first.
For micrograd: is there more documentation available than just the source code in the Github repo?
He (Andrej Karpathy) has got a series on youtube which goes through how he made it!
https://www.youtube.com/watch?v=VMj-3S1tku0&list=PLAqhIrjkxb...
Oh cool, thank you so much!!
Damn, his other blog posts are stellar as well, nice!
Awesome. Thanks.
Neat. Thanks.
don't you mean bernstainbear?
https://www.cosmopolitan.com/lifestyle/a7664693/mandela-effe...