The last four links I have shared above all get automated by a simple QL:QUICKLOAD call if we're using Quicklisp, and that's one of the reasons Quicklisp has become almost a de facto standard in the community.
You can achieve roughly the same by writing down the SHA256 hash the first time you download and then comparing when you download the next time.
But, yeah, while I do not like submodules, for vendoring stuff it seems a reasonable approach. There's also https://github.com/fosskers/vend if you lean that way.
I started learning Common Lisp, but ASDF and Quicklisp threw me off. I couldn't tell if you were supposed to choose one or the other or they were used together. This might revive my interest in Common Lisp if I get around to reading it. But in the meantime I drifted off to Racket, which is relatively well documented and has extensive libraries and really unique features.
The packaging story in common lisp is.... Not great.
It's hamstrung by archaic naming conventions that confuse newcomers. What CL calls a system is roughly analogous to what most other languages call a package. What CL calls a package is what other languages call a namespace.
Despite all that it's a pretty good language if you can find libraries for what you need. The de facto standard implementation (sbcl) has a very good compiler and an acceptable GC. The language itself is expressive and it makes for very quick and pleasant DX. I love writing common lisp.
> * What CL calls a system is roughly analogous to what most other languages call a package.*
Or a crate, or an artifact, or a module, or a gem, and there's probably other variations I can't remember off-hand.
> * What CL calls a package is what other languages call a namespace.*
Or a module, or a package, or... actually, I don't know what Perl or Ruby call it. I believe C calls it a header, but that's not quite the same thing as a package.
Turns out naming things is difficult (as well as cache invalidation, off-by-one errors concurrency, and).
Eggs? Goodness. And I believe Chicken is R5RS as well, so I don't know what they call libraries/modules/packages/whatever (in R6RS and R7RS they're called libraries, but R5RS didn't specify anything). I expect Racket to call them libraries considering the Racket/R6RS connections.
Is it archaic? A lisp program is a dynamic image. A collection of symbol is very aptly named a package. And third party module can be named as a system (collection of packages).
Agreed, and I think package as used by Common Lisp and Java is more common than “namespace” which the parent commenter believes is the modern word for that!
For anybody who's still confused, the tl;dr is ASDF is the actual package loading mechanism, Quicklisp doubles as an ASDF wrapper and a package manager.
Pretty good, except and I don't share the advice to use package-inferred-systems, like, at all. It hides the third-party libraries you rely on, it prevents you from using one package in multiple files (a flexibility not common out there), you can't see the project's structure at first glance… just use a simple .asd file declaration, you'll be fine.
YMMV, of course. I switched to it half a year or so ago, when doing a close read of the ASDF docs, and for my purposes it works well. But I may be odd: I have a monorepo of Lisp code which I don't intend to distribute in the sense of turning them into Open Source packages. There's an `l` subdirectory for libraries, a `p` subdirectory for "projects", and if I need something I can just import `ca.berksoft.l/math/fft` and be done. I think that having a file-per-package is not a limitation, it makes packages probably a bit more like modules in my daytime language (Elixir/Erlang), and it does save a lot of typing telling ASDF what to find where.
It's interesting to know your use case, thanks. I don't like dealing with package-inferred-systems when exploring, reading or using other people's libraries.
Quicklisp is great, it's the defacto standard, but compared to OCICL it kinda feels ancient. There's also CLPM, but last time I checked it was broken by a combination of dead links and missing functions.
Indeed, while you can use ql-https for, well, HTTPS, it's not the easiest thing to install (especially if you want to put everything somewhere else than ~/common-lisp/) and adding other distributions (like, say, Ultralisp) is a bit finicky.
What's missing from any of this, which has really confused me in the past, is any notion of dependency versioning. We get predefined dependencies as a distribution in quicklisp - which is great as far as it goes - but how do people manage without being able to say "this system depends on a version of that system greater than X"?
You can pin dependencies with Qlot or Ocicl (or vendor them with vend), but it might be a long time before you actually need this (the ecosystem is pretty darn stable).
TL;DR: If I find a library I'm using would need dependency versioning, I consider that library broken and find (or write) an alternative.
You can always just add a version check and error out if it's too outdated. The thing there isn't an easy way to do is say "this needs a version of that system lower than X" but it would be unusual for a system to intentionally break backwards compatibility (or for an unintentional break to not be fixed relatively quickly after being discovered); usually if there is the semver equivalent of a "major version" change in lisp, the system-name itself gets changed.
Yeah, the liberating thing for me in CL is that things just don’t break as much as they do in other ecosystems. So, when I get breaking changes I look for an alternative that doesn’t break.
My honest take is that if someone truly loves CL and wants it to get more hacker attention, it would greatly, greatly benefit from someone greenfielding a modern package manager for it.
That is to say, a cargo/zig/mix/golang-style all-in-one CLI tool that has opinionated defaults, reasonable basic functionality (HTTPS, hashing, lockfiles) and is approachable and frictionless. `cl init my-proj`, `cl test my-proj`, etc.
To be entirely frank though, I never got the sense that the CL community is interested in that kind of onboarding, so I expect the language to continue its steady slide into senescence, sadly.
I am aware of it. I don't think it's equivalent to the tools I mentioned, though it is definitely an improvement on mere ASDF/QL. The problem with ocicl is that it's more brownfield than greenfield -- it's fixing the existing packaging, but not doing much to rethink it.
The tool I am thinking of would need to (a) be able to download, manage, pin (etc) various CL compilers, (b) offer a REPL as interactive as e.g. irb, (c) offer comparable features and ease of use to something like cargo. That is to say - language management, project management, package management, accessible at logical keywords under an all-in-one tool.
CL's soul is the REPL, and yet by default the average newcomer can't even use arrow keys or backspace in SBCL. Sure, sure, they should use rlwrap, but they'd have to know that exists, and they generally don't. So they'll go back to Python or Node, and not worry about a language that can't even figure out backspaces (from their perspective).
CL has a lot going for it, but also has a lot of cruft, that its users are used to, but which is off-putting to newcomers. Not every bit of esoterica about the historical implementations of CL is in fact important or even useful, and CL is absolutely buried in it. Both OCaml and Haskell have invested heavily in their tooling recently, to good effect. CL... not so much.
It would take a very confident hand to hack away the weeds, but I don't see another way forward. CL can be a vibrant tool into the future, or it can become (remain?) a museum piece.
I messed around with common lisp for a while a few months ago, and I remember the packaging/dependency situation was by far the most difficult and confusing part. So thanks for writing this article, bookmarked it for the next time I write some CL :)
Another point that needs clarification is testing. Theres a lot of different test systems but they are all amateurish. Does anyone know something that works well? Stuff like rov, parachute, clunit is all really basic. Not even support for good html reports and tagging tests for example.
I considered that (author here), but how I test is way too odd to share lol.
I think that that's one of the strengths and one of the weaknesses of CL and its ecosystem. Rolling your own variation is just too easy and it almost seems to be encouraged. Which artificially steepens the learning curve. Anyway, I decided to focus on just "packaging", but I agree that testing needs attention, just like all the other topics people here touched on: secure distribution, versioning and pinning, and all these other modern comforts we're used to when doing our daytime non-Common-Lisp jobs :)
Quicklisp is great and I recommend using it along with a brief introduction in both my Common Lisp setup guides for Vim and Emacs:
https://susam.net/lisp-in-vim.html
https://github.com/susam/emacs4cl
However, for my personal projects, I usually just download the package versions I need from GitHub with curl within a simple while loop:
https://github.com/susam/susam.net/blob/0.4.0/Makefile#L83-L...
https://github.com/susam/susam.net/blob/0.4.0/meta/cldeps/fo...
Then I point ASDF to the download directory with CL_SOURCE_REGISTRY and load it in my Lisp program using good old ASDF:LOAD-SYSTEM:
https://github.com/susam/susam.net/blob/0.4.0/etc/form.servi...
https://github.com/susam/susam.net/blob/0.4.0/form.lisp#L5
The last four links I have shared above all get automated by a simple QL:QUICKLOAD call if we're using Quicklisp, and that's one of the reasons Quicklisp has become almost a de facto standard in the community.
I'd suggest you submodule in dependencies rather than curl. Supply chain attacks and version incompatibilities both happen and suck
> I'd suggest you submodule in dependencies rather than curl. Supply chain attacks and version incompatibilities both happen and suck
What kind of supply chain attack or version incompatibility would affect
but notSubmodules are pinned by commit hash. It prevents an attacker from replacing a release.
That is very handy to know.
You can achieve roughly the same by writing down the SHA256 hash the first time you download and then comparing when you download the next time.
But, yeah, while I do not like submodules, for vendoring stuff it seems a reasonable approach. There's also https://github.com/fosskers/vend if you lean that way.
I started learning Common Lisp, but ASDF and Quicklisp threw me off. I couldn't tell if you were supposed to choose one or the other or they were used together. This might revive my interest in Common Lisp if I get around to reading it. But in the meantime I drifted off to Racket, which is relatively well documented and has extensive libraries and really unique features.
The packaging story in common lisp is.... Not great.
It's hamstrung by archaic naming conventions that confuse newcomers. What CL calls a system is roughly analogous to what most other languages call a package. What CL calls a package is what other languages call a namespace.
Despite all that it's a pretty good language if you can find libraries for what you need. The de facto standard implementation (sbcl) has a very good compiler and an acceptable GC. The language itself is expressive and it makes for very quick and pleasant DX. I love writing common lisp.
> * What CL calls a system is roughly analogous to what most other languages call a package.*
Or a crate, or an artifact, or a module, or a gem, and there's probably other variations I can't remember off-hand.
> * What CL calls a package is what other languages call a namespace.*
Or a module, or a package, or... actually, I don't know what Perl or Ruby call it. I believe C calls it a header, but that's not quite the same thing as a package.
Turns out naming things is difficult (as well as cache invalidation, off-by-one errors concurrency, and).
Racket has packages (1) that work quite well. Chicken Scheme has Eggs.
(1) https://docs.racket-lang.org/pkg/index.html
Eggs? Goodness. And I believe Chicken is R5RS as well, so I don't know what they call libraries/modules/packages/whatever (in R6RS and R7RS they're called libraries, but R5RS didn't specify anything). I expect Racket to call them libraries considering the Racket/R6RS connections.
Is it archaic? A lisp program is a dynamic image. A collection of symbol is very aptly named a package. And third party module can be named as a system (collection of packages).
Agreed, and I think package as used by Common Lisp and Java is more common than “namespace” which the parent commenter believes is the modern word for that!
For anybody who's still confused, the tl;dr is ASDF is the actual package loading mechanism, Quicklisp doubles as an ASDF wrapper and a package manager.
Pretty good, except and I don't share the advice to use package-inferred-systems, like, at all. It hides the third-party libraries you rely on, it prevents you from using one package in multiple files (a flexibility not common out there), you can't see the project's structure at first glance… just use a simple .asd file declaration, you'll be fine.
more: https://lispcookbook.github.io/cl-cookbook/
libraries: https://github.com/CodyReichert/awesome-cl/
YMMV, of course. I switched to it half a year or so ago, when doing a close read of the ASDF docs, and for my purposes it works well. But I may be odd: I have a monorepo of Lisp code which I don't intend to distribute in the sense of turning them into Open Source packages. There's an `l` subdirectory for libraries, a `p` subdirectory for "projects", and if I need something I can just import `ca.berksoft.l/math/fft` and be done. I think that having a file-per-package is not a limitation, it makes packages probably a bit more like modules in my daytime language (Elixir/Erlang), and it does save a lot of typing telling ASDF what to find where.
It's interesting to know your use case, thanks. I don't like dealing with package-inferred-systems when exploring, reading or using other people's libraries.
Quicklisp is great, it's the defacto standard, but compared to OCICL it kinda feels ancient. There's also CLPM, but last time I checked it was broken by a combination of dead links and missing functions.
Last time I checked quicklisp also didn’t support https and doesn’t do any signature checking.
Quicklisp still doesn't support HTTPS, which is apparently also necessary to do signature check.
Use HTTPS instead of HTTP - https://github.com/quicklisp/quicklisp-client/issues/167
Indeed, while you can use ql-https for, well, HTTPS, it's not the easiest thing to install (especially if you want to put everything somewhere else than ~/common-lisp/) and adding other distributions (like, say, Ultralisp) is a bit finicky.
What's missing from any of this, which has really confused me in the past, is any notion of dependency versioning. We get predefined dependencies as a distribution in quicklisp - which is great as far as it goes - but how do people manage without being able to say "this system depends on a version of that system greater than X"?
You can pin dependencies with Qlot or Ocicl (or vendor them with vend), but it might be a long time before you actually need this (the ecosystem is pretty darn stable).
https://github.com/fukamachi/qlot/
https://github.com/ocicl/ocicl/
https://github.com/fosskers/vend/ (new)
TL;DR: If I find a library I'm using would need dependency versioning, I consider that library broken and find (or write) an alternative.
You can always just add a version check and error out if it's too outdated. The thing there isn't an easy way to do is say "this needs a version of that system lower than X" but it would be unusual for a system to intentionally break backwards compatibility (or for an unintentional break to not be fixed relatively quickly after being discovered); usually if there is the semver equivalent of a "major version" change in lisp, the system-name itself gets changed.
Yeah, the liberating thing for me in CL is that things just don’t break as much as they do in other ecosystems. So, when I get breaking changes I look for an alternative that doesn’t break.
My honest take is that if someone truly loves CL and wants it to get more hacker attention, it would greatly, greatly benefit from someone greenfielding a modern package manager for it.
That is to say, a cargo/zig/mix/golang-style all-in-one CLI tool that has opinionated defaults, reasonable basic functionality (HTTPS, hashing, lockfiles) and is approachable and frictionless. `cl init my-proj`, `cl test my-proj`, etc.
To be entirely frank though, I never got the sense that the CL community is interested in that kind of onboarding, so I expect the language to continue its steady slide into senescence, sadly.
You need to learn about ocicl: https://github.com/ocicl/ocicl It does all of this and more.
I am aware of it. I don't think it's equivalent to the tools I mentioned, though it is definitely an improvement on mere ASDF/QL. The problem with ocicl is that it's more brownfield than greenfield -- it's fixing the existing packaging, but not doing much to rethink it.
The tool I am thinking of would need to (a) be able to download, manage, pin (etc) various CL compilers, (b) offer a REPL as interactive as e.g. irb, (c) offer comparable features and ease of use to something like cargo. That is to say - language management, project management, package management, accessible at logical keywords under an all-in-one tool.
CL's soul is the REPL, and yet by default the average newcomer can't even use arrow keys or backspace in SBCL. Sure, sure, they should use rlwrap, but they'd have to know that exists, and they generally don't. So they'll go back to Python or Node, and not worry about a language that can't even figure out backspaces (from their perspective).
CL has a lot going for it, but also has a lot of cruft, that its users are used to, but which is off-putting to newcomers. Not every bit of esoterica about the historical implementations of CL is in fact important or even useful, and CL is absolutely buried in it. Both OCaml and Haskell have invested heavily in their tooling recently, to good effect. CL... not so much.
It would take a very confident hand to hack away the weeds, but I don't see another way forward. CL can be a vibrant tool into the future, or it can become (remain?) a museum piece.
I messed around with common lisp for a while a few months ago, and I remember the packaging/dependency situation was by far the most difficult and confusing part. So thanks for writing this article, bookmarked it for the next time I write some CL :)
ASDF (Another System Definition Facility) is my all time favorite name for a piece of software. Descriptive, funny, and easy to type!
Don't forget about UIOP (Utilities for Implementation and OS Portability) which is part of the ASDF project. Also very easy to type!
Mine is GNU WoMan, an alternative to man.
https://www.gnu.org/software/emacs/manual/html_mono/woman.ht...
thank you so kindly for sharing this it is very helpful!
Another point that needs clarification is testing. Theres a lot of different test systems but they are all amateurish. Does anyone know something that works well? Stuff like rov, parachute, clunit is all really basic. Not even support for good html reports and tagging tests for example.
I considered that (author here), but how I test is way too odd to share lol.
I think that that's one of the strengths and one of the weaknesses of CL and its ecosystem. Rolling your own variation is just too easy and it almost seems to be encouraged. Which artificially steepens the learning curve. Anyway, I decided to focus on just "packaging", but I agree that testing needs attention, just like all the other topics people here touched on: secure distribution, versioning and pinning, and all these other modern comforts we're used to when doing our daytime non-Common-Lisp jobs :)
https://sabracrolleton.github.io/testing-framework There's this pretty in-depth comparison of testing frameworks, but I'm not sure if any of the frameworks there satisfy your specifications.