--- a/content/blog/2018/07/a-road-to-common-lisp.markdown Fri Aug 03 18:17:15 2018 -0700
+++ b/content/blog/2018/07/a-road-to-common-lisp.markdown Sat Aug 04 12:55:21 2018 -0700
@@ -33,8 +33,8 @@
* [Common Lisp: the Untold Story][untold] by Kent Pitman.
* [The Evolution of Lisp][evolution] by Guy Steele and Richard Gabriel.
-I realize that you probably won't want to read all of the links above
-immediately, so here's a whirlwind tour of sixty years of Lisp.
+I realize you probably won't want to read all of the links above immediately, so
+here's a whirlwind tour of sixty years of Lisp.
Lisp began in the late 1950's. It was invented by John McCarthy at MIT.
@@ -68,11 +68,11 @@
For comparison: Python (a "modern" language many people think of as also being
"kind of old") [was released][python] for the first time the following year.
-In 1992 the X3J13 published the first draft of the new ANSI standard for Common
-Lisp for public review (see Pitman's paper). The draft was approved in 1994 and
-the approved specification was finally published in 1995. At this point Lisp is
-over thirty-five years old. The first version of Ruby [was released][rubby] in
-December of that year.
+In 1992 the X3J13 committee published the first draft of the new ANSI standard
+for Common Lisp for public review (see Pitman's paper). The draft was approved
+in 1994 and the approved specification was finally published in 1995. At this
+point Lisp was over thirty-five years old. The first version of Ruby [was
+released][rubby] in December of that year.
That's the end of the history lesson. There has not been another revision of
the ANSI specification of Common Lisp. The version published in 1995 is the one
@@ -95,25 +95,27 @@
Ruby, it's probably going to take some time to update it. My current day job is
in Scala, and if a library's last activity is more than 2 or 3 years old on
Github I just assume it won't work without a significant amount of screwing
-around on my part. The hamster wheel of backwards incompatibility we deal with
+around on my part. The Hamster Wheel of Backwards Incompatibility we deal with
every day is a fact of life in most modern languages (though some are certainly
better than others).
If you learn Common Lisp, this is not the case. In the next section of this
post I'll be recommending a book written in 1990. You can run its code,
unchanged, in a Common Lisp implementation released last month. After years of
-jogging on the hamster wheel of backwards incompatibility I cannot tell you how
+jogging on the Hamster Wheel of Backwards Incompatibility I cannot tell you how
much of a relief it is to be able to write code and reasonably expect it to
still work in twenty years.
Of course, this is only the case for the language itself — if you depend on any
libraries there's always the chance they might break when you update them. But
-I've found the stability of the core language is contagious, and the Common Lisp
-community seems overall fairly good about maintaining backwards compatibility.
+I've found the stability of the core language is contagious, and overall the
+Common Lisp community seems fairly good about maintaining backwards
+compatibility.
+
I'll be honest though: there are exceptions. As you learn the language and
start using libraries you'll start noticing some library authors who don't
bother to document and/or preserve stable APIs for their libraries, and if
-staying off the hamster wheel is important to you you'll learn to avoid relying
+staying off the Hamster Wheel is important to you you'll learn to avoid relying
on code written by those people as much as possible.
This may sound a little bleak, but it's made a bit better by some other
@@ -122,11 +124,11 @@
Lisp programmers) is 971 pages long, not including the preface, references, or
index. When programming in Common Lisp people will tend to depend on
a small(ish) number of libraries, and library writers often try to minimize
-dependencies by utilizing as much of the (large) core language as possible.
+dependencies by utilizing as much of the core language as possible.
I personally try to stick to fewer than ten or so dependencies for my
applications and no more than two or three for my libraries, but I'm probably
-a bit more conservative here than most folks (I *really* don't like the hamster
-wheel any more).
+a bit more conservative here than most folks (I *really* don't like the Hamster
+Wheel any more).
It's also worth noting that since Common Lisp has been around and stable for so
long, it has *libraries* older and more stable than many programming languages.
@@ -254,9 +256,9 @@
This historical baggage is a price paid to ensure Common Lisp had a future. It
made it practical for people using the old dialects to actually adopt Common
Lisp with a reasonable amount of effort. If the designers had tried to make it
-perfect and beautiful the language probably would have been ignored as too
-different to port implementations and code to, instead of being adopted and
-embraced.
+perfect and beautiful this could have made it too different to port
+implementations and code to and might have resulted in the language being
+ignored, instead of being adopted and embraced.
[wiki-history-cl]: https://en.wikipedia.org/wiki/Common_Lisp#History
[wiki-history-lisp]: https://en.wikipedia.org/wiki/Lisp_(programming_language)#History
@@ -339,11 +341,12 @@
To check that you've got everything set up properly, make a `hello.lisp` file
with the following contents:
- :::lisp
- (defun hello ()
- (write-line "What is your name?")
- (let ((name (read-line)))
- (format t "Hello, ~A.~%" name)))
+```lisp
+(defun hello ()
+ (write-line "What is your name?")
+ (let ((name (read-line)))
+ (format t "Hello, ~A.~%" name)))
+```
Don't worry about what this means yet, it's just a check that everything's
working properly.
@@ -516,14 +519,14 @@
things on top of it.
This cycle of making a function, compiling it, poking at it to make sure it's
-working as expected, and moving on happens constantly.
+working as expected, and moving on happens *constantly*.
In contrast, when working in languages like Scala or Python I almost never find
myself writing one single function and compiling or running the project
immediately. Spinning up the compiler or running the unit tests takes at
*least* a second or two (or, sometimes *minutes* in Scala, unfortunately) so to
avoid having a constant stream of gaps in my thought I end up writing a bunch of
-functions at once, and then run the project or tests once I know they have
+functions at once, and then I run the project or tests once I know they have
a chance of working.
But then when I get back an error I have much more surface area to check,
@@ -550,8 +553,8 @@
say you're working on the code for a particular quest. You'll start the game,
load a save file at the beginning of the quest, and start going through the
steps. All of a sudden, in the middle of killing the final monster for the
-quest, you hit the damage bug! In tradition languages, one of two things might
-happen:
+quest, you hit the damage bug! In traditional languages, one of two things
+might happen:
1. The game crashes, and you get a stack trace and maybe a core dump.
2. You've wrapped a `try` block around the main game loop that logs a stack
@@ -573,34 +576,34 @@
and open a window in your editor showing you the stack trace. Your warrior's
sword is hovering at the monster, waiting for you. At this point you can
communicate with the running process at the REPL to see what's going on. You
-can examine variables in the stack.
+can examine variables in the stack, or even run any arbitrary code you want.
Once you figure out the problem ("Oh, I see, the `calculate-armor-percentage`
function returns `0` if a shielding spell ran out during the same frame") you
can fix the code, recompile the problematic function, and *restart the execution
-of that function in call the stack*. Your warrior's sword lands, and you move
+of that function in the call stack*. Your warrior's sword lands, and you move
back to what you were doing before.
You don't have to track down the bug from just a stack trace, like a detective
trying to piece together what happened by the blood stains on the wall. You can
examine the crime *as it's happening* and intervene to save the victim. It's
like if you could run your code in a debugger with a breakpoint at every single
-line.
+line that only activates if something goes wrong!
-Maybe you don't make video games. But this process can be useful in call kinds
+Maybe you don't make video games. But this process can be useful in all kinds
of contexts. Maybe you're writing a web app that talks to an API somewhere, and
-are debugging a requests that fails somewhere between two calls to the API,
-maybe "create widget `foo`" and "add `foo` to widget list `bar`". Instead of
+are debugging a request that fails somewhere between two calls to the API, e.g.
+between "create widget `foo`" and "add `foo` to widget list `bar`". Instead of
just aborting the request, logging a stack trace, and now leaving things in
-a possibly weird state (`foo` having been created without being in the expected `bar`
-list), you can fix the problem and allow the request to finish properly.
+a possibly weird state (`foo` having been created without being in the expected
+`bar` list), you can fix the problem and allow the request to finish properly.
Of course this won't always work. If you've got a big function that does some
side effects and then crashes, restarting execution of the function would make
the side effects happen again. But if you divide up your functions well ([one
-function to a function][1ftaf]) this case is pretty rare. And even when it does
-happen, it just means you're back in the same situation you're in *by default*
-with other languages!
+function to a function!][1ftaf]) this case is pretty rare. And even when it
+does happen, it just means you're back in the same situation you're in *by
+default* with other languages!
[eager]: https://www.reddit.com/r/lisp/comments/4oo1cp/common_lisp_for_clojure_programmer/d4eec68/
[1ftaf]: https://groups.google.com/forum/message/raw?msg=comp.lang.lisp/9SKZ5YJUmBg/Fj05OZQomzIJ
@@ -643,7 +646,7 @@
Protocol][amop] (usually abbreviated as AMOP). This book will probably take you
a couple of tries to get through. Read it until you hit a mental wall, go work
on other things for a couple of months, and come back and try again. Repeat
-that process as many times as is necessary.
+that process as many times as necessary.
[Keene]: https://www.amazon.com/Object-Oriented-Programming-COMMON-LISP-Programmers/dp/0201175894
[amop]: https://www.amazon.com/Art-Metaobject-Protocol-Gregor-Kiczales/dp/0262610744
@@ -717,9 +720,9 @@
### Window Management
If you're running Linux and like tinkering with your desktop environment,
-[StumpWM][] is a X window manager written in Common Lisp. I've just recently
-switched to Linux so I've only been using it for a month or so, but it's really
-pleasant to be able to customize my working environment with Common Lisp.
+[StumpWM][] is an X window manager written in Common Lisp. I've just recently
+switched back to Linux so I've only been using it for a month or so, but it's
+really pleasant to be able to customize my working environment with Common Lisp.
StumpWM has a small but friendly community — if you're looking for a non-trivial
open source Common Lisp project to contribute to, StumpWM would be a great
@@ -751,16 +754,262 @@
Common Lisp is old and stable, but that doesn't mean it's stagnant. The
language gives you plenty of power to build on, and before I wrap this up I want
-to mention a few things that have become "defacto standards" (or at least
-"pretty commonly seen").
+go over a couple of things that often trip up new people.
+
+### Structure
-### Structure and Building
+Common Lisp's terminology for various parts of projects is often confusing to
+new people because it's old and uses a lot of words that we use now (like
+“package”) to mean different things than people mean today. Things get easier
+once you internalize what Common Lisp means by the terms.
+
+NOTE: I posted a quick-and-dirty version of this section as a [comment][] on Lobste.rs
+while I was waiting for a plane — this section of the post is an expanded
+version of that comment.
+
+[comment]: https://lobste.rs/s/fwhuz5/my_lisp_journey_1_getting_started_with#c_ebhvzq
#### Packages
+
+We often see questions in `#clnoobs` that look something like: "How do I export
+a class from a package"? Questions worded like this are a sign of a very common
+misunderstanding about what packages in Common Lisp *actually are*.
+
+A package in Common Lisp is **a container for symbols**. That's it. They're
+a way to group related names (symbols) together so you don't have to do the
+miserable prefixing of every name with `mylibrary-...` like you need to do in
+Emacs Lisp or C to avoid name clashes.
+
+You don't export a class from a package, you export a *symbol*. You don't
+import a function, you import the *symbol* it's attached to. This sounds
+pedantic, but is important to keep clear in your head as you start using the
+package system. If you're not clear on what exactly a symbol *is*, I wrote
+a [separate post][symbols] just about symbols which you might find helpful.
+
+Another major tripping point for new people is the relationship between packages
+and files. Or, rather, the completely *lack* of any relationship in Common
+Lisp.
+
+In many languages like Python, Java, or Clojure, a file's package and its
+location on the hard drive are tied together. For example: when you say `import
+foo.bar.baz` in Python, Python will look for a `baz.py` inside the `foo/bar/`
+folder (it's a little more complicated than this, but that doesn't matter for
+this example).
+
+In Common Lisp, this is not the case. **Files and packages are completely
+orthogonal in Common Lisp.** You can have many files that all work in the same
+package, or one file that switches between many packages, or even create or
+modify packages at runtime.
+
+This gives you maximum flexibility to work however you want. For example: in my
+[Prolog VM][temperance] most of the packages are each defined in their own file,
+much like you would do in modern languages. But the `temperance.compiler` TODO
+package is pretty large (the compiler is the most complicated part of the code)
+and so I split it into [a series of separate files][temperance-compiler], each
+one dealing with a single pass of the compiler, which all work in the same
+package.
+
+So if files and packages aren't related, the next question is: how does Common
+Lisp know where to *find* anything on disk when it comes time to load the code?
+
+[symbols]: TODO
+[temperance]: TODO
+[temperance-compiler]: TODO
+
#### Systems
+
+A system in Common Lisp is a collection of serveral things:
+
+* The code.
+* A description of how to load that code.
+* A list of other systems this system depends on, which need to be loaded prior
+ to loading this one.
+* Some metadata like author, license, version, homepage, etc.
+
+The Common Lisp language itself has no knowledge of systems. If you look at
+chapter TODO of CLtL2 you'll see that it was imagined that each library author
+would write their own custom file to load their code.
+
+Of course, since Common Lisp gives you the power to abstract almost anything,
+people eventually abstracted the process of loading Common Lisp code.
+
+ASDF is a Common Lisp library bundled with most (all?) modern implementations
+which handles defining and loading systems. The name ASDF stands for "Another
+System Definition Facility", so as you might guess there have been several other
+such libraries. ASDF is the one everyone uses today.
+
+ASDF standardizes the process of defining a system into something like this:
+
+* The system definition(s) for a project called `foo` would be in a file named `foo.asd`.
+* Each system is defined with a `(defsystem ...)` form inside this file.
+
+We'll talk more about what a "project" is shortly. Note the extension of the
+file is `asd`, not `asdf`, which is a little confusing, but was probably chosen
+to work in environments with three-letter-extension limits.
+
+The [ASDF manual][TODO] is the definitive resource for the syntax and semantics
+of `defproject`, but can be a little heavy to read if you're just getting
+started. Another way to get started is to read some `.asd` files of some
+small-to-medium sized open source projects and see how they handle things.
+
+Systems and packages are orthogonal in Common Lisp. Some systems (like small
+libraries) will define exactly one package. Some systems will define multiple
+packages. Rarely a system might not define any new packages, but will use or
+add to an existing one.
+
+For example:
+
+* My directed graph library [cl-digraph][] contains a system called `cl-digraph`.
+* That system has a description of how to load the code, which lives in the [`cl-digraph.asd`][cl-digraph-asd] file.
+* As part of loading the system it will load the file [`packages.lisp`][cl-digraph-packages], which creates a package called `digraph`.
+
+Even though ASDF standardizes some aspects of system definition, it still gives
+you plenty of flexibility. As you read projects by different authors you'll
+encounter different ways of organizing systems — this can be a little
+overwhelming at first, but it means you can organize a system in the way that
+works *best for that system*, which is really nice once you've got some
+experience under your belt.
+
+One example of this is how people define packages for their systems. There are
+a couple of common ways to do this you'll see in the wild:
+
+* A single `package.lisp` file which contains all the definitions for all the
+ packages in the project, and gets loaded before all other files. This is the
+ strategy I usually prefer.
+* Each file defines its package at the top of the file, much like you would in
+ Clojure or other modern languages. Care is taken in the system definition to
+ load the files in the correct order so that each package is defined before it
+ is ever used.
+
+So to recap: a system is a collection of code and a description of how to load
+it, a list of its dependencies, and some metadata. Now let's move up one level
+higher to the final layer of structure you need to know about.
+
+[cl-digraph]: TODO
+[cl-digraph-asd]: TODO
+[cl-digraph-packages]: TODO
+
#### Projects
+
+A project in Common Lisp is not an official term defined anywhere that I know
+of, but is a word that's generally used to mean something like a library,
+a framework, an application, etc.
+
+A project will usually define at least one system, because systems are where you
+describe how to load the code, and if a project didn't define a system how would
+you know how to load its code? My cl-digraph library mentioned above is
+a project that defines *three* systems:
+
+The `cl-digraph` system contains the actual data structure and API. It has no
+dependencies.
+
+The `cl-digraph.test` system contains the unit tests. It depends on the
+`cl-digraph` system (because that's the code it's going to test) and the `1am`
+system (a unit test framework). I made this a separate system because it allows
+users to load the main code without also having to load the unit testing
+framework if they're not going to be running the tests.
+
+The `cl-digraph.dot` system contains code for drawing the directed graphs to
+image files with [Graphviz][TODO]. It depends on the `cl-digraph` system and
+the `cl-dot` system (the Graphviz bindings). I made this a separate system
+because it allows users load the main code without also having to load the
+Graphviz bindings if they don't care about drawing.
+
+If I were writing this project today I'd use a forward slash in the system names
+instead of a period (e.g. `cl-digraph/test`), because ASDF has some nice [extra
+support][TODO] for that. I just didn't know about it at the time, and don't
+want to break backwards compatibility now.
+
+We saw how Common Lisp has no concept of a system — that concept comes from
+ASDF. Similarly, ASDF has no concept of the internet or of reaching out to
+somewhere to download things. ASDF assumes you have somehow acquired the
+systems you want to load and stored them on your hard drive, perhaps by sending
+a check to an address and receiving a copy of the code on floppy disk, as many
+of my old Lisp books offer in their final pages.
+
+Quicklisp is another library that works on top of ASDF to provide the "download
+projects from the internet automatically if necessary" functionality that people
+expect in the modern world. So when you say `(ql:quickload :cl-digraph)` you’re
+asking Quicklisp to download cl-digraph (and any dependencies) if necessary, and
+then hand it off to ASDF to actually load the code of the `cl-digraph` system.
+Unlike ASDF, Quicklisp is relatively new in the Common Lisp world (it's only
+about eight years old) and so is not bundled with any modern Lisp
+implementations that I know of (yet?), which is why you need to install it
+separately.
+
+#### Recap
+
+Here's a quick recap of the different layers of project structure you'll
+encounter in Common Lisp:
+
+* **Files** are files on your hard drive.
+* **Packages** are containers of symbols. They are orthogonal to files.
+* **Systems** are collections of code, instructions on how to load this code,
+ dependency lists, and metadata. They are orthogonal to packages.
+* **Projects** are high-level collections of... "stuff" such as code,
+ documentation, maybe some image assets, etc. They are (mostly) orthogonal to
+ systems (are you seeing a trend here?).
+* Common Lisp itself knows about files and packages.
+* ASDF adds systems.
+* Quicklisp adds the internet.
+
#### Naming Conventions
+One more small thing we should talk about before moving on is how to name
+things. This section is only my personal opinion, so some other folks will
+probably disagree. You should consider their arguments and your own needs and
+decide for yourself.
+
+You can name your packages anything you want, but your users will thank you if
+you follow a couple of simple rules. First, don't do the overly-verbose style
+of naming packages like `com.stevelosh.cl-digraph.digraph`. Unfortunately
+Common Lisp doesn't have a portable way to add package-local nicknames, so
+naming your packages like this means that users will have to type that entire
+giant name whenever they wanted to refer to a symbol without importing it, like
+`(com.stevelosh.cl-digraph.digraph:successors graph vertex)` instead of
+`(digraph:successors graph vertex)`. This is just as miserable to read as it is
+to write.
+
+On the flip side: don't try to be overly clever and use a one or two letter name
+for your packages. Common Lisp's packages are in a global namespace, and if
+more than one library tries to claim the same two-letter name it causes
+problems. An example of this is Bordeaux Threads which has a package called
+`bt`. It's too late to change this now, but try to avoid such easily-clashing
+names when making new packages.
+
+Another question that comes up is: "I see a lot of projects called `cl-whatever`
+— should I name my projects with the `cl-` prefix?". Some Common Lispers hate
+the `cl-` prefix and never use it. I personally prefer this style for two
+specific cases:
+
+* A Common Lisp wrapper around something else, like `cl-cudd` (not written by
+ me), which is a set of bindings to the TODO C++ CUDD library.
+* A Common Lisp implementation of a well-known data structure or protocol like
+ `cl-digraph`.
+
+Not using the `cl-` prefix in this case would feel confusing to me, because
+you'd just be calling the project `cudd` which would be easy to confuse with the
+actual library itself.
+
+For anything that's not one of these two cases I prefer to come up with a unique
+name instead. Sometimes it's a pun or just something that sounds unique — the
+critical part is that it doesn't conflict with anything already in Quicklisp.
+
+When you *do* use the prefix, the next question is what parts of the project get
+the prefix. This comes down to personal preference. Here are my personal
+rules:
+
+* The project, Mercurial/Git repository, etc have the `cl-` prefix (e.g.
+ `https://github.com/sjl/cl-digraph`).
+* The system(s) should have the same name as the project, so it also gets the
+ prefix (e.g. `cl-digraph`).
+* The packages are going to be typed frequently, and we're *firmly* in Common
+ Lisp territory at the point where we're defining packages, so it's safe to
+ drop the prefix for packages (e.g. `digraph`).
+
+Other people (including my past and future selves) might disagree on some of
+these conventions. Feel free to take them with a large grain of salt.
+
### Common Libraries
Common Lisp doesn't have a *large* of a community as some newer languages, but
@@ -769,7 +1018,7 @@
portable Common Lisp ten or twenty years ago can still run just fine.
In this section I'll give you a quick overview of some of the more popular
-libraries you might encounter as you learn the language. You don't have to use
+libraries you might run into as you learn the language. You don't have to use
all of them, of course, but it's helpful to have some idea of what's available.
#### Alexandria