Lisp: more is less

There’s recently been a resurgence of interest in Lisp (specifically, Clojure) in my neighborhood. I have some kind of a fondness for Lisp, but I think some of its characteristics make it poorly suited for use in large programming efforts. Indeed many of the unique properties that Lisp devotees (myself once included) tout over other languages, like homoiconicity, make Lisp into an unweidly, conceptual sledgehammer that’s often out of scale with the problem being solved.

I liked Lisp

At some point in college, I began following a charismatic AI professor around who was very knowledgeable and enthusiastic about Lisp. He described it in a way that made it seems like some sort of remarkably powerful, higher form of expression. So I dove in and started doing many of my projets in Common Lisp.

I eagerly read through ANSI Common Lisp, Practical Common Lisp, and SICP; all great reads. I watched talks that sold me on Rich Hickey’s vision of how Clojure tackles modeling reality with concurrency primitives. I wrote a basic robot driver and a shitty blog framework in Clojure. I strove to grok code-as-data-data-as-code, metacircular evaluators, hygenic macros; the whole enchilada.

And frankly, I got a lot of mileage out of that pursuit. SICP in particular taught me a ton about good abstraction, programming structure, and (unexpectedly) numerical analysis. Rich Hickey’s talks are fascinating works of thought, and they’re engaging as hell. There is no shortage of entertaining philosophical content in the Lisp community.

But that doesn’t make Lisp an appropriate tool for large software projects.

Remember “less is more?”

Lisps, for those unacquainted, have the unique property of homoiconicity. This means, among other things, that Lisps offer the ability to write very powerful macros which put users on roughly even footing with language implementors. In fact, most builtin functionality in Lisps are implemented as macros.

At first blush this may sound enticing, but it’s actually a property that gets progressively uglier as the number of active programmers goes up.

Most programmers are not (nor should be) language designers

A smart programmer is not necessarily an empathetic language designer; they are occupations that require different skillsets. Giving any programmer on your team the ability to arbitrarily extend the compiler can lead to a bevy of strange syntax and hard-to-debug idiosyncrasies. Introducing macros increases the conceptual surface area of a language by an unbounded amount, and it defeats compactness, which I’ve seen is an important characteristic that aids programmers in quickly and effectively understanding a code base.

Let me be clear: I think Lisp is more powerful language than, say, Python or Java. That’s what I’m arguing is its downfall. Using the simplest tool that provides enough convenience to get a job done enjoyably (which Python often does for me very nicely) has all kinds of peripheral benefits. Ones that Lisp’s power may exclude itself from, like…

Goodbye, static analysis

One of the most helpful programming tools I’ve discovered in the past few years has been in-editor static analysis tools, e.g. syntastic. Especially when developing in an interpreted language like Python, having syntactic analysis tools on hand to catch bugs before they happen and patrol code-quality in CI builds is a massive boon.

With macros, or any form of extraordinary dynamic language abilities, many of these benefits get thrown out the window. How can a static analysis tool keep up with a language that’s being arbitrarily extended at runtime? The prospect is daunting.

Unix wisdom

Benefits like static analysis and compactness are consequences of using the simplest thing that works with a reasonable amount of convenience. Obviously “a reasonable amount” is tough to pin down and I won’t attempt to do it here, but suffice to say that the only convenience I see on the margin between a Lisp and, say, Python, is a little bit of added syntactic sugar.

Projects like Korma sure look cool, and I’m sure are a ball to write, but is there really such an advantage over something like sqlalchemy in terms of readability? An advantage great enough to abandon automated sanity checks and to introduce obscure macro-based stacktraces? I don’t think so.

Perhaps it’s possible to write a linter that is informed by macro definitions within the codebase, but that’s certainly a more challenging task than just having to internalize a published language spec.

Conceptual clutter

Popular Lisps these days seem to be, optimistically, more conceptually diverse or, pessimistically, more conceptually undecided than other languages in popular usage.

Unix is in part such an effective environment because its concepts are easy and consistent. To see everything as a file, to work in terms of text streaming through pipes; these are straightforward ideas that govern the entire system. OOP is widely-used and easily comprehended because it is a fairly simple way of modeling reality that is compatible with how human beings do it.

By comparison, Common Lisp and Clojure (which rejects OOP but does, to its credit, impose some generally useful frameworks, e.g. the seq interface) seem to fall prey to the same affliction that I see in Haskell, Ruby, and Scala: they give you varied and often overlapping options for how to model a certain process or piece of state.

In Clojure, if I want to define a symbol there are nine different ways of doing so. In Python, there are three (=, def, class). By the way, those nine creational procedures only apply to Vars; there are also Refs, Agents, and Atoms with their own assortment of creation semantics; all of which are different primitive ways to reference pieces of data in Clojure.

Clojure claims to include these language features as a way to mitigate the complexity of parallelism; frankly, I’ve never found threading or interprocess communication to be any sort of conceptual bottleneck while working on some fairly complex distributed systems in Python.

Simplicity buys a lot

I attribute a lot of Python’s success to its devotion to simplicity; Python wants the conceptual machinery for accomplishing a certain thing within the language to be obvious and singular. This makes it easy for experienced Python users to quickly grok foreign code, and it makes it easy for newbies to master the language quickly.

The relatively few conceptual mechanisms in Python may not cover every single problem domain as well as picking the perfect one out in Clojure case-by-case, but they work well enough for most things to be make Python a powerful general-purpose programming language that is very effective at quickly communicating systems to programmers. This property is paramount in large engineering efforts.

Not all Lisps

This sort of excessive complexity isn’t inherent in Lisps (certain types of Scheme are wonderfully simple languages) but it does seem common to popular variants.

On the margin

On the margin, Lisp’s additional power over other common languages like Python and Java doesn’t buy much practical benefit. It does, however, impose significant costs in terms of programmer comprehension, code complexity, and automated tooling like static analysis.

I believe in Lisp and its communities as worthwhile resources and valuable for personal intellectual growth, but as, an engineer, I think it’d be irresponsible to choose Lisp for a large-scale project given the risks it introduces.

Less is more (as long as “less” is sufficiently convenient).