Introduction

Hello again, sorry for the delay, but getting Postabon (my Lisp-based startup that helps you find and share in-store discounts) off the ground over the last few weeks (both business and technology wise) has been consuming a lot of my free time :-D

This post is a simple explanation of how to setup a Common Lisp webapp. This isn't the only way to get things done, but actually building a deployable app is something that often gives Lisp newbies a bit of trouble and these are a general set of techniques that I've been using with some success for the last few years. The key trick is that everything should be as self contained as possible and not overly rely on anything outside of the app (i.e., the app should include all its own libraries). If you want to cut straight to the meat, checkout the GitHub repo - it's engineered so it can easily be 'forked' and modified to be the basis of a project.

Choosing a Lisp Implementation

Before we get started, you'll have to install a Common Lisp implementation. Out of the free ones, I've had the best experience with Steel Bank Common Lisp (SBCL) -  but I've been hearing a lot of good things about Clozure Common Lisp (CCL) lately. This example is tested with, and geared towards, SBCL - although I suspect with a one line modification in the startup script it should work with CCL as well.

The one major caveat with SBCL is that the threading support is a bit annoying - it only really works with Linux (I've occasionaly had some mixed success with other *nixs like Mac OS X or FreeBSD, but I would recommend against it). Threading is a requirement to hope to get  any semblance of reasonable performance out of a blocking webserver but, as of a few weeks ago, the binary distribution of SBCL has threading disabled (although, there seem to be plans to change this behaviour back to the more sensible default of having threads enabled for Linux in the near future).

The unfortunate consqeuence of this decision is that you'll have to install SBCL from source with threading enabled. Ironically, to compile SBCL from source, you must first have a working Common Lisp installation ;-) So, what we'll end up doing is installing an SBCL binary from somewhere and then using this crippled binary to bootstrap a proper SBCL installation from source (with threading enabled).

Install SBCL Binary

So, the first step is to install a binary SBCL. If it's possible, I'd highly recommend you use your distribution's package manager (yum, apt, emerge, etc) to install SBCL - otherwise you can install it from http://www.sbcl.org/platform-table.html (using the INSTALL document).

Once you install a binary SBCL, test it out by running 'sbcl' from the command line. When greeted by an '*' prompt (called a REPL), verify that (+ 2 2) returns '4'. You can also try typing '*features*' at the REPL - if the resulting list contains ':SB-THREAD' then your binary installation contains threading support out of the box (and you can skip the source installation). If, it doesn't you'll need to install SBCL from source.

Install SBCL from Source (only after Binary)

To install SBCL from source, first we must get a copy of the source. I personally prefer Git to SVN, so I usually get a copy of the SBCL source tree from Andreas Fuchs' Git repository with a command like:

git clone git://git.boinkor.net/sbcl

Using Git is probably overkill for a casual user (since it pulls in the entire commit history, which is ~50MB), but I prefer it. After you download the source: 'cd' into the SBCL directory and create a file called 'customize-target-features.lisp' with the following contents:

The run:

sh make.sh

sudo sh install.sh

Now that you have a customized SBCL installed from source, remove the binary copy you installed (if you used your distro's package manager) and ensure your PATH contains /usr/local/bin/. Then, run SBCL and verify that it's working and that the *features* includes :SB-THREAD.

Run the Webapp

Don't worry - I'll explain how the web app works in a moment, but now that you have a Lisp compiler installed you can try to run it. Start by grabbing a copy of the code with:

git clone git://github.com/smanek/trivial-lisp-webapp.git

Then you can 'cd' into trivial-lisp-webapp/scripts and run:

$ ./startserver.sh

After a bunch of stuff scrolls down the screen, you should see a line that says:

Webserver started on port 8080.

Verify that you can visit your site on http://localhost:8080, and then you're good to go.

Webapp Structure

There are a few noteworthy directories and files in the trivial-lisp-webapp:

• asdf-systems: This is the directory that Another System Definition Facility (ASDF) uses to look for packages it wants to load. Every package that is loaded (including trivial-lisp-webapp) has a soft link to its '.asd' file which describes the package, its files, dependencies, and so on.
• aux: This directory contains a bunch of ASDF enabled libraries that our webapp uses. It's generally more reliable to deploy libraries you want with your webapp than try to depend on libraries being installed system-wide. The two most notable libraries we use are Edi Weitz's Hunchentoot webserver and html-template - most of the other libraries are just dependencies for these two.
• document-root/
  • static/: Static files that are served up by Hunchentoot (notably CSS, Javascript, and images). Anything placed in this directory is automatically served.
  • templates/: A series of templates that are compiled to generate HTML pages that users see. In the 'trivial' web app, we're basically just using html-template for server-side includes - but see the html-template documentation for instructions on how to do more complex things with them.
• scripts: Scripts contains any scripts we need. In this case it simply contains 'startserver.sh', but in the future you could add scripts to analyze log files, update libraries, do backups, or anything else.
• src/: Finally, the actual Lisp source code of the webapp. The files are:
  • init.lisp: A small file that bootstraps the webapp. It configures ASDF, loads dependencies, etc. when loaded by startserver.sh.
  • webapp.asd: The file that tells ASDF how to load our webapp (what dependencies are required, what should be exported, what order to load files in, etc).
  • global.lisp: A few global settings, such as paths.
  • misc.lisp: Any utility functions go here
  • pages.lisp: Setups handlers for any pages in the webapp
  • control.lisp: Functions to start and stop the webapp

A more substantial webapp might have some more functionality for a test suite, advanced configuration, easier deployment, backups, and so on - but this makes a pretty good starting point.

Next Steps

Now that you've got the basics down, it's time to get hacking! Try figuring out how to add a second page to the site (hint: modify pages.lisp and add a template to document-root/templates). Make a page that accepts GET/POST parameters and saves state across visits (see the hunchentoot documentation). You'll probably want to use emacs/slime/paredit mode, and there are plenty of great tutorials out there on doing so (I'm personally a bit partial to Marco Baringer's screencast - although it's getting a bit long-in-the-tooth these days).

In production, I run my webserver (via a script similar, but slightly more powerful than, startserver.sh) under the GNU Screen multiplexor (instead of as a daemon). I also setup a reverse proxy (nginx) in front of Lisp to shield Hunchentoot from malformed requests, provide faster gzip compression, and more resilient error pages.

If people find this helpful, and there's demand for it, I can write another post on how to expand our trivial-lisp-webapp to add persistence via a database. We could eventually even work our way up to building a very minimal MVC framework for Hunchentoot.