Before we get started I just want to say I know you should, theoretically, be able to do all of these in Java. Java, like most languages, is Turing Complete - and therefore is capable of computing every computable function. However, completing these following feats in Java (or any other non-functionalish language) would be much uglier and more difficult (just like it is theoretically possible to write a web application in assembly code; but really, who would want to?). I’m moderately proficient in Java, and I can’t think of any way to do these without resorting to hacks like making pseudo-closures by hiding methods inside objects - and even that isn’t anywhere near as general and flexible as the code presented here.

I plan on discussing three techniques today: sorting (quicksort) on an arbitrary comparison (i.e., first class functions), automatic memoization, and lazy evalution. I wish I could say I invented some of these techniques, but all the code presented here is at least inspired by The Structure and Interpretation of Computer Programming and Paradigms of Artificial Intelligence Programming (both highly recommended). If you’ve read both, you probably won’t learn anything new from this post - but if you haven’t I can at least give you a taste of the great masters’ works.

I’ve chosen to write all my code in Common Lisp, just because that’s what I like. It should be relatively straightforward to translate my code to Python, Ruby, or any other language with first class functions. I’ll also make a point to explain my code so non-lispers should be able to follow along.

Quicksort

I’m assuming if you’re reading this you’re a programmer and you are, or at least were, familiar with Quicksort. As a quick refresher, if you were sorting a list in ascending order you would randomly choose a pivot element and place all the elements less than or equal to the pivot on its left and all the items greater than the pivot to its right. You then recursively call the quicksort function on the left and right sublists. On average, it runs in time, but if you get unlucky choosing your pivots it can be as poor as .

My implementation of a quicksort is extremely naive (always choosing the first element as the pivot, doing a lot of unnecessary memory allocation (consing), iterating over the list twice to split it on the pivot, etc.) but it will do for pedagogical purposes.

So, first the code:

1
2
3
4
5
6
7
8

(defun quicksort (data comparison)
  (if (null (cdr data))
      data
      (let* ((pivot (first data))
            (predicate (lambda (x) (funcall comparison pivot x))))
	(append (quicksort (remove-if predicate (rest data)) comparison)
	         (list pivot)
		 (quicksort (remove-if-not predicate (rest data)) comparison)))))

For you non-lispers, that defines a function (named quicksort) that takes two parameters, “data” and “comparison,” and sorts the data so that the comparison function holds true for any two adjacent elements in “data.” For example, if data = (1 2 3 4) and comparison = ‘<’, then the data is already sorted since ,, and .

The nice part about this function is that it can sort the data by any arbitrary criteria. For example, it can sort strings alphabetically, numbers in ascending or descending order, or numbers by absolute value. Here’s a few quick examples:

CL-USER> (quicksort (list 8 1 3 9 2) #'>;)
(9 8 3 2 1)
 
CL-USER> (quicksort (list 8 1 3 9 2) #'<)
(1 2 3 8 9)
CL-USER> (quicksort (list "Tiger" "Grape" "Red" "Apple" "Zebra" "Cat" "Bear" ) #'string<)
("Apple" "Bear" "Cat" "Grape" "Red" "Tiger" "Zebra")
CL-USER> (quicksort (list -32 3 21 -12 50) #'(lambda (a b) (< (abs a) (abs b))))
(3 -12 21 -32 50)

In a non-functional language it’s easy to write a sort function that works on any one of those criteria, but it’s far more difficult to write one general enough to sort in any way (even ways you don’t originally anticipate).

Memoization

Memoization isn’t as generally well known as quicksort, but the idea is simple: cache the results of previous function calls so that you don’t have to recompute values you have already computed. The canonical poster child for memoization is the Fibonacci sequence.

The Fibonacci sequence is defined by the recurrence relation:

As you might guess, it’s heavily recursive and often calls itself with the same arguments. For example, in order to computer the 40th Fibonacci number naively you have to computer the 2nd Fibonacci number 63,245,986 times, and the 5th Fibonacci number 14,930,352 times. A naive lisp function to compute the nth Fibonacci number might be:

1
2
3
4

(defun fibonacci (n)
  (if (< n 2)
      n
      (+ (fibonacci (1- n)) (fibonacci (- n 2)))))

To see it in use, we can use the Common Lisp debugging function trace to show all the recursive function calls and the benchmarking function time as a gross performance measure.

CL-USER> (time (fibonacci 40))
Evaluation took:
  6.385 seconds of real time
  5.000313 seconds of total run time (4.968311 user, 0.032002 system)
  78.31% CPU
  11,673,458,046 processor cycles
  0 bytes consed
102334155
CL-USER> (trace fibonacci)
(FIBONACCI)
CL-USER> (fibonacci 5)
  0: (FIBONACCI 5)
    1: (FIBONACCI 4)
      2: (FIBONACCI 3)
        3: (FIBONACCI 2)
          4: (FIBONACCI 1)
          4: FIBONACCI returned 1
          4: (FIBONACCI 0)
          4: FIBONACCI returned 0
        3: FIBONACCI returned 1
        3: (FIBONACCI 1)
        3: FIBONACCI returned 1
      2: FIBONACCI returned 2
      2: (FIBONACCI 2)
        3: (FIBONACCI 1)
        3: FIBONACCI returned 1
        3: (FIBONACCI 0)
        3: FIBONACCI returned 0
      2: FIBONACCI returned 1
    1: FIBONACCI returned 3
    1: (FIBONACCI 3)
      2: (FIBONACCI 2)
        3: (FIBONACCI 1)
        3: FIBONACCI returned 1
        3: (FIBONACCI 0)
        3: FIBONACCI returned 0
      2: FIBONACCI returned 1
      2: (FIBONACCI 1)
      2: FIBONACCI returned 1
    1: FIBONACCI returned 2
  0: FIBONACCI returned 5
5

Not too bad … but even computing the 5th Fibonacci number required a lot of repeated function calls (and the 40th required approximately 102,334,114 repeated calls).

Fortunately, we can use the power of functional languages to write a meta-function (a function that operates on functions) that will automatically and transparently transform any function into a memoized-function. Therefore, for example, every time (after the first) that we try to look up the nth Fibonacci number, the result is just looked up in a hash table instead of being recomputed. In most real world applications (web-apps, business logic, etc) we would also add some logic to ‘expire’ answers in the cache after a certain amount of time has elapsed (or certain triggers have occurred).

5
6
7
8
9
10
11
12
13
14
15

(defun make-memoize-fn (fn)
  (let ((cache (make-hash-table)))
    #'(lambda (input)
      (multiple-value-bind (output found-p)
	  (gethash input cache)
	(if found-p
	    output
	    (setf (gethash input cache) (funcall fn input)))))))
 
(defun memoize (fn)
  (setf (symbol-function fn) (make-memoize-fn (symbol-function fn))))

Then, after just running:

CL-USER> (memoize 'fibonacci)
#<CLOSURE (LAMBDA (INPUT)) {AE6A65D}>

Our Fibonacci function is automatically memoizing - no rewriting. From now on, each call to the Fibonacci function is just looked up in the hash table. If the result is there, it just returns it without ever doing any work. If the result isn’t cached in the hash table, it is computed and added to the hash table for future reference. All completely transparently, without ever rewriting Fibonacci.

The performance difference is significant:

CL-USER> (fibonacci 5)
  0: (FIBONACCI 5)
    1: (FIBONACCI 4)
      2: (FIBONACCI 3)
        3: (FIBONACCI 2)
          4: (FIBONACCI 1)
          4: FIBONACCI returned 1
          4: (FIBONACCI 0)
          4: FIBONACCI returned 0
        3: FIBONACCI returned 1
      2: FIBONACCI returned 2
    1: FIBONACCI returned 3
  0: FIBONACCI returned 5
5
CL-USER> (untrace fibonacci)
T
CL-USER> (time (fibonacci 40))
Evaluation took:
  0.000 seconds of real time
  0.000000 seconds of total run time (0.000000 user, 0.000000 system)
  100.00% CPU
  19,174 processor cycles
  0 bytes consed
102334155
CL-USER> (time (fibonacci 40))
Evaluation took:
  0.000 seconds of real time
  0.000000 seconds of total run time (0.000000 user, 0.000000 system)
  100.00% CPU
  2 processor cycles
  0 bytes consed
102334155

Even the first time it computed the 40th Fibonacci number, it was over 600,000 times faster. And the second time it only took 2 processor cycles (far less than I, personally, expected). And the truly amazing part is that we can make any other function memoizing just by calling the meta-function memoize on it. No having to rewrite any code to implement caching!

Lazy Evaluation

With Lazy Evaluation, an expression isn’t evaluated as soon as it is created; instead it is delayed until that expression’s value is needed. One of the neat tricks provided by lazy evaluation is the ability to work with infinite sets - for example, you can literally have a variable which holds all the natural numbers, all the primes, or all the Fibonacci numbers. This sometimes provides rather surprising results - such as the ability to exhaustively search over infinite spaces.

I’ll settle for just writing a bare bones lazy evaluation system. I won’t provide many of the utility functions (map, iterators, etc.) that could prove useful in the real world, but I will build enough so that we can, for example, have a variable which holds all the primes (lazily evaluated, of course).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

(defmacro make-pipe (head tail)
  `(cons ,head (lambda () ,tail))) ;;the lambda is delaying the evaluation
 
(defun head (pipe)
  (car pipe))
 
(defun tail (pipe)
  (if (functionp (cdr pipe))
      (setf (cdr pipe) (funcall (cdr pipe)))
      (cdr pipe)))
 
(defun pipe-elt (pipe n)
  (if (zerop n)
      (head pipe)
      (pipe-elt (tail pipe) (1- n))))
 
(defun integers (&key (start 0) (end nil))
  (if (or (null end) (<= start end))
      (make-pipe start (integers :start (1+ start) :end end))
      nil))

That should be enough to get us started. The make-pipe macro creates a lazily evaluated list (a ‘pipe’, as per Norvig) whose last element is a closure over a function that can be evaluated to get another actual element (we use the closure to delay evaluation, up until we call the lambda). We then define two convenience functions: pipe-elt, which will evaluate a pipe up to a particular element (e.g., compute the first N integers) and integers which can create a list of (possibly infinite) integers.

A few simple examples of our basic system at use:

CL-USER> (defvar *short-lazy* (integers :start 0 :end 5))
*SHORT-LAZY*
CL-USER> *short-lazy*
(0 . #<CLOSURE (LAMBDA #) {B1D0D05}>)
CL-USER> (pipe-elt *short-lazy* 3)
3
CL-USER> *short-lazy*
(0 1 2 3 . #<CLOSURE (LAMBDA #) {B42B795}>)
CL-USER> (pipe-elt *short-lazy* 6)
NIL
CL-USER> *short-lazy*
(0 1 2 3 4 5)
CL-USER> (defvar *infinite-lazy* (integers :start 0))
*INFINITE-LAZY*
CL-USER> (pipe-elt *infinite-lazy* 5)
5
CL-USER> *infinite-lazy*
(0 1 2 3 4 5 . #<CLOSURE (LAMBDA #) {B71B7E5}>)
CL-USER> (pipe-elt *infinite-lazy* 20)
20
CL-USER> *infinite-lazy*
(0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
 . #<CLOSURE (LAMBDA #) {B7C8DD5}>)

We could get the hundredth, thousandth, or trillionth entry from *infinite-lazy*, and they would all be there.

Well, so far this is interesting - but not really useful. But we’ve laid the foundation for the really cool stuff. My goal for this exercise will be to create an infinite list of all the primes using the Sieve of Eratosthenes. Basically, by applying filters to the closure on the end of the lazy list (i.e. function composition) that ‘cross off’ multiples of numbers already evaluated, we can generate an infinite list of primes (I used this very trick on several Project Euler problems).

21
22
23
24
25
26
27
28
29
30

(defun filter (predicate pipe)
  (if (funcall predicate (head pipe))
      (make-pipe (head pipe) (filter predicate (tail pipe)))
      (filter predicate (tail pipe))))
 
(defun lazy-eratosthenes (&optional (pipe (integers :start 2)))
  (make-pipe (head pipe) 
	     (filter #'(lambda (p) 
			 (not (zerop (mod p (head pipe)))))
		     (eratosthenes (tail pipe)))))

Now, we can simply create, and play with, an infinite list of primes:

CL-USER> (defvar *lazy-primes* (lazy-eratosthenes))
*LAZY-PRIMES*
CL-USER> *lazy-primes*
(2 . #<CLOSURE (LAMBDA #) {B60E975}>)
CL-USER> (pipe-elt *lazy-primes* 5)
13
CL-USER> *lazy-primes*
(2 3 5 7 11 13 . #<CLOSURE (LAMBDA #) {B07FB45}>)
CL-USER> (pipe-elt *lazy-primes* 100)
547
CL-USER> *lazy-primes*
(2 3 5 7 11 13 17 19 23 29 31 37 41 43 47 53 59 61 67 71 73 79 83 89 97 101 103
 107 109 113 127 131 137 139 149 151 157 163 167 173 179 181 191 193 197 199
 211 223 227 229 233 239 241 251 257 263 269 271 277 281 283 293 307 311 313
 317 331 337 347 349 353 359 367 373 379 383 389 397 401 409 419 421 431 433
 439 443 449 457 461 463 467 479 487 491 499 503 509 521 523 541 547
 . #<CLOSURE (LAMBDA #) {B3A785D}>)

So, there you have it folks. Next time some blub programmer asks you what you need first class functions and closures for, pull out some of these. While I didn’t develop any of these three ideas beyond the level of parlor-tricks, it doesn’t take much imagination to extend these simple examples into the real world.

To give you some idea of what I’m talking about take a look at these examples:

Schwartzian Transform in Perl:

1
2
3
4

@sorted = map  { $_->[0] }
          sort { $a->[1] cmp $b->[1] }
          map  { [$_, foo($_)] }
               @unsorted;

A Common Lisp Quine (extra whitespace for readability):

1
2

((lambda (x) (list x (list 'quote x)))
 '(lambda (x) (list x (list 'quote x))))

I have a few better examples in mind too (clever regexes, pointer magic, etc), but I don’t want to steal anyone’s real entry The code you submit doesn’t have to be of your own invention, but bonus points if it is. Any language and any topic (obfuscated code, polyglots, numerical methods, etc.) are fair game - if I don’t understand it, I’ll ask someone smarter than me for help.

I’ll only consider comments submitted by midnight Wednesday, and I’ll announce winners within a day of that.

If you feel like entering, please be sure to leave some way for me to contact you so I can get your invite to you if you win.

My webserver was standing up to the load just fine, I just did something stupid.

As some background, I almost always leave a terminal multiplexer (screen) running on my vps. It eats a few megs of memory, but I have enough to spare that it isn’t an issue.

And sometimes, when I’m bored, I’ll run a command like:

$tail -f apache_access_log.log| grep 'GET article-url'

It basically shows me real time access to a particular url - every time someone reads that article I see their user-agent string, referrer, and a bit of other useful information.

Of course I have real analytics software too, but sometimes it’s just fun to watch the hits scroll by - it’s like a pretty matrix-esque screen saver.

Well, I was doing that this morning, and then I left my computer. I disconnected my screen session and ssh, and went about life in the real world. When I returned ~6 hours later, I noticed my brand new blog was down. Uh-oh.

After connecting to a console to investigate (ssh was timing out), I was greeted by this beautiful sight:

#uptime
[...]load average: 20.84, 22.16, 23.84

For the record, that’s bad. I don’t think I’ve ever seen a load that high on a single cpu machine …

I was particularly surprised since I had spent a few hours the previous day setting up the site (caching, php compiling, tweaking apache/mysql, etc) to handle slashdot level loads. I had simulated well over a 1000 concurrent users without a hiccup.

After looking at ps, I realized that my little log trick was the culprit. I killed the job, and load dropped right down to 0.

I’m still looking into why it pegged the CPU so hard though - I’ve left that same command running for hours on a terminal I’m connected to with no discernible effect. My best guess is having that much information going to STDOUT in a disconnected screen session triggered some sort of pathological edge case … I’ll investigate more thoroughly tonight.

Anyways, my apologies to anyone who tried to access the site when it’s down. I’m always the first person to criticize twitter when I hear it’s down (again) - maybe there’s something is to be said about glass houses …

Has anyone else ever made a stupid mistake like this that brought a server to its knees? If so, I’d love to hear about it. I think I need a little pick me up

We all know about the classic texts of computer science (although, we may be a bit behind on our reading). We’ve heard a thousand times how important Knuth’s Art of Computer Programming, Abelson and Sussman’s Structure and Interpretation of Computer Programming, and the Gang of Four’s book on Design Patterns are.

But, I think these are sometime overemphasized - often to the neglect of other great books that have been written in the last few decades. Recently, I moved from Chicago to Cambridge and could only bring a few dozen books with me. I was forced to evaluate which are truly timeless classics, and which are just tutorials.

Now, don’t get me wrong - tutorials are great and do serve their purpose. I often reference Programming Perl or Practical Common Lisp. But they don’t change the way you think - they don’t have an impact which spans disciplines and languages and technologies.

When I sat down and thought about it, many of the books that have had the greatest impact on me as a programmer (and as a thinker in general) aren’t even Computer Science books. These books below often transcend their stated domain and impart a new way of looking at all problems.

• Godel, Escher, Bach: An Eternal Golden Braid

   GEB is ostensibly about the hidden threads that hold together Mathematics, Art, and Music. But Hofstadter goes so much deeper than that. Through a series of Socratic Dialogues, the nature of thought and computation are revealed to us. And, along the way, this book touches on recursion, formal systems, lambda calculus, undecidability, complexity theory, and far more. If you only read one book on this list, it should be this one.

• The New Turing Omnibus

   This book is extraordinary for the sheer breadth of topics it covers. I’ve noticed that many programmers stagnate and grow complacent with their existing tool set. They master a few tricks, and then just plateau - whenever any problem arises, they reach for a technique from their limited repertoire and think it’s the best solution. As the old cliche goes, when the only tool you have is a hammer, every problem looks like a nail.

   The new Turing Omnibus opens your eyes to new worlds. Everything from genetic algorithms, to pushdown automata, to computer vision techniques are fair game. This book doesn’t go into enough depth in any topic to actually teach you how to do anything - but it will teach what is possible, and will help you recognize the best tool for the job. You won’t understand the details of simulated annealing or the simplex algorithm, but when you encounter a new problem you will know the right approach to take, and will having a jumping off point for your research.

• Introduction to Algorithms

   This, to me, is the new Art of Computer Programming. Although it is nice to learn efficient arithmetic algorithms using assembly on a MIX machine, it really isn’t what I’d like to focus on. I’d rather learn, for example, about an extremely clever disjoint set data structure which provides unions (where and is the Ackermann function) in high level pseudo-code. This book showed me a lot of novel techniques for proving an algorithm’s runtime and it has become my bible for elementary algorithms.

• Algebra and Geometry

   First, this isn’t your high school algebra/geometry book (well, I suppose you could have done this in high school, but I didn’t at least). It is focused, primarily, on abstract algebra, linear algebra, and non-euclidean geometry. By the time I read this book, I had already taken courses in all those topics individually. But this book tied up a lot of loose ends and showed me many interconnections between seemingly disparate fields that I hadn’t picked up on on my own. It may be a little too intense for a first foray into higher math thought (at least I would have struggled a lot more, if I wasn’t familiar with so many of the topics going in), but it’s definitely a worthwhile read.

• The Road to Reality

   The first half of this book is, hands down, the most incredible math tutorial I’ve ever seen. In the span of a few hundred pages, the author takes a reader up from elementary algebra and arithmetic all the way through calculus and large swathes of analysis, abstract algebra, number theory, geometry, and basically the better part of an undergraduate math education. After that, Penrose uses your newly minted math skills to teach physics from simple Newtonian mechanics, up through quantum mechanics and relativity, and even touching on some cutting edge work currently taking place on a “theory of everything.” Because of the tremendous amount of ground this book covers, it is extremely difficult to get through. Over the course of about a year, I often found myself giving up on exercises, and just glossing over some of the more intense sections. Reading this book can best be described as attempting to drink from a firehose: if even 20% of what Penrose throws at you sticks, you’ll be a better man for it. The math skills that this book imparts are alone worth the price of admission. Throw in the newfound understanding of all things physical, and this book is bargain for what you put in.

Those are a few of my favorite books, that are off the beaten track. If you have any to add, please leave a note. I’m already behind on my reading, but nothing like a bit of guilt to motivate me

I’ve been playing with Python recently, and a few things have been bugging me. Before I get started, I do have to admit Python does a lot of things right; for all its flaws, I’d much rather work in Python than Java or C++.

That being said, there are a few things I’ve encountered that seem very inconsistent, and are very inconvenient. I had hoped that they were just legacy artifacts and would be fixed in Python 3000 but, for the most part, they seem to be sticking around.

Scoping
First, to show you the expected behavior of lexically scoped variables in Python:

a = "foo"
 
def f():
    print a #a is free, so it refers to the global binding "foo"
 
def g():
    a = "bar"
    f() #Python creates a closure around the function f with a reference to the global a
    print a #Here the global a is shadowed by the local a, which has a value of "bar"
 
>>>g()
foo
bar

As expected, calling the function g prints “foo” and then “bar”.

Now, when we try a slightly more interesting example:

a = "foo"
 
def f():
    print a #Should refer to the global a, declared above
    a = "bar" #Should shadow the global (statically scoped) a
    print a #Should print bar
 
>>>f()
UnboundLocalError: local variable 'a' referenced before assignment

There is no good reason for a language to exhibit this behavior: it’s completely inconsistent with what was demonstrated in the first example. Since ‘a’ is free in the function f the first time we try to print it, it should refer to the global ‘a’ with the value of “foo”.

Rounding errors
Now, maybe I’m just spoiled coming from doing Common Lisp work, but I’m used to programming languages handling numbers right. In Lisp, I don’t have to tell the compiler how big or small my numbers are going to be (although, I optionally can for better performance). They just work. Lisp transparently uses fractions or floating points or complex numbers or integers automatically - and it never makes a rounding error. To me , etc.

Technically, I understand that it is rather difficult to perfectly represent decimal numbers in binary machines, but Lisp (and many other languages) solved these problems 30 years ago. There’s no reason that we should still be struggling with them today - especially in a very high level language like Python.

>>>a = 0.1
>>>a
0.10000000000000001
>>>a - 1/10
0.10000000000000001

’s
Python’s lambdas have been emasculated. I know that lambas are never strictly necessary - anything I can do with a lambda I can do with a named function. And I also know that python provides powerful list comprehensions which serve the purpose of map/filter and lambda more often than not. But, there are often times when I just want a lambda.

The problem is, python’s lambda doesn’t work. You can’t ‘print’ inside a lambda (but you can call another function that can?!) and they can’t span multiple lines. I don’t know if these were purposeful design decisions meant to discourage the use of lambdas (maybe, especially in light of the talk I’ve seen of eventually deprecating reduce, filter, and map) but, in any case, they are annoying.

>>> a = lambda x: print x
SyntaxError: invalid syntax
 
def myPrint(x):
    print x
 
a = lambda x: myPrint(x)
>>>a("Hello World")
Hello World

And here ends my first, relatively mild, rant.