hackinghat.com

Too much choice is challenging, to say the least. Well, it is to me anyway. I get hung up on trying to pick the best choice from a huge list of items and end up doing nothing at all. All too often just making a choice, any choice, would be the best idea. However when you’re writing software and trying to pick from an array of available libraries the penalties of a wrong choice can be amplified later. This penalty comes from having to bet early on your chosen horse, but here’s the kicker, you could well have a lot of capital in the race before you realise you backed a donkey.

For instance, if you choose the wrong core library for the “Next Big Thing” now, you run the risk of tying yourself to an implementation that might have a flaw in it that you can’t fix because it’s not open source or because you don’t have the resources to undo it. So you have to adopt a work-around. Thankfully, most of the time things don’t turn out so bad. But if I stop talking to the donkeys for a minute and listen very carefully I can hear millions of developers happily using Microsoft’s tools and libraries without a care in the world. They don’t have to think about this sort of problem very often because Microsoft thought about it for them and gave them the library as part of the SDK. Those developers are real happy with what they’ve got and who can blame them, it’s like having an older brother who looks out for you. So what if he steals your lunch money, it’s part of his charm.

I don’t want to bash Microsoft here, by the way, I think they have a massively important role to play and I have to admit that some of the more recent tools they have made have been pretty awesome achievements for a gigadollar-company. It seems obvious, to me at least, that the higher Microsoft raises the bar in terms of tools the better it is for pretty much everyone. But the (real?) world outside of Redmond is a little more confusing, I know from (old) experience that when I want to use a SMTP client in Java I’ve got to go through a process of evaluation. In .NET the client that comes free in System.Net.Mail will do just fine, nothing to add. Sure sometimes .NET will sometimes be found lacking, but I’ve been pretty happy with the standard libraries in .NET so far and it’s only when I want something a little unusual that I need to go elsewhere (except for the annoying lack of a built-in Set class but that’s a different story).

Well all I wanted to do this week on my continuing Lisp learning exercise was send a multicast packet from my Common Lisp project. Guess what, network programming in Common Lisp is simply not possible. Gasp. It seems that the Common Lisp standard was set in 1994 and so predated the massive expansion of the internet. So CL projects like trivial-sockets and usocket have been created as CL vendor neutral APIs. Unfortunately, it seems trivial sockets is abandonware so I guess usockets is the way to go. But then, it seems broadcast networking is still work in progress for usocket so it seems fair to say that Common Lisp network programming for multicast will be some time off. In a platform independent way at least.

Even that’s ok. I don’t mind. I’m not producing any world-changing libraries here. Hell I might even pitch in on the usocket effort to help out. But right now I’m only messing about trying out some ideas to see what might fly, it’s what Lisp is good at after all. So I’m happy to use whatever multicast support SBCL has. Crap. They don’t seem to have implemented all the socket options needed . Then I read this very illuminating 2005 paper that lays the CL network programming problem out beautifully but when you scroll down to the “Proposed Solution” it says:

… To be written

If you stop talking to your own donkeys for a second and listen you might be able to hear me laughing. Very faintly.

Ok, I know what I’ll do. I’ll take a look at what I need to do make a foreign-function interface work. Yeah, that’s good, I’ll do that. Crap. I’ve got to chose between UFFI & CFFI. So I already know that I’m going to have to take a look at both to see what’s going on. A short while later and I’m starting to believe that CFFI is the cleaner of the two but then I stumble upon Verrazano and SWIG and my head starts to spin.

At this point it’s hard to be cross or frustrated about anything. This isn’t really anything that I can really blame CL for, it’s just how it is, if time-travel had of been available when the 1994 ANSI Common Lisp standard had been accepted then I’m sure they would have added a sockets layer. I can’t blame OpenSource because I can’t stop applauding it. I bow down to those people that have sacrificed their own personal time to make my life easier / better. I know that the fact that I have to do a little bit of work in the decision making process should make me happy because I have a choice to begin with. And anyway it’s better than having to do the whole thing from scratch myself.

But here’s the thing. When I stop shouting and beating my donkeys for being a bit lame and look up there’s millions of Microsoft developers cheerfully waving at the crazy guy who talks (and occasionally beats) his donkeys. I know if I was one of them, I’d think I was crazy too.

I think we donkey beaters can turn this around. I still believe that in general what I can hold in my hand for free is far more powerful than anything I can buy. The problem is essentially one of continuing improvement and better communication. All I have to do is stop cursing the donkeys and do something about it.

So last week I mentioned that I’d been practicing my Common Lisp by having a go at some puzzles on Project Euler.

A few days later I got an email asking me about what my experiences of the project were and how far I’d got on the puzzles. Well now I can reveal to all of you the heady extent of my genius. My correspondent also casually dropped in the fact that he was trying to do his puzzles in a functional, rather than imperative, programming style. Oh … yeah, “functional programming” I’d conveniently forgotten about that. I was so busy getting to grips with programming in CL I forgot about the important bit. I guess 15 years bashing your head against C++.Java and C# is going to have some long-term mental health effects.

Losing the plot

Now, my CS class at University (15 years ago) very briefly touched on functional languages so I’m not totally clueless. However, we learnt precious little and there is really no defense for that University to not place a larger emphasis on functional programming. I would argue, now, that it would have been a lot more useful for them to teach us a functional language than what they did teach us, which was: COBOL.

So, given my limited understanding, the point of functional programming is to write programs that behave more like mathematical functions. This has certain consequences since a mathematical function can not have side-effects like printing “Hello, World!” or delivering spam. Which surely is going to disappoint some people.

It is this “dirty” little secret of programming (i.e. that it should probably do something if we’re gonna get paid) that makes pure functional-programming a little less straightforward. Languages like Haskell have monads that address the problem of purity by effectively differentiating the pure from the impure. But I don’t think I’m quite ready to go there just yet. Maybe in another 15 years.

Taking the tablets

Ok, so where was I? Well, I’m going to try to make my code more functional too. Which means a diet of higher order functions and recursion for me. So let’s pick a trivial example and see how we do.

Problem #6 in Project Euler asks us:

Find the difference between the sum of the squares of the first one hundred natural numbers and the square of the sum.

My first solution looked like this:

(defun sum-of-arithmetic-series (n)
  (/ (* n (1+ n)) 2)

(- (expt (sum-of-arithmetic-series 100) 2)
   (loop for i from 1 to 100 sum (expt i 2)))

Which I was proud of because I’d figured out that I could use an equation to solve the first part. This isn’t terribly functional though because the second part uses a loop. So I tried again, this time I thought I could do better. I decided that what I was really doing was summing a series of natural numbers in the range 1..n. I could generalise this into: “apply a function (or the identity) over a series of numbers and sum the results”.

Looking at the problem this way generates an abstraction that did not previously exist in the earlier solution, making it a more general result and increasing the possibility of its reuse (perhaps!):

(defun sum-of-series (n &optional f)
  (let ((n-dash (if (null f) n
		    (funcall f n))))
    (if (= n 1) 1
     (+ n-dash (sum-of-series (1- n) f)))))

CL-USER> (- (expt (sum-of-series 100) 2) 
	    (sum-of-series 100 (lambda (x) (* x x))))
25164150
CL-USER> 

Ok, I know it's trivial but it made me feel better. There was only one problem though. When testing this for large n (> 30000) I got stack-overflows. Which surprised me because I was expecting it to be tail recursive and hence a tail-optimization to happen. I tried playing around with the debug settings and trying different variations but no amount of playing could make SBCL do the right-thing and optimize the recursion away.

Guess I need to do some more investigation into functional programming and that's probably an imperative! (groan)