Categories
lisp

Common Lisp and the Towers of Blub

So I’ve been teaching myself Common Lisp. It’s a slow process because I don’t get a lot of time to do it, and also I’m a bit slow. The reason I’ve been teaching myself? Well I’m wholly dissatisfied with C++, Java, C#, and to a much lesser extent Python. I still love C though. She’s my lady. We connect.

The dissatisfaction started after I read Paul Graham’s “Hackers and Painters” book. I wrote about it too. Whilst reading the book I came across this paragraph and identied myself as a card-holding member of team “Blub”:

As long as our hypothetical Blub programmer is looking down the power continuum, he knows he’s looking down. Languages less powerful than Blub are obviously less powerful, because they’re missing some feature he’s used to. But when our hypothetical Blub programmer looks in the other direction, up the power continuum, he doesn’t realize he’s looking up. What he sees are merely weird languages. He probably considers them about equivalent in power to Blub, but with all this other hairy stuff thrown in as well. Blub is good enough for him, because he thinks in Blub.

Since then I’ve been on a journey. It’s taken me through books, well a book. I’ve discovered that Emacs is like 1 million times more powerful and sexy than her slightly musty smelling older sister Vi, with whom I’ve had an almost 15 year relationship and am trying to get out of my head. :x!

If you don’t believe me about Emacs (and why should you) consider how you’d write a video editing suite in whatever code editor you use or can extend. I already knew long ago that Emacs is at least partly written in Emacs Lisp but I should have guessed that arguably the most productive Common Lisp development environment, or IDE, is also an Emacs extension called SLIME.

Soon after that I discovered some fantastic screencasts in the Ruby On Rails mould for stupid people (like me): SLIME, Hunchentoot, and Selenium. BTW, I know Selenium is not Common Lisp but the screencast presents a Common Lisp binding to the Selenium Remote Control server that makes it possible to use Selenium from Common Lisp as a testing framework.

So it feels like I’ve come along way. But today, your humble programmer-of-Blub discovered something on the HyperSpec that scared me a little. Through the PlanetLisp RSS feed I discovered that I am also really crap at mathematics. I did this by trying some of these puzzles. In attempting to solve puzzle #2 in Lisp I found I needed to brush up on some Lisp basics. After some digging, and through a series of HyperLinks, I ended up here and before that here:


...
;; This illustrates the use of SETQ on a symbol macro.
 (let ((x (list 10 20 30)))
   (symbol-macrolet ((y (car x)) (z (cadr x)))
     (setq y (1+ z) z (1+ y))
     (list x y z)))
=>  ((21 22 30) 21 22)
...

I think I stared at this code snippet for about 2 minutes before it finally made sense when I at last remembered that setf works on a place and not a symbol. Now, I’m guessing an experienced Lisp-er would look at the sample and say ‘so-what’? I’m guessing that non-Lispers would say, anything that looks that complicated must be bad.

And this is when it struck me again. And this time it hurt. I may have lost a little “Blub” to get this far but I still have way much more “Blub” to lose.

Categories
c++ rant

Why not just use Haskell?

I am a fully paid up member of the ACCU. I joined because it seemed most of my now ex-colleagues were members and I’m a sucker for peer pressure.

Anyway, as you’d expect from a magazine dedicated to C and C++ users it occasionally has articles about C and C++. This quarter’s doozy is from a chap we’ll call Stuart. Now it looks like some of the things that Stuart is doing for his PhD are quite interesting. However his article about the similarities he has managed to draw between Haskell and using partial-template specializations in C++ gives me the shudders. It is based, in part, on the work of Andrei Alexandrescu. As charming as Mr Alexandrescu is in person I wouldn’t let him anywhere near any of my production C++ code (if I had any). But that’s beside the point.

Now I have to admit that I don’t know any Haskell and I’m also not a lover of C++ much either so I can’t bring myself to actually RTFA. However the article is full of little code snippets like this:


    template <typename T> struct Head;
    template <int x, typename xs>
       struct Head<IntList<x,xs> > {
       enum { value = x };
    };

    template <typename T> struct Tail;
    template <int x, typename xs>
       struct Tail<IntList<x,xs> > {
        typedef xs result;
    };

Why anyone would hate themselves enough to actually do this and write about it I don’t know. Especially when you could just use Haskell and not have to wrap your head around this lunacy.

But I know there is a simple answer. I should just unsubscribe from the magazine if I don’t like it. But sadly for me it’s not that simple. Stuart is obviously a clever guy, it almost goes without saying. But he’s wasting his brain cycles. Surely he must have better things to do with his time.

Then I read this 2005 interview with the glorious Alan Kay. If you have the time I suggest you read all of it because he makes some really very excellent points. In reference to my pain though this one stands out:

Perhaps it was commercialization in the 1980s that killed off the next expected new thing. Our plan and our hope was that the next generation of kids would come along and do something better than Smalltalk around 1984 or so. We all thought that the next level of programming language would be much more strategic and even policy-oriented and would have much more knowledge about what it was trying to do. But a variety of different things conspired together, and that next generation actually didn’t show up. One could actually argue—as I sometimes do—that the success of commercial personal computing and operating systems has actually led to a considerable retrogression in many, many respects.

You could think of it as putting a low-pass filter on some of the good ideas from the ’60s and ’70s, as computing spread out much, much faster than educating unsophisticated people can happen.

And while Stuart carries on trying to make C++ like Haskell, rather than just using Haskell, another wasted brain drops into the soup.

Categories
java lisp ruby

Continuations and Exceptions

I very often find myself tripping over ‘old’ ideas in software technology and discovering a rich topic that I had no idea even existed. This weeks elephant in the room is the ‘Continuation’ language/compiler feature.

Let’s take a quick look at a simplified version of an example of a continuation that I found to illustrate the point.


irb(main):041:0> def ab
irb(main):042:1>    puts "a"
irb(main):043:1>    callcc { |continuation| return continuation }
irb(main):044:1>    puts "b"
irb(main):045:1> end
=> nil
irb(main):046:0> x = ab()
a
=> #<Continuation:0xb7c5ee84>
irb(main):047:0> x.call()
b
=> nil

The callcc function accepts a closure as an argument. This closure accepts the ‘continuation’ and we immediately return it from the function. The continuation contains all of the state that got us to the point before we exit, variables, stack-frames, the lot. The key is that the continuation is itself a closure and when it is called the code will continue from the next line. I think it’s easy to see that this feature has pitfalls. It’s like a GOTO but without the benefit of having a line number to lookup! This is because on the outside every continuation looks pretty much the same. But this approach does have arguably real tangible benefits in Web servers like Seaside.

So what?

Well it turns out that exceptions in C++ derivative languages are a stupider half-cousin of continuations called: escape continuations. Thinking about it the similarity is striking and not all that surprising. The reason I say they’re stupid though is because exception objects are not re-entrant in the same way as continuations are, and the context that is captured from the exception is defined by the programmer. However, it’s plain to see that the exception handling we have in most modern programming languages like C++, Java, C#, etc are a diluted form of continuation.

So what?

Well I’m not totally convinced of the practicality of continuations in web programming environments, partly because I’m a bit old-school when it comes to performance and those languages that have web-frameworks and support continuations (Smalltalk, Ruby, PHP) are slow-as-molasses. The other reason I’m not keen on the idea of continuations is the practicality of working with them, i.e. they are a variation on the GOTO in that it’s very hard to follow the flow of the program by just looking at it and reading it. But I know that one day I will discover the need to make sparing use of a continuation and be glad that I knew that I could do it.

If the truth is ever told, I think exceptions suffer from the curse of the GOTO as well. And I think the reason for that is partly because the exception mechanism as implemented in Java, C# and C++ is a teensy bit broken. The reason, again, is that the flow of control jumps up the stack to a potentially unknown place. I say unknown because if you write a library that throws an exception and after you release it you change the contract of how that exception is thrown, well then you’re in trouble.

Trouble, that’s what

The catching of exceptions is based on type, and inheritance rules apply, so the inheritance hierarchy of the exception classes you use affects the flow of control. This is clearly the intention since you use the type to differentiate a NullArgumentException and a NullPointerException and how you deal with these will probably require different code. However, if the handling code was the same well then I why couldn’t I catch the base class instead and have one code-block? This is the temptation and the danger because if someone else decides to rejig the exception hierarchy you might not catch the exception you want. Worse still, you might end up catching exceptions you don’t expect and providing incorrect handling code for them.

So what?

Well like Peter Parker says:

Whatever life holds in store for me, I will never forget these words: “With great power comes great responsibility.” This is my gift, my curse. Who am I? I’m Spider-man.

And whilst we may not be Spidey we need to recognize the gifts and curses that 50 years of programming languages has given us … oh, and we should take note about the power/responsibility thing too …

Categories
slang

Programmer’s Bladder

Main Entry: pro·gram·mers blad·der
Pronunciation: \ˈprō-ˌgra-mərs ˈbla-dər\
Function: noun phrase
Etymology: Birmingham, Richard Pinchin
Date: circa 1994

1 : A condition that requires the victim to sit at their terminal for extended periods. The condition denies the victim their normal bodily functions until: that fucking thing compiles, or more caffeine is required.

Categories
rant

You Suck, I Rock

I guess I’m going beyond the remit of technical blog here and exploring an area best left to people that know more about it than me. So I’ll keep it brief. More and more I’m coming to realise that the ‘them and us’ of software development is a problem. There are many thems and sadly just us, and since we’re obviously the good guys that makes them the bad guys.

What would Al Do?Dale Carengie in his 1933 book writes that very few people ever think of themselves as bad people. Even the really bad ones. Al Capone, arguably the most notorious gangster in history, claimed that he was just providing a public service (effectively saying that the ends justified the means).

The totally unsurprising fact is that almost everybody feels this way about their actions. The reason that we don’t all get along though is that my interests, and hence my actions, are often divergent from yours. Like when you go and commit that change to that function, that I said I didn’t want you to make, you did it because you’re you. And you probably have a very good reason for it. The problem is that I will instantly brand you a jerk, and if I don’t know you very well your jerk-ness will be hard to shrug off. Trust me, you’re pretty much screwed.

The anti-dote to this problem is for me to try and get inside your head and understand why you acted like you did. Maybe then we can find a better, mutually acceptable solution or I can persuade you do it my way. But it’s soooo much harder to do and it sounds really lame. It’s so much easier to write-off perfectly justifiable behaviour as bad and bitch about it for a while.

Welcome Trevor! Trevor is an ex-ex-ex-colleague and he’s nasty. He looks nasty, he talks trash and he has no time for me (or anyone else) because I don’t have a PhD and my IQ is so low that I may as well dribble from the left-side of my mouth and eat through a straw. Trevor is clever. Very clever (almost magical) Trevor. Part of Trevor’s problem was that he was often interrupted during the working day by people who didn’t think about what they were asking. They ended up wasting his time and that really annoyed him. Once I figured this out I made no attempt to have a conversation with him because I knew my opinion was of no interest to him. He just wanted to end whatever conversation we were about to have as quickly as possible. So, I kept our conversations to a minimum and made sure to if possible ask him questions to which he could answer yes or no. Over time it seems that this approach gained me some credits with Trevor. Although we could never be friends some of my fellow ‘retards’ admired the relationship Trevor and I had.

My advice? Hug a Trevor today. It’s worth it.

Categories
article finance programming python

Calculating peak-to-trough drawdown

Ok, so this is a little bit technical but it’s an intriguing puzzle that got me thinking quite hard. So here’s the problem. Sometimes investors want to be able to judge what the absolute worst case scenario would have been if they’d invested in something. Look at the following random graph of pretend asset prices:

Peak-To-Trough

You’ll see that there are two points on the graph (marked in red) where if you had invested at the first point and pulled out on the second point you would have the worst-case loss. This is the point of this analysis and is a way for investors in the asset to see how bad, ‘bad’ has really been in the past. Clearly past prices are not an indicator of future losses. 🙂

The upper one is the ‘peak’ and the lower one is the ‘trough’. Well, finding these two babys by eye is trivial. To do it reliably (and quickly) on a computer, is not that straight forward. Part of the problem is coming up with a consistent natural language of what you want your peak and trough to be. This took me some time. I believe what I really want is: the largest positive difference of high minus low where the low occurs after the high in time-order. This was the best I could do. This led to the first solution (in Python):


def drawdown(prices):
	maxi = 0
	mini = 0
	for i in range(len(prices))[1:]:
	   maxj = 0
	   max = 0
	   for j in range(i+1, len(prices)):
		if prices[i] - prices[j] > max:
		    maxj = j
		    max = prices[i] - prices[j]
	   if max > prices[maxi] - prices[mini]:
	   	maxi = i
		mini = maxj
	return (prices[maxi], navs[mini])

Now this solution is easy to explain. It’s what I have come to know as a ‘between’ analysis. I don’t know if that’s the proper term but it harks back to the days when I used to be a number-cruncher for some statisticians. The deal is relatively straight-forward: compare the fist item against every item after it in the list and store the largest positive difference. If this difference is also the largest seen in the data-set so far then make it the largest positive difference of all points. At the end you just return the two points you found. This is a natural way to solve the problem because it looks at all possible start points and assesses what the worst outcome would be.

The problem with this solution is that it has quadratic complexity. That is for any data-series of size N the best and worst case will result in N * N-1 iterations, in shorthand this is O(N^2). For small n this doesn’t really matter, but for any decently sized data-series this baby will be slow-as-molasses. The challenge then is to find an O(N) solution to the problem and to save-those-much-needed-cycles for something really important:


def drawdown(prices):
  prevmaxi = 0
  prevmini = 0
  maxi = 0

  for i in range(len(prices))[1:]:
    if prices[i] >= prices[maxi]:
      maxi = i
    else:
      # You can only determine the largest drawdown on a downward price!
      if (prices[maxi] - prices[i]) > (prices[prevmaxi] - prices[prevmini]):
	prevmaxi = maxi
	prevmini = i
      return (prices[prevmaxi], prices[prevmini])

This solution is a bit harder to explain. We move through the prices and the first part of the ‘if’ will find the highest part of the peak so far. However, the second part of the ‘if’ is where the magic happens. If the next value is less than the maximum then we see if this difference is larger than any previously encountered difference, if it is then this is our new peak-to-trough.

The purist in me likes that fact that the O(N) solution looks like easier code to understand than the O(N^2) solution. Although the O(N^2) solution is, I think, an easier concept to grapple with, when it’s translated into code it just doesn’t grok.

Categories
rant

The Decline of Hewlett-Packard

I remember when I was a kid that I had a certain amount of awe for some companies. I particularly remember that I used to think that HP was cool. I mean, how couldn’t they be? For one thing they made all that fancy scientific equipment but not only that they made all those neat calculators that did the whole reverse-polish thing.

The Mighty HP42s

I owned a HP42s (pictured left) and it was a very very cool item indeed. The ‘s’ stood for scientifc but it had it’s own stack based programming language that allowed you to write simple applications. You could even plot graphs onto its two line display. But only if you were sick and twisted, like me.

In the early 90s I spent a long summer trying to program an HP Deskjet 560c from a DOS program using only PCL5. It was quite a slow task and Windows would make the entire thing a subsequent waste of time but I had nothing but the utmost respect for the hardware.

But everything made by HP I’ve owned since then has been of poor or low quality. I owned an inkjet that broke down before the cartridges ran out. Then I bought a WiFi printer that has lasted longer but never loads a page straight. Then I bought a scanner that:

  • for some reason comes with about 300Mb of software that is garbage and I don’t want but the scanner can’t do without
  • the software doesn’t work very well and has a clunky non-standard interface which makes the whole experience regrettable

Last night a friend ‘dropped’ round an old HP notebook ‘nx6110’ for me to fix. The notebook was not broken just missing a driver, but the entire experience of getting that driver from HP left me sad and demoralised. Once I had discovered the product model number and located the driver I noticed that the driver was 24.7Mb (it subsequently transpired that this package contained the driver for 3 different OSs but still)! When I tried to install it it wanted to install itself into a non-standard folder in C:\ which annoyed me. Nor did the folder get automatically cleaned up afterwards.

After that joyless experience I wanted to install the WiFi drivers but it was impossible to tell from the notebook and the website which one to install. They had at least 5. Should I install one? Two? A combination? Simply unclear. I unsuccesfully tried to install 3 or 4 and just gave up in the end. I was sick, tired and angry of the low quality HP product and support.

What is going on? There’s no joy in owning an HP product anymore, and in fact I would say that this has been true since before the Compaq merger. I can’t remember the last time they made a product that generated desire like Apple or even Dell. So I guess I’ll just polish my reverse-polish HP42S and remember the gold old days and how things used to be. Perhaps new research direction can save them but as a consumer I’m not going to buying any more of their products unless they produce something exceptional.

Categories
article project management

You think your code don’t smell?

So, code reviews are great. Get the benefit of some ass-hole telling you that your comments should be C-style (/*) and not C++-style (//) and remind you that the member name ‘mSuckThis’ is not suitable, ever. No really, code reviews are great. It’s just that a lot of times they just don’t work.

The first time I encountered code-review was when my boss of the time had just read some book on how to manage programmers and was keen to inflict it on all his employees. His code-review process was to take all my work print it out and go through it line-by-line. Master and student style.

This type of code-review, in the way that he implemented it, was meaningless. It concentrated on an important but largely automatable aspect of code review and that is: adherence to coding guidelines.

As I see it there are three types of defect that code review is trying to identify:

  1. Adherence to coding guidelines (or lack of it) and inter-package dependencies.
  2. Identification of localised errors: “that loop is infite”, or “that algorithim should be log(N) and not N^2”, “that module is way too big”
  3. Identification of non-local errors. Where local means local to the code-review. For instance the impact of adding a throw on a widely used method and how that affects all the dependent code paths.

I question anyone’s ability to fully understand the dynamic nature of any reasonable sized piece of software by just looking at a small excerpt. Every time you review that code you have to ‘load’ that supporting information into your head to be able to identify whether the code is totally awesome or tragically bogus. In my experience defects of a local type (type 2) were sadly rarely identified by code review and defects of a non-local type (type 3) almost never.

The improvement of code-quality I’m passionate about. But I don’t see any realistic way to achieve what I want. To identify non-local errors you really need your code reviewer to sit next to you during the development or be as deeply involved in the code as you are. It probably would need a similar approach to reliably find local errors too. However your reviewer is rarely as involved as that. It seems that some judicious use of pair programming might be the answer but that comes with its own problems.

It seems that to get the best out of code-reviews you have to be very careful about how you implement them. Sure, let’s automate what we can automate and pair program on tricky items but the real code-review needs to be extremely skilfully handled to get the best bang-for-your-buck-chuck.

Categories
article programming

Time: the unseen global variable

Just about everyone knows that global variables need to be used sparingly. The more you use the more likely you are to capture complex state in places that are hard to maintain. Or something.

As well as all the globals you can see and measure there exists a shadowy league of ‘unseen’ globals in your programs. Some, like environment variables, are clearly designed as global variables and are desirable and understandable. However, some are wistful and ephemeral and dance round your program like wicked elves. Time is the biggest and most scarey of these elves.

For most programs you write time probably doesn’t matter, they are to all intents-and-purposes time-less. But as soon as you start entering the shadowy world of time, and the even more nebulous one of time-zones and daylight savings, a whole set of other state is being used. In my experience the programs and components that I have written that have been dependent on time have been some of the most complex to develop and maintain. This is for a variety of reasons but in summary:

time is not constant and can be interpreted in more than one way.

This leads to all manner of difficulties:

  1. Code that depends on the current system time ‘Now()’ and doesn’t pass it as a parameter is always going to be fragile. This is mostly because its behaviour can be non-deterministic unless you properly account for the fact that time is not-constant. This is especially important because your programs are susceptible to hard-to-spot boundary effects if you write expressions that use Now() more than once and depend on it returning the same value for each call. Which of course it never will.
  2. Time and date should never, ever, ever be separated from one another. You get all sorts of tricky errors when you split the two. Especially when you are performing some sort of time zone or daylight savings calculation where the two should change together but do not.
  3. Some programming languages represent the date (no time) as a date with a time of 00:00:00. Which is intuitive, but consider then what happens when you load a date (with no time) from a database in the past, when there were daylight savings, into a time when when there are no daylight savings. In the frame of reference of now your localised past time will now be an hour earlier and so will be in the final hour of the previous day. This problem clearly applies to timezones also but is because you made the mistake of not having a consistent view of time.
  4. Not only can the meaning of calendar time change after-the-fact (due to time-zones) but it can also be interpreted differently by different cultures.

There’s probably a lot of other time related pickles you can get yourself into. If Harold Lloyd was a programmer ...

You’d probably not be surprised to hear me say that unit-testing is one way of addressing at least some of these problems. This does two things. If you are to get good coverage for your unit-tests you are practically forced to make time a parameter wherever it’s used, instead of calling Now(). As a direct consequence of this your code can now be called ‘As Of’ and you will be able to offer the historical view where appropriate.

Indeed, I would say that where a piece of sotware has a time-context, then it will only be a matter of time before someone says: “Ok, that’s what it says today but what if I want to rerun it for that time in the past 3 weeks ago?”.

The time-zone and daylight savings problems can be nailed by having a consistent view on the treatment of time. For instance storing all dates/times as UTC is one thing. But if you ever need to store a local time then it should be clear what frame of reference is being used to store that time. So you might need to additionally know: the calendar, the timezone, and the daylight savings rules before you can correctly store a time.

Then and only then will time become your faithful and obedient friend.

Categories
article training

Did you learn anything? I forget …

Something that’s been concerning me for some time is the cost and benefit of courses and seminars. Most employers and employees would perceive programmer’s training as a positive benefit and I think I’d have to agree but it seems that there is a common view that all training is good because it’s personal development. To deny that training to an employee would make you a bad employer because you are stunting your employee’s professional growth. Well I’m not so sure. I’d even go as far as to say:

A lot of technical training is of limited value.

There I said it. It’s out. I’m probably never going to get to go on a course ever again, ever.

The last purely technical course I went on was a compulsory learn Java course in-or-around 1999 (yes I’ve been avoiding courses since then). I remember it not for the content, which was forgettable, but for the fact that I’d snapped my wrist 1 week before and I could only type with one hand. The course, however, was custom designed for our company and our tutors had been briefed about what we needed to know. I would say that this sort of training, i.e. directed, has good benefit but again it only teaches the how. The why is lost.

Compare this with the ‘shrink-wrapped’ course. Which is offered by a training company on a technology and is a generic product. In my experience I probably end up using a small-ish fraction of the material learnt on such courses. This is because to attract the candidates they need to give the course a broad appeal. However, the chances are that I’m going on a course for the broad appeal are low, it’s more likely I’m doing it for a very narrow reason. A narrow reason usually defined by the next biggest project of the moment. Sure it’s helpful to know all the aspects of a particular technology, but the things I don’t need to know right now will very soon be forgotten.

This is not the only problem with shrink-wrapped courses. There is also a tendency for candidates to choose ‘advanced’ courses that are sometimes beyond their current ability, secure in the knowledge that “it can’t be that hard” and they will pick it up. When this happens the tutor has to work very hard to bring everyone up to the same level so that he/she can teach some of the more advanced aspects.

So this sort of leraning is inefficient in that the information that needs to be conveyed is often greater than is needed, but there’s another deeper problem with courses. In my opinion I think that programmer’s would sometimes be better schooled if they learnt good approach first and implementation details later. Take security for instance, almost every application these days needs some sort of built in security. I’d argue that it would be useful for programmers to go on a ‘security for programmers course’ which covered lots of different security aspects relevant to programmers. More useful than, say, a technical course suited to a particular technology which attempts to teach security, amongst a lot of other things. As I’m starting to learn it’s the principles that matter, not the implementations. At least this way you’ll see the entire security picture and then when faced with a situation which could be a security risk you can say ‘here’s a potential risk’ now I need to find a way to mitigate it.

In some ways this sort of training is perhaps best delivered inside the organisation by mentors. Big-brothers (and sisters) who can guide the novice through the general principles leaving the rookie to grapple with the fine details of the implementation. Sadly, when you get to my age, big-brothers are most likely to be grand parents so I guess I’ll just have to keep getting my training from Amazon.