Categories
lisp programming

Know Your Idioms

A minor thought for today. Part of becoming an expert in any programming language is learning the idioms. Through experience, research and your peers you discover what makes your programming language hum. How to get the best performance and how to trade that against the best expressiveness. Since spoken language affects the way that what we think it’s reasonable to expect that programming languages affect the way we think about programming problems.

I have on a number of occassions wanted to write an expression in Lisp that will return true if all the elements are T and nil if any are not T. A sort of ‘and’ for a list. It seemed like a classic case of a ‘reduce’ or fold operation to me and so I went about doing that as a starting point.

CL-USER> (reduce #'(lambda (x y) (and x y)) '(t t t t t t t))
T

Hmm, because AND is a macro it can’t be an argument to reduce therefore I have to wrap it in a lambda. Which is ok but it removes the short-circuit feature from the AND.

So, I wasn’t terribly excited by this result but it served the purpose and life continued as normal. That was until today when I discovered the LOOP tutorial which inspired me with this trivial alternative …

CL-USER> (loop for x in '(t t t t t t) always (eq t x))
T

Yes it’s a little bit less functional but it it’s very clear what’s going on here and that was my original problem with the second reduce form. I was interested to see how the LOOP expanded too, so:

CL-USER> (macroexpand-1 '(loop for x in '(t t t t t t) always (eq t x)))
(BLOCK NIL
  (LET ((X NIL) (#:LOOP-LIST-1613 '(T T T T T T)))
    (DECLARE (TYPE LIST #:LOOP-LIST-1613))
    (SB-LOOP::LOOP-BODY NIL
                        (NIL
                         (SB-LOOP::LOOP-REALLY-DESETQ X (CAR #:LOOP-LIST-1613))
                         NIL
                         (SB-LOOP::LOOP-REALLY-DESETQ #:LOOP-LIST-1613
                                                      (CDR #:LOOP-LIST-1613)))
                        ((UNLESS (EQ T X) (RETURN-FROM NIL NIL)))
                        ((WHEN (ENDP #:LOOP-LIST-1613) (GO SB-LOOP::END-LOOP))
                         (SB-LOOP::LOOP-REALLY-DESETQ X (CAR #:LOOP-LIST-1613))
                         NIL
                         (SB-LOOP::LOOP-REALLY-DESETQ #:LOOP-LIST-1613
                                                      (CDR #:LOOP-LIST-1613)))
                        ((RETURN-FROM NIL T)))))
T
CL-USER> 

... Jeeeeezus.    No way that can be efficient surely?  So I ran both arguments with a temporary list of 1,000,000 elements all equal to true:


CL-USER> (time (reduce #'(lambda (x y) (and x y)) (loop for i below 1e6 collect t)))
Evaluation took:
  0.149 seconds of real time
  0.100006 seconds of user run time
  0.040002 seconds of system run time
  [Run times include 0.068 seconds GC run time.]
  0 calls to %EVAL
  0 page faults and
  7,999,488 bytes consed.
T
CL-USER> (time (loop for x in (loop for i below 1e6 collect t) always (eq t x)))
Evaluation took:
  0.059 seconds of real time
  0.052003 seconds of user run time
  0.0 seconds of system run time
  0 calls to %EVAL
  0 page faults and
  7,995,392 bytes consed.
T


Guess again brother.   The more I thought about it the more other solutions I could find to this problem.   For instance I could just count all the nulls and if I find one then the answer is false.


(eq 0 (count-if #'null 
           (loop for i below 1e6 collect t)))




This was marginally faster than the 'loop' on my implementation, but then I reasoned that on a list of a million items  'AND' should really retain that short-circuit ability to make it stop on the first false.   So perhaps the correct thing to do is return the first non-true argument:  


(not (find-if #'null 
          (loop for i below 1e6 collect t)))


This solution is less imperative in style and less complicated than all the other solutions.  Now that I've thought of it I can't imagine how I didn't think of it sooner.   A winner!


Lisp is such an old language that there seems literally dozens of ways to attack most problems and that really is the point.   Since I don't know what I don't know there's probably dozens more ways I could try to solve this trivial problem that only experience, research and interaction with my peers will get me.      


This is where books like the O'Reilly cookbook series should come in right?  They should have all that hard work done for me so I can stop making mistakes and start making systems.    Books, however, are not very easily searchable when they are on the shelf.   Therefore on-line resources are best for searching for optimal solutions.  So I was pleased to find that Common Lisp has an online cookbook.   However you still have to KNOW where to look in a resource like this because I might not have found the 'find-if' solution if I was stuck in a rut banging on the 'reduce' solution.    Not only that, every situation is different, 'reduce' definitely has its uses but perhaps not in this case.  


This further suggests to me that it's only your peers and your own research (i.e. trial and error) that can show you the way.  Just make sure you give yourself enough time for the research and pick your peers carefully 😉
Categories
article programming

Floating Point Flotsam

I have never been particularly clear about when to choose single or double floating point arithmetic. I think I have been operating on a sort of ‘trial-and-error’ approach for some time. It seems ludicrous that I should not, after all these years, know exactly when to chose single or double precision arithmetic. The truth is that I don’t, and it’s now time to fix that.

First, a little background. I was, for reasons that are too sad to explain, interested in how to calculate (accurately) the time of the Vernal Equinox. That is the exact time of the year (to the minute – if possible) that the sun is 90o above the earth at the equator. Anyway, I found some code which is an implementation of an astronomical algorithim designed expressly for the purpose of calculating the vernal equinox and is accurate to about 20 minutes. Which is good enough for now.

Now the next piece of background is that I was also doing this in Lisp and up until now I have let the reader interpret all the literals I enter. When I tried to do the calculation in the REPL I could get no closer than being in the same part of the day as I was expecting and with the result somewhat rounded to the nearest half day. After some head scratching it finally occurred to me that the the computation required more significant digits than I really had. It seems that the formula I was entering was being interpreted as single precision floating point numbers because if I had wanted double’s I would have suffixed my literal numbers with a ‘d’. It would seem that d could also stand for D’uh. Seems fair enough. Time to do some research then …

You see, according to the IEEE standard 754-1985 a single precision floating point number has 23 significant bits and a double has 53 significant bits. For me to answer the question of how many significant figures I can get in decimal would require me to know that the smallest decimal fraction that I can represent in binary is. This number would be 1/223 which is about .00000011920928955078125.

Now, floating point numbers are represented from a fractional part and an exponent part to give the decimal representation of the number. Therefore you can never get more accuracy than the smallest binary fraction multiplied by the exponent you have. This means that the significant figures should be something like:

log10(1/(2^23)) = -6.9

Therefore when a number has 7 significant figures you are already losing a little accuracy, the more significant figures you add the worse it will get. It was then clear that my astronomical antics were less than stellar since the first literal in the computation has 11 significant figures.

Indeed whilst I was thinking about this problem it occurred to me that if I wanted to continue using single precision I could split the fractional part from the integer part and continue this way. However, this is still inferior to a double because it would give me 7 significant figures for both parts and therefore a total of 14 significant figures. This is inferior because using my shiny new brain I can show that a double will give about 16 signficant figures. Of course I could have also concluded that double’s are better than two floats by noting that 23 bits+ 23 bits < 53 bits but that would never have been as much fun.

Type Word Size Mantissa Dec Sig. Figs.
Single 32 23 7
Double 64 53 16
Extended 96 63 19
Quad-Extended 128 113 34

You could argue that I could have saved myself 20 minutes of time by looking the answer up but then again I would never remember the answer unless I proved it to myself first. So, now my spring occurs at the same time as everyone else’s and I know why. Oh it happened already? Sheeeeiiiitttt.

Categories
article

A Version Aversion

I like war stories. They remind me that I’m not alone on this journey.

This is a war story. It involves a lot of entrails, questionable surgery and plenty of walking wounded. I carry the scars so perhaps you don’t have to …

… when I joined the team the system consisted of:

  1. A few thousand lines of C++ code which ran as 10 process on two Solaris hosts;
  2. Some Java code that ran on an NT4 J2EE server;
  3. A bunch of windows client PCs all running the same (but different to the server) Java VM
  4. A collection of ‘glue’ scripts written in shell script and Tcl.

It ran in two locales, and the hardware was broadly equivalent in both, from what I recall. Now compare this to what we had by the time I left:

  1. A larger body C++ code with 15 or so processes
    on one Solaris host as well as a set of additional x86 UNIX hosts (of mixed hardware pedigree) that were sprinkled with various flavours of Solaris and Linux (RedHat) that would run between 2-4 processes depending on the number of cores;
  2. 2 J2EE servers of the same vendor;
  3. A further 2 J2EE servers of a different vendor (don’t ask!);
  4. A collection of scripts written in shell script & Python (thankfully we stamped out the Tcl);
  5. 3 primary locations each running a 2 different versions of our software and a single satellite location (hanging from a primary location).

Each server release involved somewhere around 10 hosts running different hardware, OS & JVM. It’s similar and different to the problem that software vendors must have when they need to make their product run on multiple platforms. However the difference for us was that our software was a distributed system and each component needed to seamlessly interact with its peers. Something that not many software vendors make a habit of, other than Microsoft I suppose.

Against this background was a team of 10 developers in 3 timezones developing software for a constantly changing and fairly lucrative business. Quickly made enhancements could secure profits, instability and failures might secure losses so it was important to try and keep the system running as smoothly as possible. However, the large code base (>100,000 lines) and confusing deployment array made every release a roller coaster ride. In my last two years of the job the release cycle, whilst somewhat improved from when I started, had increased from 1-2 months to almost 6. This had an unforseen consequence that developers would, out of necessity, place new features onto release branches to be able to get features out faster. That’s when the madness started.

There were too many release versions, operating system versions and client library versions to contend with. Sometimes even trivial changes become enormous chess games where the order of the changes that we made would determine whether the system would actually run or not. Eventually it was bound to grind to a halt because with that many deployment configurations each release had too many testing dependencies. There were two problems here, firstly since this was now a very widely distributed system it would be difficult for us to have an accurate test deployment that worked. Further, making distributed systems work is hard anyway and the more configurations you have to manage the more complex it’s going to be. We were trying to help ourselves by retrospectively adding unit-tests but the coverage was still fairly low and so we could never have very much confidence that a built system was actually going to work. What we really desperately needed were integration tests but we never quite managed it.

That’s where Joel’s post from last week comes in. As described by Joel we essentially had a SEQUENCE-MANY situation. Where to be sure of stability we had to test many releases against many deployment configurations. It would be fair to say that we failed to do this adequately. I sometimes wonder if we could have done it a little better.

  1. Could We Have Had Tighter Control Over The Hardware? Unlike the problem of enforcing standards in Web browsers we of course had full control over the deployment environment so we could have mandated a common platform for it. As enticing as this sounds, talking to system admins now and then would tend to suggest that this is simply not possible if the hardware is to be purchased incrementally. This is because after you buy the first 2 Dell servers with a standard specification, a month later that specification will have changed. As more time passes the drift between the hardware is larger.

    If, however, you sourced a job lot of the hardware in the same place at the same time, you could buy extra (for spares and future requirements) and attempt to keep this variable constant. It would have been expensive to do but it is at least possible in this scenario. I think that this probably would have reduced the number of different cross-compilations that were required and reduced the number of different JVMs that we had to manage. The biggest problem though is that we would have, to a certain extent, needed to know the future to be able to predict what sorts and what amounts of hardware we would need when we set out. That kind of makes it a non-starter, coupled with the fact that I’ve never actually heard of anyone doing this for real.

  2. Could We Have Had Tighter Control Over The Software This is the thing that concerns me the most and is definitely a place we didn’t do as well as we should have. We let people go ahead and implement locale specifc solutions that were unworkable globally but those created internal system dependencies that would later need to be ‘undone’. Anyone who has ever worked on a system after its release will know it’s much easier to get it right first time. This is because if you create an intermediate solution that ends up being used then you have to manage the old intermediate-version ‘out’.

    Indeed, there was a story here too. The original system architect moved on a year or so after I joined. He used to worry about 80% of the code that got committed, when he left no-one really had his insight into the architecture and the rust quickly set in. Related to the loss of architect, as already mentioned, was the lack of integration tests. Both would have helped us to identify which code was bogus and have it fixed before it reached a release stage.

The one thing we did succesfully manage to do was to stop developers changing release branches. But the effect of that was to make us look like chumps when the business had to be denied features until new releases could be rolled out. Ho hum.

The idealist in me thinks we could have done a few things to make it work better but the pragmatist thinks that we did what we had to do. Whilst the idealist in my head makes a lot of noise and gets listened to an awful lot the pragmatist is the one who gets the most results. When you are faced with a daily tightrope walk, like we were, you have to try and be both idealist and pragmatist. Choosing the idealist’s course when you think you can get away with it and the pramatist when you can’t.

But when all else fails just hope for the best. The scars will heal. Eventually.

Categories
programming

The Robustness Principle

It seems I’m becoming a man of principles. Last week I wrote about the principle of least surprise. This week I was reading this article about martians over at Joel it reminded me about my previous job in some non-obvious ways. And before you say it, no I didn’t work with martians or any sort of aliens for that matter. The main topic of Joel’s blog is very interesting but a side issue is the discussion of the robustness principle as defined by Jon Postel in RFC 793.

2.10. Robustness Principle

TCP implementations will follow a general principle of robustness:  
be conservative in what you do, be liberal in what you accept 
from others.

Whilst Joel is talking about HTML standards Jon is talking about TCP standards. Anyway, whilst this difference is minor the pedant in me is forced to point it out ;-).

However it’s fair to say that such a principle exists and that I’ve been living by it, without actually having a name for it, for some time. The reason I started doing it was because I inherited a large code-base for a distributed system once and I discovered that occasionally components would simply terminate, users would complain and I would have to go and invent some bullshit to explain why our system wasn’t working.

I felt like I had to invent some bullshit because I’d found that some components had statements in them like the following:

    if (such and such)
        abort();

The ‘such and such’ that caused the termination would be some condition that should not have occurred. You see, the original programmer felt the need to have all unknown conditions delivered to him by aborting the process. This is rather like demanding a criminal’s head on a plate because aborting a process under UNIX will usually cause a core dump. This memory dump file contains the contents of all the memory and registers at the time that abort() was called. With the appropriate tools this gives something to indicate what the cause of the problem might have been. This style of programming is sometimes referred to as kamikaze programming but I think seppuku programming is probably more apt.

My sin was to systematically remove these little time-bombs and attempt to handle the condition that had occurred. The problem with this, of course, is that if any of these conditions have undesirable side-effects then they can store problems up for the future which, I guarantee, will be a lot harder to find and fix.

It seems there is no right solution to this problem either. Extending Joel’s conclusion we could say that the idealist (the original code author) wanted to control the error state and deal with each case appropriately by hand. Perhaps, eventually, all these little abort() statements would get to be unused code and the system would run very smoothly. Perhaps. The pragmatist (me) would have a hard enough time of keeping the system running as it was without having components voluntarily dying all the time.

I stand by my decision to improve stability but I think I probably should have left a couple of the more esoteric abort()’s in place. I did improve the stability of the system to some extent. But my effort to restore faith in the system by making it completely stable would be somewhat thwarted for other reasons (more closely related to Joel’s post about martians). As we shall see next week …

Categories
.NET programming

The Principle of Least Surprise

I first came across the principle of least surprise a couple of years ago when someone pointed out to me that some script that I had written had violated it. Naturally defensive, and of course mildly arrogant, I asked my critic what he was talking about. He said that a command line script should return an explanation of its accepted arguments when given a –help or -h argument. He was right and I was ashamed and so I naturally did what any respectful programmer would do, I fumed quietly for a few hours and then fixed it when he wasn’t looking.

It seems then that almost any user-interface should not violate the principle of least surprise and this is common sense. When I click on a cancel button in a dialog it should not accept my input it should return to where I had come from leaving everything unchanged. This was my first violation this week and it’s a common one, a ‘Cancel’ button in a dialogue did the opposite of what I expected. This was in a popular source control tool (that I won’t name because my version is old and the bug is doubtless fixed now) and it did in fact ‘Submit’ the broken change that I was so keen to cancel. After a short bout of tourrette’s and 5 minutes of trying to figure out how to undo the change I was back on track but my confidence shaken a little.

It’s not just user-interfaces that should not violate the principle of least surprise, APIs shouldn’t violate it either. You know the sort of thing. For instance, a ‘getter()’ should be idempotent and not change any state. Languages like C++ have support for this through the ‘const’ method modifier but I can’t think of another language that I know of that enforces this in the same way.

So I was incensed when I thought I’d discovered an example of such a violation in the .NET framework. I was having trouble using the method Uri.MakeRelative(Uri uri) because I had misunderstood exactly what it did (because I didn’t RTFM) and I expected it to simply strip the routing information from the Uri and return the document path. What it actually does is a more general case and it will effectively ‘subtract’ one Uri from the other. Which is more useful when you have things like embedded Uri’s. Anyway, the depth of my sickness isn’t important, the important thing is that I realised that I could have used the principle of least surprise to save me.

As far as I can tell I haven’t ever found a .NET framework method that didn’t do as I expected, that’s partly because of the number of people who use early-release versions of .NET and iron out these problems for me. Thanks! Not only that over the years I have entered into a love-hate relationship with Microsoft. I won’t enumerate the things I hate because disk space is finite but what I really like is the way that M$ have a vast array of tools and products that seem to co-exist with very little problems. This is no small feat, it’s a triumph of quality control and standards and I applaud them.

The conclusion is then, that I should abide and uphold the principle of least surprise wherever possible. Additionally though, I can apply the principle in reverse when I trust the source of the interface to not violate the principle. Moreover if I can succeed in not violating the principle myself then people who use my and my team’s work will have more confidence in it … I can dream, can’t I?

Categories
programming

Virtual Host Development: Part 2

Last week I made the case for using virtual hosts for software development purposes. So this week we’re going to actually try it out. There are a number of considerations to make before we get stuck in so let’s go …

Choosing a Virtual Host Manager

As far as I can tell there are two main equivalent software products for virtual host solutions. They are: Microsoft Virtual PC 2007 (MVPC) and VMware Workstation 6 (WS6). The most notable difference between the hosts is that WS6 costs (at time of writing) $189 and MVPC is free.

However, free comes at a cost. There are a number of disadvantages (for me at least) with MVPC that mean that I will be ponying up the cashwa for vmware’s offering.

  • Guest Display Hardware – Since we have a virtual host it also has a virtual display card. The one that sits inside MVPC is only capable of screen resolutions upto 1600×1200. Since I have a wide-screen display that is set to 1920×1200 it’s a bit of a waste. However vmware can also display on multiple monitors making use of dual head displays.
  • Guest Support – Yes I love Linux and I have to say (because I tried it) that trying to create an Ubuntu virtual guest on Virtual PC was like trying to herd cats. I’d get one part working only to find that something else was bust, whilst it’s possible it’s a pain. If you decide to go ahead with this you should take a look at this article which explains the hoops you must get your kitty’s through. In theory, I guess, you should be able to get any x86 OS that works in WS6 working in MVPC (as long as it isn’t 64 bit). So far though, and this is an observation based on one data point, I think the virtual hardware in WS6 must be more generic than MVPC.
  • Guest OS Tools – Both MVPC and WS6 require you to install additional tools after the core OS install that enhance the operation of the virtual machine. Why I couldn’t really figure out exactly what the tools do it seems reasonable to assume that they’re useful. Indeed it’s not until you install the tools on WS6 that you can access the higher resolutions. With WS6 then it provides tools for Linux where as MVPC does not. Whether this makes an observable improvement to the WS6 experience for Linux would require more time to tell ..
  • Host OS Support – VMware will run on Linux. So should I decide that I want to take my removable disks and put them somewhere else I could. At least in theory
  • Converters – WS6 has a free ‘converter’ which allows the creation of WS6 virtual windows (only) hosts from MVPC ones. Which is nice. Not only that though the converters are bootable so that you can create a virtual ‘copy’ of an already installed non-virtual OS. Whilst I did successfully convert MVPC hosts I didn’t try and virtualise an existing host
  • Snapshot Manager – vmware has a snapshot manager which makes the management of snapshots a lot simpler to comprehend. You get a sort of snapshot time-line that you can chose from and cycle through. MVPC has a similar concept but it’s not as clean.

Snapshot Manager

The virtual host manager you chose will depend on what you need. For me MVPC is not a choice, however there are probably a fair few people who could get good results from MVPC making the benefits of WS6 not that important. Since you can download a trial version of WS6 I would suggest you at least try it before deciding. To me WS6 product feels mature and the user interface seems more intuitive. Your mileage may vary.

For the remainder of this discussion I will talk about using WS6. Whilst you could in theory use MVPC and mostly apply the same ideas the two are not equivalent, as already discussed.

Creating The Host

Having settled on vmware I should say a little about the settings I chose. I’m not saying that these are ideal but for development purposes they seemed like sensible defaults to me. Firstly I instructed WS6 to create me a custom virtual host. For the most part I used the presented defaults but deviated on a few items:

  1. Non-Local Disk – When I say non-local I mean non-local to the host OS. If the disk of a virtual host is the same physical disk of the non-virtual host then you will have I/O performance issues. This is especially true for laptops that tend to have slower hard disks than their desktop equivalents (4200rpm vs 7200rpm). That’s why I went and bought an external drive enclosure and speedy hard disk before I started
  2. Memory – Since these hosts are development hosts I’m probably going to need some RAM, especially since I intend to run VS2005 on the windows host. Therefore I chose 1Gb RAM for the windows hosts and 512Mb for the Ubuntu host. Note that for windows changing the amount of memory after the OS is activated can force you to reactivate your license. So getting it right-enough first time would be a bonus. After running both systems on this configuration for a couple of weeks I can say that the settings are convenient enough for me and I’ve not noticed any excessive paging and performance is fine.
  3. Networking – I elected to chose NAT as my virtual networking setup. This puts my virtual hosts on their own subnet, they access the internet as the non-virtual host and then the results are routed back by vmware. The other alternative might be to bridge virtual and real host but this, I think, would make my virtual host a peer of the real one on the domain. Something I want to avoid for now.
  4. Disk Size – I allocated 80Gb disks to each host. In practice I would probably only use about 10% of that space but WS6 does not allocate the space all at once. This means I only use as much space as I need. This obviously incurs an overhead because the disk is allocated on demand. However I wanted to see if this really causes a noticeable problem before I elect to allocate all the disk at once. Keeping the space to a minimum obviously has advantages for backup purposes but also (I would guess) mean that snapshots take less time.

Building The Host

Obviously I must now load an appropriate OS onto each of my virtual hosts. You do this by simply attaching an ISO to the machine and installing in the normal way. Once installed you should install the VMware tools that improve the integration between your host and guest OS. After installing the extensions (and not before) you should activate your license on Windows based-hosts.

Next you should apply all the updates that are necessary to make your host a good netizen, either through Microsoft Update or whatever package manager your OS uses.

It was at this point that I ran into an issue with Ubuntu which relates to the networking. By default Ubuntu 7.10 turns ‘roaming’ network mode on and this doesn’t play nice with WS6. So turning it off and selecting DHCP insted was needed.

Network Roaming Mode Is Bad

Once built and patched I created checkpoints for both systems to rollback to or clone. This will be useful when I need to create a ‘test’ host for deployment or system testing.

Stack ‘Em High

One of my complaints about developing for Windows is the height of the development stack. Not only do I have to install dev studio but I have to install a bunch of service packs, add-ins and 3rd party software, and their service packs, to have anything like a workable development system. The problem is, when something goes wrong and your installation gets knackered in some way (it’s happened to me twice in the last year) you can kiss goodbye to 1 days development time whilst you sort it out.

That’s why, once I had all the development tools installed, I then took another snapshot to record the base development install so I can revert to it in seconds rather than hours. It’s my intention that when I went to install a new software tool I will clone a snapshot, install the software and see what it does before I commit to installing it into the main disk image.

Shared Disks

One important feature of having a virtual host is being able to easily share data between the guest and host OS. The clipboard works fine for a lot of things but when you want to expose a 1Gb ISO from the host to the guest it becomes a bit painful. Sure you could FTP the files from one to the other but that would be a pain. What you really need is to expose a part of the host file system to your virtual machine. Under windows and WS6 this results in the creation of \\.host URI that is accessible from the guest windows OS.

However for Ubuntu it failed because the installation of the tools failed without me noticing!! It turns out there is a bug in one of the vmware headers that incorrectly sets which kernel API to use. This is one of the problems with Linux I guess in as much as vendors have to release source code drivers because of all the different versions the guest OS could be. However, because it’s source code it’s also easy to fix! By following the instructions on the Ubuntu forums I had the matter resolved in minutes.

Accessing Domain Hosts

There’s a couple of problems with interacting with other network hosts when using virtual hosts.

  1. If you’re developing on a shared network and you are unable to make your virtual machine a peer on the network then you will probably have chosen NAT networking. This, as discussed, puts you machines on their own subnet. The consequence of this is that you will need to start using qualified domain names to be able to access the hosts that are in the same domain as your non-virtual machine. This isn’t a huge discomfort and it’s arguably the right thing to do if you’re embedding those names into software that you’re writing anyway.
  2. The second problem relates to the use of Windows authentication. Applications that use windows authentication (like SQL Server Management Studio) will fail because your virtual machine is not part of the domain. One solution is again to make your virtual host a peer on the network. Whilst this would be the ideal it might create implications for how you’re going to manage a battery of virtual machines and might result in some loss of liberty because you’re system administrator got scared. The solution is to use a feature of windows networking that allows the user to manage multiple identities. This will allow you to enter a set of credentials for each machine that you will connect to. After making these changes applications that use Windows authentication work flawlessly.

Stored Passwords

Conclusions

After spending a week of development on virtual machines I like it. I don’t notice any performance lags on the widows host everything works beautifully. On Linux I occasionally have mouse problems where the mouse refuses to roll over a particular screen area. This needs to be reset by disconnecting control from the VM (Ctrl+Alt) and then clicking (to regain the control) and trying again. It’s possible that this problem is something to do with known mouse problems, perhaps more investigation is required.

The WS6 ribbon makes switching between machines and displays simple. The ribbon also makes using WS6 analgous to using remote desktop which feels natural.

WS6 Ribbon

It had been my original intention to run only my development needs inside the virtual machines and keep the real machine for my office needs like e-mail, spreadsheet and word processing. However that becomes awkward to juggle between and means that I sometimes miss e-mail or IMs when the development host is maximized over the top of the host desktop. Having said that though maybe not getting distracted is a good thing 😉

However, I’m starting to think that I should adjust the windows virtual machine RAM and install all of the office applications and IM software into the windows development host. Reducing the host OS to simply a shell for the VMware software. All in all though, I’d recommend all developers to try it out at least. It ticks all the boxes I wanted and after the setup and configuration it makes development a saner process. Which has to be good, right?

Categories
programming

Virtual Host Development: Part 1

I got a new PC this week. Oooooh. It instantly presented me with a challenge:

Q. Is it possible to make full use of a standard mid-range specification development desktop host?
A. … errr … dunno …

As a developer I have long believed that we, me included, fool ourselves into purchasing new hardware and software that we don’t need. This constant upgrade cycle blind-sides us from separating what’s important from what is aesthetically pleasing. Since 2004 (when we approached the PC clock-speed ceiling) I can’t think of any advance in software or hardware that has really compelled me to want to go and buy something new. I discovered some time back that the biggest productivity bonus you can get from any developer is to give them two monitors (oh, and a graphics card that can support them too!). Giving those developers Vista or a bigger/faster machine won’t buy you much other less help-desk time supporting old kit and a better relationship with your hardware vendor.

The things that have changed since 2004 is the cheapness of RAM & disk and the addition of multi-core CPUs. Although 64 bit has also come online too that in itself only really gives you more addressable RAM. For the moment the circa 3Gb addressable in most 32 bit OS’s is fine for me and my development needs. This means that by default I get as much RAM as the OS can handle, 2 large disks and a multi-core CPU.

So, in the absence of a new OS to tax it the new hardware was begging for a thrashing and I thought knew just the thing. I would start doing virtual host development. That is, install a VMware style virtual host manager and run my development environment entirely inside a guest operating system. The host operating system, i.e. the real one, would be used for sending email, surfing, browsing documents and running the virtual host manager.

This way of working has some desirable consequences:

  1. Snapshots – Most virtual host managers allow the taking of snapshots. So the plan is to take a snapshot after each major milestone of the virtual host build. That is:
    • Right after the install (& patch) of the base OS
    • After the install of the standard development tools
    • After the install of personal development tools that I use

    Regardless of the OS you use your development environment will usually take you a fairly long time to build. This is because you have to load: IDEs, source control, useful editors and whatever else you fancy. The installation of the DevStudio stack alone can take the best part of a day. The ability to go back in time to anyone of those points will be very useful if something goes wrong. It’s not only that though, most virtual host managers will allow you to ‘clone’ a snapshot so that you can take your current environment, clone it, fire it up, install something wacky, and see if it works. If it doesn’t work as you expected then you live to fight another day.

    2. Security – I have never particularly liked installing my custom tools and utilities on my company’s hardware. It complicates matters when system administrator’s are needed to try and resolve problems. This is because, and rightly so, the first thing they’ll blame when something goes wrong is that piece of custom software I installed last week. Therefore, I think, at least in theory that I will no longer need administrator privileges on my own machine because I will have an administrator account on my virtual hosts. I’m not convinced of this view just yet, but I aim to demonstrate it here!
  2. Testing – Deployment of my developed code is always hard to test. It’s hard because you need a clean machine to work with to be able to prove that the test was indeed a success. With the aid of snapshots I can boot a clean machine and attempt a deployment to it. If it succeeds great, if it fails I just start again safe in the knowledge that I haven’t soiled my clean machine and the test is still a good one.
  3. OS Choice – I still think UNIX is far and away the most powerful and transparent OS, but that might be because I’ve been banging my head against it for a very long time. Anyway, I like having UNIX around and when I don’t have it it makes me kind-of sad and angry. Angry that I can’t: grep a file for a regex pattern, pipe it through gawk to get a particular column, pipe that through sort, then pipe that through uniq -c and pipe the whole lot through sort -g. How else would you answer the question of how can you do a: 
    SELECT Value, COUNT(*) 
WHERE Value LIKE '%XYZ%'
GROUP BY Value 
ORDER BY COUNT(*) DESC

    … on a flat file? On second thoughts don’t answer that.

In Part 2 I will look at what I discovered. The pitfalls of running a development host within a virtual host and more on why I really do love UNIX an awful lot.

(Ok perhaps I’ll leave that bit out :-))

Categories
lisp

Fundamentally Lacking

A while’s back I wrote up my experiences of trying to learn Lisp through various books and I got this comment:

If you want to go down the rabbit hole however (and since you asked), here’s my advice: read SICP. If you’ve been programming in Algol-inspired languages for the last 15 years, your head will probably explode. It is worth it. Do the examples. Follow along with the videos: they contain all kinds of little nuggets of wisdom that may take time to sink in.

SICP, I thought, what the hell is that? It transpires that this book is, as they say in colloquial circles, like frikkin’ huge man. Huge in fame (it’s taught in a lot of places that I could only dream of attending when I was as an undergraduate) and, as it turns out, also huge in terms of sheer quantity of information inside it. So I ordered it on Amazon, waited a few weeks for it to arrive and then queued it up on my bookshelf.

About a month ago I started gently plodding my way through it, scratching my head, doing the exercises and all. It really is very good, but I have no idea how I would have done this as an introductory programming course at Uni. Let’s just say the exercises can be a little challenging, to me at least.

The amazing thing is I’ve only really just finished Chapter 1 and I feel like I’ve put my head through a wringer. So I’d really like it if you can come and visit me at the funny farm after I’ve done Chapter 5. Visiting hours are 9-5 and please don’t shout at the other patients it freaks them out. Anyway, today’s mind melt is Exercise 2.5 and it asks us to …

Exercise 2.5. Show that we can represent pairs of nonnegative integers using only numbers and arithmetic operations if we represent the pair a and b as the integer that is the product (2^a * 3^b). Give the corresponding definitions of the procedures cons, car, and cdr.

Up until now all the really hard problems had ‘Hints:’ in parentheses. That’s usually how I know when I’m going to be screwed. I wrote some numbers down, first on paper then in Egg-Smell but I just couldn’t see how I was going to extract two numbers from the encoding. Then, bizarrely, whilst reading about Church numerals (eek) I discovered that there’s a fundamental theorem of arithmetic that I never knew about. How does that happen? How did something that fundamental escape me all these years? Reading it was like being told by a German co-worker that the only time an apostrophe is appropriate in “it’s” is for a contraction of “it is”. Scheiße.

Not only is there a fundamental theorem but it suggests the answer to Exercise 2.5. Perhaps one day I’ll be able to program & punctuate. One day.

Categories
programming

The Free & The Damned

I sometimes wonder if I worked in a company that made software for others, instead of a company that makes software for itself, whether I would be a better: programmer, ventroliquist and lover. Ok, scratch the bit about the lover.

There are many potential reasons why this might be so but I want to focus on just two.

  1. When the software is the company’s business then as a developer you are closer to that business with all the benefits that brings. Essentially it’s the difference between supporting the business and being the business
  2. The second point is the related point that when software is the business, rather than for internal use, it is necessarily of a higher standard.

Item #2 is because your internal users pay for their software only indirectly. They don’t sign a EULA, and the cheques that they do sign are for salaries and aren’t budgeted in quite the same way. Our internal users will put up with sub-standard sofrware that people who have to sign a EULA and pay hard-green for will not.

The thing is that working for a company that doesn’t produce software as its primary function is still great fun. For much of what I’ve been doing for that last few years the business area I work in is only tangentially computer related. That is computers & technology are critical to the business because of the level of automation they provide not for any other reason. This makes the technology they use the means and not the ends.

This is both liberating and damning all at once. It’s liberating because I’m free to ignore decades of good software engineering practice if it is profitable and expedient to do so. Put trivially, if my software could directly generate revenues then time spent making it compile without warnings is time that it could be accumulating wealth. It’s damning because all those unfixed warnings are going to make an expensive mistake one day. I, as the programmer, have to choose where to draw the line.

Good quality software should be well-designed on very many levels: interaction, architecture, performance, etc. Anyone who buys software should demand good quality software or their money back. Therefore I would think (obviously I don’t know :-)) that being both free & damned isn’t necessarily a bad place to be. At least you’ll get the chance to make a bit of lucre while you’re free and when you’re in hell you’ll always have a job refactoring.

Categories
programming

Constrained By Types (In Another Dimension)

I just read this. Which was interesting. I love the way that Steve has a simple point to make and spends 000s of words doing it, the posts always seem to ramble a bit (a little like mine!) but they’re usually fully of interesting tidbits and insights into software development so it’s usually worth spending 30 minutes or so on his issuances.

You can write C++ like straight C code if you like, using buffers and pointers and nary a user-defined type to be found. Or you can spend weeks agonizing over template metaprogramming with your peers, trying to force the type system to do something it’s just not powerful enough to express. Guess which group gets more actual work done? My bet would be the C coders. C++ helps them iron things out in sticky situations (e.g. data structures) where you need a little more structure around the public API, but for the most part they’re just moving data around and running algorithms, rather than trying to coerce their error-handling system to catch programmatic errors. It’s fun to try to make a bulletproof model, but their peers are making them look bad by actually deploying systems. In practice, trying to make an error-proof system is way more work than it’s worth.

This post raises a point that I hadn’t really considered before which is that perhaps we should consider static types to be a form of metadata, much like comments. The more static types you have the more constraining your model will be on the system that you’re creating. This is as it should be because that’s why you created those static types in the first place, right? But that model could just as well not exist. You could have created a system without all that new-fangled OOP crap and it might be a lot less complex. You could have the whole thing written in half-a-day and still be home in time for the footy.

A few years ago I was assigned to a trading system project that was to replace an existing legacy system. The existing trading system was single threaded, multi-user and suffered all sorts of performance & concurrency problems. One of it’s strengths, though, was that it was partly written in Tcl. Now Tcl isn’t one of the worlds greatest languages but it is dynamically typed and that gives it a certain flexibility. For instance, the shape and content of our core data was changing fairly constantly. Now, because that data was basically a bunch of name-value pairs inside the system it was possible to change the shape of the data in the system while it was running. I doubt that this ‘feature’ was ever consciously designed in to the legacy system from the beginning, but the flexibility and simplicity it gave was really very powerful.

When the replacement system came it was written in C++ and Java and had its own language embedded within it. It was a technical triumph in many ways and represented a tremendous leap forwards in many other ways. But the flexibility had gone because it would have taken extra effort to preserve that flexibility using statically typed languages. As a result the new system was faster and cleaner but more fragile and much harder to maintain. It was only when we started turning the legacy system off that it occurred to me that we had sacrificed something really quite important without really knowing it.

This flexibility, then, obviously has the downside that if our system is not very constrained it will most likely be a total mess. That was one of the drivers for replacing the legacy system in the first place. Moreover, this is especially likely to be true if there’s more than one person working on the system. Those static types clearly served a purpose after all in allowing us to categorise our thoughts about the behaviours of the system and then communicate them to others.

I suspect the solution is like almost everything, it’s a compromise. Experience will tell you when you need to constrain and when not to. Indeed, this is pretty close to the actual conclusion Steve comes to as well. In practice though I suspect that on large complex projects you would have to be very disciplined about the programming idioms you are going to use in the absence of those static types. It’s another dimension to the programming plane, and I don’t need to tell you that there are quite a few other dimensions already.