There seems to be a common, not unfounded, view that programming is really hard. The response from tool vendors is to produce products that make programming simpler. They do this by introducing a user interface that ‘simplifies’ what would ordinarily be a programming task into something that mere mortals can understand and program without understanding too many of the fundamentals of the task at hand. I think the view that such tools succeed at what they set out do is broadly untrue.
Up until this week I had intimate experience of only one such tool and that was BusinessObjects. Now I’m going back a bit here but in 1998 BusinessObjects made a tool that allowed a designer to take your ugly enterprise database and turn it into ‘Business Objects’. These objects contain a little more ‘information’ about the data you were looking at and made it possible to compose those objects in ways of your choosing, but crucially, they did this with the aid of graphical tools that didn’t make it look like you were really writing SQL queries at all. This then, in-turn lets a common Jo-user compose a view of that data for a report or particular query they need without having to learn SQL. In concept the idea has tremendous power because you can do things that you could never do before without a lot of expensive IT expertise. The reality, for us at least, was a little different and this was I think for two reasons.
Unapparent Inefficiencies
A normal SQL programmer would know about the indices and relationship cardinalities on a table, hence they would know which joins would suck. The abstraction provided by BusinessObjects would happily hide those inefficiencies of the underlying representation. That made it it really easy to kill the database with a single mouse-click and drag. You can’t just blame Jo-user for this either, when you hide the inefficiencies I would not be surprised if a seasoned programmer would sometimes make the same mistakes that Jo did.
Apparent deficiencies
BusinessObjects, of the time, was a language in itself. Indeed the BusinessObjects language is in fact an abstraction of another much more general purpose language that we know as: SQL. Programming languages that are developed for a specific domain (or Domain-Specific Languages) tend to exhibit their own behaviour. They make some things easy at the expense of making other things less so. The trick, with these tools, is to make sure that all the things you make hard are things people don’t really do very often anyway. The problem for us, at the time, was that we were putting a clean layer of abstraction on-top of a not-so-great database design. BusinessObjects wasn’t able to cope very well with those apparent deficiencies and so we had to go hunting for workarounds to put into place until we could get back and refactor the pimple on our software arse away.
In the end the project limped on for a bit and then I got moved onto another task and I lost track of it. Perhaps it lives on, but I doubt it. This week I discovered that Microsoft have a related but different purpose tool: SQL Server Integration Services (SSIS). Apparently it’s been going on for years under a previous guise of DTS but I’d never seen-nor-heard of it before. Anyway, I was initially very excited when I started working with it, I really believed that I could take my complicated data import requirements and transform them into a few diagrams and have all the work done for me. Yeah right. The reality was somewhat different, and like our friend BusinessObjects, SSIS coped badly with the unapparent inefficiencies and the apparent deficiencies.
The conclusion is that tools like this are complex and that complexity includes, but is not limited to, a lot of the complexity of the lower-level underlying representation that they must use. Often then, it will be far better to just use a general purpose tool (like SQL or Perl) to get you your reports or data transformations done. Don’t mess around with some factory-made ill fitting suit if you think you can tailor one yourself for less. No matter how pretty the buttons are.
In the end I surmised that using gawk and bash I could produce something equivalent to what I wanted to do in Integration Services in a fraction of the time and hence cost. If I’d used Perl or Python I could have done it even better and/or faster. I had been hoodwinked my friends. Hoodwinked into thinking that I could write a program that was not there and discovered in the end that it was far easier to just knuckle down and make my own suit and have it fit nice, than use the pretty off-the-shelf one with the gaping crotch.
The problem, it seems, is that there is a market for tools that allow us to create code that seemingly isn’t. We still believe, as I did, that tools will somehow save the day. Dammit, why didn’t someone remind me earlier that there is still not, and never has and never will be any silver bullet.
4 replies on “The Code That Was Not There and the Badly Fitting Suit”
While there is no silver bullet, that doesn’t mean the tools that are available are always terrible.
It’s not always about what “mere mortals” can’t do either… sometimes it’s about efficiency. If I didn’t have certain tools to help me write Java, SQL, etc… it would take me far longer to get through an assignment.
It’s because I know how these tools work that allows me to take advantage of that too. I imagine many of the features I used would be useless to someone who didn’t understand exactly what that feature did.
Anyhow, tools save my day all the time.
While your never able to take away the essence of the work, that doesn’t mean you can’t shift it downstream a bit. There are heavy-weight, medium-weight and light-weight programmers. With a bit of magic, you can make it easy for light-weight programmers to do things like create web-sites. You can make it easier to code in Java then in C or definitely assembler. A lot of the modern IDEs seek to reduce time spent in refactoring. These types of tools reduce the entry point for getting the work done, thus allowing more people to qualify for the job.
We all want to believe there is something exceptionally difficult about assembling instructions for a computer that explain how to manipulate some set of data. There is to some degree, but we choose to make the problems harder than necessary. The difficulty doesn’t come from the list of steps itself, it comes from trying to extract them from the real world in a way that is general. It is the intelligence that we can’t mimic.
Paul.
http://theprogrammersparadox.blogspot.com/
Oh, BusinessObjects. That brings back memories. I have worked on a project that used and still uses BusinessObjects extensively to give users the ability to do reports against a whole lot of personnel data. It actually worked quite well for most users, both the novices and the more advanced. The real experts learned a bit of SQL and tinkered with the SQL BO generated to get what they need. But BO was and is extremely expensive. It’s quite probable that we could whip up something that allowed users to write and manage SQL queries and dump the results into Excel for the fraction of the cost.
First of all, with OO these days, you are never necessarily going to really be able to solve all of your “inefficiencies”. The “newfangled” OODBs and ORMs like Zope and ActiveRecord, respectively, will just tell you to throw more hardware at the problem (if there was a problem). But is this really a Bad Thing(tm)?
In most cases, people do want the logical schema to be abstracted away by some sort of flexible framework. The expectation is that the end-user *shouldn’t* have to know anything about database organization to be able to pick out the data they want to see or analyze, hence the development of the ability to run ad-hoc queries which can be built around metadata-driven interfaces.
In essence, worrying about the “underlying representation” ought to be obselete (or heading in that direction). This might even turn out to ultimately be a case of “premature optimization”.
In terms of automation, the logical extension of Model-Driven Architecture would be autogenerated boilerplate from UML structure and state views. You want the ability to have a tool like Visio able to generate your class definitions straight after you’ve designed it by drag-and-drop.