Software Development and the Memory Palace of Matteo Ricci

One of my regular goals as a software developer is the intentional reduction of cognitive load—that is, I try to reduce the number of things I have to think about to get the job done. This applies to interfaces used by the customer, but it applies in spades to the interfaces used by the programmer: which is to say, APIs of the modules of code in the system, and the interfaces of the tools used during development.

It’s like this. A large software system is fiendishly, horrendously, amazingly, enormously, annoyingly complex. It is simply not possible, even if you’re the only author, to keep all of the details in your head over a long period of time. (I’ve been working on one project at work for almost ten years, and every so often I run across pieces of code that I know I must have written but that I have no memory of.) Consequently, I try to design my systems so that I don’t need to keep all of the details in mind.

Despite the title of this post, there are many tricks. The first is picking the right architecture. In this sense, your system’s architecture is the way you split it up into pieces. Your architecture should be simple enough that you can keep it in mind; and then, when you’re faced with a bug, or with a new feature to add, you can easily say, “Aha—it has to be over here, in this part of the system.” And then, once you dive into that part of the system it should have its own architecture as well that helps you navigate around in it; and so on.

The result is not unlike The Memory Palace of Matteo Ricci. Ricci was a Jesuit missionary to China in the long and long ago; and he came up with a scheme for memorization. You imagine a castle of many rooms; and in your imagination you furnish each of the rooms in great detail; and you attach the things you need to remember to the furnishings in the room. If you wanted to remember the names of the Seven Wonders of the World, you might attach them to seven colored handkerchiefs in the little drawer at the top of the baroque cabinet in the niche on the left side of the blue marble room just inside the garden door, the one with the carving of Alexander the Great on the front.

Now me, my skills at envisioning such complex scenes in great visual detail are lacking; which is no doubt part of way I’m not a Jesuit. But Ricci could do it, and I’ve seen the same scheme described often enough that it must work for somebody.

In my case, I do it differently. I partition up the system, and remember the pieces; and I partition up the pieces; and so on. And there has to be a logic to the partitioning, because that logic means that I can find my way about, and keep the bits of complexity in different boxes. And that’s key—you only want to deal with one or two boxes at a time.

That’s decreasing the cognitive load.

Another aspect is the careful design of code interfaces. Module A calls module B using module B’s “application programming interface” or API. (Module A and Module B are two of the boxes, and the API is where I drew the line between them.) Now APIs can be designed to make things easy for the caller; or they can be thrown together such that the caller has to do a lot of work understanding them each and every time he looks at them. You want to do the former.

Here’s an example (some of you might to gloss over this, down to the bottom line). On most systems, the Tcl interpreter (or “Tcl Shell”) is a program called tclsh. Quill often calls the tclsh to do things: run tests, run the user’s application, or any number of other things. Suppose I want to run a test script, mytest.test. Tcl makes it really easy to call external programs, and so I might just do this:

 exec tclsh mytest.test 

That executes mytest.test as if I’d entered tclsh mytest.test at the system command line. And on many systems, it will work. On Windows, though, the Tcl shell is called tclsh.exe.

if {[os flavor] eq "windows"} {
    set tclsh "tclsh.exe"
} else {
    set tclsh "tclsh"
}

exec $tclsh mytest.test

Well, now, that’s ugly. I certainly don’t want to have to do that everytime I want to use the tclsh to execute a script. So I added a command to add the appropriate extension to the program name, when running on Windows:

 exec [os exename tclsh] mytest.test 

Since the logic to determine whether “.exe” is needed or not is in the os exename command, the user doesn’t need to think about it.

But there’s another problem. The exec command will search for the tclsh program, but might not find it. Things would be smoother if we found the Tcl shell ourselves, and gave exec its precise location. And then, we could search for the Tcl shell in a number of ways, or even read its location from a configuration file. So we get this:

 exec [plat pathto tclsh] mytest.test 

The plat pathto command not only looks for the tclsh, who cares how, it also takes care of adding the “.exe” if its needed.

So this is better. But what if the Tcl shell cannot be found? Maybe we’ve been given an explicit location to use, but the file isn’t there, or perhaps none of our search mechanisms worked, and we found nothing. In that case, the above command will fail with an error; and we want it to be a useful error that makes the problem clear to the user.

So try this:

 exec [plat pathto tclsh -required] mytest.test 

The -required option tells plat pathto to output a error message to the user when it cannot find the Tcl shell program, explaining that it cannot find it and what they might do about it.

Now, we usually need to set up some things before we invoke the Tcl shell. In particularly, we often need to set the TCL_LIB_PATH environment variable so that the Tcl shell can find the project’s libraries. So we get this:

set ::env(TCL_LIB_PATH) [project libpath]
exec [plat pathto tclsh -required] mytest.test

When there’s work that the caller of an API needs to do every single time, that’s a sign that you’re not providing the right API. Every time I call the Tcl shell in this way, I’m going to need to do all of these things. Why don’t I just define a proxy command that represents the Tcl shell program and handles all of this unpleasantness? Then my program can simply say,

 tclshell mytest.test 

The new tclshell command looks up the path to the tclsh program, taking the operating system into account, and displays a useful error if it cannot find it. But if it can, it sets the TCL_LIB_PATH environment variable, and executes the Tcl shell, passing its own arguments along, and returns the result.

Having to think about whether the Tcl shell executable is available, and how to find out, and what to do if it isn’t, and setting it up properly, all of that places an enormous cognitive load on the programmer. The new tclshell command, by contrast, “does the right thing”. It does what’s wanted under the circumstances, and so the programmer no longer needs to think about it. It reduces the cognitive load—which frees up mental resources for other things.

And here’s the bottom line. That’s really what the Quill project is all about: reducing cognitive load. Quill automates common development tasks such that the user can rely on them without having to think about them too much. For example, I want to start work on a new project. There’s a lot to put together to make that happen; but at the highest level, it’s all pretty much the same as the previous project. So let’s automate that:

$ kite new app my-project myapp

Bang! I’ve got a project tree that I know works just like my last project tree; all of the things I’ve learned to do by reflex still apply, because everything is in the same place and follows the same standards.

____
photo credit: campra via photopin cc