For the past few years I’ve been responsible for maintaining a subsystem that allows a point-of-sale system to communicate with external applications. These external applications may display information gathered from the POS and poke inputs back into it. I was involved in the initial development of the subsystem back in 2004 or so, and when the other developers left for other positions it fell to me to continue enhancing and supporting the subsystem.

I wasn’t happy with the first implementation because of some design problems, but I couldn’t change the design too much without impacting existing integrations with the subsystem. Instead, over the course of a year or so I made several improvements to the bits of code that integrated the subsystem with other applications and improved the API that the subsystem exposes.

Considering the number of transactions it supports every day across hundreds of installations, I think the subsystem is rather fast and stable. Occasionally, however, whenever there is a performance issue with an application that uses the subsystem, the subsystem itself gets blamed, and I am usually called upon to investigate the issue. While there are some scenarios that can trigger bottlenecks with the subsystem, the problems I investigate are pretty much never the fault of the subsystem.

Finally, after having to explain the intricate technical details yet again, I decided to come up with an analogy to explain how the subsystem works, how it could be a bottleneck is some scenarios, and why it usually wasn’t the bottleneck in spite of the possibility. I e-mailed the analogy to all interested parties, and it seems to have had the desired effect. I’ve edited the analogy to remove specific product and subsystem names and provided it here. In the wrap-up, I changed the name of the subsystem to NPS, for Note-Passing Subsystem.

The Story of Olympic Note Passing

We’re going to imagine a strange sport called Olympic Note Passing. Two runners run around adjacent circular tracks: Runner A, and runner F. Each time runner A reaches the outside portion of the track he picks up a message that he will deliver to runner F. Runner F will then deliver this message to the other side of track F. Never mind why this happens; some sports are just unusual (think golf or curling).

Now, the first time this sport was introduced into the Olympics, the messages were passed from runner A to runner F via a specially designed, and somewhat magical, box. This box accepts messages via a device like a bill acceptor, and on the other side of the box is a device like a bill dispenser that spits out formatted, translated copies of the messages that were put into the other side. This way, runners from different nations that speak different languages can still participate in the sport and understand the messages being passed from A to F.

Runners on the two tracks could run at independent speeds, so that runner A could stop for just a moment and place a message into the box before taking off around the track again, and runner F could stop for just a moment to pick up any messages that were waiting.

The original practice of Olympic Note Passing

There was a problem with this magic box, however. If runner A was placing messages into one side of the box, runner F could not take any messages out of the box until A was finished. Likewise, if runner F was taking messages out, runner A couldn’t place any messages into the box. This turned out to be a problem because the runners had to keep stopping and waiting for each other.

This state of affairs threatened to kill the fledgling sport before it ever got started, since spectators didn’t want to see the two runners stopping, starting, and waiting for each other all the time, so the Olympic committee had to come up with a solution. One intrepid committee member suggested designing new magic boxes so that messages could be inserted and extracted at the same time, but the committee had already spent a lot of money designing the existing boxes, and any new boxes would have to be the same size, shape, and color as the existing boxes and would have to work EXACTLY the same way in every other respect. Furthermore, the guy that designed the original boxes was now working for NASCAR and doing equally silly things to the Car of Tomorrow.

The way the committee decided to fix this problem was to station a person on either side of the magic box: helper A, and helper F. Now, runner A just hands messages to helper A on the way around the track, and helper A inserts the message into the machine while runner A keeps moving. Likewise, helper F extracts any waiting messages from the machine and hands them off to runner F when he comes around the track. If helper A sees that the machine is busy and he can’t insert any messages, he will just put the new messages in his pocket until the machine is available. Helper F will pull messages out of the machine as soon as they are available and keep them in his pocket until runner F comes around the track. If helper F has no messages, he will wave runner F on around the track.

New and improved Olympic Note Passing

Now, we’ll bring all of this around to the actual NPS design. Runner A is an AwesomePOS system, and Runner F is FooApp. The magic box is NPS. The messages being passed through the magic box are NPS messages. The redesigned sport, with helper A and helper B, represents the new NPS integration design introduced at SuperCustomer in late 2009 and early 2010. The people stationed on either side of the magic box are the independent threads that are now running in AwesomePOS and FooApp that service the NPS message queue.

With this picture in mind, it should be a little more apparent why NPS is pretty much never the source of any performance issues in current implementations, but why it can be a performance bottleneck if the integration is not done carefully. If AwesomePOS or FooApp have to keep stopping and starting to deal with NPS, it can impact the performance of the entire system. With the new design, however, NPS is much less of a bottleneck since runners A and F can now keep moving around the track a lot more smoothly, but messages can still get from one side of track A to the other side of track F no more quickly than the slower runner can move.

Please don’t design sites that look like this:

Waste of space!

Rather, please design sites that look like this:

This uses space more effectively.

I’m not knocking Herb Sutter, here. I’m pretty sure he’s just using a stock template, but templates that don’t flow to fit the browser width really puzzle me. Is there a good reason for them that I’m just not aware of?

My wife and daughters, bless them, are not. Well, my daughters are somewhat nerdy, in a hip kind of way, meaning that they can quote large portions of Star Wars and Lord of the Rings, but they can also put together an attractive ensemble to wear out on the weekends. They aren’t as hopeless as their father.

Some days when I come home and sit down to dinner, I want to tell the family about some really silly thing that happened to me at work, but I don’t want to have to explain the intricacies of programming to set up every story. Instead, on the drive home I’ll try to come up with an analogy. Here’s the story that I came up with for something that actually happened to me last week. See if you can figure out what the actual technical issue was.

The Story of the Wrong Fish Food

E-mail #1: “Paul, I just fed a batch of fish food to the fish, and the fish are dying. A chemical analysis showed that the batch contained hydrogenated frobulaxis. What’s wrong?”

Reply to e-mail #1: “Our fish are allergic to hydrogenated frobulaxis. Mix up a batch of food that doesn’t include that ingredient, and then the fish won’t die.”

E-mail #2: “Paul, I’m confused. Mix what up? I used a batch that I found in the supply cabinet. Why are the fish dying now?”

Reply to e-mail #2: “Either someone put the wrong batch in the cabinet, or someone used the wrong recipe to make the batch that’s supposed to be in the supply cabinet. See if you can track down the recipe that was used for those batches.”

E-mail #3: “Instead, I gave the fish some medicine that keeps them from dying when they consume hydrogenated frobulaxis. Easier.”

Reply to E-mail #3: “The problem is, that’s still wrong. There shouldn’t be any hydrogenated frobulaxis in the batches that are fed to the fish. If that’s being added to the production batches, that’s bad.”

Of course, I never got an e-mail #4. The fish were alive (barely), and nobody had to do the nasty work of mixing up a new batch of food, because that would be icky. All I could do was just pound my head on the desk.

Sigh.

This is a high-school student.

This is a high school student.

See the difference? The first picture is of a high-school student (my elder daughter, Madeline). The second picture is of a high school student (Sean Penn as Jeff Spicoli in Fast Times at Ridgemont High). That’s the difference a hyphen can make in the meaning of a sentence. The first picture depicts a typical 16-year-old girl; the second picture shows the kind of guy she won’t be allowed to date.

American English is really lenient about the use of hyphens in compound adjectives, but I’m seeing a lot more sentences lately where the missing hyphen makes a difference in the meaning of the sentence.

(I’m pretty sure there are some grammar mistakes in this post. Fire away!)

Before I launch into the body of this essay, let me say that I think meetings are wonderful things. I’ll even go so far as to say that people in my field, software development, probably don’t have enough meetings. I have to qualify these opinions, however, by saying that they only apply to the right kinds of meetings. Regularly scheduled project status meetings that involve an entire team are definitely the wrong kind of meeting. They provide very little useful communication, and they kill productivity. There are better ways to accomplish the purpose of status meetings without actually having them.

What’s Wrong with Project Status Meetings?

A project status meeting usually involves a manager and a team. Each member of the team, in turn, provides an update on the tasks that member has been working on since the last meeting. The manager or other team members may ask for additional information or provide comments on these updates. Team members that have little or no involvement with the tasks that are being discussed just sit and listen (or sleep, or doodle, or do what I do occasionally — write code long-hand on a pad of paper).

In a particularly malignant version of the status meeting, several teams and their managers are in a meeting with a higher-level manager, but each individual team member still reports on his tasks. On one project, I had meetings like this on a weekly basis, with upwards of 40 different people in one room. It was dreadful.

What makes these meetings so dreadful for me is that I’m only an active participant for two or three minutes. For the rest of the time (an hour and a half, or more, once a week, on my last project) I don’t have any real involvement with what’s being discussed. I’m a captive. I can neither provide any useful commentary on the discussion nor gain any useful insight from it. Even worse, I’m a consultant that bills on an hourly basis. I hate wasting my client’s time on uselessness when I could be doing something more productive with it.

Why I Think We Do It This Way

For several reasons, holding a management position is thought of as being a few steps up the career ladder, above the lowly status of the worker bee. I happen to think this situation is unfortunate, but that will be the topic of a later essay. In any case, people currently in management are usually older than the people that they manage, and they usually learned what they know about office interaction when they were younger, or at the very least from people who are older than they are. This means that the habits that are ingrained in managers may reflect a technological state of the art that was current ten or twenty years ago, maybe more.

Before the advent of telephones, e-mail, instant messaging, and wikis, the only way to get a timely update from several members of a team was to gather the members of the team together at a pre-arranged time and place. Fortunately, the telephone pre-dates most people currently in management, so for widely dispersed teams the conference call is in common use.

I try to take advantage of status meetings held both in-person and via conference calls by arranging my schedule so that my virtual working hours fall during the times that the status meetings are held. I dial into the meeting on my cell, using a headset so that my hands can stay on the keyboard. I listen for my name, provide my update and any comments that are relevant, and for the rest of the time I’m writing code, debugging, documenting, or testing. I’m less productive than I would be if I didn’t have to split my brain between the meeting and other tasks, but at least my downtime isn’t a total loss.

A Better Way

Rather than holding status meetings in person or via conference calls, why not take advantage of that wonderful new invention called e-mail? Every team I’ve worked on since 1993 has had e-mail. It’s safe to say, in fact, that I would refuse to work on a team in which one or more members didn’t have an e-mail address. E-mail has reached a comfortable level of ubiquity.

Here is what I propose: On a regular basis (for example, before noon every Monday) each team member would send an e-mail to the team’s manager and copy every other member of the team. The subject of the e-mail would clearly state that it contains a weekly status update: "Status Update for 2011/01/31." The body of the e-mail would contain a suitably detailed list of that team member’s assigned tasks and their status. It might also list the accomplishments of the past week and the problems that are retarding progress on current tasks.

Each team member could read these status updates at some point in the day and perhaps send a follow-up e-mail to other relevant team members to provide additional comments or ask questions. A meeting could be scheduled among interested parties if a topic needed to be discussed in more depth, or a meeting may just happen spontaneously in a phone call or around a whiteboard.

This kind of arrangement provides several advantages. First of all, it eliminates the need to synchronize everyone’s schedule around an event at a particular time and place every week (or every day, if a team is particularly dysfunctional). Second, it allows a great deal of flexibility in providing status updates to the team. Some people might compose their e-mails on Friday afternoon, some on Sunday evening, and some at 11:47 on Monday morning. Whatever fits into each person’s style and schedule will work.

Finally, and most importantly to me, if one is really in a flow and madly banging away at the keyboard in a productive state that I like to call "code falling out of the fingers," there’s no need to stop that state of productivity to listen to someone else drone about something which has little or no connection to whatever problem that person is thinking about. Each team member can reach a comfortable stopping point and deal with writing or reading the status e-mails on his own time.

An Aside: Flow Is King

In software development, particularly, achieving and maintaining flow is crucial for productivity. I worked on a project a while back that had a project status meeting every day at 11:00 AM. This REALLY killed my flow, because I like to spend the first part of each morning taking care of busy work, and by 9:30 or 10:00 I want to be in a really deep flow that may last until 1:00 or 2:00 in the afternoon, or whenever my hunger pangs finally remind that it’s time for lunch. Once, when my wife and I were engaged, she surprised me at my office while I was in this flow state. It took several seconds for me to even recognize her, and a few more seconds to be able to shift gears into human-to-human conversation. The flow state can be that deep!

If you’re a manager, you want your team members to be able to achieve and maintain a state of flow. Please, try to avoid doing anything that might interrupt it.

The Right Kinds of Meetings

As I wrote earlier, not all meetings are bad. I am a big fan of meetings in which all of the participants are focused on a particular issue. The meeting need not have a published agenda; that can often be a bad sign. All that’s needed is a clear statement of what the meeting needs to accomplish. For example, "We need to figure out what databases contain the data we need to gather for the migration from product ABC to product XYZ." That’s a meeting that has a chance of getting something done.

Another favorite of mine is the ad-hoc meeting, where two or three people might be discussing an idea or a problem, and before long they’re scribbling on a whiteboard or sitting around a computer trying out ideas.

What these kinds of meetings have in common is that all of the participants are engaged. They’re working toward a common goal so that they can get work done based on the output of the meeting. Project status meetings, in my experience anyway, never meet this description.

Why Not Try It?

I don’t know why, but I have yet to convince a manager that holds conventional status meetings to switch to e-mail updates instead. I’d like to hear from some managers about this idea. Why might it not work? Are there downsides that I’m just not seeing?

When you click on that shortcut that says, “Command Prompt” in Windows XP, or you run cmd.exe from the “Run” box, you’re not starting a “DOS prompt.” What you are starting is a command line interface, or just “command line” if you prefer. If you haven’t actually run COMMAND.COM, it’s not DOS!.

So, please, next time don’t tell me to run your favorite utility “at the DOS prompt.” Let DOS rest in peace.

The first person to call Powershell a DOS prompt will get the lecture in real time.

I gave my opinion a couple of times, coming down on the side of non-flat buttons. The whole discussion, though, reminded me of a couple of passages from The Hitchhiker’s Guide To the Galaxy that describe the design of the ship The Heart of Gold.

The cabin was mostly white, oblong, and about the size of a smallish restaurant. In fact it wasn’t perfectly oblong: the two long walls were raked round in a slight parallel curve, and all the angles and corners of the cabin were contoured in excitingly chunky shapes. The truth of the matter is that it would have been a great deal simpler and more practical to build the cabin as an ordinary three-dimensional oblong room, but then the designers would have got miserable.

That sounds like everything I dislike about UI design these days, particularly in Flash and Silverlight apps where designers feel compelled to recreate UI widgets that behave almost, but not quite, entirely unlike standard widgets.

A loud clatter of gunk music flooded through the Heart of Gold cabin as Zaphod searched the sub-etha radio wave bands for news of himself. The machine was rather difficult to operate. For years radios had been operated by means of pressing buttons and turning dials; then as the technology became more sophisticated the controls were made touch-sensitive — you merely had to brush the panels with your fingers; now all you had to do was wave your hand in the general direction of the components and hope. It saved a lot of muscular expenditure, of course, but meant that you had to sit infuriatingly still if you wanted to keep listening to the same program.

That sounds a lot like where flat buttons are headed. In fact, it sounds strikingly like an iPod.

So, I’m dying to know why gadget companies feel that it’s necessary to load down every product with useless stuff like this? If a professional geek like me doesn’t even bother to use these features, then who is using them?

I’d much rather see gadget makers spend some time and resources on the core functionality of their gadgets (do you hear me, phone people?) than on browsing, social networking, or photo viewing.