Wednesday, May 9, 2007

Teaching is the Best Software Test


With all of the emphasis that has been recently placed on software testing methodology, I thought I would share my final acceptance test that never fails to find those last lingering bugs and badly designed features. All you have to do is offer an advance training course to some of the typical users of the software before you do final delivery. Some of you already know what I'm talking about and are nodding your head as you read this. The rest of you, read on.

In my experience, teaching the first training class can be one of the most humbling experiences an application designer will ever encounter. First you will have to prepare a lesson plan, and once you start putting down on paper what the steps are to accomplish some example scenario, you will likely discover, as I did, that it doesn't seem so simple as when you were designing and testing. But suppose you are so supremely confident that you decide to wing it and just go in cold. After all, you designed the application, so what could you possibly need to look up?

"OK class, first I guess I'll have to walk you through creating some sample data so that we can have something to try out all of the features on. Hmm, we'll need a blank database to get started, so we'll just copy the one from the examples folder."

"I can't find the examples folder. Where is it?" someone asks.

"It's right there in the C:\MegaApp folder."

"I don't see it."

"Ok let me look. Hmm, are you sure you installed it in the same place as I demonstrated?"

"I don't know, I never pay much attention to that installer stuff. It's always the same."

"Ok here it is in the My Documents folder. I don't have time to fix it so you'll just have to remember to look here whenever I say to look in the MegaApp folder."

"OK class, let's move on. Use explorer to copy the blank database to a new folder."

"I can't find the blank database. What is it called?"

"Blank.dbs"

"I don't see it. Wait, there is an icon shaped like a cylinder labeled blank. Could that be it?"

"You must not have file extensions showing. Not really surprising really since that is the default, I suppose. OK class, listen up. Everyone make sure you have file extensions showing. The option is under folder Options under the Tools menu in explorer. Not finding it? I'll be around to show you in a minute."

Later: "OK, you should now have a copy of the Blank database in a new folder. Let's rename it Sample1.dbs. Done? OK, now double-click on the icon to open it in MegaApp. Problem?"

"Mine opened in Microsoft Word. Looks like garbage"

"Mine opened in something called PRODAS, and crashed."

"Mine just opened the folder in explorer. Were we supposed to rename the file instead of the folder?"

"OK," you say, starting to get rattled, "Apparently these PCs we borrowed from operations aren't as clean as the ones we tested with, and the DBS file extension might be used by a few other apps. Let's take a break while I work this out."

Much later: "Is everyone ready to get started again? OK, now as I was saying, the blank database Sample1 is now open in MegaApp. Question?"

"Mine isn't blank any more. I got bored waiting and filled out some data already."

"OK, just exit MegaApp and throw the file in the trash and make a new copy."

"I threw it away and emptied the trash, but I can't find another one to copy now. Wait, I remember, I decided to skip the copy step and just open the original. It didn't seem important at the time."

"Fine. Will someone please pass him a copy on a floppy? These machines don't have floppies do they? How about a USB stick?"

"I've got one, but when I insert it, it says I need to install a driver. Do we have an internet connection?"

"No, I'm afraid not. I'll tell you what, just uninstall MegaApp and reinstall it. It'll only take a few minutes. While we're waiting, are there any questions? Yes?"

"My screen just went all black. What did I do wrong?"

You're really starting to panic now, "Uh, I'm not sure. What were you doing before it went black?"

"Not a thing. In fact, I haven't touched it for over twenty minutes."

Relief, "Ah, just move the mouse a little. Is that better? It was just the screen saver. Ha, ha."

"OK, I'm finished uninstalling and reinstalling, and maybe I'm just dumb but couldn't all this have been avoided if there was just a New Database option in MegaApp?"

"Oh, well I suppose so. It didn't seem very important with all of the real features that we had to implement, and it's just a few steps after all."

...

And that's just the beginning. By the end of the day total exhaustion sets in from having to constantly troubleshoot 10 problems simultaneously. After a day or two like that you learn that you will do anything to keep the class out of Windows explorer where data magically vanishes down a black hole. Stay in your own app and odds are you won't be surprised more than five times in an hour, but avoid Windows like the plague.

There is a good reason why many modern software packages still include the tried and true installation instructions: "Insert the disk and click on the Start button and choose Run... Type D:\install where D: is the drive letter of your CD-ROM."

As retro as it sounds, it almost always works. If autoplay is enabled , you're usually golden, given you get past the virus checker or the Add or Remove Programs wizard if the user happens to fancy that method, but if not, those simple words printed on the cover of the CD-ROM will save you many support hours, because no one can talk someone through the steps of using a GUI to copy a file. Believe me, even if you can forget your muscle memory and you mirror each step yourself, it will all go horribly wrong because it can.

So schedule that advance training class. You'll be glad you did, but only after a lengthy recovery period.