原创 How I test software, again and again (Part 2)

[Continued from How I test software, again and again (Part 1)]

Hand checks

Now we get to the real meat of the issue, which is how to do hand checks. I've been telling you for some time now that I do a lot of testing, including testing every single line of code, usually as I write it. When I say testing, I mean give the code under test some inputs and verify that it gives the right outputs.

But where do those inputs come from? From a test case, of course.

Now, almost all of my software tends to be mathematical in nature—old-time readers know that. So my test cases tend to be numeric, and so are the expected results. But the nature of the software really doesn't matter. Regardless of the type of data, your code should still process it correctly.

Remember the Adams dictum for software testing:

Never ask the computer a question you don't already know the answer to.

Just picking inputs at random (that word again), looking at the results and saying "Yep, that looks right" won't get it done. You have to know it's right.

In the good old days, I used to generate the expected outputs by laborious hand calculation. Typically, the only aid I had was a pocket calculator or, worse yet, a Friden (www.oldcalculatormuseum.com/fridenstw.html). Sometimes, I even had to compute the expected results in octal, because I was dependent on core dumps to see the internal computations. It was not fun, but it had to be done.

Fortunately, today there's a much better approach. Today, we can choose between math-oriented applications such as MATLAB, Mathcad, Mathematica, and Maple. I usually find myself using Matlab or Mathcad, or both.

Enter Mathcad

Long-time readers might remember my heated rants against Mathcad (now sold by PTC). I complained loudly and longly how frustrating it was to use, and how much it was plagued with bugs.

I haven't changed my mind. But a funny thing happened. One day I realized that, despite my frustration with Mathcad, I use it almost daily. It's by far the most oft-used application in my computer. The reason is simple. Despite its faults and my complaints, it's still the best tool of its type, bar none. It's like the old joke: it's the only game in town.

Yes, I know, the other tools I mentioned tend to be much more powerful (and a whole lot more expensive) than Mathcad. But only Mathcad lets me both enter math equations in a WYSIWG format and get the results back the same way. When I'm doing math analysis, either as a pure analysis or to generate test cases, I like to use the application to both do the calculations and document the process. With Mathcad, I can interleave the math equations, text, and results to get a worksheet that looks like a page out of someone's textbook.

And yes, I still curse Mathcad sometimes. It's a love-hate relationship. I hate the frustrations, but love the final results. Come to think of it, it's the same relationship I have with Microsoft Word or Windows.

Back to testing

So here's how it works. To test a given bunch of software, I'll start with a reasonable set of inputs. They may be "photo-realistic," meaning that they're exactly the same values I'd expect in the real world. But this isn't necessary. Simpler inputs will do. If, for example, I use an initial radius of 6,000 km and the software operates on it properly, I feel reasonably satisfied that it will also work for 6378.137. Arithmetic units are good at that sort of thing.

Anyhow, I'll start with those inputs and let Mathcad calculate the expected outputs. I can also make it show me the expected value of every single intermediate variable in the calculation. When I'm done with this step, I save the file, and now I have a detailed, permanent copy of the test case. I can revisit it whenever I like.

Now it's time to go to the UUT. I'll invoke it with the same inputs. Now it's time for me to do my single-stepping thing. Sitting in the source-level debugger, I can step down through the code, checking the results of every single computation against the value Mathcad got. When I'm done, I promise you, I'm just about 100% satisfied that my software is correct.

Draw a graph

Well, make that 98%. How do I get the remaining 2%? By doing it lots more times, with lots of different sets of input values.

My next trick is one I only got into in the last few years, and I'm sharing it for the very first time.

This stunt is like the random-number thing, only done with more elegance, and for a far more useful purpose. In general, a given bit of code will not have to perform for every single input value, from +ì to –ì. There will be some range of input values we can expect. The random-number buffs got that part right.

Now suppose I give one of the parameters a value on or near one end of the range. I get a result that agrees with the Mathcad model. Now let me give it a value near the other end of the range. I get a different result, that also agrees with my test case. Let me then sprinkle some values in between, and I can see how the outputs vary when I change the inputs. I can even graph them.

This is where the math aid earns its keep. Like its rivals, Mathcad will operate on a whole range of input values, as easily as it does for a single set. I can make the mesh as dense as I choose. And I can make it draw a graph of the way the outputs respond to changes in the inputs.

Now let me do the same thing with my UUT. I'll set up an input file (hardware folk would call it a test vector), and feed the entire set to the UUT. I can then store the resulting outputs in a file and plot them.

If the two sets of graphs agree, I'm happy. Now I'm 100% sure the code is good.

As you can tell, over the years I've learned some really neat things about how to test software. Now you know them too. Use them in good health.