Economics of Software Testing

Testing code is expensive. But without testing we cannot be sure that our software will perform reliably and predictably especially under failure or fault conditions.

It is critical therefore that testing resources are targeted in the most effective way which ensures the software is reliable, robust and ft for purpose.

Unit Testing

This article is about the economics of software testing. It is about where and what to test and in particular where you get most bangs for your bucks if you're or working to a budget.

What Makes a Good Test

In order to understand where the greatest return on investment (ROI) will be achieved from the Test Budget, we need to quantify what makes a good test. A good test is charaterised by the following:

  • It test some funcationality that is pertinant to a use-case.
  • It's purpose is to find bugs or prevent bugs creaping being introduced it latter iterations.

Tests should be Pertinant to one or more Use Cases

This means that there is no point in testing code that is never going to execuited - dead code. There is nothing to be gained writting and running tests which test code that is never run. Waste of time.

Tests should Find Bugs or Prevent New Bugs

This means that if you write a test just to show that a bit of code runs and produces a result you expect, it's not a very valuable test. You can do this just by stepping through the code. For example, if you have a bit of code that adds two integers:

public int add(int a, int b)
    return (a + b);

then write a test to show that 5 + 5 = 10, this is not a good test. In fact is a rubbish test because it does not even attempt to find any errors nor does it attempt to prevent errors from creeping into the code.

A better test will be to test what happens if the sum of a and b will be greater than the maximum value of an integer (Integer.MAX_VALUE). Such a test would completly change the mechansim of calculating the sum.

Unit Testing

Lets start of with the worst place to get return on those testing dollers to be invested. Unit testing is probably the most overrated biggest waste of time ever since the rise of the Design Committee. This is because these sort of tests test small bits of code in issolation without referance to other sub-systems and without referance to the over all business objectives.

Lets think about that: We have a developer who spends a couple of days developing some functionality. Then he spends the next few days writing tests to show that the class he just wrote does what he wrote it to do in the first place.

However, the class he's written depend on other sub-systems, and in the Unit Tests he's mocked these out. So now the tests he's written demonstrates that the object does what it should do only when its connected to a mock object of the sub-system, and the mock object behaves in a way that he's told it to.

In other words a Unit Test is a test that shows that a class will do all the things it should do by mocking out all the things that may cause the class to do behave in a way that it shouldn't do.

To some this may sound like sacrilege, but think about this analogy: You get a plumber round your house to fit a bathroom. You're already ordered the taps and several lengths of water piping. The plumber gets a tap out of the box and connects it to a bottle of water, turns it on and then turns it off. He nods smugly as water came out when it was tuned on and stopped when turned off. You notice that the water bottle has printed on it in bold letters WATER SUPPLY MOCK OBJECT.

That's how useful mock objects are.

He then takes each length of water pipe and pours water down it and checks that it comes out the other end. "Gee", you think to yourself, "What is this plumber doing?".

You're paying him £200 per day,

“What are you doing?”. “I'm unit testing” he says. “Listen mate”, you say, “Why don't you just fit the bath up, connect it to the mains and then see if it works and check for leaks.”

A shocked look comes over his face, “I can't do that, that's Integration Testing, you can't integration test until you've completed your unit tests. Don't worry” he says in a chirpy way, “It will only take a couple more days”.

In the meantime his mate is connecting your new Double Condensing Gas Boiler to a portable gas bottle and trying to get it to fire up. “What are you doing?” you ask wondering if you've been beamed to a parallel universe. “Oh, I mocking out the gas utility company, so I can unit test your boiler”.

You wonder if connecting the gas bottle to his ass could induce a flame to come out his mouth.

“Look”, you say, “I just want you to connect the pieces together, turn on the water and we'll see if it works. If any parts are broken then we'll find the broken bits and fix them”.

“What?”, says the plumber, “You just want us to fit the parts together and see if it works – without testing all the individual units and without mock object?” He holds up the jug of water with 'WATER SUPPLY MOCK OBJECT' printed on it .

At this point you realise what you thought was going to be a 2 day job is in fact going to take a week. And also that when you connect all the parts there is still no guarantee that the system will work as a whole.

In fact all that has been established is that your double condensing gas boiler will work when it's connected to a portable gas bottle, and when you pour water down a pipe it will come out the other end.

You still don't know whether it will work as a whole, so really, you dont know anything.

This, my friends is Unit Testing – expensive and mostly a waist of time.  But, on the bright side, you do know is that you've spent a lot of money.

BTW Unit Testing is a favourite trick if you outsource your software development. It's a cash cow! (that and Change Requests).


EasyMock is basically a framework that helps  you mock out the important stuff (so your test can work against mocked systems that behave in the way to tell them to behave). This is like testing how a soldier reacts under fire by pointing your finger at him and shouting Bang! Bang!

Lets look at an example:

    public void findAddressTest()
        // Create the mock object
        final AddressDB database = EasyMock.createMock(AddressDB.class);
        // Create the data that we expect to be returned
        final Address foundAddress = new Address();
        // Tell EasyMock to return the 'address' object when
        // it gets a zipcode of 'abc'
        // Tell EasyMock to setup the test
        // Create the target test object
        final FindAddress addressFinder = new FindAddress(database);
        // Call the method... and....
        final Address found = addressFinder.findAddress("abc");
        // Hey presto EasyMock returned the object that we told it to.
        // What a surprise!
        // What a useless test!
        Assert.assertEquals("Bristol", found.getTown());

The comments say it all. We setup the test to pass. We tell EasyMock what data we want returned and when we want it to be returned. In other words there is no test. Just a confirmation that the code runs. 

Just like poring water in a pipe and watcing it come our the other end.

EasyMock even provides a method org.easymock.EasyMock.verify(Object...), which has the stunning ability to check that the methods on the mock objects were invoked in the order specified. In other words it checks (I suspect they would call it validates.) the objects internal algorithm - white box testing.

So what ever happened to that fantastic idea of interface contracts and encapsulation - you know, that small thing where you should be able to change the implementation of a method, and so long as it abides by the contract then nothing should break! Oh - apart from all your verified unit tests!

Software Engineering Idiot test Number 1: If you're evangelical about Unit Tests then you are probably an idiot.

So here is a question:

What is the purpose of testing:

  1. To get good code coverage metrics.
  2. To show that the code compiles and runs.
  3. To show that if you put X in you get Y out.
  4. To find bugs.

The correct answer is number 4. Any other answer gets you a job at Mcdonald's. The whole purpose of testing is to find bugs. If your tests don't find bugs then they're crap tests. The test above is a crap test.

Where Unit Tests are Useful

So where are unit tests useful? Good question, and it may supprise you to know that there are a few places where unit tests are good.

Testing small bits of code that don't depend on external systems (databases and the like). For example the int add(int a, int b) method we mensioned earlier. Also where complicated algorithms or helper classes are used. So you may write a Unit Test to test a class that converts a object from one format to another.

Another good place for unit tests it to test any algorithms that use reflection. This is because reflection can lead to fragile code (if you using strings to find methods for example).

If you write a bit of complicated code (you know one of those pieces that can't be broken down into simpler methods), then write a unit test for it. But write the test to find bugs: what happens of some of the parameters are null? What happens under failure conditions.

Don't forget, writing tests is about finding bugs by probing the contract and therefor improving and making the software more robust.

To stick with the analogy of plumbing, when a plumber builds a complicated piece of pipe work (say connecting a bunch of radiators together), then they'll pressure test that part of the system mocking out the water supply with a pressure test pump.

Summing Up Unit Testing

Unit testing does have it's place, but when you start unit testing and using mock objects as a matter of course during your development methodology then your getting into the realms of wasting time and money. Just like the plumber, your better of waiting until you've put all the pieces together and then checking that they work as a system.

Integration Testing

Now this is where you get much more bangs for your bucks (excluding the pay-off at the end of the project). So integration testing is where you get all those bits and pieces of code, put them together and make sure they work as expected.

To stick with our plumbing analogy, integration testing is what the plumber will do when he's installed your central heating system: He'll pressure test the entire system as a single unit.

Note that he will not yet connect it to the mains, or the gas supply. He'll test it using a water pressure testing kit. He'll test the gas pipes and boiler by using a Manometer. And during these tests he will make sure the pipework can withstand upto 3 times the expected gas pressure (a stress test).

So the point here is that if all your integration tests works, then what was the point of all that time spent unit testing? And just remember that writting good unit tests will take up to 50% of development time.

So what happens if something doesn't work? Well its quite easy, you script a test that will demonstrate your fault (use Selenium or use JUnit if you want to make lower level method calls), and then investigate and fix. Most importantly make sure the test can be run automatically as part of your build/deploy process so you know other changes wont reintroduce this fault.

And voilà you have not only fixed your fault, you have created a test that is repeatable and runs through the whole stack, and ensures that all the various components work together happily.

Economics of Software Testing

OK, so this article is about the economics of software testing. So the question I need to answer now is the economic one.

Firstly if you've got crap coders who cant write a few modules without introducing a bunch of system threatening bugs, then you'll have to face the fact that no matter how little you're paying them, you're still gonna be spending a whole bunch of cash developing even the most simple system.

However, if you've got some decent engineers who know how to write a screen, validate values and send them safely to a data store then you could be onto something good.

Firstly, lets look at the economics of Unit Testing. Unit tests, if they are written really well (i.e. to probe the edge cases and look for bugs), then they will add between 60% to 100% effort your code writing development stage. You also need to be aware that when some code changes and 'breaks' the unit test, you'll have to the go fix the tests.

Get that for a statement of how useful Unit Tests are. You write some good code and your unit tests break. I thought it was supposed to be the other way round!

Take a look at the graph below. The orange bit is how much cost is added to writing, maintaning and updating unit tests. Good luck with that project.

Focus On Integration Tests

Integration tests are far more effective. Why? Because they focus on getting the parts to work together. They are also not fudged with data used to show they pass. Integration test will use test data loaded into a real database, and that will hopefully be a large enough set data so you can uncover error and edge conditions.

Think about what you're testing. You're testing that the components work together, you're testing that the server can talk to the database, you're testing the cluster, session management, external fail-over etc.

With Integration Tests you can test everything that is important to the system. Just like a Plumber doing his final checks before commissioning your central heating system.

Look at it this way: If a plumber does not Unit Test the parts he uses, no-one cares. If he did he would be the most expensive plumber on the planet. However, if the Plumber didn't Integration Test then he could end up in prison. Not Integration Testing a central heating system is a criminal offence that can result in 6 month in jail sentance.

So why does nobody care about Unit Testing outside of software engineering? Because it cost too much, and because everything that's important can be tested via Integration Tests or User Tests.

Take a look at the chart below. This is for the same size project as the chart above. The coding costs are the same (£100, 000), but the cost of testing and going live is about half. This is true of any project. If you don't believe me try it out and I'll pay the differance (conditions apply - such as you can't be a dickhead).



So, good luck, and focus on Integertion Testing and User Testing. Automate everything that you can. And remember the point of software engineering is not to develope code that is bug free, but to develope code that does not have bugs in all aticipated usages, but is easy to repair when a bug is found.

So the bottom line is, if you want to run a project with lots of Unit Tests, then double your development budget, and add zero to your quality.

Alos add 20% to your ongoing maintainance budget for fixing all those Unit tests as your code changes.

Add comment