Gee, the Software is Working on My Computer

Crappy software that crashes, is hard to use or does not work as you expect can be quite a nightmare. The frustration is partly from the expectation that software should help you work faster and improve your productivity. It can be quite a letdown when it doesn’t.

And it can be embarrassing when you are the person responsible for developing the software that turns out to have poor quality. 

Do any of these scenarios sound familiar?

• You specify the user interface of your software but the developers create their own that is completely different than the rest of the program and is difficult to use.
• You get a software release with new functionality but things that used to work are broken.
• The software works fine on the developer’s computer (workstation, webserver, etc.) but not yours.

At one of my software startups we worked for weeks to create the first internal release of our software. It was a web application for an online marketplace. Late on a Wednesday night we fixed the “last” bug and let the CEO try it out. The app immediately crashed.

“Gee,” I said, “It’s working on my computer. . . Oh. No. It’s not.”

We had synchronization problems common in multi-threaded applications that access the same information for multiple users. We made the mistake of letting the CEO be the first person to simultaneously use the web application. And adding a yet another person made it crash even faster.

Of course, the real mistake was not designing the application right to begin with. Our passion for the functionality of the application exceeded our ability to create robust software.

What you need is a passion for both.

Without passion for both quality and functionality you will lose customers who get a bad first impression of your software. In a business setting the cost of buggy software is lost productivity and extra work by the users.

At my startup, it took us a year to find and hire the members of our development team with the experience we needed to take responsibility for both the functionality and quality.  Eventually we were successful and the company was sold for $75M three years later. We were lucky.

Today it is tempting to try and duplicate what we did with offshore programmers and testers instead. It’s not really outsourcing – it’s staff augmentation at a distance. And it usually doesn’t work; especially if you do the development in-house and try to outsource what you think is QA independently with cheap testers elsewhere. 

Quality assurance needs to be integrated software development. 

There are two different approaches, or procedures, needed for testing software. First is the more traditional test of the full system, sometimes called Integration Testing. Second is a Test Driven Development (TDD) approach popular these days in Agile and Scrum methodologies.

The fundamental difference between the two approaches is in how the software code is tested.  During Integration Testing, developers review and test the code.  Then, for critical artifacts, a peer review is completed.  Afterwards, testers create test scripts and deploy an array of integration and system tests (such as scalability testing, load testing, stress testing, and very importantly, security testing –i.e. checking of code vulnerability, SQL injections, malicious code, etc.).
 
Under TDD, the order in which testing appears in the cycle is reversed.  Each developer first writes “unit tests” and then writes the code.  This is done iteratively in such a way that every discrete section of code is automatically tested.  Furthermore, when any new code is sent to the integration server, the tests go with it. The server itself runs all the unit tests for all the code that developers have deposited in the server, automatically.

On top of this, testers dedicate themselves to designing system and integration tests that are also run automatically by the integration server.  Finally testers will perform manual tests that are not possible to automate.  In this manner, the integrity and quality of the code is constantly guaranteed.
 
Academic studies and field experience suggest that TDD increases development time by approximately 15%.  However, experience also shows that the Total Cost of Ownership of software developed under TDD is lower than for software tested using Integration Testing only.
 
It is worth noting that by using the above QA methodologies (TDD and Integration Testing), one Accelerance partner* is able to deploy software of world-class quality that exhibits less than 0.3 errors per 1,000 lines of code during 18 months of use.

If you are outsourcing QA then make sure it is integrated with your software development process. If you are outsourcing all of your software development then make sure testing and quality assurance is part of the process. With the right vendor your results can be stunning – much better, faster and cheaper than what you can do on your own.

As you can see, testing requires a systematic approach performed by professionals. Just having a few extra people on the project to click the mouse and pound the keyboard is not enough. A good QA engineer is as valuable as a good software developer.