Don'ts in Software Testing

Sounds like a book title, doesn’t it? No, I haven’t written a book on software testing. I’m on my way though, I just need fifteen or twenty years of Alan Page’s, Ken Johnston’s and BJ Rollisson’s experience and I’m ready to go with a new testing bible like the just announced “How We Test Software at Microsoft®”. Check out Alan’s and BJ’s posts on the subject here and here.

I’m sure the authors, who lead the Microsoft testing division of about 8000 testers, have a lot to share, but I was also happy to see the post by Federico Silva Armas from the ASP.NET QA team about the “Evolution of the ASP.NET Test Process (part 1)”. I’m not happy because the QAs from the ASP.NET team experienced so many problems in the past several years, rather because Federico points to some important aspects of the QA process in general. While I’m eager to see his next post on the learnt lessons, I also found myself thinking of my own list of “don’ts” in software testing I had come to though the years. Please note it doesn’t come from what the various methodologies around proclaim -- the only one I actually blindly believe is called “practice”. So here we go…

1. Don’t leave all the testing to the QA department!

Yeah, of course this means writing unit tests and doing it as early as possible. We all know it’s much cheaper to fix the early bugs found.

Let me jump immediately to the second one:

2. Don’t leave the testing to the end!

Really, start testing when you have something to test! This includes involving testers even in design and early development – after all, the better we (the testers) understand the product, the better it will be tested! Ideally, devs and testers can discuss how the entire functionality can be tested – what can be left to the unit tests, what should be automated via functionality tests as well as prioritize the regression, load, performance, stress, etc.

3. Don’t try to automate everything!

I like how Federico call it – “automation frenzy”! Surely the automation almost always helps in regression. However, it has its own cost as well, so instead of throwing yourself in test automation immediately, first analyze the test value. Is this a simple and common case you’d like to cover? If so, consider integrating it into the unit tests suite. Or if it is a too complex test case, don’t you think of the test creation cost and future maintainability?!

To help the previous, here comes the fourth one:

4. Don’t forget to test manually!

Don’t forget your automated tests would most probably catch the bugs they are supposed to catch!

After all, the human brain can’t be substituted, or at least that wouldn’t be cheap.

You have some ideas on testing a specific feature more, but there are other coming tasks? Write it down in your ToDo list and come back on that in a week or two. Or probably you are out of ideas about additional testing of this feature? Take the developer who has worked on that for a couple of minutes, discuss how the feature has been tested so far, look at the related unit tests together and go to the next one if you both agree this is ready to go.

5. Don’t think the product is completely tested!

And you might be wrong! I still sometimes hear devs around crying out “Jesus, this has never been working!” for functionality of a product, being on the market for years! Sooner or later customers will report all the bugs. What about you?

Did I mention the customers?

6. Don’t get satisfied with the number of the passed tests!

Furthermore, regularly get back to think of improving what you’ve done till now.

Why? Because the customer comes and says “I hit this and that issue while working with your product. I can’t or it’s hard to proceed if you don’t address those. That doesn’t seem to be well tested!” Would you answer “Hey, sorry to hear about that but we have 500 tests passing so the product has relatively high quality!”? That won’t help at all and instead it can easily frustrate her majesty, the customer!

Okay, I think those are enough for now as I’d love to see you adding your own points. Don’t hesitate to disagree on something I’ve mentioned above as well.

Finally, while writing the post I was happy to find several excellent articles, discussing similar thoughts. Please note the authors are highly experienced professionals as well as great writers and speakers so those worth reading!

“Testing and Quality” by Alan Page:

http://blogs.msdn.com/alanpa/archive/2008/04/10/testing-and-quality.aspx

“Manual v. Automated Testing Again” by James Whittaker:

http://blogs.msdn.com/james_whittaker/archive/2008/10/29/manual-v-automated-testing-again.aspx

“Automate this!” by Michael Hunter:

http://blogs.msdn.com/micahel/archive/2007/04/11/AutomateThis.aspx

“Manual tests cannot be automated” by James Bach:

http://www.satisfice.com/blog/archives/58