Unit Tests Don't Find Bugs: the Death of QA
Unit tests don't find bugs. They find regressions. This is a painful lesson I learned when I first started doing TDD (test-driven development), and it's well known among most TDD circles.
TDD's goal is to prevent programmers from introducing new bugs into working code. However, when you're writing code from scratch, your tests won't help you find all the bugs in your code. That's because you can't possibly write tests for all the ways your software will be used (or abused). When I first started doing TDD, I had really good tests, but I was too tired to do much exploratory QA. However, my boss wasn't, and I was very embarrassed to find that my software had lots of bugs. Simply put, he used my software in ways that I hadn't intended.
I've seen a lot of companies that don't bother writing any tests or doing any QA. They just let their users find all the bugs. Needless to say, I've never had respect for those companies.
However, it's growing more and more popular to destaff the QA department and just require engineers to write lots and lots of tests. Often, these are in the form of unit tests. Even though integration tests can conceivably catch more bugs, they take much longer to run. Hence, even integration tests are often deprioritized.
What I'm discovering is that a lot of projects have both lots of unit tests and lots of bugs. These are bugs that could have been found manually by a QA engineer, but it seems that manual QA testing (i.e. exploratory testing) has gone out of vogue.
I used to think that code that was well-documented, well-styled, well-tested, and code reviewed would rarely have bugs. Sadly, I no longer believe that to be the case. I think we need to go back to the days when we had decently-sized QA departments, perhaps in addition to all the other things we do.
To tweak what Knuth said, "Beware of the above code. I have only tested that it works. I haven't actually tried it."
Comments
- Mykola
Bug of this sort are only bugs in the particular context of a user having produced such and unexpected use case. Testing, by devs or QA specialists, can only hope to mitigate the risk of these inevitable outcomes and protect you from the known knowns. Intelligent testing is the practical management of risk. As a QA professional, I feel well trained to provide that service, but I've met plenty of developers who do that well, and lots of QA staff who don't.
I also agree with the above post that QA staff left to rot in mindless repetition of mechanical tasks are probably less useful than well written unit and integration tests. Still, any QA manager worth the name will find the means to prevent this abuse of his staff's faculties, and more effective use of their talents.
As soon as the code base grows larger, regression will provide a better safety net for refactoring than no tests at all.
The lack of automated QA testing is frustrating and a waste of money. QA tackles a completely different class of bugs due to use-case based testing. And even it turns into regression at some point, it will at least improve the safety net.
Frisian