Fry: If your programming told you to jump off a bridge, would you do it?
Bender: I dunno, I’d have to check my programming… yup.Futurama, Space Pilot 3000
I was testing an application where I needed to check that a popup window appeared on a certain occasion, and it closed down when the close button on the window was clicked. This is a very simple check that had been automated. One day, this test was chosen to be run overnight.
Unbeknown to me, there was a bug where the application would crash when the popup window was closed. For some reason, the test still passed.
Fortunately, the next step in the test could not be run because the software was no longer running. Normally, this is not a good thing as it is still good to know if the remaining steps in the test pass. On this occasion, it was a blessing as otherwise I may not have known about such a critical bug. It also made investigating the bug much easier.
My immediate assumption was that the failed step was where the issue was. I soon found that this was not the case. It didn’t take long to find the bug by rerunning the previous step. The step still passed but I could clearly see that the software was no longer running.
Once the bug was reported, I took a look at the automated test to determine why the test passed despite the application crashing. I soon found that while I included a check that the popup window was closed, I’d neglected to include a check that the software was still running. All the validation checks had been successful, so the test passed.
Automated tests will only do what its told, but it still has its uses
This highlights an important issue with test automation. It will only do what its programming tells it to do and not deviate in any way. If the test was being run manually, a human would have immediately noticed that the software was no longer running.
On the other hand, even though the test was passed incorrectly and stopped the remaining steps in the test from being run, it still provided the tester with enough information to find the bug. With the many other tests that need to be run, it might have been some time before this specific test was run. At least with test automation, the tester was made aware of a particular area within the software where the bug existed.
Automated testing allows scripted tests to be run quicker, freeing up more time for exploratory testing. It can also highlight areas of the software that may not be working as expected, enabling us to target exploratory testing to at-risk areas of the application.
Our aim is not to automate, but to enhance testing
An automated test will only run the test the way it is programmed to, but these tests can still be extremely useful. Michael Bolton wrote a brilliant article on the value and cost of Automated Checking in which he states that:
“Automation is NOT the goal. Tools and automation are a means of advancing some aspects of your test strategy.”Michael Bolton, Value and Cost in Automated Checking or “Don’t Fall into GeMPuB”“
Like software, test automation is not perfect and will contain bugs. We must not be completely reliant on it. Test automation should only be a part of the overall test strategy. I don’t think I’ve ever found a bug using test automation alone. At best, it highlights an area within the software that may not be working as expected. Additional exploratory testing had to be carried out to find out if there was an bug or not, and the precise details of the bug.
What is a good testing strategy?
For me, a good test strategy contains a mixture of scripted automated and manual tests, and some exploratory testing.
There should be scripted tests that cover the checks that MUST take place. These are the ones that cover the core features within the software. It is up to the tester to decide if these should be automated or not. There are a lot of benefits to automation, but the value must exceed the cost.
There MUST be exploratory testing. The tester can examine the software without being limited to what the script says. The freedom to explore the software may allow the tester to find hidden issues which could negatively effect the overall user experience. It is also useful to do more rigorous testing on areas of the software that have been changed. Perform extra checks so that we can be certain that the software works.
Automated testing is useful, but it can only enhance the testing, not replace it.