This is the final blog posts in which I talk about what I learnt while attending Emily Bache’s workshop ‘Getting High Coverage Regression Tests Quickly’. This was a half day workshop that took place at Test Bash Manchester on 2nd October 2019.
In my previous blog posts, we adapted a test so that assert statements were replaced with approval tests. This test was updated further so that we could increase the test coverage. In this blog post, we will use combination approvals to increase the test coverage even faster.
This exercise was completed using C# in Visual Studio with the ‘dotCover’ and ‘Resharper’ plugins.
Legacy code from Hell
The application we are testing is called Gilded Rose. It is an old, over-complicated and poorly designed application with several nested if-statements. Here is a section of the function ‘UpdateQuality’ which includes several of these it-statements.
Getting high coverage of code like this is a challenge. To get the required level of coverage, we would need to develop a large number of individual unit tests.
The use of combination approvals allows the same test to be run with a combination of different inputs. For this test, we wrote the function ‘DoUpdateQuality’ which sets the name, sellin and quality for an item and then updates the item by running ‘UpdateQuality’.
The test is then written which includes the combination approval. Three lists are included:
Name => “+5 Dexterity Vest”
SellIn => 0, 5, 10
Quality => -1, 0
When the test is run, ‘DoUpdateQuality’ is run using 1 item from each list. With combination approvals, several iterations of ‘DoUpdateQuality’ is run – 1 for each possible combination.
This is the approval snapshot used during the test. Each line is a separate iteration of the test. It includes the input and the expected output.
Getting High Coverage Regression Tests Quickly using Combination Approvals
By increasing the number of possible combinations, we can increase the overall code coverage very quickly.
So far, the current test has just 53% code coverage.
To increase the coverage, I first looked at the which item name values were not yet covered in the approval tests:
I took note of each possible name values that are not yet covered in the test.
Missing names were:
- “Sulfuras, Hand of Ragnaros”
- “Aged Brie”
- “Backstage passes to a TAFKAL80ETC concert”
The item name ‘Sulfuras, Hand of Ragnaros’ is also included in the if-statements. However, this is not included in the tests because it is only used when checking that the name is not equal to ‘Sulguras, Hand of Ragnaros’. The total number of combinations can drastically increase for every new item added to a list. This slows down the time required for the tests to run. It is best to avoid redundant iterations where possible.
Including these additional names increased the number of iterations. The new iterations, which have not been added to the approval snapshot file, are marked in red on the new output file generated when the test is run.
These additional tests increased the code coverage increasing from 53% to 86%.
Issues with Code Coverage
The point of this exercise was to increase code coverage. However, some assumptions were made while completing this exercise. Notably, that this code works as expected.
Achieving a high rate of code coverage is advisable. However, this should not be the sole method of assessing the quality of the application. In real life situations, 100% code coverage may be unfeasible and unnecessary. It can also provide a false sense of security. Just because a statement is covered in the test, doesn’t mean that this section of code is working correctly or even interacting with the rest of the code correctly.
Most automated tests will not find all the possible errors. It will only tell you if something has changed in the code that affects the outcome of the test. It is up to you to verify if the application is working as the business expects.
100% Statement Coverage Achieved
I managed to achieve 100% code coverage using the following combinations. The values not included in the original test are in bold italics.
Name => “+5 Dexterity Vest”, “Backstage passes to a TAFKAL80ETC concert“, “Aged Brie“
SellIn => -1, 0, 5
Quality => -1, 0, 2
Note: ’10’ was removed from SellIn as it covered some lines of code that were also covered when SellIn was ‘5’