RiskStorming is a collaborative activity that gets the team analysing and discussing the risks that might affect the quality of a product. The team then takes these risks and identify ways to mitigate these risks.

With the arrival of the new TestSphere expansion pack, I decided to run by own RiskStorming session on the new COVID-19 tracker app. I will be picking 6 quality aspects, identifying potential risks associated with these aspects and then analyzing potential methods for mitigating these risks.

The COVID-19 tracker app is designed to detect and record nearby smart phone users who have installed the app. If one of these users reports that they have experienced symptoms of COVID-19, then all other users who have been close by at some point will be alerted and advised to self isolate. Many countries are releasing similar apps. I am looking at the UK version which is currently being tested on the Isle of Wight.

The 6 quality aspects I identified for this exercise were:

  • Changeability
  • Data
  • Confidentiality
  • Adaptability
  • Impartiality
  • Accessibility

Changeability

While rolling out this new app to the Isle of Wight, the development team are likely to identify a few issues that need to be fixed to improve the quality of the app.

Risks

  • An update results in the loss of previously stored data.
  • Users being unable or unwilling to update the app and the app fails to record devices that have already been updated.
  • An update failure causes the app to stop recording nearby devices

Event Logging (Observability)

Event logging records each state change in the application, including the history of each data point. This would allow the development team to revert back to earlier states to help with debugging if any issues are reported.

Circuit Breakers (Dealing with Change)

In electronics, a circuit breaker is tripped if a fault is detected, protecting the circuit from further damage. When the fault has been detected and fixed, the circuit breaker is reset and normal operations resume. A similar concept can be applied to software development. If a new or updated feature stops working correctly, the circuit breaker is tripped and the original feature is run instead.

Should a change to the app cause it to stop working correctly, a circuit breaker could be used to allow the original version to be used instead so the app continues to log essential data.

Data

When two people who are carrying devices with the tracker app installed are near to each other, both apps need to record that these devices were near to each other. It also needs to record anyone else who happened to be nearby at the same time. A lot of data is being sent to and from the device at the same time.

Risks

  • The app fails to send out a signal, so nearby devices fail to log that it was nearby
  • The app fails to record that its detected another nearby device.
  • The app fails to record every nearby device when in a crowded location
  • The app logs incorrect or invalid information.

Stress Testing (Techniques)

Stress Testing could be used to ensure that the app is able to accurately log large amounts of data being sent to it at once. Checking that the app continues to send and receive data when running for long periods of time would also be beneficial.

Comparable Products (Heuristics)

Several countries in the world are also developing similar COVID-19 tracker apps. Analyzing some of the alternative designs, and learning about some issues that they have faced, could help develop a better product that can handle large amounts of data much more efficiently.

Confidentiality

Confidentiality is a commonly raised concern when it comes to any application that tracks someones movements. Some might be willing to overlook this for the sake of controlling the virus, but this doesn’t mean it should be neglected. The data logged using this app should only be used for purposes of monitoring and controlling the spread of the COVID-19 virus.

Risks

  • Details of a vulnerable user, like a domestic violence victim, being made available
  • The name and address of someone with COVID-19 becomes available to the local population.

Standards and regulations

When it comes to privacy and confidentiality, there are already laws in place designed to protect the public. Making sure that the app adheres to these laws is one of the best ways to ensure that vulnerable users are protected.

Adaptability, Impartiality and Accessibility

I’ve grouped these three together because I identified the same card for mitigating risks to these quality aspects.

Adaptability Risks

With so many types of smart phones available on the market, making sure the app works with every single one is going to be a challenge. We can’t just ignore a device because its only being used by 0.5% of the population. If someone in that group reports having COVID-19 symptoms, then many users will be unaware that might be at risk of having the virus.

  • The app fails to record signals from a device that is a different make or model

Impartiality Risks

This app is being tested on the Isle of Wight, an island off the south coast of England. I understand why they chose the Isle of Wight. It is one of the largest and most populated islands on the British Isles (excluding the mainland of course). There is only one way off the island and that is via the ferry, meaning there will be fewer people travelling to and from the area.

Lets compare this area with London (a major city which has seen a high number of COVID-19 cases) and Gwynedd (a more rural county in North Wales).

AreaPopulation DensityConfirmed COVID-19 Cases
Isle Of Wight372/km2155
London5,671/km225,734
Gwynedd49/km2277
The population density information was taken from Wikipedia, and the number of confirmed COVID-19 cases were taken from the BBC website and Gov.co.uk on the 10th May 2020.

The lifestyles of those living on the Isle of Wight is going to be different to other areas of the UK.

  • The app fails to record every device in a densely populated area
  • The app fails to record devices when there is no phone signal such as a rural area or on the London Underground.
  • The app can only be used by those who can read English (the main language in Gwynedd is Welsh)

Accessibility Risks

Currently, the application is only accessible to those with a smart phone. This is a good plan as it is a popular tool which most people already own. Furthermore, the financial barrier that might dissuade people from downloading the app is removed because the app is free.

However, this method relies heavily on the assumption that everyone owns a smartphone.

  • Someone without a smartphone remains unaware that they have been in contact with someone with COVID-19 symptoms.
  • Someone with poor computer literacy skills is unable to download the app or report that they have COVID-19 symptoms.

Personas (Techniques)

Creating personas can allow the team to test and analyse from the perspective of different user groups. These personas could be based on geographical locations, population densities in different locations, languages, accessibility, lifestyle, health, jobs, accessibility, and different devices.

For example, a 32 year old living in London with a Samsung Galaxy S9 who travels to work on the Jubilee line of the London Underground, and occasionally goes jogging in Regents Park.

Testing would need to ensure that the app is able to send and receive data when installed on a Samsung Galaxy S9 while in a crowded location on the London Underground, and also while out jogging.

Personas could also be used to identify groups who may not have access to the app and develop alternative solutions to tracking who they have been in contact with.

Bonus – Business Metrics (Observability)

This card really caught my eye when completing this activity. It really got me thinking about the business metrics that could be used to ensure that the tracker app is producing the desired results. The key stakeholders aren’t just those who work for the business (which in this case would be the government). The entire population of the UK need this app to be a success. There is going to be a lot of interest in the business metrics before, during and after the app is fully launched.

Key metrics could include:

  • Percentage of local population who’ve downloaded the app
  • Change in the number of COVID-19 cases nationally and locally
  • Number of people who have reported having COVID-19 symptoms using the app
  • Number of people using the app advised to self isolate
  • Number of people advised to self isolate who have reported having COVID-19

Final Thoughts…

The idea of RiskStorming is to identify potential risks and generate ideas to mitigate these risks. I completed this task alone, however this activity tends to be much more successful when the entire team get involved. This includes testers, developers, business analysts and anyone else involved in developing, running and maintaining the application. Relying on the mindset of just one tester is not going to be enough to fully mitigate all risks.

Further Reading

Reinvent Your Test Strategy By RiskStorming With TestSphere
A guide for running a RiskStorming session

Ministry of Testing Awesome Merch Store
You can purchase your own pack of TestSphere cards here, including the expansion pack.

Jan #MidTest – RiskStorming with Ben Fellows
Details of a RiskStorming workshop I attended at the Midlands Testers meetup in January.