Day 3 – Share how you evaluate a new tool

There are 3 things that I would consider when evaluating the tool.

  1. Activity/Task
  2. Improvement
  3. Cost

Activity/Task

For a tool to be useful to us, the team or the business, it needs to be able to help someone complete a specific task in a quicker or more efficient way. The first step in evaluating the tool is to consider the task that its helping us complete, why we need to complete it, and the potential repercussions of not being able to complete the task in the required timeframe and to the desired standard.

In the case of testing, we want to check that all areas of the application still work as expected after a change has been made. We need to do this so that we don’t unknowingly introduce a bug that makes the application harder to use. Not completing testing within a timeframe will delay releasing new features or bug fixes. Not completing testing with the desired level of coverage increases the risk of new bugs being introduced into the application.

Improvement

It is possible that the task is impossible to complete without the tool. It is also possible that the tool only makes the activity easier to complete. An essential part of the evaluation needs to include the actually improvement it makes to the completion of the task or activity.

The questions I’d ask when evaluating a tool is:
– How long does it take for me to complete the task with and without the tool?
– What improvements are gained from completing the task with and without the tool?
– What are the benefits of completing the task with the tool, compared to completing the task without the tool?
– Is there any specific problem that the tool solves?

It is important to compare the outcome both with and without the tool to see if there is any actual improvement. If the task can be completed in the same amount of time, with the same efficiency and to the same standard without the tool, then it isn’t likely to be a particularly useful one.

Cost

Even if there is obvious improvement to the way to the task or activity is performed using the tool, we need to consider if this is enough to justify the cost. It isn’t always up to us. There is probably another colleague who is in charge of the budget.

If the task is something that cannot be completed easily without the tool, and its a task essential to our jobs, then justifying the cost should be easy. However, in cases where the tool is making our jobs easier, then more evidence may be required. Providing quantifiable evidence of potential time saved or improved quality can help demonstrate that the tool will be a useful addition to our testing toolkit.

Questions from the challenge

How do you discover all the things you like/dislike about a new tool?

The main thing I like to look at is how the tool is useful to ME. Not everyone will find the tool useful or easy to use and will have varying opinions regarding its usefulness. A lot of times, we’re told what tools to use but our line managers should still be willing to listen to feedback. If an individual doesn’t personally find the tool useful, and believe they could do the same job better without the tool or with a different tool, then they shouldn’t be forced to use it.

What do you rely on to make a reasonable evaluation of a new tool?

Activity/Task – What specific activity or task does this tool help me complete?

Improvement – What are the actual benefits of using the tool? This should be a quantifiable value like time saved. We’d also want to assess if the task could be completed without the tool.

Cost – How much does the tool cost? Is there a cheaper tool that could produce the similar outcome?

How do you know you’ve found “the one”?

Many are not given the luxury of choosing the tools they use in their jobs. It might be that the tool selection is decided by someone else, in which case all they can do is make recommendations. Often people start a new job to find that all decisions regarding tools have already been made and the tools paid for, so are forced to make do with what they are given.

If able to choose which tools to use, I would simply consider the task I am trying to complete and how that tool helps me complete the task.

What is the biggest turn-off when evaluating a new tool?

Poor UX!

Even if the tool is fit for purpose, when the user experience is so poor then any benefit to using the tool is not likely to be worth the cost.

Further Reading