I’ve always enjoyed the Ministry of Testing’s 30 Days events. It has been a fun way to explore a topic and learn something new. I’ve often struggled to complete all 30 tasks, but the tasks I have completed have always helped change my mindset around a topic. The AI theme has particularly piqued my interest as it is a topic I know very little about (at least from a testing standpoint), so I am really excited to explore this topic in more detail.

In this blog post, I am completing the following tasks from the 30 Days of AI Testing event:

  • Day 1: Introduce yourself and your interest in AI
  • Day 2: Read an introductory article on AI in testing and share it
  • Day 3: List ways in which AI is used in testing

Here is a link to the Ministry of Testing event page where you can find out more about the event:
30 Days of AI in Testing event page.

Why I Want To Learn More About AI In Testing (Day 1)

AI will always be a fascinating topic, from books and films with speculation on the support and dangers AI can bring to society, to practical real world applications where AI has helped enhance an application. At the moment I have no idea on how AI can help with testing, so I am taking part in this event to learn more about AI from a testing point of view and hopefully learn something new which can be applied to my daily testing activities.

Interesting Articles on AI in Testing (Day 2)

For Day 2 of the 30 Days of AI in Testing event, I decided to cheat a little. I asked ChatGPT to generate an article for me. You can read the full article here, which I feel gives a good summary on the applications, benefits and potential challenges of AI in Testing.
Read An Introductory Article on AI in Testing as generated by ChatGPT

In the article, the following key applications are highlighted as being useful for AI in testing:

  • Test Case Generation
  • Test Execution
  • Defect Detection
  • Test Result Analysis

Several useful articles were posted by other people on the Ministry of Testing forum. I’ve included a few at the end of this blog post in the useful resources section.

How AI can be used in Testing (Day 3)

This is definitely a part of the 30 days event which I am most excited about as it involves exploring how AI can be used in testing. This will be useful for identifying potential ways in which my own testing activities could be improved.

For this task I took the 4 applications identified in Day 2 and analysed current tools available that could support this activity. In my current job, I mostly develop automated tests – so this analysis is going to be more focussed on the uses of AI with regards to automation.

Please note that I’ve not yet tried out all of the applications listed. This is purely for reference, so they can be investigated in the future to determine if its a tool that would be able to support future test automation efforts.

Test Case Generation

Developing automated tests can be a slow task. An application that could generate automated tests for you would definitely be useful. There are already features available in many automation frameworks that allow automatic code generation (examples include Record and Playback, and low/no code applications). However, these still require a human to make the decisions about which tests to develop. If that person develops the wrong tests, then the framework is not meeting its full potential.

Some tools specialize in generating unit tests by analysing code structure and execution paths. Examples include Diffblue Cover or Randoop.

Others can generate test cases by analysing user interactions and UI elements. Examples of this include Test.AI.

Another useful feature that I came across was a tool that generated test ideas rather than the test cases themselves. An example of this includes TestCraft. This would be particularly useful as it encourages the tester to be cautious and ensure the tests are actually useful and correct. Once they’ve reviewed the test ideas, there is an option to convert to automated tests. As a bonus, TestCraft can also be used to identify potential accessibility issues and make test suggestions based around accessibility.

Currently, I am skeptical of AI tools that claim to generate tests for us. As good as it is that an AI tool could write tests us, a human programmer would also need to review what is written to ensure what is being written is correct, useful and relevant. Even when AI-tools have evolved enough to generate tests for us, it is essential that we continue to review the generated code before fully trusting them to provide adequate test coverage of the application.

Test Execution

There are tools that can be added to the CI/CD pipeline to run automated tests whenever there’s a code change.

One tool, Tricentis Tosca, offers ‘risk-based test optimization’ which prioritizes tests based on risk and business-critical functionality.

Automated testing was created to tackle the challenges of manually testing a whole software application. We had to concentrate manual testing efforts on high-risk areas to ensure the most important features were tested. As more automated tests are developed with the help of AI, there may come a time where we don’t have time to run all the tests.

Defect Detection

Static analysis tools like SonarQube have been in use for a substantial period of time. They serve the essential purpose of ensuring that the code adheres to coding standards and does not compromise code quality.

Applitools has gained significant popularity for its application in visual regression testing. Leveraging AI technology, it compares visual components within a UI against screenshots from previous iterations of the same UI.

In essence, the timely identification of defects significantly reduces the complexity and cost associated with their resolution. These tools prove instrumental in detecting defects early on, a task that may pose challenges even for the most vigilant human software tester.

Test Result Analysis

This is definitely an application of AI powered testing that I want to see the most. Improved test result analysis could help:

  • Identify trends, patterns and anomalies in tests
  • Identify potential causes of flakiness in automated tests.
  • Identify ways to speed up automated test runs

Tools that offer support for test result analysis include Applitools and Tricentis QTest

Conclusion

After just three days, I’ve made significant progress in understanding the potential benefits of AI-driven testing tools for our testing efforts. I’ve also pinpointed some tools that can assist me in exploring how AI could enhance our testing activities. This marks a significant departure from previous years, during which we could only speculate about the potential of AI and machine learning.

While I remain cautious about the true benefits of some of the tools I’ve encountered thus far, I’m confident that further research will unveil new opportunities to enhance our software testing activities.

Useful Resources

Introductory articles on AI in Testing

AI in Software Testing – How can AI be used in software testing?
This article by Testsignma Inc. provides a more extensive list (compared to the ChatGPT article) of how AI can be used to support software testing.

What is AI Testing? A Complete Guide
This article by katalon provides an explanation of what AI is, specifically what AI testing is, focusses on the benefits more and provides a list of AI-powered tools that can support AI in Testing. Crucially, it also provides a section on using ChatGPT for software testing.

Introduction to Artificial Intelligence with LinkedInLearning
Not followed the course yet, but this will provide a good outline to what AI is. It focusses on AI in general, rather than AI in Testing so it does delve into AI uses that may not be relevant. However, knowledge of AI patterns, neural networks and machine learning algorithms can help provide you with an understanding of the terminology that AI applications use when explaining how it works.

AI tools listed in day 3

Diffblue Cover
An AI tool that specializes in generating unit tests by analysing code structure and execution paths

Randoop
Another AI tool that specializes in generating unit tests

Test.AI
A tool that analyses UI elements and user interactions to generate test cases.

TestCraft
A tool that generates test ideas which generates test suggestions, including ones that focus on accessibility testing.

Tricentis Tosca
A tool that offers risk-based optimization to speed up test runs by prioritizing the most essential tests based on business risk.

Tricentis qTest
Another tool supported by Tricentis which includes offering insights into analytics and reporting.

Applitools
An AI powered application that can support early defect detection through visual regression testing and test result analysis

Links to pages in the Ministry of Testing forum, where poeple have discussed completing each task

Day 1: Introduce yourself and your interest in AI
Day 2: Read an introductory article on AI in testing and share it
Day 3: List ways in which AI is used in testing