The Current State & Future Trends of AI in Software Testing

As you are likely aware, artificial intelligence (AI) and machine learning (ML) are transforming multiple sectors of the economy, and impacting several aspects of our daily lives.

Workplaces such as those in finance, healthcare, retail, education, and technology are leveraging AI to automate tasks, reduce costs, and make data-driven decisions. In our homes, AI is powering television and movie recommendations, personal digital assistants, security cameras, and home automation.

In this blog, I will discuss what I wish I knew about AI in software testing with a focus on how the coming together of artificial intelligence (AI) and software testing have already made a huge impact on test automation. I look forward to discussing this developing trend with you in more depth. 

What I Wish I Knew About AI in Software Testing

AI software testing uses automated software testing tools to leverage AI (machine learning) to produce better results. By incorporating AI into automated testing tools, teams will be able to overcome many of the traditional challenges of automated software testing.

As AI continues to permeate our world, it is becoming clear to me that validating the functionality, safety, security, performance, and resiliency of AI software systems is more critical than ever. In other words, AI needs testing.

Related Viewing: The Future of Testing: A Conversation About the Use of AI and ML

Researchers and practitioners are recognizing the potential for AI and ML to bridge the gap between human and machine-driven testing capabilities. As a result, several organizations are developing AI-powered automated testing tools. But there is still work to be done. 

Since 2014, there has been a spike in the number of vendors offering AI-driven test automation services. The majority of these tool vendors are start-up companies targeting system-level testing of mobile applications, and the subject is generating some much-needed buzz in the industry.

Hence, it seems that just like AI needs testing, testing needs AI.

Although some aspects of AI for Software Testing (AIST) are still not receiving enough attention, in the last decade we have seen it emerge as a discipline centered at the intersection of three areas:

• AI-driven automation testing — developing AI tools to test software.
• Testing AI systems — devising methods to test AI systems.
• Self-testing systems — designing software that is capable of self-testing and self-healing.

The intersection of AI and software testing

How I View the Current State of AI in Software Testing

The current state of the practice uses autonomous and intelligent agents, referred to as “test bots,” to automate activities such as application discovery, modeling, test generation, and failure detection. A combination of machine learning techniques are used to implement test bots. These include but are not limited to decision tree learning, neural networks, and reinforcement learning.

Machine learning allows the test bots to be robust and act under conditions of uncertainty unlike traditional test automation tools and frameworks. Examples of AI-driven automation testing approaches that have formed over the last decade include:

• Differential testing — comparing application versions overbuilds, classifying the differences, and learning from feedback on the classification.
• Visual testing — leveraging image-based learning and screen comparisons to test the look and feel of an application.
• Declarative testing — specifying the intent of a test in a natural or domain-specific language, and having the system figure out how to carry out the test.
• Self-healing automation — auto-correcting element selection in tests when the UI changes.

My Plug: Perfecto's AI-Powered Root Cause Analysis

Now is a great time for me to point out the latest developments with Perfecto's AI and ML capabilities. With Perfecto’s AI-powered Root Cause Analysis, teams can quickly pinpoint the true causes of test failures, significantly reducing time and effort spent on troubleshooting. For a deeper dive into available root cause analysis tools and techniques, explore this guide that outlines methods to diagnose and address failure patterns more effectively.

Harness the power of Perfecto’s AI-Powered Root Cause Analysis, which includes the following key features:

• Smarter Failure Reasons: AI identifies real failure reasons and groups similar causes for easier resolution.
• “Failed vs. Last Good” Execution Comparisons: Automatically highlights differences between failed and successful executions.
• Backend Root Cause Identification: Analyzes API call traces to find backend-related test failures.
• Test Failure Impact Analysis: Prioritizes the most critical failures and suggests corrective steps.

Related Reading: The Future of Testing Innovation With Perfecto

Let's Take a Closer Look At AI in Software Testing & the Open Source Community

Element location using AI has been incorporated into the open-source mobile testing framework Appium. The project, which is a collaboration between Jason Arbon, CEO of Test.ai, and Appium’s creator Jonathan Lipps, is worth taking a closer look at. It uses AI to identify web elements in mobile applications.

Leveraging AI for web element location makes the tests quicker to create and less brittle. Furthermore, the testing community can improve the AI by adding new training data, alternative training methods, more rigorous relevance testing, or new labels.

Related Reading: How to Use Page Object Model in Selenium

The training data is open and available on Kaggle, thereby allowing individuals or organizations to fork the data, clean it up, add it to their proprietary test frameworks, or compete to improve the classifiers.

Related Reading: Test Automation Strategy For Beginners

Tariq King and his team over at Ultimate Software developed and released an open-source framework called AGENT — AI Generation and Exploration iN Test.

AGENT is an ML-based, multi-agent prototype for web application testing. The prototype consists of a collection of intelligent agents that work in conjunction with abstract test generation and web UI classification subsystems. The agents incorporate long short-term memory units trained by human testers.

For any given subsequence of steps through the application, the agents predict sequences of the next possible set of actions and assertions that a human tester might make and automatically execute them on the system under test.

AI in Software Testing: How I See The Future of Test Automation

One of the trends that started this past decade, which is expected to continue, is the use of AI to enhance existing tools and frameworks that target specific testing problems.

Examples at this time include functional testing of web and mobile applications, visual testing of user interfaces, and UI element location and auto-correcting element selectors. For enterprise systems, especially in complex environments like desktop testing sap virtual setups, AI-driven automation can significantly reduce maintenance and increase accuracy. Beyond this we should expect to see AI starting to replace entire technology stacks for automated testing.

Related Reading: ML & AI in DevOps: Experts Predict 7 Big Changes

At all testing levels, AI automation testing will take over tasks that require decisions that a human could make in less than a second. Initially, higher-order testing tasks may still require human input or intervention. These are the tasks that require a bit more thought such as test generation, usability testing, security testing, and edge cases.

However, over time as technology progresses, and as the machines are trained on the actions of these higher-order tasks, I think that AI is likely to take over those activities as well and tackle problems that require deeper context. The image below depicts this progression of the AI testing singularity as we move into the next decade.

The AI testing singularity: a projection of the next decade of software testing

How Can We Prepare for Testing AI-Based Apps?

So now that I've covered the current and future state of AI in software testing, you may be asking yourself: what can testers do to prepare for a future in which AI tests software? In my opinion, the initial step is to determine whether you are interested in designing and developing AI-driven systems, or being an end user of these tools.

Related Reading: 5 Types of ML & AI Testing Tools

An engineer who is building these types of systems needs to have a good understanding of AI and machine learning. Both MIT and Stanford University have freely available introductory courses on machine learning.

Tariq King and Jason Arbon have open-sourced the materials for an introductory course on AI and ML skills for the testing world, including the slides and tools used during the exercises. It is unlikely that end users of AI-driven testing tools will need to have deep knowledge of machine learning.

Vendors are likely to provide easy-to-use interfaces and APIs for using AI-driven testing features and customizing pre-trained AI models.

This trend is already being followed with the advent of AItools and platforms like Google’s AutoML Vision, H2O.ai, and DataRobot.

Bottom Line

AI is already altering the landscape of testing and continues to accelerate. While I cannot know exactly what the future of software testing holds, I know that we can prepare for it by stabilizing and scaling test automation to mature our processes. In my opinion, Perfecto is the ideal partner for all of your AI and ML needs. Perfecto can help you take your test automation to the next phase, no matter where you are at. 

With Perfecto, you'll experience:

• AI-powered testing and reporting capabilities.
• AI-powered root cause analysis.
• Unified testing for web and mobile apps.
• Advanced automation.
• Full toolchain integration.
• All results in one place.
• Shift left testing.
• And more.

Whether you’re just beginning on automation or you’ve reached continuous testing, I recommend giving Perfecto a try.

See Perfecto in Action