1

The rate at which AI tools keep coming up feels like they are going to replace Selenium and other conventional automation tools sooner than we think. But, as of now what are the real life practical limitations of such tools? Can somebody share their project experience?

EDIT:

This question is basically asking to address the limitations of AI backed "codeless" automation tools like the one described above when compared to Selenium (or other "codeful" automation tools).

  • Regarding you edit: You first asked for limitations of AI-based tools, but now of "codeless" tools. IMHO this is a quite different question and probably already answered here: sqa.stackexchange.com/q/465/21973 – beatngu13 Nov 25 '18 at 13:42
  • @beatngu13 The answers to that question doesn't address the AI backed tools (exactly what your answer to that question says). That's why while comparing codeless automation tools we need to give a separate attention to these new tools. – Shivam Mishra Nov 25 '18 at 13:58
  • I tried to address codeless vs. codeful (including AI-based tools) as an answer to the linked question: sqa.stackexchange.com/a/30657/21973. – beatngu13 Nov 25 '18 at 14:09
  • I did read your answer while going through the link and it doesn't really answers the question @beatngu. Using ML solves the maintenance problem of such tools. So, what are the other limitations of them when compared to Selenium? – Shivam Mishra Nov 25 '18 at 14:26
1

First of all: "Selenium automates browsers. That's it!" (Quote from SeleniumHQ.) It provides an open-source, standardized, and widely-used interface to drive browsers, respectively, web applications. This not just enables developers/testers/… to use it for testing purposes, but also it can be used by vendors to build different kinds of testing tools/frameworks/… on top of it. Because of this, I think that a) Selenium is gonna stay for a very long time and b) the progress in the field of AI-based testing is mostly unrelated to Selenium's lifetime expectations.

"Because some things will be better done using them. Other things will be better done using Selenium" My question is to know exactly what those things are.

Since you mention products like mabl and Testim, I assume you focus on GUI-based system testing for web applications? Today, most of such (off-the-shelf) tools that incorporate some sort of AI basically can be divided into the following two main categories (although some also fit in both):

  1. Test maintenance: the tool takes care of maintenance tasks such as updating locators after changes or carrying out special checks to detect visual differences.
  2. Test generation: the tool generates actual tests that can be executed. For instance, automated exploration to find broken links or the creation of regression test suites.

When it comes to test maintenance, machines can—as already pointed out by João Farias—easily process vast amounts of data, which is a huge advantage here. For instance, when the system under test (SUT) changes, you often find yourself fixing a bunch of locators. In contrast, machines can use historical data to assign new and old locators in order to "auto-heal" tests.

However, in the case of test generation, humans are usually superior. While machines can use various techniques to create non-functional tests, they have a hard time to generate actual functional tests that effectively cover particular use cases and business needs. In general, I think this relates to the testing vs. checking debate (see my answer here).

  • In the last paragraph, by "test generation" do you mean test cases generation or test code generation? – Shivam Mishra Nov 25 '18 at 12:29
  • @ShivamMishra how would you distinguish between them? – beatngu13 Nov 25 '18 at 12:56
  • test cases are the test cases written in English or human readable language and test code is the Selenium (along with other supporting frameworks) code in a scripting language. AI tools like testim.io aims to eliminates the need for writing test code(except few cases where you can write JS). – Shivam Mishra Nov 25 '18 at 13:12
  • @ShivamMishra IMHO test cases specify data inputs, execution conditions, testing procedures, and expected results. They aren't bound to a specific format, (informal) human language or (formal) Selenium code are just two possible formats. Therefore, I think the ability to generate tests is more abstract and doesn't dependent on a particular format. Testim uses historical data to rank the locators for each element individually to stabilize the tests over time, you still create tests manually (via a recorder, code, or both). A human decides what to test and how, no test generation. – beatngu13 Nov 25 '18 at 13:30
  • Ofcourse a human will design the teat cases, prepare the testing strategy and decide what to test. AI tools do not intend to replace the ones who do this, instead as I said they intend to replace the task of writing code to automate those tests which clearly brings them head to head with Selenium. So my question is what are their limitations when compared with Selenium (or "codeful" automation)? – Shivam Mishra Nov 25 '18 at 13:36
1

Comparing Selenium to AI driven testing is not exactly the right way to think about it. But an interesting proposition and question.

Selenium and Cypress and CodedUI and others are simply ways to automate browser tests at the UI level. We all know these are brittle and challenged with modern UI libraries. But they are mature (Selenium is 15 years old now) and function as a language to automate UI actions. When Selenium was introduced it was a godsend. But that was 2004.

I read a study recently that on average 7 selenium scripts can be fully written and debugged per man-day. Of course some take all day to just get one to work...others are faster. But the 1 script an hour rule isn't a bad way to think about any of these languages.

Codeless script generation (ie recording) was tossed out years ago as unreliable and unmaintainable. But a whole new generation popped up in the last few years that changed that (for example Test Designer in Appvance IQ). It generates your test in open-source javascript as the tester uses the application. And self-heals broken accessors at each subsequent run. Data driven of course. We have measured this at 4X - 10X faster than writing selenium scripts. But it depends on the application and skill of the QA engineer. Others have introduced recorders as well with different languages and features (Functionize, Mabel etc). I really suggest you look in to some of these.

AI test generation was first introduced in 2017. The technology has rapidly progressed to where AIQ can generate 600 scripts per minute unaided by humans. Thats wicked fast. We cannot compare 7 scripts a day to 6000 in 10 minutes. But how good are these? Are they valid and valuable user flows?

To date we have found that a mix of maybe 10-20 QA designed scripts alongside 5000 or so AI generated scripts provides the right balance of coverage and assurance of very specific user flows. In the end the QA team must report back up the chain that certain items completely work in the current build. Whether or not AI thought it was a valuable exercise to test that flow. So we have found this combination is critical for assurance.

For the dozens of clients using AI test generation daily from us, they indeed have dropped all use of Selenium. That was their own choice over some months. They found that specific cases can be generated faster in AIQ Test Designer (recorder) and self-healed. And the rest of the coverage generated by AI. So that left no place for Selenium.

Since Selenium is so pervasive, someone will still be using it a decade from now. But as AI test generation and advanced new recorders become more commonplace and more reliable, it does seem to squeeze out the need for true manual selenium coding as we have done for the past 15 years.

Of course, Selenium is free. AI isn't. For small dev teams free is a pretty great choice. but for large enterprises with budget, AI driven testing can have a meaningful ROI despite the cost of the technology.

disclaimer - I work for appvance.ai and I am sharing my own experience helping teams roll out AI driven testing.

0

Talking about the general limitations of AI would fit at least a book - given only our current understanding of it. 5 more years and half the book can be re-written.

In summary, to the context software testing in general, I would raise two points:

1 - AI system do not learn as humans do. AI systems are algorithms - they are data-driven, which allow more flexibility than path/logical-expressions driven algorithms. They do somethings better than humans: Finding patterns in a large dataset and classifying new data, for instance. But they cannot put in context the subjective aspects of human behavior - one needs to encode it somehow, so the AI system can use this data; this is transforming subjective into objective - thus, losing information in the process.

2 - Projects have a life-time: A project is implemented in a certain time-frame. Therefore, it would bound to the tools available of the time, or that can be efficiently built during this time-frame. Therefore, some problems would not be solved by AI for certain projects because the system would not be available of the time.

About Selenium itself, considering the Lindy Effect, it should be around for some time yet. However, for the same principle, some of these AI tools will surely take some of the marketshare.

Why? Because some things will be better done using them. Other things will be better done using Selenium - probably some tools will use Selenium as helper for some things, instead of re-creating the wheel.

Cobol is around there until today - and probably will still be around for some time.

  • "Because some things will be better done using them. Other things will be better done using Selenium" My question is to know exactly what those things are. Thanks. – Shivam Mishra Nov 25 '18 at 8:49
0

First of all, lets set some context on why this is the situation, then I will share my thoughts on limitations of AI based tools.

For the past several decades, the biggest challenge with automated testing has been the issue of maintenance. How many times have you been in this situation where, say you write 10 tests, you run them and it passes. You are super happy!!! Then, you come in the next day only to find out that all the 10 tests are failing because the elements in the pages have changed. I have been in this situation so many times. Research suggests that on an average, a tester spends 30% of his/her time in just maintaining these tests. Can you imagine the opportunity cost associated with this effort?

Then, we have couple of other issues. One is the availability of skilled resources, as they are expensive and getting the right kind of people to do test automation takes considerable amount of time and effort. Then, we have the cost of getting these resources and training them as they quite often do not come cheap.

Selenium is a great open source framework to do test automation and it is FREE, but suffers because of the above mentioned issues. That is why we have AI based testing tools and they have become popular in the past few years. Companies started thinking “When someone can provide all these features for us, why do I have to suffer by doing everything on my own with the Selenium framework?”. This is exactly what happened which has resulted in people moving away from the Selenium framework to more robust vendor solutions. This is just like how people thought “Wow I love Blackberry’s” and then the iPhone came in and made Blackberry’s non-existent because the iPhone could provide all the features in one package; and some of the features even the customers did not know they needed them but got hooked on to it.

This being said, it does not mean Selenium is going to get replaced. It is just going to force the open source community (The chief committers to the Selenium project) to do more work and try to fix the existing problems with the framework and make it more easier to use even for non-technical people. Again remember that, the Selenium framework is maintained by people who like to help the community and most of them do not get paid to do it. So what you see is what you get and we cannot complain about it. Rather, look at the advantages and limitations and make our automation suite better.

Speaking of limitations, AI tools do have their own limitations just like Selenium.

  • Each vendor tool you see currently are good in one aspect of test automation like either authoring, execution or maintenance. There are very few tools that do all 3 seamlessly in an integrated way
  • Most of the tools have some algorithm behind the scenes trying to do self-healing i.e fixing issues proactively before they occur. In order for the algorithms to get smarter, it needs more data and more users. So, until the vendors have a lot of users using their product the algorithm may not have sufficient data to get smarter. In summary, it is going to take some more time to have complete autonomous testing
  • Most of the tools are focused on making test automation easier and more stable but do not do complete test management with fancy dashboards and reporting like HP/ALM, TestRail etc.
  • Most of the tool vendors do not have 24/7 customer support to help users get used to the tool and the AI behind the scenes.
  • There are no open source AI based tools. So you will have to pay to use the features of these tools unlike Selenium which is FREE

So it all boils down to what you want to do, how much time, effort and cost you are willing to put in for test automation.

Hope this helps

-Raj

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.