How to use OCR in Selenium to identify dynamic objects

Question

I need to implement OCR (tesseract, Abbyy, MODI, Aprise, etc..) to identify dynamically changing web elements in application page at automation runtime. This way the Selenium webdriver automation script will always work without ever changing the object locators in the code.

Any directions to get started?

Answer 1

The flow for something like that would look something like this:

1. Take a screenshot of your full browser window
2. Analyse the screenshot with your OCR software and let it return coordinates
3. Interact with the element on that location, e.g. click item coordinates.

You say you want this because:

The Selenium script will always work without changing object locators in code

I think your thought process is interesting, but I think this is a dream and a lie. What if the application changes so that your OCR software finds two or more objects with the same text? Which one should it use? What if you have multiple elements that look the same, but the order has changed. Think about edit buttons. I think you will introduce new maintainability issues, while you increase the complexity of the testing framework hugely.

I assume you want to solve maintainability issues of Selenium tests, I would try:

• Learn to write good Selenium locators
• Centralize actions and locators in Page Objects
• Force developers to run and update test during development, not afterwards by testers. I would also let the developers create the happy-path tests and extend those by testers if needed.
• Write less Selenium tests, follow the guideline test automation effort spread as described by the test-pyramid.

Answer 2

This answer assumes that you have ruled working on the DOM level with selectors.

So if you want to do visual testing, I see several options:

1. If you are not tied to Selenium, use the free Kantu web testing tool, which works visually, just the way you want it to work. It has built-in OCR. At the very least, you can use this tool test if the visual testing approach itself is the right one for your test case - before spending much time to get Selenium to work this way.

If Selenium:

2. OCR for use with Selenium: Go for an online ocr solution, it will be much easier to use than implementing Tesseract. Here is some Java code to use it.

3. Maybe you do not need OCR, but "only" image recognition? In this case Selenium + Sikuli is an option.