How to query the Canvas using Selenium with JavaScript Executor or Protractor?

Question

I like to know if it's possible to automate this 'graph' which is created on the canvas, if yes, how to automate it. I have to interact with all the elements on the 'graph or a map like thing' by clicking on an element and dragging the elements and performing other actions like getting the text, etc. Is this possible in automation using selenium with JavaScript executor, can anyone help.

I already scour the internet but didn't find anything useful or which even remotely answer the question I have. if anyone can help, please do write an answer. I cannot share the real picture due to security reasons, but this is almost the same not identical.

Please do note - everything on the 'map' are responsive elements even the lines are clickable.

Answer 1

The short answer is no, it's not possible to use Selenium or the JavaScript Executor in Selenium to automate the HTML canvas. I've tried to do this on a project once with Puppeteer, and it's the same problem.

Keep in mind that the HTML canvas is a black box, so Selenium, Puppeteer, or any other browser automation tool that uses the DOM to locate and interact with web elements won't work with the canvas tag. They can see in the DOM, but not what's inside the canvas.

The only option I had on this canvas project to automate it was to have the developer create a way to expose "custom web elements" to be visible outside the canvas. It was a custom approach that worked for that project. Ask your devs to see if they have a way to expose elements outside the canvas. Note, it's been a while since I was on that project so I don't remember the exact details of the implementation.

Answer 2

Canvas is just like any other element, see the below code that uses google map and prints the outer html of the canvas tag.

so you can get the element attributes like you would for any elements:

          var EC = protractor.ExpectedConditions;
           
          await browser.get('https://www.google.com/maps/search/microchip/@51.8959843,-8.5330895,12z?hl=en')
          await  browser.wait(EC.presenceOf(element(by.xpath("//canvas"))), 5000);
          console.log( await element(by.xpath("//canvas")).getAttribute('outerHTML'));
           
          await browser.sleep(5000)

Output:

Interact with canvas:

Canvas elements are dynamically created.

If you want to interact with the canvas, then you can use the browser.actions class in protractor:

My config.js

    exports.config = {
    
    specs: ['test.js'],
    
    /*capabilities: {
      'browserName': 'firefox' // or 'safari'
    },
    */
    onPrepare: function(){
        browser.waitForAngularEnabled(false);
        browser.driver.manage().window().maximize();
    },
    
      jasmineNodeOpts: {
        showColors: true, // Use colors in the command line report.
        defaultTimeoutInterval: 5000000,
    
    },
    SELENIUM_PROMISE_MANAGER: false,
    }

My test

    'use strict';
    
    describe('Validate dfsfdsf 1 behaviour', function () {
    
    
      it('test {Regression} {Sanity} {Sanity}', async function () {
        var EC = protractor.ExpectedConditions;
    
        await browser.get('https://www.google.com/maps/@52.6099788,-2.8433538,8z')
        await browser.wait(EC.presenceOf(element(by.xpath("//canvas"))), 10000);
    
       
    
        await $("#content-container").click()
        await browser.sleep(5000)
        console.log("writing somehting....");
    
        let a = await element(by.xpath("//canvas"))
    
        await browser.actions().
        dragAndDrop(await $('#widget-expand-button-pegman-icon'), { x: 1134, y: 503 }).
          click().perform();
    
        await browser.actions().
          mouseMove(a, { x: 500, y: 0 }).
          click().perform();
    
        await browser.actions().
          doubleClick().
          perform();
    
        await browser.sleep(15000)
    
      });
    
    });

The above code will open google map, and add the 'little man' icon into the map and zoom into the map

To know more about available methods in actions class

As you know, protractor is a wrapper around selenium so whatever selenium-js supports, protractor supports

so goto:

https://www.selenium.dev/selenium/docs/api/javascript/module/selenium-webdriver/

and search for actions, and choose legacyactionssequence

https://www.selenium.dev/selenium/docs/api/javascript/module/selenium-webdriver/lib/actions_exports_LegacyActionSequence.html

this will have full list of supported methods