4

I like to know if it's possible to automate this 'graph' which is created on the canvas, if yes, how to automate it. I have to interact with all the elements on the 'graph or a map like thing' by clicking on an element and dragging the elements and performing other actions like getting the text, etc. Is this possible in automation using selenium with JavaScript executor, can anyone help.

I already scour the internet but didn't find anything useful or which even remotely answer the question I have. if anyone can help, please do write an answer. I cannot share the real picture due to security reasons, but this is almost the same not identical.

please view picture

Please do note - everything on the 'map' are responsive elements even the lines are clickable.

6
  • What does the code look like for the "map"? If there is a way to uniquely and consistently identify an element on the map, it can be done, but there may be easier ways to do what you need to do.
    – Kate Paulk
    Mar 5, 2020 at 18:16
  • "if there is way to uniquely and consistently identify an element on the map." I don't have any way to uniquely identify them, Maybe through java script i may able to query the Canvas, i don't know. I would highly appreciate if anyone can tell me how to do that. Only the code i have is the canvas... <canvas height="689" width="1685" id="KeyLines-chart-1" style="display: block; width: 1787px; height: 453px; touch-action: none; outline: none; background: white; cursor: auto;" tabindex="8888" class="web-inspector-hide-shortcut"></canvas>
    – Im Nobody
    Mar 5, 2020 at 18:24
  • 1
    So if you were to right-click on one of the elements then select "Inspect element", what would your browser give you?
    – Kate Paulk
    Mar 5, 2020 at 18:32
  • @ImNobody see the update answer
    – PDHide
    Mar 5, 2020 at 23:09
  • Hi Kate, html canvas code which I posted in my previous comment that's the only code I saw. Right clicking on an element won't give you the option to inspect element.
    – Im Nobody
    Mar 6, 2020 at 2:38

2 Answers 2

3

The short answer is no, it's not possible to use Selenium or the JavaScript Executor in Selenium to automate the HTML canvas. I've tried to do this on a project once with Puppeteer, and it's the same problem.

Keep in mind that the HTML canvas is a black box, so Selenium, Puppeteer, or any other browser automation tool that uses the DOM to locate and interact with web elements won't work with the canvas tag. They can see in the DOM, but not what's inside the canvas.

The only option I had on this canvas project to automate it was to have the developer create a way to expose "custom web elements" to be visible outside the canvas. It was a custom approach that worked for that project. Ask your devs to see if they have a way to expose elements outside the canvas. Note, it's been a while since I was on that project so I don't remember the exact details of the implementation.

0
2

Canvas is just like any other element, see the below code that uses google map and prints the outer html of the canvas tag.

so you can get the element attributes like you would for any elements:

          var EC = protractor.ExpectedConditions;
           
          await browser.get('https://www.google.com/maps/search/microchip/@51.8959843,-8.5330895,12z?hl=en')
          await  browser.wait(EC.presenceOf(element(by.xpath("//canvas"))), 5000);
          console.log( await element(by.xpath("//canvas")).getAttribute('outerHTML'));
           
          await browser.sleep(5000)

Output:

Screenshot of output from console window

Interact with canvas:

Canvas elements are dynamically created.

If you want to interact with the canvas, then you can use the browser.actions class in protractor:

My config.js

    exports.config = {
    
    specs: ['test.js'],
    
    /*capabilities: {
      'browserName': 'firefox' // or 'safari'
    },
    */
    onPrepare: function(){
        browser.waitForAngularEnabled(false);
        browser.driver.manage().window().maximize();
    },
    
      jasmineNodeOpts: {
        showColors: true, // Use colors in the command line report.
        defaultTimeoutInterval: 5000000,
    
    },
    SELENIUM_PROMISE_MANAGER: false,
    }

My test

    'use strict';
    
    describe('Validate dfsfdsf 1 behaviour', function () {
    
    
      it('test {Regression} {Sanity} {Sanity}', async function () {
        var EC = protractor.ExpectedConditions;
    
        await browser.get('https://www.google.com/maps/@52.6099788,-2.8433538,8z')
        await browser.wait(EC.presenceOf(element(by.xpath("//canvas"))), 10000);
    
       
    
        await $("#content-container").click()
        await browser.sleep(5000)
        console.log("writing somehting....");
    
        let a = await element(by.xpath("//canvas"))
    
        await browser.actions().
        dragAndDrop(await $('#widget-expand-button-pegman-icon'), { x: 1134, y: 503 }).
          click().perform();
    
        await browser.actions().
          mouseMove(a, { x: 500, y: 0 }).
          click().perform();
    
        await browser.actions().
          doubleClick().
          perform();
    
        await browser.sleep(15000)
    
      });
    
    });

The above code will open google map, and add the 'little man' icon into the map and zoom into the map

To know more about available methods in actions class

As you know, protractor is a wrapper around selenium so whatever selenium-js supports, protractor supports

so goto:

https://www.selenium.dev/selenium/docs/api/javascript/module/selenium-webdriver/

and search for actions, and choose legacyactionssequence

https://www.selenium.dev/selenium/docs/api/javascript/module/selenium-webdriver/lib/actions_exports_LegacyActionSequence.html

this will have full list of supported methods

3
  • Thanks, but how to interact with the element on the canvas which are not visible in Html DOM all I can see is the canvas tag. For an example , in the Google Maps we see the little man which you can drag and drop on the map and zoom in and zoom out buttons, statelite view, etc. If I want to interact with those elements how to do it.
    – Im Nobody
    Mar 6, 2020 at 2:50
  • @ImNobody u can't access the element directly u have to know the coordinate and click on that . Click the little man and specify to what coordinate u have to drop him
    – PDHide
    Mar 6, 2020 at 7:11
  • @ImNobody i have updated the code to drag and drop the person into the canvas , u ,,ight have to change the drop location according to you screen size
    – PDHide
    Mar 6, 2020 at 7:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.