0

When you search for something in Google and get the results, it is not easy to locate the elements, using Selenium, since Google uses classes like

<div class="kno-ecr-pt kno-fb-ctx gsmt">

What are the best practices working with that?

  • 2
    What would you like to locate exactly? First enty of search result or result containing particular words or something else? – Alexey R. Feb 2 at 14:08
  • 3
    Google does not want many people to use robots against its site. It's intentionally made to make scraping difficult. – João Farias Feb 2 at 17:13
  • @João Farias, the problem is that the companies make test assignments for the candidates to google the company name and check if the result is according to the requirements. – Jaroslav Tavgen Feb 3 at 10:21
  • @Alexey R., in this specific case I need to locate elements that are located on the right side of the google search results page. These elements present the company that you google. There is there logo, name, address. when the office is opened, website etc. – Jaroslav Tavgen Feb 3 at 10:22
  • 1
    What exactly do you mean "not easy" do you mean the codes are not english readable? Do you mean the codes are uniquely generated (dynamic) each time? Start with what exactly you mean by "not easy" will probably lead to better results – Michael Durrant Feb 3 at 14:02
2

One of the main reasons Google and similar top ranking pages make their page elements dynamic is to hinder automation. Most of the attributes like id, class name e.t.c are dynamic and hence out of context here. However, their HTML structure remains almost always the same. That is why we need to make use of xpath and css locators since they rely more on the structure of pages. You need to find a structure for the page you are trying.

In your case, I assume you are trying to find the company name and website after searching for the company name.

Here's the xpath for both:

Company name = //div[@data-ved]/div[@lang][2]

Company website = //div[@data-ved]/div[@lang][3]

If you can specify more about the requirement, we might be able to help with more helpful answers. And it is always suggested to try first and post your findings or issues.

0

Basically you just have to adapt to the page of interest. What approach works best depends on what you're trying to achieve.

If you're looking for specific results, I'd suggest looking for link text or partial link text. This searches the text contained in the <a> tag, not the URL in the href attribute. If you want to look for the URL, use a CSS selector for the href attribute's contents.

If the summary text is more reliable, grab the summary text of the results and look thru that. If you look at the structure of the page, you'll find that each result is wrapped in <div class="g">, within which is <div class="r"> containing the links and <div class="s"> containing the summary text. You can use this structure to always select the nth result, or to go thru the text summaries and select the corresponding link.

Those are some ideas to get you unblocked.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.