I'm trying to scrape some image href attributes from the <a> tags from a Google search using Selenium, ChromeDriver and Chrome. When the url is accessed manually I'm able to locate the required href attributes but when I initiate a Google Chrome session through ChromeDriver those href attributes aren't available in the DOM and I get empty results.

Code trials:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

keyword = "cats"
url = "https://www.google.com/search?q=" + keyword + "&source=lnms&tbm=isch&tbs="
options = webdriver.ChromeOptions() 
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
print([my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.islrc>div>a")))])
print([my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='islrc']/div/a")))])

HTML of manually launched Chrome observed through Inspect:

<a class="wXeWr islib nfEiy mM5pbd" jsname="sTFXNd" jsaction="click:J9iaEb;" data-nav="1" tabindex="0" style="height: 180px;" href="/imgres?imgurl=https%3A%2F%2Ficatcare.org%2Fapp%2Fuploads%2F2018%2F07%2FThinking-of-getting-a-cat.png&amp;imgrefurl=https%3A%2F%2Ficatcare.org%2Fadvice%2Fthinking-of-getting-a-cat%2F&amp;tbnid=0V922RrJgQc9SM&amp;vet=12ahUKEwi8rsyAvKrqAhVPBysKHfuyCwgQMygAegUIARDPAQ..i&amp;docid=5qEHfJOysK_DwM&amp;w=1200&amp;h=600&amp;q=cats&amp;ved=2ahUKEwi8rsyAvKrqAhVPBysKHfuyCwgQMygAegUIARDPAQ" data-navigation="server">



HTML of Selenium launched Chrome:

<a class="wXeWr islib nfEiy mM5pbd" jsname="sTFXNd" jsaction="click:J9iaEb;" data-nav="1" tabindex="0" style="height: 180px;"><div class="bRMDJf islir" jsname="DeysSe" style="margin-top: 11px; margin-left: 19px; margin-right: 18px;" jsaction="mousedown:npT2md; touchstart:npT2md;">



I'm using the Selenium Python client. Can someone help me to understand the difference?


Few things to remember:

  1. You are doing visibility of all elements located by, this will check the visibility of elements which was identified. If the element with href was not loaded during this time then it won't consider that
  2. Your locator div.islrc>div>a detects many elements and few doesn't have href and if href is not available then getAttribute will return empty string

Correct Approach:

  1. use the locator div.islrc>div>a.VFACy.kGQAp (Just included the anchor elements class
  2. Add a sleep before trying explicit wait. Explicit wait is the actual fix but for debugging don't use it . For debugging purpose always use time.sleep(10000) or a break point. Give a long sleep and then see whats the behavior
  3. If this works, then use explicit wait

There are some difference between the HTML shown through View Source i.e. using ctrl + U and the HTML shown through Inspector i.e. using ctrl + shift + I.

Both the methods are two different browser features which allows users to look at the HTML of the webpage. However, the main difference is the View Source shows the HTML that was delivered from the web server (application server) to the browser. Where as, Inspect element is a Developer Tool e.g. Chrome DevTools to look at the state of the DOM Tree after the browser has applied its error correction and after any Javascript have manipulated the DOM. Some of those activities may include:

  • HTML error correction by the browser
  • HTML normalization by the browser
  • DOM manipulation by Javascript

In short, using View Source you will observe the Javascript but not the HTML. The HTML errors may get corrected in the Inspect Elements tool. As an example:

  • With in View Source you may observe:

    <h1>The title</h2>
  • Whereas through Inspect Element that would have corrected as:

    <h1>The title</h1>

In manual launch you would have checked the a tag attributes after clicking image preview flyout

Attribute snapshot before image preview enter image description here

Attribute snapshot with image preview enter image description here

Attribute snapshot after closing the image preview enter image description here

  • Okay, great point. actually in the manual process as the href attribute was available, so while automating when I tried to extract the hrefs there's where I was stumped. – DebanjanB Jul 1 '20 at 5:00
  • Yeah!! i guess you in manual launch would have captured these attributes after Image preview – Mohamed Sulaimaan Sheriff Jul 1 '20 at 5:02
  • Well, no, I tried without the image preview step. Updated the question with my last line of code trials. – DebanjanB Jul 1 '20 at 5:05
  • But for me the manual launch the Image attributes(without image preview) comes same as shown in snapshot(i.stack.imgur.com/czBCv.png) – Mohamed Sulaimaan Sheriff Jul 1 '20 at 5:15
  • The href attributes is visible only when an user manually Inspects – DebanjanB Jul 1 '20 at 14:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.