0

I'm trying to scrape some image href attributes from the <a> tags from a Google search using Selenium, ChromeDriver and Chrome. When the url is accessed manually I'm able to locate the required href attributes but when I initiate a Google Chrome session through ChromeDriver those href attributes aren't available in the DOM and I get empty results.

Code trials:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

keyword = "cats"
url = "https://www.google.com/search?q=" + keyword + "&source=lnms&tbm=isch&tbs="
print(url)
options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get(url)
print([my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.islrc>div>a")))])
print([my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='islrc']/div/a")))])

HTML of manually launched Chrome observed through Inspect:

<a class="wXeWr islib nfEiy mM5pbd" jsname="sTFXNd" jsaction="click:J9iaEb;" data-nav="1" tabindex="0" style="height: 180px;" href="/imgres?imgurl=https%3A%2F%2Ficatcare.org%2Fapp%2Fuploads%2F2018%2F07%2FThinking-of-getting-a-cat.png&amp;imgrefurl=https%3A%2F%2Ficatcare.org%2Fadvice%2Fthinking-of-getting-a-cat%2F&amp;tbnid=0V922RrJgQc9SM&amp;vet=12ahUKEwi8rsyAvKrqAhVPBysKHfuyCwgQMygAegUIARDPAQ..i&amp;docid=5qEHfJOysK_DwM&amp;w=1200&amp;h=600&amp;q=cats&amp;ved=2ahUKEwi8rsyAvKrqAhVPBysKHfuyCwgQMygAegUIARDPAQ" data-navigation="server">

Snapshot:

manual


HTML of Selenium launched Chrome:

<a class="wXeWr islib nfEiy mM5pbd" jsname="sTFXNd" jsaction="click:J9iaEb;" data-nav="1" tabindex="0" style="height: 180px;"><div class="bRMDJf islir" jsname="DeysSe" style="margin-top: 11px; margin-left: 19px; margin-right: 18px;" jsaction="mousedown:npT2md; touchstart:npT2md;">

Snapshot:

selenium

I'm using the Selenium Python client. Can someone help me to understand the difference?

1

In manual launch you would have checked the a tag attributes after clicking image preview flyout

Attribute snapshot before image preview enter image description here

Attribute snapshot with image preview enter image description here

Attribute snapshot after closing the image preview enter image description here

| improve this answer | |
  • Okay, great point. actually in the manual process as the href attribute was available, so while automating when I tried to extract the hrefs there's where I was stumped. – DebanjanB Jul 1 at 5:00
  • Yeah!! i guess you in manual launch would have captured these attributes after Image preview – Mohamed Sulaimaan Sheriff Jul 1 at 5:02
  • Well, no, I tried without the image preview step. Updated the question with my last line of code trials. – DebanjanB Jul 1 at 5:05
  • But for me the manual launch the Image attributes(without image preview) comes same as shown in snapshot(i.stack.imgur.com/czBCv.png) – Mohamed Sulaimaan Sheriff Jul 1 at 5:15
  • The href attributes is visible only when an user manually Inspects – DebanjanB Jul 1 at 14:05
1

Few things to remember:

  1. You are doing visibility of all elements located by, this will check the visibility of elements which was identified. If the element with href was not loaded during this time then it won't consider that
  2. Your locator div.islrc>div>a detects many elements and few doesn't have href and if href is not available then getAttribute will return empty string

Correct Approach:

  1. use the locator div.islrc>div>a.VFACy.kGQAp (Just included the anchor elements class
  2. Add a sleep before trying explicit wait. Explicit wait is the actual fix but for debugging don't use it . For debugging purpose always use time.sleep(10000) or a break point. Give a long sleep and then see whats the behavior
  3. If this works, then use explicit wait
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.