Why does Google Chrome initiated through Selenium shows less attributes for anchor tags compared to normal Chrome session?

Question

I'm trying to scrape some image href attributes from the <a> tags from a Google search using Selenium, ChromeDriver and Chrome. When the url is accessed manually I'm able to locate the required href attributes but when I initiate a Google Chrome session through ChromeDriver those href attributes aren't available in the DOM and I get empty results.

Code trials:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

keyword = "cats"
url = "https://www.google.com/search?q=" + keyword + "&source=lnms&tbm=isch&tbs="
print(url)
options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get(url)
print([my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.islrc>div>a")))])
print([my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='islrc']/div/a")))])

---

HTML of manually launched Chrome observed through Inspect:

<a class="wXeWr islib nfEiy mM5pbd" jsname="sTFXNd" jsaction="click:J9iaEb;" data-nav="1" tabindex="0" style="height: 180px;" href="/imgres?imgurl=https%3A%2F%2Ficatcare.org%2Fapp%2Fuploads%2F2018%2F07%2FThinking-of-getting-a-cat.png&amp;imgrefurl=https%3A%2F%2Ficatcare.org%2Fadvice%2Fthinking-of-getting-a-cat%2F&amp;tbnid=0V922RrJgQc9SM&amp;vet=12ahUKEwi8rsyAvKrqAhVPBysKHfuyCwgQMygAegUIARDPAQ..i&amp;docid=5qEHfJOysK_DwM&amp;w=1200&amp;h=600&amp;q=cats&amp;ved=2ahUKEwi8rsyAvKrqAhVPBysKHfuyCwgQMygAegUIARDPAQ" data-navigation="server">

Snapshot:

---

HTML of Selenium launched Chrome:

<a class="wXeWr islib nfEiy mM5pbd" jsname="sTFXNd" jsaction="click:J9iaEb;" data-nav="1" tabindex="0" style="height: 180px;"><div class="bRMDJf islir" jsname="DeysSe" style="margin-top: 11px; margin-left: 19px; margin-right: 18px;" jsaction="mousedown:npT2md; touchstart:npT2md;">

Snapshot:

I'm using the Selenium Python client. Can someone help me to understand the difference?

Answer 1

Few things to remember:

1. You are doing visibility of all elements located by, this will check the visibility of elements which was identified. If the element with href was not loaded during this time then it won't consider that
2. Your locator div.islrc>div>a detects many elements and few doesn't have href and if href is not available then getAttribute will return empty string

Correct Approach:

1. use the locator div.islrc>div>a.VFACy.kGQAp (Just included the anchor elements class
2. Add a sleep before trying explicit wait. Explicit wait is the actual fix but for debugging don't use it . For debugging purpose always use time.sleep(10000) or a break point. Give a long sleep and then see whats the behavior
3. If this works, then use explicit wait

Answer 2

There are some difference between the HTML shown through View Source i.e. using ctrl + U and the HTML shown through Inspector i.e. using ctrl + shift + I.

Both the methods are two different browser features which allows users to look at the HTML of the webpage. However, the main difference is the View Source shows the HTML that was delivered from the web server (application server) to the browser. Where as, Inspect element is a Developer Tool e.g. Chrome DevTools to look at the state of the DOM Tree after the browser has applied its error correction and after any Javascript have manipulated the DOM. Some of those activities may include:

• HTML error correction by the browser
• HTML normalization by the browser
• DOM manipulation by Javascript

In short, using View Source you will observe the Javascript but not the HTML. The HTML errors may get corrected in the Inspect Elements tool. As an example:

• With in View Source you may observe:

  <h1>The title</h2>
  

• Whereas through Inspect Element that would have corrected as:

  <h1>The title</h1>
  

Answer 3

In manual launch you would have checked the a tag attributes after clicking image preview flyout

Attribute snapshot before image preview

Attribute snapshot with image preview

Attribute snapshot after closing the image preview