0

Trying to scrape data from https://www.sgx.com/research-education/securities and the code below works but only for the first 10 iterations after which it throws a NoSuchElement Exception. Starting from the 2nd iteration will also throw the same error after completing the 10th iteration. However if I set my range to start from 11, it is able to find that single element(but doesn't go on to 12). Since selenium is able to find individual elements, it appears to me there isn't a problem with xpath... Can someone tell me what's wrong? I have tried googling the problem which advised me to add an explicit wait time. I did try that but it did not work and I'm doubting that's the issue. Help is greatly appreciated!

Line with error: selectdate = driver.find_element_by_xpath('//*[@data-key='+str(i)+']')

Error message: raise exception_class(message, screen, stacktrace) selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@data-key=10]"} (Session info: chrome=83.0.4103.116)

import os
from selenium import webdriver #for navigating through the pages
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

import time
from bs4 import BeautifulSoup #webscraping fundamentals
import urllib.request
import pandas as pd #Data manipulation
import re
import os
from selenium.common.exceptions import NoSuchElementException

options = webdriver.ChromeOptions() 
options.add_argument('--ignore-certificate-errors')
options.add_argument('--incognito')
options.add_argument('--headless')
driver = webdriver.Chrome(executable_path=r'C:\\Users\\User\\Downloads\\chromedriver.exe') 

URL = "https://www.sgx.com/research-education/securities"
driver.get(URL)
driver.implicitly_wait(5) 

response = driver.page_source
soup = BeautifulSoup(response,"lxml")

#CLICK COOKIES
Cookies = driver.find_element_by_xpath('//*[@id="gdpr-banner"]/div/button')
Cookies.click()
driver.implicitly_wait(5)


#DATE


dates = []
dropdowndate = driver.find_element_by_name('date') #date dropdown element
DownloadButton=driver.find_element_by_xpath('//*[@id="page-container"]/template-base/div/div/section[1]/div/sgx-widgets-wrapper\
/widget-research-and-reports-download[1]/widget-reports-securities/div/button')

for i in range(20):        
    dropdowndate.click() #clicking date dropdown
    driver.implicitly_wait(20)
    selectdate = driver.find_element_by_xpath('//*[@data-key='+str(i)+']') #date element
    dates.append(selectdate.get_attribute('title'))
    actions = webdriver.ActionChains(driver)
    actions.move_to_element(selectdate).perform()
    driver.implicitly_wait(20)
    driver.execute_script("arguments[0].click();", selectdate) #not rly what arguments[0] and why i have to do this instead of .click() but i read apparently that it's somehow better
    driver.implicitly_wait(20)
    driver.execute_script("arguments[0].click();", DownloadButton)
    driver.implicitly_wait(20)```

    
  • At which line your getting the erro – PDHide Jul 1 at 9:39
  • Please add the exception when adding such errors – PDHide Jul 1 at 9:39
  • please add comments to your code , what happens when you do argument.clikc() the drop down closes ? – PDHide Jul 1 at 9:41
  • okay I've done that. – ping Jul 1 at 12:49
  • I am not able to reproduce your issue but I have another error: "selenium.common.exceptions.ElementClickInterceptedException: Message: Element <sgx-input-select class="text-strong sgx-input" name="date"> is not clickable at point (452,546) because another element <sgx-loader> obscures it ". Could it be that you trimmed down your code for posting here ? – Anonymous Jul 1 at 16:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.