1

I am trying to scroll down a drop-down menu, but since scrolling with a mouse will just scroll the page instead of the drop-down, I thought of using sendkeys by sending DOWN_ARROW multiple times. However this doesn't work very well/is pretty inconsistent.

Does anyone have a solution? I need to scroll down in order to find the element's path. Trying to extract from all the dates from the first "Dates" dropdown menu here. Currently can only do so for the latest 14 https://www.sgx.com/research-education/securities Anything beyond that python reports that it's unable to find the element...

for i in range(10):
    dropdowndate.click()
    driver.implicitly_wait(5)
    selectdate = driver.find_element_by_xpath('//*[@data-key='+str(i)+']')
    dates.append(selectdate.get_attribute('title'))
    webdriver.ActionChains(driver).move_to_element(selectdate).perform()
    selectdate.click()
    driver.implicitly_wait(5)   
    driver.execute_script("arguments[0].click();", DownloadButton)
    driver.implicitly_wait(20)


time.sleep(5)

#11th to 14th elements
for j in range(10,20):
    dropdowndate.click()
    driver.implicitly_wait(5)
    webdriver.ActionChains(driver).send_keys(Keys.ARROW_DOWN * j).perform()
    #dropdowndate.send_keys(Keys.ARROW_DOWN*(13))
    driver.implicitly_wait(30)
    selectdate = WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, '//*[@data-key='+str(j)+']')))
    dates.append(selectdate.get_attribute('title'))
    driver.implicitly_wait(10)
    webdriver.ActionChains(driver).move_to_element(selectdate).perform()
    driver.implicitly_wait(10)
    selectdate.click()
    driver.implicitly_wait(5)
    driver.execute_script("arguments[0].click();", DownloadButton)
    driver.implicitly_wait(10)

Full code is here:

from selenium import webdriver #for navigating through the pages
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

import time
from bs4 import BeautifulSoup #webscraping fundamentals
import urllib.request
import pandas as pd #Data manipulation
import re
import os
import json #to save in multiple folder for looping
from pandas import Series, ExcelWriter #write output to excel in different tab
from selenium.common.exceptions import NoSuchElementException

options = webdriver.ChromeOptions() #selecting the webdriver
options.add_argument('--ignore-certificate-errors')
options.add_argument('--incognito')
options.add_argument('--headless')
driver = webdriver.Chrome(executable_path=r'C:\\Users\\user\\Downloads\\chromedriver.exe') #to always download and update accordingly
#If prompted error on version: go to your browser, type in 'chrome://version' and check version.
#If eg, Google Chrome 80.0.3987.132, go to 'https://chromedriver.chromium.org/downloads' to download
#the same version and update the link above accordingly. Note that browser version should be same as driver version




URL = "https://www.sgx.com/research-education/securities"
driver.get(URL)
driver.implicitly_wait(5) #asking machine to wait 5 seconds before performing other action
#manually block pop-up if any before running next


response = driver.page_source
soup = BeautifulSoup(response,"lxml")


#CLICK ACCEPT COOKIES
Cookies = driver.find_element_by_xpath('//*[@id="gdpr-banner"]/div/button')
Cookies.click()
driver.implicitly_wait(5)

#TYPE OF DATA
dropdowntype = driver.find_element_by_name('type')
dropdowntype.click()
driver.implicitly_wait(3)

selecttype = driver.find_element_by_xpath('//*[@id="sgx-select-dialog"]/div[2]/sgx-select-picker\
/sgx-list/div/div/sgx-select-picker-option[1]/label/span')
#current path is for price, change option[1] to option [2] and option [3] for trade amendment and price adjustment respectively
selecttype.click()


#DATES

dates = []
dropdowndate = driver.find_element_by_name('date')
DownloadButton=driver.find_element_by_xpath('//*[@id="page-container"]/template-base/div/div/section[1]/div/sgx-widgets-wrapper\
/widget-research-and-reports-download[1]/widget-reports-securities/div/button')





#1st to 10th elements

for i in range(10):
    dropdowndate.click()
    driver.implicitly_wait(5)
    selectdate = driver.find_element_by_xpath('//*[@data-key='+str(i)+']')
    dates.append(selectdate.get_attribute('title'))
    webdriver.ActionChains(driver).move_to_element(selectdate).perform()
    selectdate.click()
    driver.implicitly_wait(5)   
    driver.execute_script("arguments[0].click();", DownloadButton)
    driver.implicitly_wait(20)


time.sleep(5)

#11th to 14th elements
for j in range(10,20):
    dropdowndate.click()
    driver.implicitly_wait(5)
    webdriver.ActionChains(driver).send_keys(Keys.ARROW_DOWN * j).perform()
    #dropdowndate.send_keys(Keys.ARROW_DOWN*(13))
    driver.implicitly_wait(30)
    selectdate = WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, '//*[@data-key='+str(j)+']')))
    dates.append(selectdate.get_attribute('title'))
    driver.implicitly_wait(10)
    webdriver.ActionChains(driver).move_to_element(selectdate).perform()
    driver.implicitly_wait(10)
    selectdate.click()
    driver.implicitly_wait(5)
    driver.execute_script("arguments[0].click();", DownloadButton)
    driver.implicitly_wait(10)

1
  • Is your drop-down list such that when you scroll to the very bottom, it loads more data? Mar 5, 2021 at 8:36

1 Answer 1

1

Revising this again to give you the tested/working answer. So you have to get the scroll bar of the dropdown to get to the bottom of the list. In here, you can use drag and drop action or below action that I used.

#DATES

dates = []
dropdowndate = driver.find_element_by_name('date')
DownloadButton=driver.find_element_by_xpath('//*[@id="page-container"]/template-base/div/div/section[1]/div/sgx-widgets-wrapper\
/widget-research-and-reports-download[1]/widget-reports-securities/div/button')

footer = driver.find_element_by_id('website-footer')

try:
    for i in range(0,1000): #0 since it's the start of data-key value
        dropdowndate.click()
        driver.implicitly_wait(2)
        try:
            selectdate = driver.find_element_by_xpath('//*[@data-key='+str(i)+']')
            dates.append(selectdate.get_attribute('title'))
        except Exception as ex:
            dragElem = driver.find_element_by_xpath("//*[@class='list-view']//*[@class='vertical-scrolling-bar']")
            ActionChains(driver).click_and_hold(dragElem).move_to_element(footer).release(footer).perform()
            selectdate = driver.find_element_by_xpath('//*[@data-key='+str(i)+']')
            dates.append(selectdate.get_attribute('title'))
        driver.execute_script("arguments[0].click();", selectdate)
        driver.execute_script("arguments[0].click();", DownloadButton)
except NoSuchElementException:
    print('Downloaded: ' , dates)
else:
    traceback.print_exc()
5
  • Thank you for the reply. But your code only helps me get the first 10 elements as the remaining elements really have to be scrolled before the elements can appear in the html. Another question: How did you get your xpath? (Sorry as I am still relatively new to this) I did get the "class="list-view" part but not the rest.
    – ping
    Jul 7, 2020 at 2:03
  • I wonder why it's not working on your end as it's working on mine. Anyhow, if you check it, elements will appear in html even if you don't scroll it down. Provided you'd click "dropdowndate" Jul 8, 2020 at 3:31
  • If you really want to scroll into element, try this: 'driver.execute_script("arguments[0].scrollIntoView(true);", selectdate)' And here's part of your html where I got the xpath: <label> <input type="radio" name="sgx-select-picker-value" value="19" data-index="19" tabindex="-1" class="sgx-select-picker-index"> <span class="sgx-icon--before sgx-select-picker-label">10 Jun 2020</span> </label></sgx-select-picker-option> Jul 8, 2020 at 3:43
  • When I click on the dropdown box without scrolling and use Ctrl+F to find a date that is at the bottom, (e.g. 12 Jun) it is unable to find anything. But when I scroll all the way down again, it is able to find it. I cannot use scrollintoview with selectdate because python won't be able to find the element before scrolling...
    – ping
    Jul 8, 2020 at 3:54
  • Thank you for trying to help though!
    – ping
    Jul 8, 2020 at 3:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.