0

I am trying to scrape some data using the following code. When I run the code line by line, it works fine. However, when I want to run all code at one go, the dropdown options go blank and as a result, the last line returns error. Your help would be much appreciated.

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.select import Select
import os

path = os.path.join(r"D:\ScrapedData\TN\SocialAudit")
path_to_chromedriver = 'D:\ScrapedData/chromedriver'
options = webdriver.ChromeOptions()
prefs = {'download.default_directory' : path}
options.add_experimental_option('prefs', prefs)
browser = webdriver.Chrome(chrome_options=options ,executable_path=path_to_chromedriver)    

url = "http://mnregaweb4.nic.in/netnrega/SocialAudit/StateList.aspx"
browser.get(url)
browser.set_page_load_timeout(45)
browser.maximize_window()

browser.find_element_by_link_text("BIHAR").click()
browser.implicitly_wait(5)

year=['2016-2017', '2017-2018', '2018-2019', '2019-2020']
elem2 = browser.find_element_by_name("ctl00$ContentPlaceHolder1$ddlFin")
elem2.send_keys(year[0])
browser.implicitly_wait(5)

select_dist = browser.find_element_by_name("ctl00$ContentPlaceHolder1$ddldist")
options = [x for x in select_dist.find_elements_by_tag_name("option")]
dist=[]
for e in range(len(options)):
    select_dist = Select(browser.find_element_by_name("ctl00$ContentPlaceHolder1$ddldist"))
    select_dist.select_by_index(e)

    select_block = Select(browser.find_element_by_name("ctl00$ContentPlaceHolder1$ddlblock"))
    options1 = select_block.options
    for f in range(len(options1)):
        select_block = Select(browser.find_element_by_name("ctl00$ContentPlaceHolder1$ddlblock"))
        select_block.select_by_index(f)

        select_gp = Select(browser.find_element_by_name("ctl00$ContentPlaceHolder1$ddlpanchayat"))
        options2 = select_gp.options
        for g in range(len(options2)):
           select_gp = Select(browser.find_element_by_name("ctl00$ContentPlaceHolder1$ddlpanchayat"))
           select_gp.select_by_index(g)

           browser.find_element_by_css_selector("#ctl00_ContentPlaceHolder1_rbLoginLevel_1").click()
           browser.implicitly_wait(10)

           elem6 = browser.find_element_by_name("ctl00$ContentPlaceHolder1$txtperiodFrom")
           elem6.send_keys('01/04/2016')
           browser.implicitly_wait(10)

           elem7 = browser.find_element_by_name("ctl00$ContentPlaceHolder1$txtperiodTo")
           elem7.send_keys('31/03/2017')
           browser.implicitly_wait(10)

           browser.find_element_by_css_selector("#ctl00_ContentPlaceHolder1_login").click()
           browser.implicitly_wait(10)

           browser.find_element_by_link_text("Download All Reports").click()
  • can you put a executable code , it doesn't have browser or options defined. Just add the entire code that could be executed so that we can debug it – PDHide Apr 15 at 12:57
  • Hi, I have edited the above codes to include the whole module – Bidhu Apr 16 at 3:33
  • why are you iterating over options ? its throwing erro option doesn't have property length – PDHide Apr 16 at 6:21
  • These are conditional dropdowns. I am iterating so that I don't need to select the district, block and Panchayat name everytime I complete downloading the file. When I run these codes line by line, it works very much fine, but in a lop environment it does not work – Bidhu Apr 16 at 11:14
  • in your code options is chrome.options , why u want to iterate over it . – PDHide Apr 16 at 11:17
0

Working code:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.select import Select

import time
import os

path = os.path.join(r"D:\ScrapedData\TN\SocialAudit")
path_to_chromedriver = r'C:\Users\prave\Downloads\chromedriver_win32 (2)\chromedriver.exe'
options = webdriver.ChromeOptions()
prefs = {'download.default_directory' : path}
options.add_experimental_option('prefs', prefs)
browser = webdriver.Chrome(chrome_options=options ,executable_path=path_to_chromedriver)    

url = "http://mnregaweb4.nic.in/netnrega/SocialAudit/StateList.aspx"
browser.get(url)
browser.set_page_load_timeout(45)
browser.maximize_window()

browser.find_element_by_link_text("BIHAR").click()


year=['2016-2017', '2017-2018', '2018-2019', '2019-2020']
elem2 = browser.find_element_by_name("ctl00$ContentPlaceHolder1$ddlFin")
elem2.send_keys(year[0])
time.sleep(4)

select_dist = browser.find_element_by_name("ctl00$ContentPlaceHolder1$ddldist")
options = [x for x in select_dist.find_elements_by_tag_name("option")]
len(options)
dist=[]
for e in range(1,len(options)):
    select_dist = Select(browser.find_element_by_name("ctl00$ContentPlaceHolder1$ddldist"))
    select_dist.select_by_index(e)
    time.sleep(4)
    select_block = Select(browser.find_element_by_name("ctl00$ContentPlaceHolder1$ddlblock"))
    a =select_dist
    options1 = select_block.options
    for f in range(1,len(options1)):
        select_block = Select(browser.find_element_by_name("ctl00$ContentPlaceHolder1$ddlblock"))
        select_block.select_by_index(f)
        time.sleep(4)
        select_gp = Select(browser.find_element_by_name("ctl00$ContentPlaceHolder1$ddlpanchayat"))
        options2 = select_gp.options
        for g in range(1,len(options2)):
           select_gp = Select(browser.find_element_by_name("ctl00$ContentPlaceHolder1$ddlpanchayat"))
           select_gp.select_by_index(g)

           time.sleep(4)

           browser.find_element_by_css_selector("#ctl00_ContentPlaceHolder1_rbLoginLevel_1").click()
           time.sleep(4)

           elem6 = browser.find_element_by_name("ctl00$ContentPlaceHolder1$txtperiodFrom")
           elem6.send_keys('01/04/2016')
           time.sleep(4)

           elem7 = browser.find_element_by_name("ctl00$ContentPlaceHolder1$txtperiodTo")
           elem7.send_keys('31/03/2017')
           time.sleep(4)

           browser.find_element_by_css_selector("#ctl00_ContentPlaceHolder1_login").click()
           time.sleep(4)

           browser.find_element_by_link_text("Download All Reports").click()

Explanation:

  1. Your range was starting from '0' and the 0th element was "select_something". So changed it to range(1,len(a)) , so range starts from 1.
  2. Don't use implicit wait, it will make the entire code slow, use time.sleep() or explicit wait instead. Here we used time.sleep() ( Recommended is an explicit wait )

Note:

I am not sure how your code worked for you when executed line by line, it should fail due to an alert saying "Select all drop-down" before proceeding.

| improve this answer | |
  • Wonderful. It's working fine. Thank you so much. So, the key was using time.sleep. – Bidhu Apr 19 at 15:39
  • @Bidhu no the issue was that you used range starting from 0 – PDHide Apr 19 at 15:50
  • Okay. A follow-up question: when I run the "download all reports", it opens a new tab. As a result, the code interrupts. I am wondering if there is a way to prevent changing the focus of tab to the one with dropdowns – Bidhu Apr 19 at 15:59
  • read about switching tab in selenium – PDHide Apr 19 at 16:09
  • Yes, I am already into that. – Bidhu Apr 19 at 16:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.