1

https://seffaflik.epias.com.tr/transparency/tuketim/gerceklesen-tuketim/gercek-zamanli-tuketim.xhtml

from the above website, I've achieved to enter date picker for two date range and then clicking "uygula" or "ok" the green execute button and successfully store table in the page as a pandas dataframe.

Everything perfect untill now. But if data range is not selected as same days, table size is going bigger and eventually creating next pages.

from selenium import webdriver
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.keys import Keys
import time
from selenium.webdriver import ActionChains
import pandas as pd
driver=webdriver.Firefox(executable_path=r'C:\Chrome\geckodriver.exe')
url=str('https://seffaflik.epias.com.tr/transparency/tuketim/gerceklesen-tuketim/gercek-zamanli- 
tuketim.xhtml')
driver.get(url)
driver.maximize_window()
date_value1='01.07.2020'
date_value2='01.07.2020'
 try:
 date1=WebDriverWait(driver,15).until(EC.presence_of_element_located((By.ID,'j_idt205:date1_input')))

  except:
        driver.quit()

 date1.clear();
 date1.send_keys(date_value1)
 try:
 date2=WebDriverWait(driver,15).until(EC.presence_of_element_located((By.ID,'j_idt205:date2_input')))

  except:
    driver.quit()

  date2.clear();
  date2.send_keys(date_value2)
  date2.submit()
  time.sleep(10)

  driver.find_element_by_name('j_idt205:goster').click()

  if date_value1==date_value2:
   df = pd.read_html(driver.page_source)[0]
   print(df)
  else:
    ???????

I want to write a loop in the above else part to scrap all pages table values and storing in dataframes as iteration. Then concatanate all the dataframes as one table.

The problem here we also dont know max page number. If we choose for example:01.07.2020-02.07.2020 it would create a few pages but if we choose 01.07.2019-01.07.2020 it would creates much more pages than the previous one.

So please could you help me to give a hint to solve this dynamic problem.

  • I couldn't identify easily because the site loads a bunch of media resources, but what about, instead of going through the UI, you simply call the endpoint from which this data comes from, fetching all the data you need at once? – João Farias Jul 3 at 21:52
  • Of course it would be better but i dont know how to do that:) – PerfectLikeNew Jul 4 at 11:14
0

I have made it big progress but there is last minor problem left. "df.append(pd.read_html(driver.page_source)[0])"

with the code above want to append all the tables in the pages but it append only the first page repetitively.

df = pd.read_html(driver.page_source)[0]

    while True:

        next_page_btn = driver.find_elements_by_xpath('//*[@id="j_idt205:dt_paginator_bottom"]/a[3]/span')


        try:
                  time.sleep(7)
                  next_page_btn[0].click()
                  df.append(pd.read_html(driver.page_source)[0])
        except (TimeoutException, WebDriverException) :

                        print("Last page reached")
                        print(df.tail())
                        break

How can i modify that part?

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.