Using loop how to get dataframe from each next pages and store table values using Selenium Python

Question

https://seffaflik.epias.com.tr/transparency/tuketim/gerceklesen-tuketim/gercek-zamanli-tuketim.xhtml

from the above website, I've achieved to enter date picker for two date range and then clicking "uygula" or "ok" the green execute button and successfully store table in the page as a pandas dataframe.

Everything perfect untill now. But if data range is not selected as same days, table size is going bigger and eventually creating next pages.

from selenium import webdriver
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.keys import Keys
import time
from selenium.webdriver import ActionChains
import pandas as pd
driver=webdriver.Firefox(executable_path=r'C:\Chrome\geckodriver.exe')
url=str('https://seffaflik.epias.com.tr/transparency/tuketim/gerceklesen-tuketim/gercek-zamanli- 
tuketim.xhtml')
driver.get(url)
driver.maximize_window()
date_value1='01.07.2020'
date_value2='01.07.2020'
 try:
 date1=WebDriverWait(driver,15).until(EC.presence_of_element_located((By.ID,'j_idt205:date1_input')))

  except:
        driver.quit()

 date1.clear();
 date1.send_keys(date_value1)
 try:
 date2=WebDriverWait(driver,15).until(EC.presence_of_element_located((By.ID,'j_idt205:date2_input')))

  except:
    driver.quit()

  date2.clear();
  date2.send_keys(date_value2)
  date2.submit()
  time.sleep(10)

  driver.find_element_by_name('j_idt205:goster').click()

  if date_value1==date_value2:
   df = pd.read_html(driver.page_source)[0]
   print(df)
  else:
    ???????

I want to write a loop in the above else part to scrap all pages table values and storing in dataframes as iteration. Then concatanate all the dataframes as one table.

The problem here we also dont know max page number. If we choose for example:01.07.2020-02.07.2020 it would create a few pages but if we choose 01.07.2019-01.07.2020 it would creates much more pages than the previous one.

So please could you help me to give a hint to solve this dynamic problem.

Answer 1

I have made it big progress but there is last minor problem left. "df.append(pd.read_html(driver.page_source)[0])"

with the code above want to append all the tables in the pages but it append only the first page repetitively.

df = pd.read_html(driver.page_source)[0]

    while True:

        next_page_btn = driver.find_elements_by_xpath('//*[@id="j_idt205:dt_paginator_bottom"]/a[3]/span')


        try:
                  time.sleep(7)
                  next_page_btn[0].click()
                  df.append(pd.read_html(driver.page_source)[0])
        except (TimeoutException, WebDriverException) :

                        print("Last page reached")
                        print(df.tail())
                        break

How can i modify that part?