Unable to extract table content from webpage using selenium

Question

I am trying to extract the tables generated by selecting "Branches", a city and a district from this site: https://www.acb.com.vn/wps/portal/en/atm

So far, I have been able to write the code to parse through each city and district:

from selenium.webdriver.support.ui import Select
from selenium.webdriver import Chrome
import pandas as pd
import time

webdriver = "chromedriver.exe"

driver = Chrome(webdriver)
driver.get('https://www.acb.com.vn/wps/portal/en/atm')
    
branch_selector = driver.find_element_by_xpath('//*[@id="branch"]')
branch_selector.click()

city = Select(driver.find_element_by_id('cityId'))

for i in range(len(city.options)):
    city.select_by_index(i)
    time.sleep(1)
    
    district = Select(driver.find_element_by_id('districtId'))

    for j in range(len(district.options)):
        district.select_by_index(j)
        time.sleep(1)

        try:
            find_btn = driver.find_element_by_xpath('//*[@id="frm-filter"]/div[3]/a[1]')
            find_btn.click()
            time.sleep(1)
            
        except:
            close_btn = driver.find_element_by_xpath('//*[@id="close-send-email"]/span[2]')
            close_btn.click()
            time.sleep(1)

Now, I want to extract the table that's displayed in each iteration of the 2 loops. However, if you look at the HTML for the table, it does not make use of the "table" tag:

So, how do I extract the table for each city-district pair?

I tried the following:

    try:
        click_btn = driver.find_element_by_xpath('//*[@id="frm-filter"]/div[3]/a[1]')
        click_btn.click()
        time.sleep(1)
        
        table = driver.find_elements_by_class_name('tbody')
        for table_row in table:
            row = table_row.find_elements_by_class_name('row')
            print ([r.text for r in row])
        
    except:
        close_btn = driver.find_element_by_xpath('//*[@id="close-send-email"]/span[2]')
        close_btn.click()
        time.sleep(1)

But it prints a list of blank elements for each city-district pair, the length of the list being as many addresses are present in the table for the corresponding city-district pair:

['', '', '', '']
['', '', '', '']
['', '', '', '']
['', '', '', '']
['', '', '', '']
['', '', '', '']
['', '', '', '', '']
['', '', '', '', '']
['', '', '', '', '']
['', '', '', '', '']
['', '', '', '', '']
['', '', '', '', '']
['', '', '', '']
['', '']
['', '']

I also tried to access each element in each row of the table individually:

    try:
        find_btn = driver.find_element_by_xpath('//*[@id="frm-filter"]/div[3]/a[1]')
        find_btn.click()
        time.sleep(1)
        
        table = driver.find_elements_by_class_name('tbody')
        for table_row in table:
            row = table_row.find_elements_by_class_name('row')
            
            for element in row:
                time.sleep(1)
                
                Type.append(element.find_element_by_class_name('col type'))
                Address.append(element.find_element_by_class_name('col address'))
                District.append(element.find_element_by_class_name('col district'))
                Tel_Fax.append(element.find_element_by_class_name('col tel-fax'))
                Hours.append(element.find_element_by_class_name('col hours'))
        
    except:
        close_btn = driver.find_element_by_xpath('//*[@id="close-send-email"]/span[2]')
        close_btn.click()
        time.sleep(1)

But this gives the following error:

---------------------------------------------------------------------------
NoSuchElementException                    Traceback (most recent call last)
<ipython-input-41-2d73f0dc931c> in <module>
     39 
---> 40                     Type.append(element.find_element_by_class_name('col type'))
     41                     Address.append(element.find_element_by_class_name('col address'))

NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="col type"]"}

Since it says css selector in the error, I tried the following:

element.find_element_by_css_selector('div.col.type').text

This outputs a blank string, ''.

So, how do I do this?

EDIT: The HTML of the table, for one district-city selection, is:

        <div class="tbody">
            
            <div class="row" id="row1">
                <div class="col stt">1</div>
                <div class="col type">
                
                PGD Hai Bà Trưng</div>
                <div class="col address">56-58-60 Hai Bà Trưng, P. Bến Nghé, Quan 1, Ho Chi Minh</div>
                <div class="col district">1</div>
                <div class="col tel-fax">(028) 6291 3690<br>(028) 6291 3691</div>
                <div class="col hours">      07:00-16:30</div>
                <div class="col control"><a href="#" title="Direction" class="btn-direction" onclick="showDialogDirection('56-58-60 Hai Bà Trưng, P. Bến Nghé, Quan 1, Ho Chi Minh', '10.77714,106.704325', 1); return false;">Direction</a></div>
            </div>
            
            <div class="row" id="row2">
                <div class="col stt">2</div>
                <div class="col type">
                
                PGD Đa Kao</div>
                <div class="col address">45 Võ Thị Sáu, P. Đa Kao, Quan 1, Ho Chi Minh</div>
                <div class="col district">1</div>
                <div class="col tel-fax">(028) 6290 5980<br>(028) 6290 5981</div>
                <div class="col hours">  07:30 – 16:30</div>
                <div class="col control"><a href="#" title="Direction" class="btn-direction" onclick="showDialogDirection('45 Võ Thị Sáu, P. Đa Kao, Quan 1, Ho Chi Minh', '10.790715,106.69486', 2); return false;">Direction</a></div>
            </div>
            
            <div class="row" id="row3">
                <div class="col stt">3</div>
                <div class="col type">
                
                PGD Nguyễn Công Trứ</div>
                <div class="col address">74 - 76 Nguyễn Công Trứ, P. Nguyễn Thái Bình, Quan 1, Ho Chi Minh</div>
                <div class="col district">1</div>
                <div class="col tel-fax">(028) 3914 4470 <br>(028) 3914 4471</div>
                <div class="col hours">  07:30 – 16:30</div>
                <div class="col control"><a href="#" title="Direction" class="btn-direction" onclick="showDialogDirection('74 - 76 Nguyễn Công Trứ, P. Nguyễn Thái Bình, Quan 1, Ho Chi Minh', '10.76972,106.703142', 3); return false;">Direction</a></div>
            </div>
            
            <div class="row" id="row4">
                <div class="col stt">4</div>
                <div class="col type">
                
                PGD Lê Lợi</div>
                <div class="col address">72 Lê Lợi, P. Bến Thành, Quận 1, TP.Hồ Chí Minh</div>
                <div class="col district">1</div>
                <div class="col tel-fax">(028) 3821 4619<br>(028) 3821 4618</div>
                <div class="col hours">    07:00-16:30</div>
                <div class="col control"><a href="#" title="Direction" class="btn-direction" onclick="showDialogDirection('72 Lê Lợi, P. Bến Thành, Quận 1, TP.Hồ Chí Minh', '10.773541,106.699635', 4); return false;">Direction</a></div>
            </div>
            
            <div class="row" id="row5">
                <div class="col stt">5</div>
                <div class="col type">
                
                CN Sài Gòn</div>
                <div class="col address">41 Mạc Đỉnh Chi, P. Đakao, Quan 1, Ho Chi Minh</div>
                <div class="col district">1</div>
                <div class="col tel-fax">(028) 3824 3770<br>(028) 3824 3946</div>
                <div class="col hours">  07:30 – 16:30</div>
                <div class="col control"><a href="#" title="Direction" class="btn-direction" onclick="showDialogDirection('41 Mạc Đỉnh Chi, P. Đakao, Quan 1, Ho Chi Minh', '10.786191,106.697818', 5); return false;">Direction</a></div>
            </div>
            
            <div class="row" id="row6">
                <div class="col stt">6</div>
                <div class="col type">
                
                PGD Nguyễn Thái Bình</div>
                <div class="col address">176 – 178 Ký Con, P. Nguyễn Thái Bình, Quan 1, Ho Chi Minh</div>
                <div class="col district">1</div>
                <div class="col tel-fax">(028) 3915 1310<br>(028) 3915 1311</div>
                <div class="col hours">  07:30 – 16:30</div>
                <div class="col control"><a href="#" title="Direction" class="btn-direction" onclick="showDialogDirection('176 – 178 Ký Con, P. Nguyễn Thái Bình, Quan 1, Ho Chi Minh', '10.768917,106.696863', 6); return false;">Direction</a></div>
            </div>
            
            <div class="row" id="row7">
                <div class="col stt">7</div>
                <div class="col type">
                
                PGD Bến Chương Dương</div>
                <div class="col address">328 Võ Văn Kiệt, phường Cô Giang, Quận 1, Tp.HCM</div>
                <div class="col district">1</div>
                <div class="col tel-fax">(028) 3837 0586<br>(028) 3837 0584</div>
                <div class="col hours">   7h30-16h30</div>
                <div class="col control"><a href="#" title="Direction" class="btn-direction" onclick="showDialogDirection('328 Võ Văn Kiệt, phường Cô Giang, Quận 1, Tp.HCM', '10.76161,106.695998', 7); return false;">Direction</a></div>
            </div>
            
            <div class="row" id="row8">
                <div class="col stt">8</div>
                <div class="col type">
                
                PGD Trần Khắc Chân</div>
                <div class="col address">48-50 Nguyễn Hữu Cầu, P.Tân Định, Q.1, TP.HCM</div>
                <div class="col district">1</div>
                <div class="col tel-fax">(028) 3820 9990<br>(028) 3526 7738</div>
                <div class="col hours"> 07:30 -16:30</div>
                <div class="col control"><a href="#" title="Direction" class="btn-direction" onclick="showDialogDirection('48-50 Nguyễn Hữu Cầu, P.Tân Định, Q.1, TP.HCM', '10.790724, 106.690976', 8); return false;">Direction</a></div>
            </div>
            
            <div class="row" id="row9">
                <div class="col stt">9</div>
                <div class="col type">
                
                PGD Cống Quỳnh</div>
                <div class="col address">106  108 Cống Quỳnh, P. Nguyễn Cư Trinh, Q.1</div>
                <div class="col district">1</div>
                <div class="col tel-fax">(028) 38385464<br>(028) 3925 6645</div>
                <div class="col hours"> 07:30 -16:30</div>
                <div class="col control"><a href="#" title="Direction" class="btn-direction" onclick="showDialogDirection('106  108 Cống Quỳnh, P. Nguyễn Cư Trinh, Q.1', '10.764772,106.687505', 9); return false;">Direction</a></div>
            </div>
            
            <div class="row" id="row10">
                <div class="col stt">10</div>
                <div class="col type">
                
                CN Bến Thành</div>
                <div class="col address">96 Lý Tự Trọng, P. Bến Thành, Quan 1, Ho Chi Minh</div>
                <div class="col district">1</div>
                <div class="col tel-fax">(028) 3825 7949<br>(028) 3825 7950</div>
                <div class="col hours"> 07:30-16:30</div>
                <div class="col control"><a href="#" title="Direction" class="btn-direction" onclick="showDialogDirection('96 Lý Tự Trọng, P. Bến Thành, Quan 1, Ho Chi Minh', '10.774379, 106.697395', 10); return false;">Direction</a></div>
            </div>
            
            <div class="row" id="row11">
                <div class="col stt">11</div>
                <div class="col type">
                
                PGD Tân Định   </div>
                <div class="col address">261 Trần Quang Khải, Phường Tân Định, Quận 1, TP.HCM</div>
                <div class="col district">1</div>
                <div class="col tel-fax">(028) 3848 0520<br></div>
                <div class="col hours"> 07:30 - 16:30</div>
                <div class="col control"><a href="#" title="Direction" class="btn-direction" onclick="showDialogDirection('261 Trần Quang Khải, Phường Tân Định, Quận 1, TP.HCM', '10.791284, 106.688080', 11); return false;">Direction</a></div>
            </div>
            
            <div class="row" id="row12">
                <div class="col stt">12</div>
                <div class="col type">
                
                PGD Nguyễn Du</div>
                <div class="col address">Tầng hầm 1, tầng trệt, tầng lửng và tầng 2 tòa nhà 480 đường Nguyễn Thị Minh Khai, Phường 2, Quận 3, TP.Hồ Chí Minh</div>
                <div class="col district">1</div>
                <div class="col tel-fax">(028) 35218626<br>(028) 35218627</div>
                <div class="col hours"> 07:30 -16:30</div>
                <div class="col control"><a href="#" title="Direction" class="btn-direction" onclick="showDialogDirection('Tầng hầm 1, tầng trệt, tầng lửng và tầng 2 tòa nhà 480 đường Nguyễn Thị Minh Khai, Phường 2, Quận 3, TP.Hồ Chí Minh', '10.777328,106.698459', 12); return false;">Direction</a></div>
            </div>
            
        </div>

Does this answer your question? How to get text from an element , when getText fails — PDHide, Commented Jul 13, 2020 at 15:19
@PDHide No, it doesn't. I tried getAttribute('textContent') at both the table level as well as the table row level, but it gives the error ElementNotInteractableException. — Kristada673, Commented Jul 15, 2020 at 2:47

Millie Anne Volante · Accepted Answer · 2020-07-14 01:52:27Z

1

Your problem might be sames as https://stackoverflow.com/questions/20888592/gettext-method-of-selenium-chrome-driver-sometimes-returns-an-empty-string

But upon initial checking on the attributes of your element, I would suggest to use "outerText" attribute. You can check your element's properties in developer's tool Console by typing $x("your_xpath").

answered Jul 14, 2020 at 1:52

Millie Anne Volante

1063 bronze badges

1

Where do I use this outerText? Because I tried using them on the table, doesn't work. I tried using them on each table row, doesn't work. I tried using them on each element in a table row, doesn't work either. So, if you don't mind, could you please test the code yourself and edit your answer accordingly?
– Kristada673
Commented Jul 15, 2020 at 2:41
Yes, I tested it. if you do it by table or row, it will return even the "\n or \t" and also the button label which is included. Performance wise, it's faster but you still need to parse it. If you do it by element, you're more controlled on which data to get. I tried this code: print (element.find_element_by_css_selector('div.col.type').get_attribute("outerText"))
– Millie Anne Volante
Commented Jul 15, 2020 at 10:05

Add a comment |

Stack Exchange Network

Unable to extract table content from webpage using selenium

1 Answer 1

Your Answer

Not the answer you're looking for? Browse other questions tagged
selenium-webdriver
python
or ask your own question.

Linked

Hot Network Questions

Unable to extract table content from webpage using selenium

1 Answer 1

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged selenium-webdriverpython or ask your own question.

Linked

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
selenium-webdriver
python
or ask your own question.