Get a specific text value from an HTML table using python and selenium

Question

CROSS-POSTED here: https://stackoverflow.com/questions/71003815/get-a-specific-text-value-from-an-html-table-using-python-and-selenium

This website (https://egov.uscis.gov/processing-times/) shows the processing time for different immigration applications for different service centers. There are two drop-down menus; first is the Form (immigration application), and second is the service center. Once you select a value from these two menus and hit the "Get processing time" button, a table will be displayed which shows the details for different immigration categories/types (you have to scroll down a bit to be able to see the table)(the table has three column titles: "Estimated time range", "Form type", "Receipt date for a case inquiry"). I want to use selenium with python to automate printing out the current range for processing time for form I-765 in Texas Service Center for the category of "Based on a pending I-485 adjustment application [(c)(9)]", which is the sixth row in the table.

As of 2/5/2022, I want my code to print out a text as: "11.5 Months to 13.5 Months".

Here is my attempt so far, but it only prints an empty string.

It opens the website, selects the right Form and then the right service center, and hits the button. But it fails to fetch the text I want from the table.

What if I want to print out all the text values in the 6th row?

 from selenium import webdriver
 from selenium.webdriver.support.ui import Select
 import time

 browser = webdriver.Chrome('C:\ChromeDriver\chromedriver')
 browser.get('https://egov.uscis.gov/processing-times/')

 form_menu = browser.find_element_by_id('selectForm')
 select_form_menu = Select(form_menu)
 select_form = select_form_menu.select_by_visible_text('I-765 | Application for Employment Authorization')

 time.sleep(2)

 office_menu = browser.find_element_by_id('officeOrCenter')
 select_office_menu = Select(office_menu)
 select_office = select_office_menu.select_by_visible_text('Texas Service Center')

 time.sleep(2)

 search_btn = browser.find_element_by_id('getProcTimes')
 search_btn.click()

 time.sleep(2)

 table = browser.find_element_by_id('resultList')
 items = table.find_elements_by_id('est') #here I am trying to return a list of all the values in the first column.

 print(items[5].text) #my value of interest is the 6th element in the list (6th row in the table).

Answer 1

The Element is hidden as the table is collapsed. The text is not visible in UI. so webdriver doesnt show the text as .text will show only visible text.

While innerText will consider the the visibility style but of child elements only not on the element it self:

Eg:

console.log("outer.textContent:", outer.textContent);
console.log("outer.innerText:", outer.innerText);
console.log("inner.textContent:", inner.textContent);
console.log("inner.innerText:", inner.innerText);

<div id="outer" style="display: none;" >This is visible <span id="inner" >This is invisible</span></div>

https://stackoverflow.com/a/68809991/6793637

Here outer display is none but we still see the innerText . Because:

https://developer.mozilla.org/en-US/docs/Web/API/HTMLElement/innerText#value

Value

A DOMString representing the rendered text content of an element. If the element itself is not being rendered (e.g detached from the document or is hidden from view), the returned value is the same as the Node.textContent property.

Answer 2

So, it seems no one is interested in answering my question, I will go ahead and answer my own question, for future reference.

If I change

print(items[5].text)

to

print(items[5].get_attribute("innerText"))

I get the result I want.

I just don't know why??