How to unfreeze debug while getting hidden-li elements of DOM into the list using Selenium WebDriver?

Question

I need to get a sheet of li elements on the site. The problem is that by clicking on "View page code" the block collapsed to ul in which the li elements are empty.

Suppose the <li> elements are not immediately visible in the DOM it's possible that they are loaded dynamically or are hidden behind some interactions.

An example of it:

Clarification:

1. if you open the site, the block with the necessary li-elements is loaded by JS normally;

2. if you click on "View code", then I see all li:

   

   But when I try options by type, as for instance:

    WebDriverWait wait = new WebDriverWait(driver, 10); // 10 seconds timeout
   
    // Replace the following with the actual locator of the parent ul element
    By ulLocator = By.cssSelector("body > div.new-arrivals > div.newest-for-you > ul");
   
    // Wait for the ul element to be present in the DOM
    WebElement ulElement = wait.until(ExpectedConditions.presenceOfElementLocated(ulLocator));
   
    // Now, find the li elements within the ul
    List<WebElement> liElements = ulElement.findElements(By.tagName("li"));
   
    ... 
   
    for (WebElement element : liElements) { ... }
   

As far as I understood, in such cases, should be used explicit waits to ensure that the elements are present and visible before attempting to interact with them, but the example above still didn't help at all to resolve my situation.

or accordingly:

List<WebElement> liElements = driver.findElements(By.cssSelector("body > div.new-arrivals > div.newest-for-you > ul > li"));
         // Find all <li> elements under the <ul>
         //List<WebElement> liElements = ulElement.findElements(By.tagName("li"));
         //List<WebElement> liElements = ulElement.findElements(By.xpath(".//li"));
         //List<WebElement> liElements = ulElement.findElements(By.cssSelector("li"));
         /*List<WebElement> liElements = (List<WebElement>) ((JavascriptExecutor) driver)
                 .executeScript("return arguments[0].getElementsByTagName('li')", ulElement);*/

or:

// Assuming 'driver' is your WebDriver instance
        WebDriverWait wait = new WebDriverWait(driver, 30); // 10 seconds timeout

// Replace the following with the actual locator of the parent ul element
        By ulLocator = By.cssSelector("body > div.new-arrivals > div.newest-for-you > ul");

// Wait for the ul element to be present in the DOM
//WebElement ulElement = wait.until(ExpectedConditions.presenceOfElementLocated(ulLocator));

// Now, find the li elements within the ul
//List<WebElement> liElements = ulElement.findElements(By.tagName("li"));
//List<WebElement> liElements = ulElement.findElements(By.xpath( ".//li"));

// Wait for the ul element to be present in the DOM
WebElement ulElement = wait.until(ExpectedConditions.presenceOfElementLocated(ulLocator));

// Now, find the li elements within the ul using explicit wait
List<WebElement> liElements = wait.until(ExpectedConditions.visibilityOfNestedElementsLocatedBy(
                ulElement, By.tagName("li")));

My debug completely dies and does not go through the line before entering the loop to process the elements:

To clarify:

1. "View code" shows me li-elements in the browser
2. "View page code" doesn't show me li-elements in the browser.

Can I get the li elements, even though I can't expand the ul in the "View page code"? If yes, how can this be implemented?

About innerHtml element: I don't quite understand how it should fix the issue with hidden-li elements while going through debugging inside IDE.

Answer 1

Solved this problem after adding the following construction with FluentWait:

    FluentWait<ChromeDriver> fluentWait = new FluentWait<>(driver)
            .withTimeout(Duration.ofSeconds(60))
            .pollingEvery(Duration.ofMillis(500))
            .ignoring(NoSuchElementException.class);

    List<WebElement> liElements = fluentWait.until(webDriver -> driver.findElements(By.tagName("li")));