How to get all existing li-hidden-elements under ul tag using Selenium while loading JS?

Question

I'm struggling with the following code using Selenium 3.141.59 and Java 11:

@Override
public void run() {
    WebDriverManager.chromedriver().setup();

    ChromeOptions options = new ChromeOptions();
    //WebDriverManager.chromedriver().browserVersion("118.0.5993.118").setup();
    //WebDriverManager.chromedriver().browserVersion("114.0.5735.90").setup();

    /*options.addArguments("--headless");
    options.addArguments("--disable-gpu");
    options.addArguments("--disable-extensions");*/

    ChromeDriver driver = new ChromeDriver(options);
    driver.get("...");

    List<Article> articleList = new ArrayList<>();

    /**
     * Need to improve it somehow, I need around 300 (as it should work on the website) instead of 140-160 goods
     */
    /*FluentWait<ChromeDriver> fluentWait = new FluentWait<>(driver)
            .withTimeout(Duration.ofSeconds(60))
            .pollingEvery(Duration.ofMillis(500))
            .ignoring(NoSuchElementException.class);

    List<WebElement> liElements = fluentWait.until(webDriver -> driver.findElements(By.tagName("li")));*/

    /*WebDriverWait wait = new WebDriverWait(driver, 60);
    List<WebElement> liElements = wait.until(ExpectedConditions.presenceOfAllElementsLocatedBy(By.tagName("li")));*/

    /*List<WebElement> liElements = (List<WebElement>) ((JavascriptExecutor) driver)
            .executeScript("return document.getElementsByTagName('li')");*/

    driver.manage().timeouts().implicitlyWait(60, TimeUnit.SECONDS);
    List<WebElement> liElements = driver.findElements(By.tagName("li"));


    for (WebElement element : liElements) {

        /**
         * Need somehow to improve this snippet because of incorrect working of getting price
         */
        WebElement priceElement = element.findElement(
                By.cssSelector("div.new-arrivals > div.newest-for-you > ul > li"));

        // Assuming that the 'oriprice' is under a specific class, modify the selector accordingly
        String price = priceElement.findElement(
                By.cssSelector("body > div.new-arrivals > div.newest-for-you > ul > li:nth-child(1) > a > div.text > div.price"))
                .getText();

        //String price = element.findElement(By.cssSelector("div.price-box > span.price.notranslate")).getText();
        String title = element.findElement(By.cssSelector("p.title")).getText();
        String url = element.findElement(By.cssSelector("span.img > a")).getAttribute("href");
        String newUrl = url.replace("...", url);
        ((JavascriptExecutor)driver).executeScript("window.open()");
        ArrayList<String> tabs = new ArrayList<>(driver.getWindowHandles());
        driver.switchTo().window(tabs.get(1));
        driver.get(newUrl);

        WebElement img = driver.findElement(By.id("landingImage"));
        String srcImg = img.getAttribute("src");

        articleList.add(new Article(srcImg, url, title, price));
        /*((JavascriptExecutor)driver).executeScript("window.close()");
        driver.switchTo().window(tabs.get(0));*/
    }

    driver.close();
    driver.quit();

    articleList.forEach(System.out::println);
    articleList.forEach(article -> feedAdd(article.getImg(), article.getTitle(), article.getUrl(), article.getPrice()));
}

And my problem is I could not extract/receive all loaded elements/goods on the provided website.

To clarify:

while I'm debugging my own code, I'm getting around 140-160 elements, but, in fact, there's 300 elements (50 rows x 6 goods per each row):

and accordingly:

What I have already tried to use to change my situation:

1. WebDriverWait with ExpectedConditions:

WebDriverWait wait = new WebDriverWait(driver, 60);
List<WebElement> liElements = wait.until(ExpectedConditions.presenceOfAllElementsLocatedBy(By.tagName("li")));

2. JavaScript Executor:

List<WebElement> liElements = (List<WebElement>) ((JavascriptExecutor) driver)
        .executeScript("return document.getElementsByTagName('li')");

3. Implicit Wait:

driver.manage().timeouts().implicitlyWait(60, TimeUnit.SECONDS);
List<WebElement> liElements = driver.findElements(By.tagName("li"));

and it still doesn't work for me, I still receive only half of elements, just 160 instead of 300.

I'm looking for any ideas how to avoid it, if I find the way on my own, as always, I'll update my solution, I would really appreciate, if someone help with it if have time.

I suppose it could be improved with any cssSelector-s or xpath.

Important to say: the only question here is in DOM, because there are existing hidden-li elements, for this reason, this forces me use various delays, as I provided above.

DOM looks like:

Thank you in advance for your help.

Answer 1

Needed to add Scroll down the page, as in the following attached example below:

    // Scroll down the page
    for (int i = 0; i < 10; i++) {
        ((JavascriptExecutor) driver).executeScript("window.scrollBy(0, 2000);");
        try {
            Thread.sleep(1500);
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        }
    }