I'm struggling with the following code using Selenium 3.141.59 and Java 11:

public void run() {

    ChromeOptions options = new ChromeOptions();


    ChromeDriver driver = new ChromeDriver(options);

    List<Article> articleList = new ArrayList<>();

     * Need to improve it somehow, I need around 300 (as it should work on the website) instead of 140-160 goods
    /*FluentWait<ChromeDriver> fluentWait = new FluentWait<>(driver)

    List<WebElement> liElements = fluentWait.until(webDriver -> driver.findElements(By.tagName("li")));*/

    /*WebDriverWait wait = new WebDriverWait(driver, 60);
    List<WebElement> liElements = wait.until(ExpectedConditions.presenceOfAllElementsLocatedBy(By.tagName("li")));*/

    /*List<WebElement> liElements = (List<WebElement>) ((JavascriptExecutor) driver)
            .executeScript("return document.getElementsByTagName('li')");*/

    driver.manage().timeouts().implicitlyWait(60, TimeUnit.SECONDS);
    List<WebElement> liElements = driver.findElements(By.tagName("li"));

    for (WebElement element : liElements) {

         * Need somehow to improve this snippet because of incorrect working of getting price
        WebElement priceElement = element.findElement(
                By.cssSelector("div.new-arrivals > div.newest-for-you > ul > li"));

        // Assuming that the 'oriprice' is under a specific class, modify the selector accordingly
        String price = priceElement.findElement(
                By.cssSelector("body > div.new-arrivals > div.newest-for-you > ul > li:nth-child(1) > a > div.text > div.price"))

        //String price = element.findElement(By.cssSelector("div.price-box > span.price.notranslate")).getText();
        String title = element.findElement(By.cssSelector("p.title")).getText();
        String url = element.findElement(By.cssSelector("span.img > a")).getAttribute("href");
        String newUrl = url.replace("...", url);
        ArrayList<String> tabs = new ArrayList<>(driver.getWindowHandles());

        WebElement img = driver.findElement(By.id("landingImage"));
        String srcImg = img.getAttribute("src");

        articleList.add(new Article(srcImg, url, title, price));


    articleList.forEach(article -> feedAdd(article.getImg(), article.getTitle(), article.getUrl(), article.getPrice()));

And my problem is I could not extract/receive all loaded elements/goods on the provided website.

To clarify:

while I'm debugging my own code, I'm getting around 140-160 elements, but, in fact, there's 300 elements (50 rows x 6 goods per each row):

enter image description here

and accordingly:

enter image description here

What I have already tried to use to change my situation:

  1. WebDriverWait with ExpectedConditions:
WebDriverWait wait = new WebDriverWait(driver, 60);
List<WebElement> liElements = wait.until(ExpectedConditions.presenceOfAllElementsLocatedBy(By.tagName("li")));
  1. JavaScript Executor:
List<WebElement> liElements = (List<WebElement>) ((JavascriptExecutor) driver)
        .executeScript("return document.getElementsByTagName('li')");
  1. Implicit Wait:
driver.manage().timeouts().implicitlyWait(60, TimeUnit.SECONDS);
List<WebElement> liElements = driver.findElements(By.tagName("li"));

and it still doesn't work for me, I still receive only half of elements, just 160 instead of 300.

I'm looking for any ideas how to avoid it, if I find the way on my own, as always, I'll update my solution, I would really appreciate, if someone help with it if have time.

I suppose it could be improved with any cssSelector-s or xpath.

Important to say: the only question here is in DOM, because there are existing hidden-li elements, for this reason, this forces me use various delays, as I provided above.

DOM looks like:

enter image description here

Thank you in advance for your help.

1 Answer 1


Needed to add Scroll down the page, as in the following attached example below:

    // Scroll down the page
    for (int i = 0; i < 10; i++) {
        ((JavascriptExecutor) driver).executeScript("window.scrollBy(0, 2000);");
        try {
        } catch (InterruptedException e) {
            throw new RuntimeException(e);

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.