how to Iterate a List of links?

Question

This "Books" link shows that it has the following xpath:

/html/body/div/div/div/aside/div[2]/ul/li/a

Then follows the catalogue categories: Travel, Mystery, etc. Reading the xpath above I see three consecutive div, an aside, then another div before we start to get to the list.

Within that last division there seems some a priori knowledge is required that "Books" is the title for the list and not a list item itself.

To get the categories, I would want the list items? Currently, I'm getting all the links for the PageObjectModel as here.

1.) How can I get just the list items and the link for that item?

2.) How reasonable is my approach towards scraping data from this sample site?

code:

package dur.bounceme.net.SeleniumBase;

import java.util.List;
import java.util.logging.Logger;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.support.FindBy;
import org.openqa.selenium.support.PageFactory;

class WelcomePage {

    private static final Logger LOG = Logger.getLogger(App.class.getName());
    private WebDriver webDriver = null;

    @FindBy(partialLinkText = "Books")
    private WebElement books;

    @FindBy(tagName = "a")
    List<WebElement> links;

    private WelcomePage() {
    }

    WelcomePage(WebDriver webDriver) {
        this.webDriver = webDriver;
        PageFactory.initElements(this.webDriver, this);
        LOG.info(webDriver.getCurrentUrl());
    }

    static WelcomePage init(WebDriver webDriver) {
        return new WelcomePage(webDriver);
    }

    void populateCatalogue() {
        LOG.fine("start..");
        for (WebElement webElement : links) {
            LOG.info(webElement.getText());
        }
        LOG.fine("..done");
    }
}

Answer 1

"Books" is a single list item, which contains a new list with the categories. So you need to go deeper into that list like this:

//div[@class='side_categories']/ul/li/ul/li/a

In test code, it should look like something this:

@FindBy(xpath= "//div[@class='side_categories']/ul/li/ul/li/a")
List<WebElement> links;

Then to get the link and the item name:

for (WebElement webElement : links) {
    LOG.info(webElement.getText());
    LOG.info(webElement.getAttribute("href"));
}