1

This "Books" link shows that it has the following xpath:

/html/body/div/div/div/aside/div[2]/ul/li/a

Then follows the catalogue categories: Travel, Mystery, etc. Reading the xpath above I see three consecutive div, an aside, then another div before we start to get to the list.

Within that last division there seems some a priori knowledge is required that "Books" is the title for the list and not a list item itself.

To get the categories, I would want the list items? Currently, I'm getting all the links for the PageObjectModel as here.

1.) How can I get just the list items and the link for that item?

2.) How reasonable is my approach towards scraping data from this sample site?

code:

package dur.bounceme.net.SeleniumBase;

import java.util.List;
import java.util.logging.Logger;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.support.FindBy;
import org.openqa.selenium.support.PageFactory;

class WelcomePage {

    private static final Logger LOG = Logger.getLogger(App.class.getName());
    private WebDriver webDriver = null;

    @FindBy(partialLinkText = "Books")
    private WebElement books;

    @FindBy(tagName = "a")
    List<WebElement> links;

    private WelcomePage() {
    }

    WelcomePage(WebDriver webDriver) {
        this.webDriver = webDriver;
        PageFactory.initElements(this.webDriver, this);
        LOG.info(webDriver.getCurrentUrl());
    }

    static WelcomePage init(WebDriver webDriver) {
        return new WelcomePage(webDriver);
    }

    void populateCatalogue() {
        LOG.fine("start..");
        for (WebElement webElement : links) {
            LOG.info(webElement.getText());
        }
        LOG.fine("..done");
    }
}
2

"Books" is a single list item, which contains a new list with the categories. So you need to go deeper into that list like this:

//div[@class='side_categories']/ul/li/ul/li/a

In test code, it should look like something this:

@FindBy(xpath= "//div[@class='side_categories']/ul/li/ul/li/a")
List<WebElement> links;

Then to get the link and the item name:

for (WebElement webElement : links) {
    LOG.info(webElement.getText());
    LOG.info(webElement.getAttribute("href"));
}
  • how did you find the division name, side_categories? – Thufir Dec 23 '18 at 23:43
  • @Thufir I have used the inspect element (Ctrl + Shift + C) functionality of my browser. – Bence Kaulics Dec 24 '18 at 7:59
  • I know. It's just a bunch jibberish to me. I meant, how did you read the source code? Just knowledge of html, css, javascript? – Thufir Dec 26 '18 at 19:29
  • @Thufir Just HTML, if you inspect the "Books" element you can observe a div with class attribute of "side_categories", then you can copy the inner HTML to narrow down the source. There are many online xpath testers to experiment with. – Bence Kaulics Dec 26 '18 at 21:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.