Scrape items from an HTML list

Question

When Catalog receives a fragment for a book category, it seems to my eyes, to be just list items. For space, I didn't include that output. How can I ensure that only categories which belong to the catalog are passed to the Category instances which Catalog creates?

debug output for the project:

thufir@dur:~/NetBeansProjects/com.toscrape.books.selenium$ 
thufir@dur:~/NetBeansProjects/com.toscrape.books.selenium$ gradle clean run

> Task :run
1546177894344   mozrunner::runner       INFO    Running command: "/usr/lib/firefox/firefox" "-marionette" "--headless" "-foreground" "-no-remote" "-profile" "/tmp/rust_mozprofile.64P2JcqFPQGb"
*** You are running in headless mode.
1546177894956   [email protected]     WARN    Loading extension '[email protected]': Reading manifest: Invalid host permission: resource://pdf.js/
1546177894956   [email protected]     WARN    Loading extension '[email protected]': Reading manifest: Invalid host permission: about:reader*
1546177898508   Marionette      INFO    Listening on port 41957
1546177898575   Marionette      WARN    TLS certificate errors will be ignored for this session
Dec 30, 2018 5:51:38 AM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
JavaScript warning: http://ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js, line 1: Using //@ to indicate sourceMappingURL pragmas is deprecated. Use //# instead
Dec 30, 2018 5:51:44 AM dur.bounceme.net.SeleniumBase.Catalog iterateCatalogElements
INFO: trying to iterate..
Dec 30, 2018 5:51:45 AM dur.bounceme.net.SeleniumBase.Catalog iterateCatalogElements
INFO: li
Dec 30, 2018 5:51:45 AM dur.bounceme.net.SeleniumBase.Catalog iterateCatalogElements
INFO: Books
Travel
...
Cultural
Erotica
Crime
Dec 30, 2018 5:51:45 AM dur.bounceme.net.SeleniumBase.Category <init>
INFO: new category..
Dec 30, 2018 5:51:45 AM dur.bounceme.net.SeleniumBase.Category <init>
INFO: 
            <a href="index.html">Home</a>

Dec 30, 2018 5:51:45 AM dur.bounceme.net.SeleniumBase.Catalog iterateCatalogElements
INFO: li
Dec 30, 2018 5:51:45 AM dur.bounceme.net.SeleniumBase.Catalog iterateCatalogElements
INFO: Historical Fiction
Dec 30, 2018 5:51:45 AM dur.bounceme.net.SeleniumBase.Category <init>
INFO: new category..
Dec 30, 2018 5:51:45 AM dur.bounceme.net.SeleniumBase.Category <init>
INFO: 
                        <a href="catalogue/category/books/travel_2/index.html">

                                Travel

                        </a>



..                        
Dec 30, 2018 5:51:46 AM dur.bounceme.net.SeleniumBase.Catalog iterateCatalogElements
INFO: li
Dec 30, 2018 5:51:46 AM dur.bounceme.net.SeleniumBase.Catalog iterateCatalogElements
INFO: Cultural
Dec 30, 2018 5:51:46 AM dur.bounceme.net.SeleniumBase.Category <init>
INFO: new category..
Dec 30, 2018 5:51:46 AM dur.bounceme.net.SeleniumBase.Category <init>
INFO: 
                        <a href="catalogue/category/books/health_47/index.html">

                                Health

                        </a>


Dec 30, 2018 5:51:46 AM dur.bounceme.net.SeleniumBase.Catalog iterateCatalogElements
INFO: li
Dec 30, 2018 5:51:46 AM dur.bounceme.net.SeleniumBase.Catalog iterateCatalogElements
INFO: A Light in the ...
£51.77
In stock
Add to basket
Dec 30, 2018 5:51:46 AM dur.bounceme.net.SeleniumBase.Category <init>
INFO: new category..
Dec 30, 2018 5:51:46 AM dur.bounceme.net.SeleniumBase.Category <init>
INFO: 
                        <a href="catalogue/category/books/erotica_50/index.html">

                                Erotica

                        </a>


Dec 30, 2018 5:51:46 AM dur.bounceme.net.SeleniumBase.Catalog iterateCatalogElements
INFO: li
Dec 30, 2018 5:51:46 AM dur.bounceme.net.SeleniumBase.Catalog iterateCatalogElements
INFO: Sharp Objects
£47.82
In stock
Add to basket
Dec 30, 2018 5:51:46 AM dur.bounceme.net.SeleniumBase.Category <init>
INFO: new category..
Dec 30, 2018 5:51:46 AM dur.bounceme.net.SeleniumBase.Category <init>
INFO: 






    <article class="product_pod">

            <div class="image_container">


..

Dec 30, 2018 5:51:46 AM dur.bounceme.net.SeleniumBase.Catalog iterateCatalogElements
INFO: li
Dec 30, 2018 5:51:46 AM dur.bounceme.net.SeleniumBase.Catalog iterateCatalogElements
INFO: The Dirty Little Secrets ...
£33.34
In stock
Add to basket
Dec 30, 2018 5:51:46 AM dur.bounceme.net.SeleniumBase.Category <init>
INFO: new category..
Dec 30, 2018 5:51:46 AM dur.bounceme.net.SeleniumBase.Category <init>
INFO: 




..


    <form>
        <button type="submit" class="btn btn-primary btn-block" data-loading-text="Adding...">Add to basket</button>
    </form>



            </div>

    </article>



BUILD SUCCESSFUL in 15s
4 actionable tasks: 4 executed
thufir@dur:~/NetBeansProjects/com.toscrape.books.selenium$ 
thufir@dur:~/NetBeansProjects/com.toscrape.books.selenium$

Too much is being sent to a Category -- it should only handle list items.

The Catalog class should only iterate the categories of the catalog container:

package dur.bounceme.net.SeleniumBase;

import java.util.List;
import java.util.ListIterator;
import java.util.logging.Logger;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.support.FindBy;
import org.openqa.selenium.support.PageFactory;

public class Catalog {

    private static final Logger LOG = Logger.getLogger(Catalog.class.getName());
    private WebElement container = null;
    private final WebDriver webDriver = null;

    @FindBy(xpath = "//li")
    List<WebElement> catalogElements = null;

    private Catalog() {
    }

    public Catalog(WebDriver webDriver, WebElement container) {
        this.container = container;
        LOG.fine(container.getAttribute("innerHTML"));
        PageFactory.initElements(webDriver, this);
    }

    public void iterateCatalogElements() {
        LOG.info("trying to iterate..");
        LOG.fine(catalogElements.toString());
        ListIterator<WebElement> li = catalogElements.listIterator();
        WebElement listItem = null;
        while (li.hasNext()) {
            listItem = li.next();
            LOG.info(li.next().getTagName());
            LOG.info(li.next().getText());
            Category c = new Category(webDriver, listItem);
        }
    }
}

The Category class, which only deals with one category:

package dur.bounceme.net.SeleniumBase;

import java.util.logging.Logger;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.support.PageFactory;

public class Category {

    private static final Logger LOG = Logger.getLogger(Category.class.getName());
    private WebElement container = null;
    private final WebDriver webdriver = null;

    private Category() {
    }

    public Category(WebDriver webDriver, WebElement container) {
        LOG.info("new category..");
        this.container = container;
        PageFactory.initElements(webDriver, container);
        LOG.info(container.getAttribute("innerHTML"));
    }

    public void iterateThisCategory() {
    }
}

While Category should confirm that the fragments it receives are of the valid form, the real problem is that the Catalog instance somehow is sending more than what's in the HTML list itself.

For clarity, and pardon the long output, only HTML list items from this list should be passed from Catalog to the Category instances (I only changed what was logged to the console):

thufir@dur:~/NetBeansProjects/com.toscrape.books.selenium$ 
thufir@dur:~/NetBeansProjects/com.toscrape.books.selenium$ gradle clean run

> Task :run
1546179316703   mozrunner::runner       INFO    Running command: "/usr/lib/firefox/firefox" "-marionette" "--headless" "-foreground" "-no-remote" "-profile" "/tmp/rust_mozprofile.a0uEgT6zm0Y4"
*** You are running in headless mode.
1546179317387   [email protected]     WARN    Loading extension '[email protected]': Reading manifest: Invalid host permission: resource://pdf.js/
1546179317392   [email protected]     WARN    Loading extension '[email protected]': Reading manifest: Invalid host permission: about:reader*
1546179321898   Marionette      INFO    Listening on port 46537
1546179321925   Marionette      WARN    TLS certificate errors will be ignored for this session
Dec 30, 2018 6:15:22 AM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
JavaScript warning: http://ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js, line 1: Using //@ to indicate sourceMappingURL pragmas is deprecated. Use //# instead
Dec 30, 2018 6:15:27 AM dur.bounceme.net.SeleniumBase.Catalog <init>
INFO: 


                    <li>
                        <a href="catalogue/category/books/travel_2/index.html">

                                Travel

                        </a>

                        </li>


                    <li>
                        <a href="catalogue/category/books/mystery_3/index.html">

                                Mystery

                        </a>

                        </li>



..



                    <li>
                        <a href="catalogue/category/books/erotica_50/index.html">

                                Erotica

                        </a>

                        </li>


                    <li>
                        <a href="catalogue/category/books/crime_51/index.html">

                                Crime

                        </a>

                        </li>



BUILD SUCCESSFUL in 15s
4 actionable tasks: 3 executed, 1 up-to-date
thufir@dur:~/NetBeansProjects/com.toscrape.books.selenium$

JAINAM · Accepted Answer · 2020-07-16 10:38:29Z

1

This kinda works:

Obviously, that's one result -- I would want to iterate the list for all results.

One angle is just to see what's possible from withing basex itself. I believe it can grab the html directly from the website. (I'll probably update the answer.)

edited Jul 16, 2020 at 10:38

JAINAM

1,8351 gold badge17 silver badges39 bronze badges

answered Dec 31, 2018 at 19:51

Thufir

3671 gold badge5 silver badges18 bronze badges

Add a comment |

Stack Exchange Network

Scrape items from an HTML list

1 Answer 1

Your Answer

Not the answer you're looking for? Browse other questions tagged
selenium-webdriver
java
xpath
page-objects
page-factory
or ask your own question.

Hot Network Questions

Scrape items from an HTML list

1 Answer 1

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged selenium-webdriverjavaxpathpage-objectspage-factory or ask your own question.

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
selenium-webdriver
java
xpath
page-objects
page-factory
or ask your own question.