Tidy HTML with BaseX

Question

How do I tidy this HTML for baseX? Working from a basic example.

Looks like the HTML isn't valid, or at least baseX has apparent problems with parsing it:

thufir@dur:~/NetBeansProjects/helloWorldBaseX$ 
thufir@dur:~/NetBeansProjects/helloWorldBaseX$ gradle clean run

> Task :run FAILED
Jan 01, 2019 8:11:21 PM org.basex.examples.local.ScraperForHTML fetch
INFO: name  Resources  Size  Input Path  
---------------------------------

0 Databases.

Exception in thread "main" org.basex.core.BaseXException: "http://books.toscrape.com/" (Line 665): </a> found, </img> expected.
        at org.basex.core.Command.execute(Command.java:78)
        at org.basex.core.Command.execute(Command.java:90)
        at org.basex.examples.local.ScraperForHTML.fetch(ScraperForHTML.java:38)
        at org.basex.examples.local.App.helloWorld(App.java:24)
        at org.basex.examples.local.App.main(App.java:14)

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':run'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 2s
4 actionable tasks: 3 executed, 1 up-to-date
thufir@dur:~/NetBeansProjects/helloWorldBaseX$ 

code:

package org.basex.examples.local;

import java.net.MalformedURLException;
import java.net.URL;
import java.util.Iterator;
import java.util.Properties;
import java.util.logging.Logger;
import org.basex.core.BaseXException;
import org.basex.core.Context;
import org.basex.core.Databases;
import org.basex.core.cmd.CreateDB;
import org.basex.core.cmd.List;
import org.basex.core.cmd.Set;
import org.basex.util.list.StringList;

public class ScraperForHTML implements Scraper{

    private static final Logger LOG = Logger.getLogger(App.class.getName());
    private Properties properties = new Properties();

    private ScraperForHTML() {
    }

    public ScraperForHTML(Properties properties) {
        this.properties = properties;
        LOG.fine(properties.toString());
    }

    @Override
    public void fetch() throws BaseXException, MalformedURLException   {
        URL url = new URL(properties.getProperty("htmlURL"));
        String databaseName = properties.getProperty("databaseName");

        Context context = new Context();
        LOG.info(new List().execute(context));

        new Set("parser", "html").execute(context);
        new CreateDB(databaseName, url.toString()).execute(context);


        Databases databases = context.databases();
        StringList stringListOfDatabases = databases.listDBs();
        String currentDatabaseName = null;

        Iterator<String> databaseIterator = stringListOfDatabases.iterator();

        while (databaseIterator.hasNext()) {
            currentDatabaseName=databaseIterator.next();
            LOG.info(currentDatabaseName);
            //not quite sure how to query a database...
        }


      //  new DropDB(databaseName).execute(context);
        context.close();
    }


}

Notably, this XQuery seems to work fine:

thufir@dur:~/basex$ 
thufir@dur:~/basex$ cat html_fetch_parse.xq 


fetch:xml("http://books.toscrape.com/", map {
  'parser': 'html',
  'htmlparser': map { 'html': false(), 'nodefaults': true() }
})

thufir@dur:~/basex$ 

Answer 1

Looks like you are missing TagSoup in your classpath. As it is stated here you should have one to be able to successfully import html format. Since html is often not a well-formed XML, it should be pre-processed before importing.

When one does not have TagSoup the input format is considered as XML and has to be well-formed. Your example is missing closing </img> tag which makes importing without TagSoup not possible.