How to massage or format html in order to parse with xmlstarlet?

Question

Just looking to parse html found "in the wild":

Very simple script:

curl -s http://www.foo.com  > 1.html
xmlstarlet fo  -H -R 2> /dev/null 1.html > 2.html
xmlstarlet sel -t -v "//header[@name='title']" -n 2.html > 3a.html
xmlstarlet sel -t -v '//table//tr//h3/span' -n 2> /dev/null 2.html > 3b.html

but xmlstarlet is barfing with:

     ^
2.html:324.11: Opening and ending tag mismatch: div line 241 and tr
</td></tr></table>&#13;
          ^
2.html:324.19: Opening and ending tag mismatch: div line 240 and table
</td></tr></table>&#13;
                  ^

what else can I do to massage the html for xmlstarlet?

Answer 1

I think you have to pass it the HTML spec or also it does not understand the format.

-N n="http://www.w3.org/1999/xhtml"

Also see this question for some examples: https://stackoverflow.com/questions/22021494/how-to-xmlstarlet-to-extract-html-data-by-id