Some Help Scraping A Page In Java

I need to scrape a web page using Java and I've read that regex is a pretty inefficient way of doing it and one should put it into a DOM Document to navigate it. I've tried reading

Solution 1:

You can try jsoup: Java HTML Parser. It is an excellent library with good sample codes.

Solution 2:

1. Transform the web page you are trying to scrap into an XHTML document. There are several options to do this with Java, such as JTidy and HTMLCleaner. These tools will also automatically fix malformed HTML (e.g., close unclosed tags). Both work very well, but I prefer JTidy because it integrates better with Java's DOM API;
2. Extract required information using XPath expressions.

Here is a working example using JTidy and the Web Page you provided, used to extract all file names from the table.

publicstaticvoidmain(String[] args)throws Exception {
    // Create a new JTidy instance and set optionsTidytidy=newTidy();
    tidy.setXHTML(true); 

    // Parse an HTML page into a DOM documentURLurl=newURL("http://www.cs.grinnell.edu/~walker/fluency-book/labs/sample-table.html");        
    Documentdoc= tidy.parseDOM(url.openStream(), System.out);

    // Use XPath to obtain whatever you want from the (X)HTMLXPathxpath= XPathFactory.newInstance().newXPath();
    XPathExpressionexpr= xpath.compile("//td[@valign = 'top']/a/text()");
    NodeListnodes= (NodeList)expr.evaluate(doc, XPathConstants.NODESET);
    List<String> filenames = newArrayList<String>();
    for (inti=0; i < nodes.getLength(); i++) {
        filenames.add(nodes.item(i).getNodeValue()); 
    }

    System.out.println(filenames);
}

The result will be [Integer Processing:, Image Processing:, A Photo Album:, Run-time Experiments:, More Run-time Experiments:] as expected.

Another cool tool that you can use is Web Harvest. It basically does everything I did above but using an XML file to configure the extraction pipeline.

Solution 3:

Regex is definitely the way to go. Building a DOM is overly complicated and itself requires a lot of text parsing.

Solution 4:

If all you are doing is scraping a table into a datafile, regex will be just fine, and may be even better than using a DOM document. DOM documents will use up a lot of memory (especially for really large data tables) so you probably want a SAX parser for large documents.