3

I am using my linux box to try and periodically download my local skateboard park's webpage and search for whether there are free skate-school lessons available.

I'm currently falling at the first hurdle as I am only getting a partially rendered page. The crucial dynamic content is missing.

I'm using a bash command to get an html file:

$ phantomjs get_page.js [my local skateboard park's URL] > [a_file.html]

My save_page.js looks like this:

var system = require('system');
var page = require('webpage').create();

page.open(system.args[1], function()
{
    setTimeout(function(){
    console.log(page.content);
    phantom.exit();
}, 10000);
});

There is a ten second delay in there - but it also fails to load the whole page with a 3 or 5 second delay.

  • Whats the URL? You script seems to work with google.com here locally. – Niels van Reijmersdal Aug 9 '17 at 17:00
1

Your example script seems to work fine here.

Added some extra loggin:

var system = require('system');
var page = require('webpage').create();
console.log(system.args[1]);

page.open(system.args[1], function(status) {
    console.log(status);
    setTimeout(function(){
        console.log(page.content);
        phantom.exit();
    }   , 10000);
});

Output for node_modules/phantomjs/lib/phantom/bin/phantomjs get_page.js http://www.google.com/ > test.html:

http://www.google.com/
success
<!DOCTYPE html><html itemscope="" itemtype="http://schema.org/WebPage"...

What is your URL and output? if you add this extra console.logs?

  • My HTML file is now blank apart from Status: success... URL:houseofvanslondon.com/book-skate-session Incidentally the content appears if I use another phantom.js example where the page gets rendered to .png but that is not what I am trying to acheive. – user137385 Aug 9 '17 at 17:55
  • Weird the same code works here i get: houseofvanslondon.com/book-skate-session success <!DOCTYPE html><head><title>Book a Skate or BMX Session - House of Vans London</title>... – Niels van Reijmersdal Aug 9 '17 at 18:11
  • can u wget or curl the url from the command-line? maybe it is a network or proxy issue, dont really know. Your code seems good. I do see another hurdle after you fixed this one. That is that the skate dates are loaded into in iFrame. Maybe it is easier to write a selenium-webdriver script in javascript using the new chrome-driver --headless argument which has been added in Chrome version 59 (current version is 60) developers.google.com/web/updates/2017/04/headless-chrome – Niels van Reijmersdal Aug 9 '17 at 18:12
  • 1
    @KatePaulk No, PhantomJS is dead, the webkit engine hasnt been updated in years. Doesnt support cross-domain iframes sources. The maintainer has stepped down and he suggest to use Chrome headless instead :) groups.google.com/forum/#!topic/phantomjs/9aI5d-LDuNE – Niels van Reijmersdal Aug 12 '17 at 7:38
  • 1
    @NielsvanReijmersdal - well, that's a shame. PhantomJS was a useful tool. Hopefully Chrome headless will serve as a viable replacement. – Kate Paulk Aug 14 '17 at 12:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.