3

I am using my linux box to try and periodically download my local skateboard park's webpage and search for whether there are free skate-school lessons available.

I'm currently falling at the first hurdle as I am only getting a partially rendered page. The crucial dynamic content is missing.

I'm using a bash command to get an html file:

$ phantomjs get_page.js [my local skateboard park's URL] > [a_file.html]

My save_page.js looks like this:

var system = require('system');
var page = require('webpage').create();

page.open(system.args[1], function()
{
    setTimeout(function(){
    console.log(page.content);
    phantom.exit();
}, 10000);
});

There is a ten second delay in there - but it also fails to load the whole page with a 3 or 5 second delay.

Improve this question
1
  • Whats the URL? You script seems to work with google.com here locally.
    – Niels van Reijmersdal
    Commented Aug 9, 2017 at 17:00

1 Answer 1

Reset to default
1

Your example script seems to work fine here.

Added some extra loggin:

var system = require('system');
var page = require('webpage').create();
console.log(system.args[1]);

page.open(system.args[1], function(status) {
    console.log(status);
    setTimeout(function(){
        console.log(page.content);
        phantom.exit();
    }   , 10000);
});

Output for node_modules/phantomjs/lib/phantom/bin/phantomjs get_page.js http://www.google.com/ > test.html:

http://www.google.com/
success
<!DOCTYPE html><html itemscope="" itemtype="http://schema.org/WebPage"...

What is your URL and output? if you add this extra console.logs?

Improve this answer
10
  • My HTML file is now blank apart from Status: success... URL:houseofvanslondon.com/book-skate-session Incidentally the content appears if I use another phantom.js example where the page gets rendered to .png but that is not what I am trying to acheive.
    – user137385
    Commented Aug 9, 2017 at 17:55
  • Weird the same code works here i get: houseofvanslondon.com/book-skate-session success <!DOCTYPE html><head><title>Book a Skate or BMX Session - House of Vans London</title>...
    – Niels van Reijmersdal
    Commented Aug 9, 2017 at 18:11
  • can u wget or curl the url from the command-line? maybe it is a network or proxy issue, dont really know. Your code seems good. I do see another hurdle after you fixed this one. That is that the skate dates are loaded into in iFrame. Maybe it is easier to write a selenium-webdriver script in javascript using the new chrome-driver --headless argument which has been added in Chrome version 59 (current version is 60) developers.google.com/web/updates/2017/04/headless-chrome
    – Niels van Reijmersdal
    Commented Aug 9, 2017 at 18:12
  • 1
    @KatePaulk No, PhantomJS is dead, the webkit engine hasnt been updated in years. Doesnt support cross-domain iframes sources. The maintainer has stepped down and he suggest to use Chrome headless instead :) groups.google.com/forum/#!topic/phantomjs/9aI5d-LDuNE
    – Niels van Reijmersdal
    Commented Aug 12, 2017 at 7:38
  • 1
    @NielsvanReijmersdal - well, that's a shame. PhantomJS was a useful tool. Hopefully Chrome headless will serve as a viable replacement.
    – Kate Paulk
    Commented Aug 14, 2017 at 12:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.