Phantomjs to scrape webpage function not working

I am using phantomjs to learn how to scrape a webpage, so far I have developed the following code below.. I know that I am able to connect to the site, but I am unable to get data from the table at I on the right track?

My goal is to scrape data from the table on this site. I also understand that I need to use includeJs or injectJs to wait for the table to load else I would be scraping an empty html page. I am trying to put these concepts together, but am stuck for over 3 days now..please give some guidance..

var page = require('webpage').create();
console.log('Welcome to scraping...');'', function(status) {
    page.includeJs("", function() {
  if (status !== 'success') {
    console.log('Unable to access network');
  } else {
    var ua = page.evaluate(function() {
      var DATA = [];
      var res= document.getElementById((this).find('td')).eq(0).text();
      console.log(res); //**See output below for what is happening here**
            var $tds = $(this).find('td');

                rank:     $tds.eq(0).text(),
                company:  $tds.eq(1).text(),
                growth:   $tds.eq(2).text(),
                revenue:  $tds.eq(3).text(),
                industry: $tds.eq(4).text()

      console.log('Got into func');  
      //return document.getElementById('myagent').textContent;
    console.log('Got out of function');


The output I am getting is:

Welcome to scraping...

ReferenceError: Can't find variable: angular

TypeError: 'null' is not an object (evaluating 'document.getElementById(this.find('td')).eq')

phantomjs://webpage.evaluate():3 //what does this mean??



Got out of function




Recent Questions

Top Questions

Home Tags Terms of Service Privacy Policy DMCA Contact Us

©2020 All rights reserved.