How can I scrape pages with dynamic content using node.js?

I am trying to scrape a website but I don't get some of the elements, because these elements are dynamically created.

I use the cheerio in node.js and My code is below.

var request = require('request');
var cheerio = require('cheerio');
var url = "";

request(url, function (err, res, html) {
    var $ = cheerio.load(html);
    $('.listMain > li').each(function () {

This code returns empty response, because when the page is loaded, the <ul id="store_list" class="listMain"> is empty.

The content has not been appended yet.

How can I get these elements using node.js? How can I scrape pages with dynamic content?



Here you go;

var phantom = require('phantom');

phantom.create(function (ph) {
  ph.createPage(function (page) {
    var url = "";, function() {
      page.includeJs("", function() {
        page.evaluate(function() {
          $('.listMain > li').each(function () {
        }, function(){

Use the new npm module x-ray, with a pluggable web driver x-ray-phantom.

Examples in the pages above, but here's how to do dynamic scraping:

var phantom = require('x-ray-phantom');
var Xray = require('x-ray');

var x = Xray()

x('', 'title')(function(err, str) {
  if (err) return done(err);
  assert.equal('Google', str);

Easiest and reliable solution is to use puppeteer. As mentioned in which is suitable for both static + dynamic scrapping.

Only change the timeout in Browser.js, TimeoutSettings.js, Launcher.js 300000 to 3000000


Check out GoogleChrome/puppeteer

Headless Chrome Node API

It makes scraping pretty trivial. The following example will scrape the headline over at (assuming .npm-expansions remains)

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  await page.goto('');

  const textContent = await page.evaluate(() => {
    return document.querySelector('.npm-expansions').textContent

  console.log(textContent); /* No Problem Mate */


evaluate will allow for the inspection of the dynamic element as this will run scripts on the page.


Recent Questions

Top Questions

Home Tags Terms of Service Privacy Policy DMCA Contact Us

©2020 All rights reserved.