JavaScript leaking memory (Node.js/Restify/MongoDB)

Update 4: By instantiating the restify client (see controllers/messages.js) outside of the function and calling global.gc() after every request it seems the memory growth rate has been reduced a lot (~500KB per 10secs). Yet, the memory usage is still constantly growing.

Update3: Came across this post: https://journal.paul.querna.org/articles/2011/04/05/openssl-memory-use/

It might be worth noting that I'm using HTTPS with Restify.

Update 2: Updated the code below to the current state. I've tried swapping out Restify with Express. Sadly this didn't make any difference. It seems that the api call at the end of the chain (restify -> mongodb -> external api) causes everything to retain to memory.

Update 1: I have replaced Mongoose with the standard MongoDB driver. Memory usage seems to grow less fast, yet the leak remains..


I've been working on trying to locate this leak for a couple of days now.

I'm running an API using Restify and Mongoose and for every API call I do at least one MongoDB lookup. I've got about 1-2k users that hit the API multiple times in a day.

What I have tried

  • I've isolated my code to just using Restify and used ApacheBench to fire a huge amount of requests (100k+). The memory usage stays around 60MB during the test.
  • I've isolated my code to just using Restify and Mongoose and tested it the same way as above. Memory usage stays around 80MB.
  • I've tested the full production code locally using ApacheBench. Memory usage stays around 80MB.
  • I've automatically dumped the heap on intervals. The biggest heap dump I had was 400MB. All I can see that there are tons of Strings and Arrays but I cannot clearly see a pattern in it.

So, what could be wrong?

I've done the above tests using just one API user. This means that Mongoose only grabs the same document over and over. The difference with production is that a lot of different users hit the API meaning mongoose gets a lot of different documents.

When I start the nodejs server the memory quickly grows to 100MB-200MB. It eventually stabilizes around 500MB. Could this mean that it leaks memory for every user? Once every user has visited the API it will stabilize?

I've included my code below which outlines the general structure of my API. I would love to know if there's a critical mistake in my code or any other approach to finding out what is causing the high memory usage.

Code

app.js

var restify = require('restify');
var MongoClient = require('mongodb').MongoClient;

// ... setup restify server and mongodb

require('./api/message')(server, db);

api/message.js

module.exports = function(server, db) {

    // Controllers used for retrieving accounts via MongoDB and communication with an external api
    var accountController = require('../controllers/accounts')(db);        
    var messageController = require('../controllers/messages')();

    // Restify bind to put
    server.put('/api/message', function(req, res, next) {
        // Token from body
        var token = req.body.token;

        // Get account by token
        accountController.getAccount(token, function(error, account) {

            // Send a message using external API
            messageController.sendMessage(token, account.email, function() {
                res.send(201, {});
                return next();
            });
        });
    });
};

controllers/accounts.js

module.exports = function(db) {

    // Gets account by a token
    function getAccount(token, callback) {
        var ObjectID = require('mongodb').ObjectID;

        var collection = db.collection('accounts');

        collection.findOne({
            token: token
        }, function(error, account) {

            if (error) {
                return callback(error);
            }

            if (account) {
                return callback('', account);
            }

            return callback('Account not found');
        });
    }
};

controllers/messages.js

module.exports = function() {

    function sendMessage(token, email, callback) {

        // Get a token used for external API
        getAccessToken(function() {}

            // ... Setup client

            // Do POST
            client.post('/external_api', values, function(err, req, res, obj) {
                return callback();
            });

        });
    }

    return {
        sendMessage: sendMessage
    };
};

Heap snapshot of suspected leak enter image description here

Answers:

Answer

Might be a bug in getters, I got it when using virtuals or getters for mongoose schema https://github.com/LearnBoost/mongoose/issues/1565

Answer

It's actually normal to only see string and arrays, as most programs are largely based on them. The profiler that allow sorting by total object count are therefore not of much use as they many times give the same results for many different programs.

A better way to use the memory profiling of chrome is to take one snapshot for example after one user calls an API, and then a second heap snapshot after a second user called the API.

The profiler gives the possibility to compare two snapshots and see what is the difference between one and the other (see this tutorial), this will help understand why the memory grew in an unexpected way.

Objects are retained in memory because there is still a reference to them that prevents the object from being garbage collected.

So another way to try to use the profiler to find memory leaks is to look for an object that you believe should not be there and see what is it's retaining paths, and see if there are any unexpected paths.

Answer

Not sure whether this helps, but could you try to remove unnecessary returns?

api/message.js

        // Send a message using external API
        messageController.sendMessage(token, account.email, function() {
            res.send(201, {});
            next(); // remove 'return'
        });

controllers/accounts.js

module.exports = function(db) {

    // Gets account by a token
    function getAccount(token, callback) {
        var ObjectID = require('mongodb').ObjectID;

        var collection = db.collection('accounts');

        collection.findOne({
            token: token
        }, function(error, account) {

            if (error) {
                callback(error); // remove 'return'
            } else if (account) {
                callback('', account); // remove 'return'
            } else {
                callback('Account not found'); // remove 'return'
            }
        });
    }

    return { // I guess you missed to copy this onto the question.
        getAccount: getAccount
    };
};

controllers/messages.js

            // Do POST
            client.post('/external_api', values, function(err, req, res, obj) {
                callback(); // remove 'return'
            });
Answer

Your issue is in the getAccount mixed with how GC work's.

When you chain lots of function the GC only clears one at a time and the older something is on memory the less chances it has of being collected so on your get account you need at least that I can count 6 calls to global.gc() or auto executes before it can be collected by this time the GC assumes its something that it probably wont collect so it doesn't check it anyway.

collection{
   findOne{
      function(error, account){
         callback('', account)
            sendMessage(...)
               getAccessToken(){
                  Post
               }
            }
         }
      }
   }
}

as suggested by Gene remove this chaining.

PS: This is just a representation of how the GC works and depends on Implementation but you get the point.

Tags

Recent Questions

Top Questions

Home Tags Terms of Service Privacy Policy DMCA Contact Us

©2020 All rights reserved.