Load Balancing Node.js Application

In this article, we will cover Horizontally scaling of your node.js application on a single server. Basically, we will fork our main application process as many times as we have CPU cores, and load balance all requests to all processes. Thus we need a multi-core environment.

Load Balancing with PM2 cluster mode

PM2 is a production process manager for Node.js applications with a built-in load balancer. It allows you to keep applications alive forever, to reload them without the downtime and to facilitate common system admin tasks.

lets first install

$ npm install -g pm2

Let's start our application using pm2

$ pm2 start app.js -i max --name "Backend"

This command will run the app.js file on the cluster mode to the total no of core available on your server. 

app on cluster mode

For your application to work perfectly your app needs to be stateless, meaning no local data is stored in the process, for example, sessions/WebSocket connections, session-memory and related.

If you have a RESTful and follows REST philosophy, your application is already stateless and should function correctly. But you could still encounter race condition on the database.

One of the most common reasons to add state to REST is for authentication. For an efficient approach to stateless authentication, you can make use of JSON Web Tokens.

PM2 cluster mode for node application with socket.io

We need a stateless system to work on properly in the cluster mode. But socket is stateful (not stateless). so how can we achieve the load-balancing?

Let us introduce you to Redis. So what is Redis? Redis is an open-source (BSD licensed), in-memory data structure store, used as a database, cache and message broker.

Redis

Redis is an in-memory key-value store known for its flexibility, performance, and wide language support

Installing Redis

$ sudo apt install redis-server

We need to edit the config file so we can start/stop the redis with systemctl. Open the /etc/redis/redis.conf file and replace supervised no with supervised systemd.

$ sudo systemctl restart redis.service

you can connect to the Redis server using.

$ redis-cli

more on securing Redis: How To Install and Secure Redis on Ubuntu 18.04.

Using socket adapter socket.io-redis

Let's install socket.io-redis

$ npm install socket.io-redis

import it as redis.

Once socket is initalized use redis adapter

1 2	io = socketIO(httpServer); io.adapter(redis({ host: 'localhost', port: 6379 }));

Before next step lets understand Polling vs WebSocket.

Pooling -> Client pull — client asking server for updates at certain regular intervals

WebSocket -> Server push — server is proactively pushing updates to the client (reverse of client pull)

For loadbalancing to work on pm2 cluster mode we need to use websocket as a transports for the socket. By default, a long-polling connection is established first, then upgraded to “better” transports (like WebSocket).

1 2 3 4 5

io = socketIO(httpServer, { transports: [ 'websocket' ] } );

in the client side: 

1 2 3

io = socketio(url, { transports: ['websocket'] });

java client

1 2 3

IO.Options opts = new IO.Options(); opts.transports = new String[]{WebSocket.NAME}; mSocket = IO.socket(url, opts);

Your application should function correctly by now.

But

Long Polling is HTTP based and it's basically request --> wait --> response and the wait isn't very long, as it can be dropped by load balancers on EOF or stale connections. Nevertheless, it's still useful when the websockets protocol (TCP based) isn't available and socket.io automatically re-establishes the connection for you. Notice that websockets is a relatively new protocol, ratified in 2011, so older browsers don't support it. Well, socket.io detects that and then resorts to long polling.

Thus using only websocket as transports on socket.io means that there is NO FALLBACK to long-polling when the websocket connection cannot be established, which is, in fact, one of the key features of Socket.IO.

The main reason to work with socket.io is to get the fallback feature of socket.io. With the above method, we are not getting the benefits of using socket.io over raw websocket. Even socket.io says and I quote,

In that case, you should maybe consider using raw WebSocket, or a thin wrapper like robust-websocket.

that -> using transports: [ 'websocket' ].

Above configuration should work. but still, there could be some issue with servers on reverse proxy. The following process about loadbalancing with nginx could solve the issue.

Loadbalancing node.js with nginx

Before load balancing let's create a simple server and run it.

install nginx

$ sudo apt install nginx

start nginx

$ sudo systemctl start nginx

create a record on /etc/hosts

1	127.0.2.1 bloggernepal.local

Now we can use bloggernepal.local as a domain pointed to this server. Since we are testing on local mechine the above step is required.

when you visit bloggernepal.local you will get the nginx default page Now create a conf file on /etc/nginx/conf.d lets create bloggernepal.conf

             

1 2 3 4 5 6 7 8 9 10 11 12

server { server_name bloggernepal.local; location / { proxy_pass http://localhost:3000; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; } }

Now lets test if there is any syntax error.

$ sudo nginx -t 

reload nginx

$sudo systemctl reload nginx

Now when you access bloggernepal.local you will get the node server. The node server in this step could be using just fork mode or cluster node, does not matter.

Now for loadbalancing with nginx, we will distribute the request to multiple instances of node through nginx. for that, we need to run the nodejs app in such a way that each process run and have different ports.

Let's again run the app on cluster mode. Please delete previous if you had followed the tutorials above with pm2 delete app (here app is name) or pm2 delete 0 1 2 3 ...

$ pm2 start app.js -i max --name "Backend"

This will run the app in cluster mode to the total no of core available on your server.

When you run an app in cluster mode with pm2 each worker will receive a value in process.env.NODE_APP_INSTANCE which goes from 0 to workers-1

Now we can use that value to listen in different ports.

          

1 2 3

var server = app.listen(300 + process.env.NODE_APP_INSTANCE, () => { console.log("Server running on port: ", server.address().port); });

Restart the application on pm2, Also on cluster mode, it will be zero downtime reload

$ pm2 reload Backend

Now lets configure the nginx. Open the /etc/nginx/conf.d/bloggernepal.conf and edit as follows.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

upstream backend { server localhost:3000; server localhost:3001; server localhost:3002; server localhost:3003; server localhost:3004; server localhost:3005; server localhost:3006; server localhost:3007; } server { server_name bloggernepal.local; location / { proxy_pass http://backend; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; } }

If your application is stateless, this setup would work perfectly for your application. There are several methods for load balancing. Round Robin is the default one.

• Round Robin
• Least Connections
• IP Hash
• Generic Hash
• Random

Load balancing node.js with socket.io with nginx

We can use simple the IP Hash methods for loadbalancing, which will point to the server that was connected based on the last request. for that simply add ip_hash; to upstream: 

1 2 3 4 5 6 7 8 9 10 11

upstream backend { ip_hash; server localhost:3000; server localhost:3001; server localhost:3002; server localhost:3003; server localhost:3004; server localhost:3005; server localhost:3006; server localhost:3007; }

Now we don't need to to use transports: [ 'websocket' ] for the socket.io, which will use the default setting of long-polling.

By default, a long-polling connection is established first, then upgraded to “better” transports (like WebSocket).

But Sticky sessions are a violation of twelve-factor. Twelve-factor app is a methodology for building distributed applications that run in the cloud and are delivered as a service.

It is a triangulation on ideal practices for app development, paying particular attention to the dynamics of the organic growth of an app over time, the dynamics of collaboration between developers working on the app’s codebase, and avoiding the cost of software erosion.

Conclusion 

We can use pm2 cluster mode for load balancing our node.js application, which simply spawn one process for each core of your machine. But if we are using socket.io, we need more configurations. We can use nginx load-balancing which will works fine for stateless application, but for stateful app we need to use the method which allow sticky season, here we use ip hash.