Optimizing Node.js Application Concurrency

English — 日本語に切り替える

Node has a limited ability to scale to different container sizes. It’s single-threaded, so it can’t automatically take advantage of additional CPU cores.

Heroku Enterprise customers with Premier or Signature Success Plans can request in-depth guidance on this topic from the Customer Solutions Architecture (CSA) team. Learn more about Expert Coaching Sessions here or contact your Salesforce account executive.

Node versions < 12 cannot automatically take advantage of additional memory. Since Node is built on top of V8, which has a hard memory limit of about 1.5 GB, the process will allocate the same.

Instead, Node.js apps must fork multiple processes to maximize their available resources. This is called “clustering,” and is supported by the Node.js Cluster API. You can invoke the Cluster API directly in your app, or you can use one of many abstractions over the API. Here, we’ll use throng.

With Cluster, you can optimize your app’s performance across various dyno types. The Heroku Node.js buildpack provides environment variables to help.

Enabling concurrency in your app

We recommend that all applications support clustering. Concurrency, even if you don’t anticipate running more than a single process, grants you greater control and flexibility over your app’s performance in the future. Let’s take a look at an example.

First, we determine how many processes we should cluster:

var WORKERS = process.env.WEB_CONCURRENCY || 1;

Second, we define a start function that will be the entry point for each newly clustered process:

function start() {
  // ...
}

Finally, we use throng to cluster the app into multiple processes. We specify a lifetime of Infinity to tell throng that, if a worker dies, it should be respawned – so we will always have WORKERS processes running:

throng({
  workers: WORKERS,
  lifetime: Infinity
}, start);

Testing locally

With that implemented, we can observe the cluster:

$ npm start

> [email protected] start example-concurrency
> node server.js

Listening on 3000

$ WEB_CONCURRENCY=4

$ npm start

> [email protected] start example-concurrency
> node server.js

Listening on 3000
Listening on 3000
Listening on 3000
Listening on 3000

Tuning the concurrency level

Each app has unique memory, CPU, and I/O requirements, so there’s no such thing as a one-size-fits-all scaling solution. The Heroku buildpack provides reasonable defaults through two environment variables: WEB_MEMORY and WEB_CONCURRENCY. Both of these can be overridden to fit your specific application.

• WEB_MEMORY specifies, in MB, the expected memory requirements of your application’s processes. It defaults to 512MB.

• WEB_CONCURRENCY specifies the recommended number of processes to cluster for your application. It’s essentially MEMORY_AVAILABLE / WEB_MEMORY.

Read more about configuring your application’s memory use when clustering

Defaults:

Common Runtime

Dyno TypeNumber of Cluster workers

eco, basic, standard-1×1
standard-2×2
performance-M5
performance-L28

Private Spaces

Dyno TypeNumber of Cluster workers

private-S2
private-M5
private-L28

For Performance-L dynos, applications will fare well with the 28 workers suggested for its 14 GB of memory. However, it is prudent to test an application to see whether it can in fact support so many concurrent processes.

These are reasonable defaults for most apps. In most cases, clustering more than one worker on a Standard-1x dyno will hurt, rather than help performance. However, feel free to try any combination of WEB_CONCURRENCY with any dyno size to see what works best for your workload.

Decreasing the WEB_MEMORY will increase WEB_CONCURRENCY. Increasing WEB_MEMORY will, similarly, reduce concurrency. When the size of your dyno changes, WEB_CONCURRENCY will be recalculated automatically to fill available memory.

You can also set WEB_CONCURRENCY directly, but this will prevent your app from automatically re-clustering when you change dyno sizes.

See it in action

To log Node concurrency settings on startup, set the LOG_CONCURRENCY config var:

$ heroku config:set LOG_CONCURRENCY=true

Once you’ve deployed a cluster-able app to Heroku, you can tail its log to observe it scaling into different container sizes:

$ heroku logs --tail

$ heroku scale web=1:standard-1x

heroku[api]: Resize web to standard-1x
heroku[api]: Scale to web=1
heroku[web.1]: State changed from up to starting
heroku[web.1]: State changed from up to starting
heroku[web.1]: Starting process with command `npm start`
app[web.1]: Detected 512 MB available memory, 512 MB limit per process (WEB_MEMORY)
app[web.1]: Recommending WEB_CONCURRENCY=1
heroku[web.1]: Stopping all processes with SIGTERM
app[web.1]:
app[web.1]: > [email protected] start /app
app[web.1]: > node server.js
app[web.1]:
app[web.1]: Listening on 51077
heroku[web.1]: State changed from starting to up

$ heroku scale web=1:performance-l

heroku[api]: Resize web to performance-l
heroku[api]: Scale to web=1
heroku[web.1]: State changed from up to starting
heroku[web.1]: Starting process with command `npm start`
app[web.1]: Recommending WEB_CONCURRENCY=12
app[web.1]: Detected 6144 MB available memory, 512 MB limit per process (WEB_MEMORY)
heroku[web.1]: Stopping all processes with SIGTERM
app[web.1]:
app[web.1]: > [email protected] start /app
app[web.1]: > node server.js
app[web.1]:
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
app[web.1]: Listening on 50092
heroku[web.1]: Process exited with status 143
heroku[web.1]: State changed from starting to up