By Pablo Gonzalez in Node.js — Jun 15, 2023

Redis 7 with TLS, NodeJS, Heroku, Docker Compose, and Bull

I'm writing this article because I spent too many days trying to solve this issue and want to help others.

I own HappySoup.io, a nodejs app hosted on Heroku, which uses Redis for session management and as a queue system using Bull.

What happened?

Heroku forced an upgrade of the Redis database to version 7, which requires TLS for its connections.

Here's the email I received

Your Redis database redis-octagonal-62067 on sfdc-happy-soup is running a >deprecated version (5.0.14) and will not be supported after 30 Jun, 2023.

⏰ Upgrade now to avoid a forced upgrade on 30 May, 2023. Forced upgrades >start a month before the EOL date.

Upgrading from 5.0.14 to 7.0.11 may break your app. Take action now!

7.0.11 %> enforces TLS connections, you will need to update your application to >use TLS connections, see Dev Center articles: Connecting to Heroku Redis and >Heroku Redis Version Upgrade.

The email had a link to this knowledge article

This explains that if you are using node-redis, all you have to do is add this code to your Redis client

const client = redis.createClient({
  url: process.env.REDIS_URL,
  tls: {
    rejectUnauthorized: false
  }
});

This leaves way too many questions unanswered, and I couldn't believe that adding that bit of code was all that was needed So I started my spiral down the rabbit hole of hopelessness.

Here are some of the questions I had about this upgrade:

Do I need to create my own certificate? and how do I make Heroku use it?

You don't need to create your own certificate when the app runs on Heroku, as in when it's running in the cloud.

Heroku will use a self-signed certificate and will instantiate the Redis server with that certificate.

But what about when the app runs locally on my PC?

Whether you need a certificate when using the app locally depends on your version of Redis.

I was using Redis 5 on Docker, so I didn't need a certificate. However, I upgraded my docker image to version 7 because I wanted to have the same infrastructure as when the app runs on the cloud, to avoid the typical "it works on my computer" problem.

So yes, if you are using Redis 7 locally, you'll need to provide a certificate...but where?

Where do I specify the certificate?

This is where I started spiraling. Do I specify the certificate when I instantiate the server, or when the client connects to it?

The client connects to the server using code similar to this

const client = redis.createClient({
  url: process.env.REDIS_URL,
  tls: {
    rejectUnauthorized: false
  }
});

You do NOT need to specify the certificate here.

Initially, I thought this was where I had to somehow specify the certificate because if you look at the node-redis documentation for this configuration (using TLS), they mention the use of a certificate like this:

createClient({
  socket: {
    tls: true,
    ca: '...',
    cert: '...'
  }
});

I don't know when you need this configuration, but I presume is when you want mutual TLS, i.e. you want the client also to provide a certificate.

This is not what we need for Heroku. For Heroku, we need a server to present a certificate, and Heroku takes care of that.

So really, when the app is running on Heroku, you don't need to do anything other than using tls.rejectUnauthorised = false in the Redis configuration object.

If you are using Bull

Well, despite using the above configuration, I was still getting this error when starting the app

Error accepting a client connection: error:1408F10B:SSL routines:ssl3_get_record:wrong version number

The reason was that Bull also connects to Redis, i.e, it's also a client. So Bull as well needs to ask Redis to provide a certificate.

Looking at the Bull documentation, they have an example of how you can pass the Redis configuration object to the queue constructor, like this

const audioQueue = new Queue('audio transcoding', { redis: { port: 6379, host: '127.0.0.1', password: 'foobared' } }); // Specify Redis connection using object

And this was the missing bit. My initial Bull config was only taking the Redis URL.

So I modified the constructor to take the entire Redis config object like this:

let redisConfig = require('../redisConfig');
let workQueue = new Queue(process.env.QUEUE_NAME, 
    {redis:redisConfig}
);

where my redisConfig looks like this

...
other code here
...

else{
    config.port = 6379;//default redis port
    config.host = '127.0.0.1';
    config.url = `redis://${config.host}:${config.port}`;
}

config.tls = {
    rejectUnauthorized: false
}

Creating the certificate to run the app locally

As I said, if you want to run the app locally, then you do need to provide a certificate.

I will explain later exactly where that certificate is provided. I had ChatGPT provide me with the steps to create the certificate.

It's very easy, just use this prompt:

I need to create a self-signed certificate for my redis server. The certificate needs to be signed by a CA (self-signed as well). Please give me detailed steps to create all the required certificates

Once I followed the steps, I ended up with the following files

Where do I use the certificate?

We go back to this question. Where is the certificate used!?

Well, it turns out it is meant to be used when you start the Redis server, as seen in the Redis documentation for TLS

If you have Redis installed on your computer, you typically boot it by running redis-server on the terminal.

The idea is that this command accepts a list of parameters, and is it in these parameters, you specify the CA and cert files.

But what if you are using Docker?

I was using Docker Compose to create a Redis instance like this

services:
  redis:
    image: 'redis:7.0.11'
   
   
  webapp:
    build:
      context: .
      dockerfile: ./docker/web/Dockerfile
    ports:

The problem is, how do I tell Docker to use those cert files that I created on my computer (the host computer, as its called)

Docker Compose Volumes

Again, thanks to ChatGPT, I learned there's a volumes property of Docker compose that you can use to move files between the host and the container, so I used this to move the certs over to the Redis container

services:
  redis:
    image: 'redis:7.0.11'
    volumes:
      - ./certs/redis-server.crt:/usr/local/etc/redis/redis-server.crt
      - ./certs/redis-server.key:/usr/local/etc/redis/redis-server.key
      - ./certs/ca.crt:/usr/local/etc/redis/ca.crt

Now, how do I tell Redis to use these certs?

Docker Compose Command

I didn't know exactly how Docker was booting the Redis instance, I just knew it worked.

I learned that the image comes with a default command that runs to boot it, which is just redis-server, but you can override this command so you can specify your own, so I wrote this:

services:
  redis:
    image: 'redis:7.0.11'
    volumes:
      - ./certs/redis-server.crt:/usr/local/etc/redis/redis-server.crt
      - ./certs/redis-server.key:/usr/local/etc/redis/redis-server.key
      - ./certs/ca.crt:/usr/local/etc/redis/ca.crt
    command: >
      redis-server
      --tls-port 6379
      --port 0
      --tls-cert-file /usr/local/etc/redis/redis-server.crt
      --tls-key-file /usr/local/etc/redis/redis-server.key
      --tls-ca-cert-file /usr/local/etc/redis/ca.crt
      --tls-auth-clients no
      --tls-replication yes

The command runs the redis-server command while passing the configuration options to read the CA files that I moved earlier with the volumes property.

Note that I also used tls-auth-clients and set it to no because I don't want the client to have to authenticate itself.

After all this, I finally got the app to work locally and on Heroku as well.

Conclusion

The biggest takeaway here is the dangers of using libraries and tools like a black box that you don't understand, for example:

I didn't know (or forgot) that Bull was using a separate connection to Redis, and that it needed to ask for a certificate
I didn't know how Docker Compose was booting the Redis server, and that I could overwrite that
The Heroku docs don't explain anything about how Heroku passes their own self-signed certificate
I didn't know that a CA certificate is just any other certificate, what makes it a CA certificate is that it is used to sign another certificate
etc

So the lesson learned is, moving forward, I will spend more time understanding any tools/framework/technique that I use instead of treating them like a black box that "just works."