Want to bring more performance, speed, and scalability to your website? Or scale your sites for real-time services or message passing? Learn how, and get practical real-world tips in this exploration of Redis, part of a series on choosing the right data storage.
Steven Edouard and I (Rami Sayar) show you how to get up and running with Redis, a powerful key-value cache and store. In this tutorial series, you can check out a number of practical and advanced use cases for Redis as cache, queue, and publish/subscribe (pub/sub) tool, look at NoSQL and data structures, see how to create list sets and sorted sets in the cache, and much more. You can watch the course online on Microsoft Virtual Academy.
Level: Beginner to Intermediate.
By the end of this installment, you will:
- Learn about publish-subscribe
- Learn about how to setup Redis for pub/sub
- How to consume and produce messages with Node.js
- How to Scale a Messaging Application with Redis
Publish-subscribe or pub/sub for short is a messaging pattern where there are multiple publishers sending messages to multiple subscribers listening for messages on specific channels through server software. It is one of the most common message queue patterns. Many systems have been created for this purpose among them Redis, RabbitMQ, Azure MessageBus and more...
Redis, a high-performance data structure store, has added support for pub/sub on top of its existing functionality. Redis pub/sub is built on three commands:
SUBSCRIBE command lets a client subscribe to the channels passed to the command start receiving messages e.g.
SUBSCRIBE analytics will subscribe the client to the analytics channel.
UNSUBSCRIBE command will unsubscribe the client from the specific channels.
All messages received take the form of a three-element array. The first element is the kind of message: "subscribe", "unsubscribe" and "message". The "subscribe" message indicates if the client was successful in subscribing with the "unsubscribe" message indicating the opposite. The "message" message contains the payload sent to the channel with the
Redis developers suggest for you to create separate clients for publishers and subscribers.
Redis Client #1:
Redis Client #2:
Redis Client #1:
Redis pub/sub also has support for pattern matching when subscribing in clients. The
PSUBSCRIBE command takes glob-style patterns. For example, if you wanted to subscribe to all messages on the "news." channel, you can use
PSUBSCRIBE news.*. If you want to be more specific and only want to subscribe to art news, you can use
Using Redis Publish-Subscribe with Node.js
To use the Redis pub/sub mechanism with Node.js, the first thing you will want to do is select a native Redis client. The preferred Node client is the node_redis package.
The first thing you want to do is run
npm install redis to install the node package into your project.
Second, you want to require the node_redis project and create a Client for both the publisher and subscriber. Note, we are using the default client settings here.
Next, you want to set event handlers to the client to handle receiving messages and subscribe to the right channel.
Next, you want to set up the publisher to send messages on the channel.
That's all you need to use Redis pub/sub!
Scaling a Real-time Messaging Application with Pub/Sub
By itself it may be unclear why Redis' Pub/Sub feature is useful. However we can explain why this is so useful for scalability with a simple chat application. We'll be using Socket.IO and Node.js to do this application example but the architectural principal transcends languages.
We won't be going into much of the application details but if you want to learn more about the inner workings of Socket.IO and Node Express, check out one of our previous MVAs.
Creating the Application
First let's quickly build a chat application using the Yeoman generator. Yeoman allows us to quickly scaffold an app using a variety of technologies that work together.
Execute these commands:
Now run the application by executing:
Alright and if everything went well for you, your default web browser should have launched with the live chat app where you can send a message:
You'll notice the debug output when you send a message:
Scaling the Application
Notice that multiple clients can chat with each other when they connect to the same server. This is a high-level look at the application messaging architecture:
A client sends a message to everyone else connected to the server by dispatching a message to the server. Then the server broadcasts it to its connected clients.
Running locally we have 1 server, however when you roll out a service, you want more than just 1 instance of the server running. Say we add another Socket.IO server to scale the chat service:
Now, the way things are, Client D would never get any messages from clients A B or C.
Creating the Messaging Application
We can try this out for real by just copying the application we made to another directory:
Now change the port you run the application by modifying the top couple lines in the
Grunt.js file generated to run on port
public/js/app.js modify this first line of code to tell the browser to connect to the new port number we just set:
Notice, how we can't send a message from one window to the other:
Scaling With Redis
A great way to solve this is by using Redis' pub/sub feature to facilitate messages between our Socket.IO servers. With this here's how our architecture looks:
In this scenario, client D can now receive a message from client A because the backend servers are connected with Redis.
Let's actually implement this. Luckily the nice folks at Socket.IO has already created a Socket.IO Redis adapter just for this use.
In our case we need to make the following changes for each version of the application. In the real-world deployment we wouldn't have to do this because we would have a load balancer in place.
First, we have to install the Redis adapter and update the Socket.IO package. Execute this command to install the adapter:
npm install socket.io-redis --save
As of writing this the Yeoman Socket.IO generator uses an outdated version of the package. Change your
package.jsondependencies to look like this:
Now we need to tell the Node.js servers to use Redis. Change
server.js first few
require statements to use the Redis adapter:
Remember all these changes need to be made to both versions of your app to make this demo work.
Now run both applications by using two terminal windows and executing
grunt. Navigate one browser to
http://localhost:1338 and the other to
Notice now, we can send messages between the windows, even though they are both connected to two different Socket.IO servers:
Redis allows us to horizontally scale our service allowing multiple servers to publish and subscribe to messages so that users connected to different instances of the app can communicate.
We won't actually do this in our walkthrough but when you deploy to Azure or any other cloud service your final architecture would have a load balancer, which logically would look like a single server to your clients. Azure abstracts the load balancer from you and makes it easy to setup load balancing without having to do any low-level web server configuration.
Here's what this architecture would look like in an actual cloud deployment: