This demo is based on docker and spins up the following containers:
- Redis which is a popular open source in-memory database. We create 1 master & 2 slave containers
- Redis Sentinel which is used to provide HA and clustering for Redis
- and of course NGINX Plus which will be used as a TCP load balancer for the Redis nodes
The demo is based off the work described in this blog post: [Load Balancing Redis with NGINX Plus](Add Link later)
The following software needs to be installed:
- Docker for Mac if you are running this locally on your MAC OR docker-compose if you are running this on a linux VM
- redis-cli on the Docker host
-
Clone demo repo
$ git clone https://github.com/nginxinc/NGINX-Demos.git
-
Copy
nginx-repo.key
andnginx-repo.crt
files for your account to~/NGINX-Demos/redis-demo/nginxplus/
-
Move into the demo directory
$ cd ~/NGINX-Demos/redis-demo
-
If you have run this demo previously or have any docker containers running, start with a clean slate by running
$ ./clean-containers.sh
-
Spin up Redis, Sentinel and NGINX Plus containers
$ docker-compose up -d
-
Now follow the steps under section 'Running the demo'
-
You should have a bunch of containers up and running now:
$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 578f9e662203 redisdemo_sentinel "sentinel-entrypoi..." 4 seconds ago Up 3 seconds 6379/tcp, 26379/tcp redisdemo_sentinel_1 7aeee69035e7 redis:3 "docker-entrypoint..." 5 seconds ago Up 4 seconds 0.0.0.0:63793->6379/tcp redis-slave2 77530c277fdd redis:3 "docker-entrypoint..." 5 seconds ago Up 4 seconds 0.0.0.0:63792->6379/tcp redis-slave1 4c06e6236587 redisdemo_nginxplus "nginx -g 'daemon ..." 6 seconds ago Up 5 seconds 0.0.0.0:6379->6379/tcp, 0.0.0.0:8080->8080/tcp nginxplus 0ab4d54bd353 redis:3 "docker-entrypoint..." 6 seconds ago Up 5 seconds 0.0.0.0:63791->6379/tcp redis-master
NGINX Plus is listening on TCP port 6379 on the docker host which is the standard Redis port and load balance the traffic across the 3 Redis containers (one Master & two Slave). One Redis Sentinel container which is used to set up Clustering and for providing High Availability.
-
Go to the NGINX Plus Monitoring dashboard by going to http:https://127.0.0.1:8080 in your favourite web browser if you used Docker for Mac. If you set this up on a Linux VM, replace 127.0.0.1 with the IP address of your Linux VM. You will see all 3 Redis containers under TCP/UDP Upstreams
-
Now run
redis-cli -h <DOCKER-HOST-IP>
and execute couple of different Redis commands. After executing each redis command, notice the number of 'Active' & 'Total' connections getting incremented in the Dashboard for each upstream Redis node in a round-robin manner. This is because of the default round-robin LB$ redis-cli -h 127.0.0.1 127.0.0.1:6379> info replication # Replication role:master connected_slaves:2 slave0:ip=172.17.0.2,port=6379,state=online,offset=901913,lag=0 slave1:ip=172.17.0.4,port=6379,state=online,offset=901778,lag=1 master_repl_offset:901913 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:2 repl_backlog_histlen:901912 127.0.0.1:6379> 127.0.0.1:6379> set foo abc (error) READONLY You can't write against a read only slave. 127.0.0.1:6379> set foo abc OK 127.0.0.1:6379> 127.0.0.1:6379> get foo "abc" 127.0.0.1:6379> config get maxclients 1) "maxclients" 2) "10000"
-
Notice above that the 'set foo abc' command fails once since it lands on the Redis slave which cannot handle SET. It succeeds on the 2nd attempt when it lands on the Redis master. You could do GET/SET splits by configuring 2 virtual servers listening on 2 different ports (one for GET & another for SET) & then proxy_pass to just the Redis master in case of SET and to all Redis nodes in case of GET