Highlighting its role as an data-store in the enterprise, Redis offers two operational tools; clustering, available in Redis 3.0, and Redis Sentinel, for high availability management of Redis Instances. Both tools increased Redis use and reliability as a core IT infrastructure.
Redis Cluster provides one method of running Redis where data is sharded across multiple Redis instances while providing a degree of availability for continuing operations in the event of a node failing or a communication issue.
Nodes in the Redis Cluster use the Redis cluster protocol to connect with every other node in the Redis cluster for a mesh network topology. Nodes communicate using a TCP gossip protocol and along with a configuration update mechanism in order to reduce the number of messages being exchanged between nodes.
Redis cluster uses a form of composite partitioning called consistent hashing that calculates what Redis instance any particular key is assigned called a hash slot.
The hash slot is the CRC16 hash algorithm applied to the key and then the computation of a modulo using 16384. The specific algorithm used by Redis cluster to calculate the hash slot for a key:
Each master node is assigned a sub-range of hash slots and the key and value will reside on that Redis instance.
To illustrate all of the features of Redis Cluster, the minimum recommended Redis cluster setup is to have a 3 Master Redis nodes with each master node replicated with a single Redis slave instance node. Each Master Node hash slots are broken down as
We'll start by manually create a minimal cluster with three master nodes and three slaves.
$ cd introduction-to-redis/clustering-and-ha/ $ mkdir cluster-test $ cd cluster-test $ mkdir 7000 7001 7002 7003 7004 7005 $ touch 7000/redis.conf $ vi 7000/redis.conf $ cp 7000/redis.conf 7001/. $ vi 7001/redis.conf ... $ cp ~/redis-3.0.2/src/redis-server . $ cd 7000 $ ../redis-server redis.conf # New terminal tab $ cd ../7001 $ ../redis-server redis.conf ... $ ../redis-server redis.conf
Install the Ruby Redis gem and run the redis-trib.rb
$ sudo gem install redis $ cd ../../ $ .~/redis-3.0.2/src/redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 \ 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005
Here is the example redis.conf we are
using for each of Redis cluster nodes
port 7000
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
We need to change the port number to the correct value for the node, i.e.
for node 7002, we need specify the correct port number for that node's redis.conf
configuration file.
After successfully running redis-trib.rb, you can interact with the cluster
using redis-cli
$ ~/redis-3.0.2/src/redis-cli -c -p 7000 127.0.0.1:7000> SET Book:1 "Moby Dick" -> Redirected to slot [14335] located at 127.0.0.1:7002 OK 127.0.0.1:7002> SET Book:2 "Infinite Jest" -> Redirected to slot [1948] located at 127.0.0.1:7000 OK 127.0.0.1:7000> GET Book:1 -> Redirected to slot [14335] located at 127.0.0.1:7002 "Gone with the Wind" 127.0.0.1:7002> GET Book:2 -> Redirected to slot [1948] located at 127.0.0.1:7000 "Infinite Jest" 127.0.0.1:7000>
Redis commands that involve multiple keys, like MSET will not work with redis cluster
127.0.0.1:7000> MSET Book:3 "Moby Dick" Book:4 "Pride and Prejudice" Book:5 "Tom Sawyer" (error) CROSSSLOT Keys in request don't hash to the same slot
Instead of manually setting up and running a Redis
cluster, you can instead use the create-cluster located in the
redis/utils/create-cluster.
Starting in the create-cluster directory, follow these steps to get a 6-node Redis Cluster with 3 masters and 3 slaves.
config.sh script:
#!/bin/bash
PORT=7000
TIMEOUT=2000
NODES=6
REPLICAS=1
~/redis/utils/create-cluster$ ./create-cluster start
Starting 7001
Starting 7002
Starting 7003
Starting 7004
Starting 7005
Starting 7006
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
127.0.0.1:7001
127.0.0.1:7002
127.0.0.1:7003
Adding replica 127.0.0.1:7004 to 127.0.0.1:7001
Adding replica 127.0.0.1:7005 to 127.0.0.1:7002
Adding replica 127.0.0.1:7006 to 127.0.0.1:7003
M: 1379949e7d8eaa27a0634285e521079eccc0cc1f 127.0.0.1:7001
slots:0-5460 (5461 slots) master
M: 470bf4397e0002f211df09dadcd5ec12b458e9c3 127.0.0.1:7002
slots:5461-10922 (5462 slots) master
M: 7e343391d165ccee34e0f1cf43590270130a9d5b 127.0.0.1:7003
slots:10923-16383 (5461 slots) master
S: 3d8c532367f0846f292b538d09b7cafdc6b3c6b9 127.0.0.1:7004
replicates 1379949e7d8eaa27a0634285e521079eccc0cc1f
S: 491abeb14973c0c9495f1b045b4e5d3f0729bcc8 127.0.0.1:7005
replicates 470bf4397e0002f211df09dadcd5ec12b458e9c3
S: 933258e1d5ed8752c7e4ff7ce377dfd63543977f 127.0.0.1:7006
replicates 7e343391d165ccee34e0f1cf43590270130a9d5b
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join..
>>> Performing Cluster Check (using node 127.0.0.1:7001)
M: 1379949e7d8eaa27a0634285e521079eccc0cc1f 127.0.0.1:7001
slots:0-5460 (5461 slots) master
M: 470bf4397e0002f211df09dadcd5ec12b458e9c3 127.0.0.1:7002
slots:5461-10922 (5462 slots) master
M: 7e343391d165ccee34e0f1cf43590270130a9d5b 127.0.0.1:7003
slots:10923-16383 (5461 slots) master
M: 3d8c532367f0846f292b538d09b7cafdc6b3c6b9 127.0.0.1:7004
slots: (0 slots) master
replicates 1379949e7d8eaa27a0634285e521079eccc0cc1f
M: 491abeb14973c0c9495f1b045b4e5d3f0729bcc8 127.0.0.1:7005
slots: (0 slots) master
replicates 470bf4397e0002f211df09dadcd5ec12b458e9c3
M: 933258e1d5ed8752c7e4ff7ce377dfd63543977f 127.0.0.1:7006
slots: (0 slots) master
replicates 7e343391d165ccee34e0f1cf43590270130a9d5b
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
$ ./create-cluster stop
Stopping 7001
Stopping 7002
Stopping 7003
Stopping 7004
Stopping 7005
Stopping 7006
Redis Sentinel manages Redis instances through four main ways:
Snippet from sentinel.conf
# dirdir /tmp # sentinel monitor sentinel monitor mymaster 127.0.0.1 6379 2 # Default is 30 seconds. sentinel down-after-milliseconds mymaster 30000
Running Redis Sentinel can be accomplished either with the redis-sentinel binary or
by passing the --sentinel parameter when running redis-server. Both
options require a sentinel.conf file.
$ redis-sentinel /path/to/sentinel.conf
$ redis-server /path/to/sentinel.conf --sentinel
You can connect directly to the sentinel with redis-cli and query to find the status of
monitored Redis instances
$ ../redis-3.0.2/src/redis-cli -p 26379
127.0.0.1:26379> PING
PONG
127.0.0.1:26379> SENTINEL masters
127.0.0.1:26379> SENTINEL slaves mymaster