Day 2

Focused Topics in Redis Day 1 - #4

Introduction to Redis Cluster

In this topic …

Redis Cluster provides one method of running a Redis installation where data is sharded across multiple Redis instances while providing a degree of availability for continuing operations in the event of partial node failure or communication failure.

This topics starts with an overview of a minimal Redis cluster with an demonstration of the composite hashing approach used by Redis

Partitioning

Composite Partitioning

Redis cluster uses a form of composite partitioning called consistent hashing that calculates what Redis instance any particular key is assigned called a hash slot.

The hash slot is the CRC16 hash algorithm applied to the key and then the computation of a modulo using 16384. The specific algorithm used by Redis cluster to calculate the hash slot for a key:

  1. The cyclic redundancy check (CRC) using a polynomial length of 17-bits or CRC16 is calculated on the Redis key
  2. The modulo 16384 of the value is then calculated to get the hash slot for the key.

Each master node is assigned a sub-range of hash slots and the key and value will reside on that Redis instance.


Cluster Topology & Redis Cluster Protocol

Mesh Network

Nodes in the Redis Cluster use the Redis cluster protocol to connect with every other node in the Redis cluster for a mesh network topology. Nodes communicate using a TCP gossip protocol and along with a configuration update mechanism in order to reduce the number of messages being exchanged between nodes.

When a Redis cluster is running, each node has two TCP sockets open:

Clients should never need to connect directly with the cluster bus port but with the lower, standard port.


Six-Node Redis Cluster

To illustrate all of the features of Redis Cluster, the minimum recommended Redis cluster setup is to have a 3 Master Redis nodes with each master node replicated with a single Redis slave instance node. Each Master Node hash slots are broken down as

Master One (M1) - allocated hash slots 0-5460
Slave One (S1) replicates Master One, is promoted if a quorum of nodes cannot reach Master One.
Master Two (M2) - allocated hash slots 5461-10922
Slave Two (S2) replicates Master Two, is promoted if a quorum of nodes cannot reach Master Two.
Master Three (M3) - allocated hash slots 10923-16383
Slave Three (S3) replicates Master Three, is promoted if a quorum of nodes cannot reach Master Three.

Manually Running a Redis Cluster

We'll start by manually create a minimal cluster with three master nodes and three slaves.

$ mkdir cluster-test
$ cd cluster-test
$ mkdir 7000 7001 7002 7003 7004 7005
$ touch 7000/redis.conf
$ vi 7000/redis.conf
$ cp 7000/redis.conf 7001/.
$ vi 7001/redis.conf
...
$ cp ~/redis/src/redis-server .
$ cd 7000
$ ~/redis/src/redis-server redis.conf
# New terminal tab
$ cd ../7001
$ ~/redis/src/redis-server redis.conf
...
$ ~/redis/src/redis-server redis.conf
    

Install the Ruby Redis gem and run the redis-trib.rb

$ sudo gem install redis
$ cd ../../
$ .~/redis/src/redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 \
127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005
    
Sample redis.conf

Here is the example redis.conf we are using for each of Redis cluster nodes

port 7000
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
                

We need to change the port number to the correct value for the node, i.e. for node 7002, we need specify the correct port number for that node's redis.conf configuration file.


Easier Redis Cluster with create-cluster script

Instead of manually setting up and running a Redis cluster, you can instead use the create-cluster located in the redis/utils/create-cluster.

Starting in the create-cluster directory, follow these steps to get a 6-node Redis Cluster with 3 masters and 3 slaves.

  1. Create a config.sh script:
    #!/bin/bash
    PORT=7000
    TIMEOUT=2000
    NODES=6
    REPLICAS=1
            
  2. Start Redis Cluster:
    ~/redis/utils/create-cluster$ ./create-cluster start
    Starting 7001
    Starting 7002
    Starting 7003
    Starting 7004
    Starting 7005
    Starting 7006
            
  3. Link and create the Redis Cluster
  4. >>> Creating cluster
    >>> Performing hash slots allocation on 6 nodes...
    Using 3 masters:
    127.0.0.1:7001
    127.0.0.1:7002
    127.0.0.1:7003
    Adding replica 127.0.0.1:7004 to 127.0.0.1:7001
    Adding replica 127.0.0.1:7005 to 127.0.0.1:7002
    Adding replica 127.0.0.1:7006 to 127.0.0.1:7003
    M: 1379949e7d8eaa27a0634285e521079eccc0cc1f 127.0.0.1:7001
       slots:0-5460 (5461 slots) master
    M: 470bf4397e0002f211df09dadcd5ec12b458e9c3 127.0.0.1:7002
       slots:5461-10922 (5462 slots) master
    M: 7e343391d165ccee34e0f1cf43590270130a9d5b 127.0.0.1:7003
       slots:10923-16383 (5461 slots) master
    S: 3d8c532367f0846f292b538d09b7cafdc6b3c6b9 127.0.0.1:7004
       replicates 1379949e7d8eaa27a0634285e521079eccc0cc1f
    S: 491abeb14973c0c9495f1b045b4e5d3f0729bcc8 127.0.0.1:7005
       replicates 470bf4397e0002f211df09dadcd5ec12b458e9c3
    S: 933258e1d5ed8752c7e4ff7ce377dfd63543977f 127.0.0.1:7006
       replicates 7e343391d165ccee34e0f1cf43590270130a9d5b
    Can I set the above configuration? (type 'yes' to accept): yes
    >>> Nodes configuration updated
    >>> Assign a different config epoch to each node
    >>> Sending CLUSTER MEET messages to join the cluster
    Waiting for the cluster to join..
    >>> Performing Cluster Check (using node 127.0.0.1:7001)
    M: 1379949e7d8eaa27a0634285e521079eccc0cc1f 127.0.0.1:7001
       slots:0-5460 (5461 slots) master
    M: 470bf4397e0002f211df09dadcd5ec12b458e9c3 127.0.0.1:7002
       slots:5461-10922 (5462 slots) master
    M: 7e343391d165ccee34e0f1cf43590270130a9d5b 127.0.0.1:7003
       slots:10923-16383 (5461 slots) master
    M: 3d8c532367f0846f292b538d09b7cafdc6b3c6b9 127.0.0.1:7004
       slots: (0 slots) master
       replicates 1379949e7d8eaa27a0634285e521079eccc0cc1f
    M: 491abeb14973c0c9495f1b045b4e5d3f0729bcc8 127.0.0.1:7005
       slots: (0 slots) master
       replicates 470bf4397e0002f211df09dadcd5ec12b458e9c3
    M: 933258e1d5ed8752c7e4ff7ce377dfd63543977f 127.0.0.1:7006
       slots: (0 slots) master
       replicates 7e343391d165ccee34e0f1cf43590270130a9d5b
    [OK] All nodes agree about slots configuration.
    >>> Check for open slots...
    >>> Check slots coverage...
    [OK] All 16384 slots covered.
            
  5. Stopping Redis Cluster
    $ ./create-cluster stop
    Stopping 7001
    Stopping 7002
    Stopping 7003
    Stopping 7004
    Stopping 7005
    Stopping 7006
            

Using Redis Cluster with redis-cli

After successfully launching your Redis Cluster, an easy way to interact with it is to use the default Redis command-line client, redis-cli in cluster mode.