Introduction to Redis

Replication and Pub/Sub

Redis supports two models of distributed computing. Redis's master-slave replication is simple to use with horizontal scaling across multiple slaves connecting asynchronous with a master Redis instance. In addition, Redis is well suited to supporting the Publish/Subscribe model where a publisher application sends messages to one or more channels that subscriber applications monitor and then react to the messages in the channel.

Operating a Redis System Pipelining and Mass Insertions of Data

In this topic …

  • Master-Slave Replication
  • Pub/Sub

    • Exercises

  • Runtime set-up of Master-Slave Replication
  • Set-up of Master-Slave Replication with redis.config
  • Pub/Sub

    • References and Resources

    Master-Slave Replication

    Redis's Master-Slave replication starts with at least two Redis server instances; a master and one or more slave instances. All Redis server instances store a copy of data. The difference between the two is that the slaves are usually read-only while the master instance accepts reads and writes.

    Exercise: Runtime Master and Slave instances

    Open four command-line windows to run a Redis master, slave, and redis-cli instances

    Redis Master

    $ cd introduction-to-redis/replication-and-pub-sub/
$ ~/redis-3.0.2/src/redis-server --dbfilename master.rdb --port 6379

    Redis Slave

    $ cd introduction-to-redis/replication-and-pub-sub/
$ ~/redis-3.0.2/src/redis-server --dbfilename slave.rdb --port 6380

    Redis CLI

    $ ~/redis-3.0.2/src/redis-cli -p 6379
127.0.0.1:6379> DBSIZE
(integer) 0
127.0.0.1:6379> exit
$
$ ~/redis-3.0.2/src/redis-cli -p 6380
127.0.0.1:6380> DBSIZE
(integer) 0
127.0.0.1:6380> SLAVEOF localhost 6379
OK
127.0.0.1:6380> exit
$
$ ~/redis-3.0.3/src/redis-cli -p 6379
127.0.0.1:6379> HMSET Book:1 name "Infinite Jest" author "David Foster Wallace"
OK
127.0.0.1:6379> exit
$
$ ~/redis-3.0.2/src/redis-cli -p 6380
127.0.0.1:6380> KEYS *
127.0.0.1:6380> KEYS *
1) "Book:1"
127.0.0.1:6380> HGETALL Book:1
1) "name"
2) "Infinite Jest"
3) "author"
4) "David Foster Wallace"
127.0.0.1:6380> HSET Book:1 copyrightYear 1996
(error) READONLY You can't write against a read only slave.

    Exercise: Set-up of Master-Slave Replication with redis.config

    Master-Slave replication can be set-up in the redis.conf file for each slave.

    # Master-Slave replication. Use slaveof to make a Redis instance a copy of
# another Redis server.
slaveof <masterip> <masterport>
    Facts about Master-Slave Replication
    • Master and Slaves use asynchronous replication
    • Masters can have multiple slaves
    • Slaves can accept connections from other slaves (i.e. a slave could have as it's master a Redis instance that is slave to a master instance)
    • Replication is non-blocking on both master and slave instances; a master can continue to respond to queries while slaves are synchronizing, a slave can be configured to respond to queries with old data while synchronizing with the master

    Critical Commands for Master-Slave Replication

    Caveats when using Replication

    If persistence is turned off on the master A and it has two slaves B, C the following problem can occur:

    • We have a setup with node A acting as master, with persistence turned off, and nodes B and C replicating from node A.
    • A crashes, however it has some auto-restart system, that restarts the process. However since persistence is turned off, the node restarts with an empty data set.
    • Nodes B and C will replicate from A, which is empty, so they'll effectively destroy their copy of the data.

    SYNC, Partial Resynchronization & Diskless Replication

    Redis Replication

    Steps:
    1. The master then starts background saving, and starts to buffer all new commands received that will modify the dataset.
    2. When the background saving is complete, the master transfers the database file to the slave, which saves it on disk, and then loads it into memory.
    3. The master will then send to the slave all buffered commands. This is done as a stream of commands and is structured in the Redis.

    Partial Resynchronization

    Replication is non-blocking on both master and slave instances; a master can continue to respond to queries while slaves are synchronizing, a slave can be configured to respond to queries with old data while synchronizing with the master

    If the link between the master and slave goes down, the old behavior of Redis is to do a full synchronization with the slave when the connection is re-established. As of Redis 2.8 experimental support was added with PSYNC command that will only do a partial synchronization with MASTER based on if a time threshold has been exceeded or not. If time has been exceeded, a full SYNC is triggered between the master and slave.

    Diskless replication

    An experimental feature that sends the full RDB file to the slave without forking the master and using the disk as an intermediate storage.

    Pub/Sub (Publication/Subscription)

    Critical Commands for Pub/Sub

    Exercise: Pub/Sub

    In this exercise, we will use redis-cli as a publisher and use the a Python command line to run a consumer application using PUBLISH and SUBSCRIBE commands. The Python client redis-py has its own Publish/Subscribe class with its own method calls for responding to messages published to a channel.

    Redis Python Subscription Client

    >>> import redis
>>> local_redis = redis.StrictRedis()
>>> circ_consumer = local_redis.pubsub()
>>> circ_consumer.subscribe("checkouts", "checkins")
>>> circ_consumer.get_message()

    Redis redis-cli Publisher Client

    127.0.0.1:6379> publish checkouts Book:3
(integer) 1
127.0.0.1:6379> publish checkouts Book:1
(integer) 1
127.0.0.1:6379> publish checkins Book:3
(integer) 1
127.0.0.1:6379> publish checkins Book:3
(integer) 1

    Pub/Sub Message Syntax

    Redis Pub/Sub messages are defined as 3 or 4 part RESP (REdis Serialization Protocol) array. RESP is the communication protocol that Redis client communicate with Redis server and is used in other projects beside Redis. We see the raw RESP protocol if we use telnet and connect directly to Redis.

    First, we will connect to a single Redis instance running at port 6379 with telnet. 

    $ telnet localhost 6379
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

    Now, we will subscribe to a chat_room channel

    and from a redis-cli session, we will post a message to the channel. 
    SUBSCRIBE chat_room
*3
$9
subscribe
$9
chat_room
:1
*3
$7
message
$9
chat_room
$18
Anybody out there?

    Pattern Matching with PSUBSCRIBE & PUNSUBSCRIBE

    To subscribe to more than one channel, you can use the PSUBSCRIBE that take a pattern to match to existing channels.

    PSUBSCRIBE chat_*
*3
$10
psubscribe
$6
chat_*
:1
*4
$8
pmessage
$6
chat_*
$9
chat_room
$18
What is your name?

    To unsubscribe to multiple channels at once, you can use the PUNSUBSCRIBE 

    PUNSUBSCRIBE chat*
*3
$12
punsubscribe
$5
chat*
:1

    References and Resources

    1. From the redis.io website: topic on Replication
    2. Topic on *Pub/Sub