Focused Topics in Redis Day 1 - #1

Redis data types

A real strength of Redis is its powerful data structures that can be used to model and solve complex problems even with such primitive structures as Strings, Lists, Hashes, Sets, and Sorted Sets. Other data structures are stored as strings but have Redis commands for special uses. These are the Bitmap data type and the HyperLogLogs data type.

it all begins with the Key

Redis keys are binary safe -any binary stream can be used as a key- although the most common (and recommended) stream to use is a string key, like "Person", other file formats and binary streams like images, mp3, or other file formats, can be used.

The flexibility of Redis allows for a wide diversity in how keys are structured and stored. Performance and maintainability of the Redis instance can be either positively or negatively impacted by the choices made in designing and constructing the Redis keys using in your database.

A good general practice when designing your Redis keys is to construct at least a rough outline of what information you are trying to store in Redis, which Redis data structure to use, and finally how your data structures relates to other information stored at different keys in your Redis database.

Critical Commands for Keys

Exercise: Key Commands> SET testkey hello
OK> EXISTS testkey
(integer) 1> TYPE testkey
string> KEYS *
1) "testkey"> DEL testkey
(integer) 1> EXISTS testkey
(integer) 0> TYPE testkey
none> SCAN 0
1) "0"
2) 1) "testkey"> SCAN 0 MATCH "hell*"
1) "0"
2) (empty list or set)

String the most basic data-type

In Redis, a string is not merely alphanumeric characters as strings are normally understood to be in higher-level programming languages, but are serialized characters in C. Integers are stored in Redis as a string.

Critical Commands for String

Exercise: Using STRING data type

Setting a string value with a characters and an integer using the SET and GET commands.> SET Book:1 "Infinite Jest"
OK> GET Book:1
"Infinite Jest"> SET Book:1:ReadAction 1
OK> GET Book:1:ReadAction

The INCR command will increase an string "integer" value by 1, the INCRBY command is similar but increase the value by a integer. The DECR and DECRBY commands decrease an string "integer" value by 1 or more.

NOTE: the INCR, INCRBY, DECR, and DECRBY are atomic so other clients cannot change the value when the command is run.> INCR Book:1:ReadAction
(integer) 2> GET Book:1:ReadAction
"2"> INCRBY Book:1:ReadAction 20
(integer) 22> GET Book:1:ReadAction
"22"> DECR Book:1:ReadAction
(integer) 21> GET Book:1:ReadAction
"21"> DECRBY Book:1:ReadAction 5
(integer) 16> GET Book:1:ReadAction

What happens when INCR is used as follows?> INCR Book:1 "Infinite Jest"

You can set multiple key-string values using MSET command. The MGET command retrieves multiple values from one or more keys.> MSET Person:2 "Kurt Vonnegut" Person:3 "Jane Austen" Person:4 "Mark Twain"
OK> MGET Person:2 Person:3 Person:4
1) "Kurt Vonnegut"
2) "Jane Austen"
3) "Mark Twain"

A collection of one or more values is a List

Lists in Redis are ordered collections of Redis strings that allows for duplicates values. Because Redis lists are implemented as linked-lists, adding an item to the front of the list with LPUSH or to the end of the list with RPUSH are relatively inexpensive operations that are performed in constant time of O(1).

Exercise: Using LIST data type

The LPUSH command adds one or more elements to the front of a list, the RPUSH command adds one or more elements to the end of a list.> LPUSH Book:1:comment "This was a fun read"
(integer) 1> LRANGE Book:1:comment 0 -1
1) "This was a fun read"> LPUSH Book:1:comment "Way too long!"
(integer) 2> LRANGE Book:1:comment 0 -1
1) "Way too long!"
2) "This was a fun read"> RPUSH Book:1:comment "Tennis anyone?"
(integer) 3> LRANGE Book:1:comment 0 -1
1) "Way too long!"
2) "This was a fun read"
3) "Tennis anyone?"

The LPOP command remove the first element from the list and returns it the calling client while the RPOP command removes and returns the last element of the list.> LPOP Book:1:comment
"Way too long!"> LRANGE Book:1:comment 0 -1
1) "This was a fun read"
2) "Tennis anyone?"> RPOP Book:1:comment
"Tennis anyone?"> LRANGE Book:1:comment 0 -1
1) "This was a fun read"

The LTRIM command replaces a list with a range from an existing list.> RPUSH Organization:1:members Person:1 Person:2 Person:3 Person:4
(integer) 4> LRANGE Organization:1:members 0 -1
1) "Person:1"
2) "Person:2"
3) "Person:3"
4) "Person:4"> LTRIM Organization:1:members 0 2
OK> LRANGE Organization:1:members 0 -1
1) "Person:1"
2) "Person:2"
3) "Person:3"

To add in implementing simple queues in Redis using the List data-type, the BLPOP and BRPOP commands are similar to LPOP and RPOP commands only they will block sending a return back to client if the list is empty. These blocking commands return two values, the key of the list and an element.> BRPOP Organization:1:members 5
1) "Organization:1:members"
2) "Person:3"> LRANGE Organization:1:members 0 -1
1) "Person:1"
2) "Person:2"

Critical Commands for List

Hash is a "dictionary" of fields and values

Hash data structure maps one or more fields to corresponding value pairs. In Redis, all hash values must be Redis strings with unique field names.

Critical Commands for Hash

Exercise: Using Hash data type

To set the value of a single field, use the HSET, to retrieve a single field for a Hash key, use the HGET command.> HSET Book:3 name "Cats Cradle"
(integer) 1> HGET Book:3 name
"Cats Cradle"

You can set multiple field-values using the HMSET command and retrieve multiple field-values with the HMGET command. To retrieve all of the values of a Redis Hash, use the HGETALL command.> HMSET Book:4 name "Slaughterhouse-Five" author "Kurt Vonnegut" copyrightYear 1969 ISBN 29960763
OK> HMGET Book:4 author ISBN
1) "Kurt Vonnegut"
2) "29960763"> HGETALL Book:4
1) "name"
2) "Slaughterhouse-Five"
3) "author"
4) "Kurt Vonnegut"
5) "copyrightYear"
6) "1969"
7) "ISBN"
8) "29960763"

The HEXISTS determine if a field exists in a Redis Hash. The HLEN returns the number fields in a Redis Hash.> HEXISTS Book:4 copyrightYear
(integer) 1> HEXISTS Book:4 barcode
(integer) 0> HLEN Book:4
(integer) 4

The HKEYS returns all of the field names for a Redis Hash and the HVALS returns all of the values in the Hash.> HKEYS Book:4
1) "name"
2) "author"
3) "copyrightYear"
4) "ISBN"> HVALS Book:4
1) "Slaughterhouse-Five"
2) "Kurt Vonnegut"
3) "1969"
4) "29960763"

HDEL deletes a field from a hash and the HINCRBY increases the integer value of a hash field by a given number.> HDEL Book:4 copyrightYear
(integer) 1> HGETALL Book:4
1) "name"
2) "Slaughterhouse-Five"
3) "author"
4) "Kurt Vonnegut"
5) "ISBN"
6) "29960763"> HSET Book:4 copyrightYear 1968
(integer) 1> HGET Book:4 copyrightYear
"1968"> HINCRBY Book:4 copyrightYear 1
(integer) 1969> HGET Book:4 copyrightYear

An unique and unordered collection of values is a Set

A set is a collection of string values where uniqueness of each member is guaranteed but does not offer ordering of members. Redis sets also implement union, intersection, and difference set semantics along with the ability to store the results of those set operations as a new Redis.

Exercise: Using Set data type

The SADD command adds one or members to a set, to display all of the members of a set use the SMEMBERS command.> SET Organization:5 "Beatles"> SADD Organization:5:member Paul John George Ringo
(integer) 4> SMEMBERS Organization:5:member
1) "Ringo"
2) "John"
3) "Paul"
4) "George"

The SISMEMBER command determines if a value is a member of a set, the SCARD command returns the number of members in a set.> SISMEMBER Organization:5:member "John"
(integer) 1> SISMEMBER Organization:5:member "Ralph"
(integer) 0> SCARD Organization:5:member
(integer) 4

Set operations, union and intersection, are available with the SUNION and SINTER commands respectively. The SDIFF command subtracts multiple sets.> SET Organization:6 "Wings"
OK> SET Organization:7 "Traveling Wilburys"
OK> SADD Organization:6:member Paul Linda Denny
(integer) 3> SADD Organization:7:member Bob George Jeff Roy Tom
(integer) 5> SUNION Organization:5:member Organization:6:member
1) "Ringo"
2) "John"
3) "Paul"
4) "George"
5) "Denny"
6) "Linda"> SUNION Organization:6:member Organization:7:member
1) "Paul"
2) "George"
3) "Roy"
4) "Bob"
5) "Denny"
6) "Tom"
7) "Linda"
8) "Jeff"> SUNION Organization:5:member Organization:6:member Organization:8
 1) "Roy"
 2) "George"
 3) "Bob"
 4) "Denny"
 5) "Linda"
 6) "Ringo"
 7) "Paul"
 8) "John"
 9) "Tom"
10) "Jeff"> SINTER Organization:5:member Organization:6:member
1) "Paul"> SINTER Organization:5:member Organization:7:member
1) "George"> SINTER Organization:6:member Organization:7:member
(empty list or set)> SDIFF Organization:5:member Organization:6:member
1) "John"
2) "Ringo"
3) "George"> SDIFF Organization:6:member Organization:5:member
1) "Linda"
2) "Denny"

Critical Commands for Set

Sorted Set an ordered collection of unique values

A sorted set combines characteristics of both Redis lists and sets. Like a Redis list, a sorted set's values are ordered and like a set, each value is assured to be unique. The flexibility of a sorted set allows for multiple types of access patterns depending on the needs of the application.

Critical Commands for Sorted Set

Exercise: Using Sorted Set data type

The ZADD command adds a member with a score to a sorted set. To retrieve a range of members by index, use the ZRANGE command. To retrieve a range of members from high score to low score, use the ZREVRANGE command.> ZADD copyrightYear 1996 Book:1 2014 Book:2 1963 Book:3
(integer) 3> ZADD copyrightYear 1969 Book:4
(integer) 1> ZRANGE copyrightYear 0 -1
1) "Book:3"
2) "Book:4"
3) "Book:1"
4) "Book:2"> ZRANGE copyrightYear 0 -1 WITHSCORES
1) "Book:3"
2) "1963"
3) "Book:4"
4) "1969"
5) "Book:1"
6) "1996"
7) "Book:2"
8) "2014"> ZREVRANGE copyrightYear 0 -1
1) "Book:2"
2) "Book:1"
3) "Book:4"
4) "Book:3"> ZREVRANGE copyrightYear 0 -1 WITHSCORES
1) "Book:2"
2) "2014"
3) "Book:1"
4) "1996"
5) "Book:4"
6) "1969"
7) "Book:3"
8) "1963"

In Redis sorted sets, the order of members is first determined by the score. If two members have identical scores, the members are then sorted lexicographically by value.> ZADD Book:names 0 "Into the Wild" 0 "Cat's Cradle" 
(integer) 2> ZADD Book:names0 "Time Machine, The" 0 "Gravity's Rainbow"
(integer) 2> ZRANGE Book:names 0 -1 WITHSCORES
1) "Cat's Cradle"
2) "0"
3) "Gravity's Rainbow"
4) "0"
5) "Into the Wild"
6) "0"
7) "Time Machine, The"
8) "0"

The ZRANK and ZSCORE commands retrieves the index (rank) and score of a member in a sorted set. The ZREM command removes the member from the sorted set.> ZRANK copyrightYear Book:3
(integer) 0> ZRANK copyrightYear Book:1
(integer) 2> ZRANK copyrightYear Book:56
(nil)> ZSCORE copyrightYear Book:3
"1963"> ZSCORE copyrightYear Book:1
"1996"> ZRANGE copyrightYear 0 -1 WITHSCORES
 1) "Book:3"
 2) "1963"
 3) "Book:1"
 4) "1996"
 5) "Book:5"
 6) "1996"
 7) "Book:6"
 8) "1996"
 9) "Book:2"
10) "2014"

Other useful sorted set commands include: ZCARD which returns the total number of members in the sorted set. ZCOUNT that returns the number of members that are in a range of scores. ZRANGEBYSCORE returns all members in a sorted set that have score between a minimum and maximum score. Both ZCOUNT and ZRANGEBYSCORE use -inf for all scores from negative infinity and +inf for all score to infinity.> ZCOUNT copyrightYear 1900 1970
(integer) 2> ZRANGEBYSCORE copyrightYear 1900 1970 WITHSCORES
1) "Book:3"
2) "1963"
3) "Book:4"
4) "1969"> ZRANGEBYSCORE copyrightYear -inf 2000 WITHSCORES
 1) "Book:3"
 2) "1963"
 3) "Book:4"
 4) "1969"
 5) "Book:1"
 6) "1996"
 7) "Book:5"
 8) "1996"
 9) "Book:6"
10) "1996"> ZRANGEBYSCORE copyrightYear 1998 +inf WITHSCORES
1) "Book:2"
2) "2014"

treating a string as sequence of bits is a Bit array or Bitmap

Redis bitstrings is a very memory efficient data structures in Redis. In a bitstring, 8 bits are stored per byte, with the first bit at position 0 being the significant one that is either set to either 0 or 1. The maximum size for Redis bitstrings is 512 MB.

Exercise: Bitmap Commands

SETBIT and GETBIT commands operate on a single bit offset in a Redis string. Bitmaps are highly efficient in storing boolean information on a range of related> SET Movie:1 "Blade Runner"
OK> HSET Movie:2 name "Star Wars"
OK> SADD Movie:3 "2001 a Space Odyssey"
(integer) 1> SETBIT Movie:UserPlays:2014-12-11 2 1
(integer) 0> GETBIT Movie:UserPlays:2014-12-11 2
(integer) 1

The BITCOUNT command returns the total number of bits set to 1 in Redis bitmap string. The BITPOS command returns the first offset that matches either 1 or 0.> BITCOUNT Movie:UserPlays:2014-12-11
(integer) 1> BITPOS Movie:UserPlays:2014-12-11 1
(integer) 2> BITPOS Movie:UserPlays:2014-12-11 0
(integer) 0

The BITOP command performs the following bit-wise operations on different strings: AND, OR, XOR, and NOT.> SETBIT Movie:UserPlays:2014-12-12 2 1
(integer) 0> SETBIT Movie:UserPlays:2014-12-12 1 1
(integer) 0> BITOP AND and_result Movie:UserPlays:2014-12-11 Movie:UserPlays:2014-12-12
(integer) 1> GETBIT and_result 1
(integer) 0> GETBIT and_result 2
(integer) 1> GETBIT and_result 3
(integer) 0> BITOP OR or_result Movie:UserPlays:2014-12-11 Movie:UserPlays:20
(integer) 1> GETBIT or_result 1
(integer) 1> GETBIT or_result 2
(integer) 1> BITOP OR or_result Movie:UserPlays:2014-12-11 Movie:UserPlays:2014-12-12
(integer) 1> GETBIT or_result 1
(integer) 1> GETBIT or_result 2
(integer) 1> GETBIT or_result 3
(integer) 0> BITOP XOR xor_result Movie:UserPlays:2014-12-11 Movie:UserPlays:2014-12-12
(integer) 1> GETBIT or_result 1
(integer) 1> GETBIT or_result 2
(integer) 1> GETBIT or_result 3
(integer) 0> BITOP NOT not_result Movie:UserPlays:2014-12-11
(integer) 1> GETBIT not_result 1
(integer) 1> GETBIT not_result 2
(integer) 0> GETBIT not_result 3
(integer) 1

Critical Commands for Bitmap

HyperLogLogs special probabilistic data type similar to a Sorted Set

Redis data type is a probabilistic data structure that provides an estimated count of unique items in a collection.

HyperLogLogs offer a reduced

Critical Commands for HyperLogLogs

  • PFADD key element [element …]
  • PFCOUNT key [key …]
  • PFMERGE destination_key key1 [key2 …]

Exercise: Bitmap Commands

PFADD adds one or more elements to a HyperLogLogs. The PFCOUNT returns an approximation of the count in the HyperLogLogs with an standard error of .81%. With PFMERGE multiple HyperLogLogs can be merged into a single HyperLogLogs. The advantage of the HyperLogLogs is that is very efficient in memory (maximum size is 12k bytes) and does not require an proportional amount of memory (memory to items in the set) to perform a population count.> PFADD EducationEvent:1:attendee Person:1 Person:2 Person:3 Person:44556
(integer) 1> PFCOUNT EducationEvent:1:attendee
(integer) 4> PFADD LiteraryEvent:1:attendee Person:4 Person:1
(integer) 1> PFMERGE Event:attendee EducationEvent:1:attendee LiteraryEvent:1:attendee
OK> PFCOUNT Event:attendee
(integer) 5


Downloading and Using Geo Commands

With a running 3.2 Redis instance, we'll open a second terminal window and launch Redis-cli. We'll then add a couple of data points to the BayArea key with the GEOADD and then calculate the distance between San Francisco and San Jose with the GEODIST> GEOADD BayArea 121.8863 37.7833 "San Jose" 122.4167 37.7833 "San Francisco" 122.2708 37.8044 Oakland> GEODIST BayArea "San Francisco" "San Jose"
"46624.876174299716"> GEODIST BayArea "San Francisco" "San Jose" km
"46.624876174299715"> GEODIST BayArea "San Francisco" "San Jose" mi

With the GEORADIUS and GEORADIUSBYMEMBER returns the geospatial information that are within the borders of an area specified with a central location and a maximum distance from the center.> GEORADIUS BayArea 121.9692 37.3544 100 mi
1) "San Jose"
2) "Oakland"
3) "San Francisco"> GEORADIUS BayArea 121.9692 37.3544 30 mi
1) "San Jose"

Use Case: Linked Data Fragments Server

Overview of Linked Data

Based upon Sir Tim Berners-Lee idea of the Semantic Web, libraries are adopting various RDF graph-based vocabularies to describe their collections while linking with other institutions and projects to extend and enrich the discovery of their materials by patrons.

RDF Graphs

RDF Graphs are made up of triples, statement made up one each of Subjects, Predicates, and Objects.

Subjects are made up either IRIs (most commonly an URL), or Blank Nodes, unique identifiers within a single RDF graph.

Predicates must be IRIs and is the graph edge that connects the subject and object entities with a relationship.

Objects are made up of either URLs, Literal values, or Blank Nodes.

Data structure evolution

During the development of the Linked Data Fragments Server, the variety and types of Redis data-structures that were used changed as further design, development, and testing continued in the project.

Redis Strings

The initial schema design for the Linked Data Fragments Server was to generate a SHA1 hash digest for each subject, predicate, and object and then storing a simple composite key made up of the triple.



SHA1 Hash digest: e61da38f50b68b932ddcde7d3ec6de5baddba0e9



SHA1 Hash digest: 4dab97eaa98fcba6bd7a204da2fea672e51f6a84


Literal Value: Denver Broncos

SHA1 Hash digest: 0a84cadbc87cbca90ddf0f90e4af57d6f0753335> SETNX e61da38f50b68b932ddcde7d3ec6de5baddba0e9
(integer) 1> SETNX 4dab97eaa98fcba6bd7a204da2fea672e51f6a84
(integer) 1> SETNX 0a84cadbc87cbca90ddf0f90e4af57d6f0753335 "Denver Broncos"
(integer) 1> SETNX e61da38f50b68b932ddcde7d3ec6de5baddba0e9:4dab97eaa98fcba6bd7a204da2fea672e51f6a84:0a84cadbc87cbca90ddf0f90e4af57d6f0753335 1
(integer) 1

References and Resources

  1. From the website section on commands