Why we migrated from Memcached to Redis?
Introduction
Most of the time, caching solutions like Memcached, Redis, Guava, etc. are used to increase data retrieval performance. This is achieved by storing some of the data in the cache layer and thus reducing the need to access the underlying slower data-storage layer. But there is another usage pattern of caches which is called “Cache-as-datastore”. In this pattern, a cache is used as the primary data store itself. This is usually done to store ephemeral data that may have an expiration timestamp.
Memcached and Redis are two of the most popular caching solutions. Both of them are offered by AWS as fully managed caching services under Amazon ElastiCache. For the rest of this article, Memcached and Redis will imply AWS ElastiCache offering of Memcached and Redis respectively.
In our use case, we were using Memcached as a data store to store ephemeral data. As the overall load on the system increased, we started running into some performance issues with our Memcached cluster. We decided to migrate from Memcached and move to Redis because Memcached’s architecture does not support some of the capabilities that could have helped mitigate these performance issues.
In the following sections, we will deep dive more into the issues we faced with Memcached and how we used Redis to overcome them.
Memcached Architecture
Memcached is an in-memory key-value store service that can be used as a cache or a data store and provides sub-millisecond latency.
Memcached supports data sharding, thus allowing to partition data across multiple nodes and distributing the load in a cluster. Each node in a Memcached cluster is totally independent and does not communicate with the others. Instead of the cluster being aware of the data partitioning, the logic for data partitioning resides with the client. As a result, the addition or removal of a shard would result in cache misses until the client repopulates all the shards as per the new shard count in the cluster.
Memcached is multithreaded, and it therefore can make use of multiple processing cores. This means that you can handle more operations by scaling up compute capacity of a Memcached cluster. This is a good technique to handle an increase in the load, but it is usually upper bound by the monetary budget.
Drawbacks of Memcached
While Memcached is very powerful and easy to use, there are two capabilities that are not supported by Memcached.
First, Memcached does not offer Shard Replication. A Memcached cluster can only have one node per shard. In absence of any replica nodes, a node failure implies that the entire shard being handled by that node has failed.
The second issue is that Memcached does not support Online Re-sharding. In a distributed cache system, keys are spread across multiple shards. With online re-sharding, a shard can be seamlessly added or removed from the cluster, without causing any downtime. The cluster would handle all the requests gracefully while it is redistributing the keys as per the new shard count. But in absence of online re-sharding, any change in the number of shards will result in a partial downtime till all the keys are properly redistributed.
Why migrate to Redis?
Redis is an in-memory data store that provides sub-millisecond latency. Redis also offers functionalities like transactions, pub/sub, Lua scripting, etc. that are not supported by Memcached. With fewer functionalities, Memcached is designed for simplicity and hence it was our choice when we started.
Our Memcached setup had multiple shards in the cluster. Each shard is comprised of only one node (no replication). Thus, a single node per shard serves both the read and write requests.
As the business grew, so did the number of read and write operations, resulting in an increased overall load on our MemcacheD cluster. With the increased load, the nodes in our Memcached cluster will occasionally run very hot. The only way to fix such situations was to add more shards to the Memcached cluster. But as we have discussed before, Memcached does not support online re-sharding. Hence the addition of a new shard implies cache misses until all the keys are redistributed. As a result, we were forced us to carry out these shard additions only during off-peak traffic windows, when we could afford a maintenance downtime.
As the frequency of nodes running hot and requiring the addition of a new shard to the Memcached cluster increased, we started to look for alternate solutions. This is how Redis came into the picture.
Redis Architecture
ElastiCache for Redis allows three configurations, (1) single node, (2) cluster mode disabled, and (3) cluster mode enabled. The diagram below explains these configurations in more detail. Note that in cluster mode enabled setup, ElastiCache for Redis offers both shard replication and online re-sharding.
For our use case, we decided to use (3) cluster mode enabled mode with data replication and partitioning. This allowed us to distribute the data across multiple shards thus reducing the load on a single shard. And with data replication, any failure in the primary node of a shard can be handled by falling back on the replica node for that shard.
Redis Clients for Java
Redisson and Jedis are two of the most popular Java clients for Redis.
Redisson works as an in-memory data grid over Redis. In other words, Redisson provides an object interface that is native to Java and stores these objects in Redis. Jedis, on the other hand, provides an object interface that is native to Redis.
Let us take an example of storing key-value pairs in Redis. Redisson provides RMap object for this use case. The interface of RMap object is similar to Java’s native Map object. Under the hood, Redisson will manage the keys and values of the RMap object and store them in Redis. On the other hand, Jedis also supports the map object, but uses the native commands of Redis like “SET”, “GET”, etc. Thus, the user needs to understand Redis Command Line language to use the Jedis client efficiently.
Redisson v/s Jedis
There are pros and cons to both Redisson and Jedis. In our use case, we wanted to ensure that our data is partitioned and replicated and that the client supports online re-sharding.
Redisson Java objects are Redis cluster compatible, but their state is not partitioned to multiple primary nodes in the Redis cluster. Redisson PRO (paid) offers data partitioning for some of these objects. As we were not willing to compromise on data sharding, hence we did not move forward with Redisson (free version).
Our next choice was Jedis. Jedis supports data partitioning allowing the storage of data across multiple shards. Jedis also supports read traffic redirection in cluster mode disabled setup (single shard, multiple replicas). This allows writes to hit the primary node and the reads to hit the replica node(s). Lastly, Jedis supports the graceful handling of requests during cluster re-sharding. Note that to fully utilize this feature, you will have to write some extra code in your Jedis client wrapper.
In our case, we were using Redis in cluster mode enabled setup (multiple shards, multiple replicas). Unfortunately, Jedis does not support read redirection to replica nodes in the cluster mode enabled setup. But we were ok with making this tradeoff, as we were still getting the benefits of data partitioning and online re-sharding. Hence, we decided to use Jedis client for our Redis cluster.
Conclusion
Depending on your use case, you can decide between Redis and Memcached. While this is not a one size fits all problem, still, a basic rule of thumb would be to use Memcached if your use case is simple. Furthermore, if you are using Redis, you can decide between Redisson and Jedis, based on the trade-offs that you are comfortable making for your use case.
