> Rate limiting is one of the most common Redis use cases. Traditionally, users implemented rate limiters using server-side Lua scripts combined with client logic. In Redis 8.8, we introduce a window counter rate limiter (by @raffertyyu, together with the Redis team).
I had a look for this and it turns out it's slightly mis-described there - it's not a window counter, it's a "GCRA (Generic Cell Rate Algorithm)" - a leaky bucket algorithm. Code here: https://github.com/redis/redis/blob/unstable/src/gcra.c
It's a neat algorithm (I just learned about it today) - it only needs to store a single integer for each rate-limited key, which is the "Theoretical Arrival Time" when the bucket would next be empty.
Also, the “cell” in Generic Cell Rate Algorithm is an ATM cell. GCRA is 1990s telecom, the scheduling algorithm ATM switches used to check that 53-byte cells were arriving on the wire at the agreed rate.
I've never liked token buckets (vs sliding window counters) due to the extra work CPU cycles required to "fill" them. It seems like doing an atomic incr on a key based on a `time % 1 minute` or something would be more efficient and then let that key TTL expire X duration later. This results in zero work for rate limiters not in use and only a single push change -> resulting count for ones in use. Nothing but setting the TTL extra is required.
Thanks for the links, I'll checkout the Generic Cell Rate Algorithm!
While I love Redis as a versatile tool for external data structures, it's still lacking in two areas IMHO:
One, it would be cool to be able to embed it, similar to sqlite, directly into applications.
Two, the HA story is so much more complicated than it should be. I totally acknowledge that concurrency and distributed computing is hard, but it should not require reading heaps of documentation and understanding two entirely separate multi-node approaches only to figure out there are lots of subtle strings attached that make it impractical for many applications.
What would be the point of embedding Redis into an application? What's the advantage of using Redis over using the builtin (or third party) data structures of the language the application is developed in?
I'm asking as a non-webdev who never quite got what Redis actually does, but would love to learn.
To me the thing I like about Redis is that it gives you a storage engine very suitable for caches; it handles TTLs and memory pressure, as well as built-in serialization with the ability to get better performance by allowing for some data loss. At the same time, many users will be deploying small programs to individual machines. If you could just have Redis be embedded this would make it very operationally simple: no additional daemons and a single file to backup if you want to.
It would also be useful because of the ability to switch modalities. When running a multi node service, you can use Redis to share data between nodes and use Redis pubsub as a communication bus. If you wanted to support a simple single node configuration too, then it wouldn't need to be a special case, it could just go through the same mechanism but with an embedded Redis instance.
It's pretty similar to SQLite: being able to embed more or less a complete storage engine into your app can be very convenient and powerful.
Well the most basic redis replacement would be just a global hashmap to replace GET and SET, possibly with a background thread to periodically delete expired keys. But obviously that stops working as soon as you get a second node.
The entire value of redis IMO is that is ISN'T inside your normal application, but rather some shared storage that all nodes can use to coordinate and that survives deploys, but that provides more ergonomic data structures than SQL databases. Caches are only one type of such shared data, but things like feature flags, circuit breakers and rate limiters are also super common (and super useful).
Neat. Write that up, match parity, and give all the function calls with the same name as redis, and you're both happy! You get to hand roll something, he gets to use a library that others have perfected over the years!
Unfortunately I have never really used Erlang outside of deploying RabbitMQ. I mostly use Go, Rust, Python, sometimes C/C++.
However, Mnesia seems like it is quite a bit more of a complete distributed database engine than Redis. To me the nicest thing about Redis is just the convenience of what it offers: very fast data structures, serialized, optimized (at least by default) for cases where speed is more important than durability. It is simple on many levels and somewhat constrained in scope. Mnesia seems to be aiming more generally in the distributed database category.
Really it would be more like Nebulex/Cachex which provide a really nice caching interface across ETS (what Mnesia is built off of) or other data stores.
Probably because Redis gives you a very well-defined/understood set of rich data structures with built-in behavior like TTL, atomic operations, eviction, and persistence. These things are otherwise usually scattered across native types, helper classes, or entirely separate libraries.
Language's own native data-structures are generally much more capable and vast. 99%+ developers use only a very limited set of those capabilities. This approach packages those most used ones into a nice, consistent DSL.
It's similar in effect to what busybox does to shell utilities, though the motives are different.
Doesn’t depend on the language? Actually I am thinking of the standard library… Python’s in kinda huge and some are hard (for me) to grasp. Golangs seem pretty simple.
Can you name a single language that can talk to redis and doesn't have these in a form of a library that integrates with an app better than mystical embedded redis?
Every language you can talk to redis most likely has a library to do that, and it probably works much better with the rest of application than "embedded redis". If it doesn't, it probably has C-FFI and there is "fast, robust and well understood" implementations in C.
Sure. But if Redis was embeddable you'd get a robust C-FFI style implementation of those data structures which has been tested a lot more than some random library that has almost no existing users or active maintenance.
(I'm not personally sold on embedded Redis myself, but the question was "Aren’t your own programming language’s constructs much more well-defined / understood?")
> you'd get a robust C-FFI style implementation of those data structures which has been tested a lot more than some random library that has almost no existing users or active maintenance.
What are these mystical random libraries you're talking about? There is a solid C implementation of every data structure on this planet.
A few nice things about doing this in no particular order:
Embedding would make local dev/CI integration testing convenient.
Embedding replicated Redis with each application instance would give you HA benefits while infra-management complexity.
Embedded redis (even via local RPC) is still going to be faster than a lot of languages or frameworks’ built-in data structures. Large array operations in, say, Python are gonna slower than RPCing to Redis (assuming that the data structures are built gradually and not built all at once); to beat Redis you’d have to use numpy or something—-which is definitely preferable, but is extra work if your app already uses Redis for other things.
Just like choosing SQLite over e.g. LMDB or RocksDB, embedded Redis would be a nice future proofing option for small apps during the prototype phase; less would have to be changed to move Redis out of the app than if a different cache or persistence service were chosen.
I have 15 processes handling API requests over HTTP. I want rate limiting, so users don't DoS my API. I need shared state across these processes to store information about rate limits for each client key. I store those in redis.
Or queues, caching, pub/sub... Redis can do a lot, it's really (and i mean really) fast, very easy to get up and running and the protocol is pretty simple.
Network hops are not free! Those milliseconds are an eternity compared to local function calls.
The optimal architecture is something like what Service Fabric or Orleans can do with their distributed dictionary types: reads are generally in-process and take only nanoseconds (but writes require a synchronous replica copy to a remote host.)
Obviously this requires load balancers to steer traffic consistently, but that’s a common feature… outside of the public clouds where they forgot latency exists.
A typical (meaningful) example might be communication between threads or actors in a single process, or idempotent tests.
As with SQLite, an external xxx that does this for you is certainly better, etc. but it’s convenient sometimes, to have an application that doesn’t go “now before you run this install Postgres…”.
It’s seldom useful for a web app where you control everything.
In practice, mostly scaling sessions and ephemeral data (caching) across multiple intances of a microservice on multiple machines. Seperating the kv store and the application allows upgrading each application while retaining availability and avoiding loss of session data.
I mostly use redis for pub/sub communication between services. If the app wasn't a collection of knative functions, and instead a monolith, it would be cool to also use redis for event based communication.
For simple cases, it is probably a total overkill to even consider it, but for something heavier, embedding the database gives you a chance to trivially migrate later to a separate database server.
A key-value database, or key-value store, is a data storage paradigm designed for storing, retrieving, and managing associative arrays, a data structure more commonly known today as a dictionary.
> One, it would be cool to be able to embed it, similar to sqlite, directly into applications.
I've found myself wanting this on several occasions too. I.e. wanting all my rust backend processes (k8s pods) to have some minimal shared state, without having to spin up a Redis cluster. I've talked to Claude about it a couple of times, and it descends into something like, "you gotta use Raft or CRDTs, and pick 2 out of 3 from CAP". Which honestly seems pretty fair, and indicates to me that I'm dreaming for something magical.
Nonetheless, it is nice to hear someone else asking for this. If this is indeed feasible (even if simple/limited), then I'd be interested to try it.
I don't know if that'll make you feel any better but yeah, you're indeed asking for the impossible! You need consensus between your nodes that store state _somehow_, either these nodes are Redis and it does that for you, or these nodes are your pods and you need to do consensus yourself (zookeeper might help, but you're definitely in "complicated stuff" territory).
Spinning up an in-memory (no persistence) Redis cluster in your k8s should be easy enough, hopefully?
And yes, adding a Redis cluster is fine, it is just another moving part to manage. But given that the alternative is made out of unobtainium, I guess that is just the way of it :-)
Genuinely interested why we need HA in redis, just not read round robin from multiple non-HA instances?
Redis (and memcache) are memory caches and should be treated like that, not like highly consistent distributed session store.
> Redis (and memcache) are memory caches and should be treated like that
If you haven't come across Kvrocks yet, it may be worth a look: https://github.com/apache/kvrockshttps://kvrocks.apache.org/ . It's a database with a Redis-compatible wire protocol, but the database is stored on disk. This means your working set is not limited by RAM and can be a few orders of magnitude larger! On modern SSDs this is still very fast. I think it improves the durability story as well. But the big win is the orders of magnitude larger database space.
As I've been improving my side project https://totalrealreturns.com/ recently I've ended up using both Redis and Kvrocks together. Redis is great for small global state that needs to be super fast. Kvrocks is great for larger bulk data storage (large precomputed datasets), but also supports a lot of the Redis data structures as well as Lua scripts.
It’s not. Imagine a web app that stores your user information in a session store, mapped by your cookie-provided session ID. Your web app searches redis 1 for the session id, but since that key is on redis 2, the lookup fails and the application thinks there is no such session, and rejects the request.
Now you could solve this specific case by sharding by prefix, or by querying all instances, but then you still do not have high availability: if the instance a specific session is on is down, these users cannot authenticate. At that point you’re better off with a single instance.
But that is his point.
If you cannot find the session id in redis, you login again.
If your Redis server crash, you start a new one and everyone just login again. No data is lost.
No two processes can guarantee data consistency unless using shared memory with some kind of locking on update. And given two servers don't share memory, two processes running on these servers can not guarantee consistency either.
To put the simple terms...
App writes to node-A, node-A (/process on node-A) crashes before change is synced from node-A to node-B, data is lost.
This is true for redis and true for postgresql/ mysql or any similar database. Difference between redis and a "database" is that database protects against this problem by writing change to durable storage before telling app that write is successful. Redis
First up, if I wanted to talk to a machine, I would've asked one myself.
Then, I don't understand your point really: Yes, the CAP theorem is a thing. There are compromise solutions available however to enable highly available data storage. Some of them for Redis too, but they are more complicated than those for other database engines. Which is the point of this discussion.
Point is... with AOF and RDB enabled, and wait command used in sane manner, one can get reasonable consistency with a significant speed tradeoff and increase in application complexity. So if consistent cache is needed, one can have that with some compromises, but then probably one could use a database straight away.
Again: Redis is a database, not just a cache - it doesn’t care if you store ephemeral cache artifacts or customer records within it. Redis doesn’t pose any constraints on the type of data. Inversely, you can use Postgres as a semi or fully consistent cache.
And yes, what you’re saying is technically correct, even a well-tuned single node doesn’t solve the availability problem: if it goes down, you have an outage. To avoid that, you need multiple instances to provide the same data, avoiding downtime if one of the nodes breaks eventually.
> doesn’t pose any constraints on the type of data.
logic, raw disk is also a database. One just need to add block level replication to other nodes to build a replicated/ HA database.
We may not agree, but anything not providing transactions across logically related multiple data read/ update operations is not a database.
> multiple instances to provide the same data
is easy and done by a bunch of software out there, but
> multiple instances to provide the same data on non-shared memory computers, with consistency
is a really hard problem, and no one has been able to solve it yet without introducing other problems to be considered (giving up on fast performance being one of the most visible one) by architects/ developers.
This discussion is a bit weird. We started off from, Redis should have better availability guarantees. Specifically to avoid the degradation of service you described.
But that requires running on multiple instances, which in turn requires to share the data across all replicas.
These two concerns are not mutually exclusive, the kind of database or data stored within it doesn't give any availability guarantees on its own. Even a single Postgres instance, which I suppose fits your understanding of a real database, is a single point of failure and not a highly available setup: If your database server goes down, clients get errors and the database is thus unavailable.
> The app would look up in both databases. If it exists in any, there would be a session.
And if you find the session with differing values in both databases, how do you know which one is up-to-date?
You need an algorithm to pick which data is right, such as electing a master instance.
And that brings us back to the original discussion: to manage sessions (unlike caches) in a highly available way, you need to setup HA (or reimplement it, which obviously is a bad idea). You can't read round robin from multiple non-HA instances.
For the project I've been working on for more than 15 years, we make extensive use of the pub/sub functionality for distributing live data. Pub/sub scales well across the cluster. Publish to one, and it goes out to subscribers on any of the nodes that they've connected to.
Will millions of users, high availability is critical for this functionality.
Redis doesn't necessarily have to be used as a cache. Streams, for example, make it a great message queue; but a single-node message queue is a single point of failure and thus not viable for many setups.
That you do. Until you realise that there is only a single writer in that scenario, it doesn’t address any sharding concerns, you need to use compatible clients that opt into the sentinel protocol, during failover you’ll see client errors… there’s lots of room for improvement on redis HA.
With the amount of problems I had using Redis Sentinel, I really wish there was another way. On multiple occasions, with completely different deployments, it got itself into a non-repairable state where the only option was to drop it and setup the replicas manually. I was hoping someone would do a Patroni-like project for Redis, but I've not found it yet. I've moved all persistent data to PostgreSQL and use a number of Valkeys behind Envoy proxy as a cache.
I suggest you to take a look at rdsync (https://github.com/yandex/rdsync), exactly what you want: Patroni-like high-availability tool for Valkey/Redis. Uses ZooKeeper for external coordination. We use it in our large deployment and with a couple patches you will forged about the need to take manual actions to resolve broken states.
To be honest - at any scale, this really does help me not wake up at night to fix broken states by hand as sometimes on-call engineer. Although note that rdsync is mainly for Valkey up to 9.1, there were Redis patches for 7.2 (last BSD version).
Redis have many use cases, and acting as a cache is only one of them. One very common usage is as a backend for background worker jobs. That can need HA.
What kind of an answer is that? This software is perfect the way it is, you’re just to inept to hold it right?
A high availability protocol should not leak into the client. It should be able to discover other nodes. It should not land in broken states so easily. It should not limit the number of writers. It should not error during failover.
Are these hard problems? Yes. Should we just accept that things are hard because that’s how the gods have given them to us? No.
High availability and abstraction complexity are orthogonal.
Redis is a low-level concurrency primitive, and it made certain choices in dealing with CAP.
It might be single-threaded, but it can easily absorb 100,000+ requests per second.
I've built systems that handle billions of dollars of online payments flow, active-active, with six nines of uptime reliability on top of Redis. It does what it says on the tin, and it doesn't need to be everything for everybody. This is a hard domain and you're going to have to deal with different problems and tradeoffs.
If you want something higher level, there are other systems to reach for.
> Two, the HA story is so much more complicated than it should be.
Really? I am curious, how would simplify it? It’s a very well defined problem and all the “solutions” are very complicated and with many strings attached. I have managed one or two systems that came in different modalities and you had to pick your poison and had to make sure the other engineering teams understand the trade offs. Some were more successful than others, but “easy” never crossed my mind.
Redis is single threaded and doesn’t concern itself with these things, directly, exactly because Antitez understood the trade offs and made all the right choices.
How would you improve the HA story without sacrificing ease of use and performance on a single thread?
We switched to Valkey two years ago. I haven't really looked back. I think both projects have done a lot of nice stuff since the split but it's not really impacting anything I use. The feature set was fine five years ago and I don't think we're using anything in Valkey that wouldn't work in Redis. There are probably a lot of projects that never switched over because they had no real need.
But most of the cloud providers now offer Valkey because of the license changes. Of course, cloud providers not offering Redis was the intention of the license change from the Redis point of view. So mission accomplished for Redis.
But the flip side of course is that if you want to deploy on standard infrastructure rather than self hosting Redis, Valkey is now the easy, low risk path that probably should be the default for most companies that target AWS, Azure, GCP, etc. Same with Elasticsearch vs. Opensearch and a few other products where the community forked because of license changes.
Mentioning Elasticsearch because I know people in both communities and I'm deeply familiar with the stack. A few years on, Opensearch has taken a lot of the momentum from Elasticsearch.
Valkey, because our cloud provider is hosting it and that's obviously what they prefer.
I feel like we're using about 1% of its features at this point - really just as a fast K/V store - so it would be easy to switch if needed, but I can't see a case where we would.
We use almost exclusively Valkey now, mostly because we host on AWS and Render, which both use Valkey. It's faster, cheaper and compatible. I'd consider Garnet too but I believe it doesn't support LUA(or didn't at the time we needed it).
We switched to Valkey after the Redis license kerfuffle happened, discovered we were saving money on our AWS bill, and have no motivation to go back to Redis.
Valkey continues to be the real redis, released under a permissive open source license like redis originally was. The fork of redis known as redis because those that decided to fork it had the trademark at first wasn't under an OSI-certified open source license and now is under a non-permissive OSI-certified open source license.
There's also a separate blog post that goes into the details of why existing data structures Redis already supported, which could provide array-like behavior, weren't good enough:
@antirez wrote about the development of that data structure last month, which includes how he used LLMs to do it (which was before ds4 for the co-comment mentioning it ;). The PR he linked goes into the motivation.
Possibly, but the array type code was implemented using GPT/Claude models before DS4 was a thing. I really recommend this write up on how he used LLMs which I think is a more sane/safe way to code with them vs the YOLOing even I'm subject to unfortunately...
The experimental SSD streaming feature (author's demo @ https://x.com/antirez/status/2062536214675067322 - recently merged into the main branch) is great news for that project, allowing for SOTA inference (DeepSeek V4 Flash and Pro!) on RAM-limited machines. Now we need work on large-ish scale batching in order to recover tok/s under the SSD streaming scenario. It's not helpful when running normally (at least not on Apple Silicon) since thermal/power throttling is the constraint in that case, but SSD streaming is a whole other consideration.
> Rate limiting is one of the most common Redis use cases. Traditionally, users implemented rate limiters using server-side Lua scripts combined with client logic. In Redis 8.8, we introduce a window counter rate limiter (by @raffertyyu, together with the Redis team).
I had a look for this and it turns out it's slightly mis-described there - it's not a window counter, it's a "GCRA (Generic Cell Rate Algorithm)" - a leaky bucket algorithm. Code here: https://github.com/redis/redis/blob/unstable/src/gcra.c
The code comments say it was heavily influenced by https://github.com/brandur/redis-cell by Brandur Leach.
It's a neat algorithm (I just learned about it today) - it only needs to store a single integer for each rate-limited key, which is the "Theoretical Arrival Time" when the bucket would next be empty.
Also, the “cell” in Generic Cell Rate Algorithm is an ATM cell. GCRA is 1990s telecom, the scheduling algorithm ATM switches used to check that 53-byte cells were arriving on the wire at the agreed rate.
Asynchronous Transfer Mode (I had to look it up too.)
https://en.wikipedia.org/wiki/Asynchronous_Transfer_Mode
I've never liked token buckets (vs sliding window counters) due to the extra work CPU cycles required to "fill" them. It seems like doing an atomic incr on a key based on a `time % 1 minute` or something would be more efficient and then let that key TTL expire X duration later. This results in zero work for rate limiters not in use and only a single push change -> resulting count for ones in use. Nothing but setting the TTL extra is required.
Thanks for the links, I'll checkout the Generic Cell Rate Algorithm!
While I love Redis as a versatile tool for external data structures, it's still lacking in two areas IMHO:
One, it would be cool to be able to embed it, similar to sqlite, directly into applications.
Two, the HA story is so much more complicated than it should be. I totally acknowledge that concurrency and distributed computing is hard, but it should not require reading heaps of documentation and understanding two entirely separate multi-node approaches only to figure out there are lots of subtle strings attached that make it impractical for many applications.
What would be the point of embedding Redis into an application? What's the advantage of using Redis over using the builtin (or third party) data structures of the language the application is developed in?
I'm asking as a non-webdev who never quite got what Redis actually does, but would love to learn.
To me the thing I like about Redis is that it gives you a storage engine very suitable for caches; it handles TTLs and memory pressure, as well as built-in serialization with the ability to get better performance by allowing for some data loss. At the same time, many users will be deploying small programs to individual machines. If you could just have Redis be embedded this would make it very operationally simple: no additional daemons and a single file to backup if you want to.
It would also be useful because of the ability to switch modalities. When running a multi node service, you can use Redis to share data between nodes and use Redis pubsub as a communication bus. If you wanted to support a simple single node configuration too, then it wouldn't need to be a special case, it could just go through the same mechanism but with an embedded Redis instance.
It's pretty similar to SQLite: being able to embed more or less a complete storage engine into your app can be very convenient and powerful.
Well, if you have a single instance than using language libraries and structures will be better in most cases.
If you use multiple nodes, then you probably want your redis lifecycle not be tied to application lifecycle.
I am not aware of an in-process alternative similar to what Redis offers.
Well the most basic redis replacement would be just a global hashmap to replace GET and SET, possibly with a background thread to periodically delete expired keys. But obviously that stops working as soon as you get a second node.
The entire value of redis IMO is that is ISN'T inside your normal application, but rather some shared storage that all nodes can use to coordinate and that survives deploys, but that provides more ergonomic data structures than SQL databases. Caches are only one type of such shared data, but things like feature flags, circuit breakers and rate limiters are also super common (and super useful).
Neat. Write that up, match parity, and give all the function calls with the same name as redis, and you're both happy! You get to hand roll something, he gets to use a library that others have perfected over the years!
Ehm no? Do it yourself if you want. I'm already happy.
I would Cache with TTL? A cache that can store lists?
Mnesia, if you’re using Erlang or Elixir.
Unfortunately I have never really used Erlang outside of deploying RabbitMQ. I mostly use Go, Rust, Python, sometimes C/C++.
However, Mnesia seems like it is quite a bit more of a complete distributed database engine than Redis. To me the nicest thing about Redis is just the convenience of what it offers: very fast data structures, serialized, optimized (at least by default) for cases where speed is more important than durability. It is simple on many levels and somewhat constrained in scope. Mnesia seems to be aiming more generally in the distributed database category.
So how do you feel they compare?
Really it would be more like Nebulex/Cachex which provide a really nice caching interface across ETS (what Mnesia is built off of) or other data stores.
Probably because Redis gives you a very well-defined/understood set of rich data structures with built-in behavior like TTL, atomic operations, eviction, and persistence. These things are otherwise usually scattered across native types, helper classes, or entirely separate libraries.
It doesn’t seem like the right tool for the job, though. Aren’t your own programming language’s constructs much more well-defined / understood ?
Language's own native data-structures are generally much more capable and vast. 99%+ developers use only a very limited set of those capabilities. This approach packages those most used ones into a nice, consistent DSL.
It's similar in effect to what busybox does to shell utilities, though the motives are different.
Doesn’t depend on the language? Actually I am thinking of the standard library… Python’s in kinda huge and some are hard (for me) to grasp. Golangs seem pretty simple.
Language’s own native data structures isn’t limited to just the standard library.
agreed but depends on then language. for instance, the .NET equivalent (MemoryCache) is pretty poor.
Redis has some pretty useful primitive that many languages don't:
- HyperLogLog, bloom filter, other probabilistic data structures
- Geospatial operations on stored points and polygons
- Expiring keys, for creating caches
These aren't in most standard libraries, and the Redis implementations tend to be fast, robust and well understood.
Can you name a single language that can talk to redis and doesn't have these in a form of a library that integrates with an app better than mystical embedded redis?
Every language you can talk to redis most likely has a library to do that, and it probably works much better with the rest of application than "embedded redis". If it doesn't, it probably has C-FFI and there is "fast, robust and well understood" implementations in C.
Sure. But if Redis was embeddable you'd get a robust C-FFI style implementation of those data structures which has been tested a lot more than some random library that has almost no existing users or active maintenance.
(I'm not personally sold on embedded Redis myself, but the question was "Aren’t your own programming language’s constructs much more well-defined / understood?")
> you'd get a robust C-FFI style implementation of those data structures which has been tested a lot more than some random library that has almost no existing users or active maintenance.
What are these mystical random libraries you're talking about? There is a solid C implementation of every data structure on this planet.
Show me a better C library for HyperLogLog than the one baked into Redis.
I went looking and the pure C ones all appeared to be untouched in 10+ years - https://github.com/ivitjuk/libhll and https://github.com/avz/hll for example.
I use PHP. None of the language tools or constructs available to me are adequate.
https://blog.codinghorror.com/the-php-singularity/
And you want to embed Redis inside PHP as a solution?? That’s nuts.
Where else could they store their serialized PHP data structures? (just kidding)
A few nice things about doing this in no particular order:
Embedding would make local dev/CI integration testing convenient.
Embedding replicated Redis with each application instance would give you HA benefits while infra-management complexity.
Embedded redis (even via local RPC) is still going to be faster than a lot of languages or frameworks’ built-in data structures. Large array operations in, say, Python are gonna slower than RPCing to Redis (assuming that the data structures are built gradually and not built all at once); to beat Redis you’d have to use numpy or something—-which is definitely preferable, but is extra work if your app already uses Redis for other things.
Just like choosing SQLite over e.g. LMDB or RocksDB, embedded Redis would be a nice future proofing option for small apps during the prototype phase; less would have to be changed to move Redis out of the app than if a different cache or persistence service were chosen.
I have 15 processes handling API requests over HTTP. I want rate limiting, so users don't DoS my API. I need shared state across these processes to store information about rate limits for each client key. I store those in redis.
Or queues, caching, pub/sub... Redis can do a lot, it's really (and i mean really) fast, very easy to get up and running and the protocol is pretty simple.
Locality and latency.
Network hops are not free! Those milliseconds are an eternity compared to local function calls.
The optimal architecture is something like what Service Fabric or Orleans can do with their distributed dictionary types: reads are generally in-process and take only nanoseconds (but writes require a synchronous replica copy to a remote host.)
Obviously this requires load balancers to steer traffic consistently, but that’s a common feature… outside of the public clouds where they forgot latency exists.
Why would you embed SQLite?
It’s the same use case with a different api.
A typical (meaningful) example might be communication between threads or actors in a single process, or idempotent tests.
As with SQLite, an external xxx that does this for you is certainly better, etc. but it’s convenient sometimes, to have an application that doesn’t go “now before you run this install Postgres…”.
It’s seldom useful for a web app where you control everything.
In practice, mostly scaling sessions and ephemeral data (caching) across multiple intances of a microservice on multiple machines. Seperating the kv store and the application allows upgrading each application while retaining availability and avoiding loss of session data.
I mostly use redis for pub/sub communication between services. If the app wasn't a collection of knative functions, and instead a monolith, it would be cool to also use redis for event based communication.
For simple cases, it is probably a total overkill to even consider it, but for something heavier, embedding the database gives you a chance to trivially migrate later to a separate database server.
Redis is not a database. It’s a key / value store.
It kind of is a database:
A key-value database, or key-value store, is a data storage paradigm designed for storing, retrieving, and managing associative arrays, a data structure more commonly known today as a dictionary.
https://en.wikipedia.org/wiki/Key–value_database
that's still a database.
it's not a relational database.
you are confusing redis with memcached
> One, it would be cool to be able to embed it, similar to sqlite, directly into applications.
I've found myself wanting this on several occasions too. I.e. wanting all my rust backend processes (k8s pods) to have some minimal shared state, without having to spin up a Redis cluster. I've talked to Claude about it a couple of times, and it descends into something like, "you gotta use Raft or CRDTs, and pick 2 out of 3 from CAP". Which honestly seems pretty fair, and indicates to me that I'm dreaming for something magical.
Nonetheless, it is nice to hear someone else asking for this. If this is indeed feasible (even if simple/limited), then I'd be interested to try it.
I don't know if that'll make you feel any better but yeah, you're indeed asking for the impossible! You need consensus between your nodes that store state _somehow_, either these nodes are Redis and it does that for you, or these nodes are your pods and you need to do consensus yourself (zookeeper might help, but you're definitely in "complicated stuff" territory).
Spinning up an in-memory (no persistence) Redis cluster in your k8s should be easy enough, hopefully?
Yeah, fair enough.
And yes, adding a Redis cluster is fine, it is just another moving part to manage. But given that the alternative is made out of unobtainium, I guess that is just the way of it :-)
Genuinely interested why we need HA in redis, just not read round robin from multiple non-HA instances? Redis (and memcache) are memory caches and should be treated like that, not like highly consistent distributed session store.
> Redis (and memcache) are memory caches and should be treated like that
If you haven't come across Kvrocks yet, it may be worth a look: https://github.com/apache/kvrocks https://kvrocks.apache.org/ . It's a database with a Redis-compatible wire protocol, but the database is stored on disk. This means your working set is not limited by RAM and can be a few orders of magnitude larger! On modern SSDs this is still very fast. I think it improves the durability story as well. But the big win is the orders of magnitude larger database space.
As I've been improving my side project https://totalrealreturns.com/ recently I've ended up using both Redis and Kvrocks together. Redis is great for small global state that needs to be super fast. Kvrocks is great for larger bulk data storage (large precomputed datasets), but also supports a lot of the Redis data structures as well as Lua scripts.
Redis is used for plenty of things, not just memory caches.
For example if you use it for session storage, you can't have your application read from a random instance that may or may not contain the session.
This case is exactly what he talks about. To get HA just setup more than one redis cache - or rebuild the session if it was lost in the redis cache.
It’s not. Imagine a web app that stores your user information in a session store, mapped by your cookie-provided session ID. Your web app searches redis 1 for the session id, but since that key is on redis 2, the lookup fails and the application thinks there is no such session, and rejects the request.
Now you could solve this specific case by sharding by prefix, or by querying all instances, but then you still do not have high availability: if the instance a specific session is on is down, these users cannot authenticate. At that point you’re better off with a single instance.
But that is his point. If you cannot find the session id in redis, you login again. If your Redis server crash, you start a new one and everyone just login again. No data is lost.
Sure the data is lost. A session commonly holds arbitrary state, and even if it’s just the login information. This is ridiculous.
No two processes can guarantee data consistency unless using shared memory with some kind of locking on update. And given two servers don't share memory, two processes running on these servers can not guarantee consistency either.
To put the simple terms... App writes to node-A, node-A (/process on node-A) crashes before change is synced from node-A to node-B, data is lost.
This is true for redis and true for postgresql/ mysql or any similar database. Difference between redis and a "database" is that database protects against this problem by writing change to durable storage before telling app that write is successful. Redis
So if one wants to have a consistent session storage, one should use a proper database or use AOF redis persistense with single node (https://chatgpt.com/s/t_6a24ab818e2881918db959cec8d8cc2d)
First up, if I wanted to talk to a machine, I would've asked one myself.
Then, I don't understand your point really: Yes, the CAP theorem is a thing. There are compromise solutions available however to enable highly available data storage. Some of them for Redis too, but they are more complicated than those for other database engines. Which is the point of this discussion.
Point is... with AOF and RDB enabled, and wait command used in sane manner, one can get reasonable consistency with a significant speed tradeoff and increase in application complexity. So if consistent cache is needed, one can have that with some compromises, but then probably one could use a database straight away.
Again: Redis is a database, not just a cache - it doesn’t care if you store ephemeral cache artifacts or customer records within it. Redis doesn’t pose any constraints on the type of data. Inversely, you can use Postgres as a semi or fully consistent cache.
And yes, what you’re saying is technically correct, even a well-tuned single node doesn’t solve the availability problem: if it goes down, you have an outage. To avoid that, you need multiple instances to provide the same data, avoiding downtime if one of the nodes breaks eventually.
If redis is a database, then by
> doesn’t pose any constraints on the type of data.
logic, raw disk is also a database. One just need to add block level replication to other nodes to build a replicated/ HA database.
We may not agree, but anything not providing transactions across logically related multiple data read/ update operations is not a database.
> multiple instances to provide the same data
is easy and done by a bunch of software out there, but
> multiple instances to provide the same data on non-shared memory computers, with consistency
is a really hard problem, and no one has been able to solve it yet without introducing other problems to be considered (giving up on fast performance being one of the most visible one) by architects/ developers.
Obviously these are application decisions.
You, obviously, don't commit important data only to a session that you can loose, if the application does not allow it.
We use redis as infrastructure. To route events and as a cache.
For us redis could go down and we would merely see a degradation of our service with no data loss.
I recommend using redis like that. And then use a database that supports transactions for real data problems.
But we are different. And that's OK.
This discussion is a bit weird. We started off from, Redis should have better availability guarantees. Specifically to avoid the degradation of service you described.
But that requires running on multiple instances, which in turn requires to share the data across all replicas.
If you consider it important, you have to store it in a real database. No buts. If you don't consider it important, sharded redis works fine.
Redis is a real database. If I wasn’t convinced it could retain data I hand it, I wouldn’t use it in the first place.
Just because it works for your use case right now doesn’t mean there isn’t room for improvements to support others too.
> Redis is a real database
No, redis is a memory cache, with some ACID like features bolt on.
Even real databases have hard time maintaining consistency across nodes. CAP is a real constraint.
> Redis is a real database.
Oh good, then you don't need to do any of the stuff that you suggested to do
These two concerns are not mutually exclusive, the kind of database or data stored within it doesn't give any availability guarantees on its own. Even a single Postgres instance, which I suppose fits your understanding of a real database, is a single point of failure and not a highly available setup: If your database server goes down, clients get errors and the database is thus unavailable.
I don't think you understand what HA means.
The app would look up in both databases. If it exists in any, there would be a session.
Thisnis strictly different from partitioning which I think you are mixing it up with.
Paritioning is for performance not HA
> The app would look up in both databases. If it exists in any, there would be a session.
And if you find the session with differing values in both databases, how do you know which one is up-to-date?
You need an algorithm to pick which data is right, such as electing a master instance.
And that brings us back to the original discussion: to manage sessions (unlike caches) in a highly available way, you need to setup HA (or reimplement it, which obviously is a bad idea). You can't read round robin from multiple non-HA instances.
Yes, you are pointing out exactly how HA is difficult.
There is a whole slew of downstream things you need to take into consideration.
and as you've just confirmed... there is a need for good HA support in Redis. Which was the entire premise of this thread.
That’s the precise point I’m making
For the project I've been working on for more than 15 years, we make extensive use of the pub/sub functionality for distributing live data. Pub/sub scales well across the cluster. Publish to one, and it goes out to subscribers on any of the nodes that they've connected to.
Will millions of users, high availability is critical for this functionality.
Redis doesn't necessarily have to be used as a cache. Streams, for example, make it a great message queue; but a single-node message queue is a single point of failure and thus not viable for many setups.
That's why you run Redis Sentinel in production
That you do. Until you realise that there is only a single writer in that scenario, it doesn’t address any sharding concerns, you need to use compatible clients that opt into the sentinel protocol, during failover you’ll see client errors… there’s lots of room for improvement on redis HA.
With the amount of problems I had using Redis Sentinel, I really wish there was another way. On multiple occasions, with completely different deployments, it got itself into a non-repairable state where the only option was to drop it and setup the replicas manually. I was hoping someone would do a Patroni-like project for Redis, but I've not found it yet. I've moved all persistent data to PostgreSQL and use a number of Valkeys behind Envoy proxy as a cache.
I suggest you to take a look at rdsync (https://github.com/yandex/rdsync), exactly what you want: Patroni-like high-availability tool for Valkey/Redis. Uses ZooKeeper for external coordination. We use it in our large deployment and with a couple patches you will forged about the need to take manual actions to resolve broken states.
At which scale would you recommend this over a Sentinel setup?
To be honest - at any scale, this really does help me not wake up at night to fix broken states by hand as sometimes on-call engineer. Although note that rdsync is mainly for Valkey up to 9.1, there were Redis patches for 7.2 (last BSD version).
Redis have many use cases, and acting as a cache is only one of them. One very common usage is as a backend for background worker jobs. That can need HA.
Years ago I enabled durability on redis & used it as database for an online card game
Redka can be used embed, in Go: https://github.com/nalgeon/redka
> it's still lacking in two areas
This is entirely different than what Redis is and tries to solve.
Sqlite is embedded. It's not a distributed SQL. Redis is a distributed data structure store and concurrency primitive. These are worlds apart.
> HA story is so much more complicated than it should be
It is precisely as complicated as it needs to be. You don't want data loss.
If you're in the business of high available fault tolerance, you read the manual and learn how to Redis.
What kind of an answer is that? This software is perfect the way it is, you’re just to inept to hold it right?
A high availability protocol should not leak into the client. It should be able to discover other nodes. It should not land in broken states so easily. It should not limit the number of writers. It should not error during failover.
Are these hard problems? Yes. Should we just accept that things are hard because that’s how the gods have given them to us? No.
High availability and abstraction complexity are orthogonal.
Redis is a low-level concurrency primitive, and it made certain choices in dealing with CAP.
It might be single-threaded, but it can easily absorb 100,000+ requests per second.
I've built systems that handle billions of dollars of online payments flow, active-active, with six nines of uptime reliability on top of Redis. It does what it says on the tin, and it doesn't need to be everything for everybody. This is a hard domain and you're going to have to deal with different problems and tradeoffs.
If you want something higher level, there are other systems to reach for.
> Two, the HA story is so much more complicated than it should be.
Really? I am curious, how would simplify it? It’s a very well defined problem and all the “solutions” are very complicated and with many strings attached. I have managed one or two systems that came in different modalities and you had to pick your poison and had to make sure the other engineering teams understand the trade offs. Some were more successful than others, but “easy” never crossed my mind.
Redis is single threaded and doesn’t concern itself with these things, directly, exactly because Antitez understood the trade offs and made all the right choices.
How would you improve the HA story without sacrificing ease of use and performance on a single thread?
[dead]
Where did everyone end up on the Redis/Valkey split? Is there still a reason to use Redis after the license kerfuffle?
We switched to Valkey two years ago. I haven't really looked back. I think both projects have done a lot of nice stuff since the split but it's not really impacting anything I use. The feature set was fine five years ago and I don't think we're using anything in Valkey that wouldn't work in Redis. There are probably a lot of projects that never switched over because they had no real need.
But most of the cloud providers now offer Valkey because of the license changes. Of course, cloud providers not offering Redis was the intention of the license change from the Redis point of view. So mission accomplished for Redis.
But the flip side of course is that if you want to deploy on standard infrastructure rather than self hosting Redis, Valkey is now the easy, low risk path that probably should be the default for most companies that target AWS, Azure, GCP, etc. Same with Elasticsearch vs. Opensearch and a few other products where the community forked because of license changes.
Mentioning Elasticsearch because I know people in both communities and I'm deeply familiar with the stack. A few years on, Opensearch has taken a lot of the momentum from Elasticsearch.
For those who may not know, you can cut your costs in AWS by going with Valkey over Redis for about 33% savings.
https://aws.amazon.com/blogs/database/reduce-your-amazon-ela...
We're planning to go to Valkey for exactly that reason.
But what about Geico?
It's so easy a grug brain can do it.
I've switched to Valkey and I'm not really looking back. I'm much more comfortable with those people maintaining the software.
Valkey, because our cloud provider is hosting it and that's obviously what they prefer.
I feel like we're using about 1% of its features at this point - really just as a fast K/V store - so it would be easy to switch if needed, but I can't see a case where we would.
They prefer it because they don't have to pay to use it.
Sure. I prefer that because they charge me less.
We use almost exclusively Valkey now, mostly because we host on AWS and Render, which both use Valkey. It's faster, cheaper and compatible. I'd consider Garnet too but I believe it doesn't support LUA(or didn't at the time we needed it).
https://microsoft.github.io/garnet/docs/welcome/compatibilit...
> Garnet now has full-fledged and efficient support for Lua scripting. You can enable Lua with the --lua switch.
We switched to Valkey after the Redis license kerfuffle happened, discovered we were saving money on our AWS bill, and have no motivation to go back to Redis.
So we’ve stayed with Valkey.
Valkey continues to be the real redis, released under a permissive open source license like redis originally was. The fork of redis known as redis because those that decided to fork it had the trademark at first wasn't under an OSI-certified open source license and now is under a non-permissive OSI-certified open source license.
I'm taking liberties with the concept of forking.
We're a self hosted shop, we went with Valkey. Valkey also has support for RDMA, which we already is running in our infrastructure.
Went with 100% ValKey, if you are solely on AWS it is a no-brainer
Most people seems to have switched to Valkey, and it's backed by the Linux foundation.
We went with DragonFlyDB
Are you happy with it? Is it faster as claimed?
There's also a separate blog post that goes into the details of why existing data structures Redis already supported, which could provide array-like behavior, weren't good enough:
https://redis.io/blog/diving-deep-into-rediss-new-array-data...
@antirez wrote about the development of that data structure last month, which includes how he used LLMs to do it (which was before ds4 for the co-comment mentioning it ;). The PR he linked goes into the motivation.
https://antirez.com/news/164
https://github.com/redis/redis/pull/15162
given his ds4 project, likely collaborated with DeepSeek for this release:
https://github.com/antirez/ds4
Possibly, but the array type code was implemented using GPT/Claude models before DS4 was a thing. I really recommend this write up on how he used LLMs which I think is a more sane/safe way to code with them vs the YOLOing even I'm subject to unfortunately...
https://antirez.com/news/164
The experimental SSD streaming feature (author's demo @ https://x.com/antirez/status/2062536214675067322 - recently merged into the main branch) is great news for that project, allowing for SOTA inference (DeepSeek V4 Flash and Pro!) on RAM-limited machines. Now we need work on large-ish scale batching in order to recover tok/s under the SSD streaming scenario. It's not helpful when running normally (at least not on Apple Silicon) since thermal/power throttling is the constraint in that case, but SSD streaming is a whole other consideration.
Are we still using Redis? License change, no more Kube operators.
window counter rate limiter!
This is awesome!
And arrays look great too. Lots to play with.
[dead]
And here we see the reason for the sudden AI enthusiasm of Redis authors: array data structures are used in AI. This was clear weeks ago.
The website looks like openclaw's website.