Have you considered using this as a test bed to explore alternative architectures and approaches? The classical Volcano architecture used here has been established for many decades, and it could be fun exercise to see what can be done to improve on it, while remaining true to the original goal of having something that's purely for educational purposes.
It’s the first productionization of leaderless consensus protocols. It merges strict serializable transactions into the consensus layer (in 1RTT on the fast path), and has sharding built-in including cross-shard transactions.
It doesn’t support interactive transactions though, so it’s not a drop-in for SQL systems (but I don’t think they are suitable in modern systems anyway, given the latency cost). And it requires quorum reads, since it can’t use leader leases (for obvious reasons), so reads may end up being too slow for many georeplicated setups. But it’s a neat design.
Super cool project and the code is delightfully readable as well!
I had the pleasure of having Erik advise me for my thesis on distributed databases a few years ago. I’m very lucky to have had the chance to tap into his knowledge on all things distributed systems. On top of being super knowledgeable he’s also a terrific teacher!
Thanks a lot, Erik. I learnt so much from digging into your code and attempting to implement parts of it by referring to your code, especially around Raft. Looking forward to doing it again. Absolutely fantastic work.
It's been abandoned for some years, the author was working on a new engine for it and in the last 5 days they started working again on sled proper. However, it's pretty good the way it is (the 0.34.7 release from 2021 https://crates.io/crates/sled), despite the beta warnings
It already uses a key/value store for on-disk storage, but you’ll have to write the server API and client yourself, along with a Raft state machine layer. It’s not a big lift though, and could make a fun weekend project.
Hi, Erik! Nice to see this on the HN front page.
Have you considered using this as a test bed to explore alternative architectures and approaches? The classical Volcano architecture used here has been established for many decades, and it could be fun exercise to see what can be done to improve on it, while remaining true to the original goal of having something that's purely for educational purposes.
Hi Alex!
I quite like the Volcano model, it has a certain elegance to it. :) With vectorization and distribution you can get pretty far with it.
For my next real project, I want to build something based on the Accord consensus protocol from Cassandra: https://cwiki.apache.org/confluence/download/attachments/188...
It’s the first productionization of leaderless consensus protocols. It merges strict serializable transactions into the consensus layer (in 1RTT on the fast path), and has sharding built-in including cross-shard transactions.
It doesn’t support interactive transactions though, so it’s not a drop-in for SQL systems (but I don’t think they are suitable in modern systems anyway, given the latency cost). And it requires quorum reads, since it can’t use leader leases (for obvious reasons), so reads may end up being too slow for many georeplicated setups. But it’s a neat design.
Super cool project and the code is delightfully readable as well!
I had the pleasure of having Erik advise me for my thesis on distributed databases a few years ago. I’m very lucky to have had the chance to tap into his knowledge on all things distributed systems. On top of being super knowledgeable he’s also a terrific teacher!
Thanks for the kind words, Fabian — it was fun working together on your thesis!
There were quite a few interesting projects linked to in the prior comment threads from 2021 https://news.ycombinator.com/item?id=27874992
Really cool project, your code is beautifully neat and commented. I'm also building a Rust DB so was really nice reading through yours
Thanks a lot, Erik. I learnt so much from digging into your code and attempting to implement parts of it by referring to your code, especially around Raft. Looking forward to doing it again. Absolutely fantastic work.
Love it, though if it’s sufficiently instructive, I bet it will go beyond education.
Is there a way to use this as a simple key value store without sql?
Maybe consider sled? https://github.com/spacejam/sled
It's been abandoned for some years, the author was working on a new engine for it and in the last 5 days they started working again on sled proper. However, it's pretty good the way it is (the 0.34.7 release from 2021 https://crates.io/crates/sled), despite the beta warnings
More info here https://dbdb.io/db/sled
(Note, non affiliated to the author, just liked the project)
These days I've been using fjall, which has the benefit of being actively worked on: https://github.com/fjall-rs/fjall
It already uses a key/value store for on-disk storage, but you’ll have to write the server API and client yourself, along with a Raft state machine layer. It’s not a big lift though, and could make a fun weekend project.
Simple question but is this OLAP or OLTP?
OLTP