With Litestream v0.5.0 doing the replication, you'll get some transient read failures that really shouldn't happen.
If you build Litestream from head (or wait for v0.5.1), that'll be fixed. You'll still have similar issues around the time of a snapshot, if you keep only one snapshot.
Other than that, I'm pretty confident it just works. But the performance probably sucks. I need to add a layer of caching. But that's it.
The idea behind Litestream is that you have a “primary” which has the SQLite database on a local disk, and you use Litestream to replicate it asynchronously to (e.g.) object storage.
What's in object storage is not an SQLite file, but enough information to restore your SQLite file at various points in time. The way it's stored is optimized for frequent incremental updates and less frequent point restores.
The lightweight read replicas VFS allows you open an SQLite database read-only directly from the data in object storage, without having to first do a point restore, downloading all the data, and creating a local copy of the database.
You “stream” the data directly from object storage as needed, get a consistent view of it, and can query it while the “primary” is concurrently updating it.
It works, but SQLite kinda expects data to be local and low latency, which this might not be.
SQLite has a page cache, which you can use, but everytime there is a write, the entire cache is dropped.
Which means if you poll for updates regularly (which for various reasons you have to) you'll get frequent latency spikes.
One way to improve this is to add a page cache that has an understanding of which pages have changed and which have stayed the same. Prefetching would help to, but SQLite is really unhelpful there.
This is the piece I'm still missing. The rest mostly works already with Litestream v0.5.1.
To clarify, SQLite does this for local databases in WAL mode, that's what the shared memory WAL index file is for: it allows multiple processes to cooperate and figure out which pages changed since when, which also helps SQLite manage its page cache more efficiently.
Rollback mode needs to drop its page cache whenever there is any change to the database, because what changed – and what didn't – is not stored anywhere.
Confusingly, the read-replica VFS needs to pretend the database is in rollback mode even if it isn't. Otherwise, we'd have to fake a database file, a WAL and a WAL index, instead of just a database.
The change log is massive, so this is not unexpected to have a few issues. Maybe the two-year backlog of features and updates should have been batched into a few releases instead of this one big one? But there was a beta, and this is ripping the bandaid.
Litestream author here. Yes, that's correct. LTX was the biggest hurdle to get over and was impossible to switch over incrementally. The storage layer change brings a lot of benefits and enables a lot of future work that we're really excited about.
Thanks to mtlynch and everyone else who has submitted bug reports. We're squashing issues and working to get everything stable as quickly as possible.
Feels like a side-effect of forever 0.x version symptom (I am guilty of as well). Even though semi-ver says 0.x can do whatever, people don't associate enough disruptive changes to it, whereas 0.4.x if it is 1.x, then it is much clearer this is a 2.x release.
All things considered, this is probably just a tiny footnote in this software's life.
> One of the benefits of Litestream 0.5.0 is that there’s now an official litestream Docker image. All of my previous Docker containers required a lot of boilerplate to download the correct version of Litestream and make it available in my container, but now it reduces to a single Dockerfile line
There’s been an official Litestream container image for over 3 years at this point (since version 0.3.4, it’s at the same Docker Hub as 0.5.0).
Oh, thanks for the correction! I can't believe I never noticed that. I've updated the post.
After your comment, I thought, "Oh, I should contribute a PR to the repo to add the Docker badge so the Docker image is obvious to everyone," but it turns out the badge has been right there for four years.[0]
What I suspect happened is that I tried to use the Litestream Docker image once, discovered that image was amd64-only (until 0.3.9), so I didn't use it because I needed ARM, and then I just kept copy/pasting my workaround from project to project.
Extremely helpful. I've been eagerly awaiting v0.5 but have been holding off on deploying it until I had more confidence that it would work and be stable. Reading this, I'm definitely glad that I waited.
No, I saw that bug in the issue log, but I never hit that personally.
If you're running into it, I'd test again with the latest release (0.5.1) and see if it's still present. If so, I suspect it would get more traction from the dev team if the report was complete, as it's currently missing repro steps and the litestream.yml is not compatible with 0.5.0 (it still has "replicas" plural).
I've been trying to get a headstart on the lightweight read replicas for my Go SQLite driver and, yes, there are issues.
https://fly.io/blog/litestream-revamped/#lightweight-read-re...
But the concept is really cool, and it's workable. I already have a version that mostly works.
https://github.com/ncruces/go-sqlite3/tree/litestream/litest...
With Litestream v0.5.0 doing the replication, you'll get some transient read failures that really shouldn't happen.
If you build Litestream from head (or wait for v0.5.1), that'll be fixed. You'll still have similar issues around the time of a snapshot, if you keep only one snapshot.
Other than that, I'm pretty confident it just works. But the performance probably sucks. I need to add a layer of caching. But that's it.
"But the performance probably sucks" The performance of what sucks? Replication? Why would you need to cache that?
The idea behind Litestream is that you have a “primary” which has the SQLite database on a local disk, and you use Litestream to replicate it asynchronously to (e.g.) object storage.
What's in object storage is not an SQLite file, but enough information to restore your SQLite file at various points in time. The way it's stored is optimized for frequent incremental updates and less frequent point restores.
The lightweight read replicas VFS allows you open an SQLite database read-only directly from the data in object storage, without having to first do a point restore, downloading all the data, and creating a local copy of the database.
You “stream” the data directly from object storage as needed, get a consistent view of it, and can query it while the “primary” is concurrently updating it.
It works, but SQLite kinda expects data to be local and low latency, which this might not be.
SQLite has a page cache, which you can use, but everytime there is a write, the entire cache is dropped.
Which means if you poll for updates regularly (which for various reasons you have to) you'll get frequent latency spikes.
One way to improve this is to add a page cache that has an understanding of which pages have changed and which have stayed the same. Prefetching would help to, but SQLite is really unhelpful there.
This is the piece I'm still missing. The rest mostly works already with Litestream v0.5.1.
That's a rabbit hole. Thank you for sharing so much context. I see clearly that SQLite needs a page-updated-aware page cache, even if it was optional.
To clarify, SQLite does this for local databases in WAL mode, that's what the shared memory WAL index file is for: it allows multiple processes to cooperate and figure out which pages changed since when, which also helps SQLite manage its page cache more efficiently.
Rollback mode needs to drop its page cache whenever there is any change to the database, because what changed – and what didn't – is not stored anywhere.
Confusingly, the read-replica VFS needs to pretend the database is in rollback mode even if it isn't. Otherwise, we'd have to fake a database file, a WAL and a WAL index, instead of just a database.
And, meanwhile, v0.5.1 has been released. :)
The change log is massive, so this is not unexpected to have a few issues. Maybe the two-year backlog of features and updates should have been batched into a few releases instead of this one big one? But there was a beta, and this is ripping the bandaid.
https://github.com/benbjohnson/litestream/releases/tag/v0.5....
I think the most disruptive part is the migration to the new LTX format,[0] which I'd imagine is hard to do incrementally.
[0] https://fly.io/blog/litestream-v050-is-here/#the-ltx-file-fo...
Litestream author here. Yes, that's correct. LTX was the biggest hurdle to get over and was impossible to switch over incrementally. The storage layer change brings a lot of benefits and enables a lot of future work that we're really excited about.
Thanks to mtlynch and everyone else who has submitted bug reports. We're squashing issues and working to get everything stable as quickly as possible.
Feels like a side-effect of forever 0.x version symptom (I am guilty of as well). Even though semi-ver says 0.x can do whatever, people don't associate enough disruptive changes to it, whereas 0.4.x if it is 1.x, then it is much clearer this is a 2.x release.
All things considered, this is probably just a tiny footnote in this software's life.
They jumped from v0.3.x to v0.5.0 after a couple of years of v0.3.x (with an unreleased v0.4.x in between).
That alone should hint everyone it was a big leap.
Thanks for the heads up. I encountered similar issues so staying on 0.3.13 for now. But excited to give 0.5.x a go once the dust settles.
Livestream, sqlite and caddy are incredibly at making single box/vps ops a breeze.
> One of the benefits of Litestream 0.5.0 is that there’s now an official litestream Docker image. All of my previous Docker containers required a lot of boilerplate to download the correct version of Litestream and make it available in my container, but now it reduces to a single Dockerfile line
There’s been an official Litestream container image for over 3 years at this point (since version 0.3.4, it’s at the same Docker Hub as 0.5.0).
Oh, thanks for the correction! I can't believe I never noticed that. I've updated the post.
After your comment, I thought, "Oh, I should contribute a PR to the repo to add the Docker badge so the Docker image is obvious to everyone," but it turns out the badge has been right there for four years.[0]
What I suspect happened is that I tried to use the Litestream Docker image once, discovered that image was amd64-only (until 0.3.9), so I didn't use it because I needed ARM, and then I just kept copy/pasting my workaround from project to project.
[0] https://github.com/benbjohnson/litestream/commit/6acfbcbc64d...
Extremely helpful. I've been eagerly awaiting v0.5 but have been holding off on deploying it until I had more confidence that it would work and be stable. Reading this, I'm definitely glad that I waited.
mtlynch, since you're a litestream+backblaze user, did you encounter this with 0.5.0? https://github.com/benbjohnson/litestream/issues/747
No, I saw that bug in the issue log, but I never hit that personally.
If you're running into it, I'd test again with the latest release (0.5.1) and see if it's still present. If so, I suspect it would get more traction from the dev team if the report was complete, as it's currently missing repro steps and the litestream.yml is not compatible with 0.5.0 (it still has "replicas" plural).
No need for all the “I love Litestream” disclaimers, these are serious issues that speak for themselves.