I was expecting this to say “S3 append is buggy” or something. But really it just is saying “my specific application cannot replace making new files with append for a very specific reason”
The issue cited is that using append instead of writing a new file means losing put-if-absent consistency guarantee.
So… just don’t do that? Make the rule that a leader can append to a file it already wrote, but new process or newly elected leader must always start the next file in the sequence, and don’t design your system so that the append use-case needs at-most-once delivery or something. You can put a causal clock in each append chunk header or something if you want to stay as close as possible to the original system design.
Like, I will happily write pages of a WAL using append with no issues as long as my system isn’t multi-leader. ¯\_(ツ)_/¯
Ok, S3 is not a file system replacement. Trying to fit a square peg into a round hole and claiming there are “issues” with the peg is disingenuous.
S3 has and always will be object storage, that’s how it’s architected, and unless there’s a full rewrite, effectively making it something else, that’s not going to change.
Multipart upload is implemented as a grouping of a list of objects. This append operation is implemented the same way. It does share the name with the block storage append, but it’s a different use case.
Think of this as the objects being comprised as a list of sub objects, like a multipart upload. This feature allows you to add new sub-objects to the end of the list. Same rules continue to apply.
There are some cases where this is useful, like logs batched with Kinesis or any other batcher, live streaming, chunked video footage, etc. For those use cases it works fine. It’s not a file system.
Don’t hold it like that. It’s not a replacement for a disk. A simple fix is to ensure a single writer per object, e.g keyed by a unique node id. You can then merge objects on the backend using some aggregation process (if that is even necessary for your use case).
I was expecting this to say “S3 append is buggy” or something. But really it just is saying “my specific application cannot replace making new files with append for a very specific reason”
The issue cited is that using append instead of writing a new file means losing put-if-absent consistency guarantee.
So… just don’t do that? Make the rule that a leader can append to a file it already wrote, but new process or newly elected leader must always start the next file in the sequence, and don’t design your system so that the append use-case needs at-most-once delivery or something. You can put a causal clock in each append chunk header or something if you want to stay as close as possible to the original system design.
Like, I will happily write pages of a WAL using append with no issues as long as my system isn’t multi-leader. ¯\_(ツ)_/¯
Ok, S3 is not a file system replacement. Trying to fit a square peg into a round hole and claiming there are “issues” with the peg is disingenuous.
S3 has and always will be object storage, that’s how it’s architected, and unless there’s a full rewrite, effectively making it something else, that’s not going to change.
Multipart upload is implemented as a grouping of a list of objects. This append operation is implemented the same way. It does share the name with the block storage append, but it’s a different use case.
Think of this as the objects being comprised as a list of sub objects, like a multipart upload. This feature allows you to add new sub-objects to the end of the list. Same rules continue to apply.
There are some cases where this is useful, like logs batched with Kinesis or any other batcher, live streaming, chunked video footage, etc. For those use cases it works fine. It’s not a file system.
Don’t hold it like that. It’s not a replacement for a disk. A simple fix is to ensure a single writer per object, e.g keyed by a unique node id. You can then merge objects on the backend using some aggregation process (if that is even necessary for your use case).
I happen to know the content on this blog, the tone of this article and quality of content is not usually like this. What happened?