Hi HN, I'm Sagi, founder of Varve (https://varve.io). For the past few months I've been working with S3 buckets in the Registry of Open Data on AWS. Some of them are enormous, with many billions of objects, and don't have a public inventory. Listing them serially with the AWS CLI would have taken days, or sometimes weeks, and if the process fails in the middle it's not always easy to resume from the right place without losing data.
I tested a bunch of existing tools that list S3 buckets in parallel (s3-fast-list, s3p, s7cmd and others). Each of them worked well in particular cases and crashed hard in others.
The challenge here is that S3 lists keys serially: you get the continuation token for page N+1 from page N, and each page contains up to 1000 keys. There's no API that tells you how many keys exist between two points, so you can't divide the keyspace evenly up front. The most common naive approach is to list each prefix individually, but this only works well for buckets that have a uniform key distribution. Another creative technique is using recursive bisection, that is, generating synthetic midpoint keys between two known keys.
Swath was born by researching how existing parallel listers work: the algorithms they use for splitting the keyspace, when they work well and when they fail, and then synthesizing those findings into a new architecture that combines those techniques into something that works well on most bucket shapes out of the box. Swath combines prefix seeding with adaptive work stealing, using observed density, structure probes and bounded bisection to place splits.
Swath also has a lot of operational goodies like crash-safe checkpoint and resume, very detailed metrics and the ability to write its output in JSONL/TSV/Parquet formats. It also supports producing globally sorted Parquet output. There's also a replay server that can take those sorted outputs and serve them back in an S3-compatible API, allowing testing of Swath and other tools against pathological buckets locally without hitting S3 again.
Swath was also an experiment in how far I could push AI-assisted development on systems code. Everything about its development was AI-driven, from the initial study of the existing tools and their shortcomings, through designing the new architecture and implementing it, to iterating in a loop and testing against hundreds of buckets, fine-tuning the algorithm until it handled any bucket that I threw at it. I'm thinking about writing more about the process and maybe releasing some of the transcripts and internal development notes later if this is of interest.
This is the first public release of Swath. I'm sure it's not perfect yet but I trust the list and resume paths enough to put them in other people's hands. I'm most interested in buckets that break it. If you run it on a large or unusual bucket and it doesn't do well, I'd love to hear about it.
Hi HN, I'm Sagi, founder of Varve (https://varve.io). For the past few months I've been working with S3 buckets in the Registry of Open Data on AWS. Some of them are enormous, with many billions of objects, and don't have a public inventory. Listing them serially with the AWS CLI would have taken days, or sometimes weeks, and if the process fails in the middle it's not always easy to resume from the right place without losing data.
I tested a bunch of existing tools that list S3 buckets in parallel (s3-fast-list, s3p, s7cmd and others). Each of them worked well in particular cases and crashed hard in others.
The challenge here is that S3 lists keys serially: you get the continuation token for page N+1 from page N, and each page contains up to 1000 keys. There's no API that tells you how many keys exist between two points, so you can't divide the keyspace evenly up front. The most common naive approach is to list each prefix individually, but this only works well for buckets that have a uniform key distribution. Another creative technique is using recursive bisection, that is, generating synthetic midpoint keys between two known keys.
Swath was born by researching how existing parallel listers work: the algorithms they use for splitting the keyspace, when they work well and when they fail, and then synthesizing those findings into a new architecture that combines those techniques into something that works well on most bucket shapes out of the box. Swath combines prefix seeding with adaptive work stealing, using observed density, structure probes and bounded bisection to place splits.
Swath also has a lot of operational goodies like crash-safe checkpoint and resume, very detailed metrics and the ability to write its output in JSONL/TSV/Parquet formats. It also supports producing globally sorted Parquet output. There's also a replay server that can take those sorted outputs and serve them back in an S3-compatible API, allowing testing of Swath and other tools against pathological buckets locally without hitting S3 again.
Swath was also an experiment in how far I could push AI-assisted development on systems code. Everything about its development was AI-driven, from the initial study of the existing tools and their shortcomings, through designing the new architecture and implementing it, to iterating in a loop and testing against hundreds of buckets, fine-tuning the algorithm until it handled any bucket that I threw at it. I'm thinking about writing more about the process and maybe releasing some of the transcripts and internal development notes later if this is of interest.
This is the first public release of Swath. I'm sure it's not perfect yet but I trust the list and resume paths enough to put them in other people's hands. I'm most interested in buckets that break it. If you run it on a large or unusual bucket and it doesn't do well, I'd love to hear about it.