Hi HN, I'm Isaac. This is Duckstring, an open source data engineering platform, using DuckDB as the engine. I've built it to address some of the main frustrations I've had in my years as a data engineer: modularising logic so that it can grow in complexity over time, methods for upgrading components without breaking everything, and implying the DAG through dependencies so that I don't have to manage it.
I'm convinced DuckDB is the only engine needed to cover most real-world data pipelines. I've tested it on data scales up to terabytes, and running on appropriately-sized single machines, it's consistently beaten distributed engines like Spark. I think the age of distributed computing for most pipelines might be waning.
- Data transformations are treated like software packages, allowing the DAG to be implied by dependencies
- A pull-based orchestration system, throttling every process both upstream and downstream of the bottleneck so that nothing runs if it can't be consumed
- An incremental framework based on DBSP that handles updates and deletes efficiently
- CLI-first, so that you can point your own AI agents at the platform rather than relying on a janky integration
The link is to an intro story, which passes on to a Playground app to help get a feel for the orchestration method. Link if you want that directly:
https://playground.duckstring.com
Hi HN, I'm Isaac. This is Duckstring, an open source data engineering platform, using DuckDB as the engine. I've built it to address some of the main frustrations I've had in my years as a data engineer: modularising logic so that it can grow in complexity over time, methods for upgrading components without breaking everything, and implying the DAG through dependencies so that I don't have to manage it.
I'm convinced DuckDB is the only engine needed to cover most real-world data pipelines. I've tested it on data scales up to terabytes, and running on appropriately-sized single machines, it's consistently beaten distributed engines like Spark. I think the age of distributed computing for most pipelines might be waning.
You can see the benchmark results for some tests here, where at 1 TB DuckDB 2.0 ran 10 times faster than (untuned) Spark: https://duckstring.com/blog/duckdb-pipeline-engine
Some specific features of Duckstring:
- Data transformations are treated like software packages, allowing the DAG to be implied by dependencies
- A pull-based orchestration system, throttling every process both upstream and downstream of the bottleneck so that nothing runs if it can't be consumed
- An incremental framework based on DBSP that handles updates and deletes efficiently
- CLI-first, so that you can point your own AI agents at the platform rather than relying on a janky integration
The link is to an intro story, which passes on to a Playground app to help get a feel for the orchestration method. Link if you want that directly: https://playground.duckstring.com
You can also just go straight to the site or the project on GitHub: https://duckstring.com https://github.com/duckstring-dev/duckstring
It's pre-1.0, though has been in active development since January, and is ready to try on real pipelines. I'd love to hear your feedback!
Nice work! This is very interesting (as a heavy Dagster and DuckDB user myself)