DuckDB 2.0 as a Pipeline Engine
In my circles at least, DuckDB seems to be mostly understood as either an SQLite equivalent for analytical queries, or as an equivalent to pandas but somehow more suited to handling SQL. It's rarely mentioned in the same breath as Spark or other 'real' data processing engines, and if it is, always with the addendum that it's 'for small to medium sized data'.
I think this is underselling it. With version 2.0, released on 21 October, I think DuckDB has become among the best of the engines available for batch data pipelines for any realistic data size, up to at least a terabyte. This is especially the case if truly enormous data is handled in the only sane way: incrementally. In this post I compare DuckDB 2.0 with two leading engines: Spark and StarRocks. Then I show how Duckstring's incremental runs compare against this baseline.
In short, the tests confirmed that one DuckDB machine beat same-sized Spark and StarRocks clusters, at both 100 GB and 1 TB scales, by between 3 and 10 times. The 100 GB pipeline (which I suspect is already bigger than most real-world data pipelines) ran comfortably on my 32 GB laptop in less than 30 seconds. For Duckstring's incremental workloads (sub-1% changes per pass) this was cut dramatically again, by up to 13 times.