Duckstring

There is no DAG.

pip install duckstring

Build data pipelines the way you build software. Version your transforms, declare your dependencies, and let Duckstring execute only the paths your data demands.

Quickstart →Try the playground

Apache-2.0 · pure Python · no service to stand up

Declare, don't govern

Know your family, not the world.

Duckstring operates on a core decision: treat each transform as a versioned package (a Pond) that declares its upstream dependencies, exactly the way a library declares the packages it imports. You need only work with your immediate sources and consumers — not the entire lineage. Make that one decision and you get three things that are normally hand-built and hand-tended for free:

DAG is implied — upgrade atomically. ↓ The pipeline is the union of every Pond's declared dependencies. There's no central DAG to build, wire, or govern — it's already in the graph. A new Pond (or a breaking change to an existing one) won't execute until there's a consumer ready to use it.
Demand-driven execution. ↓ Runs are driven from the outputs, not the inputs — paths with no downstream consumers sit idle, and each path runs only as often as its bottleneck, throttled both downstream and upstream.
Native incremental processing. ↓ Run history is metadata, which makes change detection and incremental processing trivial. Duckstring bundles Trickle: a DBSP-based incremental engine over DuckDB. Blazing-fast execution on a single node — perfect for the 90% of cases where you don't actually need distributed compute.

Duckstring is generic ↓ — attach any Python code (even calls out to external services) and get the full benefit immediately.

Upgrade atomically

Ship a breaking change without a meeting.

Ponds use SemVer, and a new major version runs concurrently with the old one. Deploy a breaking v2 and it comes up alongside v1: existing consumers keep pulling v1, which keeps running, while consumers migrate to v2 one at a time by changing a single line in their own manifest. The old major retires when nothing depends on it. No lockstep, no choreographed release, no freeze.

Upgrading a complex sequence of transformations can paralyze development, especially if upstream changes are needed. Just deploy breaks as a separate Pond, know that it won't run until downstream also upgrades, and be sure you won't have broken anything. See Versioning.

Run only what's demanded

Bottleneck-aware execution. No wasted compute.

Most schedulers can throttle work downstream of a slow step. Duckstring throttles everything upstream of it too. Execution is strictly demand-driven: a transform runs only when something downstream has actually asked for it. The result is a pipeline that re-paces itself to its real bottleneck — and never over-produces results no one is waiting for.

No sophisticated prediction of run times is required — flipping to control by consumers rather than suppliers means the entire path naturally throttles to its slowest process. See Orchestration Theory.

Keep tasks small with deltas

Discrete stages for incremental processing.

Incremental processing becomes very natural once a Pond has a clear lineage, runs only when parents change, and tracks an epoch throughout. Bundled with Duckstring is the Trickle engine — a DBSP implementation over DuckDB that cuts processing to the absolute minimum by focussing only on changes.

Done well, incremental processing lets you stay single-node, in-memory and blazing fast — real streaming performance with minimal infrastructure. See Incremental processing.

Start small

Drop it into the stack you already run.

You don't have to move your compute to get value. A Pond is just Python, so it can wrap anything: a SQL transform, a local script, or a call out to a remote system that it kicks off and polls to completion. Point Duckstring at a sequence you already run and it becomes the coordinator — firing each step only when its inputs have actually changed and something downstream wants the result. The redundant compute you stop paying for lands on day one, with no rewrite.

Coordinate, don't migrateWrap remote jobs behind a start-and-poll code chunk; Duckstring sequences them by demand, skipping the runs whose inputs are unchanged.

Cut wasted runsNo part of the pipeline runs unless a consumer actually needs it. If nothing upstream moved, nothing runs — the compute saving lands on day one.

Grow into the modelPromote a wrapped step to a native transform when it earns it; start with API calls to existing systems and shift it to Duckstring whenever you feel.

Go deeper

Where to next.

Quickstart →A running four-Pond pipeline in a few minutes — install, deploy, trigger, query.Playground →Feel freshness-based execution in your browser — build a graph, send demand, nothing to install.Theory →The full demand-and-freshness model, worked step by step. The part to read if the demo intrigued you.GitHub →The source: the packaging standard, the CLI, and the reference runtime. Apache-2.0.

Looking for cloud hosting — a Catchment run for you? I'd like to hear from you: dev@duckstring.com