Parallel work with durable fan-in

Spawn N tasks in parallel, collect their results — every spawn and every result is a durable promise.

A workflow spawns four notification channels (email, SMS, Slack, push) in parallel and joins their results. Total wall time approaches the slowest channel, not the sum. If one channel fails and retries, the others stay checkpointed — they don't re-send.

SDK versions

TypeScript: @resonatehq/sdk v0.10.2 (current). Python: resonate-sdk v0.7.0 (current). Rust: 0.4.0, in active development. Go: pre-release — no semver tag yet, so the example pins a specific commit (see Go SDK).

example-fan-out-fan-in-tsTypeScript

Notification fan-out across email, SMS, Slack, and push using ctx.beginRun.

example-fan-out-fan-in-pyPython

Notification fan-out across email, SMS, Slack, and push with durable children.

example-fan-out-fan-in-rsRust

Three-way parallel processing using ctx.run().spawn() + DurableFuture.

example-fan-out-fan-in-goGo

Notification fan-out across email, SMS, Slack, and push using ctx.RPC + Future.Await.

The problem#

Sequential calls — even three or four — turn into the latency tax you don't notice until you do. Fan them out manually with Promise.all (or its language equivalent) and you've collapsed the latency, but you've also given up checkpointing: a crash mid-batch re-runs everything when the workflow recovers, and a retry of one branch blasts the same email twice to the customer.

Resonate's solution#

Each branch becomes its own durable promise. Spawn them with beginRun (TypeScript), unawaited ctx.run(...) handles (Python), .spawn() (Rust), or ctx.RPC (Go) — all branches run concurrently. If a single branch fails and retries, the others are already checkpointed; they don't re-execute. Crash recovery only re-runs whatever was in flight when the worker died.

Code walkthrough#

The fan-out is N spawn calls. The fan-in is N awaits. The fact that each is durable is what makes this safe — every other implementation has subtle re-execution bugs.

src/workflow.ts·typescript

import type { Context } from "@resonatehq/sdk";
import { sendEmail, sendSms, sendSlack, sendPush, type OrderEvent } from "./channels";

export function* notifyAll(ctx: Context, event: OrderEvent, simulateCrash: boolean) {
  const start = Date.now();

  // Fan-out: spawn all 4 channels at once. Each beginRun returns a future.
  const emailFuture = yield* ctx.beginRun(sendEmail, event);
  const smsFuture   = yield* ctx.beginRun(sendSms, event);
  const slackFuture = yield* ctx.beginRun(sendSlack, event);
  const pushFuture  = yield* ctx.beginRun(sendPush, event, simulateCrash);

  // Fan-in: each future is durable. If push retries, email/sms/slack stay
  // checkpointed and do NOT re-send.
  const results = [
    yield* emailFuture,
    yield* smsFuture,
    yield* slackFuture,
    yield* pushFuture,
  ];

  return {
    orderId: event.orderId,
    channelsNotified: results.filter((r) => r.success).length,
    totalMs: Date.now() - start,
    results,
  };
}

workflow.py·python

from __future__ import annotations

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from resonate.context import Context

from channels import send_email, send_push, send_slack, send_sms


async def notify_all(ctx: Context, event: dict, simulate_crash: bool):
    # Fan-out: ctx.run(...) returns a handle immediately, so all four
    # channels start at once — each is its own durable child.
    f_email = ctx.run(send_email, event)
    f_sms = ctx.run(send_sms, event)
    f_slack = ctx.run(send_slack, event)
    f_push = ctx.run(send_push, event, simulate_crash)

    # Fan-in: await each handle to collect its result. If push retries, the
    # others stay checkpointed and do NOT re-send.
    email = await f_email
    sms = await f_sms
    slack = await f_slack
    push = await f_push
    results = [email, sms, slack, push]
    return {
        "order_id": event["order_id"],
        "channels_notified": sum(1 for r in results if r.get("success")),
        "results": results,
    }

In v0.7.0, ctx.run(...) returns a handle immediately without blocking, so issuing all four calls first starts every channel concurrently; awaiting each handle then collects the results. Each ctx.run is a durable child checkpoint, so if push retries, email/SMS/Slack stay checkpointed and do not re-execute.

src/main.rs·rust

use resonate::prelude::*;
use serde::{Deserialize, Serialize};

#[derive(Serialize, Deserialize, Clone)]
struct WorkItem { id: u32, data: String }

#[derive(Serialize, Deserialize, Debug)]
struct WorkResult { id: u32, output: String }

#[resonate::function]
async fn fan_out_fan_in(ctx: &Context, items: Vec<WorkItem>) -> Result<Vec<WorkResult>> {
    // Fan-out: spawn three items in parallel.
    let h1 = ctx.run(process_item, items[0].clone()).spawn().await?;
    let h2 = ctx.run(process_item, items[1].clone()).spawn().await?;
    let h3 = ctx.run(process_item, items[2].clone()).spawn().await?;

    // Fan-in: each handle is individually durable.
    let r1 = h1.await?;
    let r2 = h2.await?;
    let r3 = h3.await?;
    Ok(vec![r1, r2, r3])
}

#[resonate::function]
async fn process_item(item: WorkItem) -> Result<WorkResult> {
    Ok(WorkResult {
        id: item.id,
        output: format!("Processed: {} (item #{})", item.data, item.id),
    })
}

main.go·go

type FanoutArgs struct {
	Channels []string `json:"channels"`
	Message  string   `json:"message"`
}

type FanoutResult struct {
	Delivered []Delivery `json:"delivered"`
}

type SendArgs struct {
	Channel string `json:"channel"`
	Message string `json:"message"`
}

type Delivery struct {
	Channel string `json:"channel"`
	OK      bool   `json:"ok"`
	Reason  string `json:"reason,omitempty"`
}

// fanout dispatches one ctx.RPC call per channel without awaiting in between,
// then awaits all of them. The dispatches happen in series in user code but
// run concurrently on the server side; the Awaits block until every child
// promise settles.
func fanout(ctx *resonate.Context, args FanoutArgs) (FanoutResult, error) {
	futures := make([]*resonate.Future, 0, len(args.Channels))
	for _, ch := range args.Channels {
		f, err := ctx.RPC("send", SendArgs{Channel: ch, Message: args.Message})
		if err != nil {
			return FanoutResult{}, err
		}
		futures = append(futures, f)
	}

	out := FanoutResult{Delivered: make([]Delivery, 0, len(futures))}
	for i, f := range futures {
		var d Delivery
		if err := f.Await(&d); err != nil {
			out.Delivered = append(out.Delivered, Delivery{
				Channel: args.Channels[i],
				OK:      false,
				Reason:  err.Error(),
			})
			continue
		}
		out.Delivered = append(out.Delivered, d)
	}
	return out, nil
}

// send is the child function that "delivers" a message to one channel. In a
// real system this would call a provider API (Twilio, SES, Slack, ...). Here
// it just prints and returns a Delivery record.
func send(_ *resonate.Context, args SendArgs) (Delivery, error) {
	fmt.Printf("  [send] %-8s <- %q\n", args.Channel, args.Message)
	return Delivery{Channel: args.Channel, OK: true}, nil
}

Go's fan-out uses ctx.RPC — each call creates a durable child promise for the registered send function and returns a *Future without waiting for the child to finish. The second loop awaits each future to fan in; the children run concurrently as separate durable promises, dispatched by the server to any worker subscribed to send.

The TypeScript example includes a --crash flag that simulates the push channel failing on its first attempt. When you run it with --crash, the workflow retries the push channel — but watching the logs, email/SMS/Slack don't re-send. They're each their own durable promise, already resolved.

Run it locally#

shell

git clone https://github.com/resonatehq-examples/example-fan-out-fan-in-ts
cd example-fan-out-fan-in-ts
bun install

Run the happy path (all 4 channels in parallel):

shell

bun run src/index.ts

Then run with the crash flag — push fails, retries, but the other three stay completed:

shell

bun run src/index.ts -- --crash

The log output shows the wall time vs the sum of the individual channel times — fan-out collapses the total to roughly max(channels).

shell

git clone https://github.com/resonatehq-examples/example-fan-out-fan-in-py
cd example-fan-out-fan-in-py
uv sync

Terminal 1Resonate Server·shell

brew install resonatehq/tap/resonate
resonate dev

Terminal 2happy path·shell

uv run python main.py

Or with the crash flag — push fails, retries, but the other three stay completed:

shell

uv run python main.py --crash

shell

git clone https://github.com/resonatehq-examples/example-fan-out-fan-in-rs
cd example-fan-out-fan-in-rs
cargo build

Terminal 1Resonate Server·shell

brew install resonatehq/tap/resonate
resonate dev

Terminal 2worker·shell

cargo run

Terminal 3invoke·shell

resonate invoke fan-out.1 \
  --func fan_out_fan_in \
  --arg '[{"id":1,"data":"a"},{"id":2,"data":"b"},{"id":3,"data":"c"}]'

shell

git clone https://github.com/resonatehq-examples/example-fan-out-fan-in-go
cd example-fan-out-fan-in-go
go mod download

Terminal 1Resonate Server·shell

brew install resonatehq/tap/resonate
resonate dev

Terminal 2workflow·shell

go run .
# or with custom channels
go run . -channels=email,sms,slack,push,webhook -message="ship it"

The single process registers both fanout and send, then invokes the workflow. The [send] lines arrive in non-deterministic order — that's the concurrency — while the final aggregation respects the input channel order.

Try the recovery story#

While the fan-out is mid-flight, kill the worker. Resonate replays only the branches that hadn't checkpointed; everything that completed before the kill is reused from durable promises. The wall time goes up by however long the worker was down, but no branch double-executes.

Recursive factorial — the same primitive, applied to a recursive call graph.
Deep research agent — fan-out applied to LLM tool calls.

Parallel work with durable fan-in

The problem#

Resonate's solution#

Code walkthrough#

Run it locally#

Try the recovery story#

Related#