Kafka integration

How to integrate Resonate with Kafka today (the worker pattern) and what to expect when native Kafka transport ships.

TL;DR#

Resonate does not ship a native Kafka transport in the current server. The supported integration today is the Kafka worker pattern: your code consumes Kafka with a normal client library and dispatches a Resonate workflow per message. Native kafka:// transport is specified in the Message Passing Protocol and is on the roadmap — when it ships, the server will deliver task messages over Kafka directly.

The Kafka worker pattern#

Use a regular Kafka consumer (kafkajs, confluent-kafka-python, rdkafka, etc.) and call beginRun on each message. A stable per-message identifier becomes the Resonate promise ID, so the dispatch is idempotent — duplicate deliveries reconnect to the in-flight execution rather than starting over.

src/consumer.ts

import { Kafka } from "kafkajs";
import { resonate } from "./workflow";

const kafka = new Kafka({ brokers: ["localhost:9092"] });
const consumer = kafka.consumer({ groupId: "workers" });

await consumer.connect();
await consumer.subscribe({ topic: "records_to_process", fromBeginning: true });

await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    // topic-partition-offset uniquely identifies the message even when no key is set
    const messageId = `${topic}-${partition}-${message.offset}`;
    await resonate.beginRun(messageId, "workflow", messageId, message.offset);
  },
});

If your producer sets a stable message key (or your payload carries a domain ID like a record UUID), that's also a valid promise ID — the only requirement is that the same logical message always resolves to the same string.

Full TypeScript / Python / Rust walkthroughs and runnable repos are on the Kafka worker example page. Source repos:

Idempotent dispatch#

Every Kafka message has a stable identifier you can derive from its coordinates: ${topic}-${partition}-${offset} is unique by construction, and a producer-supplied key (or a domain ID inside the payload) works equally well when present. Pass that identifier as the Resonate promise ID:

code

await resonate.beginRun(messageId, "workflow", ...args);

Resonate dedupes by promise ID. A consumer that crashes after starting a workflow but before committing the offset will redeliver the same message on restart. The redelivery hits beginRun with the same ID, Resonate recognizes the in-flight execution, and the workflow resumes from its last checkpoint instead of starting over. No per-message dedup table, no idempotency key bookkeeping — the workflow ID is the idempotency key.

This is the property that makes the worker pattern crash-safe: the consumer can be at-least-once, and the dispatch into Resonate stays exactly-once.

What about `@resonatehq/kafka`?#

The @resonatehq/kafka NPM package (and the resonatehq/resonate-transport-kafka-ts repo) targets the legacy Go server, not the current Rust server. It expects server flags like --api-kafka-enable that don't exist in the current server binary. Don't install it expecting it to wire up against a current Resonate Server — it won't.

If you want native Kafka transport, watch:

The kafka:// address scheme in the protocol specification.
The Planned section of the Message transports page — it gets updated when transports land in the server.

Roadmap#

Native Kafka transport is on the roadmap. See Message transports — Planned for status, and the Kafka worker example for the supported integration today.