Troubleshooting

Common problems and how to fix them.

This guide covers common issues you might encounter when using Resonate and how to resolve them.

If your problem isn't listed here, check the Errors reference for error codes or ask in Discord.

Server won't start#

Database connection failures#

Symptoms:

Server fails to start with connection errors
Logs show failed to connect to database or similar

Causes:

PostgreSQL not running
Wrong connection credentials
Network/firewall blocking connection
Database doesn't exist

Solutions:

Check PostgreSQL is running:

code

# PostgreSQL status
pg_isready -h localhost -p 5432

# For managed services, check cloud console

Verify connection string:

resonate.toml

[storage]
type = "postgres"

[storage.postgres]
url = "postgres://resonate:secret@localhost:5432/resonate"

Or set RESONATE_STORAGE__POSTGRES__URL in the environment. Check the host, port, database, user, and password all match what your Postgres instance expects.

Create database if missing:

code

psql -h localhost -U postgres -c "CREATE DATABASE resonate;"

Check firewall/security groups:

Ensure server can reach database on port 5432
Check cloud security group rules
Verify VPC/network configuration

Port already in use#

Symptoms:

Server fails to start
Logs show address already in use or bind: address already in use

Cause: Another process is using port 8001 (HTTP).

Solutions:

Find what's using the port:

code

# macOS/Linux
lsof -i :8001

# Kill the process if needed
kill -9 <PID>

Use a different port: Pass --server-port to resonate serve to bind a different port:

code

resonate serve --server-port 8003

Configuration file errors#

Symptoms:

Server fails to start
Logs show a TOML parse error from figment or an unknown field name

Cause: Invalid TOML syntax in resonate.toml, or a field name that doesn't match the schema.

Solutions:

Validate TOML syntax with any TOML linter, or paste it into toml-lint.com.
Check section and field names against Run a server. Sections are nested with dots: [storage.postgres], not [storage_postgres].
Check env var nesting. Environment variables use the RESONATE_ prefix and __ (double underscore) for nesting:

code

# WRONG — single underscore
RESONATE_STORAGE_POSTGRES_URL=...

# RIGHT — double underscore between sections
RESONATE_STORAGE__POSTGRES__URL=...

Workers not receiving tasks#

Workers not connecting#

Symptoms:

Workers start but never process tasks
No worker registration in server logs
Tasks remain pending indefinitely

Causes:

Wrong server URL
Network/firewall blocking connection
Server not running
Authentication misconfigured

Solutions:

Verify server URL:

code

const resonate = new Resonate({
  url: "http://resonate-server:8001",  // Must match server address
  group: "workers",
});

Test connectivity:

code

# From worker machine, test server reachability
curl -s -o /dev/null -w "%{http_code}\n" http://resonate-server:8001/health

# Should return 200 when the server is alive

Check authentication: If server has auth enabled, workers must provide credentials:

code

const resonate = new Resonate({
  url: "http://resonate-server:8001",
  group: "workers",
  token: "your-jwt-token",
});

Check server logs:

code

# Look for worker registration messages
resonate serve --level debug

# Should see: "worker registered" or similar

Tasks not reaching workers#

Symptoms:

Workers connected but not processing tasks
Tasks created but remain pending
No task distribution happening

Causes:

Wrong worker group name
Worker polling misconfigured
Task routing rules don't match workers

Solutions:

Verify group names match:

code

// When registering worker:
const resonate = new Resonate({
  group: "workers",  // Note the group name
});

// When creating task:
resonate.rpc(
  taskId,
  "processOrder",
  data,
  resonate.options({
    target: "poll://any@workers",  // Must match worker group
  })
);

Check worker is polling: Workers must actively poll for tasks. Ensure your worker code calls functions that poll:

code

// Worker polls when you register functions
resonate.register("processOrder", async (ctx, data) => {
  // Task execution
});

// Worker polls automatically after registration

Inspect promise state:

code

# Query promises to see if tasks are being created
curl http://localhost:8001/promises?state=pending

# Check if tasks exist for those promises
curl http://localhost:8001/tasks?state=pending

Promises not resolving#

Promise stuck in pending state#

Symptoms:

Promise created but never completes
No worker picks up the task
resonate.promises.get() shows state: "pending" indefinitely

Causes:

No workers available for the task's group
Worker crashed mid-execution and task not reassigned
Task timeout not configured (waits forever)
Routing misconfiguration

Solutions:

Confirm workers are running:

code

# Check worker processes
ps aux | grep worker

# In Kubernetes:
kubectl get pods -l app=resonate-worker

Check promise/task state:

code

# Get promise details
curl http://localhost:8001/promises/{promiseId}

# Check if task exists
curl http://localhost:8001/tasks?promiseId={promiseId}

Set task timeouts:

code

resonate.rpc(
  taskId,
  "processOrder",
  data,
  resonate.options({
    target: "poll://any@workers",
    timeout: 60000,  // 60 second timeout
  })
);

Check worker heartbeats: If worker crashed, server should detect via heartbeat timeout (default: 60s) and reassign. Check server logs for heartbeat failures.

Promise failed but retry not working#

Symptoms:

Promise fails once and doesn't retry
Expected automatic retry but it didn't happen

Cause: Resonate doesn't automatically retry failed promises unless you configure retry logic.

Solutions:

Implement retry logic explicitly:

code

async function processOrderWithRetry(ctx, data) {
  let attempts = 0;
  const maxAttempts = 3;

  while (attempts < maxAttempts) {
    try {
      const result = await ctx.run(() => processOrder(data));
      return result;
    } catch (error) {
      attempts++;
      if (attempts >= maxAttempts) throw error;
      await ctx.sleep(1000 * attempts);  // Exponential backoff
    }
  }
}

Check error type: Some errors shouldn't retry (e.g., invalid input). Handle appropriately:

code

catch (error) {
  if (error.code === "INVALID_INPUT") {
    throw error;  // Don't retry
  }
  // Retry for transient errors
}

Performance issues#

Slow task execution#

Symptoms:

Tasks complete but take longer than expected
High latency between task creation and completion

Causes:

Not enough workers (tasks queue up)
Worker resource constraints (CPU/memory)
Database performance issues
Network latency

Solutions:

Scale workers horizontally:

code

# Docker Compose
docker-compose up -d --scale worker=10

# Kubernetes
kubectl scale deployment resonate-workers --replicas=20

See Scaling for details.

Monitor worker resources:

code

# Check CPU/memory usage
top
htop

# Kubernetes:
kubectl top pods -l app=resonate-worker

Optimize database:

Add indexes for frequently queried promise/task fields
Increase PostgreSQL connection pool size
Use managed PostgreSQL with IOPS scaling

Check network latency:

code

# Measure round-trip time to server
ping resonate-server

# Test HTTP latency
time curl -s http://resonate-server:8001/health

High database load#

Symptoms:

Slow promise creation/resolution
Database CPU/IOPS maxed out
Connection pool exhausted

Causes:

Too many concurrent promises
Inefficient queries (missing indexes)
Insufficient database resources

Solutions:

Upgrade database resources:

Increase CPU/RAM
Add IOPS capacity (for cloud databases)
Use managed PostgreSQL with auto-scaling

Tune connection pool:

resonate.toml

[storage.postgres]
url = "postgres://resonate:secret@localhost:5432/resonate"
pool_size = 50  # Increase pool size (default: 10)

Add database indexes: Check PostgreSQL slow query log and add indexes for common queries.
Batch operations: If creating many promises, batch them when possible to reduce database round-trips.

Authentication issues#

Unauthorized errors#

Symptoms:

Workers can't connect
API requests return 401 Unauthorized
Logs show authentication failures

Causes:

Wrong credentials
Auth enabled on server but not configured in client
Token expired (JWT)

Solutions:

Verify credentials:

code

const resonate = new Resonate({
  url: "http://resonate-server:8001",
  token: "your-jwt-token",  // Check this matches server config
});

Check server auth config:

The server uses JWT bearer auth signed with an Ed25519 key. Start the server with the public key path:

code

resonate serve --auth-publickey /etc/resonate/auth/public.pem

Or in resonate.toml:

code

[auth]
publickey = "/etc/resonate/auth/public.pem"

In the SDK, pass a JWT signed by the matching private key:

code

const resonate = new Resonate({
  url: "http://resonate-server:8001",
  token: process.env.RESONATE_TOKEN,
});

For JWT tokens, verify:

Token hasn't expired
Signature was produced with the private key matching the server's public key
iss / aud claims match what the server expects (if configured)

See Security for the full auth setup.

Development workflow issues#

Changes not taking effect#

Symptoms:

Code changes don't appear when running
Old behavior persists after updates

Causes:

Using wrong binary (old version still running)
Cache issues
Docker image not rebuilt

Solutions:

Verify process is new:

code

# Kill old processes
pkill -f resonate

# Restart with fresh binary
resonate serve

Rebuild Docker images:

code

docker-compose build --no-cache
docker-compose up -d

Clear SDK caches (if applicable):

code

# Node.js
rm -rf node_modules && npm install

# Python
rm -rf __pycache__ && pip install -r requirements.txt

SQLite "database is locked" errors#

Symptoms:

SQLite errors about locked database
Concurrent access failures

Cause: SQLite doesn't handle high concurrency well. Multiple processes/threads trying to write simultaneously.

Solution: Use PostgreSQL for any deployment with concurrent writers:

resonate.toml

[storage]
type = "postgres"

[storage.postgres]
url = "postgres://resonate:secret@localhost:5432/resonate"

SQLite is best for development and single-tenant deployments where there is no concurrent write contention against the server.

Getting more help#

If these troubleshooting steps don't resolve your issue:

Check error codes: See Errors for detailed error information
Enable debug logging:

code

resonate serve --level debug

Collect diagnostics:
- Server logs
- Worker logs
- Promise/task state from API
- Database connection status
- Network connectivity tests
Ask in Discord: Share diagnostics in the Resonate Discord
File a bug: If you've found a bug, open an issue on GitHub

Quick diagnostic checklist#

When debugging, check these in order:

Most issues fall into one of these categories. Work through the checklist systematically.