Dhara is a distributed task queue I built from scratch in Go. Tasks are submitted over HTTP, persisted in PostgreSQL, and executed by a pool of goroutines. The system handles failure recovery, stale worker detection, dead-letter queueing, and operational metrics — the core mechanics you’d find in production systems like Sidekiq, Faktory, or Asynq, built independently to understand how they work at the layer below the API.

Architecture

PostgreSQL as the Queue Backend

Most task queues use Redis or a dedicated broker. Dhara uses PostgreSQL as the single source of truth for task state. This is a deliberate constraint: it eliminates an operational dependency while giving you durable storage, transactional guarantees, and the ability to query task history with SQL. Workers claim tasks atomically — a pending task is either claimed by exactly one worker or not at all, with no double-execution.

Heartbeat and Reaper

A running task must periodically send a heartbeat. If a worker crashes or hangs, its task stops being updated. The reaper runs on a separate goroutine, scanning for tasks whose last heartbeat is stale. Depending on how many attempts remain, a stale task is either requeued or moved to the dead-letter state. This is the same mechanism production systems use to survive process crashes without manual intervention.

Worker Pool and Task Lifecycle

A configurable pool of goroutines continuously polls PostgreSQL for pending tasks. Each worker atomically claims one task, dispatches it to the registered handler, and updates its status. Failed tasks are retried with exponential backoff. Tasks that exhaust retries become DEAD and can be manually retried via the API. All state transitions are recorded in task_logs.

Observability

The service exposes a Prometheus-format /metrics endpoint covering task lifecycle counters (enqueued, completed, failed, retried, dead, reaped), current queue state by status, worker counts, and in-flight executions. Structured logging via Go’s slog. Liveness and readiness probes for orchestration. No external monitoring dependency required.

Design Decisions

• No HTTP framework. Routing and middleware written against Go’s standard library. Intentional: forces you to understand what frameworks abstract.

• No external broker. PostgreSQL only. One less operational dependency; task durability is guaranteed by the database, not a separate system.

• Minimal dependencies. The Go module graph is intentionally lean — no ORMs, no queue libraries, no web frameworks.

• Correctness before optimization. The reaper, heartbeat interval, retry backoff, and worker shutdown are all designed to avoid data loss before they’re designed to be fast.

Tech Stack

Status

Core queue lifecycle is complete and working. Planned: stronger retry semantics with jitter, richer dead-letter handling, task execution histograms, and improved cancellation.

• Source: https://github.com/Md-Talim/dhara
• Stack: Go · PostgreSQL · Docker · Prometheus · slog