Back to projects
2026

Distributed Task Queue

A high-performance, educational distributed task queue built with Go, Redis, and Next.js, featuring priority queuing, delayed tasks, worker state visualization, and high-density metrics.

Source Code

Technology Stack

GoRedisNext.jsReactTailwind CSSDocker

Overview

Core Features

  • •Robust Go Backend: Implements an at-least-once delivery queue using Go and Redis with priority queuing, delayed task scheduling, retries & Dead Letter Queues (DLQ) for failed jobs, and dynamic worker pool management.
  • •Real-time Next.js Dashboard: A beautifully designed frontend to visualize live task flow pipelines, real-time worker status (idle/busy/offline), activity logs tracking exactly what stage tasks are in, and high-density metrics and success rates.
  • •Docker Orchestrated: Multi-container workspace setup for instantaneous local development.

System Architecture

dist-task-queue architecture
  • •Backend (Go): Exposes a clean REST API on port 8080. Manages the worker pool lifecycle, orchestrating concurrent worker routines that safely dequeue, process, and transition tasks.
  • •Broker & State Store (Redis 7): Serves as the high-throughput message broker. Uses Redis List structures for FIFO/LIFO queues, ZSETs (Sorted Sets) for delay queues, and Hash maps for maintaining worker heartbeat and state metadata.
  • •Frontend Dashboard (Next.js 16 + React 19 + TailwindCSS v4): Streams state transitions directly to the browser, displaying reactive metrics, system health, and worker lifecycles.

Design Decisions

  • •At-Least-Once Delivery: Uses Redis transaction mechanisms to ensure tasks are only acknowledged and popped once workers have safely finished processing, preventing task loss during crashes.
  • •Dynamic Worker Registry: Workers announce themselves to Redis on startup and maintain an active heartbeat. If a worker goes offline, its state is garbage collected, and remaining tasks are re-queued.
  • •High-Density React dashboard: React 19 concurrent features keep the dashboard responsive even under high-frequency server-sent updates.

Key Challenges

  • •Atomic Dequeue Operations: Preventing race conditions when multiple concurrent workers pull from the priority queue. Solved by implementing Redis Lua scripts for atomic search, lock, and pop operations.
  • •Accurate Delayed Scheduling: Designing a low-overhead ticker routine in Go that periodically transfers ready tasks from the Redis delayed set into the active processing queue without creating execution lag.
  • •State Reconciliation: Keeping the real-time dashboard UI perfectly synchronized with database states under thousands of short-lived task transitions.

Key Learnings

  • •Lua Scripting in Redis: Offloading state transition logic directly to Redis using Lua scripts guarantees atomicity and dramatically reduces round-trip network times.
  • •Go Concurrency Patterns: Channels and sync primitives in Go provide excellent control for building robust, self-healing worker pools.
  • •Real-time UX Strategy: Buffering high-frequency state updates on the frontend prevents UI thread blocking and preserves rendering performance.

Impact and Results

  • •Zero Task Loss: Achieved reliable execution with automatic retry logic and safe DLQ fallback for persistent errors.
  • •Sub-millisecond Dequeue: Dequeue latency remains under 1ms under high concurrent worker pool tests.
  • •High Observability: Delivered a beautiful dashboard that makes complex distributed system patterns instantly understandable.