MANOHAR.

PROJECT / 002

FlowForge

Distributed background job processing platform.

PythonFastAPIPostgreSQLRedisSQLAlchemy

TYPE

Backend application

STATUS

Prototype

01 / THE PROBLEM

The engineering problem

Monolithic API performance degrades rapidly when CPU-intensive or slow I/O tasks run synchronously inside the HTTP lifecycle. Built-in FastAPI BackgroundTasks vanish on server restarts because they run in-process memory. Applications require independent, decoupled worker processes backed by durable state.

CONSTRAINTS

HTTP request lifecycle limits

In-memory task loss risk

Worker crash during task

ENGINEERING FOCUS

Decoupled async worker processes

PostgreSQL persistent state record

Heartbeat detection and retry engine

02 / ARCHITECTURE

How the system works

FLOWFORGE / DISTRIBUTED TASK EXECUTION — NORMAL SCENARIO

FastAPIJob submitenqueueRedisJob queuedequeueWorkerExecutionstate syncPostgreSQLPersistent StateQUEUEDDISPATCHEDPROCESSINGCOMPLETED

03 / IMPLEMENTATION

What was built

  • 01Built clean REST APIs for asynchronous job submission, progress querying, and cancellation.
  • 02Integrated Redis as a lightweight, low-latency task queue broker.
  • 03Designed an independent worker loop that polls Redis and executes tasks isolated from the API process.
  • 04Persisted job lifecycle states (QUEUED → PROCESSING → COMPLETED / FAILED) in PostgreSQL.
  • 05Engineered worker heartbeat tracking to detect dead workers and re-queue abandoned jobs.
  • 06Implemented exponential backoff retry handling with configurable attempt limits.

04 / DECISIONS

Technical decisions

DECISIONREASONING
Dual storage: PostgreSQL + RedisPostgreSQL provides an auditable, durable history of job states; Redis handles fast queue operations.
Independent worker daemonGuarantees that long-running jobs never block the API event loop or consume HTTP worker threads.
Heartbeat mechanismWorkers update a Redis TTL key; absence of heartbeat marks worker offline and triggers job recovery.
Attempt tracking in PostgreSQLRetains complete error diagnostics and stack traces across failed attempts for post-mortem analysis.

05 / RELIABILITY

Reliability and failure handling

SCENARIOEXPECTED BEHAVIOURVERIFICATION
Worker crashes mid-execution (SIGKILL)Heartbeat timeout identifies unacknowledged job; supervisory loop resets job to QUEUED for retry.Worker termination simulation test
Job raises uncaught exception repeatedlyAttempt counter increments until max retries reached, then marks job permanently FAILED.Retry exhaustion test
Redis broker temporarily unavailableFastAPI rejects submission with 503 Service Unavailable; existing database records remain intact.Redis downtime injection test

06 / LIMITATIONS

Known limitations and future work

—Delivery guarantee is at-least-once; tasks must be idempotent to tolerate edge-case duplicate execution.
—Heartbeat recovery window incurs a detection delay (configurable, default 30s).
—Dynamic worker autoscaling based on queue depth is not yet implemented.

Asynchronous architecture, decoupled worker daemon design, queue management, and distributed failure recovery.

← PREVIOUS

OrderFlow

All projects

NEXT →

IncidentHub