PROJECT / 002
FlowForge
Distributed background job processing platform.
01 / THE PROBLEM
The engineering problem
Monolithic API performance degrades rapidly when CPU-intensive or slow I/O tasks run synchronously inside the HTTP lifecycle. Built-in FastAPI BackgroundTasks vanish on server restarts because they run in-process memory. Applications require independent, decoupled worker processes backed by durable state.
CONSTRAINTS
HTTP request lifecycle limits
In-memory task loss risk
Worker crash during task
ENGINEERING FOCUS
Decoupled async worker processes
PostgreSQL persistent state record
Heartbeat detection and retry engine
02 / ARCHITECTURE
How the system works
FLOWFORGE / DISTRIBUTED TASK EXECUTION — NORMAL SCENARIO
03 / IMPLEMENTATION
What was built
- 01Built clean REST APIs for asynchronous job submission, progress querying, and cancellation.
- 02Integrated Redis as a lightweight, low-latency task queue broker.
- 03Designed an independent worker loop that polls Redis and executes tasks isolated from the API process.
- 04Persisted job lifecycle states (QUEUED → PROCESSING → COMPLETED / FAILED) in PostgreSQL.
- 05Engineered worker heartbeat tracking to detect dead workers and re-queue abandoned jobs.
- 06Implemented exponential backoff retry handling with configurable attempt limits.
04 / DECISIONS
Technical decisions
05 / RELIABILITY
Reliability and failure handling
06 / LIMITATIONS
Known limitations and future work
Asynchronous architecture, decoupled worker daemon design, queue management, and distributed failure recovery.