ENGINEERING NOTES / BACKGROUND PROCESSING
Designing a background job lifecycle
How to model job state, track execution attempts, and handle worker failures in a distributed background processing system.
MANOHAR · 7 MIN READ · Python · Redis · Architecture · 2026-11
KEY ARCHITECTURAL TAKEAWAYS
- 01.In-memory task queues (like FastAPI BackgroundTasks) vanish during container restarts; durable state belongs in PostgreSQL.
- 02.Redis is an exceptional, low-latency coordination broker, but should be treated as ephemeral queue transport.
- 03.Distributed retries must track attempt counters and failure diagnostics to distinguish transient network blips from poison payloads.
01 / The Vulnerability of In-Process Background Tasks
FastAPI provides a BackgroundTasks utility that executes functions after returning an HTTP response. While convenient for trivial tasks, relying on it for mission-critical jobs in cloud environments (like Cloud Run or Kubernetes) creates severe reliability risks.
Because in-process tasks live entirely in application RAM, any autoscaling downscale, container restart, or unexpected unhandled exception immediately kills the task without a trace. The client receives a 200 OK, but the work never finishes.
02 / Modeling the Job State Machine in FlowForge
To solve this, FlowForge separates job tracking from job execution. The job state machine is explicitly modeled with discrete lifecycle stages: QUEUED, PROCESSING, COMPLETED, and FAILED.
PostgreSQL stores the durable state machine record, execution attempts, timestamps, and error traces. Redis acts strictly as the high-throughput queue transport notifying workers of work to claim.
03 / Worker Heartbeats and Failure Recovery
What happens if a worker suffers a hard crash (e.g. OOM killed or SIGKILL) while processing a job? Without a recovery mechanism, the job would remain stuck in PROCESSING forever.
FlowForge workers send periodic heartbeat signals to Redis with a 30-second TTL. A background supervisor checks for active jobs whose assigned worker heartbeat has expired, resetting the job status back to QUEUED and incrementing the retry counter.