MANOHAR.

ENGINEERING NOTES / BACKGROUND PROCESSING

Designing a background job lifecycle

How to model job state, track execution attempts, and handle worker failures in a distributed background processing system.

MANOHAR · 7 MIN READ · Python · Redis · Architecture · 2026-11

KEY ARCHITECTURAL TAKEAWAYS

  • 01.In-memory task queues (like FastAPI BackgroundTasks) vanish during container restarts; durable state belongs in PostgreSQL.
  • 02.Redis is an exceptional, low-latency coordination broker, but should be treated as ephemeral queue transport.
  • 03.Distributed retries must track attempt counters and failure diagnostics to distinguish transient network blips from poison payloads.

01 / The Vulnerability of In-Process Background Tasks

FastAPI provides a BackgroundTasks utility that executes functions after returning an HTTP response. While convenient for trivial tasks, relying on it for mission-critical jobs in cloud environments (like Cloud Run or Kubernetes) creates severe reliability risks.

Because in-process tasks live entirely in application RAM, any autoscaling downscale, container restart, or unexpected unhandled exception immediately kills the task without a trace. The client receives a 200 OK, but the work never finishes.

02 / Modeling the Job State Machine in FlowForge

To solve this, FlowForge separates job tracking from job execution. The job state machine is explicitly modeled with discrete lifecycle stages: QUEUED, PROCESSING, COMPLETED, and FAILED.

PostgreSQL stores the durable state machine record, execution attempts, timestamps, and error traces. Redis acts strictly as the high-throughput queue transport notifying workers of work to claim.

workers/execution_loop.py
def process_next_job(redis_client: Redis, db_session: Session):
    # Blocking pop from Redis queue with 5-second timeout
    job_payload = redis_client.blpop("flowforge:job_queue", timeout=5)
    if not job_payload:
        return

    job_id = parse_job_id(job_payload)
    job = db_session.query(Job).filter(Job.id == job_id).with_for_update().first()

    try:
        job.status = JobStatus.PROCESSING
        job.attempts += 1
        job.started_at = datetime.utcnow()
        db_session.commit()

        # Execute isolated worker payload
        result = execute_task(job.task_name, job.parameters)

        job.status = JobStatus.COMPLETED
        job.result = result
        job.completed_at = datetime.utcnow()
        db_session.commit()
    except Exception as exc:
        db_session.rollback()
        handle_job_failure(job, exc, db_session, redis_client)

03 / Worker Heartbeats and Failure Recovery

What happens if a worker suffers a hard crash (e.g. OOM killed or SIGKILL) while processing a job? Without a recovery mechanism, the job would remain stuck in PROCESSING forever.

FlowForge workers send periodic heartbeat signals to Redis with a 30-second TTL. A background supervisor checks for active jobs whose assigned worker heartbeat has expired, resetting the job status back to QUEUED and incrementing the retry counter.

← All notes