Why Background Jobs Sometimes Run Twice: At-Least-Once Delivery in Practice

1. The Myth of Exactly-Once Delivery

When developers first integrate message queues like RabbitMQ, AWS SQS, or Redis-based background workers like BullMQ or Sidekiq, they frequently believe that pushing a job onto a queue guarantees it will be executed precisely one time.

In real-world networks, this assumption is false. Almost every scalable message queue in existence guarantees At-Least-Once Delivery. That means the queue promises that your job will not be lost, but it reserves the right to deliver it two, three, or more times.

2. The Mechanism: Queue Acknowledgments & Visibility Timeouts

To understand why duplicate executions happen, examine how workers communicate with queue brokers:

1. Worker requests a job from the queue.

2. The broker hides the message from other workers by starting a 'visibility timeout' (e.g., 30 seconds).

3. The worker executes the task.

4. The worker sends an acknowledgment (`ACK`) back to the broker, instructing it to permanently delete the message.

[Queue Broker] ────── (1) Deliver Job ──────> [Worker Node A]
      │                                                │
[Visibility Timer (30s)]                         [Processing Job...]
      │                                                │
      ▼                                                ▼
  [Timer Expires!]                                (Takes 35s!)
      │                                                │
      ├─────── (2) Redeliver Job ──────> [Worker Node B]
      │                                    (Runs simultaneously!)
      ▼
Worker A finishes & sends ACK (Too late!)

If Worker A experiences garbage collection pauses, slow database queries, or a momentary freeze, the broker's 30-second visibility timeout will expire. Believing Worker A has died, the broker releases the job to Worker B.

Now, both Worker A and Worker B are executing the exact same task simultaneously. If this job generates a PDF invoice and emails a client, the client receives two identical emails.

3. Making Background Jobs Idempotent

Because queue redelivery cannot be prevented at the transport level, workers must be designed to be safe when invoked repeatedly:

interface InvoiceJobData {
  jobId: string;
  invoiceId: string;
}

async function processInvoiceEmail(job: InvoiceJobData) {
  // 1. Atomic database state check
  const invoice = await db.invoices.findUnique({
    where: { id: job.invoiceId }
  });

  if (!invoice || invoice.emailSentAt !== null) {
    // Already sent! Acknowledge and discard safely.
    console.log(`Invoice ${job.invoiceId} already emailed. Skipping duplicate execution.`);
    return;
  }

  // 2. Acquire short-lived distributed lock or conditional update
  const updated = await db.invoices.updateMany({
    where: {
      id: job.invoiceId,
      emailSentAt: null // Optimistic concurrency check
    },
    data: {
      emailSentAt: new Date()
    }
  });

  if (updated.count === 0) {
    // Another worker grabbed it first
    return;
  }

  // 3. Perform the external side effect
  await mailService.sendInvoice(invoice);
}

4. Conclusion: Embrace At-Least-Once Delivery

Designing systems assuming failures will happen is the hallmark of senior software engineering. By acknowledging that background workers can and will execute jobs multiple times, you build resilient applications with atomic checks and state transitions that keep data accurate regardless of network turbulence.