Message Queues and Async ProcessingLesson 5.2

Message queue guarantees: at-least-once vs exactly-once delivery

at-most-once delivery, at-least-once delivery, exactly-once delivery, idempotent consumers, message acknowledgment, dead letter queue, duplicate handling

Message queue guarantees: at-least-once vs exactly-once delivery

The Three Guarantees

At-most-once: delivered zero or one times. Fire-and-forget. Fast but messages can be lost on failure. Acceptable for metrics, logs, non-critical events.

At-least-once: delivered one or more times. No loss, but duplicates are possible if the consumer crashes after processing but before acknowledging. This is the default for most queues including RabbitMQ, Redis Streams, and SQS.

Exactly-once: delivered precisely once. Requires distributed transactions. Very hard to achieve; Kafka transactions approximate this.

Idempotent Consumers

Since exactly-once is expensive, build consumers that handle duplicates gracefully:

async function processPayment(messageId, data) {
  const alreadyDone = await redis.sismember('processed:payments', messageId);
  if (alreadyDone) return; // skip duplicate

  await chargeCard(data);
  await redis.sadd('processed:payments', messageId);
  await redis.expire('processed:payments', 86400);
}

Dead Letter Queues

When a message fails processing repeatedly after exceeding max retries, move it to a Dead Letter Queue (DLQ) instead of discarding it. DLQs let you inspect failed messages, fix the consumer bug, and reprocess without losing the original payload. Always configure a DLQ in production queuing systems.