Resilient Webhook Idempotency: Handling Stripe Financial Events with Redis Locks & PostgreSQL Audit Trails
Guaranteeing exactly-once billing state mutations under network retries and distributed concurrency.
1. The Reality of At-Least-Once Delivery
When designing financial systems, developers must assume that every webhook will be received more than once. Stripe, PayPal, and banking gateways use exponential backoff retry algorithms; if your server takes 5.1 seconds to respond due to a database lock, the gateway considers the delivery failed and immediately sends an identical payload.
If your endpoint processes `invoice.payment_succeeded` by crediting the client's balance or unlocking subscription tiers without idempotency guards, duplicate events will credit the user twice and desynchronize your revenue reporting.
In ClientForge, we built a three-stage idempotency pipeline: Cryptographic signature verification → Redis atomic locking → PostgreSQL transactional event record.
2. Cryptographic Signature Verification First
Before inspecting the payload JSON or querying any database, the endpoint verifies the cryptographic `Stripe-Signature` header against the raw request body using HMAC-SHA256.
Crucially, the raw byte stream must be preserved before Django or Next.js parses the body into a dictionary; modifying a single byte or whitespace invalidates the HMAC signature.
3. Atomic Redis Distributed Locking
Once verified, the event ID (e.g., `evt_3L...`) is acquired as an atomic lock in Redis using `SET key value NX EX 300` (set if not exists with a 5-minute expiry).
If the key already exists in Redis, another worker thread is currently processing the event or has just finished doing so. The endpoint immediately returns `HTTP 200 OK` to acknowledge the gateway, terminating duplicate processing without touching PostgreSQL.
import stripe
import redis
from django.db import transaction
from django.http import HttpResponse
from .models import ProcessedWebhookEvent, Invoice
redis_client = redis.Redis.from_url(settings.REDIS_URL)
def handle_stripe_webhook(request):
payload = request.body
sig_header = request.headers.get("Stripe-Signature")
webhook_secret = settings.STRIPE_WEBHOOK_SECRET
try:
event = stripe.Webhook.construct_event(payload, sig_header, webhook_secret)
except (ValueError, stripe.error.SignatureVerificationError):
return HttpResponse(status=400)
event_id = event["id"]
lock_key = f"webhook_lock:{event_id}"
# Atomic distributed lock: Set only if key does not exist (NX), expires in 300s
acquired = redis_client.set(lock_key, "processing", nx=True, ex=300)
if not acquired:
# Event is already being processed or was processed; acknowledge gateway
return HttpResponse(status=200)
try:
with transaction.atomic():
# Check permanent audit log inside database transaction
if ProcessedWebhookEvent.objects.filter(event_id=event_id).exists():
return HttpResponse(status=200)
event_type = event["type"]
if event_type == "invoice.payment_succeeded":
invoice_data = event["data"]["object"]
invoice = Invoice.unscoped_objects.select_for_update().get(
stripe_invoice_id=invoice_data["id"]
)
invoice.mark_as_paid(
transaction_id=invoice_data.get("payment_intent"),
amount_paid=invoice_data["amount_paid"] / 100
)
# Record event in audit table to permanently prevent duplicate replay
ProcessedWebhookEvent.objects.create(
event_id=event_id,
event_type=event_type,
payload=event
)
redis_client.set(lock_key, "completed", ex=86400)
return HttpResponse(status=200)
except Exception as e:
# Release lock so retry can be handled after transient error resolution
redis_client.delete(lock_key)
logger.error(f"Webhook processing error {event_id}: {str(e)}")
return HttpResponse(status=500)4. Transactional Database State Mutations
Inside the database execution block, we wrap all mutations within `transaction.atomic()` and lock the relevant invoice row using `select_for_update()`. This prevents concurrent race conditions if an administrative user is editing the invoice simultaneously.
Finally, the `ProcessedWebhookEvent` record is inserted within the exact same database transaction. If the database crashes mid-execution, the entire mutation rolls back, the Redis lock expires, and the gateway's subsequent retry safely reprocesses the event.
Core Engineering Takeaways
- •Always design webhook consumers for at-least-once delivery; assume every event will retry.
- •Preserve raw request byte streams for HMAC-SHA256 signature verification.
- •Use atomic Redis SET NX locks to eliminate concurrent processing across worker threads.
- •Persist processed event IDs in an ACID transaction with row-level locks on mutated records.
View the complete architectural case study and source implementation.
Continue reading technical architecture insights.