Stripe Webhook Reconciliation: Ensuring Idempotent Subscription Billing

Integrating Stripe into a cloud software application is deceptively straightforward during initial testing. You generate a checkout session, listen for checkout.session.completed, and grant the customer their subscription tier.

In production, however, payment gateways communicate across an unreliable public internet. Webhooks are delivered with at-least-once semantics: network hiccups cause Stripe to retry the identical webhook multiple times; transient latency spikes cause events to arrive out of chronological sequence; and deployment restarts cause dropped webhook payloads.

If your webhook handler is not strictly idempotent and monotonically ordered, disaster follows:

• A customer who canceled their account is downgraded, but a delayed invoice.payment_succeeded retry arrives minutes later, resurrecting the subscription and billing them again.
• An upgrade webhook fires twice, creating duplicate invoice records in your accounting database and double-counting revenue.
• A database lock timeout drops a webhook, leaving an enterprise customer stuck on a free trial despite paying $10,000.

This guide provides the complete architectural framework for building a bulletproof, idempotent Stripe webhook reconciliation engine using monotonic state machines and the Transactional Outbox pattern.

The Core Rule: Verify Signatures Immediately

Before your application deserializes or processes any JSON payload, it must verify the Stripe-Signature header using HMAC-SHA256 and your endpoint's secret key (whsec_...). This prevents attackers from forging fake payment webhooks to gain unauthorized access to paid features.

package billing

import (
	"io"
	"net/http"

	"github.com/stripe/stripe-go/v78"
	"github.com/stripe/stripe-go/v78/webhook"
)

func (s *BillingServer) HandleStripeWebhook(w http.ResponseWriter, r *http.Request) {
	const maxBodyBytes = int64(65536) // Protect against oversized payloads
	r.Body = http.MaxBytesReader(w, r.Body, maxBodyBytes)

	payload, err := io.ReadAll(r.Body)
	if err != nil {
		http.Error(w, "Error reading request body", http.StatusBadRequest)
		return
	}

	sigHeader := r.Header.Get("Stripe-Signature")
	event, err := webhook.ConstructEvent(payload, sigHeader, s.webhookSecret)
	if err != nil {
		http.Error(w, "Invalid webhook signature", http.StatusBadRequest)
		return
	}

	// Signature is cryptographically verified; proceed to idempotent persistence
	if err := s.processIdempotentEvent(r.Context(), event); err != nil {
		s.logger.Printf("Failed to process webhook %s: %v", event.ID, err)
		http.Error(w, "Processing failed", http.StatusInternalServerError)
		return
	}

	// Always return HTTP 200 OK immediately
	w.WriteHeader(http.StatusOK)
}

Idempotent Ingestion via Database Unique Constraints

Never perform business logic directly upon receiving a webhook. The first and only synchronous action your HTTP handler should perform is writing the raw event to an append-only stripe_events table in PostgreSQL.

PostgreSQL enforces uniqueness on stripe_event_id:

CREATE TABLE stripe_events (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    stripe_event_id TEXT NOT NULL UNIQUE,
    event_type TEXT NOT NULL,
    payload JSONB NOT NULL,
    status TEXT NOT NULL DEFAULT 'pending', -- 'pending', 'processed', 'ignored'
    stripe_created_at TIMESTAMPTZ NOT NULL,
    received_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

When Stripe sends a retry for an event you have already received, the database insertion safely ignores it using ON CONFLICT DO NOTHING:

INSERT INTO stripe_events (stripe_event_id, event_type, payload, stripe_created_at)
VALUES ($1, $2, $3, $4)
ON CONFLICT (stripe_event_id) DO NOTHING;

If zero rows were affected, the event was already recorded; return HTTP 200 OK immediately. This guarantees that duplicated deliveries from Stripe are eliminated at the database door.

The Out-of-Order Trap: Building Monotonic State Machines

A severe bug in naive webhook consumers is processing events based on arrival time rather than event generation time.

Consider this real-world production race condition:

1. Monday 10:00:00: User downgrades their plan. Stripe generates customer.subscription.updated (Plan: Free, created = 10:00:00). Network fails; Stripe schedules a retry.
2. Monday 10:02:00: User changes their mind and upgrades. Stripe generates customer.subscription.updated (Plan: Enterprise, created = 10:02:00). This delivery succeeds immediately. Your database records the user as Enterprise.
3. Monday 10:05:00: Stripe's retry from step 1 finally arrives.

If you blindly apply the webhook that arrived at 10:05:00, you downgrade the user to Free, overwriting their subsequent upgrade!

Solution: Timestamp-Guarded Monotonic Updates

Every record in your subscriptions table must store the last_stripe_event_created_at timestamp. When applying a state update, execute a conditional SQL update:

UPDATE subscriptions
SET 
    status = $1,
    tier = $2,
    current_period_end = $3,
    last_stripe_event_created_at = $4,
    updated_at = NOW()
WHERE 
    stripe_subscription_id = $5
    AND last_stripe_event_created_at < $4; -- MONOTONIC GUARD!

If Stripe delivers an older event out-of-order, last_stripe_event_created_at < $4 evaluates to false. Zero rows are updated, and your current, newer state is preserved.

The Transactional Outbox Pattern for Downstream Sync

When a customer upgrades to your Enterprise tier, two things must happen:

1. Your database record is updated to Enterprise.
2. The user's cached permission flags in Redis must be invalidated immediately.

If you update the database and then attempt to call Redis, what happens if your application pod crashes between the two calls? The database is updated, but Redis retains stale Free-tier entitlements indefinitely (the classic Dual-Write Problem).

Solving Dual-Writes with the Transactional Outbox

Wrap your subscription update and an outbox event insertion inside the same atomic database transaction:

BEGIN;

-- 1. Update subscription state
UPDATE subscriptions 
SET tier = 'enterprise', status = 'active'
WHERE id = 'sub_9918';

-- 2. Insert event to outbox table in same transaction
INSERT INTO outbox_events (aggregate_type, aggregate_id, event_type, payload)
VALUES (
    'subscription',
    'sub_9918',
    'tier_upgraded',
    '{"tenant_id": "c1209b55...", "tier": "enterprise"}'
);

COMMIT;

A lightweight background worker polls the outbox_events table (or streams from the PostgreSQL WAL via Debezium), broadcasts the invalidation to Redis, and marks the outbox row as delivered. If the worker crashes, it resumes without ever dropping a message.

Production Reconciliation Runbook

Even with rigorous architecture, upstream payment networks occasionally experience total outages. What if Stripe experiences a multi-hour webhook delivery interruption?

Maintain an automated Daily Reconciliation Worker:

1. Run a nightly cron job that queries Stripe's List Invoices API for all subscriptions updated in the last 24 hours.
2. Compare Stripe's authoritative subscription status with your local database records.
3. If a discrepancy is detected (e.g. an invoice marked paid in Stripe is still past_due in your database), trigger an automated reconciliation and alert your engineering channel.

Practical Next Steps

1. Verify Webhook Signatures: Ensure your webhook endpoint verifies Stripe-Signature before reading payload fields.
2. Implement the stripe_events Unique Table: Insert every event with ON CONFLICT (stripe_event_id) DO NOTHING.
3. Add Monotonic Guards: Add last_stripe_event_created_at to your subscriptions schema and enforce timestamp checks on all updates.
4. Deploy the Daily Reconciliation Job: Build a periodic health check comparing local subscription status against Stripe's authoritative API.