Usage Metering Without Lost or Double-Billed Events: Where to Deduplicate

Usage-based pricing turns an engineering pipeline into part of the invoice. Every API call, stored gigabyte, or processed token you count becomes money, so a dropped event undercharges a customer and a counted-twice event overcharges one. Both are visible: the first in your revenue, the second in a dispute.

Event pipelines deliver at least once almost everywhere, so duplicates are normal. The design question is where you remove them. This guide walks through a metering pipeline from event to invoice and includes a lab comparing three common deduplication designs under the same failures. The most popular one, a Redis "seen this key" marker set before the event is written, lost about 1% of usage in the lab.

What a meter event needs

Make each event self-describing and stable, so it can be stored, replayed, and audited without guessing:

The idempotency key has to be created where the usage happens and stay the same across retries. A key generated by the ingestion endpoint is useless, because each retry would get a new one.

Where duplicates and losses come from

• Client retries. The client sends an event, the response is lost, and the client sends it again.
• Broker redelivery. A consumer processes a message and crashes before committing its offset, so the message is delivered again.
• Late re-sends. A device that was offline for days uploads a buffer that partly overlaps what it already sent.
• Partial failures inside your service. Your code records "seen" and then fails before the event is stored durably.

The last one is the subtle one, and it is what the lab is about.

Lab: where to deduplicate

The simulation sends 200,000 one-unit events through three designs using the same random sequence (seed 42), so the comparison is exact. The failure rates are made up but deliberately small: 1% transient broker-write errors, a 0.05% chance the process crashes between steps, 2% duplicate deliveries from client retries, and 0.1% of events re-sent by a device five days later. Clients retry errors up to five times.

• Design A sets a Redis marker with SET NX and a 72-hour expiry, then writes to the broker. A duplicate is any event whose marker exists.
• Design B is the same, but deletes the marker if the broker write returns an error.
• Design C writes every delivery to the broker and deduplicates in the billing ledger with a unique constraint on (tenant_id, idempotency_key), kept for the whole billing period.

Run it with go run .; it uses only the standard library.

// Lab: where should usage events be deduplicated?
//
// 200,000 metered events (1 unit each) pass through an ingestion endpoint.
// Failure rates are illustrative, not measured from a real system:
//   - 1% of broker writes fail transiently and the client retries (up to 5 tries)
//   - 0.05% of requests crash the ingest process after the dedup marker is set
//     but before the broker write; the client retries
//   - 2% of events are delivered twice by the client's network retry
//   - 0.1% of events are re-sent by an offline device 5 days later
//
// Three designs are compared:
//
//	A. SET NX marker (72 h TTL) first, then write to the broker
//	B. Same, but delete the marker when the broker write returns an error
//	C. Accept and write every delivery; deduplicate in the billing ledger with a
//	   unique (tenant, idempotency_key) constraint kept for the billing period
package main

import (
	"fmt"
	"math/rand/v2"
)

const (
	events       = 200_000
	pBrokerFail  = 0.01
	pCrash       = 0.0005
	pNetDup      = 0.02
	pLateResend  = 0.001
	markerTTLh   = 72
	lateResendAt = 5 * 24 // hours
	maxTries     = 5
)

type delivery struct {
	key  int
	hour int
}

type design struct {
	name         string
	markerFirst  bool
	clearOnError bool
	ledgerUnique bool
}

func run(d design, seed uint64) (billed, lost, doubled int) {
	rng := rand.New(rand.NewPCG(seed, 1))
	markers := map[int]int{} // key -> hour set
	ledger := map[int]int{}  // key -> times billed
	var deliveries []delivery
	for k := 0; k < events; k++ {
		h := rng.IntN(24 * 30)
		deliveries = append(deliveries, delivery{k, h})
		if rng.Float64() < pNetDup {
			deliveries = append(deliveries, delivery{k, h})
		}
		if rng.Float64() < pLateResend {
			deliveries = append(deliveries, delivery{k, h + lateResendAt})
		}
	}
	for _, dl := range deliveries {
		for try := 0; try < maxTries; try++ {
			if d.markerFirst {
				if set, ok := markers[dl.key]; ok && dl.hour-set < markerTTLh {
					break // treated as a duplicate and dropped with HTTP 202
				}
				markers[dl.key] = dl.hour
				if rng.Float64() < pCrash {
					continue // process died before the broker write; client retries
				}
			}
			if rng.Float64() < pBrokerFail {
				if d.clearOnError {
					delete(markers, dl.key)
				}
				continue // error returned; client retries
			}
			// durable write succeeded; the consumer posts to the ledger
			if d.ledgerUnique && ledger[dl.key] > 0 {
				break // unique constraint: ON CONFLICT DO NOTHING
			}
			ledger[dl.key]++
			break
		}
	}
	for k := 0; k < events; k++ {
		switch n := ledger[k]; {
		case n == 0:
			lost++
		case n > 1:
			doubled += n - 1
		}
		billed += ledger[k]
	}
	return
}

func main() {
	fmt.Printf("%d true units; same random sequence (seed 42) for every design\n\n", events)
	fmt.Printf("%-52s %9s %7s %13s\n", "design", "billed", "lost", "double-billed")
	for _, d := range []design{
		{"A  marker first, then broker write", true, false, false},
		{"B  marker first, cleared on broker error", true, true, false},
		{"C  write every delivery, unique key in ledger", false, false, true},
	} {
		b, l, x := run(d, 42)
		fmt.Printf("%-52s %9d %7d %13d\n", d.name, b, l, x)
	}
}

Output recorded on November 3, 2026 (Go 1.26.5, windows/amd64):

200000 true units; same random sequence (seed 42) for every design

design                                                  billed    lost double-billed
A  marker first, then broker write                      198173    2018           191
B  marker first, cleared on broker error                200104      92           196
C  write every delivery, unique key in ledger           200000       0             0

Why A loses usage

In design A, the marker means "accepted", but it is written before anything is accepted. When the broker write fails, the client correctly retries, finds the marker, and gets a success response for an event that was never stored. That happened to 2,018 of 200,000 events, about 1%: almost all from the 1% broker failure rate, plus a few from the much rarer crash case. In production nobody notices, because nothing errors. It shows up only as revenue that never existed in your data.

Design B fixes the error path but not the crash path: if the process dies after setting the marker and before writing, there is no code left to delete it. That lost 92 events.

Both A and B also double-billed about 190 events: the late re-sends arrived after the 72-hour marker had expired. Any time-limited dedup window has this problem for data older than the window.

Design C billed exactly 200,000 units. It accepts duplicates into the stream and removes them at the one place that decides money, using a constraint that lasts as long as the billing period. That is the general rule: deduplicate where the effect is committed, with a key retention at least as long as you accept late data.

The lab does not model a consumer that crashes between writing the ledger and committing its offset. That case is safe in design C for the same reason: the redelivered event hits the unique constraint.

A ledger table that enforces it

In PostgreSQL, the ledger can be one table with a primary key that includes the idempotency key. This schema is illustrative and was not executed as part of the lab:

CREATE TABLE usage_ledger (
    tenant_id       uuid        NOT NULL,
    idempotency_key text        NOT NULL,
    meter           text        NOT NULL,
    quantity        bigint      NOT NULL CHECK (quantity >= 0),
    event_time      timestamptz NOT NULL,
    billing_period  date        NOT NULL,   -- first day of the period it is billed in
    received_at     timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (tenant_id, idempotency_key)
);

-- Consumer: insert, and treat a conflict as "already counted".
INSERT INTO usage_ledger (tenant_id, idempotency_key, meter, quantity, event_time, billing_period)
VALUES ($1, $2, $3, $4, $5, $6)
ON CONFLICT (tenant_id, idempotency_key) DO NOTHING;

Keep ledger rows at least until the invoice for that period can no longer change, and longer if your contracts or tax rules require you to reproduce invoices. A Redis marker in front of the ledger is still fine as an optimization to skip obvious duplicates early, as long as the ledger constraint remains the authority.

Ordering and late events

If you use Kafka or a similar log, partitioning by tenant keeps one tenant's events in one partition, which preserves the order in which they were appended. That is not the order in which the usage happened: two producers, a retry, or an offline device can append an old event after a new one. Martin Kleppmann's Designing Data-Intensive Applications covers the difference between event time and processing time in its stream-processing chapter.

For billing, write down three rules:

1. Which period an event belongs to. Usually the period containing its event_time, not its arrival time.
2. When a period closes. For example, 48 hours after period end. Events arriving before the close are billed normally.
3. What happens after the close. Either bill late usage in the next invoice as a dated adjustment, or ignore usage older than a stated limit. Put whichever you choose in the customer agreement.

Do not reject events purely because their timestamp is far from the server clock unless your contract says so. Clamp obviously impossible times (future dates, times before the account existed) and flag them for review instead.

Rollups for invoices and dashboards

Summing raw events at invoice time works until the table is large. Pre-aggregate into hourly or daily buckets. TimescaleDB's continuous aggregates refresh incrementally and recalculate only the buckets whose raw rows changed, which suits late events: an event for last Tuesday updates last Tuesday's bucket. Check how far back your refresh policy looks; a late event older than the refresh window will not appear until you refresh that range.

Whatever you use, keep the raw ledger. When a customer questions an invoice, you need to show the individual events behind each bucket.

Sending usage to a billing provider

If Stripe rates your usage, you report usage as billing meter events. Three details from its documentation shape the design:

• Each meter event can carry an identifier, and Stripe enforces uniqueness for it over a rolling period of at least 24 hours. Use a deterministic identifier, such as tenant:meter:hour, so a retry of the same report is not counted twice.
• The event timestamp must be within the past 35 calendar days and no more than 5 minutes in the future. Usage older than that cannot be reported as a normal meter event, so your late-data rule has to fit inside this window.
• A meter event adjustment can cancel a single event by identifier, but only within 24 hours of Stripe receiving it. After that, corrections go through credits or invoice adjustments.

Report from your ledger or rollups, not directly from the ingestion path. That way the provider receives totals that have already been deduplicated, and you can reconcile its invoice totals against your own sums every period.

Reconciliation

Once per period, compare three numbers for every tenant and meter: the sum in your ledger, the sum you reported to the billing provider, and the quantity on the provider's invoice. Investigate any difference before the invoice is finalized. Small, frequent differences usually point to a bug like design A's; large, rare ones usually point to a backfill or a misconfigured meter.

The tenant unit economics calculator can then turn those usage totals and your costs into a margin estimate, and the proration guide covers what happens when a tenant changes plans in the middle of a metered period.

Where to start

Check where your current pipeline decides that an event is a duplicate, and what has been durably written at that moment. If the answer is "a cache marker, before the event is stored", you are probably losing usage at roughly your write-failure rate. Add a unique constraint where usage is committed, keep it for the billing period, and run a reconciliation for last month before changing anything else.

Frequently Asked Questions

Is a Redis marker in front of the ledger still useful?

Yes, as an optimization to drop obvious duplicates before they reach the broker, provided it is set only after the event is durably written and the ledger's unique constraint stays the authority. Losing the marker should never change what is billed.

Should I send every event to the billing provider or send totals?

Totals from your deduplicated ledger, per tenant and meter, at a fixed interval such as hourly, usually work better. They keep provider API volume predictable, let you use deterministic identifiers for retries, and make reconciliation a comparison of sums.

How long should I keep raw usage events?

At least until the invoices that use them can no longer change, plus any period your contracts, audit needs, or tax rules require for reproducing an invoice. Older raw events can move to cheaper storage as long as they can be queried during a dispute.

What if a customer's clock is wrong?

Use event time for period assignment, but sanity-check it against the server's receive time. Clamp impossible values, such as future timestamps or times before the account existed, and flag them for review instead of silently dropping usage.

Can Kafka's exactly-once features replace ledger deduplication?

They help within Kafka, for example avoiding duplicate writes from producer retries, but they do not see a client that sends the same usage twice or a device that re-uploads days later. Business-level deduplication by idempotency key is still needed.

Sources

Documents cited in this guide. Each link was opened and its title compared with the label below on the date shown. A source supports the specific statements it is linked from; design recommendations and lab results are Velynoriva's own.

• Create a billing meter event (Stripe) • Link checked 2026-11-03

  Identifier uniqueness over at least 24 hours; timestamps within the past 35 days or up to 5 minutes ahead.

• Billing Meter Event Adjustments (Stripe) • Link checked 2026-11-03

  Cancelling a single meter event by identifier within 24 hours of receipt.

• Basic usage-based billing (Stripe) • Link checked 2026-11-03

  Overview of meters and recording usage.

• Understand continuous aggregates (Tiger Data (TimescaleDB)) • Link checked 2026-11-03

  Incremental refresh of only the time buckets whose raw data changed.

• Designing Data-Intensive Applications (Martin Kleppmann / O'Reilly) • Link checked 2026-11-03

  Book site; the stream processing chapter discusses event time versus processing time.

Revision history

• 2026-11-03 — Added a reproducible lab showing that the previous marker-before-write deduplication loses usage. Moved the authoritative deduplication to the billing ledger, removed the conflicting 24-hour rejection and 48-hour settlement rules, removed an unsupported ingestion latency figure, and corrected the Stripe meter event example (identifier, timestamp limits).
• 2026-10-12 — Reframed exactly-once billing as business idempotency and corrected ordering claims.