Reproducible Labs

FIFO vs. a published DRR dispatcher vs. corrected DRR

Question: When one tenant enqueues 2,000 jobs, how long does another tenant's single job wait?

What the output showed: FIFO: 24.9 s. The published dispatcher never ran the other tenants' jobs and then looped forever. Corrected DRR: 40 ms.

From the guide: Fair Background Job Queues for Multi-Tenant Systems: Testing Deficit Round-Robin

One hypothetical month, three allocation rules

Question: How much does the choice of allocation rule change tenant margins on the same $30,000 of shared cost?

What the output showed: One tenant ranged from a 54% margin to -245%. An equal split showed small tenants at -40% as a group. Keeping idle capacity separate removed $11,400 from tenant costs.

From the guide: Per-Tenant Cost Attribution on Shared Infrastructure: Rules, Idle Capacity, and Reconciliation

One waiting ALTER TABLE and the lock queue

Question: While a long report holds the table, what happens to application queries when a migration waits for its lock?

What the output showed: No lock_timeout: 7,500 of 12,000 queries failed. A 500 ms lock_timeout with retries: none failed, worst wait 490 ms, migration done 2.5 s after the report.

From the guide: PostgreSQL Schema Changes Without Downtime: Locks, the Lock Queue, and Expand-Contract

Downtime budgets and a credit-table gap

Question: How much does month length change a downtime budget, and do common credit bands cover every value?

What the output showed: At 99.9%, February allows 40 min 19 s and 31-day months 44 min 38 s. Bands '99.0-99.89%' and '99.9-100%' matched nothing for 2,600 to 2,851 s of monthly downtime.

From the guide: Status Pages and SLA Reporting: Independent Hosting, Real Probes, and Gap-Free Credit Tables

Does W3C baggage reach your spans?

Question: If the gateway puts tenant.id in baggage, which spans carry it as an attribute?

What the output showed: None by default, although the header reached the downstream service. With a span processor copying tenant.id from baggage, all three spans carried it.

From the guide: Tenant-Aware Observability with OpenTelemetry: Baggage, Spans, and Metric Cardinality

Local AES-GCM cost and moving ciphertext between tenants

Question: How expensive is local encryption, and what stops an encrypted value from being copied to another tenant's row?

What the output showed: AES-256-GCM sealed 4 KiB in 0.96 µs and 1 MiB in 0.23 ms on a laptop CPU. With tenant and row IDs as associated data, a copied ciphertext failed authentication.

From the guide: Customer-Managed Encryption Keys: Envelope Encryption and Real Revocation Time

Parsing and testing a tenant-aware Rego policy

Question: Does the previously published policy compile under OPA 1.x, and does the corrected policy's batch rule agree with its single-decision rule?

What the output showed: The old policy failed to parse as Rego v1 and as v0. The corrected policy passed 7 of 7 tests, including a viewer getting no batch permissions.

From the guide: RBAC and ABAC for Multi-Tenant Products: Custom Roles and Tested Rego Policies

Concurrent refresh-token replay against two stores

Question: When the same refresh token is presented twice at once, how often does a read-then-write store miss the reuse?

What the output showed: About 72% of 20,000 trials ended with two live refresh tokens and no alert. An atomic check-and-consume detected reuse in every trial.

From the guide: Session Revocation and Refresh Token Rotation: Where the Design Breaks

Six ways to tamper with a hash-chained log

Question: Which edits and deletions does a hash chain detect with and without an external checkpoint?

What the output showed: A full rehash after an edit and an edit to record 0 both passed a chain-only verifier. A genesis check plus an external checkpoint caught them. Deletions after the last checkpoint passed both.

From the guide: Tamper-Evident Audit Logs: What a Hash Chain Detects and What It Misses

No limit vs. a global limit vs. per-tenant token buckets

Question: During a 2,000 req/s flood from one tenant, how many of the other tenants' requests are served?

What the output showed: No limit: 24.4%. Global limit: 17.2%, because the flooder takes most tokens. Per-tenant buckets: 100%, with a 34 ms p99.

From the guide: Per-Tenant Rate Limiting: Containing Noisy Neighbors in a Shared API

Row-level security on PostgreSQL 18.6

Question: Does the policy fail closed, what leaks through a reused connection, and who bypasses it?

What the output showed: A session-level SET leaked tenant A's rows to the next request, and a string-formatted SET LOCAL let tenant B read tenant A. set_config with a bind parameter prevented both.

From the guide: PostgreSQL Row-Level Security for Multi-Tenant Tables: Policies, Pools, and Bypass Rules

A tenant_id string linter vs. a structural rule

Question: Across eight realistic call sites, which unsafe queries does each rule catch?

What the output showed: The string rule missed queries in constants, built by concatenation, or with tenant_id only in a comment, and flagged two safe calls. The structural rule caught every unsafe call.

From the guide: Preventing Cross-Tenant Data Leaks: Structure, Database Policies, and Probes

Shared pool vs. dedicated databases: capacity

Question: For 200 bursty tenants in eight time zones, how much less capacity does one shared pool need than per-tenant databases?

What the output showed: 76% less at the peak minute (594 vs. 2,482 units); average utilization 53% shared vs. 13% dedicated.

From the guide: Shared Tables, Schema per Tenant, or Database per Tenant: Choosing a PostgreSQL Isolation Model

Host normalization edge cases and singleflight cancellation

Question: What does a lowercase-and-cut-at-colon normalizer get wrong, and what happens to 500 waiting lookups when the first caller cancels?

What the output showed: The shortcut turned an IPv6 host into "[2001" and kept duplicate keys. Cancelling the first caller failed all 500 shared lookups; a detached context failed only that one.

From the guide: Tenant Routing by Hostname: Subdomains, Customer Domains, and a Safe Resolver

Cached entitlements after a lost invalidation message

Question: With 0.5% of Pub/Sub invalidations lost across 20 servers, how long can a server serve old entitlements?

What the output showed: Messages only: up to 58 minutes. With a 60 s cache lifetime: 38 s. With a 5 s version poll: 1.6 s.

From the guide: Entitlements for Subscription Plans: Data Model, Caching, and Stale Permissions

Five ways to apply out-of-order webhook events

Question: After 42,126 unordered and duplicated deliveries, how many of 10,000 subscriptions does each strategy leave wrong?

What the output showed: Arrival order: 30.5%. created guard with a NULL start: 100%. created guard otherwise: about 14%. Fetching the current object: 0%.

From the guide: Stripe Webhooks Out of Order: Why created Timestamps Fail and What to Do Instead

One upgrade, twelve months, three shortcuts

Question: How much do 30-day months, rounding order, daylight saving, and 365-day years change a proration?

What the output showed: 30-day months: +$4.76 in February, -$2.16 in 31-day months. Rounding order: 1 cent. A 365-day year overcharged a leap-year true-up by $16.44.

From the guide: Subscription Proration in Practice: Month Lengths, Rounding, and Plan Changes