π Language / NgΓ΄n ngα»―: English β’ TiαΊΏng Viα»t
FlashPass is an ultra-low latency, distributed ticket booking and flash-sale reservation engine engineered to sustain 20,000+ requests/second (RPS) during high-demand concert ticket drops without overselling, database saturation, or race-condition financial leakage.
- Architectural Overview
- Key Performance Indicators (KPIs)
- Core Invariants & Design Principles
- 12 Distributed Edge Cases & Battle-Tested Solutions
- Technology Stack
- System Architecture & Data Flows
- Clean Architecture & Project Structure
- API Reference Overview
- Production Quickstart & Deployment
- Load Testing & Benchmark Proof
- Operational Runbook & Disaster Recovery
- Contributing & License
FlashPass separates the high-speed in-memory ingestion tier from the persistent relational database storage tier through asynchronous queue decoupling, atomic Lua scripts, and robust distributed state machines.
flowchart TD
Client["π Web / Mobile Client (Next.js 15)"]
subgraph Edge ["Gateway & Traffic Control"]
RateLimiter["β‘ Redis Sliding-Window Rate Limiter (ZSET)"]
SingleFlight["π‘ SingleFlight Cache Stampede Guard"]
end
subgraph MemoryTier ["In-Memory Atomic Layer (Redis 7)"]
Sentinel["π‘ Redis Sentinel Cluster (Port 26379)"]
MasterRedis["β‘ Redis Master (Port 6379) - Atomic Lua Engine"]
ReplicaRedis["π Redis Replica (Port 6380) - Async / WAIT Replication"]
WatchdogLock["π Watchdog Distributed Lock (Lease Renewal TTL/3)"]
end
subgraph Ingestion ["Core Ingestion Engine (Go Fiber)"]
BookingService["Booking Service"]
OrderService["Order Service (Debounce & State Machine)"]
ReconcileService["Stock Auditor (Discrepancy Self-Healer)"]
CircuitBreaker["β‘ Database Circuit Breaker (Closed / Open / Half-Open)"]
end
subgraph AsyncPipeline ["Asynchronous Broker (RabbitMQ 3.13)"]
OrderExchange["Direct Exchange: flashpass.order.exchange"]
DelayedExchange["Delayed Exchange: flashpass.order.delayed.exchange (TTL 10m)"]
DLX["DLX: flashpass.dlx -> flashpass.order.dlq"]
RefundQueue["Refund Queue: flashpass.order.refund.queue"]
end
subgraph Persistence ["Persistent Storage (PostgreSQL 16)"]
DB[("PostgreSQL 16 Engine")]
OrdersTable["orders (UUID, Status, IdempotencyKey)"]
RefundsTable["refund_orders (UUID, OrderID, RefundID)"]
EventsTable["events (ID, TotalTickets, AvailableTickets)"]
end
Client -->|POST /api/v1/tickets/book| RateLimiter
RateLimiter -->|Pass| BookingService
RateLimiter -->|Burst Exceeded| Client
BookingService -->|Atomic Lua Check & DECRBY| MasterRedis
MasterRedis -.->|Asynchronous Replication| ReplicaRedis
Sentinel -.->|Auto-Failover Monitoring| MasterRedis
BookingService -->|Publish order.create| OrderExchange
OrderExchange -->|Worker Consumer| DB
OrderExchange -->|Failed Retry x2| DLX
OrderExchange -->|Schedule 10m Expiration| DelayedExchange
DelayedExchange -->|Timeout Check & Expiration| DB
DB -->|Rollback Stock on Expiration| MasterRedis
Client -->|POST /api/v1/orders/:id/pay| OrderService
OrderService -->|Acquire Lock| WatchdogLock
OrderService -->|Circuit Breaker Guard| CircuitBreaker
CircuitBreaker -->|Persist Paid / Expired| DB
OrderService -->|Late Payment Detected| RefundQueue
| Metric | Target SLA | Production Result | Validation Status |
|---|---|---|---|
| Overselling Invariant | Exactly 0.00% (Zero oversold tickets) | 0.00% (Strict 1,000 / 1,000) | β PASSED |
| Throughput Capacity | > 8,000 RPS (Single Node) | 8,285 RPS | β PASSED |
| Fail-Fast Latency | < 2.00 ms (Directly in RAM) | 947 Β΅s | β PASSED |
| p95 Response Time | < 25.00 ms | 18.83 ms | β PASSED |
| p99 Response Time | < 50.00 ms | 29.42 ms | β PASSED |
| Server Error Rate | < 0.01% HTTP 5xx | 0.00% (0 errors / 20k requests) | β PASSED |
| Deduplication Reliability | 100% duplicate suppression | 100.00% idempotent | β PASSED |
- Atomic Check-and-Decrement Invariant:
Inventory checking and deduction are performed strictly within a single atomic Redis Lua script. Application-level
GETfollowed byDECRBYis forbidden to prevent race conditions. - Fail-Fast In-Memory Rejection:
When inventory reaches
0, incoming booking requests are rejected in sub-millisecond RAM checks (HTTP409 Conflict), never touching PostgreSQL connection pools or RabbitMQ channels. - Strict Idempotency:
Every reservation requires a unique client-side
idempotency_key(persisted across user tab reloads insessionStorage). Duplicate submissions return the cached reservation result without double-decrementing stock. - Decoupled Database Persistence:
The HTTP booking API does not write directly to PostgreSQL. Requests write to Redis and enqueue a message in RabbitMQ (
202 Accepted), protecting relational databases from write saturation. - Authoritative Expiration & Grace Windows:
Payment deadlines are strictly enforced by database timestamps (
expires_at <= NOW()), coupled with a 5-second grace window to absorb distributed network transit delays and NTP clock drift.
FlashPass is hardened against the full spectrum of distributed systems failures encountered in mission-critical payment and flash-sale infrastructures:
| # | Edge Case Scenario | Real-World Failure Mode | FlashPass Solution | Automated Test Suite |
|---|---|---|---|---|
| 1 | Redis Master Failover | Master crashes before async replication reaches replica; promoted replica double-sells stock. | Dual-Node Sentinel failover, synchronous WAIT 1 verification, and PostgreSQL database constraint guard. |
TestEdgeCase1_RedisFailover |
| 2 | Timeout Worker vs Late Webhook Race | Order expires at 10m00s releasing stock; payment arrives at 10m01s causing duplicate seat allocation. | Optimistic state machine rejection, ticket retention for buyer B, and automatic background refund pipeline (order.refund). |
TestEdgeCase2_TimeoutRaceCondition |
| 3 | Dual-Write Partial Failure (Ghost Inventory) | Redis stock decremented, but network drops before RabbitMQ publish; tickets vanish from inventory. | Stock Auditor (ReconcileService) with 2-way clamping, Hit-Zero trigger, and orphaned user lock cleanup. |
TestEdgeCase3_DualWriteGhostInventory |
| 4 | Redis Cluster Cross-Slot Partitioning | Keys mapped to different slots in a 16,384-slot Redis cluster cause Lua multi-key CROSSSLOT fatal errors. |
Hash Tag co-location (flashpass:{eventId}:stock, flashpass:{eventId}:user_booked:{userId}) mapped to identical CRC16 hash slots. |
TestEdgeCase4_CrossSlotHashTag |
| 5 | RabbitMQ Poison Pill & Queue Deadlock | Corrupted payload or DB constraint violation causes infinite NACK retry loops, stalling worker. | Dead Letter Exchange (flashpass.dlx), automatic quarantine to DLQ on 2nd retry, and Redis stock rollback. |
TestEdgeCase5_RabbitMQDeadLetterQueue |
| 6 | Cache Stampede & Thundering Herd | Cache expires during peak surge; 10,000 requests hit PostgreSQL simultaneously, exhausting connections. |
singleflight.Group request coalescing collapsing 100 concurrent read queries into exactly 1 database query. |
TestEdgeCase6_CacheStampedeSingleFlight |
| 7 | Double Payment Webhook Concurrency | Payment gateway retries webhooks concurrently; parallel threads process double credit / double refunds. | Distributed lock (SETNX 5s), 100ms settle wait, winner-only lock release, and DB row-count idempotent fallback. |
TestEdgeCase7_DoublePaymentWebhookConcurrency |
| 8 | NTP Clock Drift / Skew | Worker clock running ahead causes premature order cancellation before the 10-minute user window elapses. | 5-second Clock Drift Grace Window and PostgreSQL authoritative server time evaluation (expires_at <= NOW()). |
TestEdgeCase8_ClockDriftGraceWindow |
| 9 | Bot Traffic Burst & Scalper Swarms | High-frequency bot scripts overwhelm booking API, degrading service for legitimate users. | Distributed Sliding-Window Log rate limiter in Redis Lua (ZSET), returning HTTP 429 with Retry-After: 2. |
TestEdgeCase9_SlidingWindowRateLimiter |
| 10 | Out-of-Order Message Inversion | Network reordering causes Payment/Cancellation to arrive before asynchronous Order Creation. | Early Payment retry debounce (100ms) + Cancellation Tombstone pattern preventing late order resurrection. | TestEdgeCase10_OutOfOrderMessageProcessing |
| 11 | Database Cascading Failure | Slow DB queries saturate connection pool (50 conns); threads lock up, bringing down the entire API. | State-machine Circuit Breaker (Closed |
TestEdgeCase11_DatabaseCircuitBreaker |
| 12 | Split-Brain Distributed Lock Hijacking | Long GC pause or slow network causes lock TTL to expire while task runs; another thread acquires lock. | Heartbeat Watchdog goroutine renewing lease every |
TestEdgeCase12_DistributedLockWatchdog |
- Programming Language: Go (Golang) 1.22+
- HTTP Web Framework: Fiber v2 (FastHTTP engine, zero-allocation routing)
- In-Memory Cache & Distributed Lock: Redis 7.x (Alpine) with Redis Sentinel (port 26379)
- Message Broker: RabbitMQ 3.13+ with
rabbitmq_delayed_message_exchangeplugin - Database Engine: PostgreSQL 16+ using connection pooling (
jackc/pgx/v5/pgxpool) - Resilience & Concurrency: Go
singleflight, distributed locks, atomic Lua scripts, sync primitives
- Framework: Next.js 15 (React 19, App Router)
- Styling: Tailwind CSS, Lucide Icons, Canvas Confetti
- Real-Time Streaming: Server-Sent Events (SSE) & WebSocket fallbacks
- Client Storage:
sessionStoragefor session idempotency,localStoragefor load test benchmarks
Before ticket sales commence, an administrator invokes the pre-warming endpoint:
POST /api/v1/admin/events/:id/prewarm- Backend queries PostgreSQL for
total_tickets. - Sets
flashpass:{event_id}:stockin Redis memory. - Initializes event metadata cache for zero-DB reads during sale surge.
sequenceDiagram
autonumber
actor User as Buyer
participant API as Fiber Ingestion API
participant Rate as Redis Rate Limiter
participant Redis as Redis Master (Lua)
participant RMQ as RabbitMQ (Order Exchange)
User->>API: POST /api/v1/tickets/book (idempotency_key)
API->>Rate: Check Sliding Window (ZSET)
alt Rate Limit Exceeded
Rate-->>API: Limit Exceeded (Burst > 5)
API-->>User: HTTP 429 Too Many Requests (Retry-After: 2)
end
API->>Redis: EVALSHA atomic_decrement(event_id, user_id)
alt User Already Booked
Redis-->>API: Return -2 (Already Booked)
API-->>User: HTTP 409 Conflict ("You have already reserved a ticket")
else Out of Stock
Redis-->>API: Return -1 (Sold Out)
API-->>User: HTTP 409 Conflict ("Event sold out")
else Stock Decremented
Redis-->>API: Return 1 (Success)
API->>RMQ: Publish order.create event (Thread-Safe Mutex)
API-->>User: HTTP 202 Accepted { reservation_id, expires_in: 600 }
end
- Fulfillment Consumer: Reads
order.createqueue, persists order into PostgreSQL withPENDING_PAYMENT, and publishes a delayed message withx-delay: 600000(10 minutes). - Timeout Consumer: Reads message after 10 minutes. Calls PostgreSQL authoritative expiration:
UPDATE orders SET status = 'EXPIRED' WHERE id = $1 AND status = 'PENDING_PAYMENT' AND expires_at <= NOW()
- If updated, the worker issues an atomic
INCRBY flashpass:{event_id}:stock 1and clears the user lock in Redis, releasing the ticket back to the public pool.
If a payment arrives after expiration:
POST /api/v1/orders/:id/payacquires a distributed lock (WatchdogLock).- Detects
status = EXPIREDor past expiration deadline + grace window. - Automatically writes a record to
refund_orderstable. - Enqueues refund task to RabbitMQ
flashpass.order.refund.queue. - Background Refund Worker calls payment gateway refund API with exponential backoff and sets status to
REFUNDED.
FlashPass adheres to Hexagonal / Clean Architecture principles, decoupling transport layers from domain services and database repositories:
FlashPass/
βββ backend/
β βββ cmd/
β β βββ api/ # Main HTTP API Entry Point (Fiber)
β β βββ worker/ # Background Worker (Consumers & Cron Reconciler)
β βββ internal/
β β βββ api/ # HTTP Transport Controllers & Modular Integration Tests
β β β βββ handlers.go # Thin Fiber Controllers
β β β βββ router.go # API Route Registrations
β β β βββ test_helpers_test.go # Shared Test Fixtures & HTTP Setup
β β β βββ booking_flow_test.go # Core Booking Flow Integration Tests
β β β βββ payment_flow_test.go # Payment Lifecycle & Auto-Refund Tests
β β β βββ edge_case_1..12_*_test.go # 12 Modular Distributed Chaos & Edge Case Tests
β β βββ config/ # Viper / Environment Configuration
β β βββ consumer/ # RabbitMQ Consumers (Order, Timeout, Refund)
β β βββ database/ # PostgreSQL Connection Pool & Embedded Migrations
β β βββ models/ # Domain Entities & Status Enums
β β βββ rabbitmq/ # AMQP Connection, Thread-Safe Producer & DLX
β β βββ redis/ # Redis Pool, Sentinel, Lua Scripts, Watchdog Lock
β β βββ repository/ # Persistence Layer (Event, Order, Refund)
β β βββ resilience/ # Circuit Breaker & Concurrency Utilities
β β βββ service/ # Core Business Logic (Booking, Order, Reconcile)
βββ frontend/ # Next.js 15 Client & Interactive Simulation Lab
β βββ app/ # Next.js App Router Pages
β βββ components/
β βββ edge-cases/ # Interactive Distributed Chaos Lab Components
β βββ ...
βββ docs/ # Technical Specs, Diagrams & Production Runbooks
β βββ API_REFERENCE.md # Comprehensive REST API Specification
β βββ RUNBOOK.md # Day-2 Operations & Incident Playbook
β βββ DISTRIBUTED_EDGE_CASES.md # Deep-Dive Engineering Analysis (12 Scenarios)
β βββ benchmark_chart.png # k6 Visual Benchmark Results
βββ scripts/ # Load Testing & Automation Scripts
β βββ load-test.js # k6 20,000-request Surge Test
β βββ visualize_benchmark.py # Matplotlib Graph Generator
βββ docker-compose.yml # Multi-Node Cluster Orchestration
βββ SPEC.md # Formal System Specification
βββ ARCHITECTURE.md # Technical Design Blueprint
βββ TASKS.md # Implementation Roadmap & Verification Checklist
βββ README.md # Master Documentation
Detailed request/response contracts, headers, and schemas are available in docs/API_REFERENCE.md.
| Method | Endpoint | Description | Key Headers / Body |
|---|---|---|---|
GET |
/health |
System liveness probe | Returns 200 OK |
GET |
/api/v1/events/:id |
Event metadata (SingleFlight protected) | Cached response |
GET |
/api/v1/events/:id/stock |
Real-time ticket inventory in RAM | Returns available tickets |
GET |
/api/v1/events/:id/stream |
Server-Sent Events (SSE) live updates | Stream of inventory changes |
POST |
/api/v1/tickets/book |
Flash Sale Ticket Booking | Requires idempotency_key |
GET |
/api/v1/orders/:id |
Query order details & payment deadline | Returns order status |
POST |
/api/v1/orders/:id/pay |
Mock Payment Webhook / Confirmation | Idempotent transition to PAID |
POST |
/api/v1/orders/:id/cancel |
Explicit order cancellation | Sets CANCELLED tombstone |
POST |
/api/v1/admin/events/:id/prewarm |
Pre-load DB inventory into Redis RAM | Admin token |
POST |
/api/v1/admin/events/:id/reconcile |
Stock Auditor Discrepancy Self-Healing | Resolves ghost inventory |
POST |
/api/v1/admin/events/:id/reset |
Reset event stock & clean cache | Custom tickets parameter |
- Docker Engine 24+ & Docker Compose v2+
- Go 1.22+ (for local development & running test suite)
- Node.js 18+ & npm/pnpm (for frontend)
Clone the repository and start the cluster:
git clone https://github.com/your-org/FlashPass.git
cd FlashPass
# Start PostgreSQL 16, Redis Master, Redis Replica, Redis Sentinel, and RabbitMQ
docker compose up -dVerify that all 5 cluster services are healthy:
docker compose ps| Service | Container Name | Ports | Role |
|---|---|---|---|
| PostgreSQL | flashpass_postgres |
5432:5432 |
Relational Storage & Write Model |
| Redis Master | flashpass_redis |
6379:6379 |
In-Memory Atomic Engine |
| Redis Replica | flashpass_redis_replica |
6380:6379 |
Replication & Failover Standby |
| Redis Sentinel | flashpass_redis_sentinel |
26379:26379 |
Cluster Health & Auto-Promotion |
| RabbitMQ | flashpass_rabbitmq |
5672:5672, 15672:15672 |
Message Broker & Management UI |
cd backend
# Copy configuration
cp .env.example .env
# Run HTTP API Server
go run cmd/api/main.go
# In a separate terminal, run the Background Worker
go run cmd/worker/main.gocd frontend
npm install
npm run dev
# Open http://localhost:3000 in your browserFlashPass comes with an enterprise test suite verifying all 12 distributed edge cases against real Docker containers:
cd backend
# Execute all 12 Distributed Edge Cases tests
go test -v -run "TestEdgeCase" ./internal/api
# Run the complete test suite across all packages
go test -v ./...Expected output:
=== RUN TestEdgeCase1_RedisFailover
--- PASS: TestEdgeCase1_RedisFailover (1.05s)
=== RUN TestEdgeCase2_TimeoutRaceCondition
--- PASS: TestEdgeCase2_TimeoutRaceCondition (1.10s)
=== RUN TestEdgeCase3_DualWriteGhostInventory
--- PASS: TestEdgeCase3_DualWriteGhostInventory (1.02s)
=== RUN TestEdgeCase4_CrossSlotHashTag
--- PASS: TestEdgeCase4_CrossSlotHashTag (1.01s)
=== RUN TestEdgeCase5_RabbitMQDeadLetterQueue
--- PASS: TestEdgeCase5_RabbitMQDeadLetterQueue (1.05s)
=== RUN TestEdgeCase6_CacheStampedeSingleFlight
--- PASS: TestEdgeCase6_CacheStampedeSingleFlight (1.02s)
=== RUN TestEdgeCase7_DoublePaymentWebhookConcurrency
--- PASS: TestEdgeCase7_DoublePaymentWebhookConcurrency (1.08s)
=== RUN TestEdgeCase8_ClockDriftGraceWindow
--- PASS: TestEdgeCase8_ClockDriftGraceWindow (1.04s)
=== RUN TestEdgeCase9_SlidingWindowRateLimiter
--- PASS: TestEdgeCase9_SlidingWindowRateLimiter (1.02s)
=== RUN TestEdgeCase10_OutOfOrderMessageProcessing
--- PASS: TestEdgeCase10_OutOfOrderMessageProcessing (1.15s)
=== RUN TestEdgeCase11_DatabaseCircuitBreaker
--- PASS: TestEdgeCase11_DatabaseCircuitBreaker (1.52s)
=== RUN TestEdgeCase12_DistributedLockWatchdog
--- PASS: TestEdgeCase12_DistributedLockWatchdog (1.51s)
PASS
ok flashpass/internal/api 13.2s
A stress test simulating 20,000 concurrent requests competing for 1,000 tickets was conducted using Grafana k6:
k6 run scripts/load-test.js- Total Requests Processed: 20,000 requests in 2.41s
- Throughput: 8,285 Requests/sec (RPS)
- Accepted Reservations (HTTP 202): Exactly 1,000 (100% of inventory)
- Fail-Fast Sold Out (HTTP 409): Exactly 19,000
- Database Orders Created: Exactly 1,000
- Overselling Invariant: 0.00% (Zero Overselling)
- p95 Latency: 18.83 ms (Target SLA: < 25 ms)
- p99 Latency: 29.42 ms (Target SLA: < 50 ms)
For an in-depth breakdown of metrics, charts, and breaking limits, consult BENCHMARK.md.
For production Day-2 operations, cluster maintenance, and emergency response:
- docs/RUNBOOK.md: Alert thresholds, manual reconciliation steps, dead-letter queue reprocessing, Redis Sentinel switchover commands, and circuit breaker tuning.
- docs/DISTRIBUTED_EDGE_CASES.md: Deep-dive mathematical proofs, Lua scripts, and architecture mechanics of all 12 edge cases.
- docs/API_REFERENCE.md: Complete HTTP REST schemas and integration guides.
This project is licensed under the MIT License. Built for high-volume enterprise distributed systems engineering.
