Skip to content
janhae4Public

About

FlashPass: High-Concurrency Distributed Flash Sale & Ticketing Engine (Go Fiber, Redis Sentinel, Lua, RabbitMQ, PostgreSQL 16, Next.js 15)

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

⚑ FlashPass: High-Concurrency Distributed Flash Sale & Ticketing Engine

Go Version Fiber Next.js Redis RabbitMQ PostgreSQL Docker Overselling Rate License

🌐 Language / NgΓ΄n ngα»―: English β€’ TiαΊΏng Việt

FlashPass is an ultra-low latency, distributed ticket booking and flash-sale reservation engine engineered to sustain 20,000+ requests/second (RPS) during high-demand concert ticket drops without overselling, database saturation, or race-condition financial leakage.


πŸ“Œ Table of Contents


πŸ— Architectural Overview

FlashPass separates the high-speed in-memory ingestion tier from the persistent relational database storage tier through asynchronous queue decoupling, atomic Lua scripts, and robust distributed state machines.

flowchart TD
    Client["🌐 Web / Mobile Client (Next.js 15)"]
    
    subgraph Edge ["Gateway & Traffic Control"]
        RateLimiter["⚑ Redis Sliding-Window Rate Limiter (ZSET)"]
        SingleFlight["πŸ›‘ SingleFlight Cache Stampede Guard"]
    end

    subgraph MemoryTier ["In-Memory Atomic Layer (Redis 7)"]
        Sentinel["πŸ›‘ Redis Sentinel Cluster (Port 26379)"]
        MasterRedis["⚑ Redis Master (Port 6379) - Atomic Lua Engine"]
        ReplicaRedis["πŸ”„ Redis Replica (Port 6380) - Async / WAIT Replication"]
        WatchdogLock["πŸ”’ Watchdog Distributed Lock (Lease Renewal TTL/3)"]
    end

    subgraph Ingestion ["Core Ingestion Engine (Go Fiber)"]
        BookingService["Booking Service"]
        OrderService["Order Service (Debounce & State Machine)"]
        ReconcileService["Stock Auditor (Discrepancy Self-Healer)"]
        CircuitBreaker["⚑ Database Circuit Breaker (Closed / Open / Half-Open)"]
    end

    subgraph AsyncPipeline ["Asynchronous Broker (RabbitMQ 3.13)"]
        OrderExchange["Direct Exchange: flashpass.order.exchange"]
        DelayedExchange["Delayed Exchange: flashpass.order.delayed.exchange (TTL 10m)"]
        DLX["DLX: flashpass.dlx -> flashpass.order.dlq"]
        RefundQueue["Refund Queue: flashpass.order.refund.queue"]
    end

    subgraph Persistence ["Persistent Storage (PostgreSQL 16)"]
        DB[("PostgreSQL 16 Engine")]
        OrdersTable["orders (UUID, Status, IdempotencyKey)"]
        RefundsTable["refund_orders (UUID, OrderID, RefundID)"]
        EventsTable["events (ID, TotalTickets, AvailableTickets)"]
    end

    Client -->|POST /api/v1/tickets/book| RateLimiter
    RateLimiter -->|Pass| BookingService
    RateLimiter -->|Burst Exceeded| Client

    BookingService -->|Atomic Lua Check & DECRBY| MasterRedis
    MasterRedis -.->|Asynchronous Replication| ReplicaRedis
    Sentinel -.->|Auto-Failover Monitoring| MasterRedis

    BookingService -->|Publish order.create| OrderExchange
    OrderExchange -->|Worker Consumer| DB
    OrderExchange -->|Failed Retry x2| DLX
    OrderExchange -->|Schedule 10m Expiration| DelayedExchange

    DelayedExchange -->|Timeout Check & Expiration| DB
    DB -->|Rollback Stock on Expiration| MasterRedis

    Client -->|POST /api/v1/orders/:id/pay| OrderService
    OrderService -->|Acquire Lock| WatchdogLock
    OrderService -->|Circuit Breaker Guard| CircuitBreaker
    CircuitBreaker -->|Persist Paid / Expired| DB
    OrderService -->|Late Payment Detected| RefundQueue
Loading

🎯 Key Performance Indicators (KPIs)

Metric Target SLA Production Result Validation Status
Overselling Invariant Exactly 0.00% (Zero oversold tickets) 0.00% (Strict 1,000 / 1,000) βœ… PASSED
Throughput Capacity > 8,000 RPS (Single Node) 8,285 RPS βœ… PASSED
Fail-Fast Latency < 2.00 ms (Directly in RAM) 947 Β΅s βœ… PASSED
p95 Response Time < 25.00 ms 18.83 ms βœ… PASSED
p99 Response Time < 50.00 ms 29.42 ms βœ… PASSED
Server Error Rate < 0.01% HTTP 5xx 0.00% (0 errors / 20k requests) βœ… PASSED
Deduplication Reliability 100% duplicate suppression 100.00% idempotent βœ… PASSED

πŸ’Ž Core Invariants & Design Principles

  1. Atomic Check-and-Decrement Invariant: Inventory checking and deduction are performed strictly within a single atomic Redis Lua script. Application-level GET followed by DECRBY is forbidden to prevent race conditions.
  2. Fail-Fast In-Memory Rejection: When inventory reaches 0, incoming booking requests are rejected in sub-millisecond RAM checks (HTTP 409 Conflict), never touching PostgreSQL connection pools or RabbitMQ channels.
  3. Strict Idempotency: Every reservation requires a unique client-side idempotency_key (persisted across user tab reloads in sessionStorage). Duplicate submissions return the cached reservation result without double-decrementing stock.
  4. Decoupled Database Persistence: The HTTP booking API does not write directly to PostgreSQL. Requests write to Redis and enqueue a message in RabbitMQ (202 Accepted), protecting relational databases from write saturation.
  5. Authoritative Expiration & Grace Windows: Payment deadlines are strictly enforced by database timestamps (expires_at <= NOW()), coupled with a 5-second grace window to absorb distributed network transit delays and NTP clock drift.

πŸ›‘ 12 Distributed Edge Cases & Battle-Tested Solutions

FlashPass is hardened against the full spectrum of distributed systems failures encountered in mission-critical payment and flash-sale infrastructures:

# Edge Case Scenario Real-World Failure Mode FlashPass Solution Automated Test Suite
1 Redis Master Failover Master crashes before async replication reaches replica; promoted replica double-sells stock. Dual-Node Sentinel failover, synchronous WAIT 1 verification, and PostgreSQL database constraint guard. TestEdgeCase1_RedisFailover
2 Timeout Worker vs Late Webhook Race Order expires at 10m00s releasing stock; payment arrives at 10m01s causing duplicate seat allocation. Optimistic state machine rejection, ticket retention for buyer B, and automatic background refund pipeline (order.refund). TestEdgeCase2_TimeoutRaceCondition
3 Dual-Write Partial Failure (Ghost Inventory) Redis stock decremented, but network drops before RabbitMQ publish; tickets vanish from inventory. Stock Auditor (ReconcileService) with 2-way clamping, Hit-Zero trigger, and orphaned user lock cleanup. TestEdgeCase3_DualWriteGhostInventory
4 Redis Cluster Cross-Slot Partitioning Keys mapped to different slots in a 16,384-slot Redis cluster cause Lua multi-key CROSSSLOT fatal errors. Hash Tag co-location (flashpass:{eventId}:stock, flashpass:{eventId}:user_booked:{userId}) mapped to identical CRC16 hash slots. TestEdgeCase4_CrossSlotHashTag
5 RabbitMQ Poison Pill & Queue Deadlock Corrupted payload or DB constraint violation causes infinite NACK retry loops, stalling worker. Dead Letter Exchange (flashpass.dlx), automatic quarantine to DLQ on 2nd retry, and Redis stock rollback. TestEdgeCase5_RabbitMQDeadLetterQueue
6 Cache Stampede & Thundering Herd Cache expires during peak surge; 10,000 requests hit PostgreSQL simultaneously, exhausting connections. singleflight.Group request coalescing collapsing 100 concurrent read queries into exactly 1 database query. TestEdgeCase6_CacheStampedeSingleFlight
7 Double Payment Webhook Concurrency Payment gateway retries webhooks concurrently; parallel threads process double credit / double refunds. Distributed lock (SETNX 5s), 100ms settle wait, winner-only lock release, and DB row-count idempotent fallback. TestEdgeCase7_DoublePaymentWebhookConcurrency
8 NTP Clock Drift / Skew Worker clock running ahead causes premature order cancellation before the 10-minute user window elapses. 5-second Clock Drift Grace Window and PostgreSQL authoritative server time evaluation (expires_at <= NOW()). TestEdgeCase8_ClockDriftGraceWindow
9 Bot Traffic Burst & Scalper Swarms High-frequency bot scripts overwhelm booking API, degrading service for legitimate users. Distributed Sliding-Window Log rate limiter in Redis Lua (ZSET), returning HTTP 429 with Retry-After: 2. TestEdgeCase9_SlidingWindowRateLimiter
10 Out-of-Order Message Inversion Network reordering causes Payment/Cancellation to arrive before asynchronous Order Creation. Early Payment retry debounce (100ms) + Cancellation Tombstone pattern preventing late order resurrection. TestEdgeCase10_OutOfOrderMessageProcessing
11 Database Cascading Failure Slow DB queries saturate connection pool (50 conns); threads lock up, bringing down the entire API. State-machine Circuit Breaker (Closed $\to$ Open $\to$ Half-Open) fast-failing DB calls in 0ms with HTTP 503. TestEdgeCase11_DatabaseCircuitBreaker
12 Split-Brain Distributed Lock Hijacking Long GC pause or slow network causes lock TTL to expire while task runs; another thread acquires lock. Heartbeat Watchdog goroutine renewing lease every $\text{TTL}/3$ with atomic Lua owner-token verification. TestEdgeCase12_DistributedLockWatchdog

🧰 Technology Stack

Backend Infrastructure

  • Programming Language: Go (Golang) 1.22+
  • HTTP Web Framework: Fiber v2 (FastHTTP engine, zero-allocation routing)
  • In-Memory Cache & Distributed Lock: Redis 7.x (Alpine) with Redis Sentinel (port 26379)
  • Message Broker: RabbitMQ 3.13+ with rabbitmq_delayed_message_exchange plugin
  • Database Engine: PostgreSQL 16+ using connection pooling (jackc/pgx/v5/pgxpool)
  • Resilience & Concurrency: Go singleflight, distributed locks, atomic Lua scripts, sync primitives

Frontend Client & Monitoring

  • Framework: Next.js 15 (React 19, App Router)
  • Styling: Tailwind CSS, Lucide Icons, Canvas Confetti
  • Real-Time Streaming: Server-Sent Events (SSE) & WebSocket fallbacks
  • Client Storage: sessionStorage for session idempotency, localStorage for load test benchmarks

πŸ”„ System Architecture & Data Flows

1. Flash Sale Pre-Warming

Before ticket sales commence, an administrator invokes the pre-warming endpoint:

  1. POST /api/v1/admin/events/:id/prewarm
  2. Backend queries PostgreSQL for total_tickets.
  3. Sets flashpass:{event_id}:stock in Redis memory.
  4. Initializes event metadata cache for zero-DB reads during sale surge.

2. Sub-Millisecond Booking & Ingestion Flow

sequenceDiagram
    autonumber
    actor User as Buyer
    participant API as Fiber Ingestion API
    participant Rate as Redis Rate Limiter
    participant Redis as Redis Master (Lua)
    participant RMQ as RabbitMQ (Order Exchange)
    
    User->>API: POST /api/v1/tickets/book (idempotency_key)
    API->>Rate: Check Sliding Window (ZSET)
    alt Rate Limit Exceeded
        Rate-->>API: Limit Exceeded (Burst > 5)
        API-->>User: HTTP 429 Too Many Requests (Retry-After: 2)
    end
    API->>Redis: EVALSHA atomic_decrement(event_id, user_id)
    alt User Already Booked
        Redis-->>API: Return -2 (Already Booked)
        API-->>User: HTTP 409 Conflict ("You have already reserved a ticket")
    else Out of Stock
        Redis-->>API: Return -1 (Sold Out)
        API-->>User: HTTP 409 Conflict ("Event sold out")
    else Stock Decremented
        Redis-->>API: Return 1 (Success)
        API->>RMQ: Publish order.create event (Thread-Safe Mutex)
        API-->>User: HTTP 202 Accepted { reservation_id, expires_in: 600 }
    end
Loading

3. Asynchronous Fulfillment & Delayed Expiration

  1. Fulfillment Consumer: Reads order.create queue, persists order into PostgreSQL with PENDING_PAYMENT, and publishes a delayed message with x-delay: 600000 (10 minutes).
  2. Timeout Consumer: Reads message after 10 minutes. Calls PostgreSQL authoritative expiration:
    UPDATE orders SET status = 'EXPIRED' 
    WHERE id = $1 AND status = 'PENDING_PAYMENT' AND expires_at <= NOW()
  3. If updated, the worker issues an atomic INCRBY flashpass:{event_id}:stock 1 and clears the user lock in Redis, releasing the ticket back to the public pool.

4. Auto-Refund Pipeline for Late Webhooks

If a payment arrives after expiration:

  1. POST /api/v1/orders/:id/pay acquires a distributed lock (WatchdogLock).
  2. Detects status = EXPIRED or past expiration deadline + grace window.
  3. Automatically writes a record to refund_orders table.
  4. Enqueues refund task to RabbitMQ flashpass.order.refund.queue.
  5. Background Refund Worker calls payment gateway refund API with exponential backoff and sets status to REFUNDED.

πŸ“‚ Clean Architecture & Project Structure

FlashPass adheres to Hexagonal / Clean Architecture principles, decoupling transport layers from domain services and database repositories:

FlashPass/
β”œβ”€β”€ backend/
β”‚   β”œβ”€β”€ cmd/
β”‚   β”‚   β”œβ”€β”€ api/                  # Main HTTP API Entry Point (Fiber)
β”‚   β”‚   └── worker/               # Background Worker (Consumers & Cron Reconciler)
β”‚   β”œβ”€β”€ internal/
β”‚   β”‚   β”œβ”€β”€ api/                  # HTTP Transport Controllers & Modular Integration Tests
β”‚   β”‚   β”‚   β”œβ”€β”€ handlers.go       # Thin Fiber Controllers
β”‚   β”‚   β”‚   β”œβ”€β”€ router.go         # API Route Registrations
β”‚   β”‚   β”‚   β”œβ”€β”€ test_helpers_test.go          # Shared Test Fixtures & HTTP Setup
β”‚   β”‚   β”‚   β”œβ”€β”€ booking_flow_test.go          # Core Booking Flow Integration Tests
β”‚   β”‚   β”‚   β”œβ”€β”€ payment_flow_test.go          # Payment Lifecycle & Auto-Refund Tests
β”‚   β”‚   β”‚   └── edge_case_1..12_*_test.go     # 12 Modular Distributed Chaos & Edge Case Tests
β”‚   β”‚   β”œβ”€β”€ config/               # Viper / Environment Configuration
β”‚   β”‚   β”œβ”€β”€ consumer/             # RabbitMQ Consumers (Order, Timeout, Refund)
β”‚   β”‚   β”œβ”€β”€ database/             # PostgreSQL Connection Pool & Embedded Migrations
β”‚   β”‚   β”œβ”€β”€ models/               # Domain Entities & Status Enums
β”‚   β”‚   β”œβ”€β”€ rabbitmq/             # AMQP Connection, Thread-Safe Producer & DLX
β”‚   β”‚   β”œβ”€β”€ redis/                # Redis Pool, Sentinel, Lua Scripts, Watchdog Lock
β”‚   β”‚   β”œβ”€β”€ repository/           # Persistence Layer (Event, Order, Refund)
β”‚   β”‚   β”œβ”€β”€ resilience/           # Circuit Breaker & Concurrency Utilities
β”‚   β”‚   └── service/              # Core Business Logic (Booking, Order, Reconcile)
β”œβ”€β”€ frontend/                     # Next.js 15 Client & Interactive Simulation Lab
β”‚   β”œβ”€β”€ app/                      # Next.js App Router Pages
β”‚   └── components/
β”‚       β”œβ”€β”€ edge-cases/           # Interactive Distributed Chaos Lab Components
β”‚       └── ...
β”œβ”€β”€ docs/                         # Technical Specs, Diagrams & Production Runbooks
β”‚   β”œβ”€β”€ API_REFERENCE.md          # Comprehensive REST API Specification
β”‚   β”œβ”€β”€ RUNBOOK.md                # Day-2 Operations & Incident Playbook
β”‚   β”œβ”€β”€ DISTRIBUTED_EDGE_CASES.md # Deep-Dive Engineering Analysis (12 Scenarios)
β”‚   └── benchmark_chart.png       # k6 Visual Benchmark Results
β”œβ”€β”€ scripts/                      # Load Testing & Automation Scripts
β”‚   β”œβ”€β”€ load-test.js              # k6 20,000-request Surge Test
β”‚   └── visualize_benchmark.py    # Matplotlib Graph Generator
β”œβ”€β”€ docker-compose.yml            # Multi-Node Cluster Orchestration
β”œβ”€β”€ SPEC.md                       # Formal System Specification
β”œβ”€β”€ ARCHITECTURE.md               # Technical Design Blueprint
β”œβ”€β”€ TASKS.md                      # Implementation Roadmap & Verification Checklist
└── README.md                     # Master Documentation

πŸ“‘ API Reference Overview

Detailed request/response contracts, headers, and schemas are available in docs/API_REFERENCE.md.

Method Endpoint Description Key Headers / Body
GET /health System liveness probe Returns 200 OK
GET /api/v1/events/:id Event metadata (SingleFlight protected) Cached response
GET /api/v1/events/:id/stock Real-time ticket inventory in RAM Returns available tickets
GET /api/v1/events/:id/stream Server-Sent Events (SSE) live updates Stream of inventory changes
POST /api/v1/tickets/book Flash Sale Ticket Booking Requires idempotency_key
GET /api/v1/orders/:id Query order details & payment deadline Returns order status
POST /api/v1/orders/:id/pay Mock Payment Webhook / Confirmation Idempotent transition to PAID
POST /api/v1/orders/:id/cancel Explicit order cancellation Sets CANCELLED tombstone
POST /api/v1/admin/events/:id/prewarm Pre-load DB inventory into Redis RAM Admin token
POST /api/v1/admin/events/:id/reconcile Stock Auditor Discrepancy Self-Healing Resolves ghost inventory
POST /api/v1/admin/events/:id/reset Reset event stock & clean cache Custom tickets parameter

πŸš€ Production Quickstart & Deployment

1. Prerequisites

  • Docker Engine 24+ & Docker Compose v2+
  • Go 1.22+ (for local development & running test suite)
  • Node.js 18+ & npm/pnpm (for frontend)

2. Launch the Infrastructure Stack

Clone the repository and start the cluster:

git clone https://github.com/your-org/FlashPass.git
cd FlashPass

# Start PostgreSQL 16, Redis Master, Redis Replica, Redis Sentinel, and RabbitMQ
docker compose up -d

Verify that all 5 cluster services are healthy:

docker compose ps
Service Container Name Ports Role
PostgreSQL flashpass_postgres 5432:5432 Relational Storage & Write Model
Redis Master flashpass_redis 6379:6379 In-Memory Atomic Engine
Redis Replica flashpass_redis_replica 6380:6379 Replication & Failover Standby
Redis Sentinel flashpass_redis_sentinel 26379:26379 Cluster Health & Auto-Promotion
RabbitMQ flashpass_rabbitmq 5672:5672, 15672:15672 Message Broker & Management UI

3. Start the Backend API & Background Worker

cd backend

# Copy configuration
cp .env.example .env

# Run HTTP API Server
go run cmd/api/main.go

# In a separate terminal, run the Background Worker
go run cmd/worker/main.go

4. Start the Frontend Application

cd frontend
npm install
npm run dev
# Open http://localhost:3000 in your browser

πŸ§ͺ Automated Edge Cases Test Suite

FlashPass comes with an enterprise test suite verifying all 12 distributed edge cases against real Docker containers:

cd backend

# Execute all 12 Distributed Edge Cases tests
go test -v -run "TestEdgeCase" ./internal/api

# Run the complete test suite across all packages
go test -v ./...

Expected output:

=== RUN   TestEdgeCase1_RedisFailover
--- PASS: TestEdgeCase1_RedisFailover (1.05s)
=== RUN   TestEdgeCase2_TimeoutRaceCondition
--- PASS: TestEdgeCase2_TimeoutRaceCondition (1.10s)
=== RUN   TestEdgeCase3_DualWriteGhostInventory
--- PASS: TestEdgeCase3_DualWriteGhostInventory (1.02s)
=== RUN   TestEdgeCase4_CrossSlotHashTag
--- PASS: TestEdgeCase4_CrossSlotHashTag (1.01s)
=== RUN   TestEdgeCase5_RabbitMQDeadLetterQueue
--- PASS: TestEdgeCase5_RabbitMQDeadLetterQueue (1.05s)
=== RUN   TestEdgeCase6_CacheStampedeSingleFlight
--- PASS: TestEdgeCase6_CacheStampedeSingleFlight (1.02s)
=== RUN   TestEdgeCase7_DoublePaymentWebhookConcurrency
--- PASS: TestEdgeCase7_DoublePaymentWebhookConcurrency (1.08s)
=== RUN   TestEdgeCase8_ClockDriftGraceWindow
--- PASS: TestEdgeCase8_ClockDriftGraceWindow (1.04s)
=== RUN   TestEdgeCase9_SlidingWindowRateLimiter
--- PASS: TestEdgeCase9_SlidingWindowRateLimiter (1.02s)
=== RUN   TestEdgeCase10_OutOfOrderMessageProcessing
--- PASS: TestEdgeCase10_OutOfOrderMessageProcessing (1.15s)
=== RUN   TestEdgeCase11_DatabaseCircuitBreaker
--- PASS: TestEdgeCase11_DatabaseCircuitBreaker (1.52s)
=== RUN   TestEdgeCase12_DistributedLockWatchdog
--- PASS: TestEdgeCase12_DistributedLockWatchdog (1.51s)
PASS
ok      flashpass/internal/api  13.2s

πŸ“Š Load Testing & Benchmark Proof

A stress test simulating 20,000 concurrent requests competing for 1,000 tickets was conducted using Grafana k6:

k6 run scripts/load-test.js

Benchmark Metrics Dashboard

Benchmark Dashboard

Summary of Results

  • Total Requests Processed: 20,000 requests in 2.41s
  • Throughput: 8,285 Requests/sec (RPS)
  • Accepted Reservations (HTTP 202): Exactly 1,000 (100% of inventory)
  • Fail-Fast Sold Out (HTTP 409): Exactly 19,000
  • Database Orders Created: Exactly 1,000
  • Overselling Invariant: 0.00% (Zero Overselling)
  • p95 Latency: 18.83 ms (Target SLA: < 25 ms)
  • p99 Latency: 29.42 ms (Target SLA: < 50 ms)

For an in-depth breakdown of metrics, charts, and breaking limits, consult BENCHMARK.md.


πŸ“– Operational Runbook & Disaster Recovery

For production Day-2 operations, cluster maintenance, and emergency response:

  • docs/RUNBOOK.md: Alert thresholds, manual reconciliation steps, dead-letter queue reprocessing, Redis Sentinel switchover commands, and circuit breaker tuning.
  • docs/DISTRIBUTED_EDGE_CASES.md: Deep-dive mathematical proofs, Lua scripts, and architecture mechanics of all 12 edge cases.
  • docs/API_REFERENCE.md: Complete HTTP REST schemas and integration guides.

πŸ“„ License

This project is licensed under the MIT License. Built for high-volume enterprise distributed systems engineering.

About

FlashPass: High-Concurrency Distributed Flash Sale & Ticketing Engine (Go Fiber, Redis Sentinel, Lua, RabbitMQ, PostgreSQL 16, Next.js 15)

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages