HFT Pipeline Architecture & FlatBuffers Migration: Sub-100ms Trading with Zero-Copy Serialization

Single monolithic scanner:
- Mixed concerns (detection + validation + execution)
- JSON serialization overhead (36ms per round-trip)
- No separation between cheap filters and expensive RPC calls
- Kill switch required manual intervention (minutes to stop)
- Single NATS stream with mixed retention needs

Result: 150-200ms+ execution time, unreliable under load

Target: <100ms end-to-end execution
- Fast pre-filtering (<10ms) - detect opportunities cheaply
- Deep validation (50-100ms) - expensive RPC simulation only when needed
- Dumb execution (<20ms) - just sign and send
- Zero-copy serialization - eliminate JSON overhead
- Instant kill switch - <100ms system-wide halt
- Independent scaling - scale each stage separately

┌────────────────────────────────────────────────────────────┐
│ QUOTE-SERVICE (Go)                                          │
│ Purpose: Generate and cache market data                     │
│ Latency: <10ms per quote                                    │
│ Throughput: 10,000 quotes/sec                               │
│ Cost: Low (cached pool data)                                │
│                                                              │
│ Output: gRPC streaming + NATS MARKET_DATA stream            │
└────────────────────────────────────────────────────────────┘
                            ↓
┌────────────────────────────────────────────────────────────┐
│ SCANNER (Fast Pre-Filter)                                   │
│ Purpose: Detect potential arbitrage with simple math        │
│ Latency: <10ms                                              │
│ Throughput: 10k quotes → 500 opportunities (95% filtered)   │
│ Cost: Very low (cached data only, no RPC)                   │
│ False Positives: 35% acceptable                             │
│                                                              │
│ Output: Multiple event types → OPPORTUNITIES stream         │
│ - TwoHopArbitrageEvent (SOL→mSOL→SOL)                       │
│ - TriangularArbitrageEvent (SOL→USDC→mSOL→SOL)              │
│ - LiquidationEvent (under-collateralized positions)         │
│ - PriceDivergenceEvent (oracle vs DEX price gaps)           │
└────────────────────────────────────────────────────────────┘
                            ↓
┌────────────────────────────────────────────────────────────┐
│ PLANNER (Deep Validation) - Multiple Specialized Planners   │
│ Purpose: Validate profitability with RPC simulation         │
│ Latency: 50-100ms                                           │
│ Throughput: 500 opportunities → 50 plans (90% rejected)     │
│ Cost: High (RPC simulation required)                        │
│ Accuracy: 90%+ (after simulation)                           │
│                                                              │
│ Planners subscribe to specific opportunity types:           │
│ - TwoHopArbitragePlanner → opportunity.two_hop.*            │
│ - TriangularArbitragePlanner → opportunity.triangular.*     │
│ - LiquidationPlanner → opportunity.liquidation.*            │
│                                                              │
│ Output: ExecutionPlanEvent → EXECUTION stream               │
└────────────────────────────────────────────────────────────┘
                            ↓
┌────────────────────────────────────────────────────────────┐
│ EXECUTOR (Dumb Submission)                                  │
│ Purpose: Sign and submit transactions                       │
│ Latency: <20ms                                              │
│ Throughput: 50 executions/sec                               │
│ Cost: Very low (no validation, just submission)             │
│                                                              │
│ Output: ExecutionResultEvent → EXECUTED stream              │
└────────────────────────────────────────────────────────────┘

Input: 10,000 quotes/sec from quote-service
Processing: Simple arithmetic on cached data
- Check for A→B→A round-trip profitability
- Calculate rough profit estimate (no RPC)
- Confidence: 0.5-0.8 (rough guess is fine)

Output: 500 opportunities/sec
Filtering efficiency: 95% (10k → 500)
Cost: <10ms CPU time only

Input: 500 opportunities/sec from scanner
Processing: RPC simulation with current on-chain state
- Fetch fresh quotes from MARKET_DATA (if stale)
- Simulate transaction with actual pool reserves
- Calculate real slippage and fees
- Choose execution method (Jito/TPU/RPC)
- Confidence: 0.9+ (high accuracy required)

Output: 50 execution plans/sec
Filtering efficiency: 90% (500 → 50)
Cost: 50-100ms per validation (RPC network calls)

Input: 50 execution plans/sec from planner
Processing: Transaction building and submission
- Build transaction from pre-computed path (no re-validation)
- Sign with wallet
- Submit via Jito/TPU/RPC (as specified)
- NO simulation, NO checks (Planner already validated)

Output: 50 execution results/sec
Success rate: 80%+ (high because Planner validated)
Cost: <20ms (just signing + network)

┌────────────────────────────────────────────────────────────┐
│ Stream 1: MARKET_DATA                                       │
│ Purpose: High-volume raw market data                        │
│ Throughput: 10,000 events/sec                               │
│ Retention: 1 hour (noisy, delete fast)                      │
│ Storage: Memory (sub-1ms latency)                           │
│ Size: 1 GB buffer (100k messages)                           │
└────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────┐
│ Stream 2: OPPORTUNITIES                                     │
│ Purpose: Pre-filtered arbitrage opportunities               │
│ Throughput: 500 events/sec                                  │
│ Retention: 24 hours (debugging detection logic)             │
│ Storage: File (2-5ms latency, persistent)                   │
│ Size: 100 MB (10k messages)                                 │
└────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────┐
│ Stream 3: EXECUTION                                         │
│ Purpose: Validated execution plans                          │
│ Throughput: 50 events/sec                                   │
│ Retention: 1 hour (short-lived, execute once)               │
│ Storage: File (2-5ms latency)                               │
│ Size: 10 MB (1k messages)                                   │
└────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────┐
│ Stream 4: EXECUTED                                          │
│ Purpose: Execution results for PnL analysis                 │
│ Throughput: 50 events/sec                                   │
│ Retention: 7 days (audit trail, reporting)                  │
│ Storage: File (persistence critical)                        │
│ Size: 1 GB                                                  │
└────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────┐
│ Stream 5: METRICS                                           │
│ Purpose: Performance metrics (latency, throughput, PnL)     │
│ Throughput: 1,000-5,000 events/sec                          │
│ Retention: 1 hour (aggregated to Prometheus)                │
│ Storage: Memory (fast reads)                                │
│ Size: 100 MB (50k messages)                                 │
└────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────┐
│ Stream 6: SYSTEM                                            │
│ Purpose: Lifecycle & safety events (kill switch, shutdown)  │
│ Throughput: 1-10 events/sec                                 │
│ Retention: 30 days (compliance, audit)                      │
│ Storage: File (critical persistence)                        │
│ Size: 50 MB (10k messages)                                  │
└────────────────────────────────────────────────────────────┘

MARKET_DATA:    10,000 events/sec (raw market data)
OPPORTUNITIES:     500 events/sec (98% filtered by scanner)
EXECUTION:          50 events/sec (90% rejected by planner)
EXECUTED:           50 events/sec (results from executor)
METRICS:         1,000-5,000 events/sec (high-frequency performance data)
SYSTEM:            1-10 events/sec (lifecycle events)

Total filtering: 99.5% (10k → 50)

MARKET_DATA:     1 hour   (noisy raw data, no long-term value)
OPPORTUNITIES:  24 hours  (debug detection logic)
EXECUTION:       1 hour   (ephemeral plans, execute once)
EXECUTED:        7 days   (PnL reporting, compliance)
METRICS:         1 hour   (scraped by Prometheus)
SYSTEM:         30 days   (regulatory audit trail)

Disk savings: Massive
- Don't waste space storing 10k market events/sec for days
- Keep execution results for a week for PnL analysis
- System events retained longest for compliance

Scanner crashes:
├─ MARKET_DATA: No new data (expected)
├─ OPPORTUNITIES: No new opportunities (expected)
├─ EXECUTION: Planner processes buffered opportunities
└─ EXECUTED: Executor keeps processing buffered plans

Planner crashes:
├─ MARKET_DATA: Scanner unaffected
├─ OPPORTUNITIES: Buffer fills with new opportunities
├─ EXECUTION: No new plans (expected)
└─ EXECUTED: Executor processes buffered plans

Executor crashes:
├─ MARKET_DATA: Scanner unaffected
├─ OPPORTUNITIES: Planner unaffected
├─ EXECUTION: Buffer fills with validated plans (waits for executor restart)
└─ EXECUTED: No new results (expected)

Subjects:
- system.killswitch.excessive_loss      (stop trading immediately)
- system.killswitch.connection_failure  (RPC/websocket down)
- system.killswitch.manual              (operator triggered)
- system.lifecycle.shutdown.graceful    (clean shutdown)
- system.health.slot_drift.critical     (blockchain lag)

All frameworks MUST subscribe:
Scanner  → Stops scanning + publishes final metrics
Planner  → Stops planning + rejects pending opportunities
Executor → Cancels pending txs + finishes in-flight

Response time: <100ms system-wide halt

┌────────────────────────────────────────────────────────────┐
│ ArbitrageQuoteScanner                                       │
│ Detects: Two-hop arbitrage (SOL→mSOL→SOL)                  │
│ Publishes to: opportunity.two_hop.detected.*                │
│ Event Type: TwoHopArbitrageEvent                            │
│ Volume: ~300 events/sec                                     │
└────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────┐
│ TriangularArbitrageScanner                                  │
│ Detects: Three-hop arbitrage (SOL→USDC→mSOL→SOL)           │
│ Publishes to: opportunity.triangular.detected.*             │
│ Event Type: TriangularArbitrageEvent                        │
│ Volume: ~100 events/sec                                     │
└────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────┐
│ LiquidationScanner                                          │
│ Detects: Under-collateralized lending positions             │
│ Publishes to: opportunity.liquidation.detected.*            │
│ Event Type: LiquidationEvent                                │
│ Volume: ~50 events/sec                                      │
└────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────┐
│ OracleDivergenceScanner                                     │
│ Detects: Oracle vs DEX price divergence                     │
│ Publishes to: opportunity.oracle_divergence.detected.*      │
│ Event Type: PriceDivergenceEvent                            │
│ Volume: ~20 events/sec                                      │
└────────────────────────────────────────────────────────────┘

All publish to: OPPORTUNITIES stream
Total volume: ~470 events/sec (well within 500 target)
Stream capacity: Handles all scanner types simultaneously

┌────────────────────────────────────────────────────────────┐
│ TwoHopArbitragePlanner                                      │
│ Subscribes to: opportunity.two_hop.>                        │
│ Consumes: TwoHopArbitrageEvent                              │
│ Validation: RPC simulation for 2-hop swaps                  │
│ Output: ExecutionPlanEvent (Jito/TPU routing)               │
└────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────┐
│ TriangularArbitragePlanner                                  │
│ Subscribes to: opportunity.triangular.>                     │
│ Consumes: TriangularArbitrageEvent                          │
│ Validation: RPC simulation for 3-hop swaps                  │
│ Output: ExecutionPlanEvent (optimized routing)              │
└────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────┐
│ LiquidationPlanner                                          │
│ Subscribes to: opportunity.liquidation.>                    │
│ Consumes: LiquidationEvent                                  │
│ Validation: Check health factor, liquidation bonus          │
│ Output: ExecutionPlanEvent (flash loan execution)           │
└────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────┐
│ OracleDivergencePlanner                                     │
│ Subscribes to: opportunity.oracle_divergence.>              │
│ Consumes: PriceDivergenceEvent                              │
│ Validation: Verify oracle freshness, calculate profit       │
│ Output: ExecutionPlanEvent (oracle update + arbitrage)      │
└────────────────────────────────────────────────────────────┘

All planners run concurrently, processing only relevant events
Each planner is independent (can scale/restart independently)
EXECUTION stream receives mixed ExecutionPlanEvent types

OPPORTUNITIES stream subjects:

opportunity.two_hop.detected.<tokenA>.<tokenB>
├─ opportunity.two_hop.detected.SOL.mSOL
├─ opportunity.two_hop.detected.SOL.USDC
└─ opportunity.two_hop.detected.USDC.USDT

opportunity.triangular.detected.<tokenA>.<tokenB>.<tokenC>
├─ opportunity.triangular.detected.SOL.USDC.mSOL
└─ opportunity.triangular.detected.SOL.USDT.USDC

opportunity.liquidation.detected.<protocol>.<severity>
├─ opportunity.liquidation.detected.marinade.critical
├─ opportunity.liquidation.detected.solend.high
└─ opportunity.liquidation.detected.mango.medium

opportunity.oracle_divergence.detected.<oracle>.<dex>
├─ opportunity.oracle_divergence.detected.pyth.raydium
└─ opportunity.oracle_divergence.detected.switchboard.orca

Planner subscriptions:
- TwoHopPlanner:          opportunity.two_hop.>
- TriangularPlanner:      opportunity.triangular.>
- LiquidationPlanner:     opportunity.liquidation.>
- OracleDivergencePlanner: opportunity.oracle_divergence.>

Benefits:
✅ Each planner only receives relevant events
✅ Add new scanner types without modifying existing planners
✅ Add new planners without modifying scanners
✅ Fine-grained filtering (e.g., only SOL pairs, only critical liquidations)

Step 1: Create new scanner
└─ Implement scanner that detects new opportunity type
   └─ Publishes to: opportunity.<new_type>.detected.*

Step 2: Define FlatBuffers schema
└─ Create NewOpportunityEvent schema
   └─ Generate code with flatc

Step 3: Create specialized planner
└─ Implement planner for validation logic
   └─ Subscribes to: opportunity.<new_type>.>
   └─ Publishes to: EXECUTION stream

Step 4: Deploy
└─ Scanner, Planner run independently
   └─ No changes to existing scanners/planners
   └─ OPPORTUNITIES stream handles new event type automatically

Example new strategies:
- DCA (Dollar-Cost Averaging) scanner
- Limit order execution scanner
- MEV opportunity scanner
- Cross-DEX arbitrage scanner
- Perp-spot arbitrage scanner

Before (tightly coupled):
├─ Scanner hard-coded to call specific planner
├─ Adding new planner requires scanner changes
└─ Can't run multiple planners for same opportunity

After (NATS subject routing):
├─ Scanner publishes event to subject
├─ Any planner can subscribe to subject
├─ Multiple planners can process same opportunity
└─ No coupling: scanners don't know about planners

Scenario: High two-hop arbitrage volume

Solution:
├─ Scale TwoHopArbitragePlanner horizontally (3 instances)
│  └─ All subscribe to opportunity.two_hop.>
│  └─ NATS load-balances across instances
│
└─ Other planners unaffected
   └─ TriangularPlanner, LiquidationPlanner continue normally
   └─ No coordination needed

Testing new planner logic:
├─ Deploy experimental planner alongside production planner
├─ Both subscribe to same opportunity type
├─ Experimental planner publishes to execution.test.*
├─ Production planner publishes to execution.planned.*
└─ Compare results without affecting live trading

A/B testing:
├─ PlannerId field in ExecutionPlanEvent
├─ Track which planner generated better PnL
└─ Gradually shift traffic to better-performing planner

OPPORTUNITIES stream activity (real-time):

t=0ms:    ArbitrageScanner publishes TwoHopArbitrageEvent
          Subject: opportunity.two_hop.detected.SOL.mSOL
          → TwoHopArbitragePlanner receives (0.5ms)

t=5ms:    LiquidationScanner publishes LiquidationEvent
          Subject: opportunity.liquidation.detected.solend.critical
          → LiquidationPlanner receives (0.5ms)

t=12ms:   TriangularScanner publishes TriangularArbitrageEvent
          Subject: opportunity.triangular.detected.SOL.USDC.mSOL
          → TriangularPlanner receives (0.5ms)

t=18ms:   OracleDivergenceScanner publishes PriceDivergenceEvent
          Subject: opportunity.oracle_divergence.detected.pyth.raydium
          → OracleDivergencePlanner receives (0.5ms)

t=20ms:   ArbitrageScanner publishes TwoHopArbitrageEvent
          Subject: opportunity.two_hop.detected.USDC.USDT
          → TwoHopArbitragePlanner receives (0.5ms)

Result: 5 different opportunity types processed concurrently
Each planner only sees relevant events
No cross-talk, no interference

✅ Multiple scanner types publish to OPPORTUNITIES stream
✅ Each scanner publishes different event types (FlatBuffers)
✅ Subject patterns enable fine-grained routing
✅ Specialized planners subscribe only to relevant opportunities
✅ Zero coupling: scanners don't know about planners
✅ Independent scaling per opportunity type
✅ Easy to add new strategies without modifying existing code
✅ A/B testing and experimentation without risk

Architectural principle:
"Publish what you found, subscribe to what you handle"

JSON Problems:
1. Parsing overhead: 8-15μs per decode
2. Encoding overhead: 5-10μs per encode
3. Memory allocations: Heavy GC pressure
4. No zero-copy: Every read copies data
5. Large message size: 450-600 bytes

Total cost per event: ~30-50μs (encode + decode + GC)
Over 4 pipeline stages: 120-200μs overhead
Percentage of 200ms budget: 0.1% (small but additive)

FlatBuffers Advantages:
1. Decode time: 0.1-0.5μs (20-150x faster than JSON)
2. Encode time: 1-2μs (5-10x faster than JSON)
3. Memory: Minimal allocations (read directly from buffer)
4. Zero-copy: No data copying on read
5. Message size: 300-400 bytes (30% smaller)

Total cost per event: ~2-5μs
Over 4 pipeline stages: 8-20μs overhead
Improvement: 6-10x faster serialization

┌────────────────────────────────────────────────────────────┐
│ Operation           │ JSON    │ FlatBuffers │ Improvement  │
├────────────────────────────────────────────────────────────┤
│ Scanner publish     │ 10ms    │ 2ms         │ -8ms         │
│ Planner consume     │ 8ms     │ 0.5ms       │ -7.5ms       │
│ Planner publish     │ 10ms    │ 2ms         │ -8ms         │
│ Executor consume    │ 8ms     │ 0.5ms       │ -7.5ms       │
├────────────────────────────────────────────────────────────┤
│ Total saved         │ 36ms    │ 5ms         │ -31ms        │
└────────────────────────────────────────────────────────────┘

End-to-end improvement:
Before: 147ms (JSON baseline)
After:  116ms (FlatBuffers)
Reduction: 31ms (21% faster)

With NATS optimization: 95ms target

TwoHopArbitrageEvent (FlatBuffers):
├─ Identification
│  ├─ traceId (UUID for distributed tracing)
│  ├─ opportunityId (unique identifier)
│  └─ timestamp, slot
│
├─ Tokens
│  ├─ tokenStart (e.g., SOL)
│  └─ tokenIntermediate (e.g., mSOL, USDC)
│
├─ Execution Path (SwapHop[])
│  ├─ Hop 1: SOL → mSOL
│  │  ├─ ammKey (pool address)
│  │  ├─ label (DEX name: "raydium_amm")
│  │  ├─ inputMint, outputMint
│  │  ├─ inAmount, outAmount (as strings for u64)
│  │  └─ feeAmount, feeMint
│  │
│  └─ Hop 2: mSOL → SOL
│     ├─ ammKey, label ("orca_whirlpool")
│     ├─ inputMint, outputMint
│     └─ inAmount, outAmount, fees
│
├─ Profit Analysis
│  ├─ amountIn (starting capital)
│  ├─ expectedAmountOut (after round-trip)
│  ├─ estimatedProfitUsd, estimatedProfitBps
│  ├─ netProfitUsd (after fees)
│  └─ totalFees
│
├─ Risk Metrics
│  ├─ confidence (0.5-0.8 from scanner, 0.9+ from planner)
│  ├─ priceImpactBps (total slippage)
│  ├─ requiredCapitalUsd (for sizing)
│  └─ hop1LiquidityUsd, hop2LiquidityUsd
│
└─ Timing
   ├─ expiresAt (estimated opportunity expiration)
   └─ detectedAtSlot (blockchain slot)

Traditional JSON (copies memory):
1. Parse JSON string → JavaScript object (allocation)
2. Access fields → copies data
3. Process event
4. GC cleanup (pause)

FlatBuffers (zero-copy):
1. Wrap buffer → ByteBuffer (no allocation)
2. Access fields → reads directly from buffer (no copy)
3. Process event
4. No GC (buffer reused)

Result: Faster + less memory + no GC pauses

Manual kill switch (old way):
1. Operator notices excessive losses (30-60 seconds)
2. SSH into each service (10-20 seconds)
3. Manually stop scanner, planner, executor (10-30 seconds)
4. Hope no in-flight trades execute

Total time: 50-110 seconds
Risk: Pending trades may still execute
Result: Potentially thousands in additional losses

Automatic kill switch (new way):
1. Monitor detects excessive loss condition (<1 second)
2. Publishes system.killswitch.excessive_loss to SYSTEM stream
3. All frameworks receive event simultaneously (<100ms)
4. Scanner stops detecting
5. Planner rejects all pending opportunities
6. Executor cancels pending transactions
7. System fully halted

Total time: <100ms
Risk: Zero (all frameworks halt synchronously)
Result: Minimal additional losses

┌────────────────────────────────────────────────────────────┐
│ Scanner Framework                                           │
│                                                              │
│ Subscribes to: system.killswitch.*, system.lifecycle.*     │
│                                                              │
│ On KillSwitch:                                              │
│ 1. Stop scanning immediately                                │
│ 2. Publish final metrics                                    │
│ 3. Log kill switch trigger                                  │
│                                                              │
│ On Shutdown:                                                │
│ 1. Finish current detection cycle (<10ms)                   │
│ 2. Stop gracefully                                          │
│ 3. Cleanup resources                                        │
└────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────┐
│ Planner Framework                                           │
│                                                              │
│ Subscribes to: system.killswitch.*, system.lifecycle.*     │
│                                                              │
│ On KillSwitch:                                              │
│ 1. Stop planning immediately                                │
│ 2. Reject all buffered opportunities (publish rejections)   │
│ 3. Publish metrics                                          │
│                                                              │
│ On Shutdown:                                                │
│ 1. Finish current validation cycle (50-100ms)               │
│ 2. Publish final execution plan or rejection                │
│ 3. Cleanup                                                  │
└────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────┐
│ Executor Framework                                          │
│                                                              │
│ Subscribes to: system.killswitch.*, system.lifecycle.*     │
│                                                              │
│ On KillSwitch:                                              │
│ 1. Cancel all pending transactions immediately              │
│ 2. Stop executor                                            │
│ 3. Publish failure events for cancelled txs                 │
│                                                              │
│ On Shutdown:                                                │
│ 1. Wait for in-flight transactions (max 30s timeout)        │
│ 2. Finish pending txs                                       │
│ 3. Publish final results                                    │
└────────────────────────────────────────────────────────────┘

t=0ms:    Monitor detects excessive_loss condition
          └─ Publishes: system.killswitch.excessive_loss

t=5ms:    Scanner receives KillSwitchEvent
          ├─ Stops scanning
          └─ No new opportunities published

t=10ms:   Planner receives KillSwitchEvent
          ├─ Stops planning
          ├─ Rejects all buffered opportunities
          └─ Publishes: ExecutionRejectedEvent(reason: killswitch)

t=15ms:   Executor receives KillSwitchEvent
          ├─ Cancels pending transactions
          └─ Publishes: ExecutionFailedEvent(error: killswitch)

t=100ms:  System fully halted
          └─ All frameworks idle, no new trading activity

Result: Complete system stop in <100ms

t=0s:     Operator triggers graceful shutdown
          └─ Publishes: system.lifecycle.shutdown.manual

t=0.1s:   Scanner receives ShutdownEvent
          ├─ Finishes current detection cycle
          ├─ Stops scanning
          └─ Publishes final metrics

t=0.2s:   Planner receives ShutdownEvent
          ├─ Finishes validating current opportunity
          ├─ Publishes final plan or rejection
          └─ Stops planning

t=30s:    Executor receives ShutdownEvent
          ├─ Waits for in-flight transactions (30s timeout)
          ├─ Publishes final execution results
          └─ Stops executor

t=30-60s: System fully shut down
          └─ All resources cleaned up, state persisted

Result: Clean shutdown with no lost trades

┌────────────────────────────────────────────────────────────┐
│ Stage              │ Target   │ Critical Path │ Notes       │
├────────────────────────────────────────────────────────────┤
│ Scanner            │ <10ms    │ ✅ Yes        │ Pre-filter  │
│ NATS (OPPORT.)     │ 2-5ms    │ ✅ Yes        │ File store  │
│ Planner            │ 50-100ms │ ✅ Yes        │ RPC simul.  │
│ NATS (EXECUTION)   │ 2-5ms    │ ✅ Yes        │ File store  │
│ Executor           │ <20ms    │ ✅ Yes        │ Sign + send │
│ NATS (EXECUTED)    │ 5-10ms   │ ❌ No         │ After trade │
├────────────────────────────────────────────────────────────┤
│ TOTAL (critical)   │ <200ms   │               │ Target met  │
│ TOTAL (optimized)  │ ~95ms    │               │ FlatBuffers │
└────────────────────────────────────────────────────────────┘

Breakdown:
Scanner:    8ms   (quote → TwoHopArbitrageEvent)
NATS:       2ms   (OPPORTUNITIES stream)
Planner:   85ms   (validation + RPC simulation)
NATS:       2ms   (EXECUTION stream)
Executor:  18ms   (sign + submit)
────────────────
TOTAL:     115ms  (JSON baseline)

With FlatBuffers: -20ms (serialization savings)
Final target:      95ms  (sub-100ms achieved ✅)

Scanner Performance:
├─ Latency: <10ms per detection
├─ Throughput: 10,000 quotes → 500 opportunities/sec
├─ False positive rate: <35%
└─ Memory: Minimal (cached quotes only)

Planner Performance:
├─ Latency: 50-100ms per validation
├─ Throughput: 500 opportunities → 50 plans/sec
├─ Rejection rate: 90% (efficient filtering)
├─ Accuracy: >90% (simulation matches on-chain)
└─ Fresh quote access: MARKET_DATA stream

Executor Performance:
├─ Latency: <20ms per execution
├─ Throughput: 50 executions/sec
├─ Success rate: >80% (high validation quality)
└─ Jito bundle landing: >95%

End-to-End Performance:
├─ Total latency: 95ms (quote → execution)
├─ Profitable trades: >60% (after all filtering)
├─ Average profit: >0.05 SOL per trade
├─ Kill switch response: <100ms
└─ Graceful shutdown: 30-60s (clean state)

Add new trading strategy:
└─ Create new scanner (publishes to opportunity.<type>.*)
└─ Create new planner (subscribes to opportunity.<type>.>)
└─ Deploy independently
   └─ Zero changes to existing code
   └─ No coordination with other scanners/planners
   └─ Instant multi-strategy capability

Example strategies:
├─ Two-hop arbitrage (implemented)
├─ Triangular arbitrage (ready to add)
├─ Liquidation monitoring (ready to add)
├─ Oracle divergence (ready to add)
├─ DCA execution (ready to add)
├─ Limit order matching (ready to add)
└─ MEV opportunities (ready to add)

Result: Platform for unlimited trading strategies

High market activity:
└─ Scale up ArbitrageScanner (horizontal: 3-5 instances)
   └─ OPPORTUNITIES stream buffers for Planner

High two-hop volume:
└─ Scale up TwoHopArbitragePlanner (horizontal: 3 instances)
   └─ NATS load-balances across instances
   └─ Other planners (Liquidation, Triangular) unaffected

High execution volume:
└─ Scale up Executor (horizontal: 2-3 instances)
   └─ Each handles subset of execution plans

Per-strategy scaling:
└─ Scale only the bottlenecked strategy
   └─ Other strategies continue normally
   └─ No global coordination needed

Scanner:  "I think this might be profitable" (confidence: 0.5-0.8)
Planner:  "I verified this is profitable" (confidence: 0.9+)
Executor: "I trust the Planner completely"

Result: No redundant work, clear handoffs

Multi-strategy example:
├─ ArbitrageScanner detects 2-hop opportunities
├─ LiquidationScanner detects liquidation opportunities
├─ TwoHopPlanner validates only 2-hop (ignores liquidations)
├─ LiquidationPlanner validates only liquidations (ignores arbitrage)
└─ Executor handles all validated plans (doesn't care about type)

Result: Zero coupling, perfect separation

METRICS stream captures:
├─ Latency: Every stage publishes timing metrics
├─ Throughput: Event counts per stage
├─ PnL: Real profit from EXECUTED events
├─ Success rate: Planner accuracy, Executor success
└─ Strategy performance: Per-strategy metrics (which scanner/planner)

Distributed tracing:
└─ traceId propagated through FlatBuffers events
   └─ Complete end-to-end visibility (quote → execution)
   └─ Track opportunity from scanner → planner → executor

Per-strategy metrics:
├─ Which scanner found most profitable opportunities
├─ Which planner has highest success rate
├─ Which opportunity types generate most PnL
└─ A/B test different planner implementations

Service crashes:
└─ NATS buffers events (persistent streams)
   └─ Service restarts and processes buffered events
      └─ No lost opportunities

Kill switch:
└─ System halts in <100ms across all frameworks
   └─ All scanners, all planners, executor stop simultaneously
   └─ No runaway losses

Graceful shutdown:
└─ Clean state preservation
   └─ Resume trading without data loss

Strategy isolation:
└─ If LiquidationPlanner crashes:
   ├─ ArbitrageScanner continues (unaffected)
   ├─ TwoHopPlanner continues (unaffected)
   └─ Only liquidation opportunities buffered
   └─ Other strategies execute normally

Before (JSON + monolithic):
├─ Latency: 150-200ms (variable)
├─ Throughput: ~500 events/sec (limited)
├─ Kill switch: Minutes (manual)
└─ Scaling: Vertical only (single service)

After (FlatBuffers + pipeline):
├─ Latency: 95ms (consistent)
├─ Throughput: 10,000 events/sec (scanner)
├─ Kill switch: <100ms (automatic)
└─ Scaling: Horizontal (independent stages)

Improvement: 35% faster + 20x throughput + 600x faster safety

┌────────────────────────────────────────────────────────────┐
│ Metric           │ JSON      │ FlatBuffers │ Improvement  │
├────────────────────────────────────────────────────────────┤
│ Encode time      │ 5-10μs    │ 1-2μs       │ 5-10x faster │
│ Decode time      │ 8-15μs    │ 0.1-0.5μs   │ 20-150x      │
│ Message size     │ 450-600 B │ 300-400 B   │ 30% smaller  │
│ Memory alloc     │ High      │ Minimal     │ Less GC      │
│ Zero-copy read   │ ❌ No     │ ✅ Yes      │ No copies    │
├────────────────────────────────────────────────────────────┤
│ Total pipeline   │ 36ms      │ 5ms         │ -31ms        │
└────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────┐
│ Scenario         │ Without      │ With SYSTEM │ Improvement│
├────────────────────────────────────────────────────────────┤
│ Excessive loss   │ Minutes      │ <100ms      │ 600x faster│
│ Connection fail  │ Blind trades │ Auto pause  │ Prevents loss│
│ Slot drift       │ Unknown      │ Real-time   │ Early warn │
│ Graceful stop    │ Forced kill  │ Clean exit  │ Data safe  │
└────────────────────────────────────────────────────────────┘

Week 1: Infrastructure Setup
├─ Day 1-2: FlatBuffers schema generation and testing
├─ Day 2-3: NATS 6-stream deployment and validation
└─ Day 3-4: SYSTEM stream integration (all frameworks)

Week 2: Framework Migration
├─ Day 1: Scanner FlatBuffers migration
├─ Day 2-3: Planner FlatBuffers + MARKET_DATA integration
└─ Day 4-5: Executor FlatBuffers migration

Week 3: Testing and Deployment
├─ Day 1-2: End-to-end integration testing
├─ Day 2-3: Load testing (1000+ events/sec)
└─ Day 3-5: Production deployment and monitoring

Target: Sub-100ms HFT pipeline in production by Week 3