Quote Service Test Plan

Version: 3.0 Last Updated: December 31, 2025 Owner: Solution Architect & Testing Team Architecture Doc: 30-QUOTE-SERVICE-ARCHITECTURE.md ⭐ PRIMARY REFERENCE Review Enhancements: 26.1-TEST-PLAN-ENHANCEMENTS.md (Critical additions from Gemini/ChatGPT reviews)

1. Executive Summary

The Quote Service is the core engine of the HFT trading pipeline, responsible for sub-10ms quote calculation with 95%+ accuracy. This test plan ensures production readiness through comprehensive unit, integration, load, and HFT-specific testing.

Key Performance Targets (Updated with Critical Enhancements):

Latency:
- HTTP: p50 <1ms (cached), p95 <10ms (uncached), p99 <50ms
- Shared Memory Read: p99 <1μs (100-500x faster than gRPC)
- Arbitrage Detection: p95 <50μs (scanner integration)
Throughput: 1000 req/s sustained, 10,000 req/s peak
Availability: 99.99% uptime (52 minutes downtime/year) - Enhanced with Redis persistence
Cache Hit Rate: >95% (enhanced with pre-computation)
Quote Accuracy: >95% vs actual execution (enhanced with validation layer)
Cold Start Recovery: <3s (vs 30s with Redis persistence)

Test Coverage Target: >80% code coverage, 100% critical path coverage

New Critical Enhancement Tests (See Sections 2.7-2.12):

Shared Memory IPC: Ultra-low-latency quote access for Rust scanners (<1μs reads)
PostgreSQL + Redis Route Storage: Three-tier storage (Shared memory → Redis → PostgreSQL)
Quote Pre-computation: Background refresh for 99.9%+ cache hit rate
Parallel Quote Calculation: Worker pool pattern for 5x speedup
Redis Quote Persistence: <3s cold start recovery from snapshots
Circuit Breaker Per-Quoter: Isolated failure handling per external provider
Quote Validation Layer: Price manipulation detection and outlier filtering
Quote Versioning: Staleness detection with atomic version counters
⭐ Torn Read Prevention (ChatGPT Critical #1): Double-read verification for correctness
⭐ Confidence Scoring (ChatGPT Critical #3): 5-factor deterministic algorithm
⭐ 1s AMM Refresh (Gemini Enhancement): 10× faster opportunity capture
⭐ Parallel Paired Quotes (ChatGPT Exceptional): Eliminate fake arbitrage
⭐ Explicit Timeouts (ChatGPT Critical #2): Non-blocking local-first emission

2. Test Categories

2.1 Unit Tests (40 hours)

2.1.1 Quote Calculator (`internal/quote-service/calculator/`)

Purpose: Validate core quote calculation logic for all DEX protocols

Test Cases:

Single Protocol Quote Calculation
- Input: SOL/USDC, 1 SOL, Raydium AMM V4
- Expected: Quote with price, output amount, price impact
- Assertions:
  - Output amount > 0
  - Price impact < slippage tolerance
  - Calculation time < 1ms
Multi-Protocol Quote Comparison
- Input: SOL/USDC, 1 SOL, [Raydium, Orca, Meteora]
- Expected: Best quote selected (highest output)
- Assertions:
  - All protocols return valid quotes
  - Best quote has highest output amount
  - Calculation time < 5ms for 3 protocols
Slippage Calculation
- Input: Quote with 1% price impact
- Expected: Minimum output with 0.5% slippage buffer
- Assertions:
  - Min output = expected output * (1 - slippage)
  - Slippage BPS correctly applied
Price Impact Validation
- Input: Large trade (10 SOL in low liquidity pool)
- Expected: High price impact warning
- Assertions:
  - Price impact > 5% triggers warning
  - Price impact > 20% rejects quote
Edge Cases
- Zero amount: Should reject
- Negative amount: Should reject
- Amount > pool liquidity: Should reject or warn
- Invalid token mints: Should return error
- Pool paused/disabled: Should skip pool

Code Coverage Target: 90% for calculator package

Tools: Go standard testing, testify/assert

Example Test:

func TestQuoteCalculator_SingleProtocol(t *testing.T) {
    calc := calculator.NewQuoteCalculator()

    quote, err := calc.CalculateQuote(ctx, calculator.QuoteRequest{
        InputMint:  SOL_MINT,
        OutputMint: USDC_MINT,
        Amount:     1_000_000_000, // 1 SOL
        Protocol:   "raydium-amm",
    })

    assert.NoError(t, err)
    assert.Greater(t, quote.OutputAmount, uint64(0))
    assert.Less(t, quote.PriceImpact, 0.05) // <5%
    assert.Less(t, quote.CalculationTime, time.Millisecond)
}

2.1.2 Adaptive Refresh Manager (`internal/quote-service/cache/`)

Test Cases:

Tier Assignment - Hot Tier
- Input: 15 requests in last minute
- Expected: Promoted to Hot tier (5s refresh)
- Assertions:
  - Tier = RefreshTierHot
  - Refresh interval = 5s
  - Prometheus counter incremented
Tier Assignment - Volatility Trigger
- Input: 8% price change
- Expected: Promoted to Hot tier (volatility > 5%)
- Assertions:
  - Tier = RefreshTierHot
  - Volatility correctly calculated
Tier Demotion
- Input: Hot tier pair with <3 req/min for 15 minutes
- Expected: Demoted to Cold tier (60s refresh)
- Assertions:
  - Tier = RefreshTierCold
  - Demotion counter incremented
Manual Override
- Input: SetManualTier(SOL/USDC, Hot)
- Expected: Tier set to Hot, not auto-evaluated
- Assertions:
  - ManualOverride != nil
  - Auto-evaluation skipped
Concurrent Request Recording
- Input: 100 concurrent RecordRequest() calls
- Expected: All requests tracked, no race conditions
- Assertions:
  - Request count = 100
  - No data race (use -race flag)

Code Coverage Target: 85%

Tools: Go testing, testify, sync race detector

2.1.3 Circuit Breaker (`internal/quote-service/resilience/`)

Test Cases:

Circuit Breaker State Transitions
- Closed → Open (after 5 failures)
- Open → Half-Open (after 60s timeout)
- Half-Open → Closed (after 3 successes)
- Half-Open → Open (after 1 failure)
Failure Threshold
- Input: 5 consecutive failures
- Expected: Circuit opens, rejects requests
- Assertions:
  - State = CircuitOpen
  - Error = “circuit breaker open”
  - Prometheus gauge = 1 (open)
Recovery Test
- Input: Open circuit, wait 60s, send successful request
- Expected: Circuit closes after 3 successes
- Assertions:
  - State transitions to HalfOpen, then Closed
  - Requests succeed after recovery

Code Coverage Target: 90%

2.1.4 Slippage Calculator & Tracker

Test Cases:

Dynamic Slippage Calculation
- Input: Historical slippage data [0.1%, 0.2%, 5%]
- Expected: Recommended slippage = p95 (5%)
- Assertions:
  - Slippage between min (0.5%) and max (2%)
Slippage Tracking & Alerts
- Input: Actual slippage 10% > expected 1%
- Expected: Alert triggered
- Assertions:
  - Alert logged
  - Metric incremented

Code Coverage Target: 80%

2.2 Integration Tests (60 hours)

2.2.1 HTTP REST API (`internal/quote-service/api/http/`)

Test Cases:

GET /quote - Success Case

Request: GET /quote?input=SOL&output=USDC&amount=1000000000

Expected Response (200):

{
  "inputMint": "So11111...",
  "outputMint": "EPjFW...",
  "inputAmount": "1000000000",
  "outputAmount": "145230000",
  "priceImpact": 0.012,
  "protocol": "raydium-amm",
  "cacheAge": 3.5,
  "timestamp": 1703635200
}

Assertions:
- Status code = 200
- Response time < 10ms (cache hit)
- Headers: X-Quote-Age-Ms present
- Output amount > 0

GET /quote - Stale Quote Warning

Setup: Cache last refreshed 8s ago

Expected Response (200 with warning):

{
  "outputAmount": "145230000",
  "warning": "Quote is 8.2s old (stale threshold: 5s)",
  "cacheAge": 8.2
}

GET /quote - Price Sanity Check Failure
- Setup: Mock pool returns price 50% above oracle
- Expected Response (400):
```
{
  "error": "Price sanity check failed: 52% deviation from oracle"
}
```
GET /quote - Invalid Input
- Request: GET /quote?input=INVALID&output=USDC&amount=0
- Expected Response (400):
```
{
  "error": "Invalid input mint or zero amount"
}
```

POST /quote/batch - Batch Quote Success

Request:

{
  "quotes": [
    {"input": "SOL", "output": "USDC", "amount": "1000000000"},
    {"input": "SOL", "output": "BONK", "amount": "500000000"}
  ]
}

Expected Response (200):

{
  "results": [
    {"outputAmount": "145230000", "protocol": "raydium-amm"},
    {"outputAmount": "8234567890", "protocol": "meteora-dlmm"}
  ],
  "batchTime": 12.3
}

Assertions:
- Both quotes succeed
- Batch time < 50ms
- Parallel processing used

Rate Limiting - 429 Response
- Setup: Client exceeds 100 req/s limit
- Expected Response (429):
```
{
  "error": "Rate limit exceeded",
  "retryAfter": 1.2
}
```
- Headers:
  - X-RateLimit-Limit: 100
  - X-RateLimit-Remaining: 0
  - Retry-After: 2

Cache Tier Management Endpoints:

GET /cache/tiers - Tier Distribution

Expected Response (200):

{
  "tierCounts": {"hot": 15, "warm": 45, "cold": 120},
  "total": 180
}

GET /cache/tiers/detailed - Detailed Tier Info
- Expected Response (200): Array of pairs with tier, refresh interval, request count, volatility
POST /cache/tiers/manual - Set Manual Tier
- Request:
```
{
  "inputMint": "So11111...",
  "outputMint": "EPjFW...",
  "tier": "hot"
}
```
- Expected: Tier set to hot, manual override flag set

GET /health - Health Check

Expected Response (200):

{
  "status": "healthy",
  "cacheSize": 180,
  "uptime": 3600,
  "latencySLA": {
    "p50": 4.2,
    "p95": 8.9,
    "p99": 45.3,
    "slaBreaches": 2
  }
}

Test Setup:

Mock Solana RPC client
Mock Redis cache
Mock NATS event publisher
Use httptest.Server for API testing

Code Coverage Target: 85% for handler package

2.2.2 gRPC Streaming API (`internal/quote-service/api/grpc/`)

Test Cases:

StreamQuotes - Real-Time Updates
- Setup: Subscribe to SOL/USDC pair
- Trigger: Cache refresh every 5s (hot tier)
- Expected: Stream receives quote updates every 5s
- Assertions:
  - 10 messages received in 50s
  - No duplicate messages
  - Stream stays alive
StreamQuotes - Multiple Subscribers
- Setup: 100 concurrent subscribers
- Expected: All receive updates without blocking
- Assertions:
  - No goroutine leaks
  - Memory usage stable
StreamQuotes - Error Handling
- Trigger: Quote calculation failure
- Expected: Error message streamed to client
- Assertions:
  - Stream not closed
  - Error message contains details

Tools: grpc-go, grpc testing package

2.2.3 Redis Cache Integration

Test Cases:

Cache Write & Read
- Action: Store quote in Redis
- Expected: Quote retrieved with TTL
- Assertions:
  - Quote matches stored data
  - TTL correctly set (5s/15s/60s based on tier)
Cache Expiration
- Setup: Store quote with 5s TTL
- Action: Wait 6s, retrieve
- Expected: Cache miss, quote recalculated
Cache Invalidation
- Action: Clear cache for specific pair
- Expected: Next request triggers recalculation

Test Setup: Miniredis (in-memory Redis mock)

2.2.4 NATS Event Publishing

Test Cases:

Quote Event Publishing

Action: Calculate quote
Expected: Event published to quote.calculated

Event Schema:

{
  "inputMint": "So11111...",
  "outputMint": "EPjFW...",
  "outputAmount": "145230000",
  "protocol": "raydium-amm",
  "timestamp": 1703635200
}

Error Event Publishing
- Action: Quote calculation fails
- Expected: Event published to quote.error

Test Setup: NATS test server (nats-server -DV)

2.3 Load Tests (40 hours)

Tools: k6, Grafana, Prometheus

2.3.1 Sustained Load Test

Scenario: Simulate normal HFT trading load

Configuration:

Duration: 10 minutes
Virtual Users: 500
Target RPS: 1000 req/s
Endpoint: GET /quote

Success Criteria:

✅ p95 latency < 10ms
✅ p99 latency < 50ms
✅ 0% error rate
✅ Cache hit rate > 80%
✅ Memory usage < 1GB
✅ CPU usage < 50% (4 cores)

k6 Script:

import http from 'k6/http';
import { check, sleep } from 'k6';

export let options = {
  stages: [
    { duration: '2m', target: 500 },  // Ramp up
    { duration: '6m', target: 500 },  // Sustain
    { duration: '2m', target: 0 },    // Ramp down
  ],
  thresholds: {
    http_req_duration: ['p(95)<10', 'p(99)<50'],
    http_req_failed: ['rate<0.01'],
  },
};

export default function () {
  const pairs = [
    { input: 'SOL', output: 'USDC' },
    { input: 'SOL', output: 'BONK' },
    { input: 'USDC', output: 'USDT' },
  ];

  const pair = pairs[Math.floor(Math.random() * pairs.length)];
  const amount = Math.floor(Math.random() * 10) * 1e9; // 0-10 SOL

  const res = http.get(
    `http://localhost:8080/quote?input=${pair.input}&output=${pair.output}&amount=${amount}`
  );

  check(res, {
    'status is 200': (r) => r.status === 200,
    'response time < 10ms': (r) => r.timings.duration < 10,
    'has output amount': (r) => JSON.parse(r.body).outputAmount > 0,
  });

  sleep(0.002); // 500 users * 2 req/s = 1000 req/s
}

Monitoring:

Grafana dashboard: Quote service metrics
Prometheus queries:
- rate(http_requests_total[1m])
- histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[1m]))
- cache_hit_rate

2.3.2 Peak Load Test

Scenario: Simulate market volatility spike (10x normal load)

Configuration:

Duration: 5 minutes
Virtual Users: 2500
Target RPS: 5000 req/s

Success Criteria:

✅ p95 latency < 50ms (degraded but acceptable)
✅ p99 latency < 200ms
✅ Error rate < 1%
✅ No crashes or OOM errors
✅ Circuit breaker triggers if needed

Expected Behavior:

Cache hit rate may drop to 60-70% (more unique pairs)
Rate limiting may activate for aggressive clients
Graceful degradation (slower, not broken)

2.3.3 Spike Test

Scenario: Sudden traffic spike (0 → 10,000 req/s in 10s)

Configuration:

Duration: 2 minutes
Ramp up: 10 seconds (0 → 5000 VUs)
Sustain: 1 minute at 10,000 req/s
Ramp down: 30 seconds

Success Criteria:

✅ No crashes
✅ Circuit breaker opens gracefully
✅ Recovery after spike ends

2.3.4 Soak Test

Scenario: Long-duration stability test

Configuration:

Duration: 4 hours
Virtual Users: 200
Target RPS: 500 req/s

Success Criteria:

✅ No memory leaks (memory stable over 4 hours)
✅ No goroutine leaks
✅ Performance does not degrade over time
✅ No connection pool exhaustion

Monitoring:

Track goroutine count: runtime.NumGoroutine()
Track memory: runtime.MemStats.Alloc
Track open connections

2.4 Performance Tests (30 hours)

2.4.1 Latency SLA Compliance

Test: Measure end-to-end latency under various conditions

Scenarios:

Cache Hit - Hot Pair
- Target: p95 < 5ms, p99 < 10ms
- Setup: SOL/USDC cached 2s ago
- Method: 10,000 requests, measure distribution
Cache Miss - Cold Pair
- Target: p95 < 50ms, p99 < 100ms
- Setup: Uncached pair, requires pool data fetch
- Method: 1,000 requests
Multi-Protocol Comparison
- Target: p95 < 20ms (3 protocols)
- Setup: Compare Raydium + Orca + Meteora
- Method: 5,000 requests

Measurement Tool:

func BenchmarkQuoteLatency(b *testing.B) {
    handler := setupTestHandler()

    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        req := httptest.NewRequest("GET", "/quote?input=SOL&output=USDC&amount=1000000000", nil)
        w := httptest.NewRecorder()

        start := time.Now()
        handler.ServeHTTP(w, req)
        elapsed := time.Since(start)

        if elapsed > 10*time.Millisecond {
            b.Errorf("Request took %v, exceeds 10ms SLA", elapsed)
        }
    }
}

Assertions:

p50 < 5ms
p95 < 10ms
p99 < 50ms
p99.9 < 200ms

2.4.2 Cache Performance

Test: Cache hit rate and lookup speed

Metrics:

Cache hit rate (target: >80% for hot pairs)
Cache lookup time (target: <1ms)
Cache write time (target: <2ms)

Test Method:

Prime cache with 1000 pairs
Send 10,000 requests (80% to cached pairs, 20% new)
Measure hit rate and lookup time

2.4.3 Quote Calculation Benchmarks

Test: Measure calculation speed per protocol

Benchmarks:

func BenchmarkRaydiumAMMQuote(b *testing.B) {
    calc := calculator.NewRaydiumAMM()
    for i := 0; i < b.N; i++ {
        _, _ = calc.CalculateQuote(ctx, testInput)
    }
}

func BenchmarkMeteoraQuote(b *testing.B) { /* ... */ }
func BenchmarkOrcaQuote(b *testing.B) { /* ... */ }

Targets:

Raydium AMM: < 0.5ms
Meteora DLMM: < 1ms
Orca Whirlpool: < 1ms

2.4.4 Concurrent Connection Handling

Test: WebSocket connection scalability

Configuration:

1000 concurrent WebSocket connections
Each subscribed to 5 pairs (5000 subscriptions)
Quote updates every 5s

Success Criteria:

✅ All connections receive updates within 100ms of cache refresh
✅ No connection drops
✅ Memory usage < 500MB for 1000 connections

2.5 HFT-Specific Tests (50 hours)

2.5.1 Quote Staleness Detection

Test: Ensure stale quotes are flagged and rejected

Test Cases:

Fresh Quote (< 5s old)
- Setup: Quote cached 3s ago
- Expected: No warning, served normally
- Header: X-Quote-Age-Ms: 3200
Stale Quote (5-10s old)
- Setup: Quote cached 8s ago
- Expected: Warning in response
- Response: "warning": "Quote is 8.2s old"
Very Stale Quote (> 10s old)
- Setup: Quote cached 12s ago
- Expected: Reject with 503 Service Unavailable
- Response: "error": "Quote too stale (12.5s), recalculation failed"
Circuit Breaker During Staleness
- Setup: Quote stale, RPC circuit breaker open
- Expected: Return stale quote with strong warning
- Response: "warning": "Using stale quote (8s) due to upstream issues"

Monitoring:

Prometheus metric: quote_staleness_warnings_total
Alert: quote_staleness_warnings_total > 100/min

2.5.2 Price Sanity Checks

Test: Prevent catastrophic losses from bad quotes

Test Cases:

Price Deviation Check - Accept
- Setup: Pool price = $145, Oracle price = $148 (2% deviation)
- Expected: Quote accepted
- Assertions: No warnings
Price Deviation Check - Warning
- Setup: Pool price = $145, Oracle price = $160 (10.3% deviation)
- Expected: Quote accepted with warning
- Response: "warning": "Price deviates 10.3% from oracle"
Price Deviation Check - Reject
- Setup: Pool price = $145, Oracle price = $220 (51% deviation)
- Expected: Quote rejected
- Response: "error": "Price sanity check failed: 51% deviation"
Spread Sanity Check
- Setup: Bid = $140, Ask = $180 (28% spread)
- Expected: Quote rejected
- Response: "error": "Spread too wide: 28% (max 20%)"
Liquidity Drop Check
- Setup: Pool liquidity dropped from $1M to $300K (70% drop)
- Expected: Warning logged, quote recalculated
- Metric: pool_liquidity_drops_total incremented

Oracle Integration:

Use Pyth Network for price feeds
Fallback to Chainlink if Pyth unavailable
30s cache for oracle prices

2.5.3 Circuit Breaker Integration

Test: Verify circuit breaker protects against cascading failures

Test Cases:

RPC Failure - Circuit Opens
- Setup: Mock RPC returns 500 errors (5 consecutive)
- Expected: Circuit opens, requests fail fast
- Response Time: < 1ms (no RPC call)
- Response: "error": "Service temporarily unavailable (circuit breaker open)"
Half-Open Recovery
- Setup: Circuit open for 60s, then 1 successful request
- Expected: Circuit transitions to half-open, then closed after 3 successes
- Assertions:
  - State: Open → HalfOpen → Closed
  - Successful requests served
Partial Failure - Protocol-Level Circuit Breaker
- Setup: Raydium RPC failing, Orca working
- Expected: Raydium circuit opens, quotes fall back to Orca
- Assertions:
  - Raydium quotes fail fast
  - Orca quotes succeed
  - Overall service stays healthy

Monitoring:

Grafana panel: Circuit breaker state (0=Closed, 1=Open, 2=HalfOpen)
Alert: circuit_breaker_state == 1 (Open)

2.5.4 Rate Limiting Per Client

Test: Prevent abuse and ensure fair resource allocation

Test Cases:

Under Limit - Success
- Setup: Client sends 50 req/s (limit: 100 req/s)
- Expected: All requests succeed
- Headers:
  - X-RateLimit-Limit: 100
  - X-RateLimit-Remaining: 50
Exceed Limit - 429 Response
- Setup: Client sends 150 req/s (limit: 100 req/s)
- Expected: 50 requests rejected with 429
- Headers:
  - X-RateLimit-Remaining: 0
  - Retry-After: 1
Premium Client - Higher Limit
- Setup: Client with API key (limit: 1000 req/s)
- Expected: All 1000 req/s succeed
- Assertions:
  - No 429 responses
  - X-RateLimit-Limit: 1000
Token Bucket Refill
- Setup: Client exhausts tokens, waits 1s
- Expected: 100 new tokens available
- Assertions:
  - Tokens refill at 100/sec
  - Burst capacity = 200 tokens

Implementation:

Use golang.org/x/time/rate (token bucket)
Per-IP rate limiting by default
Per-API-key for authenticated clients

2.5.5 Failover & Graceful Degradation

Test: System remains available during partial failures

Test Cases:

Redis Failure - Degrade to In-Memory Cache
- Setup: Redis container stopped
- Expected: Service continues with in-memory cache (reduced capacity)
- Assertions:
  - Cache hit rate drops but service functional
  - Warning logged: “Redis unavailable, using in-memory cache”
NATS Failure - Continue Without Events
- Setup: NATS container stopped
- Expected: Quotes still served, events not published
- Assertions:
  - Quote API functional
  - Warning logged: “Event publishing failed”
Partial RPC Failure - Retry Logic
- Setup: 2 of 5 RPC endpoints failing
- Expected: Requests retry to healthy endpoints
- Assertions:
  - Success rate > 95%
  - Latency increases slightly
Database Failure - Read-Only Mode
- Setup: PostgreSQL down (if used for pool metadata)
- Expected: Serve from cache, no new pool discovery
- Assertions:
  - Cached quotes served
  - /health returns 503 with details

2.6 Functional Tests (20 hours)

2.6.1 Multi-Protocol Support

Test: Verify all DEX protocols work correctly

Protocols to Test:

✅ Raydium AMM V4
✅ Raydium CLMM
✅ Raydium CPMM
✅ Orca Whirlpool
✅ Meteora DLMM
✅ Pump.fun AMM

Test Per Protocol:

Calculate quote for test pair
Verify output amount > 0
Verify price impact reasonable (<5% for 1 SOL)
Verify quote matches on-chain simulation

Test Data:

{
  "raydium-amm": {
    "pair": "SOL/USDC",
    "pool": "58oQChx4yWmvKdwLLZzBi4ChoCc2fqCUWBkwMihLYQo2",
    "expectedOutput": "145000000" // ~145 USDC for 1 SOL
  },
  "orca-whirlpool": {
    "pair": "SOL/USDC",
    "pool": "HJPjoWUrhoZzkNfRpHuieeFk9WcZWjwy6PBjZ81ngndJ",
    "expectedOutput": "145500000"
  }
}

2.6.2 Error Handling

Test: Graceful error responses

Error Scenarios:

Invalid mint address → 400 Bad Request
Unsupported token pair → 404 Not Found
RPC timeout → 503 Service Unavailable
Circuit breaker open → 503 with Retry-After
Internal panic → 500 (recovered, logged)

Assertions:

Correct HTTP status code
Error message in JSON response
Error logged to observability system
No service crash

2.6.3 Health Check Endpoints

Test: Health endpoints return correct status

Endpoints:

GET /health
- Healthy: 200 with service details
- Degraded: 200 with warnings
- Unhealthy: 503 with error details
GET /health/ready
- Ready: 200 (can serve traffic)
- Not ready: 503 (dependencies unavailable)
GET /health/live
- Live: 200 (process alive)
- Dead: No response (process crashed)

GET /health/sla

Response:

{
  "latency": {
    "p50": 4.2, "p95": 8.9, "p99": 45.3, "p99.9": 189
  },
  "slaBreaches": {
    "last1h": 2,
    "last24h": 15
  },
  "status": "healthy"
}

2.7 Critical Enhancement Tests (60 hours) 🆕

Reference: See docs/32-QUOTE-SERVICE-CRITICAL-ENHANCEMENTS.md for implementation details

This section covers tests for the critical enhancements that make quote-service FAST, RELIABLE, and CORRECT as the HFT core engine.

2.7.1 Shared Memory IPC Tests (P0 - HFT Critical)

Purpose: Verify ultra-low-latency quote access for Rust scanners/planners

Test Cases:

Memory-Mapped File Creation and Initialization
- Action: Initialize SharedMemoryWriter with 10,000 entries
- Expected: File created at configured path, header initialized
- Assertions:
  - File exists at C:\workspace\solana\solana-trading-system\data\shm\quotes.mmap
  - File size = 4096 (header) + (10,000 × 208) = 2,084,096 bytes
  - Magic number = 0x514F545453 (“QOTTS”)
  - Version = 1, EntrySize = 160, MaxEntries = 10,000
Atomic Quote Write (Single Writer)
- Action: Write 100 quotes concurrently from Go service
- Expected: All writes succeed, no data corruption
- Assertions:
  - All 100 quotes readable from shared memory
  - Version numbers monotonically increasing
  - No torn reads (version validation passes)
  - Write latency <10μs (p99)
Lock-Free Quote Read (Multiple Readers)
- Setup: 10,000 concurrent readers (simulating Rust scanners)
- Action: Read same quote simultaneously
- Expected: All reads succeed, no blocking
- Assertions:
  - Read latency <1μs (p99)
  - Version validation prevents torn reads
  - No data races (race detector clean)
  - Throughput >10M reads/sec
PairID Hash Collision Test
- Action: Generate 10,000 unique PairIDs (different token pairs/amounts)
- Expected: No hash collisions
- Assertions:
  - All PairIDs unique
  - Hash distribution uniform (chi-squared test)
Docker Volume Mount Test
- Setup: Quote service runs on host, scanner in Docker container
- Action: Write quote from host, read from container
- Expected: Container reads quote successfully
- Assertions:
  - Volume mount path correct: /app/data/shm/quotes.mmap
  - Read-only mount works (:ro flag)
  - Quote data matches between host and container
Shared Memory Corruption Recovery
- Action: Simulate writer crash mid-write (version mismatch)
- Expected: Readers retry and eventually read consistent data
- Assertions:
  - Retry logic works (max 3 retries)
  - Inconsistent reads detected (version mismatch)
  - Recovery time <10μs

Performance Benchmarks:

func BenchmarkSharedMemoryWrite(b *testing.B) {
    writer := shm.NewSharedMemoryWriter("test.mmap", 10000)
    entry := &shm.QuoteEntry{/* ... */}

    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        _ = writer.WriteQuote(entry)
    }
    // Target: >1,000,000 writes/sec (< 1μs per write)
}

func BenchmarkSharedMemoryRead(b *testing.B) {
    reader := shm.NewSharedMemoryReader("test.mmap")
    pairID := [32]byte{/* ... */}

    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        _ = reader.ReadQuote(&pairID)
    }
    // Target: >10,000,000 reads/sec (< 0.1μs per read)
}

Tools:

Go: testing, unsafe, syscall.Mmap
Rust: memmap2, criterion (benchmarks)
Docker: Volume mounting validation

2.7.2 Route Storage Tests (P1 - Correctness)

Purpose: Verify atomic route storage with PostgreSQL + Redis

Test Cases:

Atomic Route-Then-Quote Write
- Action: Calculate paired quotes, store routes, write to shared memory
- Expected: RouteIDs referenced in shared memory exist in PostgreSQL
- Assertions:
  - Routes stored in PostgreSQL before shared memory write
  - No dangling RouteID references
  - Both forward and reverse routes stored
  - Operation completes in <50ms (p95)
RouteID Determinism
- Action: Calculate same route 1000 times (same pools, same order)
- Expected: Same RouteID every time
- Assertions:
  - RouteID = BLAKE3(RouteSteps) is deterministic
  - No collisions for different routes
  - Hash distribution uniform
PostgreSQL Route Persistence
- Action: Store 10,000 unique routes
- Expected: All routes retrievable by RouteID
- Assertions:
  - Indexed query <10ms (p95)
  - JSONB queries work correctly
  - 7-day retention enforced (old routes cleaned up)
  - Upsert logic works (ON CONFLICT)
Redis Route Cache Hit Rate
- Action: Store 1000 routes, access 100 hot routes 10,000 times
- Expected: >95% cache hit rate
- Assertions:
  - Cache hit latency <1ms (p95)
  - Cache miss fallback to PostgreSQL <10ms
  - TTL=30s enforced
  - Cache warming works on PostgreSQL fetch
Race Condition Prevention
- Setup: 100 concurrent GetPairedQuotes() calls for same pair
- Expected: No race conditions, routes stored exactly once
- Assertions:
  - No duplicate route inserts
  - All shared memory writes reference existing routes
  - No “route not found” errors from scanners

SQL Tests:

-- Test route insertion and retrieval
INSERT INTO route_steps (route_id, route_steps, hop_count, first_dex, last_dex, total_fee_bps)
VALUES ($1, $2, $3, $4, $5, $6)
ON CONFLICT (route_id) DO UPDATE SET
    last_used_at = NOW(),
    use_count = route_steps.use_count + 1;

-- Test indexed query performance
EXPLAIN ANALYZE SELECT route_steps FROM route_steps WHERE route_id = $1;
-- Target: Index Scan, <10ms execution time

-- Test retention cleanup
DELETE FROM route_steps
WHERE last_used_at < NOW() - INTERVAL '7 days' AND use_count < 10;

2.7.3 Quote Pre-Computation Tests (P1 - Performance)

Purpose: Verify background pre-computation achieves 99.9%+ cache hit rate

Test Cases:

Background Refresh Scheduling
- Setup: 100 monitored pairs, 10s refresh interval
- Expected: All pairs refreshed every 10s
- Assertions:
  - Background goroutine runs continuously
  - Refresh cycles complete in <10s (doesn’t accumulate delay)
  - No goroutine leaks
Pre-Computation Latency
- Action: Pre-compute 100 paired quotes
- Expected: All quotes computed within 10s interval
- Assertions:
  - Parallel computation (semaphore = 10 concurrent)
  - Average quote computation <100ms
  - Total cycle time <10s (99 pairs × 100ms ÷ 10 workers = 990ms + overhead)
Cache Hit Rate Improvement
- Setup: Pre-computation disabled vs enabled
- Action: Send 10,000 requests to 10 hot pairs
- Expected Improvement:
  - Without pre-computation: ~80% cache hit rate
  - With pre-computation: >99.9% cache hit rate
- Assertions:
  - Near-instant responses (<1ms) for all requests
  - Cache misses only for new pairs or stale data
Pre-Computation Error Handling
- Action: Simulate RPC failure during background refresh
- Expected: Service continues, stale cache served, warning logged
- Assertions:
  - Background refresh retries next cycle
  - User requests not affected (served from cache)
  - Error metric incremented: precompute_errors_total
Dynamic Pair Addition
- Action: Add new pair via /pairs/custom during pre-computation
- Expected: New pair included in next refresh cycle
- Assertions:
  - New pair quote appears in cache within 10s
  - No restart required

Performance Comparison:

Scenario: 10,000 requests to 10 hot pairs

Without Pre-Computation:
- Cache hit rate: 80%
- Cache miss latency: 10-50ms
- p95 latency: 45ms

With Pre-Computation:
- Cache hit rate: 99.9%
- Cache hit latency: <1ms
- p95 latency: <1ms

Improvement: 45x faster (45ms → 1ms)

2.7.4 Parallel Quote Calculation Tests (P1 - Performance)

Purpose: Verify worker pool pattern achieves 5x speedup for multi-pool quotes

Test Cases:

Worker Pool Concurrency
- Setup: Quote request with 50 pools
- Expected: Pools quoted in parallel (10 workers)
- Assertions:
  - 10 goroutines active during calculation
  - Total time ≈ max(pool latencies), not sum
  - Result collected correctly from all workers
Performance Comparison (Sequential vs Parallel)
- Setup: Quote calculation for 50 pools
- Sequential: 50 pools × 2ms = 100ms
- Parallel (10 workers): max(50 pools) ÷ 10 × 2ms ≈ 10ms
- Expected Speedup: 10x faster
- Assertions:
  - Parallel calculation <15ms (p95)
  - All pools processed
  - Best quote selected correctly
Worker Pool Semaphore Limit
- Setup: 100 concurrent quote requests, each quoting 50 pools
- Expected: Worker pool limits concurrent goroutines
- Assertions:
  - Total goroutines ≤ (100 requests × 10 workers) = 1000
  - No goroutine exhaustion
  - Memory usage stable
Error Handling in Parallel Workers
- Setup: 50 pools, 10 fail with RPC errors
- Expected: Failed pools skipped, best quote from remaining 40 pools
- Assertions:
  - No panic from worker errors
  - Failed pools logged
  - Quote still returns (degraded but functional)

Benchmark:

func BenchmarkParallelQuoteCalculation(b *testing.B) {
    pools := generateTestPools(50) // 50 test pools
    optimizer := calculator.NewRouteOptimizer(10) // 10 workers

    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        _, _ = optimizer.FindBestRoute(ctx, pools, inputMint, outputMint, amount)
    }
    // Target: <15ms per quote (5x faster than sequential)
}

2.7.5 Redis Quote Persistence Tests (P0 - Reliability)

Purpose: Verify <3s cold start recovery from Redis snapshots

Test Cases:

Snapshot Creation
- Setup: 1000 quotes in cache
- Action: Trigger snapshot to Redis
- Expected: All quotes written to Redis with TTL=30s
- Assertions:
  - 1000 keys created: quote:cache:{pairID}
  - Snapshot completes in <1s
  - TTL correctly set on all keys
Cold Start Recovery
- Setup: Redis contains 1000 cached quotes
- Action: Restart quote-service
- Expected: Cache loaded from Redis in <3s
- Assertions:
  - Service ready in <3s (vs 30s without persistence)
  - All 1000 quotes available immediately
  - Cache hit rate >90% on first request
Snapshot Staleness
- Setup: Redis snapshot 25s old (near TTL expiration)
- Action: Load cache on startup
- Expected: Stale quotes marked, background refresh triggered
- Assertions:
  - Stale quotes served (better than nothing)
  - Warning logged: “Loaded stale cache (age: 25s)”
  - Background refresh starts immediately
Snapshot Failure Handling
- Setup: Redis unavailable during snapshot
- Expected: Service continues, warning logged
- Assertions:
  - Snapshot error logged but doesn’t block service
  - In-memory cache still functional
  - Retry on next snapshot cycle (10s later)
Multi-Instance Cache Sharing
- Setup: 3 quote-service instances sharing Redis
- Action: Instance 1 writes snapshot, Instance 2 reads
- Expected: Cache shared across instances
- Assertions:
  - Instance 2 benefits from Instance 1’s snapshots
  - No cache duplication
  - TTL synchronized across instances

Performance Test:

func TestColdStartRecovery(t *testing.T) {
    // 1. Prime cache with 1000 quotes
    service := setupQuoteService()
    for i := 0; i < 1000; i++ {
        service.GetQuote(ctx, pairs[i].input, pairs[i].output, amounts[i], 50)
    }

    // 2. Snapshot to Redis
    service.snapshotCache()

    // 3. Restart service (simulate cold start)
    start := time.Now()
    newService := setupQuoteService() // Loads from Redis
    loadTime := time.Since(start)

    // 4. Assertions
    assert.Less(t, loadTime, 3*time.Second, "Cold start should complete in <3s")
    assert.Equal(t, 1000, newService.cacheSize(), "All quotes loaded from Redis")

    // 5. Verify immediate cache hits
    quote, err := newService.GetQuote(ctx, "SOL", "USDC", 1000000000, 50)
    assert.NoError(t, err)
    assert.True(t, quote.CacheHit, "First request should be cache hit")
}

2.7.6 Quote Validation Tests (P1 - Correctness)

Purpose: Verify price manipulation protection and outlier detection

Test Cases:

Oracle Price Comparison (±10% Threshold)
- Test 1: Quote price = $145, Oracle = $150 (3.4% deviation) → Accept
- Test 2: Quote price = $145, Oracle = $165 (13.8% deviation) → Reject
- Assertions:
  - Validation result reflects deviation
  - Warning logged for rejections
  - Metric incremented: quote_validation_failures_total{reason="oracle_deviation"}
Historical Price Outlier Detection (Z-Score)
- Setup: Historical prices = [$145, $146, $144, $147, $145] (mean=$145.4, stdDev=$1.14)
- Test 1: New quote = $150 (z-score = 4.0) → Reject (outlier)
- Test 2: New quote = $146 (z-score = 0.5) → Accept (normal)
- Assertions:
  - Z-score calculated correctly
  - Outliers rejected ( z-score > 3)
  - Normal quotes accepted
Multi-DEX Price Consistency
- Setup: Get quotes from Raydium, Orca, Meteora for same pair
- Test 1: Prices = [$145, $146, $145.5] (max deviation 0.7%) → Accept
- Test 2: Prices = [$145, $180, $146] (max deviation 24%) → Reject
- Assertions:
  - Consistency check only for large trades (>$1000)
  - Threshold = 2% max deviation
  - Rejected quotes logged
Validation Confidence Scoring
- Test: Quote passes all validations
- Expected: Confidence = 1.0
- Test: Quote fails oracle check but passes historical
- Expected: Confidence = 0.5
- Assertions:
  - Confidence field populated in Quote struct
  - Lower confidence triggers warnings
Price History Management
- Action: Store 1000 price observations
- Expected: Rolling window of last 100 prices kept
- Assertions:
  - Memory usage bounded (100 prices × pairs)
  - Old prices evicted (FIFO)

Test Implementation:

func TestQuoteValidation_OracleDeviation(t *testing.T) {
    mockOracle := &MockOracleReader{
        prices: map[string]float64{
            "SOL/USDC": 150.0, // Oracle price
        },
    }

    validator := validator.NewQuoteValidator(mockOracle)

    quote := &domain.Quote{
        InputMint:    SOL_MINT,
        OutputMint:   USDC_MINT,
        InputAmount:  1_000_000_000, // 1 SOL
        OutputAmount: 145_000_000,    // $145 (3.3% below oracle)
    }

    result := validator.Validate(ctx, quote)

    assert.True(t, result.IsValid, "3.3% deviation should be accepted (<10% threshold)")
    assert.GreaterOrEqual(t, result.Confidence, 0.9)
}

func TestQuoteValidation_Outlier(t *testing.T) {
    validator := validator.NewQuoteValidator(nil)

    // Seed price history
    for i := 0; i < 50; i++ {
        validator.priceHistory.Add("SOL", "USDC", 145.0 + rand.Float64()*2) // $145 ± $1
    }

    // Test outlier quote
    quote := &domain.Quote{
        InputMint:    SOL_MINT,
        OutputMint:   USDC_MINT,
        InputAmount:  1_000_000_000,
        OutputAmount: 180_000_000, // $180 (outlier, z-score > 3)
    }

    result := validator.Validate(ctx, quote)

    assert.False(t, result.IsValid, "Outlier should be rejected")
    assert.Contains(t, result.Reason, "price outlier")
}

2.7.7 Circuit Breaker Per-Quoter Tests (P1 - Reliability)

Purpose: Verify per-quoter circuit breakers provide isolation

Test Cases:

Per-Quoter Isolation
- Setup: 5 quoters (Jupiter, DFlow, OKX, QuickNode, BLXR)
- Action: Jupiter fails 5 times → circuit opens
- Expected: Jupiter circuit open, others remain closed
- Assertions:
  - Jupiter requests fail fast (<1ms)
  - DFlow/OKX/QuickNode/BLXR still functional
  - Metric: circuit_breaker_state{quoter="Jupiter"} = 1 (open)
Graceful Degradation with Fallback
- Setup: Jupiter circuit open, DFlow working
- Action: Request external quote
- Expected: Falls back to DFlow
- Assertions:
  - Jupiter skipped (circuit open)
  - DFlow quote returned
  - Response includes: "fallback": "DFlow (Jupiter circuit open)"
Circuit Recovery Per Quoter
- Setup: Jupiter circuit open (5 failures)
- Action: Wait 60s, send successful request
- Expected: Jupiter circuit transitions HalfOpen → Closed
- Assertions:
  - State transitions tracked correctly
  - 3 successful requests required for full recovery
  - Metric: circuit_breaker_state{quoter="Jupiter"} = 0 (closed)
Concurrent Circuit Breakers
- Setup: 100 concurrent requests to 5 quoters
- Action: Trigger failures randomly across quoters
- Expected: Each circuit breaker operates independently
- Assertions:
  - No cross-quoter interference
  - State transitions per quoter tracked correctly
  - No data races (use -race flag)

Test Implementation:

func TestCircuitBreakerPerQuoter(t *testing.T) {
    // Setup mock quoters
    jupiter := &MockQuoter{failCount: 5}  // Will fail
    dflow := &MockQuoter{failCount: 0}    // Will succeed

    manager := quoters.NewQuoterManager(map[string]Quoter{
        "Jupiter": jupiter,
        "DFlow":   dflow,
    })

    // Trigger Jupiter failures (circuit should open)
    for i := 0; i < 5; i++ {
        _, err := manager.GetQuote("Jupiter", ctx, inputMint, outputMint, amount)
        assert.Error(t, err)
    }

    // Verify Jupiter circuit open
    jupiterCB := manager.GetCircuitBreaker("Jupiter")
    assert.True(t, jupiterCB.IsOpen(), "Jupiter circuit should be open after 5 failures")

    // Verify DFlow still works
    dflowQuote, err := manager.GetQuote("DFlow", ctx, inputMint, outputMint, amount)
    assert.NoError(t, err, "DFlow should still work (circuit isolated)")
    assert.NotNil(t, dflowQuote)

    // Verify DFlow circuit still closed
    dflowCB := manager.GetCircuitBreaker("DFlow")
    assert.False(t, dflowCB.IsOpen(), "DFlow circuit should remain closed")
}

2.7.8 Quote Versioning Tests (P1 - Correctness)

Purpose: Verify staleness detection for HFT use cases

Test Cases:

Version Monotonicity
- Action: Generate 1000 quotes for same pair
- Expected: Version numbers monotonically increasing
- Assertions:
  - version[i+1] > version[i] for all i
  - No version collisions
  - Atomic increment works under concurrency
Staleness Detection
- Test 1: Quote age 3s, ExpiresAt = now + 27s → Fresh
- Test 2: Quote age 8s, ExpiresAt = now + 22s → Warning
- Test 3: Quote age 35s, ExpiresAt = now - 5s → Stale (expired)
- Assertions:
  - IsFresh flag correctly set
  - HTTP header: X-Quote-Age-Ms accurate
  - Stale quotes trigger warning or rejection
HTTP Header Validation
- Action: Request quote
- Expected HTTP headers:
  - X-Quote-Version: 12345
  - X-Quote-Age-Ms: 3200
  - X-Quote-Is-Stale: false
- Assertions:
  - Headers present in response
  - Values accurate
Staleness Metrics
- Action: Serve 100 fresh quotes, 10 stale quotes
- Expected Metrics:
  - quote_served_total{stale="false"} = 100
  - quote_served_total{stale="true"} = 10
  - quote_staleness_warnings_total = 10
- Assertions:
  - Metrics incremented correctly
  - Grafana dashboard shows staleness rate

Test Implementation:

func TestQuoteVersioning(t *testing.T) {
    service := setupQuoteService()

    // Generate 1000 quotes
    versions := make([]uint64, 1000)
    for i := 0; i < 1000; i++ {
        quote, _ := service.GetQuote(ctx, "SOL", "USDC", 1000000000, 50)
        versions[i] = quote.Version
    }

    // Verify monotonicity
    for i := 1; i < len(versions); i++ {
        assert.Greater(t, versions[i], versions[i-1], "Versions should be monotonically increasing")
    }
}

func TestQuoteStaleness(t *testing.T) {
    quote := &domain.Quote{
        GeneratedAt: time.Now().Add(-8 * time.Second), // 8s ago
        ExpiresAt:   time.Now().Add(22 * time.Second), // Expires in 22s (30s TTL - 8s age)
    }

    age := time.Since(quote.GeneratedAt)
    isExpired := time.Now().After(quote.ExpiresAt)

    assert.Equal(t, 8*time.Second, age.Round(time.Second))
    assert.False(t, isExpired, "Quote should not be expired yet")

    // Mark as stale if age > 5s
    if age > 5*time.Second {
        quote.IsStale = true
    }

    assert.True(t, quote.IsStale, "Quote >5s old should be marked stale")
}

2.7.9 Integration Tests: End-to-End HFT Pipeline

Purpose: Verify complete pipeline from quote → arbitrage detection

Test Cases:

Quote Service → Shared Memory → Scanner Integration
- Setup: Go quote-service writes quotes, Rust scanner reads
- Action: Calculate paired quotes for SOL/USDC
- Expected: Scanner detects arbitrage opportunity
- Assertions:
  - Quote written to shared memory (<10μs)
  - Scanner reads quote (<1μs)
  - RouteID lookup from Redis (<1ms)
  - Total detection time <50μs
Cold Start Performance Test
- Setup: Fresh quote-service start with Redis persistence
- Action: Measure time to first successful quote
- Expected: <3s until service ready
- Assertions:
  - Cache loaded from Redis
  - First quote request is cache hit
  - Service marked “ready” in <3s
Failover Test: Redis Down
- Setup: Quote service running, Redis stopped
- Action: Continue serving quotes
- Expected: Graceful degradation (in-memory cache only)
- Assertions:
  - Quotes still served (degraded capacity)
  - Warning logged: “Redis unavailable”
  - Snapshot fails gracefully
Failover Test: PostgreSQL Down
- Setup: Scanner requests RouteSteps, PostgreSQL down
- Action: Fallback to Redis cache only
- Expected: Cache hits work, cache misses fail gracefully
- Assertions:
  - Cached routes still accessible
  - New routes cannot be stored
  - Warning logged

Performance Benchmarks (Full Pipeline):

Metric                                Target        Achieved
───────────────────────────────────────────────────────────
Quote write (Go)                     <10μs         5-8μs ✅
Quote read (Rust)                    <1μs          0.3μs ✅
RouteID lookup (Redis)               <1ms          0.5ms ✅
Arbitrage detection                  <50μs         20μs ✅
End-to-end (quote → detection)       <100μs        30μs ✅
Cold start recovery                  <3s           2.1s ✅

2.7.10 Test Summary: Critical Enhancements

Enhancement	Test Hours	Priority	Coverage Target
Shared Memory IPC	12 hours	P0	95%
Route Storage (PostgreSQL + Redis)	8 hours	P1	90%
Quote Pre-Computation	6 hours	P1	85%
Parallel Quote Calculation	4 hours	P1	90%
Redis Quote Persistence	8 hours	P0	95%
Quote Validation Layer	8 hours	P1	90%
Circuit Breaker Per-Quoter	6 hours	P1	90%
Quote Versioning	4 hours	P1	85%
End-to-End Integration	4 hours	P0	100%
Total	60 hours	Mixed	92% avg

Expected Impact:

✅ 100-500x faster quote reads (gRPC → shared memory)
✅ 50-200x faster arbitrage detection (500μs-2ms → <10μs)
✅ 10x faster cold start (30s → <3s)
✅ 99.99% uptime (vs 99.9% with enhancements)
✅ Price manipulation protection (validation layer)

2.8 Torn Read Prevention Tests (Review Enhancement) ⭐ CRITICAL

Priority: P0 - CORRECTNESS Estimated Effort: 4 hours Review Source: ChatGPT critique #1 Design Doc: 30.2-SHARED-MEMORY-HYBRID-CHANGE-DETECTION.md

Purpose

Validate that the shared memory reader’s double-read verification protocol prevents torn reads (reading partially-written structs) under concurrent write pressure.

Test Cases

Test 2.8.1: No Torn Reads Under Heavy Contention

Scenario: Multiple concurrent writers (1000 writes/sec) while readers continuously poll

Setup:

// 10 writer threads
for i in 0..10 {
    spawn(move || {
        for j in 0..100 {
            writer.write_quote(i, create_test_quote(j));
            thread::sleep(Duration::from_millis(10)); // 100 writes/sec per thread
        }
    });
}

// 5 reader threads
for _ in 0..5 {
    spawn(move || {
        for _ in 0..10_000 {
            let quotes = reader.read_changed_quotes();
            validate_no_torn_reads(&quotes);
        }
    });
}

Validation:

fn validate_no_torn_reads(quotes: &[(u32, QuoteMetadata)]) {
    for (idx, quote) in quotes {
        let v1 = quote.version.load(Ordering::Acquire);

        // ✅ Version must be even (readable)
        assert_eq!(v1 % 2, 0, "Torn read detected: odd version");

        // ✅ All fields must be consistent
        assert!(quote.output_amount > 0, "Invalid output amount");
        assert!(quote.price_impact_bps < 10000, "Invalid price impact");

        // ✅ Oracle price must be valid
        assert!(quote.oracle_price_usd > 0.0, "Invalid oracle price");
    }
}

Assertions:

✅ 0 torn reads in 50,000 total reads
✅ All versions are even (readable state)
✅ All field values are valid (no corruption)
✅ No panics or crashes

Acceptance Criteria:

Pass: 0 torn reads detected
Fail: Any torn read (odd version or corrupted data)

Test 2.8.2: Retry Mechanism Under Active Writes

Scenario: Reader attempts to read while writer is actively writing (odd version)

Validation Pattern:

fn read_quote_safe(&self, quote: &QuoteMetadata) -> Option<QuoteMetadata> {
    let mut retry_count = 0;

    for _ in 0..10 {  // Max 10 retries
        let v1 = quote.version.load(Ordering::Acquire);

        if v1 % 2 != 0 {
            retry_count += 1;
            std::hint::spin_loop();
            continue;  // Retry if odd (writing)
        }

        let quote_copy = /* copy struct */;
        let v2 = quote.version.load(Ordering::Acquire);

        if v1 == v2 {
            METRICS.record_retry_count(retry_count);
            return Some(quote_copy);  // ✅ Success
        }

        retry_count += 1;
    }

    None  // Failed after 10 retries
}

Assertions:

✅ 100% success rate (no None returns)
✅ Average retry count < 3
✅ p99 retry count < 10
✅ Total latency < 500ns (p99)

Test 2.8.3: Performance Under No Contention

Target Latency:

Average: <50ns
p95: <100ns
p99: <200ns
0 retries (first read succeeds)

Code Coverage Target: >90% for read_quote_safe() function

Tools: Rust testing, criterion benchmarks, thread sanitizer

2.9 Confidence Score Validation Tests (Review Enhancement) ⭐ CRITICAL

Priority: P0 - HFT REQUIREMENT Estimated Effort: 4 hours Review Source: ChatGPT critique #3 Design Doc: 30.4-CHATGPT-REVIEW-RESPONSE.md

Purpose

Validate that the 5-factor confidence scoring algorithm produces deterministic, correct scores in [0.0, 1.0] range and enables proper scanner decision-making.

Test Cases

Test 2.9.1: High Confidence Quote (Fresh, On-Chain, Accurate)

Scenario: Fresh pool state, direct swap, perfect oracle match

Input:

quote := &Quote{
    PoolLastUpdate:  time.Now().Add(-3 * time.Second),  // 3s old
    RouteHops:       1,                                   // Direct swap
    OutputAmount:    154_000_000,                        // 154 USDC
    InputAmount:     1_000_000_000,                      // 1 SOL
    PriceImpactBps:  20,                                 // 0.2%
    Pool: &Pool{Depth: 5_000_000_000},                  // 5000 SOL depth
    Provider:        "local",
}

oracle := &OraclePrice{PriceUSD: 154.0}  // Matches quote

Expected Confidence Calculation:

// 1. Pool State Age: 3s old → 1.0 - (3/60) = 0.95 (weight 30%)
poolAgeFactor := 0.95

// 2. Route Hop Count: 1 hop → 1.0 - (0 * 0.2) = 1.0 (weight 20%)
routeFactor := 1.0

// 3. Oracle Deviation: 154 vs 154 → 0% → 1.0 (weight 30%)
oracleFactor := 1.0

// 4. Provider Reliability: local = 100% → 1.0 (weight 10%)
providerFactor := 1.0

// 5. Slippage vs Depth: 0.2% in 5000 SOL → expected = actual → 1.0 (weight 10%)
slippageFactor := 1.0

// Weighted sum
confidence := 0.95*0.30 + 1.0*0.20 + 1.0*0.30 + 1.0*0.10 + 1.0*0.10
            = 0.285 + 0.20 + 0.30 + 0.10 + 0.10
            = 0.985

Assertions:

✅ confidence >= 0.95 (high confidence)
✅ poolAgeFactor >= 0.95
✅ oracleFactor == 1.0
✅ Scanner decision: Strategy::Execute

Test 2.9.2: Low Confidence Quote (Stale, Multi-Hop, Oracle Mismatch)

Scenario: Stale pool (45s), 3-hop route, 9% oracle deviation

Expected Confidence: ~0.34 (low confidence) Scanner Decision: Strategy::Skip

Test 2.9.3: Deterministic Calculation

Test Pattern:

func TestConfidenceCalculator_Deterministic(t *testing.T) {
    calc := confidence.NewCalculator()
    quote := createTestQuote()
    oracle := createTestOracle()

    // Calculate 100 times
    scores := make([]float64, 100)
    for i := 0; i < 100; i++ {
        scores[i] = calc.Calculate(quote, oracle)
    }

    // All scores must be identical
    for i := 1; i < 100; i++ {
        assert.Equal(t, scores[0], scores[i],
            "Confidence calculation is not deterministic")
    }
}

Assertions:

✅ All 100 calculations produce identical score
✅ No randomness or time-dependent factors
✅ Score is pure function of inputs

Test 2.9.4: Scanner Decision Thresholds

Threshold Mapping:

Confidence ≥0.9 → Strategy::Execute
Confidence 0.7-0.9 → Strategy::Verify
Confidence 0.5-0.7 → Strategy::Cautious
Confidence <0.5 → Strategy::Skip

Code Coverage Target: >95% for ConfidenceCalculator

2.10 1-Second AMM Refresh Tests (Review Enhancement) ⭐ PERFORMANCE

Priority: P1 - QUICK WIN VALIDATION Estimated Effort: 3 hours Review Source: Gemini critique Design Doc: 30.3-REFRESH-RATE-ANALYSIS.md

Purpose

Validate that AMM pools refresh every 1 second (not 10s) and measure opportunity capture rate improvement.

Test Cases

Test 2.10.1: Refresh Frequency Validation

Test Pattern:

func TestAMMRefreshFrequency(t *testing.T) {
    manager := refresh.NewManager(1 * time.Second) // 1s interval

    refreshEvents := make(chan time.Time, 20)
    manager.OnRefresh(func(poolID string, timestamp time.Time) {
        refreshEvents <- timestamp
    })

    manager.Start()
    time.Sleep(10 * time.Second)
    manager.Stop()

    // Should have ~10 refresh events (±1 for timing jitter)
    assert.InDelta(t, 10, len(refreshEvents), 1,
        "Expected 10 refreshes in 10 seconds")

    // Validate intervals between refreshes
    timestamps := drainChannel(refreshEvents)
    for i := 1; i < len(timestamps); i++ {
        interval := timestamps[i].Sub(timestamps[i-1])
        assert.InDelta(t, 1000, interval.Milliseconds(), 100,
            "Refresh interval should be 1s ±100ms")
    }
}

Assertions:

✅ 10 refresh cycles in 10 seconds (±1)
✅ Each interval is 1s ±100ms
✅ No missed refreshes

Test 2.10.2: Opportunity Capture Rate Improvement

Expected Results:

Baseline (10s refresh): ~90% capture rate
Enhanced (1s refresh): >95% capture rate
Improvement: >5%

Test 2.10.3: Redis Load Impact

Validation:

Load increases ~10× (expected: 10s → 1s)
Absolute increase <50 reads/sec
Redis CPU usage <5% increase

Code Coverage Target: >85% for RefreshManager

2.11 Parallel Paired Quote Tests (Review Enhancement) ⭐ CRITICAL

Priority: P0 - CORRECTNESS Estimated Effort: 4 hours Review Source: ChatGPT praise #1 (Exceptional feature) Design Doc: 30-QUOTE-SERVICE-ARCHITECTURE.md Section 5.3

Purpose

Validate that parallel paired quote calculation (forward + reverse) uses the same pool snapshot and eliminates fake arbitrage from slot drift.

Test Cases

Test 2.11.1: Same Pool Snapshot for Forward + Reverse

Test Pattern:

func TestPairedQuotes_SamePoolSnapshot(t *testing.T) {
    service := local_quote.NewService()

    // Get paired quotes
    paired, err := service.CalculatePairedQuotes(
        SOL_MINT, USDC_MINT, 1_000_000_000,
    )
    assert.NoError(t, err)

    // ✅ Both quotes must reference same pool ID
    assert.Equal(t, paired.Forward.PoolID, paired.Reverse.PoolID,
        "Forward and reverse must use same pool")

    // ✅ Both quotes must have same pool state timestamp
    assert.Equal(t, paired.Forward.PoolStateAge, paired.Reverse.PoolStateAge,
        "Pool state age must be identical")

    // ✅ Verify arbitrage consistency (no slot drift)
    initialSOL := 1_000_000_000  // 1 SOL
    finalSOL := (paired.Forward.OutputAmount * initialSOL) / paired.Reverse.OutputAmount

    profit := float64(finalSOL - initialSOL) / float64(initialSOL)

    // Real arbitrage should be consistent
    if profit > 0.001 { // >0.1% profit
        assert.True(t, validateArbitrageWithPoolState(paired),
            "Arbitrage profit must be supported by pool state")
    }
}

Assertions:

✅ Same pool ID for both quotes
✅ Same pool state timestamp
✅ No fake arbitrage from slot drift

Test 2.11.2: Parallel Execution Performance (2× Speedup)

Expected Speedup: 1.5-2.5× faster than sequential Latency: Sequential >100ms, Parallel <60ms

Code Coverage Target: >90% for CalculatePairedQuotes()

2.12 Explicit Timeout Tests (Review Enhancement) ⭐ CRITICAL

Priority: P0 - TAIL LATENCY Estimated Effort: 3 hours Review Source: ChatGPT critique #2 Design Doc: 30.4-CHATGPT-REVIEW-RESPONSE.md

Purpose

Validate that aggregator enforces explicit timeouts (local: 10ms, external: 100ms) and emits local-only results immediately.

Test Cases

Test 2.12.1: Local Quote Timeout Enforcement

Test Pattern:

func TestAggregator_LocalTimeoutEnforcement(t *testing.T) {
    // Mock slow local service (20ms)
    slowLocal := &MockLocalService{Delay: 20 * time.Millisecond}
    aggregator := NewAggregator(slowLocal, fastExternal)

    start := time.Now()
    quote, err := aggregator.GetQuote(ctx, request)
    elapsed := time.Since(start)

    // Should timeout after 10ms
    assert.Error(t, err, "Expected timeout error")
    assert.Contains(t, err.Error(), "timeout")
    assert.InDelta(t, 10, elapsed.Milliseconds(), 5,
        "Should timeout at 10ms ±5ms")
}

Assertions:

✅ Timeout triggers at 10ms ±5ms
✅ Error message indicates timeout
✅ External quote not affected

Test 2.12.2: Non-Blocking Local-Only Emit

Scenario: Local fast (5ms), external slow (500ms), should emit local immediately

Expected Results:

First emit: <15ms (local-only)
Second emit: ~500ms (with external)
Local never blocks on external

Code Coverage Target: >90% for timeout logic

2.13 Test Summary: All Enhancements

Enhancement	Test Hours	Priority	Coverage Target	Source
Shared Memory IPC	12 hours	P0	95%	Architecture
Route Storage	8 hours	P1	90%	Architecture
Quote Pre-Computation	6 hours	P1	85%	Architecture
Parallel Quote Calculation	4 hours	P1	90%	Architecture
Redis Quote Persistence	8 hours	P0	95%	Architecture
Quote Validation Layer	8 hours	P1	90%	Architecture
Circuit Breaker Per-Quoter	6 hours	P1	90%	Architecture
Quote Versioning	4 hours	P1	85%	Architecture
End-to-End Integration	4 hours	P0	100%	Architecture
Torn Read Prevention ⭐	4 hours	P0	>90%	ChatGPT
Confidence Scoring ⭐	4 hours	P0	>95%	ChatGPT
1s AMM Refresh ⭐	3 hours	P1	>85%	Gemini
Parallel Paired Quotes ⭐	4 hours	P0	>90%	ChatGPT
Explicit Timeouts ⭐	3 hours	P0	>90%	ChatGPT
TOTAL	78 hours	Mixed	>91% avg	-

Review Enhancement Impact:

+18 hours testing effort
+5 critical test categories
Addresses 100% of ChatGPT/Gemini critiques
Ensures correctness (torn reads), performance (1s refresh), and HFT readiness (confidence scoring)

3. Test Environments

3.1 Local Development

Setup:

Go service: go run ./cmd/quote-service
Dependencies: Docker Compose (Redis, NATS, Prometheus)
RPC: Solana Devnet or mock RPC

Purpose:

Unit tests
Integration tests (with mocks)
Quick iteration

3.2 Staging

Setup:

Kubernetes cluster (Minikube or k3s)
Real Solana Mainnet RPC (rate-limited)
Full observability stack (Grafana, Prometheus, Jaeger)

Purpose:

Load tests (limited scale)
End-to-end tests
Pre-production validation

3.3 Production-Like (Bare Metal)

Setup:

Bare metal server (16 cores, 64GB RAM)
Multiple Solana RPC endpoints (7+)
Full production configuration

Purpose:

Performance benchmarks
Soak tests
Capacity planning

4. Test Data

4.1 Test Token Pairs

High-Volume Pairs (Hot Tier):

SOL/USDC
SOL/USDT
BONK/SOL
JUP/USDC

Medium-Volume Pairs (Warm Tier):

RAY/SOL
ORCA/USDC
MNGO/SOL

Low-Volume Pairs (Cold Tier):

Long-tail tokens from Meteora

4.2 Test Amounts

Small: 0.1 SOL ($15)
Medium: 1 SOL ($150)
Large: 10 SOL ($1,500)
Whale: 100 SOL ($15,000)

4.3 Mock Data

Mock RPC Responses:

Pool account data (binary encoded)
Oracle prices (Pyth format)
Blockhash responses

Mock Failure Scenarios:

RPC timeout
Invalid pool state
Circuit breaker open

5. Test Execution

5.1 Continuous Integration (CI)

GitHub Actions Workflow:

name: Quote Service Tests

on: [push, pull_request]

jobs:
  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-go@v4
        with:
          go-version: '1.24'
      - name: Run unit tests
        run: |
          cd go
          go test ./internal/quote-service/... -v -race -coverprofile=coverage.out
      - name: Upload coverage
        uses: codecov/codecov-action@v3

  integration-tests:
    runs-on: ubuntu-latest
    services:
      redis:
        image: redis:7
      nats:
        image: nats:latest
    steps:
      - name: Run integration tests
        run: |
          cd go
          go test ./internal/quote-service/... -tags=integration -v

  load-tests:
    runs-on: ubuntu-latest
    if: github.event_name == 'pull_request'
    steps:
      - name: Start quote service
        run: docker-compose up -d quote-service
      - name: Run k6 load tests
        run: k6 run tests/load/sustained-load.js

CI Triggers:

Every commit: Unit tests
Pull requests: Unit + Integration tests
Main branch: Full test suite + load tests
Nightly: Soak tests (4 hours)

5.2 Manual Testing

Regression Testing (Before Release):

Run full test suite locally
Deploy to staging
Run load tests for 1 hour
Verify metrics in Grafana
Check logs for errors/warnings

Exploratory Testing:

Test new features manually
Verify edge cases not covered by automated tests
User experience testing (API usability)

5.3 Performance Testing Schedule

Weekly:

Sustained load test (10 min)
Peak load test (5 min)

Monthly:

Soak test (4 hours)
Capacity planning benchmarks

Before Major Release:

Full performance test suite
Latency SLA validation
Load test at 2x expected capacity

6. Success Criteria

6.1 Code Coverage

Overall: >80%
Critical paths (calculator, cache): >90%
Handlers: >85%
Utilities: >70%

6.2 Performance SLAs

Metric	Target	Measured By
p50 latency	< 5ms	Prometheus histogram
p95 latency	< 10ms	Prometheus histogram
p99 latency	< 50ms	Prometheus histogram
p99.9 latency	< 200ms	Prometheus histogram
Throughput	1000 req/s sustained	k6 load test
Peak throughput	5000 req/s	k6 spike test
Cache hit rate	> 80%	Redis metrics
Error rate	< 0.1%	Prometheus counter
Availability	99.9%	Uptime monitoring

6.3 Functional Requirements

✅ All DEX protocols return valid quotes
✅ Price sanity checks prevent bad trades
✅ Circuit breaker prevents cascading failures
✅ Rate limiting protects against abuse
✅ Graceful degradation during failures
✅ Observability: metrics, logs, traces

7. Test Automation

7.1 Test Frameworks

Go Testing:

Standard library: testing package
Assertions: github.com/stretchr/testify
Mocking: github.com/golang/mock
HTTP testing: net/http/httptest

Load Testing:

k6 (Grafana k6)
Artillery (optional alternative)

Observability:

Prometheus test queries
Grafana test dashboards
Jaeger trace validation

7.2 Test Utilities

Mock RPC Client:

type MockRPCClient struct {
    mock.Mock
}

func (m *MockRPCClient) GetAccountInfo(ctx context.Context, pubkey string) (*rpc.AccountInfo, error) {
    args := m.Called(ctx, pubkey)
    return args.Get(0).(*rpc.AccountInfo), args.Error(1)
}

Test Data Generators:

func GenerateTestQuoteRequest() *QuoteRequest {
    return &QuoteRequest{
        InputMint:  testutil.SOL_MINT,
        OutputMint: testutil.USDC_MINT,
        Amount:     rand.Uint64() % 10_000_000_000,
    }
}

Benchmark Helpers:

func BenchmarkWithMetrics(b *testing.B, fn func()) {
    start := time.Now()
    for i := 0; i < b.N; i++ {
        fn()
    }
    elapsed := time.Since(start)

    b.ReportMetric(float64(elapsed.Nanoseconds())/float64(b.N), "ns/op")
}

8. Bug Tracking & Reporting

8.1 Bug Severity Levels

P0 - Critical (Fix within 4 hours):

Service completely down
Data corruption
Security vulnerability

P1 - High (Fix within 24 hours):

SLA breach (p95 > 10ms sustained)
Feature broken for >10% of users
Major performance degradation

P2 - Medium (Fix within 1 week):

Non-critical feature broken
Minor performance issue
Error rate elevated but <1%

P3 - Low (Fix in next sprint):

Cosmetic issues
Nice-to-have improvements
Edge case bugs

8.2 Bug Report Template

## Bug Report

**Title**: [Short description]
**Severity**: P0/P1/P2/P3
**Environment**: Local/Staging/Production

### Description
[Detailed description of the bug]

### Steps to Reproduce
1. Step 1
2. Step 2
3. Step 3

### Expected Behavior
[What should happen]

### Actual Behavior
[What actually happens]

### Logs/Screenshots
[Paste relevant logs or attach screenshots]

### System Info
- Go version: 1.24
- OS: Windows 11
- Quote Service version: v1.2.3

9. Test Maintenance

9.1 Test Review Schedule

Weekly:

Review test failures in CI
Update test data if pools change

Monthly:

Review code coverage trends
Identify untested code paths
Update performance benchmarks

Quarterly:

Review test strategy
Retire obsolete tests
Add tests for new features

9.2 Test Documentation

Required Documentation:

Test plan (this document)
Test case specifications
Performance benchmark baselines
Known issues & workarounds

Update Triggers:

New feature added → Add tests
Bug fixed → Add regression test
Performance SLA changed → Update benchmarks

10. Appendix

10.1 Useful Commands

Run All Tests:

cd go
go test ./internal/quote-service/... -v -race

Run Specific Test:

go test ./internal/quote-service/calculator -run TestQuoteCalculator_SingleProtocol -v

Run with Coverage:

go test ./internal/quote-service/... -coverprofile=coverage.out
go tool cover -html=coverage.out

Run Benchmarks:

go test ./internal/quote-service/calculator -bench=. -benchmem

Run Load Tests:

cd tests/load
k6 run sustained-load.js

10.2 Troubleshooting

Test Failure: “Redis connection refused”

Solution: Start Redis with docker-compose up -d redis

Test Failure: “RPC timeout”

Solution: Use mock RPC client or increase timeout

Load Test Failure: “Too many open files”

Solution: Increase ulimit: ulimit -n 10000

Flaky Test

Solution: Add retries with exponential backoff, or use t.Skip() if environmental

End of Quote Service Test Plan v1.0

Share on

Bluesky Facebook LinkedIn Mastodon X (formerly Twitter)

James Shen

Quote Service Test Plan

1. Executive Summary

2. Test Categories

2.1 Unit Tests (40 hours)

2.1.1 Quote Calculator (internal/quote-service/calculator/)

2.1.2 Adaptive Refresh Manager (internal/quote-service/cache/)

2.1.3 Circuit Breaker (internal/quote-service/resilience/)

2.1.4 Slippage Calculator & Tracker

2.2 Integration Tests (60 hours)

2.2.1 HTTP REST API (internal/quote-service/api/http/)

2.2.2 gRPC Streaming API (internal/quote-service/api/grpc/)

2.2.3 Redis Cache Integration

2.2.4 NATS Event Publishing

2.3 Load Tests (40 hours)

2.3.1 Sustained Load Test

2.3.2 Peak Load Test

2.3.3 Spike Test

2.3.4 Soak Test

2.4 Performance Tests (30 hours)

2.4.1 Latency SLA Compliance

2.4.2 Cache Performance

2.4.3 Quote Calculation Benchmarks

2.4.4 Concurrent Connection Handling

2.5 HFT-Specific Tests (50 hours)

2.5.1 Quote Staleness Detection

2.5.2 Price Sanity Checks

2.5.3 Circuit Breaker Integration

2.5.4 Rate Limiting Per Client

2.5.5 Failover & Graceful Degradation

2.6 Functional Tests (20 hours)

2.6.1 Multi-Protocol Support

2.6.2 Error Handling

2.6.3 Health Check Endpoints

2.7 Critical Enhancement Tests (60 hours) 🆕

2.7.1 Shared Memory IPC Tests (P0 - HFT Critical)

2.7.2 Route Storage Tests (P1 - Correctness)

2.7.3 Quote Pre-Computation Tests (P1 - Performance)

2.7.4 Parallel Quote Calculation Tests (P1 - Performance)

2.7.5 Redis Quote Persistence Tests (P0 - Reliability)

2.7.6 Quote Validation Tests (P1 - Correctness)

2.7.7 Circuit Breaker Per-Quoter Tests (P1 - Reliability)

2.7.8 Quote Versioning Tests (P1 - Correctness)

2.7.9 Integration Tests: End-to-End HFT Pipeline

2.7.10 Test Summary: Critical Enhancements

2.8 Torn Read Prevention Tests (Review Enhancement) ⭐ CRITICAL

Purpose

Test Cases

2.9 Confidence Score Validation Tests (Review Enhancement) ⭐ CRITICAL

Purpose

Test Cases

2.10 1-Second AMM Refresh Tests (Review Enhancement) ⭐ PERFORMANCE

Purpose

Test Cases

2.11 Parallel Paired Quote Tests (Review Enhancement) ⭐ CRITICAL

Purpose

Test Cases

2.12 Explicit Timeout Tests (Review Enhancement) ⭐ CRITICAL

Purpose

Test Cases

2.13 Test Summary: All Enhancements

3. Test Environments

3.1 Local Development

3.2 Staging

3.3 Production-Like (Bare Metal)

4. Test Data

4.1 Test Token Pairs

4.2 Test Amounts

4.3 Mock Data

5. Test Execution

5.1 Continuous Integration (CI)

5.2 Manual Testing

5.3 Performance Testing Schedule

6. Success Criteria

6.1 Code Coverage

6.2 Performance SLAs

6.3 Functional Requirements

7. Test Automation

7.1 Test Frameworks

7.2 Test Utilities

2.1.1 Quote Calculator (`internal/quote-service/calculator/`)

2.1.2 Adaptive Refresh Manager (`internal/quote-service/cache/`)

2.1.3 Circuit Breaker (`internal/quote-service/resilience/`)

2.2.1 HTTP REST API (`internal/quote-service/api/http/`)

2.2.2 gRPC Streaming API (`internal/quote-service/api/grpc/`)