Grafana Dashboards & Metrics for Solana HFT Trading System

Version: 1.0 Last Updated: 2025-12-20 Status: Production Ready ✅

Table of Contents

1. Overview
2. System Architecture
3. Metrics Implementation Status
4. Service Metrics Endpoints
5. Prometheus Configuration
6. Dashboard Reference
7. Key Metrics by Category
8. Alerting Guidelines
9. Troubleshooting
10. Future Enhancements

Overview

The Solana HFT Trading System uses LGTM+ Stack (Loki, Grafana, Tempo, Mimir, Prometheus) for comprehensive observability across the Scanner → Planner → Executor pipeline with FlatBuffers event-driven architecture.

Design Principles

1. Sub-500ms Latency Tracking: All metrics support the goal of < 500ms (< 200ms ideal) execution
2. Event-Driven Architecture: Full visibility into 6 NATS JetStream streams
3. Business Metrics First: P&L, win rate, and trade statistics are primary KPIs
4. Pipeline Observability: End-to-end tracking from opportunity detection to execution
5. Production-Grade: Following Prometheus best practices with proper labeling and cardinality control

Performance Targets

System Architecture

Service Topology

┌─────────────────────────────────────────────────────────────┐
│                    LGTM+ OBSERVABILITY STACK                │
│  Loki (Logs) + Grafana (Viz) + Tempo (Traces) +           │
│  Mimir (Metrics) + Prometheus (Scraping)                   │
└─────────────────────────────────────────────────────────────┘
                              ▲
                              │ Metrics, Logs, Traces
                              │
┌─────────────────────────────────────────────────────────────┐
│                      HFT PIPELINE SERVICES                   │
├─────────────────────────────────────────────────────────────┤
│  Scanner (9096) → Planner (9097) → Executor (9098)         │
│           ↓ OPPORTUNITIES    ↓ PLANNED     ↓ EXECUTED      │
└─────────────────────────────────────────────────────────────┘
                              │
                    ┌─────────┴─────────┐
                    ▼                   ▼
        ┌───────────────────┐  ┌──────────────────┐
        │  System Manager   │  │  System Auditor  │
        │  (Kill Switch)    │  │  (P&L Tracking)  │
        │      9099         │  │      9100        │
        └───────────────────┘  └──────────────────┘
                    │                   │
                    └─────────┬─────────┘
                              ▼
                    ┌──────────────────┐
                    │  NATS JetStream  │
                    │   6 Streams      │
                    └──────────────────┘

NATS JetStream Streams

Metrics Implementation Status

✅ Implemented (Production Ready)

Go Services

Quote Service (Port 8080):

• ✅ RPC Performance (requests, duration, errors, connection pool)
• ✅ Pool Query & Calculation (protocol breakdown, selection time)
• ✅ Cache Lifecycle (refresh duration, size, entries)
• ✅ Health Checks (service, cache, router, RPC pool)
• ✅ SLA & Latency Breakdown (validation, cache, calculation, serialization)

Event Logger Service (Port 9093):

• ⚠️ Needs Implementation: No metrics endpoint currently
• 📝 Action Required: Add Prometheus instrumentation

TypeScript Services

Scanner Service (Port 9096):

• ✅ Service Info & Uptime
• ✅ NATS & gRPC Connection Health
• ✅ Quote Processing (received, latency)
• ✅ Arbitrage Detection (detected, published, rejected)
• ✅ Profit Tracking (basis points)
• ✅ Event Processing (duration, queue size)
• ✅ Error Tracking

Strategy Service (Port 9097):

• ✅ Opportunities Received
• ✅ Execution Plans (created, rejected, published)
• ✅ Rejection Reasons
• ✅ Risk Score Distribution
• ✅ Validation Metrics
• ✅ NATS Connection Health

Executor Service (Port 9098):

• ✅ Execution Plans Received
• ✅ Execution Status (started, succeeded, failed)
• ✅ Execution Duration (P50, P95, P99)
• ✅ Success Rate
• ✅ In-Flight Executions
• ✅ Actual Profit & Gas Costs
• ✅ Jito Tip Tracking

System Manager (Port 9099):

• ✅ Metrics Stream Monitoring
• ✅ System Health Tracking
• ✅ Kill Switch Triggers
• ✅ Error Rate & Latency Monitoring
• ✅ P&L Threshold Tracking

System Auditor (Port 9100):

• ✅ Execution Results Processing
• ✅ P&L Calculation (realized, unrealized)
• ✅ Trade Statistics (total, winning, losing)
• ✅ Win Rate Calculation
• ✅ Profit Distribution (average, max, min)

System Initializer (Port 9091):

• ✅ Stream Creation Status
• ✅ Consumer Creation Status
• ✅ NATS Setup Health

Notification Service (Port 9092):

• ✅ Email Notifications Sent
• ✅ Alert Triggers
• ✅ Notification Queue

📋 Planned (Future Enhancements)

Week 2-3 (High Priority)

1. Concurrent Operation Metrics:
   • Goroutine counts (Go services)
   • Queue depth tracking
   • Thread pool utilization
2. Infrastructure Metrics:
   • CPU usage per service
   • Memory usage & GC stats
   • Network I/O (bytes in/out)
   • File descriptors
3. WebSocket Metrics (for Shredstream integration):
   • Subscription health
   • Message lag
   • Reconnection events
4. Distributed Tracing Enhancements:
   • Custom span attributes
   • Cross-service trace correlation

Month 2+ (Medium Priority)

1. Business Logic Metrics:
   • Quote quality scores
   • Spread analysis
   • Liquidity depth tracking
   • Slippage analysis
2. User Behavior Metrics:
   • Request patterns
   • API usage statistics
   • Rate limiting effectiveness
3. Error Context Metrics:
   • Error categories
   • Root cause analysis
   • Recovery time tracking

Service Metrics Endpoints

All services expose Prometheus metrics at /metrics endpoint:

Testing Metrics Endpoints

# Test all TypeScript services
curl http://localhost:9096/metrics | grep "service_info"  # Scanner
curl http://localhost:9097/metrics | grep "service_info"  # Strategy
curl http://localhost:9098/metrics | grep "service_info"  # Executor
curl http://localhost:9099/metrics | grep "service_info"  # Manager
curl http://localhost:9100/metrics | grep "service_info"  # Auditor

# Test Go services
curl http://localhost:8080/metrics | grep "quote_service"  # Quote Service
curl http://localhost:9093/metrics  # Event Logger (should return empty - not implemented)

Prometheus Configuration

Location: deployment/monitoring/prometheus/prometheus.yml

Global Configuration

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: 'solana-trading-system'
    environment: 'local'

Remote Write (Long-term Storage)

remote_write:
  - url: http://mimir:9009/api/v1/push
    queue_config:
      capacity: 10000
      max_shards: 5
      batch_send_deadline: 5s

Service Scrape Jobs

Each service has enhanced labels for better organization:

- job_name: 'ts-strategy-service'
  metrics_path: /metrics
  static_configs:
    - targets: ['ts-strategy-service:9097']
      labels:
        service: 'ts-strategy-service'
        environment: 'production'
        language: 'typescript'
        pipeline_stage: 'planner'

Label Schema

Dashboard Reference

Grafana: http://localhost:3000 Login: admin / (configured in .env)

Dashboard 0: Quote Service Week 2 Performance 🚀

File: deployment/monitoring/grafana/provisioning/dashboards/quote-service-week2-performance.json Purpose: Monitor P0-P1 performance optimizations for quote-service (Week 2 focus) Status: ✅ Deployed (December 22, 2025)

Context: This dashboard was specifically created to monitor the Phase 1 performance optimizations implemented in Week 1-2:

• Lock-free cache with sync.Map
• GOGC=50 memory optimization
• Circuit breaker & request hedging
• Adaptive refresh management

Deployment Status

✅ Dashboard is deployed to deployment/monitoring/grafana/provisioning/dashboards/quote-service-week2-performance.json

Auto-provisioning: Grafana will automatically load this dashboard on startup (refresh interval: 10 seconds)

Manual import (alternative):

# If you need to import manually via UI:
# Grafana → Dashboards → Import → Upload JSON file
# → Select datasource: Prometheus
# → Click "Import"

Restart Grafana to load:

cd deployment/docker
docker-compose restart grafana

Panels (10 Total)

Panel 1: Cache Latency (Optimized) - ⚡ Critical

• Metric: Cache read latency with lock-free sync.Map
• Queries:

  # p50
  histogram_quantile(0.50, rate(cache_get_duration_seconds_bucket{cache_hit="true"}[5m]))
    
  # p95
  histogram_quantile(0.95, rate(cache_get_duration_seconds_bucket{cache_hit="true"}[5m]))
    
  # p99
  histogram_quantile(0.99, rate(cache_get_duration_seconds_bucket{cache_hit="true"}[5m]))
  

• Visual: Line graph with 3 series (p50, p95, p99)
• Unit: seconds (milliseconds display)
• Thresholds:
  • Green: <3ms (target achieved)
  • Yellow: 3-5ms
  • Red: >5ms (alert triggered)
• Alert Condition: p99 > 5ms for 2 minutes
• Target: <3ms p99 ✅ (achieved in Week 1)

Panel 2: GC Pause Duration (GOGC=50) - ⚡ Critical

• Metric: Go garbage collection pause time with GOGC=50 optimization
• Queries:

  histogram_quantile(0.50, rate(gc_pause_duration_seconds_bucket[5m]))
  histogram_quantile(0.95, rate(gc_pause_duration_seconds_bucket[5m]))
  histogram_quantile(0.99, rate(gc_pause_duration_seconds_bucket[5m]))
  

• Visual: Line graph with 3 series
• Unit: seconds (milliseconds display)
• Thresholds:
  • Green: <1ms (target achieved)
  • Yellow: 1-2ms
  • Red: >2ms (alert threshold)
• Target: <1ms p99 ✅ (achieved in Week 1)
• GOGC Setting: 50 (more frequent, shorter GC cycles)

Panel 3: Memory Heap Allocation - 📊 Monitoring

• Metric: Go heap memory allocation tracking
• Queries:

  memory_heap_alloc_bytes / (1024 * 1024)    # Allocated heap (MB)
  memory_heap_inuse_bytes / (1024 * 1024)    # In-use heap (MB)
  memory_heap_sys_bytes / (1024 * 1024)      # System heap (MB)
  

• Visual: Stacked area chart
• Unit: Megabytes (MB)
• Expected Values:
  • Heap Alloc: ~200MB (steady state)
  • Heap In-Use: ~150-200MB
  • Heap Sys: ~250-300MB (Go reserves)
• Alert: > 1.5GB heap (potential memory leak)

Panel 4: Cache Hit Rate - 🎯 Performance

• Metric: Percentage of cache hits vs misses
• Query:

  rate(cache_hits_total[5m]) / (rate(cache_hits_total[5m]) + rate(cache_misses_total[5m]))
  

• Visual: Gauge with percentage (0-100%)
• Unit: Percentage
• Thresholds:
  • Green: >90% (optimal)
  • Yellow: 80-90% (acceptable)
  • Red: <80% (needs investigation)
• Target: >95% hit rate
• Alert: < 80% for 10 minutes

Panel 5: Circuit Breaker State - 🚨 Reliability

• Metric: RPC pool circuit breaker status
• Query:

  circuit_breaker_state{circuit="rpc-pool"}
  

• Visual: Stat panel with color-coded background
• Mappings:
  • 0 = CLOSED (green) - Normal operation ✅
  • 1 = OPEN (red) - Circuit tripped, failing fast 🚨
  • 2 = HALF-OPEN (yellow) - Testing recovery 🔄
• Purpose: Prevent cascade failures when RPC pool degrades
• Auto-recovery: OPEN → HALF-OPEN after 30s timeout

Panel 6: Throughput (Requests/sec) - 📈 Capacity

• Metric: Quote requests per second
• Query:

  rate(quote_requests_total[1m])
  

• Visual: Line graph
• Unit: Requests per second (req/s)
• Expected:
  • Normal load: 10-50 req/s
  • Peak load: 100-500 req/s
  • Max capacity: 1000+ req/s ✅ (achieved)
• Target: 500K ops/sec with optimized cache

Panel 7: GC Cycles Per Minute - 🔄 Memory Management

• Metric: Garbage collection frequency
• Query:

  rate(gc_cycles_total[1m]) * 60
  

• Visual: Line graph
• Unit: GC cycles per minute
• Expected with GOGC=50:
  • More frequent cycles (vs default GOGC=100)
  • Shorter pause duration (trade-off)
  • Typical: 30-60 cycles/min
• Interpretation: Higher frequency = lower pause time ✅

Panel 8: Refresh Tier Distribution - 📊 Adaptive Management

• Metric: Number of pairs in each refresh tier (Hot/Warm/Cold)
• Query:

  sum by (tier) (refresh_tier)
  

• Visual: Pie chart (donut chart)
• Tiers:
  • 🔥 Hot (5s refresh): High-volume pairs (SOL/USDC, active LSTs)
  • 🌡️ Warm (15s refresh): Moderate activity
  • ❄️ Cold (60s refresh): Low-volume pairs
• Expected Distribution:
  • Hot: 10-20% of pairs (critical pairs)
  • Warm: 30-40% of pairs (moderate)
  • Cold: 40-60% of pairs (background)
• Dynamic: Pairs auto-promote/demote based on activity

Panel 9: Quote Success Rate - ✅ Reliability

• Metric: Percentage of successful quote requests
• Query:

  (rate(quote_requests_total[5m]) - rate(quote_errors_total[5m])) / rate(quote_requests_total[5m])
  

• Visual: Line graph
• Unit: Percentage (0-100%)
• Thresholds:
  • Green: >99.9% (target) ✅
  • Yellow: 95-99.9%
  • Red: <95% (unacceptable)
• Y-axis: Zoomed to 99-100% for detail
• Target: 99.99% success rate ✅ (achieved with circuit breaker + hedging)

Panel 10: Hedged Requests - 🏃 Failover

• Metric: Request hedging trigger rate
• Query:

  rate(hedged_request_triggered_total[5m])
  

• Visual: Stat panel with area graph
• Unit: Hedges per second
• Purpose: Requests sent to 2 endpoints, use first response
• Trigger: After 500ms without response from primary
• Expected:
  • Normal: 0-1% of requests
  • Degraded RPC: 5-10% of requests
  • Alert: >5% sustained (indicates RPC issues)

Performance Validation

Success Criteria (Week 2 Monitoring):

Dashboard Refresh: 10 seconds (real-time monitoring)

Time Range: Default last 1 hour (adjustable)

Annotations: Service restarts, config changes, alerts

Usage Scenarios

Scenario 1: Normal Operations

• All panels green ✅
• Cache p99 <3ms
• GC pause p99 <1ms
• Circuit breaker CLOSED
• Cache hit rate >90%

Scenario 2: High Load

• Throughput spike to 200-500 req/s
• Cache hit rate remains >90%
• GC frequency increases (acceptable)
• Hedged requests may increase (5-10%)
• Circuit breaker remains CLOSED

Scenario 3: RPC Degradation

• Circuit breaker → OPEN 🚨
• Hedged requests spike (10-20%)
• Success rate may dip (95-99%)
• Dashboard turns yellow/red
• Action: Check RPC pool health

Scenario 4: Memory Pressure

• Heap allocation >1GB
• GC pause increases to 2-5ms
• GC frequency spikes (>100/min)
• Action: Investigate memory leak, restart service

Scenario 5: Post-Deployment

• Monitor for 1-2 hours after restart
• Verify Redis cache restored (<3s)
• Check all optimizations active
• Compare metrics vs baseline

Related Alerts

The dashboard includes built-in alert configurations (see Panel 1 alert example). Full alerting rules in prometheus-alerts.yml:

P0 Alerts (PagerDuty/SMS):

• CacheLatencyHigh: p99 > 5ms for 2min
• CircuitBreakerOpen: Circuit open for 30s
• QuoteSuccessRateLow: <95% for 5min

P1 Alerts (Slack/Discord):

• GCPauseHigh: p99 > 2ms for 5min
• CacheHitRateLow: <80% for 10min
• HedgeRateHigh: >5% for 5min

See: go/cmd/quote-service/prometheus-alerts.yml for full alert definitions

Dashboard 1: System Overview 🏠

File: system-overview-updated.json Purpose: Main dashboard with high-level system status and KPIs

Panels

Row 1: System Status

• All Services Status: Real-time UP/DOWN status for all 9 services
  • Visual: Stat panel with ✓ UP / ✗ DOWN indicators
  • Threshold: Green (UP), Red (DOWN)

Row 2: HFT Pipeline Metrics

• Scanner: Opportunities Detected: Rate of opportunities/sec
  • Query: rate(opportunities_detected_total{service="ts-scanner-service"}[5m])
  • Visual: Stat with area graph
• Planner: Plans Created: Rate of plans/sec
  • Query: rate(execution_plans_created_total{service="ts-strategy-service"}[5m])
  • Visual: Stat with area graph
• Executor: Trades Executed: Rate of successful executions/sec
  • Query: rate(executions_succeeded_total{service="ts-executor-service"}[5m])
  • Visual: Stat with area graph
• Auditor: Total Trades: Cumulative trade count
  • Query: total_trades{service="system-auditor"}
  • Visual: Stat with counter

Row 3: Trading Performance

• Realized P&L (USD): Time series of cumulative P&L
  • Query: realized_pnl_usd{service="system-auditor"}
  • Visual: Line chart with area fill
  • Thresholds: Green (profit), Red (loss)
• Win Rate: Percentage of winning trades
  • Query: (winning_trades / total_trades) * 100
  • Visual: Gauge (0-100%)
  • Thresholds: Green (>70%), Yellow (50-70%), Red (<50%)
• Trade Statistics: Total/Winning/Losing breakdown
  • Queries: total_trades, winning_trades, losing_trades
  • Visual: Stat panel with multiple values

Row 4: Performance Metrics

• End-to-End Latency (P95): Pipeline execution time
  • Query: histogram_quantile(0.95, sum(rate(execution_duration_ms_bucket[5m])) by (le))
  • Visual: Gauge (0-1000ms)
  • Thresholds: Green (<200ms), Yellow (200-500ms), Red (>500ms)
• Success Rate: Execution success percentage
  • Query: rate(executions_succeeded_total[5m]) / (rate(executions_succeeded_total[5m]) + rate(executions_failed_total[5m])) * 100
  • Visual: Gauge (0-100%)
  • Thresholds: Green (>90%), Yellow (70-90%), Red (<70%)
• System Health Score: Percentage of services up
  • Query: (sum(up{job=~".*service.*"}) / count(up{job=~".*service.*"})) * 100
  • Visual: Gauge (0-100%)
  • Thresholds: Green (>90%), Yellow (80-90%), Red (<80%)

Row 5: Infrastructure Status

• NATS JetStream: Connection health
• Prometheus: Scraping health
• Loki: Log aggregation status
• OpenTelemetry Collector: Trace collection status

Row 6: Recent Alerts

• Kill Switch Events (Last 24h): Count of kill switch triggers
  • Query: sum(increase(kill_switches_triggered_total[24h]))
  • Thresholds: Green (0), Yellow (1-4), Red (>5)
• Error Rate (All Services): Errors/sec by service
  • Query: sum(rate(errors_total[5m])) by (service)
  • Visual: Time series

Quick Links

• 🚀 HFT Pipeline Dashboard
• 📦 FlatBuffers Streams Dashboard
• 💊 System Health Dashboard
• 💱 Quote Service Dashboard
• 📡 Scanner Service Dashboard

Dashboard 2: HFT Pipeline Performance 🚀

File: hft-pipeline-performance.json Purpose: Detailed Scanner → Planner → Executor pipeline monitoring

Panels

Row 1: Pipeline Overview

• End-to-End Pipeline Latency: P50, P95, P99 execution times
  • Queries:

    histogram_quantile(0.50, sum(rate(execution_duration_ms_bucket[5m])) by (le))
    histogram_quantile(0.95, sum(rate(execution_duration_ms_bucket[5m])) by (le))
    histogram_quantile(0.99, sum(rate(execution_duration_ms_bucket[5m])) by (le))
    

  • Visual: Line graph
  • Thresholds: Green (<200ms), Yellow (200-500ms), Red (>500ms)
  • Target: < 500ms (< 200ms ideal)
• Pipeline Throughput (Ops/sec): Rate at each stage
  • Queries:

    rate(opportunities_detected_total[5m])  # Scanner
    rate(execution_plans_created_total[5m])  # Planner
    rate(executions_started_total[5m])       # Executor (started)
    rate(executions_succeeded_total[5m])     # Executor (succeeded)
    

  • Visual: Multi-value stat with graphs

Row 2: Scanner Stage

• Opportunities Detected: Detected vs Published vs Rejected
  • Queries:

    rate(opportunities_detected_total{service="ts-scanner-service"}[5m])
    rate(opportunities_published_total{service="ts-scanner-service"}[5m])
    rate(opportunities_rejected_total{service="ts-scanner-service"}[5m])
    

  • Visual: Stacked area chart
• Quote Processing Latency: P95, P99 quote latency
  • Queries:

    histogram_quantile(0.95, sum(rate(quote_latency_seconds_bucket[5m])) by (le)) * 1000
    histogram_quantile(0.99, sum(rate(quote_latency_seconds_bucket[5m])) by (le)) * 1000
    

  • Unit: milliseconds
  • Thresholds: Green (<50ms), Yellow (50-100ms), Red (>100ms)
• Profit Detected (BPS): P50, P95 profit in basis points
  • Queries:

    histogram_quantile(0.50, sum(rate(arbitrage_profit_bps_bucket[5m])) by (le))
    histogram_quantile(0.95, sum(rate(arbitrage_profit_bps_bucket[5m])) by (le))
    

  • Unit: basis points (BPS)

Row 3: Planner Stage

• Plan Creation & Rejection: Created vs Rejected vs Received
  • Queries:

    rate(execution_plans_created_total{service="ts-strategy-service"}[5m])
    rate(execution_plans_rejected_total{service="ts-strategy-service"}[5m])
    rate(opportunities_received_total{service="ts-strategy-service"}[5m])
    

  • Visual: Line chart
• Rejection Reasons: Pie chart breakdown
  • Query: sum by (reason) (increase(execution_plans_rejected_total[5m]))
  • Visual: Donut chart
  • Common reasons: low_profit, high_risk, expired, validation_failed
• Risk Scores: P50, P95 risk score distribution
  • Queries:

    histogram_quantile(0.50, sum(rate(execution_plan_risk_score_bucket[5m])) by (le))
    histogram_quantile(0.95, sum(rate(execution_plan_risk_score_bucket[5m])) by (le))
    

  • Unit: 0.0-1.0 (percentage)
  • Visual: Line chart with range 0-1

Row 4: Executor Stage

• Execution Status: Started, Succeeded, Failed rates
  • Queries:

    rate(executions_started_total{service="ts-executor-service"}[5m])
    rate(executions_succeeded_total{service="ts-executor-service"}[5m])
    rate(executions_failed_total{service="ts-executor-service"}[5m])
    

  • Visual: Multi-line chart
• Success Rate: Percentage gauge
  • Query: rate(executions_succeeded_total[5m]) / (rate(executions_succeeded_total[5m]) + rate(executions_failed_total[5m])) * 100
  • Visual: Gauge (0-100%)
  • Thresholds: Green (>90%), Yellow (70-90%), Red (<70%)
• Execution Duration: P50, P95, P99 timing
  • Queries:

    histogram_quantile(0.50, sum(rate(execution_duration_ms_bucket[5m])) by (le))
    histogram_quantile(0.95, sum(rate(execution_duration_ms_bucket[5m])) by (le))
    histogram_quantile(0.99, sum(rate(execution_duration_ms_bucket[5m])) by (le))
    

  • Unit: milliseconds
  • Thresholds: Green (<200ms), Yellow (200-500ms), Red (>500ms)

Row 5: Conversion Funnel

• Conversion Rate: Scanner → Planner → Executor: Horizontal bar gauge
  • Queries:

    sum(increase(opportunities_detected_total[5m]))  # Stage 1
    sum(increase(execution_plans_created_total[5m]))  # Stage 2
    sum(increase(executions_started_total[5m]))       # Stage 3
    sum(increase(executions_succeeded_total[5m]))     # Stage 4
    

  • Visual: Horizontal bar gauge showing drop-off
  • Shows conversion % at each stage

Use Cases

• Performance Optimization: Identify bottlenecks in pipeline
• Latency Analysis: Track sub-500ms execution goal
• Conversion Tracking: Understand opportunity → execution drop-off
• Quality Metrics: Monitor rejection reasons and risk scores

Dashboard 3: FlatBuffers Events & Streams 📦

File: flatbuffers-streams.json Purpose: NATS JetStream event flow and FlatBuffers performance

Panels

Row 1: NATS JetStream Overview

• NATS Connection Health: Server up/down status
  • Query: up{job="nats"}
  • Visual: Stat (HEALTHY/DOWN)
• Active Subscriptions (All Services): Total consumer count
  • Query: sum(active_subscriptions)
  • Visual: Stat with counter
• Event Processing Rate (Events/sec): Rate by stream
  • Queries:

    sum(rate(metrics_received_total[5m]))           # METRICS
    sum(rate(execution_results_received_total[5m])) # EXECUTED
    sum(rate(execution_plans_received_total[5m]))   # PLANNED
    sum(rate(opportunities_received_total[5m]))     # OPPORTUNITIES
    

  • Visual: Multi-value stat with area graphs

Row 2: MARKET_DATA Stream

• Event Types Published: Rate by event type
  • Query: rate(events_published_total{stream="MARKET_DATA"}[5m])
  • Legend: ``
  • Visual: Stacked area chart
• Publishing Errors: Error rate by service
  • Query: rate(event_publish_errors_total{stream="MARKET_DATA"}[5m])
  • Legend: ` - `
  • Visual: Line chart

Row 3: OPPORTUNITIES Stream

• Published vs Received: Pub/sub flow tracking
  • Queries:

    rate(opportunities_published_total{service="ts-scanner-service"}[5m])  # Published
    rate(opportunities_received_total{service="ts-strategy-service"}[5m])  # Received
    

  • Visual: Dual-line chart
  • Ideal: Lines should match (no lag)
• Message Lag (Pub→Sub): Time from publish to consume
  • Query: avg(timestamp() - opportunity_timestamp_ms / 1000)
  • Unit: seconds
  • Thresholds: Green (<0.1s), Yellow (0.1-1s), Red (>1s)

Row 4: PLANNED Stream

• Plan Flow: Published (Planner) vs Received (Executor)
  • Queries:

    rate(execution_plans_published_total{service="ts-strategy-service"}[5m])
    rate(execution_plans_received_total{service="ts-executor-service"}[5m])
    

  • Visual: Dual-line chart
• Plan Validation Status: Valid, Expired, Invalid
  • Queries:

    rate(execution_plans_valid_total[5m])
    rate(execution_plans_expired_total[5m])
    rate(execution_plans_invalid_total[5m])
    

  • Visual: Stacked area chart

Row 5: EXECUTED Stream

• Results Published: Executor → Auditor flow
  • Queries:

    rate(execution_results_published_total{service="ts-executor-service"}[5m])
    rate(execution_results_received_total{service="system-auditor"}[5m])
    

  • Visual: Dual-line chart
• Result Types: Pie chart by status
  • Query: sum by (status) (increase(execution_results_published_total[5m]))
  • Visual: Pie chart
  • Categories: Success, PartialFill, Failed

Row 6: METRICS Stream

• Event Flow: Auditor → Manager
  • Queries:

    rate(pnl_metrics_published_total{service="system-auditor"}[5m])
    rate(metrics_received_total{service="system-manager"}[5m])
    

  • Visual: Dual-line chart
• Metric Types Received: Breakdown by type
  • Query: rate(metrics_received_by_type_total{service="system-manager"}[5m])
  • Legend: ``
  • Types: PnL, Latency, Throughput, Error, SystemResource

Row 7: SYSTEM Stream

• Event Types: System events rate
  • Query: rate(system_events_published_total[5m])
  • Legend: ``
  • Types: KillSwitch, SystemShutdown, SystemStart
• Kill Switch Triggers: Last hour and 24h
  • Queries:

    sum(increase(kill_switches_triggered_total[1h]))   # Last Hour
    sum(increase(kill_switches_triggered_total[24h]))  # Last 24 Hours
    

  • Visual: Stat panel
  • Thresholds: Green (0), Yellow (1-4), Red (>5)

Row 8: Event Serialization Performance

• FlatBuffers Serialization Time: P95, P99 by event type
  • Queries:

    histogram_quantile(0.95, sum(rate(event_serialization_duration_ms_bucket[5m])) by (le, event_type))
    histogram_quantile(0.99, sum(rate(event_serialization_duration_ms_bucket[5m])) by (le, event_type))
    

  • Unit: milliseconds
  • Thresholds: Green (<1ms), Yellow (1-5ms), Red (>5ms)
  • Target: Sub-millisecond serialization
• Event Size (Bytes): Average size by event type
  • Query: avg(event_size_bytes) by (event_type)
  • Unit: bytes
  • Visual: Bar chart

Use Cases

• Event Flow Debugging: Track pub/sub lag and dropped messages
• NATS Monitoring: Health of JetStream infrastructure
• FlatBuffers Performance: Ensure sub-millisecond serialization
• Kill Switch Tracking: Monitor system safety mechanisms

Dashboard 4: System Health & Services 💊

File: system-health-services.json Purpose: Service health monitoring and infrastructure status

Panels

Row 1: All Services Health Overview

• Service Status Matrix: Historical up/down timeline
  • Query: up{job=~"system-initializer|notification-service|event-logger-service|ts-scanner-service|ts-strategy-service|ts-executor-service|system-manager|system-auditor"}
  • Legend: ``
  • Visual: Status history (green/red timeline)
  • Shows service availability over time

Row 2: TypeScript Services

• TypeScript Services - Uptime: Uptime per service
  • Query: service_uptime_seconds{language="typescript"}
  • Legend: ``
  • Visual: Stat panel
  • Unit: seconds
• TypeScript Services - Error Rate: Errors/sec by service
  • Query: rate(errors_total{language="typescript"}[5m])
  • Legend: ``
  • Visual: Multi-line chart

Row 3: Pipeline Services Health

• Scanner Service Health: Multi-metric status
  • Queries:

    up{service="ts-scanner-service"}           # Status
    grpc_connected{service="ts-scanner-service"}  # gRPC
    nats_connected{service="ts-scanner-service"}  # NATS
    

  • Visual: Stat panel with ✓/✗ indicators
• Strategy Service Health: Multi-metric status
  • Queries:

    up{service="ts-strategy-service"}
    nats_connected{service="ts-strategy-service"}
    service_info{service="ts-strategy-service"}
    

  • Visual: Stat panel with ✓/✗ indicators
• Executor Service Health: Multi-metric status with in-flight count
  • Queries:

    up{service="ts-executor-service"}
    nats_connected{service="ts-executor-service"}
    in_flight_executions{service="ts-executor-service"}
    

  • Visual: Stat panel
  • Shows active trades in progress

Row 4: Management Services

• System Manager: Metrics processed and kill switches
  • Queries:

    up{service="system-manager"}
    metrics_received_total{service="system-manager"}
    kill_switches_triggered_total{service="system-manager"}
    

  • Visual: Multi-value stat
• System Auditor: Trade count and P&L
  • Queries:

    up{service="system-auditor"}
    total_trades{service="system-auditor"}
    realized_pnl_usd{service="system-auditor"}
    

  • Visual: Multi-value stat
• System Initializer: Stream and consumer setup
  • Queries:

    up{service="system-initializer"}
    streams_created_total{service="system-initializer"}
    consumers_created_total{service="system-initializer"}
    

  • Visual: Multi-value stat

Row 5: Go Services

• Event Logger Service: Status and events logged
  • Queries:

    up{service="event-logger-service"}
    events_logged_total{service="event-logger-service"}
    

  • Visual: Stat panel
• Quote Service: Health indicators
  • Queries:

    up{service="quote-service"}
    service_healthy{service="quote-service"}
    cache_healthy{service="quote-service"}
    

  • Visual: Stat panel with HEALTHY/UNHEALTHY mappings

Row 6: System-Wide Metrics

• Total Requests/sec (All Services): Request rate by service
  • Query: sum(rate(http_requests_total[5m])) by (service)
  • Legend: ``
  • Visual: Stacked area chart
• Error Rate by Service: Error rate comparison
  • Query: sum(rate(errors_total[5m])) by (service)
  • Legend: ``
  • Visual: Line chart

Row 7: Kill Switch & System Events

• Kill Switch Status: Alert list for kill switch events
  • Visual: Alert list widget
  • Shows active and recent kill switch alerts
• System Health Score: Aggregate health percentage
  • Query: (sum(up{job=~".*service.*"}) / count(up{job=~".*service.*"})) * 100
  • Visual: Gauge (0-100%)
  • Thresholds: Green (>90%), Yellow (80-90%), Red (<80%)

Use Cases

• Service Monitoring: Real-time health of all services
• Infrastructure Status: NATS, database, observability stack
• Troubleshooting: Identify which service is down or degraded
• Capacity Planning: Track resource utilization trends

Dashboard 5: Quote Service Performance 💱

File: quote-service-dashboard.json Purpose: Go quote service detailed monitoring

Key Metrics (P0 - Critical)

RPC Performance:

# RPC request rate
rate(rpc_requests_total{endpoint, method, status}[5m])

# RPC duration (P95, P99)
histogram_quantile(0.95, sum(rate(rpc_duration_seconds_bucket{endpoint, method}[5m])) by (le))

# RPC errors by type
rate(rpc_errors_total{endpoint, method, error_type}[5m])

# Connection pool metrics
rpc_connection_pool_size
rpc_connection_pool_active
rpc_connection_pool_idle

Pool Query & Calculation:

# Pool query duration by protocol
histogram_quantile(0.95, sum(rate(pool_query_duration_seconds_bucket{protocol}[5m])) by (le))

# Pools found per protocol
pool_query_count{protocol}

# Quote calculation time per pool
histogram_quantile(0.95, sum(rate(pool_quote_duration_seconds_bucket{protocol}[5m])) by (le))

# Pool selection time
histogram_quantile(0.95, sum(rate(pool_selection_duration_seconds_bucket[5m])) by (le))

Cache Performance:

# Cache refresh duration
histogram_quantile(0.95, sum(rate(cache_refresh_duration_seconds_bucket[5m])) by (le))

# Cache entries
quote_cache_entries_total
quote_cache_size

# Cache hit/miss rate
rate(quote_cache_hits_total[5m])
rate(quote_cache_misses_total[5m])

Health Checks:

service_healthy{component}  # Overall health
cache_healthy               # Cache operational
router_healthy              # Router operational
rpc_pool_total_endpoints    # RPC pool size

Request Phase Breakdown:

# Latency breakdown by phase
histogram_quantile(0.95, sum(rate(request_phase_duration_seconds_bucket{phase}[5m])) by (le, phase))

# Phases: validation, cache_check, calculation, serialization

Use Cases

• RPC Optimization: Identify slow/failing endpoints
• Cache Tuning: Monitor refresh cycles and hit rates
• Protocol Comparison: Raydium vs Meteora vs Orca performance
• Sub-500ms Goal: Track request phase breakdowns

Dashboard 6: Scanner Service Details 📡

File: ts-scanner-service-dashboard.json Purpose: Arbitrage detection and quote processing

Key Metrics

Arbitrage Detection:

# Opportunities detected, published, rejected
rate(opportunities_detected_total[5m])
rate(opportunities_published_total[5m])
rate(opportunities_rejected_total[5m])

# Profit distribution (BPS)
histogram_quantile(0.95, sum(rate(arbitrage_profit_bps_bucket[5m])) by (le))

Quote Processing:

# Quote latency
histogram_quantile(0.95, sum(rate(quote_latency_seconds_bucket[5m])) by (le))

# Quotes received from gRPC
rate(quotes_received_total[5m])

# Active token pairs
active_token_pairs

Connection Health:

grpc_connected      # gRPC to quote service
nats_connected      # NATS JetStream
active_subscriptions  # Consumer count

Key Metrics by Category

Performance Metrics

Business Metrics

Health Metrics

Event Flow Metrics

Error Metrics

Alerting Guidelines

Critical Alerts (P0 - Immediate Action)

Kill Switch Triggered

- alert: KillSwitchTriggered
  expr: increase(kill_switches_triggered_total[5m]) > 0
  for: 0m
  labels:
    severity: critical
  annotations:
    summary: "Kill switch activated"
    description: "Kill switch triggered: "
    runbook_url: "https://docs.internal/runbooks/kill-switch"

Service Down

- alert: ServiceDown
  expr: up{job=~".*service.*"} == 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "Service  is down"
    description: "Service has been down for more than 1 minute"

High Latency

- alert: HighExecutionLatency
  expr: histogram_quantile(0.95, sum(rate(execution_duration_ms_bucket{service="ts-executor-service"}[5m])) by (le)) > 500
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Execution latency exceeds 500ms"
    description: "P95 latency: ms (target: <500ms)"

High Priority Alerts (P1)

Low Success Rate

- alert: LowSuccessRate
  expr: |
    rate(executions_succeeded_total[5m]) /
    (rate(executions_succeeded_total[5m]) + rate(executions_failed_total[5m])) * 100 < 70
  for: 10m
  labels:
    severity: high
  annotations:
    summary: "Execution success rate below 70%"
    description: "Success rate: % (target: >90%)"

System Health Low

- alert: SystemHealthLow
  expr: (sum(up{job=~".*service.*"}) / count(up{job=~".*service.*"})) * 100 < 80
  for: 5m
  labels:
    severity: high
  annotations:
    summary: "System health below 80%"
    description: "% of services are healthy"

High Error Rate

- alert: HighErrorRate
  expr: sum(rate(errors_total[5m])) by (service) > 1
  for: 10m
  labels:
    severity: high
  annotations:
    summary: " error rate > 1/sec"
    description: "Error rate:  errors/sec"

Medium Priority Alerts (P2)

NATS Connection Issues

- alert: NATSDisconnected
  expr: nats_connected == 0
  for: 2m
  labels:
    severity: medium
  annotations:
    summary: " NATS connection lost"

Cache Miss Rate High

- alert: HighCacheMissRate
  expr: |
    rate(quote_cache_misses_total[5m]) /
    (rate(quote_cache_hits_total[5m]) + rate(quote_cache_misses_total[5m])) * 100 > 50
  for: 15m
  labels:
    severity: medium
  annotations:
    summary: "Cache miss rate exceeds 50%"

Alert Routing

Critical (P0):

• PagerDuty: Immediate page
• Slack: #trading-alerts (mention @oncall)
• Email: team-leads@company.com

High (P1):

• Slack: #trading-alerts
• Email: team@company.com

Medium (P2):

• Slack: #trading-monitoring
• Email: Daily digest

Troubleshooting

Problem: Prometheus Not Scraping Service

Symptoms:

• Service shows as “down” in Prometheus targets (http://localhost:9090/targets)
• No data in Grafana dashboards for that service

Diagnosis Steps:

1. Check if service is running:

   docker-compose ps <service-name>
   

2. Test metrics endpoint directly:

   curl http://localhost:<port>/metrics
   

3. Check Prometheus logs:

   docker-compose logs prometheus | grep <service-name>
   

4. Verify network connectivity:

   docker-compose exec prometheus ping <service-name>
   

Common Causes & Solutions:

Problem: Dashboard Shows “No Data”

Symptoms:

• Grafana panels display “No data” message
• Empty graphs despite services running

Diagnosis Steps:

1. Verify Prometheus is scraping the service:

   curl http://localhost:9090/api/v1/targets | grep <service-name>
   

2. Check if metric exists in Prometheus:
   • Go to http://localhost:9090/graph
   • Run query manually (e.g., up{service="ts-scanner-service"})
3. Verify time range in dashboard (top-right corner)
   • Try “Last 5 minutes” for immediate data
   • Check if selected time range has data
4. Test metrics endpoint:

   curl http://localhost:<port>/metrics | grep <metric_name>
   

Common Causes & Solutions:

Problem: High Cardinality Warning

Symptoms:

• Prometheus memory usage increasing rapidly
• Slow query performance
• Warning in Prometheus logs: “Many time series created”

Diagnosis:

# Check cardinality by metric
curl http://localhost:9090/api/v1/label/__name__/values | jq '.data | length'

# Check series count
curl http://localhost:9090/api/v1/status/tsdb | jq '.data.seriesCountByMetricName'

Solution:

• Avoid high-cardinality labels (user IDs, transaction IDs, timestamps)
• Use aggregation where possible
• Review metric labels and remove unnecessary dimensions
• Consider recording rules for frequently-queried aggregations

Good vs Bad Labels:

# ✅ GOOD (low cardinality)
http_requests_total{service="scanner", status="200", method="GET"}

# ❌ BAD (high cardinality)
http_requests_total{user_id="123456", tx_id="abc-def-ghi", timestamp="1234567890"}

Problem: Missing Historical Data

Symptoms:

• Can only see recent data (e.g., last 2 hours)
• Long-term trends not available

Diagnosis:

# Check Prometheus retention
docker-compose exec prometheus promtool tsdb stats /prometheus

# Check Mimir (long-term storage)
curl http://localhost:9009/ready

Solutions:

1. Increase Prometheus retention:

   # docker-compose.yml
   prometheus:
     command:
       - '--storage.tsdb.retention.time=30d'
       - '--storage.tsdb.retention.size=10GB'
   

2. Verify Mimir remote write: ```yaml

   prometheus.yml

   remote_write:

   • url: http://mimir:9009/api/v1/push ```
3. Check Mimir ingestion:

   docker-compose logs mimir | grep "samples ingested"
   

Problem: Slow Dashboard Loading

Symptoms:

• Grafana dashboards take > 10 seconds to load
• Panels timeout
• Browser becomes unresponsive

Solutions:

1. Reduce query time range: Use shorter intervals for heavy queries
2. Increase query timeout in Grafana data source settings
3. Optimize queries:

   # ❌ SLOW (calculates everything then filters)
   avg(rate(metric[5m])) by (service)
   
   # ✅ FAST (filters first)
   avg(rate(metric{service="ts-scanner-service"}[5m])) by (label)
   

4. Use recording rules for expensive queries: ```yaml

   prometheus/rules/recording.yml

   groups:

   • name: performance interval: 30s rules:
     • record: job:execution_duration_ms:p95 expr: histogram_quantile(0.95, sum(rate(execution_duration_ms_bucket[5m])) by (le, service)) ```
5. Enable query caching in Grafana:
   • Settings → Data Sources → Prometheus
   • Enable “Cache timeout”: 300s

Future Enhancements

Week 2-3 (High Priority)

1. Concurrent Operation Metrics

Go Services:

// Goroutine tracking
goroutines_active                     // Gauge
goroutines_created_total              // Counter
goroutine_duration_seconds{pool}      // Histogram

// Queue depth
event_queue_size{stream}              // Gauge
event_queue_capacity{stream}          // Gauge

TypeScript Services:

// Event loop metrics
event_loop_lag_seconds                // Histogram
event_loop_utilization                // Gauge

// Promise pool
promise_pool_size                     // Gauge
promise_pool_active                   // Gauge

2. Infrastructure Metrics

# CPU & Memory
process_cpu_usage_percent{service}
process_memory_usage_bytes{service}
process_heap_usage_bytes{service}

# Network I/O
network_bytes_sent_total{service}
network_bytes_received_total{service}

# File Descriptors
process_open_fds{service}
process_max_fds{service}

3. Distributed Tracing Enhancements

• Custom span attributes for business context
• Cross-service trace correlation
• Trace sampling strategies
• Span duration histograms

Month 2+ (Medium Priority)

4. Business Logic Metrics

# Quote Quality
quote_quality_score{protocol}         // 0.0-1.0 quality score
quote_staleness_seconds{protocol}     // How old is the quote

# Spread Analysis
price_spread_bps{pair}                // Bid-ask spread
liquidity_depth_usd{pair, level}      // Liquidity at price levels

# Slippage
execution_slippage_bps{strategy}      // Expected vs actual price

5. Strategy-Specific Dashboards

Create dedicated dashboards for each strategy:

• Two-Hop Arbitrage Dashboard
• Triangular Arbitrage Dashboard
• Statistical Arbitrage Dashboard (future)

Each with:

• Strategy-specific metrics
• Performance comparison vs baseline
• Historical backtesting results
• Cost analysis (gas, Jito tips)

6. SLO/SLI Tracking

Define and track Service Level Objectives:

# SLO Dashboard
- Availability SLO: 99.9% uptime
  Indicator: sum(up) / count(up)

- Latency SLO: P95 < 200ms
  Indicator: histogram_quantile(0.95, execution_duration_ms)

- Success Rate SLO: > 95%
  Indicator: executions_succeeded / (succeeded + failed)

- Error Budget: 0.1% (43 minutes/month)
  Tracking: Remaining budget visualization

Best Practices

Metric Naming

Follow Prometheus naming conventions:

• Format: <namespace>_<metric>_<unit>_<type>
• Example: execution_duration_ms_bucket (histogram)

Units:

• _seconds for time (base unit)
• _bytes for size
• _total for counters
• No suffix for gauges

Good Examples:

rpc_requests_total                    # Counter
execution_duration_seconds            # Histogram
cache_size_bytes                      # Gauge
error_rate                            # Gauge (ratio, no unit)

Label Best Practices

Use labels for dimensions:

# ✅ GOOD
http_requests_total{service="scanner", status="200", method="GET"}

# ❌ BAD
http_requests_200_scanner_get_total

Avoid high cardinality:

# ✅ GOOD (low cardinality)
trades_total{service, strategy, status}

# ❌ BAD (high cardinality - millions of values)
trades_total{user_id, transaction_id, timestamp}

Common cardinality limits:

• Per metric: < 10 labels
• Per label: < 100 unique values (ideally < 20)
• Total series: < 10 million (for single Prometheus instance)

Query Optimization

Use recording rules for expensive queries:

# Instead of calculating P95 latency repeatedly
groups:
  - name: performance
    rules:
      - record: job:execution_latency:p95
        expr: histogram_quantile(0.95, sum(rate(execution_duration_ms_bucket[5m])) by (le, service))

Then use in dashboards:

# ✅ FAST (pre-calculated)
job:execution_latency:p95{service="ts-executor-service"}

# ❌ SLOW (calculates on every query)
histogram_quantile(0.95, sum(rate(execution_duration_ms_bucket{service="ts-executor-service"}[5m])) by (le))

Dashboard Design

Panel organization:

1. Status first: Overall health at top
2. KPIs next: Business metrics (P&L, win rate, throughput)
3. Performance: Latency, success rate
4. Details last: Deep-dive metrics

Use rows for grouping:

• Collapsible rows for optional details
• Clear row titles (e.g., “Scanner Stage”, “Planner Stage”)

Consistent time ranges:

• Use dashboard time picker
• Avoid hard-coded time ranges in queries
• Standard ranges: 1h, 6h, 24h, 7d, 30d

Related Documentation

• 18-HFT_PIPELINE_ARCHITECTURE.md - Scanner → Planner → Executor architecture
• 19-FLATBUFFERS-MIGRATION.md - FlatBuffers event migration guide
• 08-optimization-guide.md - Performance optimization roadmap (1.7s → 200ms)
• 07-hft-architecture.md - Initial HFT architecture design

Quick Reference

Common Queries

# Service uptime
service_uptime_seconds{service="ts-scanner-service"}

# Error rate (last 5 minutes)
rate(errors_total{service="ts-scanner-service"}[5m])

# Request latency P95
histogram_quantile(0.95, sum(rate(request_duration_ms_bucket[5m])) by (le, service))

# Success rate
rate(executions_succeeded_total[5m]) / (rate(executions_succeeded_total[5m]) + rate(executions_failed_total[5m])) * 100

# Event processing rate
rate(events_processed_total[5m])

# System health score
(sum(up{job=~".*service.*"}) / count(up{job=~".*service.*"})) * 100

Useful Commands

# Check Prometheus targets
curl http://localhost:9090/api/v1/targets | python -m json.tool

# Test service metrics endpoint
curl http://localhost:9096/metrics | grep service_info

# Restart Prometheus with new config
docker-compose restart prometheus

# Reload Grafana dashboards
docker-compose restart grafana

# Check Prometheus query performance
curl 'http://localhost:9090/api/v1/query?query=up&stats=true'

# View Prometheus TSDB stats
docker-compose exec prometheus promtool tsdb stats /prometheus

Summary

Implementation Status: ✅ Production Ready

Key Achievements:

• ✅ Full HFT pipeline observability (Scanner → Planner → Executor)
• ✅ All 6 NATS JetStream streams monitored
• ✅ Business metrics tracked (P&L, win rate, trade statistics)
• ✅ Sub-500ms latency tracking enabled
• ✅ Production-grade dashboards with proper labeling
• ✅ Comprehensive troubleshooting guide

Next Actions:

1. Instrument event-logger-service with Go Prometheus metrics
2. Implement alert rules in Prometheus
3. Configure notification channels (email, Slack)
4. Add strategy-specific dashboards
5. Implement SLO/SLI tracking