Inference Serving Architecture¶

Version: 2.0.0
Last Updated: 2026-06-22

Overview¶

This document describes the architecture of the inference serving system, including components, data flows, and deployment patterns.

System Architecture¶

%%{init: {'accessibility': {'title': 'Diagram showing Web Client, API Client'}}%%
graph TB
    subgraph "Client Layer"
        C1[Web Client]
        C2[API Client]
        C3[Mobile App]
    end

    subgraph "Load Balancer"
        LB[Nginx / ALB]
    end

    subgraph "Inference Cluster"
        subgraph "Blue Deployment"
            B1[Inference Server v1.0]
            B2[Inference Server v1.0]
        end

        subgraph "Green Deployment"
            G1[Inference Server v1.1]
            G2[Inference Server v1.1]
        end

        TS[Traffic Splitter]
    end

    subgraph "Model Storage"
        MS[(Model Registry)]
        MC[Model Cache]
    end

    subgraph "Monitoring"
        PM[Prometheus]
        GF[Grafana]
        AL[Alertmanager]
    end

    C1 --> LB
    C2 --> LB
    C3 --> LB

    LB --> TS
    TS -.80%.-> B1
    TS -.80%.-> B2
    TS -.20%.-> G1
    TS -.20%.-> G2

    B1 --> MS
    B2 --> MS
    G1 --> MS
    G2 --> MS

    B1 --> MC
    B2 --> MC
    G1 --> MC
    G2 --> MC

    B1 --> PM
    B2 --> PM
    G1 --> PM
    G2 --> PM

    PM --> GF
    PM --> AL

Request Flow¶

%%{init: {'accessibility': {'title': 'Sequence Diagram: >>Client: 401 Unauthorized
   '}}%%
sequenceDiagram
    participant Client
    participant Auth
    participant RateLimit
    participant CircuitBreaker
    participant ModelLoader
    participant Model
    participant Metrics

    Client->>Auth: POST /infer + API Key
    Auth->>Auth: Validate API Key

    alt Invalid API Key
        Auth-->>Client: 401 Unauthorized
    end

    Auth->>RateLimit: Check Rate Limit
    RateLimit->>RateLimit: Sliding Window Check

    alt Rate Limit Exceeded
        RateLimit-->>Client: 429 Too Many Requests
    end

    RateLimit->>CircuitBreaker: Check Circuit State

    alt Circuit Open
        CircuitBreaker-->>Client: 503 Service Unavailable
    end

    alt Circuit Half-Open
        CircuitBreaker->>CircuitBreaker: Health Probe
    end

    CircuitBreaker->>ModelLoader: Load Model

    alt Model Not in Cache
        ModelLoader->>ModelLoader: Load from Registry
        ModelLoader->>ModelLoader: Add to LRU Cache
    end

    ModelLoader->>Model: predict(input)
    Model-->>ModelLoader: prediction

    alt Prediction Success
        ModelLoader->>CircuitBreaker: Record Success
        ModelLoader->>Metrics: Update Metrics
        ModelLoader-->>Client: 200 OK + prediction
    end

    alt Prediction Failure
        ModelLoader->>CircuitBreaker: Record Failure
        ModelLoader->>Metrics: Update Error Metrics
        ModelLoader-->>Client: 500 Internal Error
    end

Circuit Breaker State Machine¶

%%{init: {'accessibility': {'title': 'State Diagram showing *'}}%%
stateDiagram-v2
    [*] --> Closed

    Closed --> Open: failures >= threshold
    Open --> HalfOpen: after backoff delay
    HalfOpen --> Closed: health probe success
    HalfOpen --> Open: health probe failure

    note right of Closed
        Normal operation
        All requests pass through
    end note

    note right of Open
        Failing fast
        No requests to backend
        Exponential backoff: 1s, 2s, 4s, ...
    end note

    note right of HalfOpen
        Testing recovery
        Limited requests
        Health probe validates
    end note

Model Loading Flow¶

%%{init: {'accessibility': {'title': 'Flowchart showing Request arrives, Cache Hit'}}%%
flowchart TD
    Start([Request arrives]) --> CheckCache{Model in cache?}

    CheckCache -->|Yes| CacheHit[Cache Hit]
    CheckCache -->|No| CacheMiss[Cache Miss]

    CacheHit --> Return[Return cached model]

    CacheMiss --> CheckRegistry{Model in registry?}
    CheckRegistry -->|Yes| Download[Download model]
    CheckRegistry -->|No| Error[Return 404]

    Download --> Validate[Validate checksum]
    Validate --> Load[Load into memory]
    Load --> Warmup[Run warmup predictions]
    Warmup --> AddCache[Add to LRU cache]

    AddCache --> CheckSize{Cache full?}
    CheckSize -->|Yes| Evict[Evict LRU model]
    CheckSize -->|No| Store[Store in cache]

    Evict --> Store
    Store --> Return

    Return --> End([Model ready])
    Error --> End

Deployment Architecture¶

Blue-Green Deployment¶

%%{init: {'accessibility': {'title': 'Flowchart showing Load Balancer, Blue v1.0'}}%%
graph LR
    subgraph "Initial State (All traffic to Blue)"
        LB1[Load Balancer] -.100%.-> Blue1[Blue v1.0]
        LB1 -.0%.-> Green1[Green v1.1]
    end

    subgraph "Gradual Rollout (20% to Green)"
        LB2[Load Balancer] -.80%.-> Blue2[Blue v1.0]
        LB2 -.20%.-> Green2[Green v1.1]
    end

    subgraph "Full Rollout (100% to Green)"
        LB3[Load Balancer] -.0%.-> Blue3[Blue v1.0]
        LB3 -.100%.-> Green3[Green v1.1]
    end

    subgraph "Promotion (Green becomes Blue)"
        LB4[Load Balancer] -.100%.-> Blue4[Blue v1.1]
    end

    style Blue1 fill:#4A90E2
    style Blue2 fill:#4A90E2
    style Blue3 fill:#4A90E2
    style Blue4 fill:#4A90E2
    style Green1 fill:#7ED321
    style Green2 fill:#7ED321
    style Green3 fill:#7ED321

Rollback Scenario¶

%%{init: {'accessibility': {'title': 'Sequence Diagram: >>Monitor: Error Rate: 8%

   '}}%%
sequenceDiagram
    participant Monitor as Health Monitor
    participant Splitter as Traffic Splitter
    participant Blue as Blue Deployment
    participant Green as Green Deployment

    Note over Splitter: Rollout in progress (60% Green)

    Monitor->>Green: Health Check
    Green-->>Monitor: Error Rate: 8%

    Monitor->>Monitor: Error rate > 5% threshold
    Monitor->>Splitter: Trigger Rollback

    Splitter->>Splitter: Set weights: Blue=100%, Green=0%

    Note over Splitter: All traffic to Blue

    Splitter->>Blue: Route all requests
    Blue-->>Splitter: Healthy responses

    Note over Green: Green deployment retired

Monitoring Architecture¶

%%{init: {'accessibility': {'title': 'Diagram showing Server 1, Prometheus'}}%%
graph TB
    subgraph "Inference Servers"
        IS1[Server 1] -->|/metrics| PM[Prometheus]
        IS2[Server 2] -->|/metrics| PM
        IS3[Server 3] -->|/metrics| PM
    end

    subgraph "Monitoring Stack"
        PM -->|scrape| PM
        PM -->|query| GF[Grafana]
        PM -->|alert| AL[Alertmanager]
    end

    subgraph "Alerting"
        AL -->|email| EM[Email]
        AL -->|slack| SL[Slack]
        AL -->|pagerduty| PD[PagerDuty]
    end

    subgraph "Visualization"
        GF -->|dashboard| DB1[Request Dashboard]
        GF -->|dashboard| DB2[Error Dashboard]
        GF -->|dashboard| DB3[Latency Dashboard]
        GF -->|dashboard| DB4[Resource Dashboard]
    end

Component Details¶

Inference Server¶

Responsibilities: - Accept HTTP requests - Authenticate requests - Apply rate limiting - Load models on-demand - Execute predictions - Return results - Expose metrics

Key Classes: - InferenceServer: FastAPI application - AuthManager: Authentication handler - CircuitBreaker: Failure protection - ModelLoader: Model management - PrometheusMetrics: Metrics collection

Configuration:

{
    "host": "0.0.0.0",
    "port": 8000,
    "workers": 4,
    "timeout": 60,
    "model_cache_size": 3,
    "rate_limit_rpm": 1000
}

Model Loader¶

Responsibilities: - Load models from registry - Cache models in memory - Evict least-recently-used models - Validate model checksums - Handle model warmup

Cache Strategy: - LRU eviction - Max 3 models (configurable) - Thread-safe access - Lazy loading

Circuit Breaker¶

Responsibilities: - Track failure rates - Open circuit on threshold - Exponential backoff recovery - Health probing - Per-model isolation

States: - Closed: Normal operation, all requests pass - Open: Failing fast, no backend requests - Half-Open: Testing recovery with health probes

Configuration:

{
    "failure_threshold": 5,
    "timeout_seconds": 60,
    "half_open_max_calls": 3,
    "backoff_base_seconds": 1,
    "backoff_max_seconds": 300
}

Traffic Splitter¶

Responsibilities: - Route requests to blue or green - Gradual traffic shifting - Health-based failover - Error rate monitoring

Routing Logic:

if not blue_healthy and green_healthy:
    return "green"
elif not green_healthy and blue_healthy:
    return "blue"
else:
    # Weighted random selection
    return weighted_choice([
        ("blue", blue_weight),
        ("green", green_weight)
    ])

Data Flow¶

Prediction Request¶

Client sends POST to /infer
Load Balancer routes to inference server
Authentication validates API key/JWT
Rate Limiter checks request quota
Circuit Breaker checks model health
Model Loader loads/retrieves model
Model executes prediction
Response returns to client
Metrics recorded to Prometheus

Model Update¶

New model uploaded to registry
Deployment manager initiates blue-green rollout
Green deployment loads new model
Health checks validate green
Traffic gradually shifts to green (0% → 100%)
Monitoring tracks error rates
Rollback if error rate exceeds threshold
Promotion if rollout successful

Scaling Strategy¶

Horizontal Scaling¶

%%{init: {'accessibility': {'title': 'Flowchart showing Load Balancer, Server 1'}}%%
graph LR
    LB[Load Balancer] --> S1[Server 1]
    LB --> S2[Server 2]
    LB --> S3[Server 3]
    LB --> S4[Server 4]
    LB --> Sn[Server N]

    S1 --> MR[(Model Registry)]
    S2 --> MR
    S3 --> MR
    S4 --> MR
    Sn --> MR

Scaling Rules: - Add replica when CPU > 70% - Remove replica when CPU < 30% - Min replicas: 2 - Max replicas: 10

Vertical Scaling¶

Memory Scaling: - Base: 4GB per server - +2GB per cached model - +4GB for GPU operations

CPU Scaling: - 2 cores for I/O operations - 4-8 cores for CPU inference - GPU recommended for large models

Security Architecture¶

%%{init: {'accessibility': {'title': 'Diagram showing TLS Termination, Authentication'}}%%
graph TB
    subgraph "Security Layers"
        TLS[TLS Termination]
        Auth[Authentication]
        RateLimit[Rate Limiting]
        Validation[Input Validation]
        Isolation[Process Isolation]
    end

    Client --> TLS
    TLS --> Auth
    Auth --> RateLimit
    RateLimit --> Validation
    Validation --> Isolation
    Isolation --> Model[Model Execution]

Security Controls: 1. TLS: All traffic encrypted 2. Authentication: API key or JWT required 3. Rate Limiting: Prevent abuse 4. Input Validation: Sanitize inputs 5. Process Isolation: Sandbox execution

Failure Modes & Recovery¶

Failure Scenarios¶

Failure	Detection	Recovery
Model crash	Circuit breaker	Reload model
High latency	Metrics	Scale up
Memory leak	Resource monitor	Restart server
Network partition	Health checks	Failover
Corrupted model	Checksum validation	Re-download

Recovery Strategies¶

Automatic: Circuit breaker, health checks
Manual: Admin intervention, rollback
Preventive: Canary deployments, chaos testing

Performance Characteristics¶

Latency Targets¶

Operation	P50	P95	P99
Health check	<10ms	<50ms	<100ms
Authentication	<5ms	<10ms	<20ms
Model loading (cache hit)	<50ms	<100ms	<200ms
Model loading (cache miss)	<5s	<10s	<15s
Inference (small model)	<100ms	<300ms	<500ms
Inference (large model)	<500ms	<1s	<2s

Throughput Targets¶

Endpoint	Target RPS
`/health`	500+
`/metrics`	100+
`/infer` (cached)	50+
`/infer` (uncached)	10+

Last reviewed: 2025-12-07