Inference Serving Troubleshooting Guide¶
Last Updated: 2026-07-11 Version: v0.2.0
Version: 2.0.0 Last Updated: 2026-06-22
Overview¶
This guide helps diagnose and resolve common issues with the inference serving infrastructure. It covers model loading, authentication, performance, circuit breakers, and deployment problems.
Quick Diagnostic Checklist¶
Before diving into specific issues, run through this checklist:
- Check
/healthendpoint status - Review Prometheus metrics at
/metrics - Check circuit breaker state
- Verify model is loaded (
GET /models) - Check logs for errors
- Verify environment variables
- Test with simple request
Common Issues¶
1. Model Loading Failures¶
Symptoms¶
- 500 errors on
/inferendpoint - "Model not found" errors
- High
/healthendpoint latency
Causes & Solutions¶
Cause: Model file not found
# Check model path
ls -lh /models/my-model/
# Verify MODEL_PATH environment variable
echo $MODEL_PATH
# Solution: Correct model path
export MODEL_PATH=/correct/path/to/model
Cause: Insufficient memory
# Check available memory
free -h
# Check GPU memory (if applicable)
nvidia-smi
# Solution: Reduce batch size or use quantization
export MODEL_QUANTIZATION=int8
Cause: Invalid model format
# Verify model format
python3 -c "
from transformers import AutoModel
model = AutoModel.from_pretrained('$MODEL_PATH')
print('Model loaded successfully')
"
# Solution: Re-export model or use correct loader
2. Authentication Failures¶
Symptoms¶
- 401 Unauthorized responses
- "Invalid API key" errors
- Authentication exhaustion warnings
Causes & Solutions¶
Cause: Missing or invalid API keys
# Check API keys are set
echo $CODEX_API_KEYS
# Solution: Set valid API keys (comma-separated)
export CODEX_API_KEYS="key1,key2,key3" <!-- pragma: allowlist secret -->
Cause: JWT token expired
# Check token expiration
import jwt
token = "your_token"
decoded = jwt.decode(token, options={"verify_signature": False})
print(f"Expires: {decoded['exp']}")
# Solution: Generate new token
Cause: Timing attack detection - Constant-time comparison may fail on clock skew - Solution: Ensure NTP sync on all servers
3. Circuit Breaker Issues¶
Symptoms¶
- 503 Service Unavailable responses
- "Circuit breaker open" in logs
/healthreports unhealthy
Diagnostic Commands¶
Solutions¶
If circuit breaker is open: 1. Check underlying model health:
# Test model directly
python3 -c "
from src.codex_ml.serving.model_loader import ModelLoader
loader = ModelLoader()
model = loader.load('model_name')
print('Model healthy')
"
- Wait for exponential backoff to complete:
- First retry: 1 second
- Second retry: 2 seconds
- Third retry: 4 seconds
-
Max: 300 seconds
-
Manual reset (if needed):
If circuit breaker is flapping (open/closed rapidly): - Increase failure threshold - Increase backoff duration - Check for intermittent network issues
4. High Latency / Slow Responses¶
Diagnostic Workflow¶
-
Check P95/P99 latency:
-
Identify bottleneck:
-
Check resource utilization:
Solutions¶
High model inference time:
- Enable quantization: export MODEL_QUANTIZATION=int8
- Reduce batch size
- Use GPU acceleration
- Optimize model architecture
High queueing time: - Increase number of workers - Enable request batching - Scale horizontally (add replicas)
Memory pressure: - Enable memory pooling - Reduce cache size - Use model quantization
Example performance tuning:
# In inference server config
from src.codex_ml.serving.optimizations import (
RequestBatcher,
DynamicBatchSizer
)
# Enable batching
batcher = RequestBatcher(
max_batch_size=32,
max_wait_ms=10
)
# Enable dynamic sizing
sizer = DynamicBatchSizer(
target_latency_ms=100
)
5. Rate Limiting Issues¶
Symptoms¶
- 429 Too Many Requests responses
- Requests rejected during load spikes
- Uneven load distribution
Diagnostic Commands¶
Solutions¶
Increase rate limits:
Implement client-side retry:
import time
import requests
def predict_with_retry(data, max_retries=3):
for attempt in range(max_retries):
response = requests.post(
json=data
)
if response.status_code == 429:
# Exponential backoff
time.sleep(2 ** attempt)
continue
return response.json()
raise Exception("Rate limit exceeded")
Use token bucket algorithm: - Current implementation: Sliding window - For bursty traffic, consider token bucket
6. Connection/Network Issues¶
Symptoms¶
- Connection refused errors
- Timeout errors
- Intermittent failures
Diagnostic Steps¶
-
Verify service is running:
-
Test connectivity:
-
Check firewall rules:
Solutions¶
Connection refused:
- Ensure server is started: python -m src.codex_ml.serving.inference_server
- Check bind address: Should be 0.0.0.0 for remote access
Timeout errors:
- Increase client timeout
- Check network latency: ping <server-ip>
- Check for packet loss: mtr <server-ip>
DNS issues:
- Use IP address instead of hostname
- Check DNS resolution: nslookup <hostname>
7. Deployment Issues¶
Symptoms¶
- Rollback triggered during blue-green deployment
- High error rate in new version
- Traffic not shifting to green
Diagnostic Commands¶
Solutions¶
Rollback triggered: 1. Check error rate threshold:
- Review logs for errors in green deployment
- Test green deployment directly before rollout
Traffic not shifting: - Check health checks pass for green - Verify minimum healthy duration met - Check gradual rollout progress
Manual traffic control:
from src.codex_ml.serving.deployment import TrafficSplitter
splitter = TrafficSplitter()
splitter.set_weights(blue=70, green=30) # 70% blue, 30% green
8. Cache Performance Issues¶
Symptoms¶
- Low cache hit rate (<50%)
- High memory usage
- Stale predictions
Diagnostic Commands¶
Solutions¶
Low hit rate: - Increase cache size (default: 3 models) - Analyze request patterns - Implement request normalization
High memory usage: - Reduce cache size - Enable cache eviction - Use quantized models
Stale predictions:
- Implement cache invalidation
- Set TTL for cached models
- Manual cache clear: loader.clear_cache()
Error Code Reference¶
| Code | Meaning | Solution |
|---|---|---|
| 400 | Bad Request | Check request payload format |
| 401 | Unauthorized | Verify API key/JWT token |
| 429 | Too Many Requests | Implement retry with backoff |
| 500 | Internal Server Error | Check logs, circuit breaker state |
| 503 | Service Unavailable | Circuit breaker open, wait for recovery |
| 504 | Gateway Timeout | Increase timeout, optimize model |
Performance Debugging¶
Latency Breakdown¶
# Example latency breakdown
total_latency = 500ms
Queue wait: 10ms (2%)
Authentication: 5ms (1%)
Preprocessing: 50ms (10%)
Model inference: 400ms (80%)
Postprocessing: 35ms (7%)
Optimization Priority¶
- Model inference (80%): Use quantization, GPU, smaller model
- Preprocessing (10%): Vectorize operations, use compiled libraries
- Postprocessing (7%): Optimize format conversion
- Queue wait (2%): Enable batching
- Authentication (1%): Use token caching
Monitoring Best Practices¶
Key Metrics to Watch¶
- Request rate: Sudden spikes may indicate attack or legitimate traffic surge
- Error rate: Should be <1% in production
- P95 latency: Should be <500ms for interactive use cases
- Circuit breaker state: Should be "closed" >99% of time
- Cache hit rate: Target >60% for cost efficiency
Alert Thresholds¶
# Recommended Prometheus alert thresholds
- error_rate > 5% for 5 minutes
- p95_latency > 1s for 10 minutes
- circuit_breaker_open > 0 for 2 minutes
- cache_hit_rate < 40% for 30 minutes
- cpu_usage > 90% for 5 minutes
- memory_usage > 90% for 5 minutes
Getting Help¶
Information to Provide¶
When requesting support, include:
- Symptoms: What's not working?
- Logs: Last 100 lines from inference server
- Metrics: Current Prometheus metrics (
/metrics) - Configuration: Environment variables
- Request example: Sample failing request
- Timeline: When did issue start?
Useful Commands¶
# Collect diagnostic bundle
./scripts/collect_diagnostics.sh
# Generate report
python -m src.codex_ml.serving.diagnostics --report
# Test model loading
python -m src.codex_ml.serving.model_loader --test
# Benchmark performance
python -m tests.serving.test_inference_performance --benchmark
Related Documentation¶
Last reviewed: 2025-12-07