Inference Serving Guide¶
Last Updated: 2026-07-11 Version: v0.2.0
**Last Updated: 2026-06-22
Overview¶
The Codex ML Inference Server provides a production-ready inference serving layer with built-in safeguards, rate limiting, and support for multiple model backends.
Features¶
- Multiple Model Backends: Support for stub (testing), HuggingFace, and ONNX models
- Configuration Management: Environment variables or config file-based setup
- Rate Limiting: Built-in request rate limiting per client
- Input Validation: Automatic validation of batch size and input length
- Health Checks: Comprehensive health monitoring endpoints
- Error Handling: Graceful error handling with detailed error messages
- Metrics Collection: Request tracking and performance metrics
Quickstart¶
Basic Setup (Stub Model)¶
The simplest way to get started is with a stub model for testing:
```python from src.codex_ml.serving.inference_server import ModelServer, ModelConfig
Create server with default stub model¶
config = ModelConfig(model_type="stub", model_name="my-test-model") server = ModelServer(config=config)
Load the model¶
server.load_model()
Make predictions¶
predictions = server.predict(["Hello world", "Test input"]) print(predictions) ```text
The bundled SimpleInferenceModel is deterministic and exercises both happy-path and error-path logic:
/predictechoes inputs with uppercasepredictionfields and returns HTTP 500 for the sentinel input"raise-error"./embedreturns lightweight one-hot embeddings (dimension 8) keyed by character sums and raises HTTP 500 for"raise-error".
FastAPI Server¶
To run a full FastAPI server:
```bash
Set environment variables¶
export CODEX_MODEL_NAME=my-model export CODEX_MODEL_TYPE=stub export CODEX_MODEL_DEVICE=cpu
Run the server¶
python -m src.codex_ml.serving.inference_server ```text
GET /- Server informationGET /health- Health checkPOST /predict- Make predictionsGET /metrics- Server metrics
Configuration¶
Environment Variables¶
Configure the server using environment variables:
| Variable | Description | Default | Options |
|---|---|---|---|
CODEX_MODEL_NAME |
Name of the model | default-model |
Any string |
CODEX_MODEL_TYPE |
Type of model backend | stub |
stub, huggingface, onnx |
CODEX_MODEL_PATH |
Path to model files | .codex/models/<model_name> |
Any valid path |
CODEX_MODEL_DEVICE |
Device to run model on | cpu |
cpu, cuda |
CODEX_MODEL_CONFIG |
Path to config file | None | Any valid file path |
CODEX_MODEL_DIR |
Base directory for models | .codex/models |
Any valid directory |
Configuration File¶
Alternatively, use a configuration dictionary:
```python config_dict = { "model_name": "my-classifier", "model_type": "huggingface", "model_path": "/path/to/model", "device": "cpu", }
config = ModelConfig.from_dict(config_dict) server = ModelServer(config=config) ```text
ModelConfig Class¶
The ModelConfig class provides a type-safe way to configure the model server:
```python from src.codex_ml.serving.inference_server import ModelConfig
Create configuration¶
config = ModelConfig( model_name="my-model", model_type="huggingface", model_path="/path/to/model", device="cpu", )
Validate configuration¶
config.validate()
Convert to dictionary¶
config_dict = config.to_dict() ```text
Model Backends¶
Stub Model (Testing)¶
Best for development and testing. Returns dummy predictions:
python
config = ModelConfig(model_type="stub", model_name="test-model")
server = ModelServer(config=config)
server.load_model()text
Output Format:
python
[
{
"text": "input text",
"label": "POSITIVE",
"score": 0.95,
"model": "test-model"
}
]text
HuggingFace Models¶
Load and serve HuggingFace Transformers models:
python
config = ModelConfig(
model_type="huggingface",
model_name="bert-classifier",
model_path="/path/to/huggingface/model",
device="cpu",
)
server = ModelServer(config=config)
server.load_model()text
Requirements:
- Model files must exist at the specified path
- transformers package must be installed: pip install transformers
Output Format:
python
[
{
"text": "input text",
"embedding": [0.1, 0.2, ...], # First 10 dimensions
"model": "bert-classifier"
}
]text
ONNX Models¶
Load and serve ONNX Runtime models:
python
config = ModelConfig(
model_type="onnx",
model_name="onnx-classifier",
model_path="/path/to/model.onnx",
device="cpu",
)
server = ModelServer(config=config)
server.load_model()text
Requirements:
- Model file must exist at the specified path
- onnxruntime package must be installed: pip install onnxruntime
Note: ONNX inference currently returns stub predictions. Full implementation requires proper tokenization for your specific model.
Making Predictions¶
Basic Prediction¶
```python
Load model¶
server.load_model()
Single or batch prediction¶
inputs = ["First input", "Second input", "Third input"] predictions = server.predict(inputs)
for pred in predictions: print(f"Text: {pred['text']}") print(f"Label: {pred['label']}") print(f"Score: {pred['score']}") ```text
With Parameters¶
Pass optional parameters to customize inference:
python
predictions = server.predict(
inputs=["Hello world"],
parameters={
"max_length": 512,
"temperature": 0.7,
}
)text
Input Constraints¶
- Batch Size: Maximum 100 inputs per request (configurable via
MAX_BATCH_SIZE) - Input Length: Maximum 10,000 characters per input (configurable via
MAX_INPUT_LENGTH)
Error Handling¶
Model Loading Errors¶
```python from src.codex_ml.serving.inference_server import ModelLoadError
try: server.load_model() except ModelLoadError as e: print(f"Failed to load model: {e}") # Handle error (fallback, retry, etc.) ```text
Common errors: - Missing model files: Model path does not exist - Invalid configuration: Invalid model_type or device - Missing dependencies: Required package not installed
Prediction Errors¶
python
try:
predictions = server.predict(inputs)
except RuntimeError as e:
print(f"Prediction failed: {e}")text
Common errors:
- Model not loaded: Call load_model() first
- Invalid input: Check batch size and input length constraints
Health Checks¶
Check server health programmatically:
```python health = server.health_check()
print(f"Status: {health['status']}") print(f"Model Loaded: {health['model_loaded']}") print(f"Uptime: {health['uptime_seconds']} seconds") print(f"Total Requests: {health['total_requests']}") print(f"Model Type: {health['model_type']}") print(f"Device: {health['device']}")
Check for load errors¶
if health['load_errors']: print(f"Load Errors: {health['load_errors']}") ```text
Rate Limiting¶
The server includes built-in rate limiting:
- Default Limit: 1000 requests per minute per client IP
- Window: 60 seconds (rolling window)
- Response: HTTP 429 (Too Many Requests) when limit exceeded
Configure rate limiting:
```python from src.codex_ml.serving.inference_server import RateLimiter
Custom rate limiter¶
limiter = RateLimiter(max_requests=500, window_seconds=60)
Check if request is allowed¶
if limiter.is_allowed(client_ip): # Process request pass else: # Reject request pass ```text
API Endpoints (FastAPI)¶
GET /¶
Server information and available endpoints.
Response:
json
{
"message": "Codex ML Inference Server",
"model": "my-model",
"endpoints": {
"health": "/health",
"predict": "/predict (POST)",
"metrics": "/metrics"
}
}text
GET /health¶
Health check endpoint.
Response:
json
{
"status": "healthy",
"model_loaded": true,
"uptime_seconds": 123.45,
"total_requests": 42,
"model_name": "my-model",
"model_type": "stub",
"device": "cpu",
"load_errors": []
}text
POST /predict¶
Make predictions.
Request:
json
{
"inputs": ["text1", "text2"],
"parameters": {
"max_length": 512
}
}text
Response:
json
{
"predictions": [
{
"text": "text1",
"label": "POSITIVE",
"score": 0.95,
"model": "my-model"
}
],
"model_name": "my-model",
"inference_time_ms": 12.34,
"metadata": {
"batch_size": 1,
"total_requests": 43
}
}text
GET /metrics¶
Server metrics.
Response:
json
{
"total_requests": 42,
"uptime_seconds": 123.45,
"model_name": "my-model",
"model_loaded": true
}text
What's NOT Covered (Advanced Features)¶
The following features are intentionally not included in this initial implementation and are reserved for future enhancements:
- Streaming Inference: Real-time streaming of predictions
- Batching Optimization: Automatic request batching for efficiency
- Multi-Model Serving: Serving multiple models simultaneously
- Model Routing: Intelligent routing between model versions
- Vector Store Integration: Integration with vector databases
- Caching: Response caching for repeated queries
- Advanced Metrics: Detailed performance profiling and traces
- GPU Optimization: Multi-GPU support and optimization
- Model Versioning: A/B testing and gradual rollouts
- Custom Preprocessing: Configurable preprocessing pipelines
These features will be addressed in subsequent pull requests as the system matures.
Examples¶
Complete Example: Stub Model¶
```python from src.codex_ml.serving.inference_server import ModelServer, ModelConfig
Configure¶
config = ModelConfig( model_name="sentiment-classifier", model_type="stub", device="cpu", )
Create server¶
server = ModelServer(config=config)
Load model¶
server.load_model()
Check health¶
health = server.health_check() print(f"Server status: {health['status']}")
Make predictions¶
inputs = [ "This product is !", "Worst purchase ever.", "It's okay, nothing special.", ]
predictions = server.predict(inputs)
for inp, pred in zip(inputs, predictions): print(f"\nInput: {inp}") print(f"Label: {pred['label']}") print(f"Score: {pred['score']:.2f}") ```text
Complete Example: Running FastAPI Server¶
```bash
1. Install dependencies¶
pip install fastapi uvicorn
2. Set environment variables¶
export CODEX_MODEL_NAME=my-classifier export CODEX_MODEL_TYPE=stub export CODEX_MODEL_DEVICE=cpu
3. Run server¶
python -m src.codex_ml.serving.inference_server
4. Test endpoints¶
5. Make prediction¶
-H "Content-Type: application/json" \ -d '{ "inputs": ["This is a test"], "parameters": {} }' ```text
Troubleshooting¶
Model Won't Load¶
- Check model path: Verify the path exists and contains model files
- Check dependencies: Ensure required packages are installed
- Check configuration: Validate model_type and device settings
- Check logs: Look for detailed error messages in server logs
Predictions Fail¶
- Check model is loaded: Call
load_model()beforepredict() - Check input format: Ensure inputs are a list of strings
- Check input size: Verify batch size and input length limits
- Check health: Use
health_check()to verify server state
Rate Limiting Issues¶
- Check client IP: Different clients have independent limits
- Wait for window: Rate limit window resets after 60 seconds
- Adjust limits: Configure custom
RateLimiterif needed
Performance Considerations¶
- Stub models: Very fast, suitable for testing
- HuggingFace models: Depends on model size and complexity
- ONNX models: Generally faster than HuggingFace for inference
- Batch size: Larger batches may improve throughput but increase latency
- Device: GPU (
cuda) will be faster than CPU for large models
Security Considerations¶
- Rate Limiting: Prevents abuse and DoS attacks
- Input Validation: Enforces size limits on inputs
- Error Handling: Prevents information leakage in error messages
- CORS: Configure appropriately for production
- Authentication: Not included; add via middleware if needed
Next Steps¶
- Try the stub model to understand the API
- Configure environment variables for your use case
- Load a real model (HuggingFace or ONNX)
- Run the FastAPI server and test endpoints
- Integrate with your application using the Python API or HTTP endpoints
For questions or issues, refer to the test suite in tests/codex_ml/test_inference_server.py for working examples.