Training Workflow¶
Last Updated: 2026-07-11 Version: v0.2.0
Last Updated: 2026-01-20 Version: v0.2.0 Reference: E2E Request Flow
Training Workflow Overview¶
%%{init: {'accessibility': {'title': 'Model Training Workflow<br/>Configuration to Checkpoint'}, 'theme': 'base'}}%%
graph TD
Start([" Training Start"]) --> LoadCfg[" Load Configuration<br/>• config.yaml<br/>• defaults resolution<br/>• override application"]
LoadCfg --> ValidCfg{"Config<br/>Valid?"}
ValidCfg -->|" Error"| CfgError["Return Error<br/>Fix and retry"]
ValidCfg -->|" OK"| PrepData[" Data Preparation<br/>• Load dataset<br/>• Tokenize text<br/>• Create batches<br/>• Compute statistics"]
PrepData --> SplitData[" Train/Val/Test Split<br/>• training: 70%<br/>• validation: 15%<br/>• test: 15%"]
SplitData --> LoadModel[" Load Model<br/>• Architecture: GPT-2 style<br/>• Init: Xavier/He<br/>• Parameters: 124M<br/>• Device: GPU/CPU"]
LoadModel --> Optimizer[" Setup Optimizer<br/>• Algorithm: AdamW<br/>• LR: 5e-4<br/>• Weight decay: 0.01<br/>• Scheduler: Cosine"]
Optimizer --> InitMonitor[" Init Monitoring<br/>• Loss tracking<br/>• Metric computation<br/>• Log directory setup<br/>• Tensorboard init"]
InitMonitor --> EpochStart([" Start Epoch Loop"])
%% Main training loop
EpochStart --> EpochNum{"/Epoch<br/>1-N"}
EpochNum -->|"Next Epoch"| BatchStart([" Start Batch Loop"])
BatchStart --> BatchNum{"/Batch<br/>1-M"}
BatchNum -->|"Next Batch"| LoadBatch[" Load Batch<br/>• Sample from loader<br/>• Move to device<br/>• Prepare tensors"]
LoadBatch --> Forward[" Forward Pass<br/>• inputs model<br/>• Logits: [B, T, V]<br/>• Context length: T"]
Forward --> Loss[" Compute Loss<br/>• Cross-entropy<br/>• Average over batch<br/>• Normalize"]
Loss --> Backward["⬅ Backward Pass<br/>• Compute gradients<br/>• Backprop through graph<br/>• Accumulate"]
Backward --> Clip[" Gradient Clipping<br/>• Max norm: 1.0<br/>• Prevent exploding grads<br/>• Check L2 norm"]
Clip --> OptStep[" Optimizer Step<br/>• Update weights<br/>• Apply learning rate<br/>• Update momentum"]
OptStep --> ZeroGrad[" Zero Gradients<br/>• Reset for next iter"]
ZeroGrad --> LogBatch[" Log Batch Metrics<br/>• Loss: X.XXX<br/>• Accuracy: XX%<br/>• Learning rate: 1e-4"]
LogBatch --> BatchCheck{"More<br/>Batches?"}
BatchCheck -->|" Yes"| BatchNum
BatchCheck -->|" No"| EvalVal[" Evaluate Validation<br/>• Forward on val set<br/>• Compute val loss<br/>• Compute metrics<br/>• No backprop"]
EvalVal --> ValMetrics[" Validation Metrics<br/>• Val loss: Y.YYY<br/>• Val accuracy: YY%<br/>• Best so far?"]
ValMetrics --> IsBest{"Better than<br/>Best?"}
IsBest -->|" Yes"| SaveBest[" Save Best Model<br/>• Save weights<br/>• Save optimizer state<br/>• Save epoch info"]
IsBest -->|" No"| CheckEarly[" Check Early Stop<br/>• No improvement<br/>• for N epochs?"]
SaveBest --> CheckEarly
CheckEarly --> StopEarly{"/No improve<br/>N epochs?"}
StopEarly -->|" Stop"| StopEpoch[" Early Stopping<br/>Trigger"]
StopEarly -->|" Continue"| LRSchedule[" Update LR Schedule<br/>• Cosine schedule<br/>• Decay: 1 - t/T<br/>• New LR: 4.9e-4"]
LRSchedule --> EpochCheck{"More<br/>Epochs?"}
EpochCheck -->|" Yes"| EpochNum
EpochCheck -->|" No"| EpochDone[" Training Done"]
StopEpoch --> EpochDone
EpochDone --> LoadBest[" Load Best Model<br/>• Restore weights<br/>• Optimizer state"]
LoadBest --> EvalTest[" Evaluate Test<br/>• Forward on test set<br/>• Final metrics<br/>• No backprop"]
EvalTest --> TestMetrics[" Final Test Metrics<br/>• Test loss: Z.ZZZ<br/>• Test accuracy: ZZ%<br/>• Precision: 0.XX<br/>• F1: 0.XX"]
TestMetrics --> SaveFinal[" Save Final Checkpoint<br/>• Weights<br/>• Config<br/>• Metadata<br/>• Hyperparameters<br/>• Training stats"]
SaveFinal --> UploadCloud[" Upload to Cloud<br/>• S3 bucket<br/>• Versioned<br/>• Metadata tagged"]
UploadCloud --> LogExp[" Log Experiment<br/>• MLflow: run params<br/>• Metrics & artifacts<br/>• Artifacts: weights<br/>• Tags: version, date"]
LogExp --> NotifyGH[" Notify GitHub<br/>• PR comment<br/>• 'Training complete'<br/>• Metrics summary<br/>• Checkpoint link"]
NotifyGH --> End([" Training Complete"])
CfgError -.error.-> End
%% Error handling
Forward -.catch.-> ErrorHandle[" Handle Error<br/>• Log error<br/>• Save checkpoint<br/>• Alert user<br/>• Cleanup"]
ErrorHandle --> End
%% Styling
style Start fill:#10b981,stroke:#059669,stroke-width:3px,color:#fff
style End fill:#10b981,stroke:#059669,stroke-width:3px,color:#fff
style LoadCfg fill:#3b82f6,stroke:#1e40af,stroke-width:2px,color:#fff
style LoadModel fill:#3b82f6,stroke:#1e40af,stroke-width:2px,color:#fff
style Optimizer fill:#3b82f6,stroke:#1e40af,stroke-width:2px,color:#fff
style PrepData fill:#f59e0b,stroke:#d97706,stroke-width:2px,color:#fff
style SplitData fill:#f59e0b,stroke:#d97706,stroke-width:2px,color:#fff
style Forward fill:#16a34a,stroke:#15803d,stroke-width:2px,color:#fff
style Backward fill:#16a34a,stroke:#15803d,stroke-width:2px,color:#fff
style OptStep fill:#16a34a,stroke:#15803d,stroke-width:2px,color:#fff
style Loss fill:#dc2626,stroke:#991b1b,stroke-width:2px,color:#fff
style Clip fill:#dc2626,stroke:#991b1b,stroke-width:2px,color:#fff
style EvalVal fill:#7c3aed,stroke:#6d28d9,stroke-width:2px,color:#fff
style SaveBest fill:#7c3aed,stroke:#6d28d9,stroke-width:2px,color:#fff
style SaveFinal fill:#7c3aed,stroke:#6d28d9,stroke-width:2px,color:#fff
style UploadCloud fill:#0891b2,stroke:#0e7490,stroke-width:2px,color:#fff
style LogExp fill:#0891b2,stroke:#0e7490,stroke-width:2px,color:#fff
style NotifyGH fill:#0891b2,stroke:#0e7490,stroke-width:2px,color:#fff
Key Phases¶
1. Configuration & Data Loading¶
- Load and validate Hydra configuration
- Prepare train/val/test datasets
- Configure optimizer and scheduler
2. Training Loop¶
- Per Epoch:
- Process batches with forward/backward
- Accumulate gradients
-
Apply optimizer step
-
Per Batch:
- Load data to device
- Forward pass
- Compute loss
- Backward pass
- Gradient clipping
- Optimizer step
3. Validation & Checkpointing¶
- Evaluate on validation set
- Compare with best model
- Save if improved
- Check for early stopping
4. Model Evaluation¶
- Evaluate on test set
- Compute final metrics
- Save final checkpoint
5. Logging & Integration¶
- Upload to cloud storage
- Log to MLflow
- Notify GitHub PR
Hyperparameter Configuration¶
# Configuration used in training
model:
architecture: gpt2
parameters: 124M
hidden_size: 768
num_layers: 12
num_heads: 12
training:
epochs: 30
batch_size: 32
learning_rate: 5e-4
weight_decay: 0.01
warmup_steps: 1000
max_grad_norm: 1.0
optimizer:
algorithm: adamw
betas: [0.9, 0.999]
eps: 1e-8
scheduler:
type: cosine
total_steps: 100000
warmup_ratio: 0.1
evaluation:
val_split: 0.15
test_split: 0.15
early_stopping_patience: 5
Typical Training Metrics¶
Epoch 1/30
Batch 1: loss=4.523, acc=0.12, lr=5.0e-4
Batch 2: loss=4.189, acc=0.18, lr=5.0e-4
...
Batch 100: loss=3.234, acc=0.35, lr=5.0e-4
Val: loss=3.156, acc=0.37 NEW BEST
Epoch 2/30
...
Val: loss=3.089, acc=0.41 NEW BEST
Epoch 15/30 (trained 6 hours)
...
Val: loss=2.123, acc=0.72 NEW BEST
Epoch 20/30
...
Val: loss=2.145, acc=0.71 No improvement (5/5)
Early stopping triggered
Final Test Results:
Test loss: 2.134
Test accuracy: 0.71
Precision: 0.73
Recall: 0.70
F1: 0.71
File Artifacts¶
| File | Purpose | Location |
|---|---|---|
| Checkpoint | Model weights + optimizer state | checkpoints/model-epoch-15.pt |
| Config | Final resolved configuration | checkpoints/config.yaml |
| Metrics | Per-epoch metrics JSON | logs/metrics.json |
| Logs | Training logs | logs/training.log |
| Tensorboard | Visualizations | logs/tensorboard/ |
Performance Characteristics¶
| Metric | Value | Notes |
|---|---|---|
| Time per epoch | ~2 hours | 124M params, GPU |
| Time per batch | ~45ms | Batch size 32 |
| GPU memory | ~24GB | A100, FP32 |
| Total time | ~30 hours | 15 epochs (with early stop) |
| Data throughput | ~700 samples/sec | GPU bound |
Next Steps¶
- Review evaluation workflow implementation in the codebase
- Explore model serving configuration
- See E2E Request Flow for full request lifecycle
Related Documentation: - 5-Layer Architecture - System architecture - Review training guides in the repository - Explore configuration management patterns - Check model management implementation