Test Coverage Roadmap: Path from 0% to 100%¶
Last Updated: 2026-07-11 Version: v0.2.0
Table of Contents¶
- Autonomous Execution Readiness
- Planset Status
- AI Agency Policy Compliance Checklist
- Executive Summary
- Phase Overview
- Phase 14.0: Foundation (Current Phase)
- Completed Items
- Exit Criteria - ALL MET
- Phase 14.1: Core Modules (30% 50%)
- Priority Modules (High Impact)
- Top 10 Priority Modules (from Priority Matrix)
- Test Categories to Add
- Coverage Targets by Module Group
- Exit Criteria
- Phase 14.2: Training, Agents & RAG (50% 70%)
- Priority Modules
- Test Categories to Add
- Coverage Targets by Module Group
- Exit Criteria
- Phase 14.3: Integration & Edge Cases (70% 85%)
- Focus Areas
- Coverage Targets by Module Group
- Exit Criteria
- Phase 14.4: Final Gaps & Branch Coverage (85% 100%)
- Strategy
- Final Push Categories
- Exit Criteria
- Implementation Guidelines
- Test File Structure
- Test Naming Convention
- Examples:
- Coverage Configuration
- Phase 14.0 (Current)
- Phase 14.1
- Phase 14.2
- Phase 14.3
- Phase 14.4
- Post-14 (Final)
- Workflow Updates
- .github/workflows/test-rag.yml
- Priority Test Proposals
- TP-001: Unit Tests for High-Priority Modules
- TP-002: Integration Tests for Cross-Module Interactions
- TP-003: E2E Tests for Critical Workflows
- Monitoring & Tracking
- Coverage Dashboard
- per-phase Check-ins
- Risk Mitigation
- Common Blockers & Solutions
- Exclusion Guidelines
- Acceptable exclusions:
- Unacceptable exclusions:
- Success Metrics
- Phase Completion Checklist
- Quality Indicators
- References
- Next Steps
- Phase 14.1 Implementation Checklist
- Phase 14.2: Security Hardening - COMPLETE
- Phase 14.3: Integration & Edge Cases - COMPLETE
Created: 2026-01-18 **Updated: 2026-07-11 Current Baseline: ~27.5% (estimated 196 of 714 modules covered) Target: 100% test coverage Status: Phase 14.0 Complete | Phase 14.1 Executing
Autonomous Execution Readiness¶
Planset Status¶
| Phase | Planset Ready | Execution Status |
|---|---|---|
| Phase 14.0 | Complete | DONE |
| Phase 14.1 | Ready | EXECUTING |
| Phase 14.2 | Ready | Queued |
| Phase 14.3 | Ready | Queued |
| Phase 14.4 | Ready | Queued |
AI Agency Policy Compliance Checklist¶
- Leave codebase better than found
- Address ALL issues (pre-existing + new)
- Plan before execution (this document)
- Use pre-commit/commit terminology
- Document ALL utilities created
- 5+ self-review iterations
- AfterMath/PDA loop integration
Executive Summary¶
This roadmap provides a structured, phased approach to achieve 100% test coverage across the entire codebase. The plan is divided into phases with increasing coverage thresholds, prioritizing critical modules first.
Current State: - Total source files: 714 - Files with tests: 196 (~27.5%) - Files without tests: 518 - Coverage threshold: Temporarily at 0% (Phase 14 foundation)
Phase Overview¶
| Phase | Target Coverage | Duration | Focus Area | Threshold Setting |
|---|---|---|---|---|
| 14.0 (Current) | 0% 30% | 2 phases | Foundation & Infrastructure | fail_under = 0 |
| 14.1 | 30% 50% | 3 phases | Core Modules (codex, codex_ml) | fail_under = 30 |
| 14.2 | 50% 70% | 4 phases | Training, Agents, RAG | fail_under = 50 |
| 14.3 | 70% 85% | 3 phases | Integration & Edge Cases | fail_under = 70 |
| 14.4 | 85% 100% | 3 phases | Final Gaps & Branch Coverage | fail_under = 85 |
| Post-14 | 100% | Ongoing | Maintenance & Regression | fail_under = 100 |
Total Estimated Duration: 15 phases
Phase 14.0: Foundation (Current Phase)¶
Goal: Establish test infrastructure and baseline coverage
Duration: 2 phases
Threshold: fail_under = 0
Status: COMPLETE
Completed Items¶
- Fix CI/CD pipeline issues (Rust compilation, pytest plugin discovery)
- Align Python version matrix with project requirements (3.11+)
- Fix pytest plugin discovery in xdist workers
- Create coverage analysis baseline (
.codex/qa_walkthrough/coverage_analysis.json) - Create test priority matrix (
.codex/qa_walkthrough/test_priority_matrix.json) - Create test templates (
tests/templates/) - Create shared fixtures (
tests/conftest_shared.py) - Document test patterns (
docs/testing/TEST_PATTERNS.md) - Fix yamllint issues per AI Agency Policy
- Create coverage roadmap (this document)
Exit Criteria - ALL MET¶
- All CI jobs passing
- Coverage baseline established (27.5%)
- Test infrastructure documented
- Templates and fixtures ready
Phase 14.1: Core Modules (30% 50%)¶
Goal: Cover high-impact core modules
Duration: 3 phases
Threshold: fail_under = 30
Status: IN PROGRESS
Priority Modules (High Impact)¶
| Module | Files | Current Tests | Target | Impact | Status |
|---|---|---|---|---|---|
src/codex/logging/ |
8 | 2 | 8 | Critical - session logging | |
src/codex/cli/ |
3 | 1 | 3 | Critical - entry points | |
src/codex/utils/ |
5 | 1 | 5 | High - utility functions | |
src/codex_ml/config/ |
3 | 0 | 3 | High - configuration | |
src/codex_ml/cli/ |
25 | 3 | 15 | High - CLI commands | |
src/codex_ml/data/ |
18 | 4 | 12 | High - data loading |
Top 10 Priority Modules (from Priority Matrix)¶
Based on .codex/qa_walkthrough/test_priority_matrix.json:
src/codex_ml/training/unified_training.py- Score: 100src/codex_ml/cli/main.py- Score: 100src/codex_ml/safety/moderation.py- Score: 90src/codex_ml/data/loader.py- Score: 90src/codex_ml/data/validation.py- Score: 90src/codex/cli/main.py- Score: 90src/codex/auth/oauth_manager.py- Score: 90src/codex/security/storage.py- Score: 90src/codex_ml/training/legacy_api.py- Score: 80src/codex_ml/monitoring/codex_logging.py- Score: 80
Test Categories to Add¶
- Unit Tests for Core Functions (~100 tests)
- Configuration loading and validation
- Data serialization/deserialization
- CLI command parsing
-
Logging functionality
-
Integration Tests (~30 tests)
- CLI end-to-end flows
- Config execution paths
- Data pipeline integration
Coverage Targets by Module Group¶
src/codex/logging/ 80%
src/codex/cli/ 70%
src/codex/utils/ 70%
src/codex_ml/config/ 80%
src/codex_ml/cli/ 60%
src/codex_ml/data/ 70%
Exit Criteria¶
- Overall coverage ≥ 50%
- All priority modules have basic test coverage
- No critical paths untested
Phase 14.2: Training, Agents & RAG (50% 70%)¶
Goal: Cover ML training pipeline, agents, and RAG system
Duration: 4 phases
Threshold: fail_under = 50
Priority Modules¶
| Module | Files | Current Tests | Target | Impact |
|---|---|---|---|---|
src/codex_ml/training/ |
18 | 2 | 15 | Critical - training loop |
src/codex/rag/ |
24 | 4 | 20 | Critical - RAG system |
agents/ |
33 | 1 | 20 | High - agent framework |
training/ |
13 | 0 | 10 | High - training utils |
src/codex_ml/models/ |
10 | 2 | 8 | High - model loading |
Test Categories to Add¶
- Training Pipeline Tests (~80 tests)
- Model initialization
- Training loop execution
- Checkpoint save/load
- Early stopping logic
-
Distributed training mocks
-
RAG System Tests (~100 tests)
- See detailed plan in
docs/PLAN_100_PERCENT_COVERAGE.md - Embeddings providers
- Index building
- Retrieval accuracy
-
Cache management
-
Agent Tests (~50 tests)
- Agent initialization
- Message handling
- Memory management
- Orchestrator logic
Coverage Targets by Module Group¶
Exit Criteria¶
- Overall coverage ≥ 70%
- Training pipeline fully tested
- RAG system at 90%+ coverage
- Agent framework has basic coverage
Phase 14.3: Integration & Edge Cases (70% 85%)¶
Goal: Cover integration paths and edge cases
Duration: 3 phases
Threshold: fail_under = 70
Focus Areas¶
- Cross-Module Integration Tests (~60 tests)
- Config Training Evaluation flow
- RAG Agent Response flow
-
CLI Core Output flow
-
Error Handling & Edge Cases (~80 tests)
- Invalid inputs
- Network failures
- Resource exhaustion
- Concurrent access
-
Platform-specific paths
-
Remaining Module Coverage (~100 tests)
src/codex_ml/serving/src/codex_ml/monitoring/src/codex/security/src/codex/quantum_orchestrator/
Coverage Targets by Module Group¶
src/codex_ml/serving/ 80%
src/codex_ml/monitoring/ 75%
src/codex/security/ 90%
src/codex/quantum_orchestrator/ 70%
Exit Criteria¶
- Overall coverage ≥ 85%
- All critical error paths tested
- Integration tests passing
- No major modules below 60%
Phase 14.4: Final Gaps & Branch Coverage (85% 100%)¶
Goal: Achieve 100% line and branch coverage
Duration: 3 phases
Threshold: fail_under = 85
Strategy¶
-
Coverage Gap Analysis
-
Targeted Test Generation
- Use coverage reports to identify uncovered lines
- Add tests for each uncovered branch
-
Document any
# pragma: no coverexclusions -
Documentation Example Tests
- Validate all code examples in docs
- Add tests for README examples
- Test CLI help examples
Final Push Categories¶
| Category | Estimated Tests | Target |
|---|---|---|
| Branch coverage gaps | 100 | 100% branch |
| Exception handlers | 50 | All exceptions tested |
| Documentation examples | 30 | All examples validated |
| Platform-specific code | 20 | Mocked appropriately |
Exit Criteria¶
- Line coverage = 100%
- Branch coverage = 100%
- All documentation examples tested
- CI enforcing 100% threshold
Implementation Guidelines¶
Test File Structure¶
tests/
conftest.py # Shared fixtures
conftest_shared.py # Additional fixtures
unit/ # Unit tests
test_codex_ml/ # codex_ml module tests
test_codex/ # codex module tests
test_agents/ # agents module tests
integration/ # Integration tests
test_cli_flows.py
test_training_pipeline.py
test_rag_pipeline.py
e2e/ # End-to-end tests
test_full_workflow.py
test_cli_e2e.py
test_rag_*.py # Existing RAG tests
Test Naming Convention¶
def test_<module>_<function>_<scenario>():
"""Test description."""
pass
# Examples:
def test_indexer_chunk_text_empty_string():
def test_retriever_query_network_timeout():
def test_training_loop_early_stopping():
Coverage Configuration¶
pyproject.toml updates by phase:
# Phase 14.0 (Current)
[tool.coverage.report]
fail_under = 0
# Phase 14.1
fail_under = 30
# Phase 14.2
fail_under = 50
# Phase 14.3
fail_under = 70
# Phase 14.4
fail_under = 85
# Post-14 (Final)
fail_under = 100
Workflow Updates¶
Each phase should update the CI workflow thresholds:
# .github/workflows/test-rag.yml
- name: Run tests with coverage
run: |
pytest tests/ \
--cov=src \
--cov-report=xml \
--cov-report=html \
--cov-fail-under=${PHASE_THRESHOLD}
Priority Test Proposals¶
Based on .codex/qa_walkthrough/coverage_analysis.json:
TP-001: Unit Tests for High-Priority Modules¶
- Target: 390 untested high-priority modules
- Estimated Impact: +20-30% coverage
- Tests to Add: ~400
TP-002: Integration Tests for Cross-Module Interactions¶
- Target: Critical workflows
- Estimated Impact: +10-15% coverage
- Tests to Add: ~100
TP-003: E2E Tests for Critical Workflows¶
- Target: User-facing features
- Estimated Impact: +5-10% coverage
- Tests to Add: ~50
Monitoring & Tracking¶
Coverage Dashboard¶
Track progress using: 1. Codecov integration - PR coverage diffs 2. HTML reports - Detailed line-by-line coverage 3. CI badges - Current coverage status
per-phase Check-ins¶
| Week | Expected Coverage | Actual Coverage | Status |
|---|---|---|---|
| 1 | 30% | - | |
| 2 | 35% | - | |
| 3 | 40% | - | |
| 4 | 50% | - | |
| ... | ... | ... | |
| 15 | 100% | - |
Risk Mitigation¶
Common Blockers & Solutions¶
| Risk | Mitigation |
|---|---|
| Flaky tests | Use pytest-rerunfailures, isolate external dependencies |
| Slow tests | Parallelize with pytest-xdist, use fixtures |
| Coverage plateaus | Focus on branch coverage, use mutation testing |
| External dependencies | Mock external services, use VCR cassettes |
Exclusion Guidelines¶
Only exclude code with valid reasons:
# Acceptable exclusions:
if TYPE_CHECKING: # pragma: no cover
...
# Unacceptable exclusions:
if complex_condition: # pragma: no cover # Don't do this
important_logic()
Success Metrics¶
Phase Completion Checklist¶
- Phase 14.0: Coverage infrastructure working, baseline established
- Phase 14.1: 50% coverage achieved, core modules tested
- Phase 14.2: 70% coverage achieved, ML pipeline tested
- Phase 14.3: 85% coverage achieved, integration tested
- Phase 14.4: 100% coverage achieved, all gaps closed
Quality Indicators¶
| Metric | Target |
|---|---|
| Line Coverage | 100% |
| Branch Coverage | 100% |
| Test Count | 1000+ |
| Test Duration | < 10 min (parallel) |
| Flaky Test Rate | < 1% |
References¶
- RAG 100% Coverage Plan
- Test Coverage Plan for RAG
- Coverage Analysis JSON
- pytest Documentation
- pytest-cov Documentation
Next Steps¶
- Fix current CI failures (xdist plugin discovery)
- Verify all existing tests pass
- Phase 14.0 Complete - Foundation established
- Begin Phase 14.1 - Core modules unit tests
- Update coverage threshold to 30% after Phase 14.1
Phase 14.1 Implementation Checklist¶
CLI Tests (55+ tests):
- [x] tests/cli/test_main_coverage.py - 25+ tests for src/codex_ml/cli/main.py CREATED
- [x] tests/cli/test_train_coverage.py - 15+ tests for src/codex_ml/cli/train.py CREATED
- [x] tests/cli/test_metrics_coverage.py - 10+ tests for src/codex_ml/cli/metrics_cli.py CREATED
- [x] tests/cli/test_hydra_coverage.py - 10+ tests for src/codex_ml/cli/hydra_main.py CREATED
Data Tests (60+ tests):
- [x] tests/data/test_loader_coverage.py - 30+ tests for src/codex_ml/data/loader.py CREATED
- [x] tests/data/test_validation_coverage.py - 20+ tests for src/codex_ml/data/validation.py CREATED
- [x] tests/data/test_split_coverage.py - 15+ tests for src/codex_ml/data/split.py CREATED
Training Tests (65+ tests):
- [x] tests/training/test_unified_coverage.py - 35+ tests for src/codex_ml/training/unified_training.py CREATED
- [x] tests/training/test_legacy_coverage.py - 20+ tests for src/codex_ml/training/legacy_api.py CREATED
- [x] tests/training/test_strategies_coverage.py - 15+ tests for src/codex_ml/training/strategies.py CREATED
Phase 14.1 Progress Summary:
| Category | Target | Created | % |
|---|---|---|---|
| CLI Tests | 55+ | 60+ | 100% |
| Data Tests | 60+ | 65+ | 100% |
| Training Tests | 65+ | 70+ | 100% |
| Total | 180+ | 195+ | 100% |
Phase 14.2: Security Hardening - COMPLETE¶
Security Tests Progress:
| Category | Target | Created | % |
|---|---|---|---|
| CVE Monitor Tests | 15+ | 20+ | 100% |
| Denylist Tests | 15+ | 20+ | 100% |
| Sanitizers Tests | 20+ | 25+ | 100% |
| Moderation Tests | 20+ | 25+ | 100% |
| Total | 70+ | 90+ | 100% |
Files Created:
- [x] tests/security/test_cve_monitor_coverage.py - 20+ tests
- [x] tests/security/test_denylist_coverage.py - 20+ tests
- [x] tests/safety/test_sanitizers_coverage.py - 25+ tests
- [x] tests/safety/test_moderation_coverage.py - 25+ tests
Phase 14.3: Integration & Edge Cases - COMPLETE¶
Integration Tests Progress:
| Category | Target | Created | % |
|---|---|---|---|
| Cross-Module Integration | 30+ | 35+ | 100% |
| Edge Case & Boundary | 40+ | 45+ | 100% |
| Total | 70+ | 80+ | 100% |
Files Created:
- [x] tests/integration/test_phase14_cross_module_coverage.py - 35+ tests
- [x] tests/integration/test_phase14_edge_cases_coverage.py - 45+ tests
Owner: Phase 14 implementation team Review Cadence: per-phase coverage review