Skip to content

Test Coverage Roadmap: Path from 0% to 100%

Last Updated: 2026-07-11 Version: v0.2.0

Table of Contents

Created: 2026-01-18 **Updated: 2026-07-11 Current Baseline: ~27.5% (estimated 196 of 714 modules covered) Target: 100% test coverage Status: Phase 14.0 Complete | Phase 14.1 Executing


Autonomous Execution Readiness

Planset Status

Phase Planset Ready Execution Status
Phase 14.0 Complete DONE
Phase 14.1 Ready EXECUTING
Phase 14.2 Ready Queued
Phase 14.3 Ready Queued
Phase 14.4 Ready Queued

AI Agency Policy Compliance Checklist

  • Leave codebase better than found
  • Address ALL issues (pre-existing + new)
  • Plan before execution (this document)
  • Use pre-commit/commit terminology
  • Document ALL utilities created
  • 5+ self-review iterations
  • AfterMath/PDA loop integration

Executive Summary

This roadmap provides a structured, phased approach to achieve 100% test coverage across the entire codebase. The plan is divided into phases with increasing coverage thresholds, prioritizing critical modules first.

Current State: - Total source files: 714 - Files with tests: 196 (~27.5%) - Files without tests: 518 - Coverage threshold: Temporarily at 0% (Phase 14 foundation)


Phase Overview

Phase Target Coverage Duration Focus Area Threshold Setting
14.0 (Current) 0% 30% 2 phases Foundation & Infrastructure fail_under = 0
14.1 30% 50% 3 phases Core Modules (codex, codex_ml) fail_under = 30
14.2 50% 70% 4 phases Training, Agents, RAG fail_under = 50
14.3 70% 85% 3 phases Integration & Edge Cases fail_under = 70
14.4 85% 100% 3 phases Final Gaps & Branch Coverage fail_under = 85
Post-14 100% Ongoing Maintenance & Regression fail_under = 100

Total Estimated Duration: 15 phases


Phase 14.0: Foundation (Current Phase)

Goal: Establish test infrastructure and baseline coverage Duration: 2 phases Threshold: fail_under = 0 Status: COMPLETE

Completed Items

  • Fix CI/CD pipeline issues (Rust compilation, pytest plugin discovery)
  • Align Python version matrix with project requirements (3.11+)
  • Fix pytest plugin discovery in xdist workers
  • Create coverage analysis baseline (.codex/qa_walkthrough/coverage_analysis.json)
  • Create test priority matrix (.codex/qa_walkthrough/test_priority_matrix.json)
  • Create test templates (tests/templates/)
  • Create shared fixtures (tests/conftest_shared.py)
  • Document test patterns (docs/testing/TEST_PATTERNS.md)
  • Fix yamllint issues per AI Agency Policy
  • Create coverage roadmap (this document)

Exit Criteria - ALL MET

  • All CI jobs passing
  • Coverage baseline established (27.5%)
  • Test infrastructure documented
  • Templates and fixtures ready

Phase 14.1: Core Modules (30% 50%)

Goal: Cover high-impact core modules Duration: 3 phases Threshold: fail_under = 30 Status: IN PROGRESS

Priority Modules (High Impact)

Module Files Current Tests Target Impact Status
src/codex/logging/ 8 2 8 Critical - session logging
src/codex/cli/ 3 1 3 Critical - entry points
src/codex/utils/ 5 1 5 High - utility functions
src/codex_ml/config/ 3 0 3 High - configuration
src/codex_ml/cli/ 25 3 15 High - CLI commands
src/codex_ml/data/ 18 4 12 High - data loading

Top 10 Priority Modules (from Priority Matrix)

Based on .codex/qa_walkthrough/test_priority_matrix.json:

  1. src/codex_ml/training/unified_training.py - Score: 100
  2. src/codex_ml/cli/main.py - Score: 100
  3. src/codex_ml/safety/moderation.py - Score: 90
  4. src/codex_ml/data/loader.py - Score: 90
  5. src/codex_ml/data/validation.py - Score: 90
  6. src/codex/cli/main.py - Score: 90
  7. src/codex/auth/oauth_manager.py - Score: 90
  8. src/codex/security/storage.py - Score: 90
  9. src/codex_ml/training/legacy_api.py - Score: 80
  10. src/codex_ml/monitoring/codex_logging.py - Score: 80

Test Categories to Add

  1. Unit Tests for Core Functions (~100 tests)
  2. Configuration loading and validation
  3. Data serialization/deserialization
  4. CLI command parsing
  5. Logging functionality

  6. Integration Tests (~30 tests)

  7. CLI end-to-end flows
  8. Config execution paths
  9. Data pipeline integration

Coverage Targets by Module Group

src/codex/logging/ 80%
src/codex/cli/ 70%
src/codex/utils/ 70%
src/codex_ml/config/ 80%
src/codex_ml/cli/ 60%
src/codex_ml/data/ 70%

Exit Criteria

  • Overall coverage ≥ 50%
  • All priority modules have basic test coverage
  • No critical paths untested

Phase 14.2: Training, Agents & RAG (50% 70%)

Goal: Cover ML training pipeline, agents, and RAG system Duration: 4 phases Threshold: fail_under = 50

Priority Modules

Module Files Current Tests Target Impact
src/codex_ml/training/ 18 2 15 Critical - training loop
src/codex/rag/ 24 4 20 Critical - RAG system
agents/ 33 1 20 High - agent framework
training/ 13 0 10 High - training utils
src/codex_ml/models/ 10 2 8 High - model loading

Test Categories to Add

  1. Training Pipeline Tests (~80 tests)
  2. Model initialization
  3. Training loop execution
  4. Checkpoint save/load
  5. Early stopping logic
  6. Distributed training mocks

  7. RAG System Tests (~100 tests)

  8. See detailed plan in docs/PLAN_100_PERCENT_COVERAGE.md
  9. Embeddings providers
  10. Index building
  11. Retrieval accuracy
  12. Cache management

  13. Agent Tests (~50 tests)

  14. Agent initialization
  15. Message handling
  16. Memory management
  17. Orchestrator logic

Coverage Targets by Module Group

src/codex_ml/training/ 80%
src/codex/rag/ 90%
agents/ 60%
training/ 70%
src/codex_ml/models/ 80%

Exit Criteria

  • Overall coverage ≥ 70%
  • Training pipeline fully tested
  • RAG system at 90%+ coverage
  • Agent framework has basic coverage

Phase 14.3: Integration & Edge Cases (70% 85%)

Goal: Cover integration paths and edge cases Duration: 3 phases Threshold: fail_under = 70

Focus Areas

  1. Cross-Module Integration Tests (~60 tests)
  2. Config Training Evaluation flow
  3. RAG Agent Response flow
  4. CLI Core Output flow

  5. Error Handling & Edge Cases (~80 tests)

  6. Invalid inputs
  7. Network failures
  8. Resource exhaustion
  9. Concurrent access
  10. Platform-specific paths

  11. Remaining Module Coverage (~100 tests)

  12. src/codex_ml/serving/
  13. src/codex_ml/monitoring/
  14. src/codex/security/
  15. src/codex/quantum_orchestrator/

Coverage Targets by Module Group

src/codex_ml/serving/ 80%
src/codex_ml/monitoring/ 75%
src/codex/security/ 90%
src/codex/quantum_orchestrator/ 70%

Exit Criteria

  • Overall coverage ≥ 85%
  • All critical error paths tested
  • Integration tests passing
  • No major modules below 60%

Phase 14.4: Final Gaps & Branch Coverage (85% 100%)

Goal: Achieve 100% line and branch coverage Duration: 3 phases Threshold: fail_under = 85

Strategy

  1. Coverage Gap Analysis

    pytest --cov=src --cov-report=term-missing --cov-branch
    

  2. Targeted Test Generation

  3. Use coverage reports to identify uncovered lines
  4. Add tests for each uncovered branch
  5. Document any # pragma: no cover exclusions

  6. Documentation Example Tests

  7. Validate all code examples in docs
  8. Add tests for README examples
  9. Test CLI help examples

Final Push Categories

Category Estimated Tests Target
Branch coverage gaps 100 100% branch
Exception handlers 50 All exceptions tested
Documentation examples 30 All examples validated
Platform-specific code 20 Mocked appropriately

Exit Criteria

  • Line coverage = 100%
  • Branch coverage = 100%
  • All documentation examples tested
  • CI enforcing 100% threshold

Implementation Guidelines

Test File Structure

tests/
 conftest.py # Shared fixtures
 conftest_shared.py # Additional fixtures
 unit/ # Unit tests
 test_codex_ml/ # codex_ml module tests
 test_codex/ # codex module tests
 test_agents/ # agents module tests
 integration/ # Integration tests
 test_cli_flows.py
 test_training_pipeline.py
 test_rag_pipeline.py
 e2e/ # End-to-end tests
 test_full_workflow.py
 test_cli_e2e.py
 test_rag_*.py # Existing RAG tests

Test Naming Convention

def test_<module>_<function>_<scenario>():
 """Test description."""
 pass

# Examples:
def test_indexer_chunk_text_empty_string():
def test_retriever_query_network_timeout():
def test_training_loop_early_stopping():

Coverage Configuration

pyproject.toml updates by phase:

# Phase 14.0 (Current)
[tool.coverage.report]
fail_under = 0

# Phase 14.1
fail_under = 30

# Phase 14.2
fail_under = 50

# Phase 14.3
fail_under = 70

# Phase 14.4
fail_under = 85

# Post-14 (Final)
fail_under = 100

Workflow Updates

Each phase should update the CI workflow thresholds:

# .github/workflows/test-rag.yml
- name: Run tests with coverage
 run: |
 pytest tests/ \
 --cov=src \
 --cov-report=xml \
 --cov-report=html \
 --cov-fail-under=${PHASE_THRESHOLD}

Priority Test Proposals

Based on .codex/qa_walkthrough/coverage_analysis.json:

TP-001: Unit Tests for High-Priority Modules

  • Target: 390 untested high-priority modules
  • Estimated Impact: +20-30% coverage
  • Tests to Add: ~400

TP-002: Integration Tests for Cross-Module Interactions

  • Target: Critical workflows
  • Estimated Impact: +10-15% coverage
  • Tests to Add: ~100

TP-003: E2E Tests for Critical Workflows

  • Target: User-facing features
  • Estimated Impact: +5-10% coverage
  • Tests to Add: ~50

Monitoring & Tracking

Coverage Dashboard

Track progress using: 1. Codecov integration - PR coverage diffs 2. HTML reports - Detailed line-by-line coverage 3. CI badges - Current coverage status

per-phase Check-ins

Week Expected Coverage Actual Coverage Status
1 30% -
2 35% -
3 40% -
4 50% -
... ... ...
15 100% -

Risk Mitigation

Common Blockers & Solutions

Risk Mitigation
Flaky tests Use pytest-rerunfailures, isolate external dependencies
Slow tests Parallelize with pytest-xdist, use fixtures
Coverage plateaus Focus on branch coverage, use mutation testing
External dependencies Mock external services, use VCR cassettes

Exclusion Guidelines

Only exclude code with valid reasons:

# Acceptable exclusions:
if TYPE_CHECKING: # pragma: no cover
 ...

# Unacceptable exclusions:
if complex_condition: # pragma: no cover # Don't do this
 important_logic()

Success Metrics

Phase Completion Checklist

  • Phase 14.0: Coverage infrastructure working, baseline established
  • Phase 14.1: 50% coverage achieved, core modules tested
  • Phase 14.2: 70% coverage achieved, ML pipeline tested
  • Phase 14.3: 85% coverage achieved, integration tested
  • Phase 14.4: 100% coverage achieved, all gaps closed

Quality Indicators

Metric Target
Line Coverage 100%
Branch Coverage 100%
Test Count 1000+
Test Duration < 10 min (parallel)
Flaky Test Rate < 1%

References


Next Steps

  1. Fix current CI failures (xdist plugin discovery)
  2. Verify all existing tests pass
  3. Phase 14.0 Complete - Foundation established
  4. Begin Phase 14.1 - Core modules unit tests
  5. Update coverage threshold to 30% after Phase 14.1

Phase 14.1 Implementation Checklist

CLI Tests (55+ tests): - [x] tests/cli/test_main_coverage.py - 25+ tests for src/codex_ml/cli/main.py CREATED - [x] tests/cli/test_train_coverage.py - 15+ tests for src/codex_ml/cli/train.py CREATED - [x] tests/cli/test_metrics_coverage.py - 10+ tests for src/codex_ml/cli/metrics_cli.py CREATED - [x] tests/cli/test_hydra_coverage.py - 10+ tests for src/codex_ml/cli/hydra_main.py CREATED

Data Tests (60+ tests): - [x] tests/data/test_loader_coverage.py - 30+ tests for src/codex_ml/data/loader.py CREATED - [x] tests/data/test_validation_coverage.py - 20+ tests for src/codex_ml/data/validation.py CREATED - [x] tests/data/test_split_coverage.py - 15+ tests for src/codex_ml/data/split.py CREATED

Training Tests (65+ tests): - [x] tests/training/test_unified_coverage.py - 35+ tests for src/codex_ml/training/unified_training.py CREATED - [x] tests/training/test_legacy_coverage.py - 20+ tests for src/codex_ml/training/legacy_api.py CREATED - [x] tests/training/test_strategies_coverage.py - 15+ tests for src/codex_ml/training/strategies.py CREATED

Phase 14.1 Progress Summary:

Category Target Created %
CLI Tests 55+ 60+ 100%
Data Tests 60+ 65+ 100%
Training Tests 65+ 70+ 100%
Total 180+ 195+ 100%

Phase 14.2: Security Hardening - COMPLETE

Security Tests Progress:

Category Target Created %
CVE Monitor Tests 15+ 20+ 100%
Denylist Tests 15+ 20+ 100%
Sanitizers Tests 20+ 25+ 100%
Moderation Tests 20+ 25+ 100%
Total 70+ 90+ 100%

Files Created: - [x] tests/security/test_cve_monitor_coverage.py - 20+ tests - [x] tests/security/test_denylist_coverage.py - 20+ tests - [x] tests/safety/test_sanitizers_coverage.py - 25+ tests - [x] tests/safety/test_moderation_coverage.py - 25+ tests


Phase 14.3: Integration & Edge Cases - COMPLETE

Integration Tests Progress:

Category Target Created %
Cross-Module Integration 30+ 35+ 100%
Edge Case & Boundary 40+ 45+ 100%
Total 70+ 80+ 100%

Files Created: - [x] tests/integration/test_phase14_cross_module_coverage.py - 35+ tests - [x] tests/integration/test_phase14_edge_cases_coverage.py - 45+ tests

Owner: Phase 14 implementation team Review Cadence: per-phase coverage review