Codebase Cognitive Map¶

Generated: 2026-06-22T08:42:00Z | Updated: 2026-06-22T18:02:00Z by @mbaetiong (S1325 variables audit) PR: #4731 | Session: S1325

🎯 Mission Overview¶

Objective: Provide a high-level cognitive map of the _codex_ repository including components, flows, dependencies, and operational context for AI agents and human contributors.

Energy Level: ⚡⚡⚡⚡ (4/5 - High Priority Reference Document)

Status: 🟢 Active

Last Updated: 2026-06-22T00:00:00Z | Version: 2.1.0 | Last Reviewed: 2026-06-22T18:02:00Z

Architecture Overview¶

Type: Modular ML/AI Platform with agent Orchestration MLOps Maturity: Level 4 (100/100 Azure MLOps) - Production Ready Stats: 21,500+ tests (100% passing), 10.7% coverage, 0 vulnerabilities, 145 active agents (post Phase-5 consolidation; registry total 159 — 14 archived), 285 workflow files under .github/workflows/ (includes dispatch-only stubs).

Repository Structure¶

_codex_/
├── src/              # Core application code
│   ├── codex/       # Ingestion pipeline (ingest→analyze→transform→verify)
│   ├── rag/         # RAG pipelines & retrieval
│   ├── verification/ # Chain-of-Verification (CoVe)
│   ├── mcp/         # Model Context Protocol adapters
│   └── tools/       # Tool registry
├── agents/          # Autonomous agents (workflow, quantum, physics)
├── scripts/         # Automation & utilities
│   └── mcp/        # ChatGPT Project packaging system
├── tests/           # 21,500+ test suite
├── docs/            # Documentation hub
│   ├── mcp/        # MCP packaging docs (93+ KB)
│   ├── system/     # Cognitive brain (this file)
│   └── capabilities/ # Capability guides
└── .github/         # CI/CD workflows & automation

Core Components¶

1. Codex Ingestion Pipeline (src/codex/)¶

Purpose: Complete Python code processing system

Commands:

python -m codex.cli ingest <source>      # Ingest code (file/ZIP/Git)
python -m codex.cli analyze <snapshot-id> # Static + runtime analysis
python -m codex.cli transform <snapshot-id> --tier A # Apply transformations
python -m codex.cli verify <snapshot-id> # Behavior verification

Flow: Source → Ingest → Analyze → Transform → Verify → PR

2. agent System (agents/)¶

Purpose: Autonomous AI agents with physics-inspired optimization

Key Agents: - workflow_navigator.py - Tokenized workflows (AUDIT_EXEC, DOC_GEN) - quantum_game_theory.py - Quantum-inspired decisions - physics_orchestrator.py - 6 physics paradigms - mental_mapping.py - Context tracking

Tokens: audit, decide, docs, organize, review, heal

Active count: 145 (post Phase-5 consolidation — see agents/AGENT_CONSOLIDATION_MATRIX.md). Registry total 159; 14 archived (5 prior coverage merges + 9 Phase-5 family merges); 1 prompt file (workflow-health-monitor.deprecated.md) archived without registry entry. Unified entry points: unified-coverage-agent, unified-doc-agent, unified-security-scanner, ci-testing-agent, ci-emergency-response-agent, cache-management-agent.

Unified entry points:

%%{init: {'accessibility': {'title': 'Flowchart showing unified-coverage-agent, unified-doc-agent'}}%%
graph LR
    subgraph "Unified Agents (canonical entry points)"
        UC[unified-coverage-agent]
        UD[unified-doc-agent]
        US[unified-security-scanner]
        UG[unified-governance-gate]
        WM[workflow-management-agent]
        CM[cache-management-agent]
        SH[self-healing-orchestrator-agent]
    end
    subgraph "Deprecated → Merged"
        D1[coverage-gapfill / -maintenance / -roadmap / test-coverage-agent / -monitor] --> UC
        D2[documentation-quality-agent / documentation-consolidator] --> UD
        D3[secret-detection / dep-vuln-scanner / dep-sec-review / security-audit] --> US
        D4[ci-failure-resolution-agent] --> SH
        D5[ci-resilience-emergency-response-agent] --> CMR[ci-emergency-response-agent]
        D6[cache-manager-integration] --> CM
    end
    UG --- WM
    SH --- WM
    CM --- WM

3. MCP Package System (scripts/mcp/)¶

Purpose: Package codebase for ChatGPT Projects

Commands:

./scripts/mcp/mcp-package --list              # List 9 topics
./scripts/mcp/mcp-package --topic agents      # Package by topic
./scripts/mcp/mcp-package --custom "patterns" # Custom patterns

Topics: zendesk, agents, quantum, docs, mcp, workflows, python_dev, testing, security

Output: Flat ZIP with manifest.json, README_dataset.md, index.md

Docs: docs/mcp/ - 93+ KB across 8 comprehensive guides

4. RAG & Verification (src/rag/, src/verification/)¶

• RAG pipelines: Chunking, embedding, retrieval
• CoVe: Chain-of-Verification fact-checking
• MCP adapters: Pinecone, Mock integrations

Data Flows¶

Code Ingestion¶

External Source → Ingest → Static Analysis → Runtime Analysis →
LLM Intent Inference → Transformation → Verification → PR Creation

agent workflow¶

Request → WorkflowNavigator → agent Orchestration →
Task Execution → Verification → State Persistence

MCP Packaging¶

Human Request → component Selection → File Flattening →
Manifest Generation → ZIP Creation → ChatGPT Upload

CI/CD¶

Git Push → Status Validation → Security Gates → Quality Gates →
Test Execution → Cache Management → Artifact Generation

Dependencies & Integrations¶

External Services¶

• OpenAI API: LLM intent inference (OPENAI_API_KEY)
• GitHub API: PR creation, workflows (GITHUB_TOKEN)
• Pinecone: Vector embeddings (optional)
• CodeQL/Semgrep: Security scanning

Python Dependencies¶

• Core: numpy, pandas, openai, httpx, pydantic, hydra-core
• Dev: pytest, black, ruff, mypy, nox, pre-commit
• ML/AI: torch, transformers, safetensors (optional)

CI/CD Pipeline¶

Key Workflows (.github/workflows/)¶

Cache Strategy (Phase 3C-Lite + Phase 5)¶

• Ruff: ~20-30 MB | MyPy: ~50-80 MB
• Pytest: ~30-50 MB | pre-commit: ~50-100 MB
• Total: 7.69 GB / 10 GB limit (23% buffer)
• Keys: ${{ runner.os }}-${{ github.workflow }}-<tool>-${{ hashFiles(...) }}
• Phase 5 introduces the 4-layer cache hierarchy and skip-rescan policy — see docs/workflows/CACHE_POLICY.md (owned by cache-management-agent).

Security & Secrets¶

Active Variable Inventory (2026-06-03 Audit)¶

Total: 113 variables/secrets across 5 scopes

%%{init: {'accessibility': {'title': 'Diagram showing CODEX_MASTER_KEY, CODEX_BACKUP_KEY'}}%%
graph TB
    subgraph "Org Secrets (13)"
        OM[CODEX_MASTER_KEY]
        OB[CODEX_BACKUP_KEY]
        OA[CODEX_ADMIN_KEY]
        OG[_GITHUB_APP_*]
        OP[PYPI_TOKEN / NPM_TOKEN]
        OH[HF_TOKEN / RAG_OPENAI_KEY]
        OC[CODECOV_TOKEN]
    end
    subgraph "Repo Secrets (7)"
        RS[OPENAI_API_KEY]
        RW[CODEX_WEBHOOK_SECRET]
        RG[CODEX_GHP_TOKEN_*]
        RB[_CODEX_BOT_RUNNER]
    end
    subgraph "Repo Variables (76)"
        RV1[agent/Autonomy: AGENT_KILL_SWITCH, AUTONOMY_*]
        RV2[Copilot: COPILOT_AGENT_*, COPILOT_WEC_*]
        RV3[Cognitive Brain: COGNITIVE_BRAIN_*]
        RV4[CI/CD: CODEX_CI_*, CODEX_COVERAGE_THRESHOLD]
        RV5[LLM/ML: CODEX_LLM_MODEL, WANDB_MODE]
        RV6[Runtime: CODEX_SESSION_ID, CODEX_LOG_LEVEL]
    end
    subgraph "Env Variables (14) — Sandbox"
        EV[CODEX_ENV_*versions, RUST_BACKTRACE, CARGO_TERM_COLOR]
    end
    subgraph "Env Secrets (3) — Sandbox"
        ES[CODEX_RUNNER_TOKEN, CODEX_ENVIRONMENT_RUNNER]
    end
    OM --> |token_chain| RS
    OB --> |fallback| RS
    RV2 --> |controls| EV

Diagram legend: token_chain = primary token source for write operations; fallback = secondary token source used only when primary is unavailable (CODEX_BACKUP_KEY fills in when CODEX_MASTER_KEY is absent).

Token Write Chain¶

GH_TOKEN = CODEX_MASTER_KEY || CODEX_BACKUP_KEY || github.token

• Defined in COPILOT_AGENT_PREFLIGHT_RULES.token_rule
• Use report_progress tool — never git push directly

Key Variable Relationships¶

Secrets (GitHub UI injected)¶

• OPENAI_API_KEY — OpenAI API for LLM calls
• CODEX_MASTER_KEY — Genesis Protocol / all write ops
• CODEX_BACKUP_KEY — Fallback write key
• _GITHUB_APP_PRIVATE_KEY — GitHub App authentication

Security Scanning¶

• Gitleaks, Trufflehog — Secret detection
• Semgrep SAST — Static analysis
• CodeQL — Code scanning

Anti-/tmp/ Protection¶

Policy: Use .github/tmp/ instead of /tmp/ for tracked artifacts Applied: emergency_cache_cleanup.sh, MCP tools Exception: CODEX_BRIDGE_DIR=/tmp/codex_secure_bridge is a runtime tmpfs mount (not tracked) Doc: docs/system/ANTI_TMP_PROTECTION_SYSTEM.md

agent Variable Expectations¶

Per-agent MUST/SHOULD variable requirements are documented in agents/VARIABLE_EXPECTATIONS.md. Key categories:

All agents universally MUST check AGENT_KILL_SWITCH at startup.

MCP & ChatGPT Integration¶

Packaging Capabilities¶

1. 9 Predefined Topics: All major capabilities covered
2. Custom Patterns: Glob-based file selection
3. Flat Structure: Optimized for ChatGPT
4. Metadata: SHA256, sizes, language detection
5. Navigation: Manifest-driven discovery

Methodology Transfer (8 Capabilities)¶

1. Python script development/deconstruction
2. workflow navigation & state management
3. Quantum game theory application
4. API integration patterns
5. CI/CD workflow optimization
6. agent-based architecture
7. TDD methodology
8. Documentation generation

Documentation (docs/mcp/)¶

• QUICK_START.md - 5-minute onboarding
• PACKAGING_GUIDE.md - Complete workflows
• PACKAGEABLE_CAPABILITIES.md - Capability transfer
• ChatGPT_Project_SYSTEM_PROMPT.md - AI prompt
• GENERIC_NAVIGATION_SYSTEM.md - Universal navigation
• ADVANCED_FEATURES_PLANSET.md - Roadmap (Future iterations)

Operational Context¶

GitHub Limits¶

• Copilot Pro+: 64K tokens/session
• GitHub Team: 10 GB cache, limited Actions minutes
• Current Cache: 7.69 GB (23% buffer)

Quality Metrics¶

• Tests: 21,500+ (100% passing)
• Coverage: 10.7% (ratchet roadmap targeting 80%+)
• Security: 0 vulnerabilities
• Cache Hit Rate: 90%+ projected

Performance Targets¶

• Test execution: <5 min
• Lint/type: <2 min
• Package creation: <2 min

Quick Reference¶

Common Commands¶

# Codex
python -m codex.cli ingest|analyze|transform|verify

# MCP
./scripts/mcp/mcp-package --list|--topic|--custom

# Testing
make docker-test
pytest tests/ --cov=src/

# Quality
nox -s lint|type|format

# agent
python -m scripts.space_traversal.audit_runner agent-interface

Entry Points¶

Navigation for AI Agents¶

Getting Started¶

1. Architecture: This doc → docs/ARCHITECTURE.md
2. Capabilities: docs/capabilities/*.md
3. Workflows: agents/TOKENIZED_WORKFLOWS.md
4. MCP: docs/mcp/QUICK_START.md
5. Contributing: docs/CONTRIBUTING.md

Finding Things¶

• Code: src/ (app), agents/ (agents)
• Tests: tests/ (mirrors src/)
• Scripts: scripts/ (automation)
• Docs: docs/ (organized by topic)
• CI/CD: .github/workflows/

Common Tasks¶

• New capability: docs/capabilities/ template
• Extend agents: agents/workflow_navigator.py
• Add CI: .github/workflows/ templates
• Package code: scripts/mcp/mcp-package
• Run tests: make docker-test

Related Documents¶

• Codebase Dashboard - Live status & next steps
• Roadmap - Feature roadmap & iterations
• Architecture - Detailed architecture
• Contributing - Contribution guide
• Admin Guide - Admin setup

Owner: DevOps + agent Development Team Review: Monthly or after major changes Last Reviewed: 2026-01-23T08:42:00Z

⚖️ Verification Checklist¶

Architecture Accuracy¶

• component structure matches current repository layout
• Data flows reflect actual implementation
• Dependencies list is up-to-date
• Integration points correctly documented

Documentation Quality¶

• All code examples are valid and tested
• Links to related documents are functional
• Tables render correctly in GitHub/browser
• Commands and paths are accurate

Currency¶

• Updated to reflect latest repository state (2026-01-23)
• Version number incremented (2.0.0)
• Iteration-based workflow language used throughout

📈 Success Metrics¶

⚛️ Physics Alignment¶

🧠 Redundancy Patterns¶

Navigation Redundancy: - Multiple access paths: By component, by workflow, by role - Cross-references between related sections - Both top-down and bottom-up navigation supported

Update Strategy: - Version-controlled documentation - Git history maintains all previous versions - Rollback available via commit history

⚡ Energy Distribution¶

Questions? → Dashboard