From 76c53cd6b73dbc3282850c75f2b86ecdc55da305 Mon Sep 17 00:00:00 2001 From: napinoco Date: Sat, 12 Jul 2025 23:55:04 +0900 Subject: [PATCH 01/13] Remove .DS_Store files and add macOS files to .gitignore MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Remove accidentally committed .DS_Store files from repository - Add .DS_Store, .AppleDouble, and .LSOverride to .gitignore - Prevent future macOS system files from being tracked πŸ€– Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude --- .DS_Store | Bin 6148 -> 0 bytes .gitignore | 5 +++++ docs/.DS_Store | Bin 6148 -> 0 bytes problems/.DS_Store | Bin 6148 -> 0 bytes 4 files changed, 5 insertions(+) delete mode 100644 .DS_Store delete mode 100644 docs/.DS_Store delete mode 100644 problems/.DS_Store diff --git a/.DS_Store b/.DS_Store deleted file mode 100644 index 75b478e450eab1f92e283c92544fec336a14cdc4..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 6148 zcmeHKF-`dbJpybi`Zb^};nV>E>}W&Z(QdIuwuF5pR)h-V-%S z0V!~*jH+wGI9ZXU-R#hIvphL^%dV iImUwJ_&$;{uW^q3UbrR(9r2(8^)oiX_k~?m#E5jBF=E0Cp74n0MfUFj=WZ~N@j?H|e;ZyO z#^W@No+6V1Qa}nw0VyB_&ZIyU*7fF0=jj+JAO$W*0slTUx??Y#6XVmtAzA?9is3NM zqn98y4-k9doX7~xl1fafRf}OsXS`KjFPsyT4y)$F>Sn7B#o~6J-y$8>6E#W!DR8R5 zbuPDF|L^F3^#7+Mt)zeyxF`i|wZGr*_@t_>i^qAbZS(`(bH3S5T0$TO(;SR3OxqA7Ob`v#7n641&ruHr6#6mFlI}WnnNk%tS{t~_&m<+ zZbfOmiAb4&*>5sGo9wq`w*vs8KaKVQ900IT2`e@ZD}>^tOHwhODxy$xPGQ{XxI8Nd$Z@1q?rCQtAtXp-fVcqx-vhedDpJx4F zaz#()QYPWT4#JCgR1BT%6Pe~goQ@`{AdW_ua(x-6ku3Uhnnszb^>x6qEqmxRXR}_f z-EuqK&b;N$jtZ(W kprNi}%%!WiiK+zsE*XfP!BQi7Q2394qJaly;7=L&0L5ihCjbBd From b0ff7fe5fb0a88e6f19b802f33ccd39ae8055876 Mon Sep 17 00:00:00 2001 From: napinoco Date: Sun, 13 Jul 2025 00:24:41 +0900 Subject: [PATCH 02/13] Update documentation for MATLAB integration completion MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - README.md: Added MATLAB/Octave solver support (SeDuMi, SDPT3) - README.md: Updated solver counts (11 total: 9 Python + 2 MATLAB) - README.md: Added component role definitions (loaders, runners, interfaces) - detail_design.md: Added clear separation of concerns documentation - Documented problem_name β†’ ProblemData β†’ SolverResult workflow πŸ€– Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude --- README.md | 54 +- docs/development/detail_design.md | 1632 ++++++++++++----------------- 2 files changed, 706 insertions(+), 980 deletions(-) diff --git a/README.md b/README.md index 4b5bff7..44c9326 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,9 @@ Both libraries are included as fixed snapshots (committed on June 25, 2025) for ## 🎯 Current Status: Production Ready - βœ… **Complete External Library Integration**: 139+ problems (DIMACS + SDPLIB) -- βœ… **Comprehensive Solver Support**: 9 solvers (SciPy, CLARABEL, SCS, ECOS, OSQP, CVXOPT, SDPA, SCIP, HiGHS via CVXPY) +- βœ… **Multi-Language Solver Support**: 11 solvers (9 Python + 2 MATLAB solvers) +- βœ… **Python Solvers**: SciPy, CLARABEL, SCS, ECOS, OSQP, CVXOPT, SDPA, SCIP, HiGHS via CVXPY +- βœ… **MATLAB/Octave Solvers**: SeDuMi, SDPT3 with unified interface architecture - βœ… **Unified ProblemData Architecture**: SeDuMi-like format for all problem types - βœ… **Professional Reporting**: Interactive HTML dashboards with problem structure analysis - βœ… **GitHub Actions CI/CD**: Automated benchmarking and report publishing @@ -44,6 +46,12 @@ cd optimization-solver-benchmark python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate pip install -r requirements.txt + +# For MATLAB/Octave solver support (optional) +# Install Octave (MATLAB-compatible environment) +# Ubuntu/Debian: sudo apt-get install octave +# macOS: brew install octave +# Windows: Download from https://octave.org/download ``` ### Run Benchmarks @@ -63,6 +71,9 @@ python main.py --report # Dry-run mode for testing python main.py --benchmark --problems nb --dry-run + +# Test MATLAB/Octave solvers specifically +python main.py --benchmark --problems nb --solvers matlab_sedumi,matlab_sdpt3 ``` ## 🎯 Design Philosophy: Fair Baseline Benchmarking @@ -78,6 +89,7 @@ This system prioritizes **unbiased solver comparison** through minimal configura ## πŸ“Š Current System Capabilities ### Supported Solvers βœ… +#### Python Solvers (9 total) - **SciPy**: Linear programming solver - **CVXPY Multi-Backend**: - CLARABEL (modern Rust-based solver) @@ -89,12 +101,22 @@ This system prioritizes **unbiased solver comparison** through minimal configura - SCIP (Mixed-integer programming) - HiGHS (Linear/mixed-integer programming) +#### MATLAB/Octave Solvers (2 total) +- **SeDuMi**: Interior-point method for conic optimization +- **SDPT3**: Semidefinite-quadratic-linear programming solver + ### Problem Type Coverage βœ… - **LP**: Linear Programming (12+ results, 100% success rate) - **QP**: Quadratic Programming (6+ results, 100% success rate) - **SOCP**: Second-Order Cone Programming (31+ results, ~43% success rate) - **SDP**: Semidefinite Programming (38+ results, ~29% success rate) +### Multi-Language Architecture βœ… +- **Unified Interface Design**: Symmetrical Python/MATLAB solver interfaces +- **Seamless Integration**: Python-MATLAB bridge via JSON data exchange +- **Fair Comparison**: Standardized metrics calculation across all solvers +- **Dynamic Solver Selection**: Runtime solver selection via configuration + ### Key Features βœ… - **External Library Integration**: DIMACS (47 problems) + SDPLIB (92+ problems) - **Unified Problem Format**: SeDuMi-like architecture for all problem types @@ -126,10 +148,13 @@ optimization-solver-benchmark/ β”‚ β”œβ”€β”€ main.py # Main entry point β”‚ └── scripts/ β”‚ β”œβ”€β”€ benchmark/ # Benchmark execution engine -β”‚ β”œβ”€β”€ solvers/python/ # Solver implementations +β”‚ β”œβ”€β”€ solvers/ +β”‚ β”‚ β”œβ”€β”€ python/ # Python solver implementations +β”‚ β”‚ └── matlab_octave/ # MATLAB/Octave solver integration β”‚ β”œβ”€β”€ data_loaders/ # Problem loading (MAT/DAT) β”‚ β”‚ β”œβ”€β”€ problem_loader.py # Unified ProblemData class -β”‚ β”‚ └── python/ # Format-specific loaders +β”‚ β”‚ β”œβ”€β”€ python/ # Python format loaders +β”‚ β”‚ └── matlab_octave/ # MATLAB format loaders β”‚ β”œβ”€β”€ external/ # External library loaders β”‚ β”œβ”€β”€ utils/ # Problem structure analysis β”‚ β”œβ”€β”€ database/ # SQLite data models @@ -157,8 +182,8 @@ optimization-solver-benchmark/ Problem Type | Total Results | Success Rate | Top Performers LP | 12 results | 100% | SciPy + CLARABEL + SCS + ECOS + OSQP QP | 6 results | 100% | SciPy + CLARABEL + SCS + ECOS + OSQP -SOCP | 31 results | ~43% | CLARABEL + SCS + ECOS -SDP | 38 results | ~29% | CLARABEL + SCS +SOCP | 31 results | ~43% | CLARABEL + SCS + ECOS + SeDuMi + SDPT3 +SDP | 38 results | ~29% | CLARABEL + SCS + SeDuMi + SDPT3 ``` ### External Library Performance @@ -212,17 +237,28 @@ python main.py --benchmark --dry-run # Test without database changes ### Architecture Highlights - **Unified ProblemData**: SeDuMi-like format with first-class cone structure support -- **Modular Loaders**: Separate MAT and DAT loaders for different formats +- **Multi-Language Support**: Symmetrical Python/MATLAB solver interfaces with unified result format +- **Modular Component Design**: + - **Loaders**: Convert problem_name β†’ ProblemData (format-specific parsing) + - **Runners**: Convert ProblemData β†’ SolverResult (solver-specific execution) + - **Interfaces**: Orchestrate problem_name β†’ SolverResult (loader + runner coordination) - **CVXPY Integration**: Unified solver interface supporting multiple backends - **Version Tracking**: Complete solver backend and Git commit recording - **Error Resilience**: Continues benchmarking despite individual solver failures ### Adding New Solvers +#### Python Solvers 1. Extend the CVXPY runner with new backend 2. Add solver configuration to `config/solver_registry.yaml` 3. Test with validation framework 4. Update documentation +#### MATLAB/Octave Solvers +1. Create solver-specific runner: `{solver}_runner.m` (ProblemData β†’ SolverResult) +2. Add solver configuration to MATLAB_SOLVER_CONFIGS in `matlab_interface.py` +3. Test with validation framework +4. Update documentation + ### Adding New Problems 1. **External Libraries**: Add as Git submodules in `problems/` 2. **Format Support**: Extend loaders in `scripts/data_loaders/python/` @@ -232,8 +268,8 @@ python main.py --benchmark --dry-run # Test without database changes ### Technical Accomplishments βœ… - **Unified Architecture**: Single ProblemData format for all optimization types +- **Multi-Language Integration**: 11 solvers (9 Python + 2 MATLAB) with symmetrical interfaces - **External Library Integration**: Professional-grade problem sets (DIMACS + SDPLIB) -- **Comprehensive Solver Support**: 9 major optimization solvers - **Production-Ready Reporting**: Interactive dashboards suitable for research publication - **Fair Baseline Benchmarking**: Minimal configuration for unbiased comparison @@ -260,7 +296,7 @@ python main.py --benchmark --dry-run # Test without database changes ### System Completeness βœ… - **139+ External Problems**: Real-world optimization challenges -- **9 Major Solvers**: Comprehensive coverage across optimization types +- **11 Major Solvers**: Multi-language coverage (9 Python + 2 MATLAB) across optimization types - **Professional Reporting**: Publication-ready HTML dashboards - **CI/CD Integration**: Automated benchmarking and deployment - **Fair Benchmarking Philosophy**: Unbiased solver comparison using defaults @@ -278,7 +314,7 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file ## πŸ™ Acknowledgments - **External Problem Libraries**: VSDP team for DIMACS and SDPLIB repositories -- **Open-Source Solvers**: SciPy, CVXPY, CLARABEL, SCS, ECOS, OSQP, CVXOPT, SDPA, SCIP, HiGHS +- **Open-Source Solvers**: SciPy, CVXPY, CLARABEL, SCS, ECOS, OSQP, CVXOPT, SDPA, SCIP, HiGHS, SeDuMi, SDPT3 - **Web Technologies**: Bootstrap 5, Chart.js for interactive visualizations - **CI/CD Platform**: GitHub Actions for automated benchmarking and deployment diff --git a/docs/development/detail_design.md b/docs/development/detail_design.md index e37ed61..0f671cb 100644 --- a/docs/development/detail_design.md +++ b/docs/development/detail_design.md @@ -1,35 +1,59 @@ -# Optimization Solver Benchmark System - Re-Architected Design +# Optimization Solver Benchmark System - Technical Design Specification -This document provides technical specifications for the **re-architected** optimization solver benchmark system focused on simplicity, reliability, and maintainability. +This document provides comprehensive technical specifications for the optimization solver benchmark system, including the complete MATLAB/Octave integration architecture. The system is designed for simplicity, reliability, and maintainability while supporting both Python and MATLAB solvers. + +--- + +## System Overview + +### Core Mission +"Regularly benchmark publicly available solvers and publish the results as data" + +### Key Features +- **Fair Baseline Benchmarking**: Minimal configuration using solver defaults +- **Multi-Language Support**: Python (CVXPY, SciPy) and MATLAB/Octave (SeDuMi, SDPT3) solvers +- **External Problem Libraries**: DIMACS (47 problems) and SDPLIB (92 problems) +- **Production Ready**: 139+ problems with comprehensive solver coverage +- **Automated Publishing**: GitHub Actions with GitHub Pages deployment + +### Current System Statistics +``` +Problem Type | Total Results | Success Rate | Solver Coverage +LP | 12 results | 100% | SciPy + CLARABEL + SCS + ECOS + OSQP +QP | 6 results | 100% | SciPy + CLARABEL + SCS + ECOS + OSQP +SOCP | 31 results | ~43% | CLARABEL + SCS + ECOS + SeDuMi + SDPT3 +SDP | 38 results | ~29% | CLARABEL + SCS + SeDuMi + SDPT3 + +External Libraries: +DIMACS | 47 problems | SeDuMi .mat format +SDPLIB | 92 problems | SDPA .dat-s format +``` --- ## System Architecture -### Simplified Data Flow Architecture +### Unified Data Flow Architecture ``` LOCAL DEVELOPMENT: β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ Problem │───▢│ Solver │───▢│ Result β”‚ -β”‚ Loading β”‚ β”‚ Execution β”‚ β”‚ Collection β”‚ +β”‚ Interface β”‚ β”‚ Interfaces β”‚ β”‚ Collection β”‚ +β”‚ (Unified) β”‚ β”‚ (Py + MATLAB)β”‚ β”‚ (Standard) β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β–Ό β–Ό β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” -β”‚ Validation β”‚ β”‚ Environment β”‚ β”‚ Database β”‚ -β”‚ & Caching β”‚ β”‚ Capture β”‚ β”‚ Storage β”‚ +β”‚ External β”‚ β”‚ Environment β”‚ β”‚ Database β”‚ +β”‚ Libraries β”‚ β”‚ Capture β”‚ β”‚ Storage β”‚ +β”‚ (DIMACS/SDL)β”‚ β”‚ (Unified) β”‚ β”‚ (SQLite) β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ Report β”‚ β”‚ Generation β”‚ - β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ - β”‚ - β–Ό - β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” - β”‚ Commit to β”‚ - β”‚ docs/ β”‚ + β”‚ (Bootstrap) β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ GITHUB ACTIONS (Publishing Only): @@ -39,1117 +63,783 @@ GITHUB ACTIONS (Publishing Only): β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` -### Component Interaction (Simplified) +### Symmetrical Interface Architecture ``` -LOCAL DEVELOPMENT: -β”œβ”€β”€ Problem Loading (Local files only) -β”œβ”€β”€ Python Solver Execution (CVXPY + SciPy) -β”œβ”€β”€ Result Collection & Database Storage -β”œβ”€β”€ HTML Report Generation -β”œβ”€β”€ Data Export (JSON/CSV) -└── Commit Generated Files - -GITHUB ACTIONS (Minimal): -β”œβ”€β”€ PR Preview (Lightweight benchmark + publish) -└── Main Branch Publishing (Static file deployment only) +BenchmarkRunner +β”œβ”€β”€ PythonInterface +β”‚ β”œβ”€β”€ cvxpy_clarabel, cvxpy_scs, cvxpy_ecos, cvxpy_osqp +β”‚ └── scipy_linprog +β”œβ”€β”€ MatlabInterface +β”‚ β”œβ”€β”€ matlab_sedumi +β”‚ └── matlab_sdpt3 +└── ProblemInterface + β”œβ”€β”€ MATLoader (DIMACS .mat files) + β”œβ”€β”€ DATLoader (SDPLIB .dat-s files) + β”œβ”€β”€ MPSLoader (LP .mps files) + β”œβ”€β”€ QPSLoader (QP .qps files) + └── PythonLoader (Python problem definitions) ``` --- -## Re-Architected Directory Structure +## Component Architecture -``` -optimization-solver-benchmark/ -β”œβ”€β”€ README.md # Project overview and quick start (updated) -β”œβ”€β”€ CLAUDE.md # AI assistant integration context -β”œβ”€β”€ LICENSE # MIT license -β”œβ”€β”€ requirements.txt # All dependencies consolidated -β”‚ -β”œβ”€β”€ .github/ # GitHub Actions workflows -β”‚ └── workflows/ -β”‚ β”œβ”€β”€ deploy.yml # Deploy reports and PR previews -β”‚ └── validate.yml # Codebase validation (no benchmarking) -β”‚ -β”œβ”€β”€ config/ # Configuration files -β”‚ β”œβ”€β”€ site_config.yaml # Site display information -β”‚ β”œβ”€β”€ solver_registry.yaml # Available solvers list -β”‚ └── problem_registry.yaml # Available problems list (moved from problems/) -β”‚ -β”œβ”€β”€ problems/ # Problem instances -β”‚ β”œβ”€β”€ DIMACS/ # External DIMACS library (50 problems) -β”‚ └── SDPLIB/ # External SDPLIB library (3 problems) -β”‚ -β”œβ”€β”€ scripts/ -β”‚ β”œβ”€β”€ benchmark/ # Benchmark execution and database storage -β”‚ β”‚ β”œβ”€β”€ __init__.py -β”‚ β”‚ β”œβ”€β”€ runner.py # Main benchmark execution logic -β”‚ β”‚ └── environment_info.py # Environment information gathering -β”‚ β”‚ -β”‚ β”œβ”€β”€ data_loaders/ # ETL: Data loading and format conversion -β”‚ β”‚ β”œβ”€β”€ __init__.py -β”‚ β”‚ β”œβ”€β”€ python/ # Python-based loaders -β”‚ β”‚ β”‚ β”œβ”€β”€ __init__.py -β”‚ β”‚ β”‚ β”œβ”€β”€ mat_loader.py # DIMACS .mat file loader -β”‚ β”‚ β”‚ β”œβ”€β”€ dat_loader.py # SDPLIB .dat-s file loader -β”‚ β”‚ └── matlab_octave/ # MATLAB/Octave loaders (future) -β”‚ β”‚ └── .gitkeep -β”‚ β”‚ -β”‚ β”œβ”€β”€ solvers/ # Solver execution with standardized output -β”‚ β”‚ β”œβ”€β”€ __init__.py -β”‚ β”‚ β”œβ”€β”€ solver_interface.py # Abstract solver interface -β”‚ β”‚ β”œβ”€β”€ python/ # Python solvers -β”‚ β”‚ β”‚ β”œβ”€β”€ __init__.py -β”‚ β”‚ β”‚ β”œβ”€β”€ cvxpy_runner.py # CVXPY solver implementation -β”‚ β”‚ β”‚ └── scipy_runner.py # SciPy solver implementation -β”‚ β”‚ └── matlab_octave/ # MATLAB/Octave solvers (future) -β”‚ β”‚ └── .gitkeep -β”‚ β”‚ -β”‚ β”œβ”€β”€ reporting/ # HTML generation and data extraction -β”‚ β”‚ β”œβ”€β”€ __init__.py -β”‚ β”‚ β”œβ”€β”€ html_generator.py # Generate overview, results_matrix, raw_data -β”‚ β”‚ β”œβ”€β”€ data_exporter.py # Export JSON/CSV data -β”‚ β”‚ └── result_processor.py # Process latest results from database -β”‚ β”‚ -β”‚ └── database/ # Database models and operations -β”‚ β”œβ”€β”€ __init__.py -β”‚ β”œβ”€β”€ models.py # Single denormalized table model -β”‚ β”œβ”€β”€ database_manager.py # Database operations and result storage -β”‚ └── schema.sql # Database schema definition -β”‚ -β”œβ”€β”€ docs/ # GitHub Pages output (published data) -β”‚ β”œβ”€β”€ pages/ # Generated HTML and data -β”‚ β”‚ β”œβ”€β”€ index.html # Overview report -β”‚ β”‚ β”œβ”€β”€ results_matrix.html # Problems Γ— solvers matrix -β”‚ β”‚ β”œβ”€β”€ raw_data.html # Raw data display -β”‚ β”‚ β”œβ”€β”€ assets/ # CSS, JS, images -β”‚ β”‚ └── data/ # JSON/CSV exports -β”‚ β”œβ”€β”€ development/ # Developer documentation -β”‚ └── guides/ # User documentation -β”‚ -β”œβ”€β”€ database/ # SQLite database -β”‚ └── results.db # Single denormalized results table -β”‚ -β”œβ”€β”€ logs/ # Log files (local only, .gitignore) -β”‚ └── benchmark.log # Structured execution logs -β”‚ -└── tests/ # Test suite - β”œβ”€β”€ unit/ # Unit tests - β”œβ”€β”€ integration/ # Integration tests - └── fixtures/ # Test data and configurations -``` +### 1. Problem Interface Module ---- +#### Component Role Definitions +The system follows a clear separation of concerns across three key components: -## Re-Architected Core Components +- **Loaders**: Convert `problem_name` β†’ `ProblemData` + - Role: Format-specific parsing and data loading + - Input: Problem name (string identifier) + - Output: Standardized ProblemData object + - Examples: `mat_loader.py`, `dat_loader.py`, `mat_loader.m`, `dat_loader.m` -### 1. GitHub Actions Workflows (Minimal) +- **Runners**: Convert `ProblemData` β†’ `SolverResult` + - Role: Solver-specific execution and result generation + - Input: ProblemData object with A, b, c, K matrices + - Output: Standardized SolverResult object + - Examples: `cvxpy_runner.py`, `sedumi_runner.m`, `sdpt3_runner.m` -#### deploy.yml - Unified Deployment System -```yaml -# Key features: -# - Triggers on push to main branch and pull requests -# - Publishes PRE-BUILT docs/ folder to GitHub Pages -# - No benchmark execution in CI -# - Unified workflow for both production and PR previews -# - Auto-deploy PR previews to gh-pages/pr-preview/pr-{number}/ -# - Auto-cleanup when PR closed -# - Preview banners and metadata injection -# - Comments with preview URLs on PRs -# - Preserves PR preview subdirectories -``` - -#### validate.yml - Codebase Validation Only -```yaml -# Key features: -# - Lightweight CI validation without benchmarking -# - Validates configuration files can be loaded -# - Checks Python dependencies installation -# - Verifies core system components initialize correctly -# - Tests solver backend availability -# - No benchmark execution or report generation -# - Fast validation for development workflow -``` - -### 2. Configuration Management (Re-architected) - -The new configuration structure eliminates `benchmark_config.yaml` and consolidates all configuration into three focused files. The system now derives configuration directly from the registries and uses sensible defaults. - -#### config/site_config.yaml - Site Display Information -```yaml -site: - title: "Optimization Solver Benchmark" - author: "Your Name" - description: "Benchmarking optimization solvers with fair comparison" - url: "https://your-username.github.io/optimization-solver-benchmark" - -github: - username: "your-username" - repository: "optimization-solver-benchmark" -``` +- **Interfaces**: Orchestrate `problem_name` β†’ `SolverResult` + - Role: Coordinate loaders and runners for complete workflow + - Input: Problem name (string identifier) + - Output: Standardized SolverResult object + - Examples: `python_interface.py`, `matlab_interface.py` -#### Solver Configuration - Interface-based Architecture (EAFP) - -**Note**: `config/solver_registry.yaml` has been **REMOVED** as part of the EAFP implementation. Solver configurations are now managed directly in the interface classes for better maintainability and performance. +This modular design ensures clean separation between data loading, solver execution, and workflow coordination, enabling independent development and testing of each component. +#### Unified Problem Management ```python -# Solver configurations are now defined directly in interface classes -# scripts/solvers/python/python_interface.py -class PythonInterface: - PYTHON_SOLVER_CONFIGS = { - "scipy_linprog": { - "class": ScipySolver, - "display_name": "SciPy linprog", - "kwargs": {} - }, - "cvxpy_clarabel": { - "class": CvxpySolver, - "display_name": "CLARABEL (via CVXPY)", - "kwargs": {"backend": "CLARABEL"} - }, - "cvxpy_scs": { - "class": CvxpySolver, - "display_name": "SCS (via CVXPY)", - "kwargs": {"backend": "SCS"} - }, - "cvxpy_ecos": { - "class": CvxpySolver, - "display_name": "ECOS (via CVXPY)", - "kwargs": {"backend": "ECOS"} - }, - "cvxpy_osqp": { - "class": CvxpySolver, - "display_name": "OSQP (via CVXPY)", - "kwargs": {"backend": "OSQP"} - } - } - -# scripts/solvers/matlab_octave/matlab_interface.py -class MatlabInterface: - MATLAB_SOLVER_CONFIGS = { - "matlab_sedumi": { - "class": SeDuMiSolver, - "display_name": "SeDuMi (MATLAB)", - "matlab_solver": "sedumi" - }, - "matlab_sdpt3": { - "class": SDPT3Solver, - "display_name": "SDPT3 (MATLAB)", - "matlab_solver": "sdpt3" - } +# scripts/data_loaders/python/problem_interface.py +class ProblemInterface: + """Centralized problem loading and registry management""" + + FORMAT_LOADERS = { + "mat": MATLoader, # SeDuMi format (DIMACS) + "dat-s": DATLoader, # SDPA format (SDPLIB) + "mps": MPSLoader, # MPS format (LP) + "qps": QPSLoader, # QPS format (QP) + "python": PythonLoader, # Python definitions } + + def load_problem(self, problem_name: str) -> ProblemData: + """Load problem using appropriate format loader""" + + def get_problem_config(self, problem_name: str) -> Dict[str, Any]: + """Get problem configuration from registry""" + + def get_available_problems(self, library_filter: List[str] = None) -> List[str]: + """Get filtered list of available problems""" ``` -**EAFP Architecture Benefits:** -- **Single Source of Truth**: Each interface manages its own solver configurations -- **Lazy Detection**: Solver availability is only detected when explicitly needed -- **Simplified Error Handling**: "Just try it" approach instead of pre-checking -- **Better Performance**: No upfront solver detection during initialization -- **Maintainability**: Solver logic is co-located with interface implementations - -#### config/problem_registry.yaml - External Problems Only -```yaml -# Flat problem structure - each problem is a top-level entry -# Only external problems from DIMACS and SDPLIB libraries -problem_libraries: - - # Small-scale test problems from DIMACS/SDPLIB - nb: - display_name: "ANTENNA NB (DIMACS)" - file_path: "problems/DIMACS/data/ANTENNA/nb.mat.gz" - file_type: "mat" - problem_type: "SDP" - library_name: "DIMACS" - for_test_flag: true # Small problem suitable for testing - known_objective_value: -12.8 # Known optimal value for validation - - arch0: - display_name: "ARCH0 (SDPLIB)" - file_path: "problems/SDPLIB/data/arch0.dat-s" - file_type: "dat-s" - problem_type: "SDP" - library_name: "SDPLIB" - for_test_flag: true # Small problem suitable for testing - known_objective_value: -5.6506 # Known optimal value - - # Larger production problems - hinf12: - display_name: "H-infinity Control 12 (DIMACS)" - file_path: "problems/DIMACS/data/HINF/hinf12.mat.gz" - file_type: "mat" - problem_type: "SDP" - library_name: "DIMACS" - for_test_flag: false - # known_objective_value: null # Unknown - omit field +#### Problem Data Standardization +```python +# scripts/data_loaders/problem_loader.py +@dataclass +class ProblemData: + """Standardized problem data structure""" + name: str + problem_class: str # 'LP', 'QP', 'SOCP', 'SDP' - control1: - display_name: "Control Problem 1 (SDPLIB)" - file_path: "problems/SDPLIB/data/control1.dat-s" - file_type: "dat-s" - problem_type: "SDP" - library_name: "SDPLIB" - for_test_flag: false - known_objective_value: 20.8 # Known optimal value + # SeDuMi format (unified internal representation) + A_eq: np.ndarray # Constraint matrix + b_eq: np.ndarray # RHS vector + c: np.ndarray # Objective coefficients + cone_structure: Dict # Cone constraints specification - gpp100: - display_name: "Graph Partitioning 100 (SDPLIB)" - file_path: "problems/SDPLIB/data/gpp100.dat-s" - file_type: "dat-s" - problem_type: "SDP" - library_name: "SDPLIB" - for_test_flag: false - # known_objective_value: null # Unknown - omit field + # Optional QP data + P: Optional[np.ndarray] = None # Quadratic term - # Linear programming problems + # Problem metadata + _num_variables: int + _num_constraints: int + metadata: Dict[str, Any] = field(default_factory=dict) ``` -This structure provides: -- **Flat hierarchy**: Direct problem access without nested library structure -- **Test problem identification**: `for_test_flag` to identify small problems for quick testing -- **Known objective values**: Optional field for result validation -- **Library attribution**: Clear source library tracking -- **No synthetic problems**: All problems are from established optimization libraries - -### 3. EAFP Benchmark Execution Architecture +### 2. Solver Interface Architecture -The EAFP (Easier to Ask for Forgiveness than Permission) implementation removes complex pre-checking and uses a "just try it" approach with graceful error handling. - -#### scripts/benchmark/runner.py - EAFP Execution Logic +#### Python Solver Interface ```python -class BenchmarkRunner: - """EAFP-based benchmark execution with unified interfaces""" +# scripts/solvers/python/python_interface.py +class PythonInterface: + """Unified interface for all Python-based solvers""" + + SOLVER_CONFIGURATIONS = { + 'cvxpy_clarabel': CvxpySolver, + 'cvxpy_scs': CvxpySolver, + 'cvxpy_ecos': CvxpySolver, + 'cvxpy_osqp': CvxpySolver, + 'scipy_linprog': ScipySolver, + } - def __init__(self, database_manager: DatabaseManager): - self.db = database_manager - self.environment_info = collect_environment_info() - self.commit_hash = get_git_commit_hash() - - # Initialize interfaces with lazy solver detection - self.problem_interface = ProblemInterface() - self.python_interface = PythonInterface( - save_solutions=save_solutions, - problem_interface=self.problem_interface - ) + def solve(self, problem_name: str, solver_name: str, + problem_data: Optional[ProblemData] = None, + timeout: Optional[float] = None) -> SolverResult: + """Unified solve method with automatic problem loading""" - # MATLAB interface with availability check - self.matlab_interface = None - if MATLAB_SOLVERS_AVAILABLE: - self.matlab_interface = MatlabInterface( - save_solutions=save_solutions, - problem_interface=self.problem_interface - ) - - def run_single_benchmark(self, problem_name: str, solver_name: str) -> None: - """Execute single problem-solver combination using EAFP approach""" + # 1. Create solver instance + solver = self.create_solver(solver_name) - try: - # EAFP: Try Python interface first (most common case) - try: - result = self.python_interface.solve(problem_name, solver_name) - except ValueError as e: - # If not a Python solver, try MATLAB interface - if self.matlab_interface: - try: - result = self.matlab_interface.solve(problem_name, solver_name) - except ValueError: - # Neither interface has this solver - raise ValueError(f"Solver '{solver_name}' not found in any interface") - else: - # No MATLAB interface available - raise ValueError(f"Solver '{solver_name}' not found (MATLAB interface not available)") - - # Success! Store result - problem_config = self.problem_interface.get_problem_config(problem_name) - self.store_result(solver_name, problem_name, result, problem_config) - - except Exception as e: - error_msg = f"Benchmark execution failed: {str(e)}" - logger.error(error_msg) - # Store error result for tracking - self.store_error_result(solver_name, problem_name, error_msg) -``` - -**Key EAFP Features:** -- **No Pre-checking**: Solvers are not validated before execution -- **Lazy Detection**: Solver availability is only checked when needed -- **Unified Interface**: Both Python and MATLAB follow the same solve() signature -- **Graceful Fallback**: Try Python first, then MATLAB if needed -- **Clear Error Messages**: Specific error messages for debugging - -#### scripts/database/database_manager.py - Database Operations -```python -class DatabaseManager: - """Handles all database operations for benchmark results""" - - def __init__(self, db_path: str = "database/results.db"): - self.db_path = db_path - self.ensure_schema() - - def store_result(self, solver_name: str, solver_version: str, - problem_library: str, problem_name: str, problem_type: str, - environment_info: str, commit_hash: str, - solve_time: float, status: str, - primal_objective: float, dual_objective: float, - duality_gap: float, primal_infeas: float, dual_infeas: float, - iterations: int) -> None: - """Store single benchmark result (append-only)""" + # 2. Load problem data if not provided + if problem_data is None: + problem_data = self.problem_interface.load_problem(problem_name) - # Insert into results table without initialization - # Preserves all historical data + # 3. Validate compatibility + if not solver.validate_problem_compatibility(problem_data): + return SolverResult.create_error_result(...) - def get_latest_results(self) -> List[BenchmarkResult]: - """Get latest results for reporting""" + # 4. Execute solver + result = solver.solve(problem_data, timeout=timeout) - # Query for results with latest commit_hash and environment_info - # Use timestamp as tiebreaker for truly latest results - query = """ - SELECT * FROM results - WHERE (commit_hash, environment_info, timestamp) IN ( - SELECT commit_hash, environment_info, MAX(timestamp) - FROM results - GROUP BY solver_name, solver_version, problem_library, problem_name - ) - ORDER BY problem_library, problem_name, solver_name - """ + # 5. Add problem class to additional_info for database storage + if not result.additional_info: + result.additional_info = {} + result.additional_info['problem_class'] = problem_data.problem_class - def get_solver_problem_history(self, solver_name: str, problem_name: str) -> List[BenchmarkResult]: - """Get historical results for analysis""" + return result ``` -#### scripts/benchmark/environment_info.py - Environment Capture +#### MATLAB Solver Interface ```python -class EnvironmentInfo: - """Capture and standardize environment information""" +# scripts/solvers/matlab_octave/matlab_interface.py +class MatlabInterface: + """Unified interface for MATLAB/Octave solvers via subprocess execution""" - @staticmethod - def gather() -> dict: - """Collect comprehensive environment information""" - return { - "platform": platform.platform(), - "python_version": platform.python_version(), - "cpu_cores": os.cpu_count(), - "memory_gb": round(psutil.virtual_memory().total / (1024**3), 1), - "hostname": platform.node(), - "user": getpass.getuser(), - "timestamp": datetime.now().isoformat(), - "timezone": str(datetime.now().astimezone().tzinfo) + MATLAB_SOLVER_CONFIGS = { + 'matlab_sedumi': { + 'matlab_solver': 'sedumi', + 'runner_function': 'sedumi_runner' + }, + 'matlab_sdpt3': { + 'matlab_solver': 'sdpt3', + 'runner_function': 'sdpt3_runner' } + } - @staticmethod - def get_git_commit_hash() -> str: - """Get current git commit hash""" + def solve(self, problem_name: str, solver_name: str, + problem_data: Optional[ProblemData] = None, + timeout: Optional[float] = None) -> SolverResult: + """Unified solve method calling matlab_interface.m directly""" + + # 1. Validate solver name + solver_config = self.MATLAB_SOLVER_CONFIGS[solver_name] + + # 2. Call MATLAB interface via subprocess + result = self._call_matlab_interface( + problem_name=problem_name, + matlab_solver=solver_config["matlab_solver"], + runner_function=solver_config["runner_function"], + timeout=timeout or self.default_timeout + ) + + # 3. Add problem class information for database storage try: - result = subprocess.run( - ["git", "rev-parse", "HEAD"], - capture_output=True, text=True, check=True - ) - return result.stdout.strip() - except: - return "unknown" + problem_interface = ProblemInterface() + problem_data = problem_interface.load_problem(problem_name) + if not result.additional_info: + result.additional_info = {} + result.additional_info['problem_class'] = problem_data.problem_class + except Exception: + result.additional_info['problem_class'] = 'UNKNOWN' + + return result +``` + +### 3. MATLAB Integration Architecture + +#### MATLAB Interface Entry Point +```matlab +% scripts/solvers/matlab_octave/matlab_interface.m +function matlab_interface(problem_name, solver_name, result_file, save_solutions, runner_function) + % Main MATLAB interface for benchmark execution + % + % Input: + % problem_name: Name of problem from problem_registry.yaml + % solver_name: Name of solver ('sedumi' or 'sdpt3') + % result_file: Path to output JSON file for results + % save_solutions: Boolean flag to save solutions to .mat file + % runner_function: Name of the runner function to use + + try + % 1. Load problem registry and resolve problem file path + [problem_config, file_path] = read_problem_registry(problem_name); + file_type = problem_config.file_type; + + % 2. Load problem data using appropriate loader + if strcmp(file_type, 'mat') + [A, b, c, K] = mat_loader(file_path); + elseif strcmp(file_type, 'dat-s') + [A, b, c, K] = dat_loader(file_path); + else + error('Unsupported file type: %s', file_type); + end + + % 3. Execute solver using dynamic function call + if exist(runner_function, 'file') + [x, y, result] = feval(runner_function, A, b, c, K); + else + error('Solver runner function not found: %s', runner_function); + end + + % 4. Calculate standardized metrics + if ~isempty(x) && ~isempty(y) + result = calculate_solver_metrics(result, x, y, A, b, c, K); + end + + % 5. Save solution vectors if requested + if save_solutions && strcmp(result.status, 'optimal') + save_solutions_if_needed(problem_name, solver_name, x, y, save_solutions); + end + + % 6. Convert result to JSON-compatible format and save + json_result = convert_to_json_result(result); + save_json_safely(json_result, result_file); + + catch ME + % Save error result to JSON file + error_result = create_error_result(ME, solver_name); + save_json_safely(error_result, result_file); + end +end +``` + +#### MATLAB Solver Runners +```matlab +% scripts/solvers/matlab_octave/sedumi_runner.m +function [x, y, result] = sedumi_runner(A, b, c, K) + % SeDuMi solver runner with standardized interface + + result = initialize_result_structure(); + result.solver_name = 'SeDuMi'; + + try + % Configure SeDuMi options + pars.fid = 0; % Suppress output for benchmarking + + % Execute SeDuMi solver + solve_start_time = tic; + [x, y, info] = sedumi(A, b, c, K, pars); + solve_time = toc(solve_start_time); + + % Map SeDuMi status to standardized format + result.status = map_sedumi_status(info.pinf, info.dinf, info.numerr); + result.solve_time = solve_time; + result.iterations = info.iter; + result.primal_objective_value = NaN; % Calculated by metrics function + result.dual_objective_value = NaN; % Calculated by metrics function + result.duality_gap = NaN; % Calculated by metrics function + result.primal_infeasibility = NaN; + result.dual_infeasibility = NaN; + result.solver_version = 'SeDuMi-1.3.7'; + + catch ME + result.status = 'error'; + result.error_message = ME.message; + x = []; + y = []; + end +end ``` -### 1. Database Architecture - Single Denormalized Table +### 4. Database Architecture -#### Simplified Schema Design +#### Unified Results Schema ```sql --- Single denormalized results table with historical retention CREATE TABLE results ( id INTEGER PRIMARY KEY AUTOINCREMENT, -- Solver information - solver_name TEXT NOT NULL, - solver_version TEXT NOT NULL, + solver_name TEXT NOT NULL, -- 'cvxpy_clarabel', 'matlab_sedumi', etc. + solver_version TEXT NOT NULL, -- Full version with backend info -- Problem information - problem_library TEXT NOT NULL, -- 'DIMACS', 'SDPLIB' - problem_name TEXT NOT NULL, - problem_type TEXT NOT NULL, -- 'LP', 'QP', 'SOCP', 'SDP' + problem_library TEXT NOT NULL, -- 'DIMACS', 'SDPLIB', 'internal' + problem_name TEXT NOT NULL, -- Problem identifier + problem_type TEXT NOT NULL, -- 'LP', 'QP', 'SOCP', 'SDP' -- Environment and execution context environment_info TEXT NOT NULL, -- JSON string with system info commit_hash TEXT NOT NULL, -- Git commit hash timestamp DATETIME DEFAULT CURRENT_TIMESTAMP, - -- Standardized solver results + -- Standardized solver results (unified across Python/MATLAB) solve_time REAL, -- Execution time in seconds - status TEXT, -- 'optimal', 'infeasible', 'error', etc. + status TEXT, -- 'OPTIMAL', 'INFEASIBLE', 'UNBOUNDED', etc. primal_objective_value REAL, -- Primal objective value - dual_objective_value REAL, -- Dual objective value (if available) - duality_gap REAL, -- Duality gap - primal_infeasibility REAL, -- Primal infeasibility measure - dual_infeasibility REAL, -- Dual infeasibility measure + dual_objective_value REAL, -- Dual objective value + duality_gap REAL, -- Primal-dual gap + primal_infeasibility REAL, -- Primal constraint violation + dual_infeasibility REAL, -- Dual constraint violation iterations INTEGER, -- Number of solver iterations - memo TEXT, -- Additional notes or metadata + memo TEXT, -- Additional solver-specific info (JSON) - -- Unique constraint to prevent exact duplicates UNIQUE(solver_name, solver_version, problem_library, problem_name, commit_hash, timestamp) ); --- Index for efficient latest results queries +-- Indexes for efficient querying CREATE INDEX idx_latest_results ON results(commit_hash, environment_info, timestamp DESC); CREATE INDEX idx_solver_problem ON results(solver_name, problem_name); +CREATE INDEX idx_problem_type ON results(problem_type); ``` -#### BenchmarkResult Model +#### Standardized Result Format ```python +# scripts/solvers/solver_interface.py @dataclass -class BenchmarkResult: - """Single denormalized benchmark result model""" - - # Primary key - id: Optional[int] = None - - # Solver information - solver_name: str = "" - solver_version: str = "" - - # Problem information - problem_library: str = "" # 'DIMACS', 'SDPLIB' - problem_name: str = "" - problem_type: str = "" # 'LP', 'QP', 'SOCP', 'SDP' - - # Environment and execution context - environment_info: Dict[str, Any] = None - commit_hash: str = "" - timestamp: Optional[datetime] = None +class SolverResult: + """Standardized solver result format for both Python and MATLAB solvers""" - # Standardized solver results - solve_time: Optional[float] = None - status: Optional[str] = None + solve_time: float + status: str # Standardized status strings primal_objective_value: Optional[float] = None dual_objective_value: Optional[float] = None duality_gap: Optional[float] = None primal_infeasibility: Optional[float] = None dual_infeasibility: Optional[float] = None iterations: Optional[int] = None - memo: Optional[str] = None + solver_name: Optional[str] = None + solver_version: Optional[str] = None + additional_info: Optional[Dict[str, Any]] = None - def to_dict(self) -> Dict[str, Any]: - """Convert to dictionary for JSON serialization""" - @classmethod - def from_dict(cls, data: Dict[str, Any]) -> 'BenchmarkResult': - """Create from dictionary (e.g., from database row)""" + def create_error_result(cls, error_msg: str, solve_time: float = 0.0, + solver_name: str = None, solver_version: str = None) -> 'SolverResult': + """Create standardized error result""" + + @classmethod + def create_timeout_result(cls, timeout: float, solver_name: str = None, + solver_version: str = None) -> 'SolverResult': + """Create standardized timeout result""" ``` -#### Database Manager +### 5. Benchmark Execution Engine + +#### Unified Benchmark Runner ```python -class DatabaseManager: - """Manages database operations for benchmark results""" +# scripts/benchmark/runner.py +class BenchmarkRunner: + """Main benchmark execution engine with symmetrical solver interfaces""" - def __init__(self, db_path: str): - self.db_path = db_path - self.init_database() + def __init__(self, database_manager: Optional[DatabaseManager] = None, + dry_run: bool = False, save_solutions: bool = False): + """Initialize with unified interfaces""" + + # Initialize all interfaces + self.python_interface = PythonInterface() + self.matlab_interface = MatlabInterface() + self.problem_interface = ProblemInterface() + + # Create explicit solver-to-interface mapping + self._solver_interface_map = self._build_solver_interface_map() - def store_result(self, result: BenchmarkResult) -> None: - """Store a single benchmark result (append-only)""" - # No database initialization - preserve historical data + def run_single_benchmark(self, problem_name: str, solver_name: str) -> None: + """Execute single benchmark with explicit interface routing""" - def get_latest_results(self, commit_hash: str, environment_info: str) -> List[BenchmarkResult]: - """Get latest results for specific commit and environment""" - # Query for results with matching commit_hash and environment_info - # Use timestamp as tiebreaker for truly latest results + # Get the appropriate interface type from mapping + interface_type = self._solver_interface_map.get(solver_name) - def get_solver_problem_history(self, solver_name: str, problem_name: str) -> List[BenchmarkResult]: - """Get historical results for solver-problem combination""" -``` - -### 2. Data Loading and Format Conversion (ETL) - -#### Format-Specific Loaders Architecture -```python -# Direct loader usage without dispatcher -class MATLoader: - """Load DIMACS .mat files in SeDuMi format""" + if interface_type == 'python': + result = self.python_interface.solve(problem_name, solver_name) + elif interface_type == 'matlab': + result = self.matlab_interface.solve(problem_name, solver_name) + else: + raise ValueError(f"Unknown solver '{solver_name}'") + + # Store result in database + problem_config = self.problem_interface.get_problem_config(problem_name) + self.store_result(solver_name, problem_name, result, problem_config) - def load(self, file_path: str) -> ProblemData: - """Load .mat file and extract problem matrices""" + def _build_solver_interface_map(self) -> Dict[str, str]: + """Build explicit mapping of solver names to interface types""" + mapping = {} + + # Add Python solvers + for solver_name in self.python_interface.get_available_solvers(): + mapping[solver_name] = 'python' + + # Add MATLAB solvers + for solver_name in self.matlab_interface.get_available_solvers(): + mapping[solver_name] = 'matlab' + + return mapping +``` -class DATLoader: - """Load SDPLIB .dat-s files in SDPA sparse format""" - - def load(self, file_path: str) -> ProblemData: - """Parse SDPA sparse format and create problem data""" +--- -class MPSLoader: - """Load MPS format files (Linear Programming)""" - - def load(self, file_path: str) -> ProblemData: - """Parse MPS format and create problem data""" +## Data Flow Specifications -class QPSLoader: - """Load QPS format files (Quadratic Programming)""" - - def load(self, file_path: str) -> ProblemData: - """Parse QPS format and create problem data""" +### 1. Problem Loading Flow -class PythonLoader: - """Load Python-defined problems""" - - def load(self, file_path: str) -> ProblemData: - """Execute Python file and extract problem definition""" +#### External Library Integration +``` +DIMACS Problems (47 problems): +β”œβ”€β”€ Source: SeDuMi .mat format +β”œβ”€β”€ Loader: scripts/data_loaders/python/mat_loader.py +β”œβ”€β”€ MATLAB: scripts/data_loaders/matlab_octave/mat_loader.m +└── Format: {A, b, c, K} SeDuMi standard -# Note: Conversion to solver format is handled directly by loaders -# Each loader produces ProblemData that is compatible with solvers +SDPLIB Problems (92 problems): +β”œβ”€β”€ Source: SDPA .dat-s format +β”œβ”€β”€ Loader: scripts/data_loaders/python/dat_loader.py +β”œβ”€β”€ MATLAB: scripts/data_loaders/matlab_octave/dat_loader.m +└── Format: Converted to SeDuMi {A, b, c, K} ``` -#### Loader Selection Logic -```python -# In BenchmarkRunner -def load_problem(self, problem_name: str, problem_library: str) -> ProblemData: - """Load problem based on registry configuration""" - - # Get problem info from config/problem_registry.yaml - problem_config = self.get_problem_config(problem_name, problem_library) - file_type = problem_config['file_type'] - file_path = problem_config['file_path'] - - # Select appropriate loader based on file type - if file_type == 'mat': - loader = MATLoader() - elif file_type == 'dat-s': - loader = DATLoader() - elif file_type == 'mps': - loader = MPSLoader() - elif file_type == 'qps': - loader = QPSLoader() - elif file_type == 'python': - loader = PythonLoader() - else: - raise ValueError(f"Unsupported file type: {file_type}") +#### Unified Problem Registry +```yaml +# config/problem_registry.yaml +problem_libraries: + nb: # DIMACS problem + display_name: "NB (DIMACS)" + file_path: "problems/DIMACS/data/ANTENNA/nb.mat.gz" + file_type: "mat" + library_name: "DIMACS" + for_test_flag: true - return loader.load(file_path) + arch0: # SDPLIB problem + display_name: "ARCH0 (SDPLIB)" + file_path: "problems/SDPLIB/data/arch0.dat-s" + file_type: "dat-s" + library_name: "SDPLIB" + for_test_flag: true ``` -### 3. Solver Architecture with Interface Symmetry - -#### Design Philosophy: Symmetrical Interface Pattern - -The solver architecture follows a **symmetrical interface pattern** where each solver ecosystem has its own dedicated interface module that manages the complexities of that ecosystem, while maintaining a unified interface to the benchmark runner. +### 2. Solver Execution Flow +#### Python Solver Flow ``` -scripts/solvers/ -β”œβ”€β”€ python/ -β”‚ β”œβ”€β”€ cvxpy_runner.py # Individual CVXPY solver implementations -β”‚ β”œβ”€β”€ scipy_runner.py # Individual SciPy solver implementations -β”‚ └── python_interface.py # Python ecosystem management interface -└── matlab_octave/ - β”œβ”€β”€ matlab_interface.py # MATLAB ecosystem management interface - └── ...individual solver files +PythonInterface.solve() +β”œβ”€β”€ 1. Create solver instance (cvxpy_clarabel, etc.) +β”œβ”€β”€ 2. Load problem data via ProblemInterface +β”œβ”€β”€ 3. Validate solver-problem compatibility +β”œβ”€β”€ 4. Execute solver.solve(problem_data) +β”œβ”€β”€ 5. Add problem_class to result.additional_info +└── 6. Return standardized SolverResult ``` -#### Benefits of Symmetrical Design -- **Architectural Consistency**: Both Python and MATLAB ecosystems have dedicated interface modules -- **Separation of Concerns**: Each interface handles its ecosystem's specific complexities -- **Maintainability**: Changes to solver ecosystems are isolated within their interface modules -- **Extensibility**: Easy to add new solver ecosystems (e.g., Julia, R) following the same pattern -- **Testability**: Each interface can be tested independently from the orchestration layer - -#### Solver Interface -```python -class SolverInterface: - """Abstract interface for all solvers with standardized output""" - - def solve(self, problem: ProblemData) -> SolverResult: - """Solve problem and return standardized result""" - -class SolverResult: - """Standardized solver result format""" - - def __init__(self): - self.solve_time: float - self.status: str # 'optimal', 'infeasible', 'unbounded', 'error' - self.primal_objective_value: float - self.dual_objective_value: float # Optional, may be None - self.duality_gap: float # Optional, may be None - self.primal_infeasibility: float # Constraint violation measure - self.dual_infeasibility: float # Dual constraint violation measure - self.iterations: int # Number of solver iterations - self.solver_info: dict # Additional solver-specific information +#### MATLAB Solver Flow +``` +MatlabInterface.solve() +β”œβ”€β”€ 1. Validate MATLAB solver configuration +β”œβ”€β”€ 2. Create temporary result file +β”œβ”€β”€ 3. Execute subprocess: octave --eval "matlab_interface(...)" +β”œβ”€β”€ 4. matlab_interface.m execution: +β”‚ β”œβ”€β”€ Load problem via mat_loader.m or dat_loader.m +β”‚ β”œβ”€β”€ Execute sedumi_runner.m or sdpt3_runner.m +β”‚ β”œβ”€β”€ Calculate standardized metrics +β”‚ └── Save JSON result to temporary file +β”œβ”€β”€ 5. Read JSON result and convert to SolverResult +β”œβ”€β”€ 6. Load problem data to get problem_class +β”œβ”€β”€ 7. Add problem_class to result.additional_info +└── 8. Return standardized SolverResult ``` -#### Python Interface Module (EAFP Implementation) +### 3. Result Storage and Processing + +#### Database Storage Flow ```python -class PythonInterface: - """Interface for managing Python solver ecosystem with EAFP approach""" - - def __init__(self, save_solutions: bool = False, problem_interface: Optional[ProblemInterface] = None): - self.save_solutions = save_solutions - self.problem_interface = problem_interface or ProblemInterface() - - # Lazy initialization - solvers detected only when needed - self._available_solvers = None +def store_result(self, solver_name: str, problem_name: str, + result: SolverResult, problem_config: Dict[str, Any]) -> None: + """Store result with unified metadata extraction""" - def solve(self, problem_name: str, solver_name: str, - problem_data: Optional[ProblemData] = None, - timeout: Optional[float] = None) -> SolverResult: - """Unified solve method that handles problem loading and solver execution""" - - try: - # 1. Create solver instance (will raise ValueError if not a Python solver) - solver = self.create_solver(solver_name) - - # 2. Load problem data if not provided - if problem_data is None: - problem_data = self.problem_interface.load_problem(problem_name) - - # 3. Validate compatibility - if not solver.validate_problem_compatibility(problem_data): - return SolverResult.create_error_result( - f"Solver {solver_name} cannot handle {problem_data.problem_class} problems" - ) - - # 4. Execute solver - result = solver.solve(problem_data, timeout=timeout) - return result - - except ValueError: - # Re-raise ValueError so EAFP pattern in runner can catch it - raise + # Determine problem library and type + problem_library = problem_config.get('library_name', 'internal') - def create_solver(self, solver_name: str) -> SolverInterface: - """Create Python solver instance (EAFP approach)""" - if solver_name not in self.PYTHON_SOLVER_CONFIGS: - raise ValueError(f"'{solver_name}' is not a Python solver") - - # Try to create solver instance directly - solver_config = self.PYTHON_SOLVER_CONFIGS[solver_name] - try: - solver = solver_config["class"](**solver_config["kwargs"]) - return solver - except Exception as e: - raise ValueError(f"Failed to create solver '{solver_name}': {e}") + # Get problem type from result.additional_info (set by interfaces) + if hasattr(result, 'additional_info') and result.additional_info: + problem_type = result.additional_info.get('problem_class', 'UNKNOWN') + else: + problem_type = problem_config.get('problem_type', 'UNKNOWN') - def get_available_solvers(self) -> List[str]: - """Get list of available Python solvers (lazy detection)""" - if self._available_solvers is None: - self._available_solvers = self._detect_available_solvers() - return self._available_solvers.copy() + # Store in database with standardized schema + self.db.store_result( + solver_name=solver_name, + solver_version=result.solver_version, + problem_library=problem_library, + problem_name=problem_name, + problem_type=problem_type, + # ... all standardized fields + ) ``` -#### MATLAB Interface Module (EAFP Implementation) +--- + +## Implementation Guidelines + +### 1. Fair Benchmarking Principles + +#### Minimal Configuration Approach ```python -class MatlabInterface: - """Interface for managing MATLAB solver ecosystem with EAFP approach""" - - def __init__(self, save_solutions: bool = False, - problem_interface: Optional[ProblemInterface] = None, - matlab_executable: str = 'matlab', - use_octave: bool = False, - timeout: Optional[float] = 300): - self.save_solutions = save_solutions - self.problem_interface = problem_interface or ProblemInterface() - self.matlab_executable = matlab_executable - self.use_octave = use_octave - - # Lazy initialization - solvers detected only when needed - self._available_solvers = None - - def solve(self, problem_name: str, solver_name: str, - problem_data: Optional[ProblemData] = None, - timeout: Optional[float] = None) -> SolverResult: - """Unified solve method matching Python interface signature""" - - try: - # 1. Create solver instance (will raise ValueError if not a MATLAB solver) - solver = self.create_solver(solver_name) - - # 2. Load problem data if not provided - if problem_data is None: - problem_data = self.problem_interface.load_problem(problem_name) - - # 3. Ensure problem data has name attribute for MATLAB resolution - if not hasattr(problem_data, 'name'): - problem_data.name = problem_name - - # 4. Execute solver - result = solver.solve(problem_data, timeout=timeout or self.default_timeout) - return result - - except ValueError: - # Re-raise ValueError so EAFP pattern in runner can catch it - raise - - def create_solver(self, solver_name: str) -> MatlabSolver: - """Create MATLAB solver instance (EAFP approach)""" - if solver_name not in self.MATLAB_SOLVER_CONFIGS: - raise ValueError(f"'{solver_name}' is not a MATLAB solver") - - # Try to create solver instance directly - solver_config = self.MATLAB_SOLVER_CONFIGS[solver_name] - try: - solver = solver_config["class"]( - matlab_executable=self.matlab_executable, - use_octave=self.use_octave, - save_solutions=self.save_solutions - ) - return solver - except Exception as e: - raise ValueError(f"Failed to create solver '{solver_name}': {e}") - - def get_available_solvers(self) -> List[str]: - """Get list of available MATLAB solvers (lazy detection)""" - if self._available_solvers is None: - self._available_solvers = self._detect_available_solvers() - return self._available_solvers.copy() +# Python solvers - use defaults with verbose=False only +cvxpy_options = {'verbose': False} +scipy_options = {'method': 'highs', 'verbose': False} + +# MATLAB solvers - use defaults with output suppression only +sedumi_options = {'fid': 0} # Suppress output +sdpt3_options = {'printlevel': 0} # Suppress output ``` -#### Individual Solver Implementations -```python -class CVXPYSolver(SolverInterface): - """CVXPY solver with multiple backend support""" +#### Standardized Metrics Calculation +```matlab +% scripts/solvers/matlab_octave/matlab_interface.m +function result = calculate_solver_metrics(result, x, y, A, b, c, K) + % Calculate standardized metrics for fair comparison - def __init__(self, backend: str): - self.backend = backend # 'CLARABEL', 'SCS', 'ECOS', 'OSQP' - self.solver_version = self.detect_version() - - def solve(self, problem: ProblemData) -> SolverResult: - """Solve using CVXPY with specified backend""" - # Convert to CVXPY format - # Solve with specified backend - # Extract standardized results - - def detect_version(self) -> str: - """Detect CVXPY and backend versions""" - # Return format: "cvxpy-1.4.0+CLARABEL-0.6.0" - -class SciPySolver(SolverInterface): - """SciPy optimization solvers""" + % Primal objective: c'*x + if ~isempty(x) && ~isempty(c) + result.primal_objective_value = c(:)' * x(:); + end - def solve(self, problem: ProblemData) -> SolverResult: - """Solve using appropriate SciPy method based on problem type""" - -class MatlabSolver(SolverInterface): - """Base class for MATLAB solvers (SeDuMi, SDPT3)""" + % Dual objective: b'*y + if ~isempty(y) && ~isempty(b) + result.dual_objective_value = b(:)' * y(:); + end - def solve(self, problem: ProblemData) -> SolverResult: - """Interface between Python and MATLAB solver systems""" - # Problem registry resolution - # MATLAB environment management - # Result conversion back to Python -``` - -### 4. Benchmark Execution and Database Storage - -#### Benchmark Runner -```python -class BenchmarkRunner: - """Main benchmark execution with database storage""" + % Duality gap: |primal - dual| + if ~isnan(result.primal_objective_value) && ~isnan(result.dual_objective_value) + result.duality_gap = abs(result.primal_objective_value - result.dual_objective_value); + end - def __init__(self, database_manager: DatabaseManager): - self.db = database_manager - self.environment_info = self.gather_environment_info() - self.commit_hash = self.get_git_commit_hash() + % Primal infeasibility: ||Ax - b|| / (1 + ||b||) + if ~isempty(x) && ~isempty(A) && ~isempty(b) + primal_residual = A * x - b; + result.primal_infeasibility = norm(primal_residual) / (1 + norm(b)); + end - def run_single_benchmark(self, problem_name: str, solver_name: str) -> None: - """Run single problem-solver combination and store result""" - # Load problem using appropriate loader - # Execute solver - # Store result in database (append-only) - - def run_benchmark_batch(self, problems: List[str], solvers: List[str]) -> None: - """Run benchmark for all problem-solver combinations""" - # Loop through problems and solvers - # Call run_single_benchmark for each combination + % Dual infeasibility: cone projection distance + % (Implementation specific to cone structure K) +end ``` -### 5. Simplified Reporting System βœ… **IMPLEMENTED** - -The reporting system has been completely simplified to exactly 3 HTML reports as specified: +### 2. Error Handling and Resilience -#### **scripts/reporting/result_processor.py** - βœ… **COMPLETE** +#### Graceful Degradation Strategy ```python -@dataclass -class BenchmarkResult: - """Standardized result data structure""" - id: int - solver_name: str - solver_version: str - problem_name: str - problem_type: str - problem_library: str - status: str - solve_time: float - primal_objective_value: Optional[float] - dual_objective_value: Optional[float] - duality_gap: Optional[float] - primal_infeasibility: Optional[float] - dual_infeasibility: Optional[float] - iterations: Optional[int] - timestamp: str - commit_hash: str - -class ResultProcessor: - """Process latest results from database for reporting""" - - def get_latest_results_for_reporting(self) -> List[BenchmarkResult]: - """Get latest results using commit_hash and environment_info with timestamp tiebreaker""" +# Individual solver failures don't affect overall system +try: + result = solver_interface.solve(problem_name, solver_name) + self.store_result(solver_name, problem_name, result, problem_config) +except Exception as e: + error_msg = f"Solver {solver_name} failed on {problem_name}: {e}" + logger.error(error_msg) + self.store_error_result(solver_name, problem_name, error_msg, problem_config) + # Continue with next solver/problem combination +``` + +#### MATLAB-Specific Error Handling +```python +def _call_matlab_interface(self, problem_name: str, matlab_solver: str, + runner_function: str, timeout: float) -> SolverResult: + """Call MATLAB with comprehensive error handling""" + + try: + # Execute MATLAB process with timeout + process = subprocess.run([ + 'octave', '--eval', matlab_command + ], timeout=timeout, capture_output=True, text=True) + + # Check for MATLAB execution errors + if process.returncode != 0: + return SolverResult.create_error_result( + f"MATLAB execution failed: {process.stderr}" + ) - def get_summary_statistics(self, results: List[BenchmarkResult]) -> dict: - """Calculate comprehensive summary statistics including success rates""" + # Read result from temporary JSON file + with open(result_file, 'r') as f: + matlab_result = json.load(f) - def get_solver_performance(self, results: List[BenchmarkResult]) -> List[dict]: - """Calculate per-solver performance metrics""" + # Convert to standardized SolverResult + return self._convert_matlab_result(matlab_result, solve_time) - def get_results_matrix_data(self, results: List[BenchmarkResult]) -> Tuple[List[str], List[str], dict]: - """Prepare data for problem Γ— solver matrix display""" + except subprocess.TimeoutExpired: + return SolverResult.create_timeout_result(timeout) + except Exception as e: + return SolverResult.create_error_result(str(e)) + finally: + # Always cleanup temporary files + cleanup_temporary_files() ``` -#### **scripts/reporting/html_generator.py** - βœ… **COMPLETE** +### 3. Performance Optimization + +#### Lazy Initialization Pattern ```python -class HTMLGenerator: - """Generate exactly 3 simplified HTML reports with professional styling""" - - def generate_overview(self, results: List[BenchmarkResult], summary: dict, solver_performance: List[dict]) -> str: - """Generate overview report showing summary statistics and solver comparison""" - # Professional CSS with gradients and modern design - # Summary cards with key metrics - # Solver performance comparison table - - def generate_results_matrix(self, results: List[BenchmarkResult]) -> str: - """Generate problems Γ— solvers results matrix with status visualization""" - # Matrix table with color-coded status indicators - # Status legend for interpretation - # Navigation between reports +class PythonInterface: + def __init__(self): + """Initialize interface without creating solver instances""" + self._solver_cache = {} # Cache for expensive solver objects - def generate_raw_data(self, results: List[BenchmarkResult]) -> str: - """Generate comprehensive raw data table for detailed inspection""" - # Complete data table with all result fields - # Sortable columns and professional formatting - # Timestamped result history + def create_solver(self, solver_name: str) -> SolverInterface: + """Create solver instance with caching""" + if solver_name not in self._solver_cache: + solver_class = self.SOLVER_CONFIGURATIONS[solver_name] + self._solver_cache[solver_name] = solver_class(...) + return self._solver_cache[solver_name] ``` -#### **scripts/reporting/data_exporter.py** - βœ… **COMPLETE** +#### Efficient Database Queries ```python -class DataExporter: - """Export data in JSON and CSV formats for external analysis""" - - def export_latest_results(self, results: List[BenchmarkResult], summary: dict, solver_performance: List[dict]) -> None: - """Export latest results to JSON and CSV files""" - # benchmark_results.json - Complete structured data export - # benchmark_results.csv - Flat CSV export for spreadsheets - # summary.json - Summary statistics and metadata -``` - -#### **Generated Reports** - βœ… **COMPLETE** -1. **docs/pages/index.html** - Overview dashboard with summary statistics -2. **docs/pages/results_matrix.html** - Problems Γ— solvers matrix with status visualization -3. **docs/pages/raw_data.html** - Detailed results table for inspection -4. **docs/pages/data/** - JSON/CSV data exports for external analysis - -**Key Implementation Features:** -- βœ… **Professional CSS** with gradients, cards, and modern typography -- βœ… **Case-insensitive status handling** for robust data processing -- βœ… **Navigation links** between all 3 reports -- βœ… **Embedded styling** for self-contained HTML files -- βœ… **Color-coded status indicators** (OPTIMAL=green, ERROR=red, etc.) -- βœ… **Complete data exports** in JSON and CSV formats -- βœ… **Responsive design** with clean, readable layouts - ---- - -## Requirements Management - -### Consolidated requirements.txt -``` -# Core dependencies -numpy>=1.24.0 -scipy>=1.10.0 -pandas>=2.0.0 -pyyaml>=6.0 -sqlalchemy>=2.0.0 -jinja2>=3.1.0 - -# Solver dependencies -cvxpy>=1.4.0 -clarabel>=0.6.0 -scs>=3.2.0 -ecos>=2.0.0 -osqp>=0.6.0 - -# Development and testing -pytest>=7.0.0 -pytest-cov>=4.0.0 - -# Optional: file format support -h5py>=3.8.0 # For .mat file loading +# Use indexed queries for latest results +def get_latest_results(self) -> List[BenchmarkResult]: + """Get latest results using optimized query""" + query = """ + SELECT * FROM results + WHERE (commit_hash, environment_info, timestamp) IN ( + SELECT commit_hash, environment_info, MAX(timestamp) + FROM results + GROUP BY solver_name, problem_name + ) + ORDER BY problem_library, problem_name, solver_name + """ + return self.execute_query(query) ``` --- -## Main Execution Flow +## Testing and Validation -### Command Line Interface (EAFP Implementation) +### 1. System Validation ```bash -# EAFP-based execution commands - no pre-filtering, just try it! -python main.py --benchmark --problems nb --solvers cvxpy_ecos # Single problem-solver -python main.py --benchmark --problems nb,arch0 --solvers cvxpy_clarabel,scipy_linprog # Multiple -python main.py --benchmark --library-names DIMACS,SDPLIB # Filter by library names -python main.py --benchmark --solvers unknown_solver # Will show clear error message -python main.py --report # Generate reports only -python main.py --all # Full benchmark + report -python main.py --validate # Validate environment -python main.py --dry-run # Show what would be executed -``` +# Complete environment validation +python main.py --validate-verbose -**EAFP Command Line Features:** -- **No Pre-validation**: Solvers are not checked before execution starts -- **Clear Error Messages**: Unknown solvers produce helpful error messages -- **Graceful Degradation**: System continues with other solver-problem combinations -- **Lazy Detection**: Solver availability is only checked when explicitly needed (e.g., --validate) -- **Unified Interface**: Same command patterns work for both Python and MATLAB solvers +# Solver-specific validation +python main.py --validate --solvers cvxpy_clarabel,matlab_sedumi -### EAFP Execution Workflow -``` -1. Configuration Loading (Simplified) - β”œβ”€β”€ Load config/problem_registry.yaml (flat problem structure) - β”œβ”€β”€ Initialize database connection (single denormalized table) - └── NO solver registry loading (managed by interfaces) - -2. Interface Initialization (Lazy) - β”œβ”€β”€ Initialize ProblemInterface - β”œβ”€β”€ Initialize PythonInterface (lazy solver detection) - β”œβ”€β”€ Initialize MatlabInterface if available (lazy solver detection) - └── No upfront solver availability checking - -3. Problem and Solver Selection (EAFP) - β”œβ”€β”€ Parse command line arguments (argparse-based CLI) - β”œβ”€β”€ Filter problems by library/type/test_flag - └── NO solver pre-filtering - just use user-specified solvers directly - -4. Benchmark Execution (EAFP) - β”œβ”€β”€ For each problem-solver combination: - β”‚ β”œβ”€β”€ Try Python interface first (most common case) - β”‚ β”œβ”€β”€ If ValueError, try MATLAB interface - β”‚ β”œβ”€β”€ If still ValueError, store error result - β”‚ └── Problem loading and database storage handled by interfaces - └── Continue execution despite individual failures (robust error handling) - -5. Report Generation (Unchanged) - β”œβ”€β”€ Query latest results from database (ResultProcessor) - β”œβ”€β”€ Generate exactly 3 HTML reports (HTMLGenerator) - β”œβ”€β”€ Export JSON/CSV data (DataExporter) - └── Save to docs/pages/ directory (GitHub Pages ready) +# Problem-specific validation +python main.py --validate --problems nb,arch0 ``` -**Key EAFP Workflow Changes:** -- **No Solver Registry**: Solver configurations are managed by interfaces -- **Lazy Detection**: Solver availability is only checked when explicitly needed -- **No Pre-filtering**: Solvers are not validated before execution -- **Unified Error Handling**: Clear error messages for unknown solvers -- **Faster Startup**: No upfront solver detection during initialization - -## Extension Points - -### Adding New Solvers (EAFP Implementation) -1. **Implement SolverInterface**: Create new solver class following the interface -2. **Add to Interface Configuration**: Update `*_SOLVER_CONFIGS` in appropriate interface class -3. **Update requirements.txt**: Add solver dependencies -4. **NO Code Changes Needed**: EAFP approach automatically handles new solvers -5. **Test Integration**: Validate with existing problems using --validate - -**Example: Adding New Python Solver** +### 2. Integration Testing ```python -# In scripts/solvers/python/python_interface.py -PYTHON_SOLVER_CONFIGS = { - # ... existing solvers ... - "cvxpy_new_solver": { - "class": CvxpySolver, - "display_name": "New Solver (via CVXPY)", - "kwargs": {"backend": "NEW_SOLVER"} - } -} +# Test Python-MATLAB parity on same problems +def test_solver_parity(): + problem_name = "nb" # SOCP problem + + # Run Python solver + python_result = python_interface.solve(problem_name, "cvxpy_clarabel") + + # Run MATLAB solver + matlab_result = matlab_interface.solve(problem_name, "matlab_sedumi") + + # Compare results (allowing numerical tolerance) + assert abs(python_result.primal_objective_value - + matlab_result.primal_objective_value) < 1e-6 ``` -**Benefits of EAFP for New Solvers:** -- **Single Source of Truth**: All solver info in one place -- **Automatic Integration**: No need to update multiple files -- **Consistent Error Handling**: Unknown solvers automatically handled -- **Lazy Detection**: New solver availability detected only when needed - -### Adding New Problem Libraries βœ… **VERIFIED WORKING** -1. **Create loader**: Implement format-specific loader in `scripts/data_loaders/` -2. **Update config/problem_registry.yaml**: Add library and problem metadata -3. **Test loading**: Ensure problems convert correctly to CVXPY format -4. **Validate results**: Check solver compatibility and result quality -5. **External Libraries**: DIMACS (.mat) and SDPLIB (.dat-s) fully supported - -### Adding New File Formats -1. **Implement loader**: Create format parser in `scripts/data_loaders/python/` -2. **Add converter**: Implement conversion to CVXPY representation -3. **Update loader selection**: Add format mapping in BenchmarkRunner.load_problem() -4. **Test pipeline**: Validate end-to-end problem loading and solving - ---- - -## Implementation Validation - -### Testing Strategy -- **Unit Tests**: Individual component validation (data loaders, solvers, database operations) -- **Integration Tests**: Complete workflow testing (problem loading β†’ solving β†’ storage β†’ reporting) -- **Format Tests**: Validate all problem file formats load correctly -- **Solver Tests**: Ensure all solvers produce standardized output format - -### Error Handling -- **Graceful Degradation**: Continue benchmark execution despite individual solver/problem failures -- **Comprehensive Logging**: Structured logging with clear error messages and context -- **Timeout Management**: Respect solver timeout limits and handle hanging processes -- **Data Validation**: Validate solver results before database storage - -### Security Considerations -- **Input Validation**: Validate problem file formats and configuration values -- **Resource Limits**: Enforce memory and execution time limits -- **Dependency Management**: Explicit version pinning for reproducible environments - ---- +### 3. Performance Benchmarking +```bash +# Measure execution time across all solvers +python tests/performance/benchmark_matlab_vs_python.py -*This re-architected design document provides technical specifications for the simplified, maintainable optimization solver benchmark system. For high-level concepts and project vision, see [basic_design.md](basic_design.md).* +# Memory usage analysis +python tests/performance/memory_profiling.py +``` --- -## Migration and Implementation Plan - -### Phase 1: Database and Configuration (Week 1) -1. **Database Restructuring**: Implement single denormalized results table -2. **Configuration Consolidation**: Move and restructure configuration files -3. **Requirements Consolidation**: Merge all requirements into single file - -### Phase 2: Core Architecture (Week 2) -4. **Data Loaders Implementation**: Create ETL system for all problem formats -5. **Solver Interface Standardization**: Implement standardized solver output format -6. **Database Manager**: Implement append-only database operations - -### Phase 3: Benchmark and Reporting (Week 3) -7. **Benchmark Runner**: Implement problem-solver execution loop with database storage -8. **Simplified Reporting**: Generate three focused HTML reports (overview, matrix, raw data) -9. **Data Export**: JSON/CSV export functionality +## Deployment and Maintenance -### Phase 4: Integration and Testing (Week 4) -10. **End-to-End Testing**: Validate complete workflow from execution to reporting -11. **Performance Validation**: Ensure no significant performance regression -12. **Documentation Updates**: Update all documentation to reflect new architecture +### 1. GitHub Actions Integration +```yaml +# .github/workflows/benchmark.yml +name: Solver Benchmark +on: + schedule: + - cron: '0 2 * * 1' # Weekly Monday 2AM + workflow_dispatch: + +jobs: + benchmark: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Setup Python + uses: actions/setup-python@v4 + with: + python-version: '3.12' + - name: Setup Octave + run: sudo apt-get install octave + - name: Install dependencies + run: pip install -r requirements.txt + - name: Run benchmarks + run: python main.py --all + - name: Deploy to GitHub Pages + uses: peaceiris/actions-gh-pages@v3 + with: + github_token: ${{ secrets.GITHUB_TOKEN }} + publish_dir: ./docs/pages +``` + +### 2. Monitoring and Alerting +```python +# Monitor solver success rates +def monitor_solver_health(): + """Alert if solver success rate drops below threshold""" + results = get_latest_results() + + for solver_name in get_available_solvers(): + solver_results = [r for r in results if r.solver_name == solver_name] + success_rate = len([r for r in solver_results if r.status == 'OPTIMAL']) / len(solver_results) + + if success_rate < 0.8: # Alert threshold + send_alert(f"Solver {solver_name} success rate: {success_rate:.1%}") +``` ---- +### 3. Data Backup and Recovery +```bash +# Automated database backup +cp database/results.db database/results_$(date +%Y%m%d).backup -*Last Updated: June 2025 - Re-Architecture Design Complete* +# Recovery from backup +cp database/results_20241201.backup database/results.db +``` --- -## Summary +## Future Extensions -This re-architected design focuses on: +### 1. Additional Solver Support +```python +# Adding new solver types +class MOSEKInterface: + """Interface for MOSEK commercial solver""" + +class GurobiInterface: + """Interface for Gurobi commercial solver""" +``` -1. **Simplicity**: Single denormalized database table, consolidated configuration files -2. **Maintainability**: Clear separation of concerns with dedicated ETL and solver modules -3. **Reliability**: Append-only database, graceful error handling, comprehensive logging -4. **Extensibility**: Modular design for adding new solvers, problems, and file formats -5. **Clean Break**: Fresh start without backward compatibility constraints +### 2. Advanced Analytics +```python +# Performance trend analysis +def analyze_solver_trends(): + """Analyze solver performance over time""" + +# Convergence analysis +def analyze_convergence_patterns(): + """Study solver convergence behavior""" +``` -### Key Benefits +### 3. Cloud Deployment +```yaml +# Kubernetes deployment configuration +apiVersion: apps/v1 +kind: CronJob +metadata: + name: solver-benchmark +spec: + schedule: "0 2 * * 1" + jobTemplate: + spec: + template: + spec: + containers: + - name: benchmark + image: solver-benchmark:latest + command: ["python", "main.py", "--all"] +``` -- **Reduced Complexity**: Eliminates multi-table relationships and complex aggregation logic -- **Historical Preservation**: Append-only database maintains complete execution history -- **Fair Benchmarking**: Standardized solver interface ensures consistent result format -- **Easy Reporting**: Latest results query using commit_hash and environment_info with timestamp tiebreaker -- **Format Flexibility**: ETL system supports multiple problem file formats with unified conversion +--- -This design provides a solid foundation for long-term development while addressing the complexity issues of the previous architecture. +This technical design provides a comprehensive foundation for the optimization solver benchmark system, supporting both current functionality and future extensions while maintaining the core principles of fair benchmarking and production reliability. \ No newline at end of file From e0f440dd0fe8fd3b879b2314c64621e74e9cb065 Mon Sep 17 00:00:00 2001 From: napinoco Date: Sun, 13 Jul 2025 00:26:23 +0900 Subject: [PATCH 03/13] Complete Phase 6 documentation updates and cleanup MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - CLAUDE.md: Updated development status to production ready with MATLAB integration - history.md: Added Phase 6 completion with comprehensive MATLAB architecture details - tasks.md: Updated to reflect production-ready status with 11 solvers - Remove matlab_integration_design.md: Content merged into detail_design.md Phase 6 achievements: - MATLAB architecture simplification (4-file β†’ 2-file for new solvers) - Field name consistency across Python/MATLAB interfaces - Unified solver interface design with symmetrical architecture - Complete documentation consolidation πŸ€– Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude --- CLAUDE.md | 4 +- docs/development/history.md | 89 ++ docs/development/matlab_integration_design.md | 1152 ---------------- docs/development/tasks.md | 1154 ++--------------- 4 files changed, 201 insertions(+), 2198 deletions(-) delete mode 100644 docs/development/matlab_integration_design.md diff --git a/CLAUDE.md b/CLAUDE.md index 384692e..ed59759 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -22,8 +22,8 @@ This is an automated benchmark system for optimization solvers (LP, QP, SOCP, SD ### **REQUIRED READING** (Read these documents carefully before any development): 1. **[πŸ“‹ Basic Design](docs/development/basic_design.md)** - High-level concepts, project vision, and development roadmap -2. **[πŸ—οΈ Detailed Design](docs/development/detail_design.md)** - Complete technical architecture and implementation specifications -3. **[πŸ“ Development Conventions](docs/development/conventions.md)** - Coding standards, git protocols, and engineering guidelines +2. **[πŸ—οΈ Detailed Design](docs/development/detail_design.md)** - Complete technical architecture and implementation specifications @docs/development/detail_design.md +3. **[πŸ“ Development Conventions](docs/development/conventions.md)** - Coding standards, git protocols, and engineering guidelines @docs/development/conventions.md 4. **[βœ… Current Tasks](docs/development/tasks.md)** - Active development tasks for current phase 5. **[πŸ“š Development History](docs/development/history.md)** - Complete record of completed development phases diff --git a/docs/development/history.md b/docs/development/history.md index 2d9a6a0..351e75f 100644 --- a/docs/development/history.md +++ b/docs/development/history.md @@ -198,6 +198,95 @@ Phase 3 focused on simplifying the system architecture to ensure reliability and - **Test**: PR preview workflow completes successfully βœ… - **Estimated Time**: 30 minutes +--- + +## Phase 6: MATLAB/Octave Solver Integration (COMPLETED βœ…) +**Timeline**: June 2025 +**Status**: Complete MATLAB integration with production-ready reliability + +### Overview +Phase 6 successfully implemented MATLAB/Octave optimization solver integration (SeDuMi and SDPT3) using a loose-coupling architecture with Python-MATLAB bridge via JSON data exchange. The integration maintains fair benchmarking principles while expanding solver coverage to include the MATLAB ecosystem. + +### Major Achievements +- βœ… **MATLAB Solver Integration**: SeDuMi and SDPT3 solvers fully integrated +- βœ… **Symmetrical Interface Architecture**: Unified Python/MATLAB solver interfaces +- βœ… **Problem Loading Unification**: MAT and DAT loaders in both Python and MATLAB +- βœ… **Architecture Simplification**: Reduced coupling, streamlined interfaces +- βœ… **Field Name Consistency**: Unified field naming across Python/MATLAB +- βœ… **Dual Calculation Fixes**: Resolved CVXPY duality calculation issues +- βœ… **Database Improvements**: Fixed problem_type classification accuracy + +### Technical Enhancements + +#### Sprint 1: MATLAB Data Infrastructure βœ… +- **matlab_json_formatter.m**: Standardized JSON output formatting +- **matlab_version_detection.m**: Version detection utilities +- **Enhanced loaders**: mat_loader.m and dat_loader.m with error handling + +#### Sprint 2: Complete Solver Integration βœ… +- **SeDuMi Integration**: Full SeDuMi solver with standardized interface +- **SDPT3 Integration**: Complete SDPT3 solver implementation +- **Git Submodules**: Proper external solver dependency management + +#### Sprint 3: Architecture Simplification βœ… +- **Unified Interfaces**: Single entry point matlab_interface.m +- **Dynamic Function Calls**: feval() for runtime solver selection +- **Reduced File Dependencies**: From 4-file to 2-file modification for new solvers + +### Current System Statistics (Post-Phase 6) +``` +Solver Coverage: 11 total solvers +β”œβ”€β”€ Python Solvers: 9 (SciPy, CVXPY backends) +└── MATLAB Solvers: 2 (SeDuMi, SDPT3) + +Problem Type Coverage: +β”œβ”€β”€ LP: 100% success rate (all solvers) +β”œβ”€β”€ QP: 100% success rate (all solvers) +β”œβ”€β”€ SOCP: ~43% success rate (CLARABEL, SCS, ECOS, SeDuMi, SDPT3) +└── SDP: ~29% success rate (CLARABEL, SCS, SeDuMi, SDPT3) + +External Libraries: 139 problems +β”œβ”€β”€ DIMACS: 47 problems (SeDuMi .mat format) +└── SDPLIB: 92 problems (SDPA .dat-s format) +``` + +### Code Quality Improvements + +#### Database and Result Consistency βœ… +- **Fixed problem_type=UNKNOWN**: Added problem_class extraction to interfaces +- **Unified field names**: primal_objective_value, dual_objective_value, duality_gap +- **CVXPY duality fixes**: Resolved SOC projection dimension mismatches + +#### Architecture Cleanup βœ… +- **Simplified BenchmarkRunner**: Removed registry dependencies +- **Clean main.py**: Streamlined entry point without unused functions +- **Test reorganization**: Updated import paths and removed obsolete tests +- **macOS cleanup**: Removed .DS_Store files and added .gitignore entries + +### Documentation Updates βœ… +- **Unified Technical Design**: Merged MATLAB integration into detail_design.md +- **Comprehensive Architecture**: Complete Python/MATLAB interface specifications +- **Implementation Guidelines**: Fair benchmarking principles and error handling + +### Final System Architecture +``` +BenchmarkRunner (Unified) +β”œβ”€β”€ PythonInterface +β”‚ β”œβ”€β”€ cvxpy_clarabel, cvxpy_scs, cvxpy_ecos, cvxpy_osqp +β”‚ └── scipy_linprog +β”œβ”€β”€ MatlabInterface +β”‚ β”œβ”€β”€ matlab_sedumi +β”‚ └── matlab_sdpt3 +└── ProblemInterface + β”œβ”€β”€ MATLoader (DIMACS .mat files) + β”œβ”€β”€ DATLoader (SDPLIB .dat-s files) + β”œβ”€β”€ MPSLoader (LP .mps files) + β”œβ”€β”€ QPSLoader (QP .qps files) + └── PythonLoader (Python definitions) +``` + +Phase 6 represents the successful completion of multi-language solver integration while maintaining system reliability, fair benchmarking principles, and production readiness. The system now supports both Python and MATLAB solvers with unified interfaces and consistent result formats. + **Task 4.3: Validate Published Site** βœ… - **Objective**: Ensure published GitHub Pages site works correctly - **Scope**: Test all aspects of published site functionality diff --git a/docs/development/matlab_integration_design.md b/docs/development/matlab_integration_design.md deleted file mode 100644 index 30b391e..0000000 --- a/docs/development/matlab_integration_design.md +++ /dev/null @@ -1,1152 +0,0 @@ -# MATLAB/Octave Solver Integration - Technical Design - -## Overview - -This document provides the detailed technical design for integrating MATLAB/Octave optimization solvers (SeDuMi and SDPT3) into the existing optimization solver benchmark system. The design maintains the core principles of fair benchmarking, modular architecture, and production reliability while extending solver coverage to include MATLAB ecosystem. - ---- - -## Design Philosophy - -### Core Principles -- **Loose Coupling**: Python-MATLAB integration via command-line execution and JSON data exchange -- **Minimal Configuration**: Use solver defaults to maintain fair benchmarking philosophy -- **Production Reliability**: No disruption to existing 139+ working problems and 9 Python solvers -- **Standardized Interface**: MATLAB solvers implement the same SolverInterface pattern -- **Error Resilience**: Individual MATLAB solver failures don't affect the overall system - -### Architecture Strategy -- **Python Entry Point**: Main execution remains in Python for consistency -- **MATLAB Worker Pattern**: MATLAB executes as a worker process with structured input/output -- **JSON Bridge**: Structured data exchange between Python and MATLAB environments -- **Temporary File Management**: Safe handling of intermediate files with automatic cleanup - ---- - -## System Architecture - -### High-Level Data Flow -``` -Python main.py - └── BenchmarkRunner.run_single_benchmark() - └── MatlabSolver.solve() - β”œβ”€β”€ Create temporary JSON input file - β”œβ”€β”€ Execute: matlab -batch "matlab_runner('problem', 'solver', 'temp_result.json')" - β”œβ”€β”€ Read JSON result file - β”œβ”€β”€ Convert to SolverResult - β”œβ”€β”€ Cleanup temporary files - └── Return standardized result -``` - -### Component Architecture -``` -scripts/ -β”œβ”€β”€ data_loaders/ -β”‚ └── matlab_octave/ # MATLAB data loading functions -β”‚ β”œβ”€β”€ mat_loader.m # SeDuMi .mat file loader -β”‚ └── dat_loader.m # SDPLIB .dat-s file loader -β”‚ -β”œβ”€β”€ solvers/ -β”‚ └── matlab_octave/ # MATLAB solver implementations -β”‚ β”œβ”€β”€ matlab_interface.py # Python interface class -β”‚ β”œβ”€β”€ sedumi_runner.m # SeDuMi solver execution -β”‚ β”œβ”€β”€ sdpt3_runner.m # SDPT3 solver execution -β”‚ └── matlab_runner.m # Main MATLAB orchestrator -β”‚ -└── benchmark/ - └── runner.py # Extended to support MATLAB solvers -``` - ---- - -## Detailed Component Design - -### 1. MATLAB Data Loaders - -#### `scripts/data_loaders/matlab_octave/mat_loader.m` -```matlab -function [A, b, c, K] = mat_loader(file_path) -% Load SeDuMi format .mat file and extract optimization problem data -% -% Input: -% file_path: Path to .mat file containing SeDuMi format data -% -% Output: -% A: Constraint matrix (sparse) -% b: Right-hand side vector -% c: Objective vector -% K: Cone structure -% -% The function handles: -% - Compressed .mat.gz files (automatic decompression) -% - Standard .mat files -% - Error handling for corrupted files - -try - % Load .mat file (handles .gz automatically in modern MATLAB) - data = load(file_path); - - % Extract SeDuMi format fields - if isfield(data, 'A') && isfield(data, 'b') && isfield(data, 'c') - A = data.A; - b = data.b; - c = data.c; - - % Extract or construct cone structure - if isfield(data, 'K') - K = data.K; - else - % Default cone structure for problems without explicit cones - K = struct(); - K.f = 0; % Number of free variables - K.l = length(c); % Number of linear inequality constraints - end - else - error('Invalid .mat file: missing required fields A, b, or c'); - end - -catch ME - error('Failed to load .mat file: %s', ME.message); -end -end -``` - -#### `scripts/data_loaders/matlab_octave/dat_loader.m` -```matlab -function [A, b, c, K] = dat_loader(file_path) -% Load SDPLIB format .dat-s file and convert to SeDuMi format -% -% Input: -% file_path: Path to .dat-s file containing SDPA sparse format data -% -% Output: -% A: Constraint matrix (sparse) -% b: Right-hand side vector -% c: Objective vector -% K: Cone structure -% -% The function parses SDPA sparse format and converts to SeDuMi format - -try - % Open file for reading - fid = fopen(file_path, 'r'); - if fid == -1 - error('Cannot open file: %s', file_path); - end - - % Parse SDPA header - m = fscanf(fid, '%d', 1); % Number of constraints - nblocks = fscanf(fid, '%d', 1); % Number of blocks - - % Read block sizes - block_sizes = fscanf(fid, '%d', nblocks); - - % Read objective vector c - c = fscanf(fid, '%f', m); - - % Initialize matrices - total_vars = sum(block_sizes.^2); % Total variables for SDP blocks - A = sparse(m, total_vars); - - % Parse constraint matrices - while ~feof(fid) - line = fgets(fid); - if ischar(line) - data = sscanf(line, '%d %d %d %d %f'); - if length(data) == 5 - % Process matrix entry: constraint, block, row, col, value - % Convert to SeDuMi format indexing - % ... (detailed parsing implementation) - end - end - end - - fclose(fid); - - % Construct cone structure for SDP - K = struct(); - K.f = 0; - K.l = 0; - K.s = block_sizes; % SDP block sizes - - % Create right-hand side vector (typically zeros for feasibility) - b = zeros(m, 1); - -catch ME - if exist('fid', 'var') && fid ~= -1 - fclose(fid); - end - error('Failed to load .dat-s file: %s', ME.message); -end -end -``` - -### 2. MATLAB Solver Runners - -#### `scripts/solvers/matlab_octave/sedumi_runner.m` -```matlab -function [x, y, info] = sedumi_runner(A, b, c, K, solver_options) -% Execute SeDuMi solver and return solutions -% -% Input: -% A, b, c, K: SeDuMi format optimization problem -% solver_options: Optional solver parameters (struct) -% -% Output: -% x: Primal solution vector -% y: Dual solution vector -% info: Solver information structure - -% Set default options for fair benchmarking -if nargin < 5 || isempty(solver_options) - solver_options = struct(); -end - -% Apply minimal configuration (fair benchmarking principle) -pars = struct(); -pars.fid = 0; % Suppress output for clean benchmarking -pars.eps = 1e-8; % Default tolerance -pars.bigeps = 1e-3; % Default feasibility tolerance - -% Override with user options if provided -if isfield(solver_options, 'eps') - pars.eps = solver_options.eps; -end - -try - % Execute SeDuMi solver with minimal configuration - [x, y, info] = sedumi(A, b, c, K, pars); - -catch ME - % Handle solver errors - return empty solutions - x = []; - y = []; - info = struct(); - info.error_message = ME.message; -end -end -``` - -#### `scripts/solvers/matlab_octave/sdpt3_runner.m` -```matlab -function [x, y, info] = sdpt3_runner(A, b, c, K, solver_options) -% Execute SDPT3 solver and return solutions -% -% Input: -% A, b, c, K: SeDuMi format optimization problem -% solver_options: Optional solver parameters (struct) -% -% Output: -% x: Primal solution vector (in SeDuMi format) -% y: Dual solution vector -% info: Solver information structure - -% Set default options for fair benchmarking -if nargin < 5 || isempty(solver_options) - solver_options = struct(); -end - -% SDPT3 default options (minimal configuration) -options = sqlparameters; -options.printlevel = 0; % Suppress output -options.gaptol = 1e-8; % Default duality gap tolerance -options.inftol = 1e-8; % Default infeasibility tolerance - -% Override with user options if provided -if isfield(solver_options, 'gaptol') - options.gaptol = solver_options.gaptol; -end - -try - % Convert SeDuMi format to SDPT3 format - [blk, At, C, b_sdpt3] = read_sedumi(A, b, c, K); - - % Execute SDPT3 solver - [obj, X, y, Z, info, runhist] = sqlp(blk, At, C, b_sdpt3, options); - - % Convert SDPT3 solution back to SeDuMi format using built-in function - if ~isempty(X) && iscell(X) - try - [x, ~, ~] = SDPT3soln_SEDUMIsoln(blk, At, C, b_sdpt3, X, y, Z); - catch - x = []; - end - else - x = []; - end - -catch ME - % Handle solver errors - return empty solutions - x = []; - y = []; - info = struct(); - info.error_message = ME.message; -end -end -``` - -### 3. MATLAB Integration Orchestrator - -#### `scripts/solvers/matlab_octave/matlab_runner.m` -```matlab -function matlab_runner(problem_name, solver_name, result_file, save_solutions) -% Main MATLAB orchestrator for benchmark execution -% -% Input: -% problem_name: Name of problem from problem_registry.yaml -% solver_name: Name of solver ('sedumi' or 'sdpt3') -% result_file: Path to output JSON file for results -% save_solutions: (optional) Boolean flag to save solutions to .mat file -% -% This function: -% 1. Loads problem_registry.yaml configuration -% 2. Resolves problem file path and type -% 3. Loads problem data using appropriate loader -% 4. Executes specified solver and gets solutions -% 5. Calculates metrics using solver_metrics_calculator -% 6. Saves results to JSON file -% 7. Optionally saves solution vectors to .mat file - -try - % Add necessary paths for solvers and loaders - addpath(genpath('.')); - - % Load problem registry configuration - config = load_problem_registry(); - - % Validate problem exists - if ~isfield(config.problem_libraries, problem_name) - error('Unknown problem: %s', problem_name); - end - - problem_config = config.problem_libraries.(problem_name); - - % Resolve file path - file_path = problem_config.file_path; - file_type = problem_config.file_type; - - % Load problem data using appropriate loader - if strcmp(file_type, 'mat') - [A, b, c, K] = mat_loader(file_path); - elseif strcmp(file_type, 'dat-s') - [A, b, c, K] = dat_loader(file_path); - else - error('Unsupported file type: %s', file_type); - end - - % Execute solver and get solutions - if strcmp(solver_name, 'sedumi') - [x, y, info] = sedumi_runner(A, b, c, K); - result = create_sedumi_result(info); - elseif strcmp(solver_name, 'sdpt3') - [x, y, info] = sdpt3_runner(A, b, c, K); - result = create_sdpt3_result(info); - else - error('Unknown solver: %s', solver_name); - end - - % Calculate metrics using shared function - if ~isempty(x) && ~isempty(y) - result = solver_metrics_calculator(result, x, y, A, b, c); - end - - % Save solution vectors if requested and solver succeeded - if save_solutions && strcmp(result.status, 'optimal') && ~isempty(x) && ~isempty(y) - save_solution_file(problem_name, solver_name, x, y, save_solutions); - end - - % Convert result to JSON-compatible format - json_result = struct(); - json_result.solve_time = result.solve_time; - json_result.status = result.status; - - % Handle optional numeric fields (convert [] to null) - if isempty(result.primal_objective_value) - json_result.primal_objective_value = []; - else - json_result.primal_objective_value = result.primal_objective_value; - end - - if isempty(result.dual_objective_value) - json_result.dual_objective_value = []; - else - json_result.dual_objective_value = result.dual_objective_value; - end - - if isempty(result.duality_gap) - json_result.duality_gap = []; - else - json_result.duality_gap = result.duality_gap; - end - - if isempty(result.primal_infeasibility) - json_result.primal_infeasibility = []; - else - json_result.primal_infeasibility = result.primal_infeasibility; - end - - if isempty(result.dual_infeasibility) - json_result.dual_infeasibility = []; - else - json_result.dual_infeasibility = result.dual_infeasibility; - end - - if isempty(result.iterations) - json_result.iterations = []; - else - json_result.iterations = result.iterations; - end - - json_result.solver_version = result.solver_version; - json_result.matlab_version = result.matlab_version; - - % Save result to JSON file - json_text = jsonencode(json_result); - fid = fopen(result_file, 'w'); - if fid == -1 - error('Cannot create result file: %s', result_file); - end - fprintf(fid, '%s', json_text); - fclose(fid); - - fprintf('MATLAB solver execution completed successfully\n'); - -catch ME - % Save error result to JSON file - error_result = struct(); - error_result.solve_time = 0; - error_result.status = 'error'; - error_result.primal_objective_value = []; - error_result.dual_objective_value = []; - error_result.duality_gap = []; - error_result.primal_infeasibility = []; - error_result.dual_infeasibility = []; - error_result.iterations = []; - error_result.solver_version = 'unknown'; - error_result.matlab_version = version(); - error_result.error_message = ME.message; - - json_text = jsonencode(error_result); - fid = fopen(result_file, 'w'); - if fid ~= -1 - fprintf(fid, '%s', json_text); - fclose(fid); - end - - fprintf('MATLAB solver execution failed: %s\n', ME.message); - exit(1); % Exit with error code -end - -% Exit successfully -exit(0); -end - -function config = load_problem_registry() -% Load problem registry YAML configuration -% This is a simplified YAML parser for the specific structure we need - -config_file = 'config/problem_registry.yaml'; -if ~exist(config_file, 'file') - error('Problem registry file not found: %s', config_file); -end - -% Read YAML file (basic parsing for our structure) -% In practice, this would use a proper YAML parser -% For now, assume we can access the data structure directly - -% Placeholder: In real implementation, parse YAML or use JSON equivalent -config = struct(); -config.problem_libraries = struct(); - -% This would be replaced with actual YAML parsing -% config = yaml.loadFile(config_file); -end -``` - -### 4. Python Integration Interface - -#### `scripts/solvers/matlab_octave/matlab_interface.py` -```python -""" -MATLAB Solver Integration for Optimization Benchmark System. - -This module provides a Python interface for MATLAB optimization solvers (SeDuMi, SDPT3) -that integrates with the existing benchmark system architecture. It uses command-line -execution and JSON data exchange for loose coupling between Python and MATLAB. -""" - -import os -import sys -import json -import subprocess -import tempfile -import time -from pathlib import Path -from typing import Optional, Dict, Any -import uuid - -# Add project root to path for imports -project_root = Path(__file__).parent.parent.parent.parent -sys.path.insert(0, str(project_root)) - -from scripts.solvers.solver_interface import SolverInterface, SolverResult -from scripts.data_loaders.problem_loader import ProblemData -from scripts.utils.logger import get_logger - -logger = get_logger("matlab_interface") - - -class MatlabSolver(SolverInterface): - """Python interface for MATLAB optimization solvers.""" - - SUPPORTED_SOLVERS = { - 'sedumi': 'SeDuMi', - 'sdpt3': 'SDPT3' - } - - def __init__(self, matlab_solver: str, matlab_executable: str = 'matlab', - timeout: Optional[float] = 300, use_octave: bool = False, **kwargs): - """ - Initialize MATLAB solver interface. - - Args: - matlab_solver: MATLAB solver name ('sedumi' or 'sdpt3') - matlab_executable: Path to MATLAB executable - timeout: Solver timeout in seconds - use_octave: Use Octave instead of MATLAB - **kwargs: Additional configuration parameters - """ - if matlab_solver not in self.SUPPORTED_SOLVERS: - raise ValueError(f"Unsupported MATLAB solver: {matlab_solver}. " - f"Supported: {list(self.SUPPORTED_SOLVERS.keys())}") - - # Generate solver name for registration - solver_name = f"matlab_{matlab_solver}" - - super().__init__(solver_name, matlab_solver=matlab_solver, - matlab_executable=matlab_executable, timeout=timeout, **kwargs) - - self.matlab_solver = matlab_solver - self.matlab_executable = matlab_executable - self.timeout = timeout - self.use_octave = use_octave - - # Verify MATLAB/Octave availability - self._verify_matlab_availability() - - logger.info(f"Initialized MATLAB solver '{self.solver_name}' " - f"using {matlab_solver} via {matlab_executable}") - - def _verify_matlab_availability(self) -> None: - """Verify that MATLAB/Octave is available and can execute.""" - try: - cmd = [self.matlab_executable, '-batch', 'disp("MATLAB OK")'] - if self.use_octave: - cmd = [self.matlab_executable, '--eval', 'disp("Octave OK")'] - - result = subprocess.run( - cmd, - capture_output=True, - text=True, - timeout=30, - cwd=project_root - ) - - if result.returncode != 0: - raise RuntimeError(f"MATLAB/Octave execution failed: {result.stderr}") - - except subprocess.TimeoutExpired: - raise RuntimeError(f"MATLAB/Octave verification timed out") - except FileNotFoundError: - raise RuntimeError(f"MATLAB/Octave executable not found: {self.matlab_executable}") - - def solve(self, problem_data: ProblemData, timeout: Optional[float] = None) -> SolverResult: - """ - Solve optimization problem using MATLAB solver. - - Args: - problem_data: Problem data in unified format - timeout: Optional timeout override - - Returns: - SolverResult with standardized fields - """ - solve_timeout = timeout or self.timeout - start_time = time.time() - - # Generate unique temporary file names - temp_id = str(uuid.uuid4())[:8] - temp_dir = tempfile.gettempdir() - result_file = os.path.join(temp_dir, f"matlab_result_{temp_id}.json") - - try: - # Verify problem data has required SeDuMi format fields - if not all(hasattr(problem_data, field) for field in ['A', 'b', 'c', 'K']): - return SolverResult.create_error_result( - "Problem data missing required SeDuMi format fields (A, b, c, K)", - solve_time=time.time() - start_time, - solver_name=self.solver_name, - solver_version=self.get_version() - ) - - # Get problem name from metadata or use default - problem_name = getattr(problem_data, 'name', 'unknown_problem') - - # Execute MATLAB solver via command line - matlab_command = f"matlab_runner('{problem_name}', '{self.matlab_solver}', '{result_file}')" - - if self.use_octave: - cmd = [self.matlab_executable, '--eval', matlab_command] - else: - cmd = [self.matlab_executable, '-batch', matlab_command] - - logger.debug(f"Executing MATLAB command: {' '.join(cmd)}") - - # Execute with timeout - result = subprocess.run( - cmd, - capture_output=True, - text=True, - timeout=solve_timeout, - cwd=project_root - ) - - solve_time = time.time() - start_time - - # Check execution success - if result.returncode != 0: - error_msg = f"MATLAB execution failed (code {result.returncode}): {result.stderr}" - logger.error(error_msg) - return SolverResult.create_error_result( - error_msg, - solve_time=solve_time, - solver_name=self.solver_name, - solver_version=self.get_version() - ) - - # Read JSON result file - if not os.path.exists(result_file): - return SolverResult.create_error_result( - "MATLAB solver did not produce result file", - solve_time=solve_time, - solver_name=self.solver_name, - solver_version=self.get_version() - ) - - with open(result_file, 'r') as f: - matlab_result = json.load(f) - - # Convert MATLAB result to SolverResult - return self._convert_matlab_result(matlab_result, solve_time) - - except subprocess.TimeoutExpired: - return SolverResult.create_timeout_result( - solve_timeout, - solver_name=self.solver_name, - solver_version=self.get_version() - ) - - except Exception as e: - solve_time = time.time() - start_time - logger.error(f"MATLAB solver execution failed: {e}") - return SolverResult.create_error_result( - str(e), - solve_time=solve_time, - solver_name=self.solver_name, - solver_version=self.get_version() - ) - - finally: - # Cleanup temporary files - try: - if os.path.exists(result_file): - os.remove(result_file) - except: - pass # Ignore cleanup failures - - def _convert_matlab_result(self, matlab_result: Dict[str, Any], solve_time: float) -> SolverResult: - """Convert MATLAB JSON result to SolverResult format.""" - - # Extract solver version information - solver_version = matlab_result.get('solver_version', 'unknown') - matlab_version = matlab_result.get('matlab_version', 'unknown') - combined_version = f"{solver_version} (MATLAB {matlab_version})" - - # Handle None/null values from JSON - def safe_float(value): - return None if value is None or value == [] else float(value) - - def safe_int(value): - return None if value is None or value == [] else int(value) - - try: - return SolverResult( - solve_time=solve_time, - status=matlab_result.get('status', 'unknown').upper(), - primal_objective_value=safe_float(matlab_result.get('primal_objective_value')), - dual_objective_value=safe_float(matlab_result.get('dual_objective_value')), - duality_gap=safe_float(matlab_result.get('duality_gap')), - primal_infeasibility=safe_float(matlab_result.get('primal_infeasibility')), - dual_infeasibility=safe_float(matlab_result.get('dual_infeasibility')), - iterations=safe_int(matlab_result.get('iterations')), - solver_name=self.solver_name, - solver_version=combined_version, - additional_info={ - 'matlab_output': matlab_result, - 'matlab_version': matlab_version - } - ) - except Exception as e: - # If conversion fails, return error result - return SolverResult.create_error_result( - f"Failed to convert MATLAB result: {e}", - solve_time=solve_time, - solver_name=self.solver_name, - solver_version=combined_version - ) - - def get_version(self) -> str: - """Get MATLAB solver version information.""" - try: - # Try to get version from MATLAB - cmd_map = { - 'sedumi': "disp(sedumi_version())", - 'sdpt3': "disp('SDPT3-4.0')" # Default version - } - - cmd_str = cmd_map.get(self.matlab_solver, "disp('unknown')") - - if self.use_octave: - cmd = [self.matlab_executable, '--eval', cmd_str] - else: - cmd = [self.matlab_executable, '-batch', cmd_str] - - result = subprocess.run( - cmd, - capture_output=True, - text=True, - timeout=10, - cwd=project_root - ) - - if result.returncode == 0: - version = result.stdout.strip() - return f"{self.SUPPORTED_SOLVERS[self.matlab_solver]} {version}" - else: - return f"{self.SUPPORTED_SOLVERS[self.matlab_solver]} (version unknown)" - - except Exception: - return f"{self.SUPPORTED_SOLVERS[self.matlab_solver]} (version detection failed)" - - def validate_problem_compatibility(self, problem_data: ProblemData) -> bool: - """Check if problem is compatible with MATLAB solver.""" - # Check for required SeDuMi format fields - required_fields = ['A', 'b', 'c', 'K'] - has_required = all(hasattr(problem_data, field) for field in required_fields) - - if not has_required: - logger.warning(f"Problem missing required SeDuMi fields for {self.solver_name}") - return False - - # Check problem type compatibility - problem_type = getattr(problem_data, 'problem_type', '').upper() - - # Both SeDuMi and SDPT3 support LP, QP, SOCP, SDP - supported_types = ['LP', 'QP', 'SOCP', 'SDP'] - - if problem_type not in supported_types: - logger.warning(f"Problem type {problem_type} not supported by {self.solver_name}") - return False - - return True - - -class SeDuMiSolver(MatlabSolver): - """Convenience class for SeDuMi solver.""" - - def __init__(self, **kwargs): - super().__init__(matlab_solver='sedumi', **kwargs) - - -class SDPT3Solver(MatlabSolver): - """Convenience class for SDPT3 solver.""" - - def __init__(self, **kwargs): - super().__init__(matlab_solver='sdpt3', **kwargs) -``` - ---- - -## Configuration Integration - -### 1. Solver Registry Extension - -Update `config/solver_registry.yaml`: -```yaml -# Existing Python solvers -solvers: - scipy_linprog: - display_name: "SciPy linprog" - - cvxpy_clarabel: - display_name: "CLARABEL (via CVXPY)" - - # ... other existing solvers ... - - # New MATLAB solvers - matlab_sedumi: - display_name: "SeDuMi (MATLAB)" - - matlab_sdpt3: - display_name: "SDPT3 (MATLAB)" -``` - -### 2. Benchmark Runner Integration - -Extend `scripts/benchmark/runner.py`: -```python -# Add import -from scripts.solvers.matlab_octave.matlab_interface import SeDuMiSolver, SDPT3Solver - -# Update create_solver method -def create_solver(self, solver_name: str) -> SolverInterface: - """Create solver instance based on solver name""" - - # Existing solver creation logic... - - # Add MATLAB solver support - elif solver_name == "matlab_sedumi": - return SeDuMiSolver() - elif solver_name == "matlab_sdpt3": - return SDPT3Solver() - else: - raise ValueError(f"Unknown solver: {solver_name}") -``` - ---- - -## Enhanced Temporary File Management System (Task 12) - -### 1. Overview - -The enhanced temporary file management system provides robust, concurrent-safe handling of temporary files for Python-MATLAB data exchange. This system was implemented as part of Task 12 to ensure reliable file-based communication without conflicts. - -### 2. Architecture - -#### Core Components -- **TempFileManager Class**: Centralized temporary file management -- **Context Manager**: Automatic cleanup with `temp_file_context()` -- **Orphaned File Cleanup**: Age-based cleanup of abandoned files -- **Concurrent Safety**: Process ID + UUID naming prevents conflicts - -#### File Naming Strategy -``` -Format: {prefix}_{process_id}_{timestamp}_{uuid}.{extension} -Example: matlab_sedumi_result_12345_1751374419_a1b2c3d4.json -``` - -### 3. Key Features - -#### Unique File Generation -```python -from scripts.utils.temp_file_manager import TempFileManager - -manager = TempFileManager("matlab_result") -temp_file = manager.create_temp_file(".json") # Atomic creation -``` - -#### Context Manager for Automatic Cleanup -```python -from scripts.utils.temp_file_manager import temp_file_context - -with temp_file_context(".json") as temp_file: - # Use temp file - with open(temp_file, 'w') as f: - json.dump(result_data, f) - # File automatically cleaned up on exit -``` - -#### Orphaned File Cleanup -```python -# Clean up files older than 1 hour -manager = TempFileManager(cleanup_age_hours=1) -cleaned_count = manager.cleanup_orphaned_files() -``` - -#### Error Handling and Fallbacks -- Multiple temp directory options (system temp, project temp, /tmp, current dir) -- Graceful handling of disk full, permission errors -- Safe cleanup that ignores missing files -- Detailed logging for debugging - -### 4. Integration with MATLAB Solver - -#### Python Side (matlab_interface.py) -```python -class MatlabSolver(SolverInterface): - def __init__(self, matlab_solver: str, **kwargs): - # Initialize with solver-specific temp manager - self.temp_manager = TempFileManager( - base_prefix=f"matlab_{matlab_solver}_result", - cleanup_age_hours=1 - ) - - def solve(self, problem_data: ProblemData, timeout: Optional[float] = None) -> SolverResult: - # Clean up old files before starting - self.temp_manager.cleanup_orphaned_files() - - # Use context manager for automatic cleanup - with temp_file_context(".json") as result_file: - # Execute MATLAB with temp file - matlab_command = f"matlab_runner('{problem_name}', '{self.matlab_solver}', '{result_file}')" - # ... execute and read result - # File automatically cleaned up -``` - -#### MATLAB Side (matlab_runner.m) -```matlab -function matlab_runner(problem_name, solver_name, result_file, save_solutions) - % MATLAB receives the result file path from Python - % Writes JSON result to the specified file - - % ... solve problem ... - - % Save result to JSON file specified by Python - json_text = jsonencode(json_result); - fid = fopen(result_file, 'w'); - fprintf(fid, '%s', json_text); - fclose(fid); -end -``` - -### 5. Concurrency and Safety - -#### Process Isolation -- Each Python process uses unique process ID in file names -- UUID component ensures uniqueness within process -- No shared state between concurrent executions - -#### File System Safety -- Atomic file creation using `os.O_CREAT | os.O_EXCL` -- Proper file permissions (0o600 - owner read/write only) -- Safe cleanup that handles missing files gracefully - -#### Resource Management -- Automatic cleanup via context managers -- Age-based cleanup of orphaned files -- Statistics tracking for monitoring - -### 6. Testing and Validation - -#### Comprehensive Test Suite -```bash -python tests/test_temp_file_management.py -``` - -**Test Coverage:** -- βœ… Unique file name generation -- βœ… Atomic file creation and cleanup -- βœ… Context manager automatic cleanup -- βœ… Orphaned file cleanup (age-based) -- βœ… Concurrent file creation without conflicts -- βœ… File statistics tracking -- βœ… Error handling scenarios - -### 7. Performance Characteristics - -#### Overhead Analysis -- File name generation: ~1ms -- File creation: ~5-10ms (system dependent) -- Cleanup: ~1-2ms per file -- Orphaned file scan: ~10-50ms (depends on temp directory size) - -#### Scalability -- Handles thousands of concurrent temp files -- Cleanup scales linearly with file count -- Memory usage minimal (no caching of file lists) - -### 8. Configuration Options - -#### TempFileManager Parameters -```python -manager = TempFileManager( - base_prefix="custom_prefix", # Base name for temp files - cleanup_age_hours=2 # Age threshold for orphaned cleanup -) -``` - -#### Environment Variables (Future Enhancement) -- `MATLAB_TEMP_DIR`: Override temp directory location -- `MATLAB_CLEANUP_AGE`: Override cleanup age threshold -- `MATLAB_TEMP_PREFIX`: Override temp file prefix - -### 9. Monitoring and Debugging - -#### Statistics Collection -```python -stats = manager.get_temp_file_stats() -# Returns: -# { -# 'total_files': 5, -# 'total_size_bytes': 12458, -# 'temp_directory': '/tmp', -# 'oldest_file': '/tmp/matlab_result_12345_1751374419_a1b2c3d4.json', -# 'oldest_age_hours': 0.25 -# } -``` - -#### Logging Integration -- All temp file operations logged at DEBUG level -- Cleanup actions logged at INFO level -- Errors logged at WARNING/ERROR levels -- Compatible with existing logger system - -### 10. Future Enhancements - -#### Planned Improvements -- **Batch Cleanup**: Cleanup multiple files in single operation -- **Compression**: Optional compression for large result files -- **Encryption**: Optional encryption for sensitive temporary data -- **Network Storage**: Support for shared/network temporary directories -- **Metrics Collection**: Integration with monitoring systems - ---- - -## Testing Strategy - -### 1. Unit Testing - -#### MATLAB Function Tests -- **Data Loader Tests**: Verify mat_loader.m and dat_loader.m correctly parse files -- **Solver Runner Tests**: Test sedumi_runner.m and sdpt3_runner.m with simple problems -- **Integration Tests**: Verify matlab_runner.m orchestrates correctly - -#### Python Integration Tests -- **Command Execution**: Test MATLAB command line execution and JSON parsing -- **Error Handling**: Verify graceful handling of MATLAB failures -- **Timeout Handling**: Test solver timeout and cleanup - -### 2. Integration Testing - -#### Small Problem Tests -- **Test Problems**: Use nb (DIMACS) and arch0 (SDPLIB) for initial validation -- **Result Consistency**: Compare MATLAB solver results with CVXPY solvers on same problems -- **Performance Baseline**: Establish execution time baselines - -#### End-to-End Workflow -- **Full Pipeline**: Test complete Python β†’ MATLAB β†’ Database workflow -- **Error Scenarios**: Test handling of corrupted files, solver failures, timeouts -- **Cleanup Verification**: Ensure temporary files are properly removed - -### 3. Performance Testing - -#### Execution Time Analysis -- **Overhead Measurement**: Quantify Python-MATLAB communication overhead -- **Solver Performance**: Compare native MATLAB performance vs CVXPY equivalents -- **Scalability Testing**: Test with larger problems from DIMACS/SDPLIB - ---- - -## Error Handling Strategy - -### 1. MATLAB Environment Errors -- **Installation Detection**: Verify MATLAB/Octave availability at startup -- **Path Configuration**: Validate solver paths and dependencies -- **License Issues**: Handle MATLAB license failures gracefully - -### 2. Execution Errors -- **Solver Failures**: Capture MATLAB solver errors and convert to standard format -- **Memory Issues**: Handle large problems that exceed MATLAB memory limits -- **Timeout Handling**: Clean termination of long-running MATLAB processes - -### 3. Data Exchange Errors -- **File I/O Errors**: Handle temporary file creation/deletion failures -- **JSON Parsing**: Validate JSON format and handle parsing errors -- **Type Conversion**: Safe conversion between MATLAB and Python data types - -### 4. Graceful Degradation -- **Individual Failures**: System continues if one MATLAB solver fails -- **Fallback Behavior**: Option to skip MATLAB solvers if environment unavailable -- **Error Reporting**: Clear error messages for debugging and user guidance - ---- - -## Performance Considerations - -### 1. Startup Overhead -- **MATLAB Initialization**: MATLAB startup can be slow (~2-5 seconds) -- **Mitigation**: Consider MATLAB session persistence for multiple problems -- **Alternative**: Use Octave for faster startup times - -### 2. Memory Management -- **Large Problems**: MATLAB may require more memory than Python solvers -- **Cleanup**: Ensure proper cleanup of MATLAB variables and temporary files -- **Monitoring**: Track memory usage during MATLAB execution - -### 3. Parallel Execution -- **Process Isolation**: Each MATLAB solver runs in separate process -- **Resource Limits**: Consider MATLAB license limits for parallel execution -- **Coordination**: Manage temporary file conflicts in parallel scenarios - ---- - -## Security Considerations - -### 1. Command Injection Prevention -- **Input Sanitization**: Validate all inputs passed to MATLAB command line -- **Path Validation**: Ensure file paths are within expected directories -- **Command Construction**: Use safe command construction methods - -### 2. Temporary File Security -- **File Permissions**: Set appropriate permissions on temporary files -- **Cleanup Assurance**: Ensure cleanup even in error scenarios -- **Name Collision**: Use unique temporary file names - -### 3. Environment Isolation -- **Working Directory**: Execute MATLAB in controlled working directory -- **Path Restrictions**: Limit MATLAB path access to required directories -- **Resource Limits**: Apply timeout and memory limits to MATLAB processes - ---- - -## Deployment Considerations - -### 1. Environment Setup -- **MATLAB Installation**: Require MATLAB R2020a+ or Octave 6.0+ -- **Solver Installation**: Document SeDuMi and SDPT3 installation procedures -- **Path Configuration**: Ensure solver paths are correctly configured - -### 2. CI/CD Integration -- **GitHub Actions**: Consider MATLAB availability in CI environment -- **Octave Alternative**: Use Octave for CI if MATLAB unavailable -- **Skip Strategy**: Graceful handling when MATLAB solvers unavailable - -### 3. Documentation -- **Installation Guide**: Step-by-step MATLAB/Octave setup instructions -- **Troubleshooting**: Common issues and solutions -- **Performance Tuning**: Optimization recommendations - ---- - -## Future Enhancements - -### 1. Additional Solvers -- **MOSEK**: Add MOSEK MATLAB interface -- **CVX**: Integrate CVX modeling language -- **YALMIP**: Support YALMIP optimization framework - -### 2. Performance Optimization -- **Session Persistence**: Maintain MATLAB session for multiple problems -- **Batch Processing**: Process multiple problems in single MATLAB session -- **Memory Optimization**: Optimize memory usage for large problems - -### 3. Enhanced Integration -- **Native MEX**: Consider MEX interface for tighter integration -- **Direct Data Transfer**: Binary data transfer instead of JSON -- **Streaming Results**: Real-time result streaming for long-running problems - ---- - -*This technical design provides a comprehensive foundation for integrating MATLAB/Octave solvers while maintaining the system's core principles of reliability, modularity, and fair benchmarking.* - -*Last Updated: December 2025* \ No newline at end of file diff --git a/docs/development/tasks.md b/docs/development/tasks.md index 45527c7..a139f40 100644 --- a/docs/development/tasks.md +++ b/docs/development/tasks.md @@ -1,1086 +1,152 @@ -# Development Tasks - Phase 6: MATLAB/Octave Solver Integration +# Development Tasks - Current Status -**Phase**: MATLAB/Octave Solver Integration -**Priority**: High - User-requested solver ecosystem expansion -**Target**: Add SeDuMi and SDPT3 solvers to expand solver coverage and validate system extensibility +**Current Phase**: Production Ready βœ… +**Status**: All major development phases completed successfully --- -## Phase Overview +## πŸ“‹ System Status -This phase implements MATLAB/Octave optimization solver integration (SeDuMi and SDPT3) using a loose-coupling architecture. The integration maintains the core principles of fair benchmarking while extending solver coverage to include the MATLAB ecosystem. +### Completed Phases +- βœ… **Phase 1**: MVP Foundation (November - December 2024) +- βœ… **Phase 2**: Data Publishing Platform (December 2024 - March 2025) +- βœ… **Phase 3**: System Simplification and Reliability (March - June 2025) +- βœ… **Phase 6**: MATLAB/Octave Solver Integration (June 2025) -**Architecture**: Python β†’ MATLAB β†’ Python with JSON data exchange -**Scope**: Production-ready integration with comprehensive testing -**Impact**: Expand from 9 to 11 solvers, add MATLAB ecosystem validation +### Current System Capabilities +``` +Solver Coverage: 11 total solvers +β”œβ”€β”€ Python Solvers: 9 (SciPy, CVXPY backends) +└── MATLAB Solvers: 2 (SeDuMi, SDPT3) ---- - -## Current System Status βœ… - -**Pre-Integration Capabilities**: -- βœ… 139+ external problems (DIMACS + SDPLIB) working correctly -- βœ… 9 Python solvers with comprehensive backend support -- βœ… Stable MAT/DAT loader architecture (Python-based) -- βœ… Production-ready reporting and database systems -- βœ… Comprehensive testing infrastructure with --dry-run mode - -**Architecture Readiness**: Phase 5 analysis confirmed compatibility for MATLAB integration - ---- - -## Sprint Schedule (Execute Sequentially per conventions.md) - -### **Sprint 1: MATLAB Environment and Data Loading** πŸš€ **WEEKS 1-2** -**Objective**: Establish MATLAB execution environment and implement data loaders -**Deliverable**: Working MATLAB data loaders with Python integration - -### **Sprint 2: MATLAB Solver Implementation** ⚑ **WEEKS 3-4** -**Objective**: Implement SeDuMi and SDPT3 solver runners with standardized output -**Deliverable**: MATLAB solver functions producing JSON results - -### **Sprint 3: Integration Orchestration** πŸ”§ **WEEKS 5-6** -**Objective**: Implement MATLAB-Python bridge and unified orchestration -**Deliverable**: Complete integration allowing Python to execute MATLAB solvers - -### **Sprint 4: Python Interface Integration** 🐍 **WEEKS 7-8** -**Objective**: Integrate MATLAB solvers into existing Python benchmark system -**Deliverable**: MATLAB solvers available through standard BenchmarkRunner interface - -### **Sprint 5: Testing and Production Deployment** βœ… **WEEKS 9-10** -**Objective**: Comprehensive testing and production-ready deployment -**Deliverable**: Production system with 11 solvers including MATLAB integration - ---- - -## Detailed Task Breakdown - -### **Sprint 1: MATLAB Environment and Data Loading (Tasks 1-5)** - -#### **Task 1: MATLAB Environment Setup and Validation** πŸ”§ HIGH PRIORITY -**Objective**: Establish and validate MATLAB/Octave execution environment -**Context**: Foundation for all MATLAB integration work - -**Steps**: -1. Verify MATLAB/Octave installation and accessibility -2. Test command-line execution (`matlab -batch` and `octave --eval`) -3. Verify SeDuMi and SDPT3 solver availability in MATLAB path -4. Test basic solver functionality (simple optimization problem) -5. Document environment requirements and setup procedures - -**Success Criteria**: -- [ ] MATLAB executable responds to command-line calls -- [ ] SeDuMi executes successfully on simple test problem -- [ ] SDPT3 executes successfully on simple test problem -- [ ] Environment detection script created and tested -- [ ] Setup documentation written with troubleshooting guide - -**Test Criteria**: -- Basic MATLAB execution: `matlab -batch "disp('Hello World')"` -- SeDuMi test: Solve simple LP problem and verify solution -- SDPT3 test: Solve simple SDP problem and verify solution -- Error handling: Test behavior with invalid MATLAB path - -**Files Modified**: -- `docs/guides/MATLAB_SETUP.md` (new) -- `scripts/utils/matlab_detection.py` (new) - -**Estimated Time**: 8-12 hours -**Dependencies**: None - ---- - -#### **Task 2: MATLAB SeDuMi Data Loader Implementation** πŸ“ HIGH PRIORITY -**Objective**: Implement MATLAB function to load SeDuMi .mat files -**Context**: Core data loading capability for DIMACS problems - -**Steps**: -1. Create `scripts/data_loaders/matlab_octave/mat_loader.m` -2. Implement .mat file loading with error handling -3. Support compressed .mat.gz files (automatic decompression) -4. Extract A, b, c, K matrices in SeDuMi format -5. Validate output format and handle edge cases - -**Success Criteria**: -- [ ] Function loads standard .mat files correctly -- [ ] Function handles compressed .mat.gz files -- [ ] Proper error handling for corrupted/missing files -- [ ] Validates required fields (A, b, c) exist -- [ ] Constructs default cone structure K when missing -- [ ] Unit tests pass for sample DIMACS problems - -**Test Criteria**: -- Load nb.mat (small DIMACS problem) and verify A,b,c,K extraction -- Test with compressed and uncompressed files -- Error handling: Test with invalid/corrupted .mat files -- Edge cases: Test with missing K field, empty matrices - -**Files Modified**: -- `scripts/data_loaders/matlab_octave/mat_loader.m` (new) -- `tests/unit/test_matlab_mat_loader.m` (new) - -**Estimated Time**: 6-8 hours -**Dependencies**: Task 1 (MATLAB environment) - ---- - -#### **Task 3: MATLAB SDPLIB Data Loader Implementation** πŸ“ HIGH PRIORITY -**Objective**: Implement MATLAB function to load SDPLIB .dat-s files -**Context**: Core data loading capability for SDPLIB problems - -**Steps**: -1. Create `scripts/data_loaders/matlab_octave/dat_loader.m` -2. Implement SDPA sparse format parser -3. Convert SDPA format to SeDuMi format (A, b, c, K) -4. Handle multiple SDP blocks and cone structures -5. Robust error handling for malformed files - -**Success Criteria**: -- [ ] Function parses SDPA sparse format correctly -- [ ] Converts to valid SeDuMi format (A, b, c, K) -- [ ] Handles multiple SDP blocks properly -- [ ] Constructs appropriate cone structure for SDP problems -- [ ] Error handling for malformed/incomplete files -- [ ] Unit tests pass for sample SDPLIB problems - -**Test Criteria**: -- Load arch0.dat-s (small SDPLIB problem) and verify conversion -- Test with multi-block SDP problems -- Verify cone structure K correctly represents SDP blocks -- Error handling: Test with malformed SDPA files - -**Files Modified**: -- `scripts/data_loaders/matlab_octave/dat_loader.m` (new) -- `tests/unit/test_matlab_dat_loader.m` (new) - -**Estimated Time**: 8-10 hours -**Dependencies**: Task 1 (MATLAB environment) - ---- - -#### **Task 4: YAML Configuration Reader for MATLAB** βš™οΈ MEDIUM PRIORITY -**Objective**: Implement MATLAB function to read problem_registry.yaml -**Context**: Enable MATLAB to resolve problem names to file paths - -**Steps**: -1. Create `scripts/utils/matlab_yaml_reader.m` -2. Implement basic YAML parsing for problem_registry structure -3. Support problem name to file path resolution -4. Handle missing problems and configuration errors -5. Test with current problem_registry.yaml structure - -**Success Criteria**: -- [ ] Function reads problem_registry.yaml successfully -- [ ] Resolves problem names to file paths correctly -- [ ] Handles missing problems gracefully -- [ ] Supports both DIMACS and SDPLIB problem entries -- [ ] Error handling for malformed YAML files -- [ ] Compatible with existing problem_registry.yaml format - -**Test Criteria**: -- Load current problem_registry.yaml and verify parsing -- Resolve known problem names (nb, arch0) to correct file paths -- Error handling: Test with missing YAML file, invalid syntax -- Validate problem metadata extraction (file_type, library_name) - -**Files Modified**: -- `scripts/utils/matlab_yaml_reader.m` (new) -- `tests/unit/test_matlab_yaml_reader.m` (new) - -**Estimated Time**: 6-8 hours -**Dependencies**: Task 1 (MATLAB environment) - ---- - -#### **Task 5: Data Loader Integration Testing** βœ… MEDIUM PRIORITY -**Objective**: Comprehensive testing of MATLAB data loading pipeline -**Context**: Validate data loaders work correctly with real problems - -**Steps**: -1. Test mat_loader.m with multiple DIMACS problems -2. Test dat_loader.m with multiple SDPLIB problems -3. Verify data consistency between MATLAB and Python loaders -4. Performance testing with larger problems -5. Create integration test suite - -**Success Criteria**: -- [ ] All DIMACS test problems load correctly via mat_loader -- [ ] All SDPLIB test problems load correctly via dat_loader -- [ ] Data consistency verified between MATLAB and Python loaders -- [ ] Performance acceptable for production use -- [ ] Comprehensive test suite covering edge cases -- [ ] Error scenarios handled gracefully - -**Test Criteria**: -- Compare MATLAB loader output with Python MAT/DAT loaders -- Test with 5+ problems from each library (DIMACS, SDPLIB) -- Performance: Measure loading time for representative problems -- Memory usage: Verify no memory leaks in MATLAB - -**Files Modified**: -- `tests/integration/test_matlab_data_loaders.m` (new) -- `tests/performance/benchmark_matlab_loaders.m` (new) - -**Estimated Time**: 4-6 hours -**Dependencies**: Tasks 2, 3, 4 (All data loaders) - ---- +Problem Coverage: 139+ problems +β”œβ”€β”€ DIMACS: 47 problems (SeDuMi .mat format) +└── SDPLIB: 92 problems (SDPA .dat-s format) -### **Sprint 2: MATLAB Solver Implementation (Tasks 6-10)** - -#### **Task 6: SeDuMi Solver Runner Implementation** πŸ”§ HIGH PRIORITY -**Objective**: Implement MATLAB function to execute SeDuMi with standardized output -**Context**: Core solver execution for SeDuMi integration - -**Steps**: -1. Create `scripts/solvers/matlab_octave/sedumi_runner.m` -2. Implement SeDuMi solver execution with minimal configuration -3. Extract standardized solver metrics (time, status, objectives, etc.) -4. Map SeDuMi status codes to standard format -5. Handle solver errors and edge cases gracefully - -**Success Criteria**: -- [ ] Function executes SeDuMi solver correctly -- [ ] Uses minimal configuration for fair benchmarking -- [ ] Extracts all required standardized metrics -- [ ] Maps SeDuMi status to standard format (optimal, infeasible, etc.) -- [ ] Handles solver errors without crashing -- [ ] Returns consistent result structure - -**Test Criteria**: -- Solve simple LP problem and verify optimal solution -- Solve infeasible problem and verify correct status -- Test error handling with invalid input -- Measure performance overhead of metric extraction - -**Files Modified**: -- `scripts/solvers/matlab_octave/sedumi_runner.m` (new) -- `tests/unit/test_sedumi_runner.m` (new) - -**Estimated Time**: 6-8 hours -**Dependencies**: Sprint 1 (Data loaders) - ---- - -#### **Task 7: SDPT3 Solver Runner Implementation** πŸ”§ HIGH PRIORITY -**Objective**: Implement MATLAB function to execute SDPT3 with standardized output -**Context**: Core solver execution for SDPT3 integration - -**Steps**: -1. Create `scripts/solvers/matlab_octave/sdpt3_runner.m` -2. Implement SDPT3 solver execution with minimal configuration -3. Extract standardized solver metrics from SDPT3 output -4. Map SDPT3 status codes to standard format -5. Handle SDPT3-specific data conversion requirements - -**Success Criteria**: -- [ ] Function executes SDPT3 solver correctly -- [ ] Handles SeDuMi to SDPT3 data format conversion -- [ ] Extracts all required standardized metrics -- [ ] Maps SDPT3 status to standard format -- [ ] Handles solver errors gracefully -- [ ] Returns consistent result structure - -**Test Criteria**: -- Solve simple SDP problem and verify optimal solution -- Test with LP/QP problems (SDPT3 multi-format support) -- Verify data conversion from SeDuMi to SDPT3 format -- Test error handling with solver failures - -**Files Modified**: -- `scripts/solvers/matlab_octave/sdpt3_runner.m` (new) -- `tests/unit/test_sdpt3_runner.m` (new) - -**Estimated Time**: 8-10 hours -**Dependencies**: Sprint 1 (Data loaders), Task 6 (SeDuMi as reference) - ---- - -#### **Task 8: Solver Version Detection and Metadata** πŸ“Š MEDIUM PRIORITY -**Objective**: Implement version detection and metadata collection for MATLAB solvers -**Context**: Required for standardized result reporting and reproducibility - -**Steps**: -1. Implement SeDuMi version detection function -2. Implement SDPT3 version detection function -3. Collect MATLAB version information -4. Create unified metadata collection function -5. Handle cases where version detection fails - -**Success Criteria**: -- [ ] SeDuMi version detected and reported correctly -- [ ] SDPT3 version detected and reported correctly -- [ ] MATLAB version information collected -- [ ] Metadata included in solver results -- [ ] Graceful fallback when version detection fails -- [ ] Version information formatted consistently - -**Test Criteria**: -- Verify version detection returns valid version strings -- Test with different MATLAB versions if available -- Handle missing solvers gracefully -- Validate version string format consistency - -**Files Modified**: -- `scripts/utils/matlab_version_detection.m` (new) -- Updated: `sedumi_runner.m`, `sdpt3_runner.m` - -**Estimated Time**: 4-6 hours -**Dependencies**: Tasks 6, 7 (Solver runners) - ---- - -#### **Task 9: JSON Result Format Implementation** πŸ“ MEDIUM PRIORITY -**Objective**: Implement standardized JSON output format for MATLAB solver results -**Context**: Enable structured data exchange between MATLAB and Python - -**Steps**: -1. Define JSON schema for solver results -2. Implement JSON encoding function in MATLAB -3. Handle null/empty values appropriately -4. Ensure compatibility with Python JSON parsing -5. Add result validation and error detection - -**Success Criteria**: -- [ ] JSON schema matches SolverResult requirements -- [ ] MATLAB jsonencode produces valid JSON -- [ ] Null/empty values handled correctly -- [ ] JSON compatible with Python json.load() -- [ ] Result validation detects invalid data -- [ ] Error results formatted consistently - -**Test Criteria**: -- Generate JSON for optimal solution and verify Python parsing -- Generate JSON for infeasible result and verify format -- Test edge cases: empty values, large numbers, special characters -- Validate JSON schema compliance - -**Files Modified**: -- `scripts/utils/matlab_json_formatter.m` (new) -- Updated: `sedumi_runner.m`, `sdpt3_runner.m` - -**Estimated Time**: 4-6 hours -**Dependencies**: Tasks 6, 7, 8 (Solver runners and metadata) - ---- - -#### **Task 10: Solver Runner Integration Testing** βœ… MEDIUM PRIORITY -**Objective**: Comprehensive testing of MATLAB solver execution pipeline -**Context**: Validate solver runners work correctly with real problems - -**Steps**: -1. Test SeDuMi with representative problems from each type (LP, QP, SOCP, SDP) -2. Test SDPT3 with representative problems from each type -3. Compare results with Python solvers for validation -4. Performance benchmarking and optimization -5. Error scenario testing and validation - -**Success Criteria**: -- [ ] SeDuMi solves all supported problem types correctly -- [ ] SDPT3 solves all supported problem types correctly -- [ ] Results consistent with Python solver results (where applicable) -- [ ] Performance meets production requirements -- [ ] Error scenarios handled gracefully -- [ ] JSON output validates correctly - -**Test Criteria**: -- Solve nb (DIMACS) with both solvers, compare results -- Solve arch0 (SDPLIB) with both solvers, compare results -- Performance: Measure solve time vs Python solvers -- Stress test: Run with larger problems, monitor memory usage - -**Files Modified**: -- `tests/integration/test_matlab_solvers.m` (new) -- `tests/performance/benchmark_matlab_solvers.m` (new) - -**Estimated Time**: 6-8 hours -**Dependencies**: Tasks 6, 7, 8, 9 (Complete solver implementation) - ---- - -### **Sprint 3: Integration Orchestration (Tasks 11-15)** - -#### **Task 11: MATLAB Integration Orchestrator** 🎯 HIGH PRIORITY -**Objective**: Implement main MATLAB orchestrator that coordinates problem loading and solving -**Context**: Central MATLAB function that Python will execute - -**Steps**: -1. Create `scripts/solvers/matlab_octave/matlab_runner.m` -2. Implement command-line argument parsing -3. Integrate problem loading, solver execution, and result output -4. Add comprehensive error handling and logging -5. Ensure clean exit codes for Python process management - -**Success Criteria**: -- [ ] Function accepts problem name, solver name, and output file as arguments -- [ ] Resolves problem name to file path using YAML configuration -- [ ] Loads problem using appropriate data loader -- [ ] Executes specified solver and collects results -- [ ] Saves results to JSON file with proper formatting -- [ ] Returns appropriate exit codes for success/failure - -**Test Criteria**: -- Execute: `matlab_runner('nb', 'sedumi', 'result.json')` and verify JSON output -- Execute: `matlab_runner('arch0', 'sdpt3', 'result.json')` and verify JSON output -- Error handling: Test with invalid problem name, solver name -- Exit codes: Verify success (0) and error (1) exit codes - -**Files Modified**: -- `scripts/solvers/matlab_octave/matlab_runner.m` (new) -- `tests/integration/test_matlab_runner.m` (new) - -**Estimated Time**: 8-10 hours -**Dependencies**: Sprint 1 (Data loaders), Sprint 2 (Solver runners) +Problem Types: LP, QP, SOCP, SDP +External Libraries: Fully integrated +Reporting: Bootstrap 5 + Chart.js interactive dashboards +``` --- -#### **Task 12: Temporary File Management System** πŸ“ MEDIUM PRIORITY -**Objective**: Implement robust temporary file management for Python-MATLAB data exchange -**Context**: Ensure reliable file-based communication without conflicts - -**Steps**: -1. Design temporary file naming strategy (unique IDs) -2. Implement file creation and cleanup in MATLAB -3. Add timeout-based cleanup for orphaned files -4. Handle concurrent execution scenarios -5. Test file system error scenarios - -**Success Criteria**: -- [ ] Unique temporary file names prevent conflicts -- [ ] Automatic cleanup of temporary files after use -- [ ] Timeout-based cleanup for orphaned files -- [ ] Concurrent execution doesn't cause file conflicts -- [ ] File system errors handled gracefully -- [ ] No temporary file leaks in normal operation +## πŸ† Recent Achievements (Latest Session) -**Test Criteria**: -- Parallel execution: Run multiple MATLAB processes simultaneously -- Cleanup verification: Confirm files removed after successful execution -- Error scenarios: Test cleanup when MATLAB crashes or times out -- File permissions: Verify appropriate file access controls +### Architecture & Code Quality βœ… +- **MATLAB Architecture Simplification**: Streamlined from 4-file to 2-file modification for new solvers +- **Field Name Consistency**: Unified field naming across Python/MATLAB interfaces +- **Database Accuracy**: Fixed problem_type=UNKNOWN issue with proper classification +- **CVXPY Duality Fixes**: Resolved SOC projection dimension mismatches in dual calculations -**Files Modified**: -- `scripts/utils/matlab_temp_manager.m` (new) -- Updated: `matlab_runner.m` - -**Estimated Time**: 4-6 hours -**Dependencies**: Task 11 (MATLAB orchestrator) +### Documentation Updates βœ… +- **Unified Technical Design**: Merged MATLAB integration documentation into detail_design.md +- **Comprehensive Specifications**: Complete Python/MATLAB interface architecture +- **Development History**: Updated with Phase 6 completion status +- **Code Cleanup**: Removed .DS_Store files and updated .gitignore --- -#### **Task 13: Command-Line Interface Validation** βœ… COMPLETED -**Objective**: Validate MATLAB command-line execution from Python environment -**Context**: Ensure reliable Python to MATLAB process execution - -**Implemented Components**: -1. **MATLAB Execution Testing Utility** (`scripts/utils/matlab_execution_test.py`) -2. **CLI Integration Test Suite** (`tests/integration/test_matlab_cli.py`) -3. **Enhanced MatlabSolver** with improved CLI reliability +## πŸ› οΈ Maintenance Tasks -**Success Criteria**: βœ… **ALL COMPLETED** -- βœ… `matlab -batch` executes MATLAB functions correctly (100% success rate) -- βœ… Argument passing with proper escaping (MATLAB quote escaping implemented) -- βœ… Process timeout and termination work correctly (5.02s vs 5s target accuracy) -- βœ… MATLAB startup delays handled appropriately (45s timeout, 15s buffer) -- βœ… Error messages captured and parsed correctly (intelligent error pattern matching) +### Ongoing Maintenance +```bash +# Regular benchmark execution +python main.py --all -**Test Results**: -- βœ… Basic execution: 5.49s average startup time -- βœ… matlab_runner execution: 33s with real problem (arch0) - **SUCCESS WITH JSON OUTPUT** -- βœ… Timeout handling: Β±2s accuracy -- βœ… Error handling: Proper error capture and reporting -- βœ… Concurrent execution: 100% success rate with 3 simultaneous processes -- βœ… **Overall Success Rate: 100%** (improved from 75% to 100%) +# Environment validation +python main.py --validate-verbose -**Key Enhancements**: -- **Enhanced Error Parsing**: Extracts meaningful MATLAB error messages -- **Safe Command Construction**: Proper argument escaping for MATLAB syntax -- **Startup Delay Handling**: Adjusted timeouts and buffering for MATLAB initialization -- **Path Management**: Automatic `addpath(genpath('.'))` for reliable function access -- **Robust Success Criteria**: Intelligent test validation considering expected behaviors +# Individual solver testing +python main.py --benchmark --problems nb --solvers cvxpy_clarabel,matlab_sedumi +``` -**Files Created/Modified**: -- βœ… `scripts/utils/matlab_execution_test.py` (comprehensive testing utility) -- βœ… `tests/integration/test_matlab_cli.py` (integration test suite) -- βœ… Enhanced `scripts/solvers/matlab_octave/matlab_solver.py` (improved reliability) +### Code Quality Monitoring +- **Linting**: Use IDE diagnostics and code analysis tools +- **Testing**: Run integration tests for Python/MATLAB solver parity +- **Documentation**: Keep design documents synchronized with implementation -**Actual Time**: 6 hours -**Dependencies**: Task 12 (Temporary File Management), existing MATLAB infrastructure +### Dependency Management +- **Python Dependencies**: Single `requirements.txt` file (no subdirectories) +- **MATLAB Submodules**: SeDuMi and SDPT3 as git submodules +- **External Libraries**: DIMACS and SDPLIB problem sets --- -#### **Task 14: Error Handling and Logging Framework** πŸ“‹ MEDIUM PRIORITY -**Objective**: Implement comprehensive error handling and logging for MATLAB integration -**Context**: Ensure reliable error detection and debugging capabilities - -**Steps**: -1. Design error handling strategy for MATLAB components -2. Implement structured logging in MATLAB functions -3. Create error classification and recovery procedures -4. Add performance monitoring and metrics collection -5. Test error scenarios and recovery mechanisms - -**Success Criteria**: -- [ ] Structured error handling across all MATLAB components -- [ ] Consistent logging format compatible with Python system -- [ ] Error classification enables appropriate recovery actions -- [ ] Performance metrics collected for monitoring -- [ ] Error scenarios tested and documented -- [ ] Debugging information available for troubleshooting +## πŸš€ Future Extensions (Ideas) -**Test Criteria**: -- Error propagation: Verify errors bubble up correctly to Python -- Logging format: Ensure MATLAB logs integrate with Python logging -- Recovery testing: Verify system continues after MATLAB errors -- Performance impact: Measure overhead of logging and error handling +### Solver Ecosystem Expansion +```python +# Commercial solver support +class MOSEKInterface: + """MOSEK commercial solver integration""" + +class GurobiInterface: + """Gurobi commercial solver integration""" +``` -**Files Modified**: -- `scripts/utils/matlab_logger.m` (new) -- `scripts/utils/matlab_error_handler.m` (new) -- Updated: All MATLAB functions +### Advanced Analytics +```python +# Performance trend analysis +def analyze_solver_convergence(): + """Study solver convergence patterns over time""" + +def benchmark_solver_scaling(): + """Analyze solver performance vs problem size""" +``` -**Estimated Time**: 6-8 hours -**Dependencies**: Tasks 11, 12, 13 (Core integration components) +### Cloud Deployment +```yaml +# Kubernetes CronJob for automated benchmarking +apiVersion: batch/v1 +kind: CronJob +metadata: + name: solver-benchmark +spec: + schedule: "0 2 * * 1" # Weekly Monday 2AM +``` --- -#### **Task 15: Integration Orchestration Testing** βœ… COMPLETED -**Objective**: End-to-end testing of complete MATLAB integration pipeline -**Context**: Validate full integration works correctly before Python interface - -**Implemented Components**: -1. **Simplified Integration Test** (`test_matlab_integration_simple.m`) -2. **Complete Pipeline Test** (`tests/integration/test_complete_matlab_pipeline.m`) -3. **Performance Benchmark** (`tests/performance/benchmark_matlab_pipeline.m`) - -**Success Criteria**: βœ… **ALL COMPLETED** -- βœ… Complete pipeline executes successfully for all test problems (100% success rate) -- βœ… JSON output format validated against Python requirements (valid JSON with all required fields) -- βœ… Performance meets production benchmarks (6.4s for arch0, within acceptable range) -- βœ… Error scenarios handled gracefully with proper logging (error JSON format implemented) -- βœ… Memory usage and cleanup verified (automatic temp file cleanup) -- βœ… Integration ready for Python interface development (**SPRINT 3 COMPLETE**) - -**Test Results**: -- βœ… **Function Availability**: All 8 required MATLAB functions accessible -- βœ… **Data Loaders**: SDPLIB (.dat-s) and DIMACS (.mat) formats working correctly - - arch0.dat-s: 26,095 variables, 174 constraints (SDP) - - nb.mat.gz: 2,383 variables, 123 constraints (SOCP) -- βœ… **Solver Runners**: Both SeDuMi and SDPT3 solve simple problems correctly - - SeDuMi: 0.32s execution time, optimal status - - SDPT3: 0.33s execution time, optimal status -- βœ… **Complete Pipeline**: Full matlab_runner integration successful - - arch0 + SeDuMi: 6.4s total time, optimal status, valid JSON output - - Temp file management: Automatic cleanup verified - - JSON validation: 446 bytes valid JSON with all required fields +## πŸ“ˆ System Health Metrics -**Key Achievements**: -- **Production-Ready Pipeline**: Complete end-to-end integration validated -- **Comprehensive Testing**: Function availability, data loading, solving, JSON output -- **Performance Validation**: Execution times within acceptable production range -- **Error Handling**: Graceful error handling with structured JSON error format -- **MATLAB Integration Complete**: Ready for Sprint 4 (Python Interface Integration) +### Success Rate Monitoring +```python +# Automated health checks +def monitor_solver_success_rates(): + """Alert if any solver success rate drops below 80%""" + +def validate_external_libraries(): + """Ensure DIMACS/SDPLIB problems still load correctly""" +``` -**Files Created**: -- βœ… `test_matlab_integration_simple.m` (simplified integration test) -- βœ… `tests/integration/test_complete_matlab_pipeline.m` (comprehensive pipeline test) -- βœ… `tests/performance/benchmark_matlab_pipeline.m` (performance validation) +### Performance Benchmarks +```bash +# Measure execution time across solvers +python tests/performance/benchmark_matlab_vs_python.py -**Actual Time**: 4 hours -**Dependencies**: Tasks 11-14 (Complete integration orchestration) +# Memory usage profiling +python tests/performance/memory_profiling.py +``` --- -### **Sprint 4: Python Interface Integration (Tasks 16-20)** +## 🎯 Current Focus: Production Stability -#### **Task 16: Python MatlabSolver Class Implementation** βœ… COMPLETED -**Objective**: Implement production-ready Python SolverInterface subclass for MATLAB solvers -**Context**: Bridge between Python benchmark system and MATLAB solvers - -**Implemented Features**: -1. **Complete SolverInterface Compliance** with standardized SolverResult format -2. **Problem Registry Integration** for DIMACS/SDPLIB problem resolution -3. **Dynamic Version Detection** with caching and fallback mechanisms -4. **Enhanced Error Handling** with intelligent MATLAB error parsing -5. **Production-Ready Integration** with comprehensive metadata and monitoring - -**Success Criteria**: βœ… **ALL COMPLETED** -- βœ… MatlabSolver inherits from SolverInterface correctly with full compliance -- βœ… solve() method executes MATLAB and returns properly formatted SolverResult -- βœ… get_version() returns dynamic version information with caching -- βœ… Timeout handling prevents hanging processes with graceful termination -- βœ… Error handling converts MATLAB errors to SolverResult errors with detailed parsing -- βœ… Temporary file cleanup works in all scenarios with enhanced management -- βœ… Problem registry integration enables seamless problem resolution -- βœ… Solver compatibility validation ensures robust problem type support -- βœ… Convenience classes (SeDuMiSolver, SDPT3Solver) provide easy instantiation - -**Test Results**: βœ… **100% SUCCESS RATE** -- βœ… **Unit Tests**: 21 comprehensive test cases covering all functionality -- βœ… **Integration Tests**: 5/5 integration scenarios passed -- βœ… **SolverInterface Compliance**: Full compatibility validated -- βœ… **Problem Registry Integration**: 142 problems loaded and validated -- βœ… **Command Construction**: Proper escaping and error handling -- βœ… **Temp File Management**: Robust concurrent execution safety -- βœ… **Version Detection**: Dynamic detection with intelligent fallbacks - -**Key Technical Achievements**: -- **Problem Resolution**: Automatic path resolution via problem registry -- **Enhanced Metadata**: Comprehensive additional_info with execution environment details -- **Concurrent Safety**: UUID-based temp file naming with cleanup management -- **Error Intelligence**: MATLAB error pattern matching and meaningful error extraction -- **Performance Optimization**: Version caching and MATLAB startup optimization -- **Registry Validation**: Problem type compatibility checking for both file formats - -**Files Created/Enhanced**: -- βœ… `scripts/solvers/matlab_octave/matlab_solver.py` (production-ready implementation) -- βœ… `tests/unit/test_matlab_solver.py` (comprehensive unit test suite) -- βœ… `tests/integration/test_enhanced_matlab_solver.py` (integration validation) - -**Actual Time**: 8 hours -**Dependencies**: Sprint 3 (Complete MATLAB integration) βœ… - -**Production Readiness**: βœ… **VALIDATED** -Ready for integration with BenchmarkRunner and main benchmark system. - ---- - -#### **Task 17: Convenience Solver Classes** βœ… COMPLETED (Implemented in Task 16) -**Objective**: Create SeDuMiSolver and SDPT3Solver convenience classes -**Context**: Provide easy instantiation of specific MATLAB solvers - -**Implementation Status**: βœ… **ALREADY COMPLETED** in Task 16 enhanced implementation - -**Implemented Features**: -- βœ… **SeDuMiSolver class** extending MatlabSolver with `matlab_solver='sedumi'` -- βœ… **SDPT3Solver class** extending MatlabSolver with `matlab_solver='sdpt3'` -- βœ… **Solver-specific configuration** through MatlabSolver constructor parameters -- βœ… **Capability detection** via enhanced `validate_problem_compatibility()` method -- βœ… **Complete integration** with existing system architecture - -**Success Criteria**: βœ… **ALL COMPLETED** -- βœ… SeDuMiSolver creates correctly configured MatlabSolver (validated in tests) -- βœ… SDPT3Solver creates correctly configured MatlabSolver (validated in tests) -- βœ… Solver-specific options handled appropriately (timeout, MATLAB executable, etc.) -- βœ… Capability detection works for problem type compatibility (DIMACS/SDPLIB support) -- βœ… Documentation provided in enhanced MatlabSolver docstrings -- βœ… Classes integrate seamlessly with existing system (validated in integration tests) - -**Test Validation**: βœ… **COMPREHENSIVE TESTING COMPLETED** -- βœ… Unit tests validate correct instantiation and configuration -- βœ… Integration tests confirm compatibility with problem registry -- βœ… Solver capability detection tested with different problem types -- βœ… BenchmarkRunner integration verified (Task 18) - -**Files Implementation**: -- βœ… `scripts/solvers/matlab_octave/matlab_solver.py` (lines 403-414) - ```python - class SeDuMiSolver(MatlabSolver): - def __init__(self, **kwargs): - super().__init__(matlab_solver='sedumi', **kwargs) - - class SDPT3Solver(MatlabSolver): - def __init__(self, **kwargs): - super().__init__(matlab_solver='sdpt3', **kwargs) - ``` -- βœ… `tests/unit/test_matlab_solver.py` (comprehensive test coverage) -- βœ… `tests/integration/test_enhanced_matlab_solver.py` (integration validation) - -**Actual Time**: 0 hours (completed as part of Task 16 enhanced implementation) -**Dependencies**: Task 16 βœ… - -**Note**: This task was automatically completed during Task 16's production-ready implementation. The convenience classes were implemented as part of the comprehensive MatlabSolver design to provide the complete interface expected by the benchmark system. - ---- - -#### **Task 18: BenchmarkRunner Integration** βœ… COMPLETED πŸ”— HIGH PRIORITY -**Objective**: Integrate MATLAB solvers into existing BenchmarkRunner system -**Context**: Enable MATLAB solvers through standard benchmark execution interface - -**Implementation Summary**: -Successfully integrated MATLAB solvers into the BenchmarkRunner system with complete functionality. The integration expands the solver count from 9 to 11 solvers, adding `matlab_sedumi` and `matlab_sdpt3` with full compatibility. - -**Key Achievements**: -1. **BenchmarkRunner.create_solver() Enhanced**: Added MATLAB solver creation with graceful degradation -2. **Solver Registry Updated**: Added MATLAB solvers to `config/solver_registry.yaml` -3. **Dynamic Availability Detection**: MATLAB solvers appear in `get_available_solvers()` when available -4. **Error Handling**: Graceful degradation when MATLAB unavailable with clear error messages -5. **Fixed Integration Bug**: Resolved duplicate `get_available_solvers()` method issue - -**Success Criteria**: βœ… **ALL COMPLETED** -- βœ… create_solver() handles 'matlab_sedumi' and 'matlab_sdpt3' correctly -- βœ… MATLAB solvers integrate seamlessly with existing benchmark workflows -- βœ… Error handling maintains system stability with graceful degradation -- βœ… Performance impact minimal - MATLAB imports only when available -- βœ… All existing functionality preserved and enhanced -- βœ… Dynamic solver availability detection working correctly - -**Test Results**: βœ… **100% SUCCESS RATE** -- βœ… **Solver Creation**: Both MATLAB solvers create successfully via BenchmarkRunner -- βœ… **Integration Testing**: MATLAB solvers appear in available solver list (11 total) -- βœ… **Compatibility Validation**: Problem compatibility validation working correctly -- βœ… **Registry Integration**: Solver registry properly includes MATLAB solvers -- βœ… **Graceful Degradation**: System handles MATLAB unavailability gracefully - -**Technical Implementation**: -- **Import Strategy**: Try/except block with `MATLAB_SOLVERS_AVAILABLE` flag -- **Registry Integration**: Added display names for MATLAB solvers -- **Availability Detection**: Enhanced `get_available_solvers()` with dynamic import checking -- **Error Messages**: Clear feedback when MATLAB not available -- **Bug Fix**: Removed duplicate method that was preventing MATLAB solver detection - -**Files Modified**: -- βœ… `scripts/benchmark/runner.py` (enhanced with MATLAB integration) -- βœ… `config/solver_registry.yaml` (added MATLAB solver entries) - -**Integration Status**: βœ… **PRODUCTION READY** -MATLAB solvers are now fully integrated and available through the standard BenchmarkRunner interface. - -**Actual Time**: 4 hours -**Dependencies**: Tasks 16, 17 (Complete Python interface) βœ… - ---- - -#### **Task 19: Configuration Integration** βœ… COMPLETED βš™οΈ MEDIUM PRIORITY -**Objective**: Update configuration files to include MATLAB solvers -**Context**: Enable MATLAB solvers through standard configuration system - -**Implementation Summary**: -Successfully integrated MATLAB solvers into the configuration system with comprehensive validation and documentation. The configuration system now properly supports both Python and MATLAB solvers with graceful degradation and enhanced validation capabilities. - -**Key Achievements**: -1. **Enhanced Validation**: Upgraded `main.py --validate` with comprehensive solver testing -2. **Documentation**: Created complete `docs/guides/CONFIGURATION.md` with MATLAB setup guide -3. **Unit Testing**: Comprehensive test suite for configuration integration -4. **Registry Integration**: MATLAB solvers properly included in `solver_registry.yaml` -5. **Graceful Degradation**: System handles MATLAB unavailability gracefully - -**Success Criteria**: βœ… **ALL COMPLETED** -- βœ… solver_registry.yaml includes matlab_sedumi and matlab_sdpt3 with proper display names -- βœ… Display names consistent with existing pattern ("SeDuMi (via MATLAB)") -- βœ… Configuration loading handles MATLAB solvers correctly with dynamic detection -- βœ… Validation checks MATLAB availability with `--validate` and `--validate-verbose` options -- βœ… Documentation explains MATLAB solver configuration with troubleshooting guide -- βœ… Backward compatibility maintained for existing configurations - -**Test Results**: βœ… **VALIDATION SUCCESSFUL** -- βœ… **Configuration Loading**: MATLAB solvers appear in solver registry (11 total solvers) -- βœ… **Enhanced Validation**: `main.py --validate` tests all solvers including MATLAB -- βœ… **Solver Filtering**: MATLAB/Python solver separation working correctly -- βœ… **Graceful Degradation**: System continues working when MATLAB unavailable -- βœ… **YAML Validation**: Configuration files valid and properly formatted - -**Technical Implementation**: -- **Validation Enhancement**: Added `validate_solver_setup()` function with detailed reporting -- **Command Line Options**: Added `--validate-verbose` for detailed solver status -- **Documentation**: Complete setup guide with troubleshooting and best practices -- **Unit Tests**: Configuration validation with mocking and error scenario testing -- **Registry Format**: Consistent display name pattern for all solvers - -**Files Modified**: -- βœ… `config/solver_registry.yaml` (added MATLAB solver entries) -- βœ… `main.py` (enhanced validation with solver testing) -- βœ… `docs/guides/CONFIGURATION.md` (comprehensive configuration guide) -- βœ… `docs/guides/README.md` (updated to include new guide) -- βœ… `tests/unit/test_config_matlab_integration.py` (comprehensive unit test suite) - -**Integration Status**: βœ… **PRODUCTION READY** -Configuration system fully supports MATLAB solvers with proper validation and documentation. - -**Actual Time**: 3 hours -**Dependencies**: Task 18 (BenchmarkRunner integration) βœ… - ---- - -#### **Task 20: Python Integration Testing** βœ… COMPLETED βœ… HIGH PRIORITY -**Objective**: Comprehensive testing of complete Python-MATLAB integration -**Context**: Validate end-to-end integration works correctly in production scenario - -**Implementation Summary**: -Successfully completed comprehensive end-to-end testing of the complete MATLAB integration. All phases passed validation, demonstrating production-ready integration with robust error handling, seamless database storage, and complete report generation capabilities. - -**Key Achievements**: -1. **End-to-End Workflow**: Complete benchmark execution validated with MATLAB solvers -2. **Database Integration**: MATLAB results seamlessly stored alongside Python results -3. **Report Generation**: HTML reports display MATLAB solvers correctly in all views -4. **Performance Analysis**: Comprehensive MATLAB vs Python performance comparison -5. **Resilience Testing**: Robust error handling and graceful degradation validated - -**Success Criteria**: βœ… **ALL COMPLETED** -- βœ… Complete benchmark workflow works with MATLAB solvers (nb problem tested successfully) -- βœ… Database storage correctly handles MATLAB solver results (45 total results, 1 MATLAB) -- βœ… HTML reports display MATLAB solver results correctly (all 3 reports + CSV/JSON exports) -- βœ… Performance acceptable for production use (detailed comparison framework created) -- βœ… System resilient to MATLAB solver failures (graceful error handling validated) -- βœ… Integration ready for production deployment (all systems operational) - -**Test Results**: βœ… **COMPREHENSIVE VALIDATION SUCCESSFUL** -- βœ… **Phase 1 - End-to-End**: 8/8 integration tests passing after fixes -- βœ… **Phase 2 - Database**: MATLAB results stored in production database format -- βœ… **Phase 3 - Reports**: MATLAB solvers appear in index.html, results_matrix.html, CSV/JSON -- βœ… **Phase 4 - Performance**: Detailed comparison shows MATLAB 11-12s creation, Python 0.01-0.02s - -**Performance Insights**: -- **MATLAB Solvers**: 11-12s creation overhead, specialized for SDP/SOCP, environment-sensitive -- **Python Solvers**: 0.01-0.02s creation overhead, general-purpose, environment-robust -- **Recommendation**: Hybrid approach - Python primary, MATLAB specialized for supported environments -- **Production Status**: Ready for deployment with proper environment configuration - -**Technical Implementation**: -- **Integration Tests**: `tests/integration/test_end_to_end_matlab.py` (comprehensive 8-test suite) -- **Performance Tests**: `tests/performance/benchmark_matlab_vs_python.py` (detailed comparison) -- **Database Integration**: MATLAB results stored with same schema as Python results -- **Report Integration**: MATLAB solvers appear seamlessly in all generated reports -- **Error Resilience**: Graceful degradation when MATLAB unavailable or fails - -**Files Created**: -- βœ… `tests/integration/test_end_to_end_matlab.py` (comprehensive integration test suite) -- βœ… `tests/performance/benchmark_matlab_vs_python.py` (performance comparison framework) -- βœ… `performance_comparison_report.json` (detailed performance analysis) - -**Integration Status**: βœ… **PRODUCTION READY** -Complete MATLAB integration validated and ready for production deployment. - -**Actual Time**: 9 hours -**Dependencies**: Tasks 16-19 (Complete Python integration) βœ… - ---- - -### **Sprint 5: Testing and Production Deployment (Tasks 21-25)** - -#### **Task 21: Comprehensive Unit Test Suite** βœ… MEDIUM PRIORITY -**Objective**: Create comprehensive unit tests for all MATLAB integration components -**Context**: Ensure code quality and regression prevention - -**Steps**: -1. Complete unit test coverage for all MATLAB functions -2. Complete unit test coverage for Python integration classes -3. Add edge case testing and error scenario coverage -4. Implement test data generation and validation -5. Set up automated test execution - -**Success Criteria**: -- [ ] >90% code coverage for MATLAB functions -- [ ] >90% code coverage for Python integration classes -- [ ] Edge cases and error scenarios covered -- [ ] Test data covers representative problem types -- [ ] Automated test execution integrated with development workflow -- [ ] Tests run successfully in CI environment (where applicable) - -**Test Criteria**: -- Run complete test suite and verify all tests pass -- Coverage report shows >90% coverage -- Tests execute in reasonable time (<5 minutes) -- Tests provide clear failure messages for debugging - -**Files Modified**: -- Complete all unit test files created in previous tasks -- `tests/run_matlab_tests.py` (new) - -**Estimated Time**: 6-8 hours -**Dependencies**: All previous tasks (Complete implementation) - ---- - -#### **Task 22: Production Problem Testing** 🏭 HIGH PRIORITY -**Objective**: Test MATLAB solvers with full production problem set -**Context**: Validate MATLAB solvers work correctly with all 139+ problems - -**Steps**: -1. Run MATLAB solvers on representative sample of DIMACS problems -2. Run MATLAB solvers on representative sample of SDPLIB problems -3. Compare results with existing Python solver results -4. Identify and resolve any compatibility issues -5. Document problem-specific behavior and limitations - -**Success Criteria**: -- [ ] MATLAB solvers execute successfully on 90%+ of test problems -- [ ] Results consistent with Python solvers where comparable -- [ ] Problem compatibility issues identified and documented -- [ ] Performance acceptable across problem range -- [ ] System stability maintained during extended execution -- [ ] Clear documentation of solver capabilities and limitations - -**Test Criteria**: -- Run MATLAB solvers on 20+ representative problems from each library -- Compare solution quality with Python solvers on same problems -- Measure execution time distribution across problem sizes -- Test system stability with continuous execution - -**Files Modified**: -- `tests/production/test_matlab_production_problems.py` (new) -- `docs/MATLAB_SOLVER_CAPABILITIES.md` (new) - -**Estimated Time**: 10-12 hours -**Dependencies**: All previous tasks (Complete system) - ---- - -#### **Task 23: Performance Optimization and Tuning** ⚑ MEDIUM PRIORITY -**Objective**: Optimize MATLAB integration performance for production use -**Context**: Ensure MATLAB integration doesn't significantly impact system performance - -**Steps**: -1. Profile MATLAB solver execution overhead -2. Optimize temporary file handling and cleanup -3. Optimize JSON serialization and parsing -4. Implement MATLAB process reuse where beneficial -5. Document performance characteristics and tuning options - -**Success Criteria**: -- [ ] MATLAB solver overhead minimized (<20% vs native execution) -- [ ] Temporary file operations optimized -- [ ] JSON processing optimized for large results -- [ ] Process reuse implemented where beneficial -- [ ] Performance characteristics documented -- [ ] Tuning options provided for different scenarios - -**Test Criteria**: -- Measure execution overhead vs native MATLAB execution -- Profile memory usage during extended operation -- Benchmark JSON processing with large result sets -- Test process reuse benefits with multiple problems - -**Files Modified**: -- Performance optimizations in existing MATLAB and Python files -- `docs/MATLAB_PERFORMANCE_TUNING.md` (new) - -**Estimated Time**: 8-10 hours -**Dependencies**: Task 22 (Production testing) - ---- - -#### **Task 24: Documentation and User Guide** πŸ“š MEDIUM PRIORITY -**Objective**: Create comprehensive documentation for MATLAB solver integration -**Context**: Enable users to understand and utilize MATLAB solver capabilities - -**Steps**: -1. Create MATLAB solver installation and setup guide -2. Document MATLAB solver capabilities and limitations -3. Create troubleshooting guide for common issues -4. Update main documentation to include MATLAB integration -5. Create examples and usage patterns - -**Success Criteria**: -- [ ] Complete installation guide for MATLAB/Octave and solvers -- [ ] Clear documentation of solver capabilities by problem type -- [ ] Comprehensive troubleshooting guide with solutions -- [ ] Updated main documentation includes MATLAB integration -- [ ] Examples provided for common usage patterns -- [ ] Documentation tested by following setup procedures - -**Test Criteria**: -- Follow installation guide and verify successful setup -- Verify troubleshooting guide addresses common issues -- Test examples and usage patterns work as documented -- Review documentation for clarity and completeness - -**Files Modified**: -- `docs/guides/MATLAB_SETUP.md` -- `docs/guides/MATLAB_TROUBLESHOOTING.md` -- `docs/guides/MATLAB_USAGE_EXAMPLES.md` -- Updated: `README.md`, `docs/development/detail_design.md` - -**Estimated Time**: 6-8 hours -**Dependencies**: Task 22 (Production testing), Task 23 (Performance optimization) - ---- - -#### **Task 25: Production Deployment and Validation** πŸš€ HIGH PRIORITY -**Objective**: Deploy MATLAB integration to production and validate complete system -**Context**: Finalize MATLAB integration and prepare for production use - -**Steps**: -1. Deploy complete system with MATLAB integration -2. Run full production benchmark with all 11 solvers -3. Generate and validate HTML reports with MATLAB results -4. Perform system stability and reliability testing -5. Create final validation report and sign-off - -**Success Criteria**: -- [ ] Production system includes working MATLAB solvers -- [ ] Full benchmark executes successfully with all solvers -- [ ] HTML reports correctly display MATLAB solver results -- [ ] System stability validated under production load -- [ ] Performance meets production requirements -- [ ] Complete validation documentation prepared - -**Test Criteria**: -- Execute: `python main.py --all` and verify all 11 solvers complete -- Verify HTML reports show matlab_sedumi and matlab_sdpt3 results -- Run extended stability test (multiple benchmark cycles) -- Validate final solver count: 11 total (9 Python + 2 MATLAB) - -**Files Modified**: -- Production deployment configurations -- `VALIDATION_REPORT_PHASE6.md` (new) - -**Estimated Time**: 6-8 hours -**Dependencies**: Tasks 21-24 (Complete testing and documentation) - ---- - -## Success Criteria for Phase 6 - -### **Technical Achievements** -- βœ… **Solver Coverage Expansion**: From 9 to 11 solvers (SeDuMi + SDPT3) -- βœ… **MATLAB Ecosystem Integration**: Production-ready MATLAB solver support -- βœ… **Architecture Validation**: Successful loose-coupling implementation -- βœ… **Fair Benchmarking**: MATLAB solvers use default configuration -- βœ… **System Stability**: No impact on existing 139+ working problems - -### **Production Readiness** -- βœ… **Performance**: MATLAB integration overhead <20% -- βœ… **Reliability**: MATLAB solver failures don't crash system -- βœ… **Documentation**: Complete setup and troubleshooting guides -- βœ… **Testing**: Comprehensive unit and integration test coverage -- βœ… **Compatibility**: Works with existing benchmark and reporting systems - -### **Validation Targets** -- **Problem Coverage**: MATLAB solvers work on 90%+ of test problems -- **Result Quality**: Solutions consistent with Python solvers where comparable -- **System Integration**: HTML reports correctly display MATLAB solver results -- **User Experience**: Clear installation guide and troubleshooting documentation - ---- - -## Risk Mitigation - -### **Technical Risks** -- **MATLAB Availability**: Graceful degradation when MATLAB not available -- **Solver Installation**: Clear documentation and validation procedures -- **Process Management**: Robust timeout and cleanup mechanisms -- **Performance Impact**: Monitoring and optimization throughout development - -### **Integration Risks** -- **System Stability**: Comprehensive testing with existing workflows -- **Backward Compatibility**: Extensive testing of existing functionality -- **Error Propagation**: Isolated error handling prevents cascade failures -- **Resource Management**: Proper cleanup prevents resource leaks - ---- - -## Dependencies and Prerequisites - -### **External Dependencies** -- MATLAB R2020a+ or Octave 6.0+ installation -- SeDuMi solver properly installed and configured -- SDPT3 solver properly installed and configured -- Command-line access to MATLAB/Octave - -### **Internal Dependencies** -- Current production system with 139+ working problems -- Existing Python solver architecture and interfaces -- Database schema and reporting system -- Testing infrastructure and validation framework - ---- +The system is now in **production-ready state** with: -*This task list provides a comprehensive roadmap for MATLAB/Octave solver integration while maintaining the system's production readiness and core principles of fair benchmarking and reliability.* +- βœ… **Stable Architecture**: Unified Python/MATLAB solver interfaces +- βœ… **Comprehensive Testing**: Integration tests and validation framework +- βœ… **Reliable Execution**: Error handling and graceful degradation +- βœ… **Clean Documentation**: Up-to-date design specifications +- βœ… **Fair Benchmarking**: Minimal configuration approach maintained -*Next Action: Begin Sprint 1, Task 1 - MATLAB Environment Setup and Validation* -*Phase Target: Complete MATLAB integration within 10 weeks* +### Next Steps (As Needed) +1. **Monitor System Health**: Regular validation runs and success rate monitoring +2. **User Feedback Integration**: Address any issues reported by users +3. **Incremental Improvements**: Small enhancements based on usage patterns +4. **Documentation Maintenance**: Keep technical specifications current --- -*Last Updated: December 2025* \ No newline at end of file +*For detailed implementation history, see [history.md](history.md)* +*For technical specifications, see [detail_design.md](detail_design.md)* \ No newline at end of file From 949fc80f872ff80f86f08a2b22df1565a26acc48 Mon Sep 17 00:00:00 2001 From: napinoco Date: Sun, 13 Jul 2025 13:10:27 +0900 Subject: [PATCH 04/13] Fix iteration extraction and pin dependency versions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Enhanced iteration extraction for CVXPY solvers with fallback methods - Document known limitations for CVXOPT and SDPA (no iteration info through CVXPY) - Add iteration_info_limitation field to memo for transparency - Pin all dependency versions for reproducible benchmarks - Update requirements.txt with documentation and version control strategy πŸ€– Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude --- requirements.txt | 38 +++++++++++++++---------- scripts/solvers/python/cvxpy_runner.py | 39 ++++++++++++++++++++++++-- 2 files changed, 59 insertions(+), 18 deletions(-) diff --git a/requirements.txt b/requirements.txt index d90a587..f203611 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,22 +1,30 @@ +# Optimization Solver Benchmark Requirements +# +# All versions are pinned to ensure reproducible benchmarks across environments. +# This is critical for fair solver comparison and consistent results. +# To update: test thoroughly and update all versions together. +# +# Last updated: 2025-07-13 + # Core dependencies -pyyaml>=6.0 -jinja2>=3.0 -psutil>=5.0 +pyyaml==6.0.2 +jinja2==3.1.6 +psutil==7.0.0 # Scientific computing -scipy>=1.9.0 -numpy>=1.21.0 -pandas>=1.5.0 +scipy==1.15.3 +numpy==2.3.0 +pandas==2.3.0 # Optimization framework -cvxpy>=1.3.0 +cvxpy==1.6.5 # Essential CVXPY backend solvers (reliable across platforms) -clarabel>=0.5.0 -scs>=3.2.0 -ecos>=2.0.0 -osqp>=0.6.0 -cvxopt>=1.3.0 -sdpa-python>=0.2.0 -pyscipopt>=5.5.0 -highspy>=1.11.0 \ No newline at end of file +clarabel==0.11.0 +scs==3.2.7.post2 +ecos==2.0.14 +osqp==1.0.4 +cvxopt==1.3.2 +sdpa-python==0.2.2 +pyscipopt==5.5.0 +highspy==1.11.0 \ No newline at end of file diff --git a/scripts/solvers/python/cvxpy_runner.py b/scripts/solvers/python/cvxpy_runner.py index 462075f..82dc4b3 100644 --- a/scripts/solvers/python/cvxpy_runner.py +++ b/scripts/solvers/python/cvxpy_runner.py @@ -460,14 +460,47 @@ def proj_onto_soc(z): import traceback self.logger.debug(f"Full traceback: {traceback.format_exc()}") - # Get iterations if available + # Get iterations if available (enhanced for multiple backends) iterations = None - if cvx_problem.solver_stats and hasattr(cvx_problem.solver_stats, 'num_iters'): - iterations = cvx_problem.solver_stats.num_iters + if cvx_problem.solver_stats: + # Primary method: standard num_iters attribute + if hasattr(cvx_problem.solver_stats, 'num_iters') and cvx_problem.solver_stats.num_iters is not None: + iterations = cvx_problem.solver_stats.num_iters + # Alternative method: check extra_stats for iteration information + elif (cvx_problem.solver_stats.extra_stats and + isinstance(cvx_problem.solver_stats.extra_stats, dict)): + extra_stats = cvx_problem.solver_stats.extra_stats + + # Try different possible keys for iteration information + possible_keys = ['iter', 'iterations', 'num_iters', 'niter'] + for key in possible_keys: + if key in extra_stats: + iterations = extra_stats[key] + break + + # For solvers that nest iteration info (e.g., SCS has info.iter) + if iterations is None and 'info' in extra_stats: + info_dict = extra_stats['info'] + if isinstance(info_dict, dict): + for key in possible_keys: + if key in info_dict: + iterations = info_dict[key] + break # Extract minimal solver timing information for memo additional_info = {} + # Log iteration extraction attempts for debugging + if iterations is not None: + self.logger.debug(f"Extracted iterations: {iterations} for backend {self.backend}") + else: + self.logger.debug(f"No iteration information available for backend {self.backend}") + # Document known limitations for specific backends + if self.backend in ['CVXOPT', 'SDPA']: + self.logger.debug(f"Backend {self.backend} does not expose iteration information through CVXPY interface") + # Add this limitation to additional_info for transparency + additional_info["iteration_info_limitation"] = f"{self.backend} does not provide iteration count through CVXPY" + # Only add solver internal timing for memo if cvx_problem.solver_stats and hasattr(cvx_problem.solver_stats, 'solve_time'): additional_info["solver_solve_time"] = cvx_problem.solver_stats.solve_time From e0b5257aa30aaee96a5bf68d1fba02da32bbd670 Mon Sep 17 00:00:00 2001 From: napinoco Date: Sun, 13 Jul 2025 13:22:06 +0900 Subject: [PATCH 05/13] Add UNSUPPORTED status for incompatible solver-problem combinations MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add SolverResult.create_unsupported_result() method for proper incompatibility handling - Make solve_time Optional to support UNSUPPORTED status (None when not applicable) - Update validation logic to handle None solve_time values - Implement pre-compatibility checking in Python interface - Return UNSUPPORTED instead of ERROR for unsupported problem types - Ensure accurate problem_type recording even for incompatible combinations - Update logging to handle None solve_time gracefully - Add problem_data parameter to store_error_result for better type detection Example: scipy_linprog on SOCP problems now returns UNSUPPORTED instead of ERROR, with correct problem_type=SOCP and descriptive reason in memo field. πŸ€– Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude --- scripts/benchmark/runner.py | 20 ++++++++-- scripts/solvers/python/python_interface.py | 10 +++-- scripts/solvers/solver_interface.py | 46 ++++++++++++++++++---- 3 files changed, 62 insertions(+), 14 deletions(-) diff --git a/scripts/benchmark/runner.py b/scripts/benchmark/runner.py index 0dd532e..edeb058 100644 --- a/scripts/benchmark/runner.py +++ b/scripts/benchmark/runner.py @@ -270,7 +270,8 @@ def store_result(self, solver_name: str, problem_name: str, logger.error(f"Failed to store result for {solver_name} on {problem_name}: {e}") def store_error_result(self, solver_name: str, problem_name: str, - error_message: str, problem_config: Dict[str, Any]) -> None: + error_message: str, problem_config: Dict[str, Any], + problem_data=None) -> None: """ Store error result in database. @@ -279,6 +280,7 @@ def store_error_result(self, solver_name: str, problem_name: str, problem_name: Name of the problem error_message: Error description problem_config: Problem configuration for metadata + problem_data: Optional problem data for type detection """ # Skip database operations in dry-run mode if self.dry_run: @@ -292,7 +294,13 @@ def store_error_result(self, solver_name: str, problem_name: str, solver_version="unknown" ) - self.store_result(solver_name, problem_name, error_result, problem_config) + # Add problem class to additional_info if available + if problem_data and hasattr(problem_data, 'problem_class'): + if not error_result.additional_info: + error_result.additional_info = {} + error_result.additional_info['problem_class'] = problem_data.problem_class + + self.store_result(solver_name, problem_name, error_result, problem_config, problem_data) def run_single_benchmark(self, problem_name: str, solver_name: str) -> None: """ @@ -326,10 +334,14 @@ def run_single_benchmark(self, problem_name: str, solver_name: str) -> None: self.store_result(solver_name, problem_name, result, problem_config) # Enhanced logging with computation time and optimal value - if result.primal_objective_value is not None: + if result.status == "UNSUPPORTED": + logger.info(f"Completed {solver_name} on {problem_name}: {result.status} (problem type not supported)") + elif result.solve_time is not None and result.primal_objective_value is not None: logger.info(f"Completed {solver_name} on {problem_name}: {result.status} in {result.solve_time:.3f}s, objective: {result.primal_objective_value:.6e}") - else: + elif result.solve_time is not None: logger.info(f"Completed {solver_name} on {problem_name}: {result.status} in {result.solve_time:.3f}s") + else: + logger.info(f"Completed {solver_name} on {problem_name}: {result.status}") except Exception as e: error_msg = f"Benchmark execution failed: {str(e)}" diff --git a/scripts/solvers/python/python_interface.py b/scripts/solvers/python/python_interface.py index c999411..52b4979 100644 --- a/scripts/solvers/python/python_interface.py +++ b/scripts/solvers/python/python_interface.py @@ -201,12 +201,16 @@ def solve(self, problem_name: str, solver_name: str, # 3. Validate compatibility if not solver.validate_problem_compatibility(problem_data): problem_type = problem_data.problem_class - return SolverResult.create_error_result( - f"Solver {solver_name} cannot handle {problem_type} problems", - solve_time=0.0, + result = SolverResult.create_unsupported_result( + problem_type=problem_type, solver_name=solver_name, solver_version=solver.get_version() ) + # Add problem class information to additional_info for database storage + if not result.additional_info: + result.additional_info = {} + result.additional_info['problem_class'] = problem_data.problem_class + return result # 4. Execute solver result = solver.solve(problem_data, timeout=timeout) diff --git a/scripts/solvers/solver_interface.py b/scripts/solvers/solver_interface.py index 2a630b3..9cf5259 100644 --- a/scripts/solvers/solver_interface.py +++ b/scripts/solvers/solver_interface.py @@ -42,8 +42,8 @@ class SolverResult: All 8 fields are required for database insertion and result analysis. """ - # Required timing information - solve_time: float # Time taken to solve the problem (seconds) + # Required timing information (may be None if not applicable) + solve_time: Optional[float] # Time taken to solve the problem (seconds) # Required solution status status: str # Solution status (e.g., "OPTIMAL", "INFEASIBLE", "UNBOUNDED", "ERROR") @@ -76,11 +76,12 @@ def validate(self) -> None: Raises: ValueError: If any required field is missing or has invalid type """ - # Validate solve_time - if not isinstance(self.solve_time, (int, float)): - raise ValueError(f"solve_time must be numeric, got {type(self.solve_time)}") - if self.solve_time < 0: - raise ValueError(f"solve_time must be non-negative, got {self.solve_time}") + # Validate solve_time (can be None) + if self.solve_time is not None: + if not isinstance(self.solve_time, (int, float)): + raise ValueError(f"solve_time must be numeric or None, got {type(self.solve_time)}") + if self.solve_time < 0: + raise ValueError(f"solve_time must be non-negative, got {self.solve_time}") # Validate status if not isinstance(self.status, str): @@ -182,6 +183,37 @@ def create_timeout_result(cls, timeout_duration: float, solver_name: str = "unkn solver_version=solver_version, additional_info={"timeout_duration": timeout_duration} ) + + @classmethod + def create_unsupported_result(cls, problem_type: str, solver_name: str = "unknown", + solver_version: str = "unknown") -> 'SolverResult': + """ + Create a standardized result for unsupported problem types. + + Args: + problem_type: Type of problem that is not supported (LP, QP, SOCP, SDP) + solver_name: Name of the solver + solver_version: Version of the solver + + Returns: + SolverResult indicating unsupported problem type + """ + return cls( + solve_time=None, + status="UNSUPPORTED", + primal_objective_value=None, + dual_objective_value=None, + duality_gap=None, + primal_infeasibility=None, + dual_infeasibility=None, + iterations=None, + solver_name=solver_name, + solver_version=solver_version, + additional_info={ + "reason": f"Solver does not support {problem_type} problems", + "problem_type": problem_type + } + ) class SolverInterface(ABC): From 51dc80c07c87fb670be71891f4534e11a4365311 Mon Sep 17 00:00:00 2001 From: napinoco Date: Sun, 13 Jul 2025 16:29:00 +0900 Subject: [PATCH 06/13] [Fix] Fix publish feature for UNSUPPORTED status and styling MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Fix results matrix to show UNSUPPORTED status correctly - Updated get_results_matrix() to use only latest results per (problem, solver) pair - Fixed nb_L1 row showing ERROR instead of UNSUPPORTED for cvxpy_osqp and scipy_linprog - Add CSS styling for OPTIMAL (INACCURATE) status - Added .status-optimal-inaccurate class with yellow background (#fff3cd) - Applied to results matrix, raw data, and overview pages - Improve data consistency by filtering duplicate results with different timestamps - Update status distribution and solver comparison to properly handle all status types Both UNSUPPORTED and OPTIMAL (INACCURATE) statuses now display correctly with appropriate styling across all report pages. πŸ€– Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude --- .gitignore | 2 + database/results.db | Bin 770048 -> 770048 bytes docs/pages/data/benchmark_results.csv | 32 +- docs/pages/data/benchmark_results.json | 573 +++++++++++++++++-------- docs/pages/data/summary.json | 155 ++++--- docs/pages/index.html | 191 +++++++-- docs/pages/raw_data.html | 230 ++++++---- docs/pages/results_matrix.html | 42 +- scripts/reporting/html_generator.py | 115 +++++ scripts/reporting/result_processor.py | 56 ++- workspace.py | 33 +- 11 files changed, 1057 insertions(+), 372 deletions(-) diff --git a/.gitignore b/.gitignore index 2c3a288..057b5a7 100644 --- a/.gitignore +++ b/.gitignore @@ -204,3 +204,5 @@ database/*_backup .DS_Store .AppleDouble .LSOverride + +workspace.py diff --git a/database/results.db b/database/results.db index 563bf976e7bc16d6b06ec52ff93056b863767179..f09c73bc2bea6c3a44756beb511e81c0e14d24f1 100644 GIT binary patch delta 37042 zcmdsg31Aad{{OtmWG0zh<>*BPLR~EgEi;qJRcHx^iin7!;;~ZFHic?wOIwg~g;c>6 zFLabq@c<82{jC?pt$TsHe_d}yMQ~LF71vw919d_Cf8U!VowNbcRD-*POx`u`_`d7= zzTeOHW!bWk%a)B?HNcK?D2nY*vR|yU@$km=ZAN%Dw;_057Fhz% z8zKwfSr}=C=R=W3c-|bTgXgW0S@66zQU%Y8A~WGx7P%0fCq&MJXItbPc$P%YhG$D; zO8cHc*(i&#QwH$=#<9XqIjr`@Q1Bib-{&&Z@goNgyVymmbI)fb^==lGnIYMr2xlsq zrQW_3HC3`J7?Rx8 zHGY@NEmr&8VpTBct*#Pkg4ID^r6fr1V!>N1N>0&L>hhJ!e(GOP68!a>XBy(pKSAyD z1{$IMbtjs@Kwy?4em8%EaLCaYS!nRn)*(3UFxyYoAyD`JR&-oP_}}Ptqz--uaWX^| zT)v3Cc*PPI9rRSXWtZ3Ia=9dLr7TtlB~Mk*BUXl>9uOml)pC_6hSW%aT`)A&)Ez^{ zviV4v@u2Cqey89ob-PPtAN4ZsxHWVKZN06sopG-5LgRR2 zKJyggH&Bk#4Cl}t_J{3u+YD>2>2}lQrs>wnmI;n)_~-ev%ug9UH;gfyq)1aw40B=b zJZ?N|WuGt#MwaRCaN3vKKCqp@uQF_~+-$A1TdOp?1vghW1|f)3&bqK4jaco=h>Hu0h&f!whW^Y6edsyCmbzeN1!U3Uf;{MKrl>P8 zR6TD_ErgOdc6aIBc-Djf13<SyQr)Q(DaY;E}C7}tm z4=3oqw|jl};CK3e**)S59DzP3&(Fy7Q~b>NBw6+!lkz_z2|i4gU^glMgYL!bBJb=> z_RhQbo!lMCqTZ))T+VyR;@-khC;h8?QU9irCEt#dd$$c@ijI#DBr;}NLd`7|&9!qH z>q68UWB|{rzqG#LvigeVhPwHoCg|F&_}!w{yBGCZ_n=pC+TmNeN4%0e@0W4<_D$1I zF&WmF-c`7nV@8;Fo8K~TG5^#2SMvt*8uJb2E6t7O8uM?>C!0r^e`6kC&N8#6gQl-c zAHxf$%)I=x>nssVi)EJOLdzMJfW>7w(UQ;q#DB%Vz;EIoc zD*inFG=4Nck}u@5d6W5o`AhSC=7r`n%_T7R*K%RAA)B6d=J=_|5_j=&mpSe-#a&$7 z#l~I6xQnqG3g~mEoId5u=~Iv)9!$qwNS}Z*hoi+)KgCi|hvKdub!n-C@!$h-*MH)! zAM{DTj|YDjckPe6zKy%SF&n~K>#48ffnVu!+NbyI)qD2nJzwfQU+6ub>ph?8J)aU! zSR)nni9YIMz2_sn=ihqIhkDO$y=Rx+vs3SRSMS+jHH726MZK?&dQb0pOYixY-t&Rp zvt94mruV#|_iWXB#=Wlhy{7lPs`qTsdtT9dUT$A=X^6379t1f)fG1V#_yERsp+j*x z|AC}&yMZ^GtgM4^(-+YfDSe!$2rON zeuM$?1Iie!Pl%U~xoVzWK$)m#DC!x$iQQzngMck)?oLYw^DsJyFyDO&q#D_PW{tLe7akYm1P|!YmDk%m#5MvSNgmzxw58O@OnZu{wjZsTW|-(sv4JASrzgL z-WpL|*Pm-GN-=TgSSMXAmrCAJw?ys9aG}0Z=;?hKE)-horS@jHP`T9QqCU%bp@K|( zn&Cn{rJ_uIk>Nt!n7p57s8F&_kSXKxQeWbuD_$q;1}h6Y)=2hTW)Z!PdIql(?ZRW* z5F>c0`@ppwwZFF!9!&dNND&sY5VRFLjOc-mTkb{s;r#6{b-AhMumEoCszCXHU1i@ zI#lU$`-GsjDW;mbbDms5bw z5K1LD2FFG~%GnX&MaroUU$|R9mRRAb(zdrf;VG5f)K?pJp{&zk3>H%NQjY5#gFrM= zZhOUcvGr4Hqh-G(Y%%dS@B_^Em`9i%HzPu+ z%Z4iYTY3cys`J@a)3EXRxys-7j=1q_Qvf6ksgfuzpo;YhCX5n3Flw|=Bg;Z{wNMlC z)l|D+Kl4{gE>W)X$R0@ydF7D5s#5T{;~9X23uZO6QJ{Xn8H`IS0~zE?5;7&U=M-p9 zD6QOF{!+n%n0nI*B2Xvz!!Whgjy$U^h->fneaN5=HgKFG?`u{5vbRaan?6+hGWe91 zr8*|`&4)4!P269pod5)B&7>RFx$$ z92$(XFsbJo;qCioQ1=%TO2r^Oh9RJ*BnM>!y<=%YT) ze4!HcQHBeZN(CRaJHv%)>wks|)h0=13l%)D6HxDEyii!z-p^2>VJz~2`2t&(oB9CH z-yw7#%J95tCifk8JvWry#7+bKT{GRs*oS$*v630(IL`jKeTwa4+vTjvvM%NENF z{&RjINbOgfJ*MXj74$y34XvQQr!Il)68`3z29MA018QQdaVCH5`Odo#(`XvvS4TR~ z;3=Rb7E0upR=78OH$Dw@Sl&Prk-5=yd>JVgYp0K!3}uK=OstSJIbk0NBoU=rY)DFT zC&oEYKdF0c=y_%irhsTfgU?E&K$?{!oBYA($}c+5P^=Iis}6ObH_bJ8)rL+$>fRk} zp_;o5m39n=^&GKvSb^RTE6|J7W!q3dkw?x{H=9utex%7w!2!nYq5jjG9K=$|O?}s! z90bgfSmn#yMbcqY>MeiXKNx; z-VD4iqT?cwRQWsYAf9UKxJbk_TH^;@n3>xeTd`8j=I1E%^$Zr;$+l*)P{QcCWNJ&s z3&q@ZrV7o{A1vaEk3TjdBovs&fkG^ZcZ*zpJX=jyGF8?grVs=LVY*Bd?XgOxDx&G7 zz-???;3jwxMmc3lQ!!TCt0nCO4+Zy)wfq<6tMfIz5FFd_DFX_&Tz)=aq`WR#dGFBB z_PZZ^5Ha>b>UijmLS!Q{Uv<^Jh(kQ+e71pzcOQdwLPIo(?%=J6#p2R;VXb-#9Y@2P zEWHonsln_+%q#Rh>&g6V)78{odm*HFlScU+H?*KFNS(O?ahyc0gp`iXAV!(F+o(>3 zDxm&aC|)Je+K#KAK(`v{XfZeLVmsI;(E-O8`$N=m^gWb)g6$pdPWm2Oz4egwI_n_I z!8NV^sFkdsv7@OfXLm#M86|J8BE|SKkB|!su zoFF0YD8RT=dJBvL9S( zI9`V|nGg)lLv)V|m*f)61WM}ca?a6NiB!jUOjZ${ESiSx*j6HXAnjcxI)_AhlmMv> zJ}Z);J14E73!)=ntQ&q;*(?VNHilp>(i+Oh@#*y>8IYs46cc4 zr|4@<7honA0|)pfsvXF)Mq_Xv-i@9|94Hh-7xkx(>z@Edx1)MDy2BE~1Mp@nM56MT zzL>eS8)BPXs;d#%=8IBH-QgU|4#2Qnj^F`6X`%^nGhC>)Ib^obs7@!tg_6x7(|xK* z_A}ekr~)ptg%V&qqBB(JK)lz2b`1JdMhFX$7?`+GJb)EArh^=?ek`3TIHxz_Uu^|) zQ9EL7PjwmVeW9@IWvI}CI75#0Xf`0Yfv%wVTe+Qd1(5qJI)CNjNOLw-G%LSP_KGDJ zySM@l|Dmf})pc7@zof<(Xqfuq0Pc#RscLGm7$6T-rkY&8okvj=f!yCOTn3dtbLu%@ z1~Pf`1K<9A@XH<<4uyWdID&V=Ut#sUU|nrXYemO~T&_Q|02L(|FCd}v ztFPv94~bwlP+1q6Qvrqo&CQTiA%+|m@=KO`B2C#8AOaGp8F8rl%9ui>Vjl-60d?yD zZgCH(7f6$OSS=nH@y$zbs#}iZ3c{043}8Z)y)pMqy6WviWo1vVFL>_7PqG(5vkNuI zxP~0BXsMlpX(W`mJ%UFNCBcVDQ+T+BudIs9&8G0-p&(%xc@qv21CDZ%$cpn6QOI@A z=Gxe7E+2(c1okBkGmyFou0YPv(QAP`x?xt{_R{rum|f$~8(2Pm&W-mh1L}SF?T(>; z?6|8uum8el29A3qpHMH_&<3?-G&h*R4OMR$%?Yd@Kt`gZZW+yuKyLM{TrPKDk`d^0 zdwoD@uw_!ixm@kf@;`Vz1=H!zkzsoPWL&c(?qC9TU7+DZKvf%Q;H`GA#;di3BM~4Gg5j58})RRHp zt^3rM%DDRI^uQz(@bp;1U?Ugx*>dj0NdVdhf@0LBhS|i(FHPkaTLDw#yWG9w(xOL_ z3>T`2v@=|&rfV#rg53)M6x$D+^!87g#qluOfqBs@Kw%0;6nZa+7N z-OXNPe9d^8Eo|M7Mp;){3oU=POyj>mPcifOE0~>to_oN2vgv(O7?=wqGNvrNN_7#? zFWPZFQoZgibkf)~v(N^v-9%4i%J6ipZY3zrhb796FNPN`Ws z{SrS5Rhots5czMS8DZ2HQ-mi`+=Y``JjePz3J25Fe`%L-q$2FVx9F^-JC?nPPBw%u zH5C;U91oK@#)T52$!Ke$Upf=YG-Z97#S!VC1lf5=eQ+z{lOCZuCu_L!T!e71G7U1QbHNo1tP<5$us&xB1j2;4;OPv=L)mIjcaKaN1I5CG5cT`k{!p+U0Ivh|rha{-1 zZI~0Po{B?i>#OJCtV#N+hQ^izLS`Eeq_5=AlLF z(t7kSl5E1Hfaa>%q3Q~^fd4l%)y~3&YGI;qm_Qtnm{fw0?x9+y%~e4dQQfj_t_pV# z$9Z&3AIrNlyrrROUUTCdSFx|yO(+l#5Z(HaS&#ykOQOLelHt(;FE?XXxQIAOl3-KS zY?2jrkiNnW(x2Et(o{9nw=^}>ks%ayN5>}#kB&>?j*eF1CWS`RlMxejd&9VHQIEa= zGMb~|BB@l&Ly8W~1qsxtqR#Cm?pckEKt7x&c1dz^JQUodqS0_tQE&rpPc9}k1&`zw z;xAx#>;>$Oy@1`Z^lqUuJ$A>^V|Oe)c9Zn8V>1ssNJ{JolXTb>Pef>iL$s*{4liyH z4j~l64niL6&}c(kq^ZOu;d?Y&ZFN$9gqDHZ9@Y9i}IKzg?a{TlDCx=iBn8LBd|s0WFH1|fx7 z6lsIU1P1OKq8OwdNOWeXIs`~X1RNOoi>U&C_^z#Gs&y4zfmW+^tLRhFN_FEZdL(L9 zcZ2VY2*gi<55CFkUijej_rd2z_}l~^5kA+!XAFEw;d2Rm7Qts(^c#mSg^vQC)#`#9 zXfL|8pQPw4SL41-xdsm^1<- zsFFlz>fIy}nl8V4L{#-1P75MmMG*o|`WYan7V2@%+SG)aVJ0hS~JCOX*UCyCI^KszJC z+I@CVI*ed2@9d0;M`)PO?h#RFS@H;wl_ib9=4Secm=TC(*rSiZ=W8rTw$BHG5z&D$8jR-^0n$WLn z49N>azLG>}5X++)vnJfj)}%*H%TL~;2g7SlTGU>$49W=Jnp#N zak=9X$5{@aW3Yp@e_`Kh|BL-@`&ITv`*eG`{RF$!_OudeNy36{E^m;ja&9nSu`8SvdZ?vqjv{`B`Q!QT0Ad3;ql3wTk%&+6G z;^*=g@MHO*yu-ZT{4QK$_AnSLErkorrkFkEL2z~1m!@r|O{VqGV%DT=3FpwBdiW|) zUJmCfcZ3CHc{pFG4`(aW!g=bLXTi8P0G<=VdCIp-1!c$5JcHY{NEx&=PZ_vWQPwTV zR?yOZ%A-p-<&!0O$}39-D0eQAl;unEl{$z!b4kAXNE=)@Shgfb83KWO7E8*u#rewL z7U!s&-v-;`-!JZmVa-a`;v1B&7Ud~}zvPsii?WozEy`E!h0nl;&B{&SZ-Tfni_FT% zMdK9PqFm*=g)F&~LOHb11|GBW{=yvPp9_yu)PPjkxJ*1pOz2w2w2D3`Y8DQAJ(*P5#gY|U5C+z-{`Tl=ZN_<@b| zf}sO2@<&7QVIT@aPp52`aW%{?dOA?^R61|DF>-c)YRD)Zh5!Kl=mK>ejfQavLbO!n znVPgPkZSVS_7e?^pf>V}HUuHZ+ukC*Xu!L6-{9e$odjJYTN6t#p znrX+RHL-;tP+C4#Yt4idS_5DLpWx<~Tw{w&O(NyJ?&Kazc~k9(sFXL=_ zxuY+Z@-8xqC0@xdOD?GcFc@{n3<7kFPcC(JY{}!6p?OH$}SkIyd&qRR_?xPBMCYjqZ9G*e$t{2}8H)E7!0 z=gz=NQ{=(~+z!l4EWj6Il5Gs-IGO>#&mz*lC{0N>>|}DF@6e4;Ld~PVoo)afbrzs+ zIx#tBzgLn)uUi(?LAjiy!Ey$uqr9A1b(V2~OyFfe!vHmxy5MCB_$8oHxgg18`^8l% zfwbu_`vW;oZwRQ9M{`RYZrSgNWA9vYqD}?QN*JTwl0Y}_$4kMZ#`TY~jaeF#{s?88#Q(`Ol70l3^t&)# zybWnB?5bFMuVfC+x#*UOhdtEQ&#SA8LElS~gZ`z+Qhi(9#DoI3^@ev) zf!gpcvZ{id_Q&Y`DG89RRbxo3|(=j36VXWsvT6XG@> zGG`zE_MnzAho~nvoDSblPuczDmCK(m`+xI}gXmGIktHcrh_MN3rFfpnMy;@TT#Cw; z=S8M~c?3Yr5(oT6M}(W{2Y!!!J|ho}O7jE>Dg_NbbJmgIGtpU5lvw*N0GeQZE8u9N zM@*O*9oaJX+3?kS&5YehKSbFk@;7q3j2F@m0g2Yo`OAkzmcWJ&Yl41##e-??o9rOv z(6_8=yUx@vqhm?(1d8|W6FSEdSyBJ7>v#D3YU@*L%Kp*HJPFa|QwKkO4SV30vJH2? z^M~5aLlY*|DR>Qu*_G&LRnp-D;*-VNq)Hqr`S>!L0ojR{-{?n3&n8&9OCv^s6MWTa z2}>jX&5fh2j3tpp&@hm4={p#KiBNDTFQ?J3fixFI+=M_$2PdCr6({Aj&* zK@$w&`ig5@k8FDK@&{iPo}|iu_#*$JZ%+Pm`R9K+W3znI_-s)(h|f$kjuJ~eZkO8& zwo$GlQ$&umCbJ5&=noxfY=`OgJiaP6FycHDg-;s=)6+5vbNgNLk;lRMMsDWn0Bbbb z##1(|&iC8$m;2i+gc7;>v4ID_&OXinSVDDbDcH!Rd{Il6>xRdLQ++eeJ$C9V0>T+5 z>DIa@Phuxdzu0hyd^ddYd3oU{U&1h=Mt|>o{mo~~pD$~EZFs?V=G;EIaZ=$X##6Dy z4ja+A-9E2dAohiaHKJR;ixUXOmR(1gKVsJb&2DDPR*Z|BlVsDpVr1gB6ZOR~+WI9i z1Dj?HlYDf{>ABhu)Hn%+nw(uXPd(05OG03#2|r3iAa(Qc?>fbPsHrmqe$S{Vj3IX< z9{=D`XB=LgJAII%p|PdYkAK}s=YyYG>huwo7kgvEDFP(d2VgmA5e6Y$lO4&KmgH=B z-MKwE8xp8+s?1C8v>X^p9w=BkdgdP^IT5=3p9Pi>Q(}kn_^x zs}eSh!x|konLS>wTauy@6Z!D{5PD-T3%uMOZ%(x}EE5YR@X6T1yW|pHi z156Dtrc6A*4JL+_>TN!wL;YQV?V&&<-SZko>1n*lG)&iJx^ahR03N9(OasE~wq*$;{%1Pw z%YHb>9r?IFGBQ6IMa;)e0D{rAfsyT3OzkB_CU+Nr2wp$AG34+YffSjSj2y>M9)ztS z0wtfF4b;#ri9g2xk-aWS1pL>L2Z$JHNk$RdF&iMnR3lJQ`8QuLDdO%nKqQ|}kd88A z)@PXsgK0OSH!14@^B&I4tVD0ZQu$Amzr3W4Cz4=IW<(v2dyS$ZAtDLxlPC#3mZBp` zJa@{pbEi%{^Xzj@o!@xBH52>V=6Q{c4NWaw^k#U}OpZ^amexjm z1(SBS)R6~NQJY!gwFrZvBN4vD0v$VPn&3TVB=NMFlF3{XB#oJqxe_ClfcNEp{DH_^ z)47Pt=av0%6_+3#aVV@XwQ)(5k!{7AKnx4Hz@^^#4k%!cfo782Y8;qYequdL;V4`$ zEzM+@Ot4zes|B%1dj%MEHoy+Uk%q{))2-Y}{L%Ewh^% z=FOVz9Dm;VQ!gkawAJK}i7Q!Gh1E&j?tWW=+SnvV;WqJDY3|7HQw6+lufC>});&53 z$4C`UN?srsbl@*WU>$u~?Q*^KLTs0yMpOJM?k8hCy@MJJ;>DA4mJf{7b%Aw(fKgpH zfOBOS%>~9e*q$d)EZ_bpnd_&erC;&5M1ZWh2$JH`Xv@^q1Gw_cHwRlG1xDObybCv{ zJO?xxpv`d05~!>hPh3Fgb~?Ve5ALQE z{2r zQkeqHlL9%<{qSU$ZRl_}xEkRiR-a!5fa1wNk0J#`-{jub`v<_1WsKDPTCmJO7%Y_! z6mQ${JA!5Qr(n|ozYPn}U;r$VUuTo(QKN{@z26Dr@VYI}KHH6hZ%o^8}<%N5vy<#Ww81HjG z&o$M!E;3Qw;pMJxq@l}l2YUtTVujdq#vwfoB(9niDE3HC63p4ylrU$&3Sd@qV5B~} z*6A*oK1K{a{)y;7@tD6nftS64^U`AQCD8?p4z@_ij?78YMP2L+GuB1;mV&^DyIy^h TbWur4T_k~ELbEd5!tee6>3G}g delta 6219 zcmd5=dvq1mxu3mf&di*dIS)cYNPy%d$V=d4-e=}$4Q)X$Eoww4Xay1?uNaa;pdjU5 z6ZICmc<6}-*(q1LKq6jgwZMT4YAgqq_WS`9b?Qehkx4-@E@B1cAO=V3@W${8kb(W&2UqibYT0`?>6H{joRkjBBGJiiH z)VhQeT%0Tk{nnR|L;I7T7Udub2V{4sD6k_Sr-qVp=-UX5Am$WQ$-qdE{9Hm)6RSS6 z8972de^|Z}2vVIGMv%YMojTTEn$>BbF;4S3)PqRnX2gwAsVxEsRB|s;(r)o;;9W6a z*yz2&T@1`l$Q6Z&ii(Ld10nxUe82Ne5aQx9(o*sMz#ieFfatIHb^3Ur)%$btsI*0V zEbxZt@%|}b_;>mK&-We45-$dREI7h#LZ0`KcWa`rB6k{(xp%n}+>6ptu{Us03Bd~9Bp+3;e>k>r$=U~FRm-blvc1|tV>({P{ zrND=VfXk}u*QUT4&mB~5R%>+{TQ!=QGN81!s-bG>vfAZUjj-HG&vq(|)>f|q4z;W5 zVh!~xQd(93J}RpPxK-E3>P>5TJ~N>pgQm%)@nvj2wJw9Z?9%9io{y=i8PuhhMymnM zOQDUh?dlA=w1Am1pgF@ZEnfw?3%^`b6t9b}TRl9#QGiex#s&*gOM73w8WGO28!i*Ig z-Gd)AqoOZhzaB7p){Kh&gCWebW>oa~pgNXSou;Vh^SFhSHVj2YzjqjlHDEABpPR#! zjLa$mGcH}N+ylDPN0j9UV8swQmmENM^O#zdnd&kF=&lh=@qo5!7r7IJW+u-6ATN1* z!R|u&Lau84QZ0Gm$CrIl- zydyX8Ki7X?uHhm3PvKyr71fQXuB~R*P@`2@HKxnvbX5(>DO|H_^JxuTj~fsD4ZMck zJDeVU)$sCi&Pw(Vl>ChaoWpJS1Eh|oH9UYDAq7|(QAMX-y?#Kmql!koLb|%~WK!9U z8FH!{=cS*1)r|{Cs1CE_*={_N{F~c-up5sdce+n{Hy%Zb7h{eTcjFv#wF|J{ci~)e z%6Q^|Iut zPCSv^uOpE>(22i8qMf)PG4^GFOzi|^Ug(iaX8;=aiI?=90qFP{U{p2*aYQ+T<0Qu& zY${{P)ehj?*MY~9LmjY4O9#wsbibB&z`}9|Z1v8}(>Q`%kw3JyBQ!8^YBz4_TM5Im9IAztxLe2T5*vqZhGSRMw~$lY3wPU&OnK8R#?v zo$YjX+L?^Z0Y~ETXDIP?hqtW)@8O-7bkvP#oq7o-_oC#v-dE|Mz$~X|>M%te=I_JX z@HQtXqlFBn{{kxeX$KAf4ElhMyp9@0U7npW#3tFkuL5(>OH_z@jS4&!DDuDTzs=X} zTO|Eaiiw}&F0oN$kxsoP>=ktH+uk4Y{rtc4qqrA1g*`#fV!!V>>#1h?nXODI{R(~z zk49ufC0CbN{CPv_Y~W!d7#WoSg>fj+{N*j==68`n0uJ1)Z2=TX?E2g!Gj@p{Oug;Y z`OwqYh3iP&F}RpP2AQOnMzfrZ?gi?f^HN{fJ%Rx#D@oGo5)Wurn~# z|B8Qsuiv-PCrYi-x5TqzgDAje74HeGp>|YB^;27c{Oy=LD!`&Eyy#YPNk9#5cNys^ zV|fx3P@gk-655N{4gC1BaZ^aocvMWbFJwn0%ERAcRsl%5#-kGF1qS6IwuUd6UXV`| z7L6m9L?n=>M06VzV@{`tY(#pd!miIuLASE?eDTzR38d*}G=|iN5li9=5#y|$h7vTh z0+f(Y8LV^WCiH@Hege7!wZ-XMX!ID+eM4#s(zAGyQ$+uFp{*#Usp*QMP79STZ&_=(U1`_NvMFnnyZ+aZD}g>VZ3U66_k55WJD4D`rn&bILyrGmJqurF`KMej}&qu z?%gs*`8(?mr{&3iSj%v`P*O(l{G)`1Q}uMpAY&~fq8hLtS-;klVVRWJ^c=FffId7@ zQ6su;+FHheR;DzgK( zvCjzi@N21+!X%K|jggd!PNe(*;*gk*Ld54_ulriza~e%_7SSs8OQ-t?I**XwIn;(e zL>TqJ>~4gFXhluPRq*6mpI)#}E+JsG(3v zXX`QaGl&SqPUG*D4usK6cZCr(~1M}%DY5XGFAKR^}8nPuZyv@`iV zv>7{fzd_|F5iDE3Gme;C4sWOMcK$xLk$FCoisr+@Kf=zlr@&~=gBQ?lF&}TEJgJZ^ z&2e*-f1-aPKa-!CC<*OybAZ=*`2s{CQPZQUK^iK#7A#3(4Jc>EfU{PnM&)9hLR4VM5tP z9(|oF9bwFKtFtVsSP^n)F1?A&xYy$$)BXtFaZ~81Bq*{0vB@&k2=P~VMoq8>*=P~V zR3et)My8=`!Ya9E1EYCWLxahN{WthnQNsw^s%a?X(N8^DJ_D$%4f5f1fKBQ+81k6p zFmcA9X_NXXbWf$a5gI*_>daDP)3z*4vsHOGW7;lPH_C$9OiMRZ%}5Cjp>Kta6n#O} zOgp04k%;M3HZhk`(3mINQC)@Kux*;AOq%C{UX!v>r&5IZ8}2vAdb zpz2DMJvB_6Zvp%fW{6j~bu8!1c>uqU82?0SuQ=6RZPS23tV>B~sQY{OpkKInKK zIm`@Br)h=S2i-TNoqI6rurd5)V1(;lslD3nQAAXWj5sLH;#FN&WlK@7^%KkW2vvz1 zT3A<1%TAw*HBUB&pq7<77s(1`V1qgd>;6{qtbx_SM2h1jXbP!~<2%qKawLw+P!72O zy*tOl`1hdQ1g#a?NN7-1QBOnL0ZoDSB(!PJqR?ufO^4P5Z4rr^0C?K<8}4%1jkDXqpaxX{Fgzmd z5Al0|p9mF~C9CiwTO?57$(t}F%Br@etB^#lOA=Zp${LU$bTgvL=_ngWB`YlwRU=_l zjcBClIeL6LR}F%eGLRcsHvMk|Amm=P9{zG5WyzBe60yDx;C6s74Z<3j-K@*tL2pX@ zyO4vS$c|XDHEh~2BZFx}Q)I(K~R?fK<

πŸ”¬ Optimization Solver Benchmark

Overview Dashboard - Latest Results

-

Generated: 2025-07-01 23:58:26

+

Generated: 2025-07-13 16:25:36