Fix CVXPY duality calculation issues and improve solver result consistency #6
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Remove unused functions across data loaders and solver interface: • dat_loader: analyze_block_structure, load_dat_problem • mat_loader: load_mat_problem • problem_loader: load_python_problem • SolverInterface: preprocess_problem - Simplify memo field to contain only solver_solve_time - Remove excessive debug information from additional_info - Update import statements to remove unused types - Fix test script references to deleted functions 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Add SeDuMi v1.3.7+ as submodule at scripts/solvers/matlab_octave/sedumi - Add SDPT3 v4.0 as submodule at scripts/solvers/matlab_octave/sdpt3 - Update .gitmodules with proper URLs and paths - Establishes foundation for MATLAB/Octave solver integration - Follows existing architecture pattern (python/ and matlab_octave/) Both packages provide semidefinite programming capabilities: - SeDuMi: Convex optimization with symmetric cones - SDPT3: Semidefinite-quadratic-linear programming solver 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
## Phase 6: MATLAB/Octave Solver Integration Planning Complete ### New Files Added: - docs/development/matlab_integration_design.md (1043 lines) * Comprehensive technical design for SeDuMi and SDPT3 integration * Loose-coupling architecture (Python → MATLAB → Python) * JSON bridge design with detailed component specifications * Performance, security, and deployment considerations ### Updated Files: - docs/development/history.md * Added Phase 5: ProblemData Architecture Analysis (completed) * Documented architecture readiness for MATLAB integration * Updated current status to Phase 6 ready - docs/development/tasks.md (complete rewrite, 1035 lines) * Phase 6: MATLAB/Octave Solver Integration (25 tasks, 5 sprints) * 10-week implementation schedule with detailed task breakdown * Each task includes: objectives, steps, success criteria, test criteria * Sprint-based execution: Environment → Solvers → Integration → Python → Testing ### Key Design Features: - **Architecture**: Loose-coupling with command-line execution and JSON exchange - **Solver Expansion**: 9 → 11 solvers (adding SeDuMi + SDPT3) - **Fair Benchmarking**: Minimal configuration using solver defaults - **Production Safety**: No impact on existing 139+ working problems - **Comprehensive Testing**: Unit, integration, and production testing strategy ### Implementation Ready: - Detailed technical specifications for all components - Clear task dependencies and execution order - Risk mitigation and error handling strategies - Complete documentation and validation plans 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
## Sprint 1: MATLAB Environment and Data Loading ✅ ### Task 1: MATLAB Environment Setup and Validation - ✅ Verified MATLAB R2024a and solver availability - ✅ Created automated setup script for SeDuMi and SDPT3 - ✅ Validated MEX compilation and solver functionality ### Task 2: MATLAB SeDuMi Data Loader Implementation - ✅ Implemented mat_loader.m for DIMACS .mat/.mat.gz files - ✅ Added compressed file support with automatic decompression - ✅ Complete SeDuMi format validation and cone structure handling ### Task 3: MATLAB SDPLIB Data Loader Implementation - ✅ Implemented dat_loader.m with manual SDPA format parsing - ✅ Avoided SDPT3 svec complexity with direct sparse matrix construction - ✅ Full support for mixed linear/SDP blocks and multiple SDP blocks ### Task 4: YAML Configuration Reader for MATLAB - ✅ Implemented matlab_yaml_reader.m without external dependencies - ✅ Complete problem name resolution to file paths - ✅ Robust error handling and metadata extraction ### Task 5: Data Loader Integration Testing - ✅ Comprehensive integration test suite (15 tests, 100% success) - ✅ Performance benchmarks (avg 0.022sec, production-ready) - ✅ End-to-end workflow validation (YAML → loading → validation) ## Technical Achievements ### Core Components - **MATLAB Data Loaders**: mat_loader.m, dat_loader.m - **Configuration System**: matlab_yaml_reader.m - **Test Infrastructure**: Full integration and performance testing - **Setup Automation**: setup_matlab_solvers.m for environment preparation ### Format Support - **DIMACS**: .mat/.mat.gz files with SeDuMi format - **SDPLIB**: .dat-s files with SDPA sparse format conversion - **Problem Libraries**: 139+ problems across both libraries ### Performance & Reliability - **Loading Speed**: 0.022sec average, handles 777K+ variables - **Memory Efficiency**: Sparse matrix processing - **Error Handling**: Comprehensive validation and recovery - **Integration**: Seamless YAML → data loading pipeline ## Next Phase: Sprint 2 - MATLAB Solver Implementation Foundation complete for SeDuMi and SDPT3 solver integration. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Add matlab_runner.m: Main orchestration for MATLAB solver execution - Add sedumi_runner.m: SeDuMi solver interface with exact version detection (1.3.7) - Add sdpt3_runner.m: SDPT3 solver interface with dynamic git version detection (4.0-20240410) - Add solver_metrics_calculator.m: Python-compatible cone-aware dual infeasibility calculation - Add save_solution_file.m: Optional solution vector storage to .mat files - Add save_json_result.m: JSON output formatting for benchmark results Key features implemented: - Accurate solve time measurement using tic/toc - Cone-specific dual infeasibility calculation matching Python logic - Dynamic version detection from git submodules and API functions - Solution vector storage functionality (problems/solutions/ directory) - Error handling and comprehensive logging - Fair benchmarking with minimal solver configuration Testing results on arch0 (26,095 variables, 174 constraints): - SeDuMi: 19.8s, optimal solution -0.566517, dual_infeasibility: 0 - SDPT3: 11.6s, optimal solution -0.566517, dual_infeasibility: 0 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Implement comprehensive MATLAB integration orchestration and testing infrastructure, completing Tasks 12-15 with production-ready validation and 100% test success rate. ## Sprint 3 Achievements ### Task 12: Temporary File Management System - Implement robust TempFileManager class with UUID-based naming - Process ID + timestamp + UUID ensures concurrent execution safety - Automatic cleanup with configurable age-based orphan management - Context manager support for guaranteed resource cleanup ### Task 13: Command-Line Interface Validation (100% Success Rate) - Enhanced MATLAB CLI execution with proper error parsing - Reliable process timeout and termination handling - Intelligent command construction with proper argument escaping - Startup delay handling with buffered timeouts (45s + 15s buffer) ### Task 14: Error Handling (Existing Implementation Sufficient) - Validation of existing robust error handling in MATLAB functions - JSON error format implementation for graceful failure reporting - Integration with Python error handling through structured results ### Task 15: Integration Orchestration Testing (100% Success Rate) - Complete end-to-end pipeline validation with real problems - Comprehensive test suite covering all integration components - Performance benchmarking confirming production readiness - MATLAB integration ready for Sprint 4 (Python Interface) ## Test Results Summary - Function Availability: All 8 MATLAB functions accessible ✅ - Data Loaders: SDPLIB (.dat-s) & DIMACS (.mat) working ✅ - Solver Runners: SeDuMi & SDPT3 optimal solutions ✅ - Complete Pipeline: arch0 solved in 6.4s with valid JSON ✅ - Error Handling: Graceful error processing verified ✅ ## Technical Infrastructure Added - TempFileManager: Concurrent-safe temporary file management - MATLAB CLI Testing: Comprehensive command-line validation - Integration Tests: Complete pipeline validation suite - Performance Benchmarks: Production readiness validation - Enhanced Error Handling: Structured error reporting ## Production Readiness ✅ MATLAB integration infrastructure complete and validated ✅ 100% test success rate across all integration components ✅ Performance within acceptable production benchmarks ✅ Ready for Sprint 4: Python Interface Integration 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
## Major Changes - **MATLAB Utility Consolidation**: Merged all 6 MATLAB utility functions into single matlab_runner.m file (1,035 lines) - **Documentation Enhancement**: Added comprehensive current_design.md (English/Japanese) explaining MATLAB integration architecture - **Code Cleanup**: Removed unused Python utilities (version_utils.py, config_loader.py) and reorganized test files - **Coding Standards**: Added MATLAB function indentation standards to development conventions - **File Reorganization**: Moved matlab_execution_test.py to proper tests/test_utils/ location ## Consolidation Details ### Integrated into matlab_runner.m: - matlab_yaml_reader.m → read_problem_registry() - matlab_json_formatter.m → format_result_to_json() - save_json_result.m → save_json_safely() - save_solution_file.m → save_solutions_if_needed() - matlab_version_detection.m → detect_versions() - solver_metrics_calculator.m → calculate_solver_metrics() ### Benefits: - Single file deployment for all MATLAB functionality - Eliminated external dependencies and path management issues - Simplified maintenance and version control - Atomic operations without external script coordination ## Documentation - current_design.md: Complete English technical documentation - current_design_ja.md: Japanese version for international collaboration - Updated conventions.md with MATLAB indentation standards (4-space rule) ## Testing and Integration - Enhanced integration tests with comprehensive coverage - Performance benchmarking framework for MATLAB vs Python solvers - Proper test file organization following Python conventions 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Apply proper 4-space indentation to all function bodies in MATLAB files:
- All code within function...end blocks now properly indented
- Control structures (try/catch, if/else, for, while) correctly indented
- Nested function definitions follow indentation rules
- Comment indentation aligned with code structure
This change implements the MATLAB coding standards specified in conventions.md
where all function body content must be indented by 4 spaces.
Example format applied:
```matlab
function test_func(test_arg)
try
display('')
catch ME
end
end
```
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Apply proper 4-space indentation to MATLAB data loader functions:
### Files Updated:
- scripts/data_loaders/matlab_octave/dat_loader.m
- scripts/data_loaders/matlab_octave/mat_loader.m
### Changes Applied:
- All code within function...end blocks now properly indented with 4 spaces
- Control structures (try/catch, if/else, for, while) correctly nested with additional 4-space indentation
- Helper functions maintain consistent indentation hierarchy
- Comment indentation aligned with code structure
### Example Format:
```matlab
function [A, b, c, K] = dat_loader(file_path)
% Function comments
try
% Code inside try block
if condition
% Code inside if block
end
catch ME
% Error handling
end
end
```
This completes the MATLAB coding standards implementation across all solver and data loader files.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
…al clarity - Rename matlab_solver.py to matlab_interface.py to better reflect its role as an interface between Python and MATLAB systems - Update all import statements across codebase to use new module name - Update documentation references and examples - Update test files and mock patches to use new module name - Update logger name in module for consistency - This naming better distinguishes the file's role as a system integration component rather than a direct solver implementation 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
…face module Design Philosophy: - Create symmetrical interface pattern where each solver ecosystem has dedicated interface module - Extract Python solver management logic from benchmark runner to python_interface.py - Maintain architectural consistency between Python and MATLAB solver management - Enable clean separation of concerns and improved maintainability Key Changes: - Add scripts/solvers/python/python_interface.py - unified Python solver management - Extract Python solver creation logic from runner.py to interface module - Update runner.py to use PythonInterface for Python solver operations - Document symmetrical interface pattern in detail_design.md - Enable dynamic backend availability detection - Provide comprehensive solver validation and statistics Benefits: - Architectural Consistency: Both Python and MATLAB have dedicated interface modules - Separation of Concerns: Each interface handles ecosystem-specific complexities - Maintainability: Changes to solver ecosystems isolated within interface modules - Extensibility: Easy to add new solver ecosystems following same pattern - Testability: Each interface can be tested independently Testing Results: - Python interface detects 9/9 available solvers (100% availability) - All CVXPY backends working (CLARABEL, SCS, ECOS, OSQP, CVXOPT, SDPA, SCIP, HIGHS) - SciPy solver working with HiGHS method - Benchmark runner successfully using symmetrical interface pattern - No regressions in existing functionality 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
…dule Complete the symmetrical interface pattern by creating a problem interface module that manages all problem loading responsibilities, eliminating duplication and creating architectural consistency across the system. Key Additions: - Add scripts/data_loaders/python/problem_interface.py - unified problem management - Centralized problem registry loading and validation - Format-specific loader management (MAT, DAT-S) - Problem statistics and monitoring capabilities - Comprehensive problem validation and environment checking Architecture Changes: - Update BenchmarkRunner to use ProblemInterface for all problem operations - Remove duplicate problem loading logic from runner.py - Update MATLAB interface to use centralized problem interface - Eliminate load_problem_registry duplication across modules - Create clean delegation pattern for backward compatibility Benefits Realized: - Complete Symmetrical Design: Python, MATLAB, and Problem interfaces - Eliminated Duplication: Single source of truth for problem management - Centralized Validation: Comprehensive problem environment checking - Clean Separation: Runner becomes pure orchestration layer - Easy Extension: Simple to add new problem formats and libraries Testing Results: - Problem interface manages 142 problems from 2 libraries (DIMACS, SDPLIB) - All problem loading operations working through centralized interface - Benchmark runner successfully using all three interfaces - MATLAB solver updated to use problem interface instead of direct registry - No regressions in existing functionality Final Architecture: ``` scripts/ ├── solvers/ │ ├── python/python_interface.py # ✅ Python solver management │ └── matlab_octave/matlab_interface.py # ✅ MATLAB solver management ├── data_loaders/ │ └── python/problem_interface.py # ✅ Problem loading management └── benchmark/runner.py # ✅ Pure orchestration layer ``` This completes the symmetrical interface architecture where each major subsystem has its own dedicated interface module for clean separation of concerns. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
…e mapping ## Major Improvements ### 1. Explicit Solver Routing - Replace trial-and-error EAFP with explicit solver-to-interface mapping - Eliminate unnecessary ValueError exceptions for cross-interface calls - Direct routing: matlab_sedumi → MATLAB interface, cvxpy_ecos → Python interface ### 2. Complete Lazy Initialization - Solver interfaces only initialize when actually accessed - Remove module-level MATLAB availability checks - Properties-based lazy initialization with proper error handling ### 3. Unified Interface Architecture - Remove solver_registry.yaml (redundant with interface configurations) - Single source of truth: interface *_SOLVER_CONFIGS - Dynamic mapping generation from interface configurations ### 4. Performance Optimizations - Zero unnecessary exceptions during normal operation - O(1) solver lookup instead of trial-and-error - Faster startup with lazy interface initialization ## Technical Changes ### Core Files Modified: - scripts/benchmark/runner.py: Add explicit mapping and lazy properties - scripts/solvers/python/python_interface.py: Implement lazy detection - scripts/solvers/matlab_octave/matlab_interface.py: Implement lazy detection - main.py: Fix import syntax error - docs/development/detail_design.md: Update documentation ### Files Removed: - config/solver_registry.yaml: Redundant with interface configurations ### Architecture Benefits: - Eliminates: matlab_sedumi → Python (ValueError) → MATLAB flow - Achieves: matlab_sedumi → mapping['matlab_sedumi'] = 'matlab' → MATLAB flow - Clear error messages with available solver lists - Extensible for future solver ecosystems (Julia, R, etc.) 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
…sary layers ## Architecture Improvements - Rename matlab_runner.m → matlab_interface.m (actual interface function) - Remove matlab_runner.py (unnecessary MatlabSolver middle layer) - Update matlab_interface.py to call matlab_interface.m directly ## Role Clarification - Loader: problem_name → A,b,c,K (mat_loader.m, dat_loader.m) - Runner: A,b,c,K → result (sedumi_runner.m, sdpt3_runner.m) - Interface: problem_name → SolverResult (matlab_interface.m orchestrates loader + runner) - Wrapper: Python wrapper for MATLAB interface (matlab_interface.py) ## Benefits - Symmetrical architecture with python_interface.py - Reduced coupling: new solvers require only 2 file changes (runner.m + config) - Eliminated 3-layer abstraction (Python wrapper → Python solver → MATLAB script) - Direct subprocess calls to matlab_interface.m - Dynamic feval() preserved for flexible solver addition ## Technical Changes - Direct MATLAB interface calls via subprocess - Maintained dynamic runner function selection - Preserved all existing functionality - Compatible with existing benchmark runner 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Fixed dimension mismatch in SOC projection function for dual infeasibility calculation - Improved SOC constraint dual value extraction with proper error handling - Enhanced debug logging for duality calculation troubleshooting - Verified fix works for both CLARABEL and ECOS backends - Results now show proper small values (~1e-13) instead of None 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
📊 Preview Reports Ready!Your preview has been deployed and is available at: Preview Details
Available Reports
Notes
Preview deployment powered by GitHub Actions |
- Add problem_class to additional_info in both Python and MATLAB interfaces - Python interface: Extract problem_class from loaded problem_data - MATLAB interface: Load problem via ProblemInterface to get problem_class - Database now correctly shows problem_type as 'SOCP' instead of 'UNKNOWN' - Verified fix works for both cvxpy_clarabel and matlab_sedumi solvers 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Update MATLAB interface and runner files to use consistent field names matching Python SolverResult - Simplify benchmark runner by removing unnecessary registry dependencies - Clean up main.py entry point and remove unused functions - Update test imports to reflect new MATLAB architecture - Remove obsolete configuration integration test file - Fix SDPT3 runner function call (sdpt3 instead of sqlp) 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
📊 Preview Reports Ready!Your preview has been deployed and is available at: Preview Details
Available Reports
Notes
Preview deployment powered by GitHub Actions |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Test plan
🤖 Generated with Claude Code