Add ruff parser backend as experimental alternative to tree-sitter

[ram-nadella]

Summary

This PR adds support for using ruff's Python parser as an experimental alternative to tree-sitter, while maintaining full backward compatibility. The tree-sitter parser remains the default.

Changes

Parser Abstraction

• Created a Parser trait to abstract symbol extraction logic
• Implemented TreeSitterParser wrapper for existing tree-sitter functionality
• Implemented RuffParser using ruff's AST for symbol extraction
• Both parsers extract identical symbols (only column positions differ slightly)

CLI and Configuration

• Added --parser CLI flag to select backend (default: tree-sitter)
• Parser backend selection flows through the entire system (LSP server, index, updater)
• Implemented standard FromStr trait for ParserBackend enum

DevTools Improvements

• Split UI into tabs: "Symbol Search" and "Parser Comparison"
• Added parser backend dropdown in Symbol Search tab
• Parser selection is passed to pylight when spawning LSP server
• Improved tab styling with blue active states and better contrast
• Removed rounded corners for cleaner, modern look

Testing and Validation

• All existing tests updated to use SymbolIndex::default()
• Created test program to verify both parsers extract same symbols
• Added parser comparison tool in devtools for side-by-side results
• Verified on multiple Python codebases - both parsers find identical symbols

Performance

Initial testing shows comparable performance between the two parsers. The ruff parser may have advantages for certain use cases, particularly when integrating with other ruff-based tooling.

Future Work

• Add benchmarks comparing parser performance on large codebases
• Consider making ruff the default parser after more testing
• Integrate with ruff's error recovery for better handling of invalid syntax

Test Plan

• All existing tests pass
• Manual testing with devtools shows identical symbol extraction
• Parser selection works correctly in LSP mode
• DevTools UI improvements tested in browser

🤖 Generated with Claude Code

Co-Authored-By: Claude noreply@anthropic.com

[ram-nadella]

File discovery down from 6 seconds to 1.3 seconds!

Python file read, parse and index down from 8.79s to 5.29s

Logs of initial bootstrap before optimizations (before switch to ruff and also before the faster file discovery changes in #14)

2025-06-29T16:06:39.443215Z  INFO pylight::lsp::server: File watcher started for workspace: /path/to/large/python-repo
2025-06-29T16:06:39.443447Z  WARN pylight_devtools: pylight stderr: 2025-06-29T16:06:39.443276Z  INFO pylight::index: Starting workspace indexing for: /path/to/large/python-repo
2025-06-29T16:06:45.438247Z  WARN pylight_devtools: pylight stderr: 2025-06-29T16:06:45.438224Z  INFO pylight::index: Found 92845 Python files to index
2025-06-29T16:06:45.438262Z  WARN pylight_devtools: pylight stderr: 2025-06-29T16:06:45.438232Z  INFO pylight::index: Starting parallel processing with 11 concurrent tasks
2025-06-29T16:06:54.517424Z  WARN pylight_devtools: pylight stderr: 2025-06-29T16:06:54.517390Z  INFO pylight::index: Parallel parsing completed in 8.79s (10561 files/sec)
2025-06-29T16:06:54.517437Z  WARN pylight_devtools: pylight stderr: 2025-06-29T16:06:54.517400Z  INFO pylight::index: Indexed 92845 files with 713082 symbols
2025-06-29T16:06:54.517438Z  WARN pylight_devtools: pylight stderr: 2025-06-29T16:06:54.517409Z  INFO pylight::lsp::server: Initial workspace indexing completed

After these fixes landed:

2025-06-30T03:26:25.446133Z  WARN pylight_devtools: pylight stderr: 2025-06-30T03:26:25.446118Z  INFO pylight::lsp::server: File watcher started for workspace: /path/to/large/python-repo
2025-06-30T03:26:25.446149Z  WARN pylight_devtools: pylight stderr: 2025-06-30T03:26:25.446142Z  INFO pylight::index::symbol_index: Starting workspace indexing for: /path/to/large/python-repo
2025-06-30T03:26:26.744525Z  WARN pylight_devtools: pylight stderr: 2025-06-30T03:26:26.744507Z  INFO pylight::index::symbol_index: Found 92845 Python files to index in 1.30s (using 11 threads)
2025-06-30T03:26:26.744536Z  WARN pylight_devtools: pylight stderr: 2025-06-30T03:26:26.744516Z  INFO pylight::index::symbol_index: Starting parallel processing with 11 concurrent tasks
2025-06-30T03:26:32.300334Z  WARN pylight_devtools: pylight stderr: 2025-06-30T03:26:32.300303Z  INFO pylight::index::symbol_index: Parallel parsing completed in 5.29s (17551 files/sec)
2025-06-30T03:26:32.300353Z  WARN pylight_devtools: pylight stderr: 2025-06-30T03:26:32.300314Z  INFO pylight::index::symbol_index: Indexed 92845 files with 713084 symbols
2025-06-30T03:26:32.300354Z  WARN pylight_devtools: pylight stderr: 2025-06-30T03:26:32.300316Z  INFO pylight::lsp::server: Initial workspace indexing completed