[Feature]: 支持图像、视频、音频等多模态资源的智能解析和访问

[summerbuild]

Problem Statement

以图搜图之类的功能什么时候能支持呢？

Proposed Solution

Alternatives Considered

Feature Area

Core (Client/Engine)

Use Case

Example API (Optional)

Additional Context

No response

Contribution

• I am willing to contribute to implementing this feature

[novvoo]

Currently, the workspace has limited capabilities regarding text content recognition and extraction from non-code documents. Users often need to extract text, metadata, or structured information from various file formats (such as PDFs, Office documents, or images) directly within the environment, but the current solution lacks comprehensive format support or performance efficiency.
Describe the solution you'd like
I propose integrating Kreuzberg, a polyglot document intelligence framework with a Rust core, into the code workspace. This integration would enable robust text content recognition and data extraction across a wide variety of file types.
Key Capabilities of Kreuzberg:
Core: High-performance Rust backend.
Format Support: Handles PDFs, Office documents, images, and 76+ other formats.
Extraction: Capable of extracting text, metadata, and structured information.
Polyglot: Designed to work seamlessly across different language environments.
Describe alternatives you've considered
Existing OCR tools: Often limited to images and lack deep structure extraction for Office/PDF files.
Python-based libraries: While flexible, they may introduce heavier dependencies and slower performance compared to a Rust-core solution.
External APIs: Relying on external services introduces latency and privacy concerns; a local integration like Kreuzberg is preferable for a workspace environment.
Potential Implementation Strategy
Dependency Management: Add Kreuzberg as a core dependency (via Cargo if the workspace uses Rust, or via FFI/WASM bindings if the workspace is primarily Electron/JS/TS).
API Exposure: Expose a unified interface (e.g., workspace.document.extractText(file)) that leverages Kreuzberg's engine.
Configuration: Allow users to enable/disable specific parsers based on their needs to optimize resource usage.
Additional Context
Integrating Kreuzberg would significantly boost the workspace's ability to handle RAG (Retrieval-Augmented Generation) pipelines, code context analysis from documentation, and general file indexing.

[MaojiaSheng]

More multi-model formats will be supported in the future, maybe in Q2.