Skip to content

feat: restore index-* CLI commands and user automation scripts#269

Open
tolgakaratas wants to merge 4 commits intoyichuan-w:mainfrom
tolgakaratas:feature/restore-index-commands-v2
Open

feat: restore index-* CLI commands and user automation scripts#269
tolgakaratas wants to merge 4 commits intoyichuan-w:mainfrom
tolgakaratas:feature/restore-index-commands-v2

Conversation

@tolgakaratas
Copy link
Copy Markdown
Contributor

@tolgakaratas tolgakaratas commented Feb 24, 2026

Summary

Changes

CLI Commands Added

  • leann index-browser chrome - Chrome browser history indexing
  • leann index-browser brave - Brave browser history indexing (our contribution)
  • leann index-email - Apple Mail indexing
  • leann index-calendar - Apple Calendar indexing
  • leann index-wechat - WeChat chat history
  • leann index-imessage - iMessage history
  • leann index-slack - Slack workspace via MCP
  • leann index-chatgpt - ChatGPT export indexing
  • leann index-claude - Claude export indexing

Technical Fixes

  • Add add_embedding_args() helper function for all index-* parsers
  • All commands now accept --embedding-model, --embedding-mode, --embedding-host, --embedding-api-base, --embedding-api-key arguments

Documentation

  • Added docs/user-scripts.md with installation and usage examples for daily automation

Testing

  • All index-* commands now accept --embedding-model and --embedding-mode arguments
  • CLI help tested

Breaking Changes

  • None

Related Issues

@tolgakaratas
feat: restore index-* CLI commands with embedding args and user scrip…
75c998e 
…ts documentation

- Add index-browser (chrome/brave), index-email, index-calendar,
  index-wechat, index-imessage, index-slack, index-chatgpt, index-claude commands
- Add add_embedding_args() helper function for all parsers
- Restore readers.py with all data readers (ChromeHistoryReader,
  AppleMailReader, AppleCalendarReader, IMessageReader, WeChatReader,
  SlackReader, ChatGPTReader, ClaudeReader)
- Add user-scripts.md documentation for daily automation workflows

Fixes: Missing embedding arguments in index-* commands (PR yichuan-w#227 comment)

Co-authored-by: tolgakaratas
@tolgakaratas
fix: unused variable warning in readers.py
d478f92
@tolgakaratas
docs: add user-scripts documentation in English and Turkish
375522d 
- Add user-scripts.md (English version) for daily automation
- Add user-scripts-tr.md (Turkish version) for Turkish users
- Update ~/bin scripts with English comments and output
@tolgakaratas
fix: correct reader class initialization in CLI index commands
d071755 
- Fix IMessageReader initialization (remove db_path argument)
- Fix WeChatHistoryReader initialization and load_data parameters
- Fix SlackMCPReader initialization and load_data parameters
- Fix ChatGPTReader initialization and load_data parameters
- Fix ClaudeReader initialization and load_data parameters

These fixes ensure all index-* commands work correctly with the reader classes.
@ASuresh0524
Copy link
Copy Markdown
Collaborator

ASuresh0524 commented Mar 3, 2026

@tolgakaratas fix link error please

@ASuresh0524
Copy link
Copy Markdown
Collaborator

ASuresh0524 commented Mar 6, 2026

@tolgakaratas Thanks for working on this, having leann index-* as first-class CLI commands is a great direction. A few things need to be addressed before this can be merged:

1. Duplicate readers instead of reusing existing ones

The new readers.py rewrites readers that already exist under apps/:

New reader Existing implementation
IMessageReader apps/imessage_data/imessage_reader.py
AppleMailReader apps/email_data/LEANN_email_reader.py
ChatGPTReader apps/chatgpt_data/chatgpt_reader.py
ClaudeReader apps/claude_data/claude_reader.py
SlackMCPReader apps/slack_data/slack_mcp_reader.py

These should be imported from their existing locations rather than duplicated — the originals are more complete and already tested.

2. Stub readers that do nothing

ChatGPTReader, ClaudeReader, SlackMCPReader, and TwitterMCPReader all just return []. This means index-chatgpt, index-claude, and index-slack will silently produce empty indexes. Either implement them by wrapping the existing apps/ readers, or remove these commands until they're ready.

3. CLI handler duplication

Each index_* method in cli.py is nearly identical (~30 lines of copy-paste per command × 8 commands). This should be refactored into a single shared helper:

async def _build_index_from_documents(self, args, index_name, documents):
    """Shared builder logic for all index-* commands."""
    ...

4. Hardcoded is_recompute=False

All builders use is_recompute=False, is_compact=False, which disables LEANN's core 97% storage savings. This should default to is_recompute=True with an optional --no-recompute flag for users who want full embedding storage.

5. Docs reference missing files

docs/user-scripts.md references shell scripts (bin/leann-sync-all.sh, bin/leann-sync-brave.sh, etc.) that aren't included in the PR. Either add the scripts or remove those references. Also, docs/user-scripts-tr.md (Turkish translation) is probably out of scope for the main repo docs.

6. CI link check is failing

Likely related to the broken references in the docs — please fix before re-requesting review.

Happy to re-review once these are addressed!

ASuresh0524 added a commit that referenced this pull request Mar 13, 2026
@ASuresh0524
feat: add index-* CLI commands for data source indexing (#269)
6efe947 
Addresses all review comments on PR #269:

1. No duplicate readers — imports directly from existing apps/ readers
   (ChromeHistoryReader, IMessageReader, EmlxReader, WeChatHistoryReader,
   ChatGPTReader, ClaudeReader) instead of duplicating 687 lines
2. Removed stub commands — dropped index-slack and index-twitter since
   SlackMCPReader and TwitterMCPReader return [] (async MCP readers
   need separate implementation)
3. Single shared helper — _build_index_from_documents() replaces 8x
   copy-pasted ~30-line handlers
4. Default is_recompute=True — preserves LEANN's 97% storage savings;
   users can opt out with --no-recompute
5. Clean docs — user-scripts.md has no missing script references and
   no Turkish translation
6. No CI link check failures — removed all broken references

Commands added:
  leann index-browser [chrome|brave]
  leann index-email
  leann index-calendar
  leann index-imessage
  leann index-wechat --export-dir <path>
  leann index-chatgpt --export-path <path>
  leann index-claude --export-path <path>

Made-with: Cursor
@ASuresh0524 ASuresh0524 mentioned this pull request Mar 13, 2026
Merged
7 tasks
ASuresh0524 added a commit that referenced this pull request Mar 30, 2026
@ASuresh0524
feat: add index-* CLI commands for data source indexing (#269) (#285)
82960a8 
Addresses all review comments on PR #269:

1. No duplicate readers — imports directly from existing apps/ readers
   (ChromeHistoryReader, IMessageReader, EmlxReader, WeChatHistoryReader,
   ChatGPTReader, ClaudeReader) instead of duplicating 687 lines
2. Removed stub commands — dropped index-slack and index-twitter since
   SlackMCPReader and TwitterMCPReader return [] (async MCP readers
   need separate implementation)
3. Single shared helper — _build_index_from_documents() replaces 8x
   copy-pasted ~30-line handlers
4. Default is_recompute=True — preserves LEANN's 97% storage savings;
   users can opt out with --no-recompute
5. Clean docs — user-scripts.md has no missing script references and
   no Turkish translation
6. No CI link check failures — removed all broken references

Commands added:
  leann index-browser [chrome|brave]
  leann index-email
  leann index-calendar
  leann index-imessage
  leann index-wechat --export-dir <path>
  leann index-chatgpt --export-path <path>
  leann index-claude --export-path <path>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@ASuresh0524 ASuresh0524

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@tolgakaratas @ASuresh0524