Feature Request:Scheduled/Automated Connector Syncing

[vaishcodescape]

Problem Statement

Currently, users must manually trigger syncs for each connector (Slack, Notion, GitHub, Linear, etc.) to index new content into their search spaces. This creates several issues:

• Data staleness: Users often forget to sync, leading to outdated search results
• Manual overhead: Tedious to sync multiple connectors regularly
• Peak load issues: Users tend to sync during business hours, causing resource spikes
• Missed updates: Important information may not be searchable until user manually syncs

Proposed Solution

Implement an automated scheduling system that allows users to configure periodic syncs for their connectors.

Core Features:

1. Schedule Types:

   • Hourly
   • Daily (with time selection)
   • Weekly (with day + time selection)
   • Custom (cron expression for advanced users)

2. Per-Connector Configuration:

   • Each connector can have its own schedule
   • Different schedules per search space
   • Enable/disable schedules without deleting them

3. Background Processing:

   • Non-blocking execution using existing background task infrastructure
   • Automatic retry logic for failed syncs
   • Logging and monitoring of scheduled jobs

4. User Control:

   • API endpoints to create/read/update/delete schedules
   • View schedule history and status
   • Manual sync still available (doesn't interfere with scheduled syncs)

Benefits

For Users:

• ✅ Always-fresh data without manual intervention
• ✅ Set-it-and-forget-it convenience
• ✅ Reduced cognitive load (no need to remember to sync)
• ✅ Better search accuracy with up-to-date content

For System:

• ✅ Distributed load (syncs spread across time)
• ✅ Predictable resource usage
• ✅ Reduced API costs (smart incremental syncing)
• ✅ Better external API rate limit management
• ✅ Improved scalability

[vaishcodescape]

I would like to work on this feature please assign this to me :)
Thanks !

[MODSetter]

@vaishcodescape I am also aware of this issue. Right now I want to shift our Messaging Queues from fasiapi's background_tasks() to celery[redis]. Let's tackle this after this.

[vaishcodescape]

Done ! 👍🏻