Bug: Connector date range filtering broken

[vaishcodescape]

Issue Summary

In this Multiple connectors have broken date range filtering, causing them to fetch ALL data instead of incremental updates within specified time periods.

---

🎯 Affected Files

• surfsense_backend/app/connectors/jira_connector.py (lines 227, 253)
• surfsense_backend/app/connectors/clickup_connector.py (line 171)
• surfsense_backend/app/connectors/linear_connector.py ✅ (working correctly)

---

🐛 Current Problems

Jira Connector

• Issue: JQL query is constructed but commented out in API request
• Lines 227 & 253: TODO comments indicating incomplete implementation
• Result: Fetches ALL issues regardless of date range

ClickUp Connector

• Issue: Date parameters accepted but completely ignored
• Line 171: TODO comment for missing date filtering
• Result: Fetches ALL tasks regardless of date range

Linear Connector

• Status: ✅ WORKING CORRECTLY - No changes needed

---

💥 Impact

• Users can't filter data by time periods
• Performance issues due to fetching all data
• API rate limit waste on already-indexed data
• No incremental data synchronisation support

---

🔧 Technical Details

Current Broken Code

Jira Connector:

# Line 227: TODO comment
# TODO : This JQL needs some improvement to work as expected

# Line 253: JQL commented out
# "jql": "",   TODO : Add a JQL query to filter from a date range

ClickUp Connector:

# Line 171: Date filtering missing
# TODO : Include date range in api request

API Requirements

Jira:

• Uses JQL (Jira Query Language)
• Date format: YYYY-MM-DD
• Fields: createdDate, updatedDate

ClickUp:

• Query parameters: date_created_gt, date_created_lt, date_updated_gt, date_updated_lt
• Requires Unix timestamps in milliseconds

---

🛠️ Proposed Fixes

Jira Connector Fix

# Use the constructed JQL query
params = {
    "jql": _jql,  # Instead of empty string
    "fields": ",".join(fields),
    "maxResults": 100,
    "startAt": 0,
}

# Enhanced JQL with OR logic
date_filter = (
    f"(createdDate >= '{start_date}' AND createdDate <= '{end_date}') "
    f"OR (updatedDate >= '{start_date}' AND updatedDate <= '{end_date}')"
)

ClickUp Connector Fix

# Convert dates to Unix timestamps
start_timestamp = int(datetime.strptime(start_date, "%Y-%m-%d").timestamp() * 1000)
end_timestamp = int(datetime.strptime(end_date, "%Y-%m-%d").timestamp() * 1000)

# Add date filter parameters
params = {
    "page": 0,
    "order_by": "created",
    "reverse": "true",
    "subtasks": "true",
    "include_closed": str(include_closed).lower(),
    # Date filtering
    "date_created_gt": start_timestamp,
    "date_created_lt": end_timestamp,
    "date_updated_gt": start_timestamp,
    "date_updated_lt": end_timestamp,
}

[vaishcodescape]

I'll work on this issue and file a PR for review you can assign this to me : )

[MODSetter]

@vaishcodescape Can you look at this now as I have shifted everything to celery.

[vaishcodescape]

Sure