feature(examples): Add Shipwire connector

[fivertran-karunveluru]

Shipwire API Connector

Created: 2025-10-31

Business Owner: Supply Chain & Logistics Operations Team

Technical Owner: Data Engineering Team

Last Updated: 2025-10-31

Business Context

• Data Source: Shipwire API for comprehensive order management, fulfillment, and inventory data
• Business Criticality: Critical - supports order fulfillment, inventory tracking, and supply chain analytics
• Data Consumers: Operations teams, warehouse managers, inventory analysts, supply chain specialists, executive leadership
• Business SLAs: Data must be fresh within 4 hours for order fulfillment, 24 hours for inventory reporting
• Compliance Requirements: GDPR compliance for customer shipping data, customs documentation for international orders
• Budget Constraints: Shipwire API access included with subscription, rate limits based on plan tier

Technical Context

• API Documentation: https://www.shipwire.com/developers/
• Authentication Method: HTTP Basic authentication with username and password credentials
• Rate Limits: Varies by Shipwire plan, typically 1000-10000 requests/hour
• Data Volume:
  • Orders: 100-100,000+ orders per month
  • Purchase Orders: 10-5,000+ purchase orders per month
  • Products: 50-50,000+ products in catalog
  • Warehouse Locations: 1-50+ fulfillment centers
  • Order Items: 200-500,000+ line items per month
• Data Velocity: Orders updated in real-time during fulfillment, purchase orders updated daily, products updated weekly
• Data Quality: Structured JSON with consistent schema, some fields may be null for incomplete records
• Network Considerations: HTTPS only, RESTful API with standard reliability, global CDN infrastructure

Operational Context

• Deployment Environment: Development (sandbox), staging, and production environments
• Monitoring Requirements: Alert on >2% error rate, >2 hour sync time, order data discrepancies
• Maintenance Windows: Off-peak hours for non-critical updates, immediate deployment for fulfillment-critical fixes
• Team Structure: Data Engineering team, Supply Chain Operations, Warehouse managers, Inventory analysts
• Escalation Path: Data Engineer → Team Lead → Supply Chain Director → COO

API-Specific Details

• Base Endpoint: https://api.shipwire.com/api/v3 (production)
• Authentication: HTTP Basic authentication (username and password in Authorization header)
• Pagination: offset and limit parameters (max 1000 per page)
• Date Format: ISO 8601 (e.g., 2024-01-15T10:30:00Z)
• Response Format: JSON with nested objects and arrays
• Key Endpoints:
  • /orders - Order management data including status, items, shipping details
  • /purchaseOrders - Purchase order tracking with vendor and warehouse information
  • /products - Product catalog with SKUs, dimensions, classifications, and inventory flags

Data Schema Overview

• orders: Order management data including status, items, shipping details, holds, and fulfillment information
• purchase_orders: Purchase order tracking with vendor, warehouse, expected dates, and line items
• products: Product catalog with SKUs, descriptions, dimensions, classifications, battery configurations, and flags

Data Replication Expectations

• Initial Sync: Last 90 days of orders, purchase orders, and products for operational baseline
• Incremental Sync: Data since last successful sync timestamp using updatedAfter and updatedBefore parameters
• Sync Frequency:
  • Production: Every 4 hours for order data, daily for purchase orders and products
  • Development: Daily for all data types
• Data Retention: 2 years of historical order data for analytics and compliance requirements
• Backfill Capability: Full historical data available based on Shipwire retention policies
• Data Consistency: Near real-time with 4-hour maximum lag for fulfillment operations

Operational Requirements

• Uptime SLA: 99.7% availability during business hours (order fulfillment critical)
• Performance SLA:
  • Initial sync: <6 hours for 90 days of order data
  • Incremental sync: <45 minutes for daily updates
• Error Handling:
  • Automatic retry with exponential backoff and jitter
  • Dead letter queue for failed order/product records
  • Alert on consecutive sync failures during peak fulfillment periods
• Monitoring:
  • API response times and error rates
  • Order count trends and anomaly detection
  • Product catalog completeness validation
  • Purchase order status tracking
• Security:
  • Credentials encrypted in transit and at rest
  • Access logs maintained for 2 years (compliance)
  • Customer shipping data handling per privacy regulations

Rate Limiting Strategy

• Starter Plan: 1,000 requests/hour, 10,000 requests/day
• Professional Plan: 5,000 requests/hour, 50,000 requests/day
• Enterprise Plan: 10,000 requests/hour, 100,000 requests/day
• Recommended: Implement exponential backoff with jitter for 429 responses
• Error Handling: 429 status code indicates rate limit exceeded, respect Retry-After header
• Monitoring: Track rate limit utilization and plan for subscription upgrades

Data Quality Considerations

• Required Fields: order id, purchase_order id, product id, sku
• Optional Fields: externalId, description, status, category, dimensions, values
• Data Validation:
  • Order IDs must be unique
  • SKUs must be unique within product catalog
  • Order status must be valid enum value
  • Total values must be non-negative
  • Dates must be valid ISO 8601 format
• Data Completeness:
  • Orders: 100% have basic order information (id, status, createdDate)
  • Purchase Orders: 95%+ have complete vendor and warehouse details
  • Products: 90%+ have SKU and description information
• Duplicate Handling: Primary key constraints prevent duplicate order/product records

Integration Points

• Fivetran Destinations: Snowflake, BigQuery, Redshift, PostgreSQL
• Downstream Systems:
  • Order management systems (OMS)
  • Warehouse management systems (WMS)
  • Inventory planning and forecasting platforms
  • E-commerce platforms for order status updates
  • Financial reporting systems for revenue recognition
• Data Dependencies: None - standalone fulfillment and logistics data source
• External Dependencies: Shipwire API availability, fulfillment center processing schedules

Disaster Recovery

• Backup Strategy: Daily snapshots of all order, purchase order, and product tables
• Recovery Time Objective: 4 hours for full data recovery
• Recovery Point Objective: 2 hours maximum data loss for fulfillment-critical data
• Failover: Automatic failover to backup API credentials
• Testing: Monthly disaster recovery drills with supply chain team validation

Compliance & Security

• Data Classification: Customer shipping data - highly sensitive, order data - business sensitive
• Retention Policy: 2 years for order data (analytics), 1 year for operational fulfillment data
• Access Controls: Strict role-based access with principle of least privilege
• Audit Trail: All data access logged and monitored for compliance audits
• Encryption: Data encrypted in transit and at rest with enterprise-grade security
• Privacy: GDPR compliance for EU customer shipping data, CCPA compliance for CA customer data

Performance Optimization

• Parallel Processing: Multiple API calls for different data types (orders, purchase orders, products)
• Caching: Product catalog data cached for 24 hours (low change frequency)
• Indexing: Order ID, product SKU, purchase order ID, and date columns indexed
• Partitioning: Order data partitioned by date and warehouse for efficient querying
• Compression: Historical order data compressed for storage efficiency
• Memory Efficiency: Generator-based streaming prevents memory accumulation for large datasets

Troubleshooting Guide

• Common Issues:
  • Rate limit exceeded: Reduce sync frequency or upgrade Shipwire plan
  • Authentication failed: Verify username and password credentials
  • Missing order data: Check order status filters and date range parameters
  • Timeout errors: Increase timeout values or reduce batch size (max_records_per_page)
  • Product SKU duplicates: Validate SKU uniqueness constraints
  • Purchase order sync gaps: Verify warehouse and vendor ID mappings
• Debug Mode: Enable detailed logging for order and product data troubleshooting
• Support Contacts:
  • Technical: Data Engineering team
  • Business: Supply Chain Operations team
  • Vendor: Shipwire support (for API and account issues)
  • Compliance: Legal/Compliance team (for privacy and regulatory issues)

Checklist

Some tips and links to help validate your PR:

• Tested the connector with fivetran debug command.
• Added/Updated example specific README.md file, refer here for template.
• Followed Python Coding Standards, refer here

[github-actions]

🧹 Python Code Quality Check

✅ No issues found in Python Files.

🔍 See how this check works

This comment is auto-updated with every commit.

[fivetran-chinmayichandrasekar]

Left a few suggestions. Thanks.

[fivertran-karunveluru]

Left a few suggestions. Thanks.

Pushed a commit to address the comments. Please re-review.

[varundhall]

Same as #256 (review)