|
| 1 | +# Connector SDK Goshippo API Connector Example |
| 2 | + |
| 3 | +## Connector overview |
| 4 | + |
| 5 | +This connector syncs shipment data from the Goshippo API to your destination using the Fivetran Connector SDK. Goshippo is a shipping platform that provides APIs to manage shipments, get shipping rates, create labels, and track packages. |
| 6 | + |
| 7 | +The connector fetches shipment records along with related data such as shipping rates, parcel details, and messages. It supports incremental synchronization by tracking the last updated timestamp of shipments and only fetching new or updated records in subsequent syncs. |
| 8 | + |
| 9 | +## Requirements |
| 10 | + |
| 11 | +- [Supported Python versions](https://github.com/fivetran/fivetran_connector_sdk/blob/main/README.md#requirements) |
| 12 | +- Operating system: |
| 13 | + - Windows: 10 or later (64-bit only) |
| 14 | + - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) |
| 15 | + - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) |
| 16 | + |
| 17 | +## Getting started |
| 18 | + |
| 19 | +Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. |
| 20 | + |
| 21 | +## Features |
| 22 | + |
| 23 | +- Incremental sync - Fetches only new or updated shipments since the last sync by tracking the `object_updated` timestamp |
| 24 | +- Pagination support - Handles API pagination using next-page URL tokens |
| 25 | +- Retry logic - Implements exponential backoff for transient errors (rate limits, timeouts, 5xx errors) |
| 26 | +- Related data handling - Syncs shipment rates, parcels, and messages along with shipment records |
| 27 | +- Regular checkpointing - Checkpoints state every 100 records to ensure sync can resume from the correct position |
| 28 | +- Comprehensive error handling - Catches specific exceptions and provides detailed error logging |
| 29 | + |
| 30 | +## Configuration file |
| 31 | + |
| 32 | +The connector requires the following configuration parameters: |
| 33 | + |
| 34 | +```json |
| 35 | +{ |
| 36 | + "api_token": "<YOUR_GOSHIPPO_API_TOKEN>" |
| 37 | +} |
| 38 | +``` |
| 39 | + |
| 40 | +Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. |
| 41 | + |
| 42 | +## Authentication |
| 43 | + |
| 44 | +This connector uses API token authentication to connect to the Goshippo API. The API token is passed in the `Authorization` header as `ShippoToken <api_token>` (refer to the `build_headers()` function). |
| 45 | + |
| 46 | +To obtain your API token: |
| 47 | + |
| 48 | +1. Log in to your [Goshippo account](https://apps.goshippo.com/login). |
| 49 | +2. Go to **Settings > API**. |
| 50 | +3. Copy your API token or generate a new one. |
| 51 | +4. Add the API token to the `configuration.json` file. |
| 52 | + |
| 53 | +## Pagination |
| 54 | + |
| 55 | +The connector implements next-page URL pagination to handle large datasets (refer to the `sync_shipments()` function). The Goshippo API returns a `next` URL in the response when more pages are available. The connector extracts the `page_token` parameter from this URL using the `extract_page_token()` function and includes it in subsequent requests via the `build_query_params()` function. |
| 56 | + |
| 57 | +The pagination loop continues until the API returns no `next` URL, indicating that all available data has been fetched. Each page fetches up to 25 shipments as defined by the `__PAGE_SIZE` constant. |
| 58 | + |
| 59 | +## Data handling |
| 60 | + |
| 61 | +The connector processes shipment data and related entities as follows: |
| 62 | + |
| 63 | +1. Main shipments - The `flatten_shipment()` function extracts and flattens shipment fields including addresses (from, to, return), carrier accounts, and metadata. Nested objects like addresses are flattened into individual columns with prefixes (`from_`, `to_`, `return_`). |
| 64 | +2. Related data - The `process_shipment()` function coordinates upserting the main shipment record and its related data: |
| 65 | + - Rates - The `process_rates()` function processes shipping rate options including carrier, amount, service level, and estimated delivery days. |
| 66 | + - Parcels - The `process_parcels()` function processes parcel dimensions, weight, and metadata. |
| 67 | + - Messages - The `process_messages()` function processes API messages with a generated unique ID combining shipment ID and message index. |
| 68 | + |
| 69 | +The connector uses the `op.upsert()` operation to insert or update records in the destination tables. Records are processed immediately as they are fetched to avoid loading large datasets into memory. |
| 70 | + |
| 71 | +## Error handling |
| 72 | + |
| 73 | +The connector implements comprehensive error handling with retry logic (refer to the `fetch_shipments_page()`, `handle_response_status()`, and `handle_request_exception()` functions): |
| 74 | + |
| 75 | +- Retry logic - The connector retries requests up to 3 times (`__MAX_RETRIES`) for transient errors including: |
| 76 | + - Network timeouts and connection errors |
| 77 | + - Rate limiting (HTTP 429) |
| 78 | + - Server errors (HTTP 500, 502, 503, 504) |
| 79 | +- Exponential backoff - The `calculate_retry_delay()` function implements exponential backoff with a maximum delay of 60 seconds. |
| 80 | +- Fail fast - The connector immediately raises exceptions for permanent errors like authentication failures (4xx errors) without retrying. |
| 81 | +- Specific exception handling - The connector catches specific exceptions (`RuntimeError`, `requests.RequestException`, `ValueError`, `KeyError`, `requests.Timeout`, `requests.ConnectionError`, `AttributeError`, `IndexError`, `TypeError`) rather than generic exceptions to avoid masking unexpected errors. |
| 82 | + |
| 83 | +All errors are logged using the SDK's logging framework before being raised. |
| 84 | + |
| 85 | +## Tables created |
| 86 | + |
| 87 | +The connector creates the `SHIPMENT`, `SHIPMENT_RATES`, `SHIPMENT_PARCELS`, and `SHIPMENT_MESSAGES` tables in the destination. |
| 88 | + |
| 89 | +Refer to the `schema()` function for more details. |
| 90 | + |
| 91 | +### SHIPMENTS |
| 92 | + |
| 93 | +| Column | Type | Description | |
| 94 | +|--------|------|-------------| |
| 95 | +| `object_id` | STRING | Unique identifier for the shipment (Primary Key) | |
| 96 | +| `object_created` | STRING | Timestamp when shipment was created | |
| 97 | +| `object_updated` | STRING | Timestamp when shipment was last updated | |
| 98 | +| `object_owner` | STRING | Owner of the shipment object | |
| 99 | +| `status` | STRING | Current status of the shipment | |
| 100 | +| `metadata` | STRING | Additional metadata | |
| 101 | +| `shipment_date` | STRING | Date of shipment | |
| 102 | +| `test` | STRING | Test mode indicator | |
| 103 | +| `order` | STRING | Associated order ID | |
| 104 | +| `carrier_accounts` | STRING | JSON array of carrier account IDs | |
| 105 | +| `customs_declaration` | STRING | Customs declaration ID | |
| 106 | +| `alternate_address_to` | STRING | JSON object of alternate delivery address | |
| 107 | +| `from_name` | STRING | Sender name | |
| 108 | +| `from_company` | STRING | Sender company | |
| 109 | +| `from_street1` | STRING | Sender street address | |
| 110 | +| `from_city` | STRING | Sender city | |
| 111 | +| `from_state` | STRING | Sender state | |
| 112 | +| `from_zip` | STRING | Sender ZIP code | |
| 113 | +| `from_country` | STRING | Sender country | |
| 114 | +| `from_phone` | STRING | Sender phone | |
| 115 | +| `from_email` | STRING | Sender email | |
| 116 | +| `to_name` | STRING | Recipient name | |
| 117 | +| `to_company` | STRING | Recipient company | |
| 118 | +| `to_street1` | STRING | Recipient street address | |
| 119 | +| `to_city` | STRING | Recipient city | |
| 120 | +| `to_state` | STRING | Recipient state | |
| 121 | +| `to_zip` | STRING | Recipient ZIP code | |
| 122 | +| `to_country` | STRING | Recipient country | |
| 123 | +| `to_phone` | STRING | Recipient phone | |
| 124 | +| `to_email` | STRING | Recipient email | |
| 125 | +| `return_name` | STRING | Return address name | |
| 126 | +| `return_company` | STRING | Return address company | |
| 127 | +| `return_street1` | STRING | Return address street | |
| 128 | +| `return_city` | STRING | Return address city | |
| 129 | +| `return_state` | STRING | Return address state | |
| 130 | +| `return_zip` | STRING | Return address ZIP code | |
| 131 | +| `return_country` | STRING | Return address country | |
| 132 | +| `return_phone` | STRING | Return address phone | |
| 133 | +| `return_email` | STRING | Return address email | |
| 134 | + |
| 135 | +### SHIPMENT_RATES |
| 136 | + |
| 137 | +| Column | Type | Description | |
| 138 | +|--------|------|-------------| |
| 139 | +| `rate_object_id` | STRING | Unique identifier for the rate (Primary Key) | |
| 140 | +| `shipment_object_id` | STRING | Associated shipment ID (Primary Key) | |
| 141 | +| `amount` | STRING | Shipping cost amount | |
| 142 | +| `currency` | STRING | Currency code | |
| 143 | +| `provider` | STRING | Shipping carrier provider | |
| 144 | +| `servicelevel_name` | STRING | Service level name | |
| 145 | +| `servicelevel_token` | STRING | Service level token | |
| 146 | +| `estimated_days` | STRING | Estimated delivery days | |
| 147 | +| `duration_terms` | STRING | Duration terms | |
| 148 | +| `carrier_account` | STRING | Carrier account ID | |
| 149 | +| `object_created` | STRING | Timestamp when rate was created | |
| 150 | + |
| 151 | +### SHIPMENT_PARCELS |
| 152 | + |
| 153 | +| Column | Type | Description | |
| 154 | +|--------|------|-------------| |
| 155 | +| `parcel_object_id` | STRING | Unique identifier for the parcel (Primary Key) | |
| 156 | +| `shipment_object_id` | STRING | Associated shipment ID (Primary Key) | |
| 157 | +| `length` | STRING | Parcel length | |
| 158 | +| `width` | STRING | Parcel width | |
| 159 | +| `height` | STRING | Parcel height | |
| 160 | +| `distance_unit` | STRING | Unit of distance measurement | |
| 161 | +| `weight` | STRING | Parcel weight | |
| 162 | +| `mass_unit` | STRING | Unit of weight measurement | |
| 163 | +| `metadata` | STRING | Additional metadata | |
| 164 | +| `object_created` | STRING | Timestamp when parcel was created | |
| 165 | +| `object_updated` | STRING | Timestamp when parcel was last updated | |
| 166 | + |
| 167 | +### SHIPMENT_MESSAGES |
| 168 | + |
| 169 | +| Column | Type | Description | |
| 170 | +|--------|------|-------------| |
| 171 | +| `message_id` | STRING | Generated unique identifier (shipment_id_index) (Primary Key) | |
| 172 | +| `shipment_object_id` | STRING | Associated shipment ID | |
| 173 | +| `source` | STRING | Message source | |
| 174 | +| `code` | STRING | Message code | |
| 175 | +| `text` | STRING | Message text | |
| 176 | + |
| 177 | +## Additional considerations |
| 178 | + |
| 179 | +The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. |
0 commit comments