Gadget Data Extraction

This is a data extraction tool designed to work with Fivetran and Google Cloud Functions to extract data from your Gadget application. It provides a standardized way to sync your Gadget data to your data warehouse.

Overview

This tool:

• Extracts data from your Gadget application using the GraphQL API
• Handles pagination and cursor-based navigation
• Manages sync state to ensure reliable incremental updates
• Supports all standard Gadget field types
• Excludes certain models (like session) by default
• Can be deployed as a Google Cloud Function

Prerequisites

• A Gadget application with an API key
• A Fivetran account
• Google Cloud Platform account (or another serverless function provider)

Environment Variables

You need to set the following environment variables:

• GADGET_APP_URL: The URL of your Gadget application (e.g., https://my-app.gadget.app)
• GADGET_API_KEY: Your Gadget API key
• GADGET_PAGE_SIZE (optional): Number of records to fetch per page (defaults to 100)

Usage

1. Deploy this code as a Google Cloud Function
2. Configure Fivetran to use this function as a custom connector
3. The function will handle the data extraction and state management automatically

Customization

If you're using a different serverless function provider or data movement platform, you may need to make minor adjustments:

• For different serverless providers: Modify the exports.run function to match your provider's expected format
• For different data movement platforms: Adjust the state management and response format in the exports.fivetran function

How It Works

1. The function receives a state object from Fivetran containing information about the last sync
2. It queries the Gadget GraphQL API to fetch the latest data
3. It manages cursors and timestamps to ensure reliable incremental updates
4. It returns the data in a format that Fivetran can process

Excluded Models

By default, the following models are excluded from extraction:

• session

You can modify the ExcludeModels array in the code to exclude additional models if needed.

Error Handling

The tool includes comprehensive error handling for:

• API request failures
• GraphQL errors
• Invalid state objects
• Missing environment variables

License

MIT