feature(examples): Add Planday connector

[fivertran-karunveluru]

Planday HR Connector

Created: 2025-10-31

Business Owner: HR & Workforce Management Operations Team

Technical Owner: Data Engineering Team

Last Updated: 2025-10-31

Business Context

• Data Source: Planday HR API for employee management, organizational structure, and skills/competency tracking
• Business Criticality: High - supports HR operations, workforce planning, and organizational structure management
• Data Consumers: HR teams, workforce management specialists, talent acquisition teams, department managers, executive leadership
• Business SLAs: Data must be fresh within 4-6 hours for HR operations, 24 hours for reporting and analytics
• Compliance Requirements: GDPR compliance for employee data, labor law compliance for employment records, data retention requirements
• Budget Constraints: Planday API access included with subscription, rate limits based on plan tier

Technical Context

• API Documentation: https://openapi.planday.com/api/hr
• Authentication Method: Bearer token (API key) authentication
• Rate Limits: Varies by Planday plan, typically 500-5000 requests/hour
• Data Volume:
  • Companies: 1-50+ companies per integration
  • Employees: 10-5,000+ employees per company
  • Departments: 5-200+ departments per company
  • Skills: 50-1,000+ skill records per company
• Data Velocity: Employee data updated on profile changes, department data updated quarterly, skills data updated periodically
• Data Quality: Structured JSON with consistent schema, some fields may be null for incomplete records
• Network Considerations: HTTPS only, RESTful API with standard reliability, European-based infrastructure

Operational Context

• Deployment Environment: Development (sandbox), staging, and production environments
• Monitoring Requirements: Alert on >2% error rate, >3 hour sync time, employee data discrepancies
• Maintenance Windows: Weekends for non-critical updates, immediate deployment for HR-critical fixes
• Team Structure: Data Engineering team, HR Operations, Workforce Management specialists, Department administrators
• Escalation Path: Data Engineer → Team Lead → HR Director → CHRO

API-Specific Details

• Base Endpoint: https://openapi.planday.com/api/hr
• Authentication: Bearer token in Authorization header (API key)
• Pagination: Offset and limit parameters (max 1000 per page, default 100)
• Date Format: ISO 8601 (e.g., 2024-01-15T10:30:00Z)
• Response Format: JSON with nested objects and arrays
• Key Endpoints:
  • /employees - Employee personal information, job details, and compensation
  • /departments - Organizational structure, departments, and hierarchies
  • /skills - Skills, competencies, and proficiency levels

Data Schema Overview

• employee: Employee personal information, job details, compensation, and employment status
• department: Department information, organizational hierarchy, budget, and cost center details
• skill: Skills and competencies with categories, levels, and certification requirements

Data Replication Expectations

• Initial Sync: Last 90 days of employee and department data (configurable up to 365 days)
• Incremental Sync: Data since last successful sync timestamp using updatedAfter and updatedBefore filters
• Sync Frequency:
  • Production: Every 4-6 hours for employee data, daily for department/skills data
  • Development: Daily for all data types
• Data Retention: 3-7 years of historical employee data for compliance requirements
• Backfill Capability: Full historical data available based on Planday retention policies
• Data Consistency: Near real-time with 4-6 hour maximum lag for HR operations

Operational Requirements

• Uptime SLA: 99.5% availability during business hours
• Performance SLA:
  • Initial sync: <4 hours for 90 days of employee/department data
  • Incremental sync: <45 minutes for regular updates
• Error Handling:
  • Automatic retry with exponential backoff (3 attempts default)
  • Dead letter queue for failed employee/department records
  • Alert on consecutive sync failures
• Monitoring:
  • API response times and error rates
  • Employee count trends and anomaly detection
  • Department structure completeness validation
  • Skills data freshness checks
• Security:
  • API keys stored securely and never logged
  • Access logs maintained for 2 years (compliance)
  • Employee PII handling per privacy regulations

Rate Limiting Strategy

• Basic Plan: 500 requests/hour, 5,000 requests/day
• Professional Plan: 2,000 requests/hour, 20,000 requests/day
• Enterprise Plan: 5,000 requests/hour, 50,000 requests/day
• Recommended: Implement exponential backoff with jitter for 429 responses
• Error Handling: 429 status code indicates rate limit exceeded, respect Retry-After header
• Monitoring: Track rate limit utilization and plan for subscription upgrades

Data Quality Considerations

• Required Fields: employee: id, employee_number, first_name, last_name, email; department: id, name; skill: id, name
• Optional Fields: employee: phone, department_id, position, manager_id, salary, hourly_rate; department: description, manager_id, parent_department_id, budget, cost_center; skill: description, category, level, certification_required
• Data Validation:
  • Employee IDs must be unique within company
  • Email addresses must be valid format
  • Hire dates must be valid and not future dates
  • Salary/hourly_rate amounts must be non-negative
  • Department hierarchies must be valid (no circular references)
• Data Completeness:
  • Employees: 95%+ have basic demographic data
  • Departments: 90%+ have name and organizational structure
  • Skills: 85%+ have name and category information
• Duplicate Handling: Primary key constraints prevent duplicate employee/department/skill records

Integration Points

• Fivetran Destinations: Snowflake, BigQuery, Redshift, PostgreSQL
• Downstream Systems:
  • HR information systems (HRIS)
  • Workforce analytics platforms
  • Payroll systems (if integrated)
  • Talent management systems
  • Organizational chart tools
  • Skills and competency tracking systems
• Data Dependencies: None - standalone HR/workforce management data source
• External Dependencies: Planday API availability, workforce management schedules

Disaster Recovery

• Backup Strategy: Daily snapshots of all HR and department tables
• Recovery Time Objective: 6 hours for full data recovery
• Recovery Point Objective: 4 hours maximum data loss for HR-critical data
• Failover: Automatic failover to backup API credentials
• Testing: Quarterly disaster recovery drills with HR team validation

Compliance & Security

• Data Classification: Employee PII - highly sensitive, department data - internal, skills data - low sensitivity
• Retention Policy: 3-7 years for employee data (compliance), 2 years for operational HR data
• Access Controls: Strict role-based access with principle of least privilege
• Audit Trail: All data access logged and monitored for compliance audits
• Encryption: Data encrypted in transit and at rest with enterprise-grade security
• Privacy: GDPR compliance for EU employees, CCPA compliance for CA employees, labor law compliance

Performance Optimization

• Parallel Processing: Multiple API calls for different data types (employees, departments, skills)
• Caching: Department and skills data cached for 24 hours
• Indexing: Employee ID, department ID, skill ID, and date columns indexed
• Partitioning: Employee data partitioned by hire date and department for efficient querying
• Memory Efficiency: Streaming-based processing prevents memory accumulation for large employee datasets

Troubleshooting Guide

• Common Issues:
  • Rate limit exceeded: Reduce sync frequency or upgrade Planday plan
  • API key expired: Verify API key validity and permissions
  • Missing employee data: Check company access and employee active status
  • Timeout errors: Increase timeout values (default 30s) or reduce batch size
  • Department hierarchy issues: Validate parent_department_id relationships
  • Skills data inconsistencies: Verify skill category and level mappings
• Debug Mode: Enable detailed logging via enable_debug_logging configuration parameter
• Support Contacts:
  • Technical: Data Engineering team
  • Business: HR Operations team
  • Vendor: Planday support (for API and account issues)
  • Compliance: Legal/Compliance team (for privacy and regulatory issues)

Checklist

Some tips and links to help validate your PR:

• Tested the connector with fivetran debug command.
• Added/Updated example specific README.md file, refer here for template.
• Followed Python Coding Standards, refer here

[github-actions]

🧹 Python Code Quality Check

✅ No issues found in Python Files.

🔍 See how this check works

This comment is auto-updated with every commit.

[fivetran-chinmayichandrasekar]

Left a few suggestions. Thanks.

[fivertran-karunveluru]

Left a few suggestions. Thanks.

Pushed a commit to address the comments. Please re-review.

[varundhall]

Same as #256 (review)

[fivetran-chinmayichandrasekar]

Left one suggestion. Thanks.

[fivetran-chinmayichandrasekar]

Suggested change

	## Features
	- Syncs employee personal information, job details, and compensation data from Planday HR API
	- Bearer token authentication with automatic retry logic for expired tokens (refer to `execute_api_request` function)
	- Offset-based pagination with automatic page traversal (refer to `get_employees`, `get_departments`, and `get_skills` functions)
	- Memory-efficient streaming prevents data accumulation for large employee datasets
	- Incremental synchronization using timestamp-based cursors (refer to `get_time_range` function)
	- Comprehensive error handling with exponential backoff retry logic (refer to `__handle_rate_limit` and `__handle_request_error` functions)
	- Data quality validation and field mapping for consistent database schema (refer to `__map_employee_data`, `__map_department_data`, and `__map_skill_data` functions)
	## Features
	- Syncs employee personal information, job details, and compensation data from Planday HR API
	- Bearer token authentication with automatic retry logic for expired tokens (refer to the `execute_api_request` function)
	- Offset-based pagination with automatic page traversal (refer to the `get_employees`, `get_departments`, and `get_skills` functions)
	- Memory-efficient streaming prevents data accumulation for large employee datasets
	- Incremental synchronization using timestamp-based cursors (refer to the `get_time_range` function)
	- Comprehensive error handling with exponential backoff retry logic (refer to the `__handle_rate_limit` and `__handle_request_error` functions)
	- Data quality validation and field mapping for consistent database schema (refer to the `__map_employee_data`, `__map_department_data`, and `__map_skill_data` functions)