Skip to content

Latest commit

 

History

History
888 lines (713 loc) · 26.4 KB

File metadata and controls

888 lines (713 loc) · 26.4 KB
layout default
title Chapter 1: System Overview
nav_order 1
has_children false
parent Teable Database Platform

Chapter 1: System Overview

Welcome to Chapter 1: System Overview. In this part of Teable: Deep Dive Tutorial, you will build an intuitive mental model first, then move into concrete implementation details and practical production tradeoffs.

Understanding Teable's multi-dimensional database platform and its approach to modern data management

🎯 Learning Objectives

By the end of this chapter, you'll understand:

  • Teable's position in the database ecosystem
  • Multi-dimensional data modeling concepts
  • Performance vs. flexibility trade-offs
  • Real-time collaboration architecture
  • Core architectural principles

🗂️ The Database Platform Evolution

From Spreadsheets to Databases

The evolution of data management tools reflects changing needs:

graph TD
    A[Spreadsheets<br/>1979 - Excel] --> B[Relational DB<br/>1970s - SQL]
    A --> C[No-Code Tools<br/>2010s - Airtable]
    B --> D[Modern Platforms<br/>2020s - Teable]

    B --> C
    C --> D

    style A fill:#e1f5fe
    style D fill:#c8e6c9
Loading

Key Evolution Drivers:

  • Accessibility: From technical experts to business users
  • Collaboration: From single-user to team workflows
  • Flexibility: From rigid schemas to dynamic structures
  • Integration: From isolated tools to connected ecosystems
  • Performance: From desktop limits to cloud-scale operations

Teable's Unique Position

Teable combines the best of multiple worlds:

Feature Spreadsheet Traditional DB No-Code Tool Teable
User Experience ⭐⭐⭐⭐⭐ ⭐⭐⭐⭐ ⭐⭐⭐⭐⭐
Data Integrity ⭐⭐ ⭐⭐⭐⭐⭐ ⭐⭐⭐ ⭐⭐⭐⭐⭐
Scalability ⭐⭐⭐⭐⭐ ⭐⭐⭐ ⭐⭐⭐⭐⭐
Customization ⭐⭐⭐⭐⭐ ⭐⭐⭐⭐ ⭐⭐ ⭐⭐⭐⭐⭐
Collaboration ⭐⭐⭐ ⭐⭐⭐⭐⭐ ⭐⭐⭐⭐⭐
API Access ⭐⭐⭐⭐⭐ ⭐⭐⭐ ⭐⭐⭐⭐⭐

🏗️ Multi-Dimensional Data Architecture

Beyond Relational Tables

Traditional relational databases organize data in two dimensions:

Table: Users
┌─────────────┬──────────────┬─────────────┐
│ id (PK)     │ name         │ email       │
├─────────────┼──────────────┼─────────────┤
│ 1           │ Alice        │ alice@...   │
│ 2           │ Bob          │ bob@...     │
└─────────────┴──────────────┴─────────────┘

Table: Orders
┌─────────────┬──────────────┬─────────────┐
│ id (PK)     │ user_id (FK) │ amount      │
├─────────────┼──────────────┼─────────────┤
│ 1           │ 1            │ $100        │
│ 2           │ 2            │ $250        │
└─────────────┴──────────────┴─────────────┘

Teable adds multiple dimensions through views, links, and computed fields:

// Multi-dimensional data structure
interface TeableBase {
  id: string;
  name: string;
  tables: TeableTable[];
  views: TeableView[];
  automations: Automation[];
}

interface TeableTable {
  id: string;
  name: string;
  fields: TeableField[];
  records: Record[];
  links: Link[]; // Relationships between tables
}

interface TeableView {
  id: string;
  name: string;
  tableId: string;
  filters: Filter[];
  sorts: Sort[];
  groupings: Grouping[];
  fields: string[]; // Which fields to display
}

interface TeableField {
  id: string;
  name: string;
  type: FieldType;
  config: FieldConfig;
  computed?: Computation; // Dynamic calculations
}

Views: Data Perspectives

Views provide different "lenses" on the same data:

// Example: Project Management Base
const projectBase: TeableBase = {
  id: "project-mgmt",
  name: "Project Management",
  tables: [
    {
      id: "tasks",
      name: "Tasks",
      fields: [
        { id: "title", name: "Title", type: "text" },
        { id: "status", name: "Status", type: "select", config: { options: ["Todo", "In Progress", "Done"] }},
        { id: "assignee", name: "Assignee", type: "link", config: { tableId: "users" }},
        { id: "due_date", name: "Due Date", type: "date" },
        { id: "priority", name: "Priority", type: "select", config: { options: ["Low", "Medium", "High"] }}
      ]
    }
  ],
  views: [
    // Kanban View
    {
      id: "kanban",
      name: "Kanban Board",
      tableId: "tasks",
      groupings: [{ field: "status" }],
      fields: ["title", "assignee", "due_date"]
    },

    // Calendar View
    {
      id: "calendar",
      name: "Calendar",
      tableId: "tasks",
      filters: [{ field: "due_date", operator: "not_empty" }],
      fields: ["title", "status", "assignee"]
    },

    // List View with Filters
    {
      id: "my-tasks",
      name: "My Tasks",
      tableId: "tasks",
      filters: [
        { field: "assignee", operator: "eq", value: "current_user" },
        { field: "status", operator: "neq", value: "Done" }
      ],
      sorts: [{ field: "due_date", direction: "asc" }],
      fields: ["title", "status", "due_date", "priority"]
    }
  ]
};

Links: Data Relationships

Unlike traditional foreign keys, Teable links are bi-directional and dynamic:

// Link field configuration
interface LinkFieldConfig {
  tableId: string;           // Target table
  viewIdForLinking?: string; // Which view to show when linking
  isMultiple: boolean;       // One-to-many or many-to-many
  symmetrical: boolean;      // Create reverse link automatically
}

// Example: Project-Task relationship
const projectTable = {
  id: "projects",
  name: "Projects",
  fields: [
    { id: "name", name: "Name", type: "text" },
    { id: "tasks", name: "Tasks", type: "link", config: {
      tableId: "tasks",
      isMultiple: true,
      symmetrical: true  // Creates "Project" field in tasks table
    }}
  ]
};

const taskTable = {
  id: "tasks",
  name: "Tasks",
  fields: [
    { id: "title", name: "Title", type: "text" },
    // Automatically created by symmetrical link
    { id: "project", name: "Project", type: "link", config: {
      tableId: "projects",
      isMultiple: false,
      symmetrical: false
    }}
  ]
};

⚡ Real-Time Collaboration Engine

Operational Transforms for Consistency

Teable uses operational transforms to handle concurrent edits:

// Operation types
interface Operation {
  id: string;
  type: 'insert' | 'update' | 'delete';
  tableId: string;
  recordId: string;
  fieldId?: string;
  oldValue?: any;
  newValue?: any;
  timestamp: number;
  userId: string;
  sessionId: string;
}

// Conflict resolution
class OperationalTransform {
  transform(operation: Operation, concurrentOps: Operation[]): Operation {
    let transformedOp = { ...operation };

    for (const concurrentOp of concurrentOps) {
      if (this.conflicts(operation, concurrentOp)) {
        transformedOp = this.resolveConflict(transformedOp, concurrentOp);
      }
    }

    return transformedOp;
  }

  private conflicts(op1: Operation, op2: Operation): boolean {
    return op1.tableId === op2.tableId &&
           op1.recordId === op2.recordId &&
           op1.fieldId === op2.fieldId &&
           op1.timestamp !== op2.timestamp;
  }

  private resolveConflict(op1: Operation, op2: Operation): Operation {
    // Last-write-wins for simple conflicts
    if (op1.timestamp > op2.timestamp) {
      return op1;
    }

    // For complex conflicts, create merge operations
    return this.createMergeOperation(op1, op2);
  }
}

WebSocket Communication

Real-time updates use WebSocket connections with room-based messaging:

// WebSocket room management
class CollaborationManager {
  private rooms: Map<string, Set<WebSocket>> = new Map();
  private userSessions: Map<string, UserSession> = new Map();

  joinRoom(tableId: string, userId: string, ws: WebSocket): void {
    if (!this.rooms.has(tableId)) {
      this.rooms.set(tableId, new Set());
    }

    this.rooms.get(tableId)!.add(ws);
    this.userSessions.set(userId, { tableId, ws, lastActivity: Date.now() });
  }

  leaveRoom(tableId: string, ws: WebSocket): void {
    const room = this.rooms.get(tableId);
    if (room) {
      room.delete(ws);
      if (room.size === 0) {
        this.rooms.delete(tableId);
      }
    }
  }

  broadcastToRoom(tableId: string, message: CollaborationMessage, excludeWs?: WebSocket): void {
    const room = this.rooms.get(tableId);
    if (!room) return;

    const messageStr = JSON.stringify(message);

    for (const ws of room) {
      if (ws !== excludeWs && ws.readyState === WebSocket.OPEN) {
        ws.send(messageStr);
      }
    }
  }

  handleOperation(operation: Operation): void {
    // Process the operation
    const result = this.processOperation(operation);

    // Broadcast to room
    this.broadcastToRoom(operation.tableId, {
      type: 'operation',
      operation: result,
      timestamp: Date.now()
    });
  }
}

interface CollaborationMessage {
  type: 'operation' | 'presence' | 'cursor';
  operation?: Operation;
  user?: UserPresence;
  cursor?: CursorPosition;
  timestamp: number;
}

interface UserPresence {
  userId: string;
  name: string;
  avatar?: string;
  status: 'online' | 'away' | 'offline';
  lastSeen: number;
}

🗃️ High-Performance Database Layer

PostgreSQL with Extensions

Teable uses PostgreSQL with specialized extensions for performance:

-- Enable required extensions
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE EXTENSION IF NOT EXISTS "pg_trgm";  -- For text search
CREATE EXTENSION IF NOT EXISTS "btree_gin"; -- For indexing

-- Optimized table structure
CREATE TABLE records (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    table_id UUID NOT NULL REFERENCES tables(id),
    data JSONB NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    created_by UUID REFERENCES users(id),
    updated_by UUID REFERENCES users(id)
);

-- Indexes for performance
CREATE INDEX idx_records_table_id ON records(table_id);
CREATE INDEX idx_records_data_gin ON records USING GIN (data);
CREATE INDEX idx_records_created_at ON records(created_at DESC);
CREATE INDEX idx_records_updated_at ON records(updated_at DESC);

-- Partial indexes for common queries
CREATE INDEX idx_records_active ON records(table_id) WHERE data->>'status' != 'archived';

-- Full-text search index
CREATE INDEX idx_records_search ON records USING GIN (to_tsvector('english', data::text));

Query Optimization Strategies

// Query builder with optimization
class OptimizedQueryBuilder {
  private query: QuerySpec;
  private hints: QueryHints;

  buildQuery(tableId: string, filters: Filter[], sorts: Sort[], pagination: Pagination): QuerySpec {
    this.query = {
      tableId,
      select: this.optimizeSelect(fields),
      where: this.optimizeWhere(filters),
      orderBy: this.optimizeOrderBy(sorts),
      limit: pagination.limit,
      offset: pagination.offset
    };

    this.hints = this.generateQueryHints(this.query);

    return this.query;
  }

  private optimizeSelect(fields: string[]): string[] {
    // Only select needed fields to reduce data transfer
    if (fields.length === 0) {
      return ['id', 'data', 'updated_at']; // Minimum required
    }

    // Add essential fields if missing
    const essential = ['id', 'updated_at'];
    return [...new Set([...fields, ...essential])];
  }

  private optimizeWhere(filters: Filter[]): WhereClause {
    // Combine filters efficiently
    const indexedFilters = filters.filter(f => this.isIndexed(f.field));
    const nonIndexedFilters = filters.filter(f => !this.isIndexed(f.field));

    return {
      indexed: this.combineIndexedFilters(indexedFilters),
      nonIndexed: nonIndexedFilters
    };
  }

  private optimizeOrderBy(sorts: Sort[]): OrderByClause[] {
    // Prefer indexed sorts
    return sorts.map(sort => ({
      field: sort.field,
      direction: sort.direction,
      indexed: this.isIndexed(sort.field)
    }));
  }

  private generateQueryHints(query: QuerySpec): QueryHints {
    const hints: QueryHints = {};

    // Index usage hints
    if (query.where.indexed.length > 0) {
      hints.useIndex = true;
    }

    // Pagination optimization
    if (query.offset > 1000) {
      hints.useCursor = true; // For large offsets
    }

    // Caching hints
    if (this.isCacheable(query)) {
      hints.cacheable = true;
      hints.ttl = this.calculateCacheTTL(query);
    }

    return hints;
  }

  private isIndexed(field: string): boolean {
    // Check if field has an index
    const indexedFields = ['id', 'created_at', 'updated_at', 'table_id'];
    return indexedFields.includes(field) || field.startsWith('data.');
  }

  private isCacheable(query: QuerySpec): boolean {
    // Static queries without user-specific filters can be cached
    return query.where.nonIndexed.length === 0 &&
           !query.where.indexed.some(f => f.operator === 'current_user');
  }

  private calculateCacheTTL(query: QuerySpec): number {
    // Shorter TTL for frequently changing data
    if (query.tableId.includes('activity') || query.tableId.includes('log')) {
      return 60; // 1 minute
    }

    return 300; // 5 minutes
  }
}

🌐 API Layer Architecture

REST and GraphQL Dual Support

Teable provides both REST and GraphQL APIs:

// REST API endpoints
const restApiRoutes = {
  // Records
  'GET /api/tables/:tableId/records': getRecords,
  'POST /api/tables/:tableId/records': createRecord,
  'GET /api/tables/:tableId/records/:recordId': getRecord,
  'PUT /api/tables/:tableId/records/:recordId': updateRecord,
  'DELETE /api/tables/:tableId/records/:recordId': deleteRecord,

  // Views
  'GET /api/views/:viewId/records': getViewRecords,
  'POST /api/views/:viewId/records': createRecordViaView,

  // Bulk operations
  'POST /api/tables/:tableId/records/bulk': bulkCreateRecords,
  'PUT /api/tables/:tableId/records/bulk': bulkUpdateRecords,
  'DELETE /api/tables/:tableId/records/bulk': bulkDeleteRecords
};

// GraphQL schema
const graphQLSchema = `
  type Record {
    id: ID!
    fields: JSON!
    createdAt: DateTime!
    updatedAt: DateTime!
    createdBy: User
    updatedBy: User
  }

  type Table {
    id: ID!
    name: String!
    fields: [Field!]!
    records(limit: Int, offset: Int, filter: JSON): [Record!]!
    record(id: ID!): Record
  }

  type Query {
    table(id: ID!): Table
    tables: [Table!]!
  }

  type Mutation {
    createRecord(tableId: ID!, fields: JSON!): Record!
    updateRecord(tableId: ID!, recordId: ID!, fields: JSON!): Record!
    deleteRecord(tableId: ID!, recordId: ID!): Boolean!
  }
`;

Authentication and Authorization

// Authentication middleware
class AuthenticationMiddleware {
  async authenticate(request: Request): Promise<User | null> {
    const token = this.extractToken(request);

    if (!token) {
      return null;
    }

    try {
      const payload = await this.verifyToken(token);
      return await this.getUserById(payload.userId);
    } catch (error) {
      throw new AuthenticationError('Invalid token');
    }
  }

  private extractToken(request: Request): string | null {
    // Check Authorization header
    const authHeader = request.headers.get('authorization');
    if (authHeader?.startsWith('Bearer ')) {
      return authHeader.substring(7);
    }

    // Check cookie
    return request.cookies.get('auth-token') || null;
  }

  private async verifyToken(token: string): Promise<JWTPayload> {
    // Verify JWT token
    return jwt.verify(token, process.env.JWT_SECRET!);
  }
}

// Authorization system
class AuthorizationSystem {
  private permissions: PermissionMap;

  constructor() {
    this.permissions = this.loadPermissions();
  }

  async authorize(user: User, resource: Resource, action: Action): Promise<boolean> {
    // Check base permissions
    const basePermission = this.checkBasePermission(user.role, resource.type, action);
    if (!basePermission) {
      return false;
    }

    // Check resource-specific permissions
    if (resource.type === 'table') {
      return this.checkTablePermission(user, resource.id, action);
    }

    if (resource.type === 'record') {
      return this.checkRecordPermission(user, resource.id, action);
    }

    return true;
  }

  private checkTablePermission(user: User, tableId: string, action: Action): Promise<boolean> {
    // Check if user has access to the table
    const tableAccess = await this.getTableAccess(user.id, tableId);

    switch (action) {
      case 'read':
        return tableAccess.read;
      case 'write':
        return tableAccess.write;
      case 'delete':
        return tableAccess.delete;
      case 'admin':
        return tableAccess.admin;
      default:
        return false;
    }
  }

  private checkRecordPermission(user: User, recordId: string, action: Action): Promise<boolean> {
    // Check record ownership and sharing permissions
    const record = await this.getRecord(recordId);

    // Owner has full access
    if (record.createdBy === user.id) {
      return true;
    }

    // Check sharing permissions
    const sharing = await this.getRecordSharing(recordId, user.id);
    return this.checkSharingPermission(sharing, action);
  }
}

interface Resource {
  type: 'table' | 'record' | 'view' | 'automation';
  id: string;
  tableId?: string; // For records
}

type Action = 'create' | 'read' | 'update' | 'delete' | 'admin';

🎨 Frontend Architecture

Dynamic UI Generation

Teable generates UI components based on field types:

// Field renderer system
class FieldRendererFactory {
  private renderers: Map<FieldType, FieldRenderer> = new Map();

  constructor() {
    this.registerRenderers();
  }

  private registerRenderers(): void {
    this.renderers.set('text', new TextFieldRenderer());
    this.renderers.set('number', new NumberFieldRenderer());
    this.renderers.set('date', new DateFieldRenderer());
    this.renderers.set('select', new SelectFieldRenderer());
    this.renderers.set('link', new LinkFieldRenderer());
    this.renderers.set('attachment', new AttachmentFieldRenderer());
  }

  getRenderer(fieldType: FieldType): FieldRenderer {
    const renderer = this.renderers.get(fieldType);
    if (!renderer) {
      throw new Error(`No renderer found for field type: ${fieldType}`);
    }
    return renderer;
  }

  renderField(field: TeableField, value: any, onChange: (value: any) => void): React.ReactNode {
    const renderer = this.getRenderer(field.type);
    return renderer.render(field, value, onChange);
  }
}

// Example field renderer
class LinkFieldRenderer implements FieldRenderer {
  render(field: TeableField, value: any, onChange: (value: any) => void): React.ReactNode {
    const config = field.config as LinkFieldConfig;

    return (
      <LinkField
        value={value}
        tableId={config.tableId}
        isMultiple={config.isMultiple}
        onChange={onChange}
        searchEndpoint={`/api/tables/${config.tableId}/records`}
        linkEndpoint={`/api/tables/${field.tableId}/records/${field.recordId}/links/${field.id}`}
      />
    );
  }
}

// Link field component
const LinkField: React.FC<LinkFieldProps> = ({
  value,
  tableId,
  isMultiple,
  onChange,
  searchEndpoint,
  linkEndpoint
}) => {
  const [searchTerm, setSearchTerm] = useState('');
  const [searchResults, setSearchResults] = useState<Record[]>([]);
  const [isSearching, setIsSearching] = useState(false);

  const searchRecords = useCallback(async (term: string) => {
    if (term.length < 2) return;

    setIsSearching(true);
    try {
      const response = await fetch(`${searchEndpoint}?search=${encodeURIComponent(term)}`);
      const results = await response.json();
      setSearchResults(results.records || []);
    } catch (error) {
      console.error('Search failed:', error);
    } finally {
      setIsSearching(false);
    }
  }, [searchEndpoint]);

  // Debounced search
  useEffect(() => {
    const timer = setTimeout(() => searchRecords(searchTerm), 300);
    return () => clearTimeout(timer);
  }, [searchTerm, searchRecords]);

  // ... rest of component implementation
};

🚀 Getting Started with Teable

Quick Start Options

Option 1: Teable Cloud (Recommended for Beginners)

# Visit https://teable.io and create a free account
# Start building bases and tables immediately

Option 2: Local Development

# Clone the repository
git clone https://github.com/teableio/teable.git
cd teable

# Install dependencies
pnpm install

# Set up environment
cp .env.example .env.local
# Configure database and other settings

# Start development server
pnpm dev

Option 3: Docker Deployment

# Run with Docker
docker run -p 3000:3000 \
  -e DATABASE_URL=postgresql://user:pass@localhost:5432/teable \
  -e NEXTAUTH_SECRET=your-secret \
  teable/teable

📊 Key Differentiators

Multi-Dimensional Views

Teable's view system provides spreadsheet-like flexibility with database performance:

  • Grid View: Traditional spreadsheet interface
  • Kanban View: Card-based project management
  • Calendar View: Time-based data visualization
  • Gallery View: Image-rich card layouts
  • Timeline View: Gantt chart functionality
  • Form View: Data entry forms
  • API View: Raw data access

Real-Time Everything

Unlike traditional databases, Teable makes real-time collaboration a core feature:

  • Live Updates: Changes appear instantly for all users
  • Presence Indicators: See who's viewing/editing what
  • Conflict Resolution: Automatic merging of simultaneous edits
  • Offline Support: Continue working without internet
  • Sync on Reconnect: Seamless offline-to-online transition

API-First Design

Teable generates comprehensive APIs automatically:

  • REST API: Full CRUD operations for all tables
  • GraphQL API: Flexible queries with relationships
  • Webhook Support: Real-time notifications for changes
  • SDK Generation: Client libraries in multiple languages
  • Third-Party Integrations: Zapier, Make, and custom connectors

🎯 Learning Path Forward

This chapter provided the foundation for understanding Teable's multi-dimensional approach. In the following chapters, we'll dive deeper into:

  • Chapter 2: Database Architecture - PostgreSQL optimization, indexing strategies, and query performance
  • Chapter 3: Real-Time Collaboration (Planned) - WebSocket implementation, operational transforms, and conflict resolution
  • Chapter 4: API Development (Planned) - REST/GraphQL APIs, authentication, and data validation

💡 Key Takeaways

  1. Multi-dimensional data: Views provide different perspectives on the same data without duplication
  2. Real-time collaboration: Operational transforms enable seamless concurrent editing
  3. Performance optimization: PostgreSQL with specialized extensions for complex queries
  4. API generation: Automatic REST and GraphQL APIs with authentication
  5. Dynamic UI: React-based interface generation from database schemas

🧪 Hands-On Exercise

Estimated Time: 45 minutes

  1. Set Up Teable: Choose one of the deployment options above and get Teable running
  2. Create a Base: Set up a new database with multiple tables and relationships
  3. Experiment with Views: Create different views (grid, kanban, calendar) for the same data
  4. Try Collaboration: Open the same base in multiple browser tabs and make simultaneous edits
  5. Explore the API: Use the automatically generated API to query and modify data

Ready to explore the database architecture? Continue to Chapter 2: Database Architecture

What Problem Does This Solve?

Most teams struggle here because the hard part is not writing more code, but deciding clear boundaries for tableId, field, name so behavior stays predictable as complexity grows.

In practical terms, this chapter helps you avoid three common failures:

  • coupling core logic too tightly to one implementation path
  • missing the handoff boundaries between setup, execution, and validation
  • shipping changes without clear rollback or observability strategy

After working through this chapter, you should be able to reason about Chapter 1: System Overview as an operating subsystem inside Teable: Deep Dive Tutorial, with explicit contracts for inputs, state transitions, and outputs.

Use the implementation notes around records, fields, tables as your checklist when adapting these patterns to your own repository.

How it Works Under the Hood

Under the hood, Chapter 1: System Overview usually follows a repeatable control path:

  1. Context bootstrap: initialize runtime config and prerequisites for tableId.
  2. Input normalization: shape incoming data so field receives stable contracts.
  3. Core execution: run the main logic branch and propagate intermediate state through name.
  4. Policy and safety checks: enforce limits, auth scopes, and failure boundaries.
  5. Output composition: return canonical result payloads for downstream consumers.
  6. Operational telemetry: emit logs/metrics needed for debugging and performance tuning.

When debugging, walk this sequence in order and confirm each stage has explicit success/failure conditions.

Source Walkthrough

Use the following upstream sources to verify implementation details while reading this chapter:

  • Teable Why it matters: authoritative reference on Teable (github.com).

Suggested trace strategy:

  • search upstream code for tableId and field to map concrete implementation paths
  • compare docs claims against actual runtime/config code before reusing patterns in production

Chapter Connections