API Gateway for analytics service

[Yedidyar]

Summary by CodeRabbit

• New Features

  • Introduced a full analytics system with modular services for collecting, aggregating, and reporting page views, including API Gateway, Aggregator, Increments, and Analytics services.
  • Added Docker and Docker Compose configurations for easy local deployment, including PostgreSQL and RabbitMQ integration.
  • Implemented robust configuration management and logging across services.
  • Added schema validation, batching, partitioned message processing, and materialized view management for scalable analytics.

• Documentation

  • Added architecture diagrams and detailed documentation describing system components, flows, and design trade-offs.
  • Service-specific READMEs and usage instructions provided.

• Tests

  • Comprehensive integration and unit tests for user, increments, and analytics services, ensuring reliable CRUD operations, data validation, and end-to-end flows.

• Chores

  • Added development scripts, editor configuration for consistent formatting, and CI/CD workflow updates.

[coderabbitai]

Caution

Review failed

Failed to post review comments.

Walkthrough

This change introduces a comprehensive analytics system built on a microservices architecture. It adds new services—including API Gateway, Partitioner, Aggregator, Increments, Analytics, and The Wolf—each with their own handlers, repositories, and configuration. Supporting infrastructure is defined via Docker Compose, database migrations, and documentation. Extensive testing, logging, and validation are also implemented.

Changes

Files / Groups	Change Summary
.github/workflows/deploy.yml	Changed workflow trigger branch from main to feature/analytics.
.vscode/settings.json	Added Prettier as default formatter and enabled format-on-save for multiple file types.
analytics-project/Dockerfile	New Dockerfile for analytics-project using Node.js 24 Alpine, with build and run steps.
analytics-project/architecture-diagram.md, analytics-project/architecture.md, docs/analytics-project.md	Added architecture diagrams and documentation describing system design, flows, and trade-offs.
analytics-project/docker-compose.yml, analytics-project/docker-compose.postgres.yml	Introduced Docker Compose files defining services for Postgres, RabbitMQ, API Gateway, Partitioner, Aggregators, Increments, Analytics, and The Wolf, with networks and volumes.
analytics-project/flyway.conf, analytics-project/migrations/V1_0__scaffolding.sql	Added Flyway config and initial SQL migration for tables, indexes, and materialized views.
analytics-project/logger/index.ts	New logger module exporting a Winston logger with Logz.io and console transports.
analytics-project/config.ts, analytics-project/api-gateway/config.ts	New configuration modules exporting typed, immutable config objects for services.
analytics-project/aggregator-service/README.md, analytics-project/aggregator-service/index.ts, analytics-project/aggregator-service/services/aggregator.service.ts, analytics-project/aggregator-service/services/aggregator.service.test.ts	Added Aggregator Service: README, entrypoint, service class for consuming, batching, aggregating, and forwarding page view data from RabbitMQ, with tests for aggregation logic.
analytics-project/analytics-service/index.ts, analytics-project/analytics-service/handlers/index.ts, analytics-project/analytics-service/repositories/page-views.ts, analytics-project/analytics-service/repositories/pool.ts, analytics-project/analytics-service/schemas/page-views.ts, analytics-project/analytics-service/schemas/page-views.test.ts, analytics-project/analytics-service/integration.test.ts	Added Analytics Service: Fastify server, handlers, schema validation, repository for querying page views, and comprehensive tests for endpoints and validation.
analytics-project/api-gateway/index.ts, analytics-project/api-gateway/handlers/index.ts, analytics-project/api-gateway/handlers/single.handler.ts, analytics-project/api-gateway/handlers/multi.handler.ts, analytics-project/api-gateway/handlers/report.handler.ts, analytics-project/api-gateway/services/partitioner.service.ts	Added API Gateway: Fastify server, partitioner service for distributing messages to partitioned RabbitMQ queues, and handlers for single/multi page view events and report proxying.
analytics-project/increments-service/index.ts, analytics-project/increments-service/constants/index.ts, analytics-project/increments-service/handlers/index.ts, analytics-project/increments-service/repositories/increments.ts, analytics-project/increments-service/repositories/pool.ts, analytics-project/increments-service/services/increments.service.ts, analytics-project/increments-service/integration.test.ts, analytics-project/increments-service/test-setup.ts	Added Increments Service: Fastify server, constants, handlers for single/multi increments, repository for upserts, service for grouping and batching, and integration tests.
analytics-project/the-wolf/handlers/index.ts, analytics-project/the-wolf/repositories/pool.ts, analytics-project/the-wolf/repositories/users.ts	Added The Wolf service: user handlers, repository, and database connection pooling.
package.json	Added scripts for service lifecycle, Docker Compose, and dependencies for amqplib, zod, and vite.
vitest.config.ts	Added Vitest configuration for environment loading.
docs/.vitepress/config.mts, docs/index.md	Updated documentation sidebar and homepage with new service links and images.

Sequence Diagram(s)

sequenceDiagram
    participant Client
    participant API_Gateway
    participant Partitioner
    participant RabbitMQ
    participant Aggregator
    participant Increments_Service
    participant Analytics_Service
    participant The_Wolf
    participant Postgres

    %% Write Flow
    Client->>API_Gateway: POST /single or /multi (page views)
    API_Gateway->>Partitioner: publishWithKey(page view data)
    Partitioner->>RabbitMQ: Publish to partitioned queue
    Aggregator->>RabbitMQ: Consume batch from queue
    Aggregator->>Increments_Service: POST aggregated increments
    Increments_Service->>Postgres: Upsert page_views

    %% Read Flow
    Client->>API_Gateway: GET /report/:page
    API_Gateway->>Analytics_Service: Proxy GET /report/:page
    Analytics_Service->>Postgres: Query page_views_report (materialized view)
    Analytics_Service-->>API_Gateway: Return analytics data
    API_Gateway-->>Client: Return analytics data

    %% Materialization
    The_Wolf->>Postgres: Refresh materialized view (page_views_report)

Loading

Possibly related PRs

• refactor(increments-service): update routes and handlers for page views functionality #15: Refactors increments-service handlers and routes to support batch increments and changes the route prefix, overlapping with this PR's introduction of increments-service endpoints and batch increment logic.

Poem

🐇
A whisk of code, a hop, a leap—
New services wake from slumber deep.
With queues and batches, views abound,
Metrics and users all around.
Compose and test, the logs alight—
Analytics magic, day and night!
🥕

✨ Finishing Touches

• 📝 Generate Docstrings

---

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Explain this complex logic.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai explain this code block.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and explain its main purpose.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[Yedidyar]

@coderabbitai summmrize the pr

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

I'll analyze the PR to provide you with a comprehensive summary. Let me gather the necessary information about the changes.

---

🏁 Script executed:

#!/bin/bash
# Get PR information and recent commits
echo "=== PR Information ==="
gh pr view --json title,body,additions,deletions,changedFiles

echo -e "\n=== Recent Commits ==="
git log --oneline -10

echo -e "\n=== Files changed in this PR ==="
git diff --name-only HEAD~1..HEAD | head -20

echo -e "\n=== Overall project structure ==="
echo "Analytics Project Structure:"
find analytics-project -type f -name "*.ts" -o -name "*.js" -o -name "*.md" | head -20

echo -e "\n=== Package.json changes ==="
git diff HEAD~1..HEAD -- package.json

Length of output: 2002

---

Let me try a different approach to gather information about this PR.

---

🏁 Script executed:

#!/bin/bash
# Get current git status and branch info
echo "=== Git Status ==="
git status
git branch -a

echo -e "\n=== Package.json content ==="
cat package.json

echo -e "\n=== Key analytics project files ==="
echo "Main entry points:"
find analytics-project -name "index.ts" | xargs -I {} sh -c 'echo "=== {} ===" && head -10 "{}"'

echo -e "\n=== Architecture documentation ==="
if [ -f "analytics-project/architecture.md" ]; then
    echo "=== Architecture Overview ==="
    head -20 analytics-project/architecture.md
fi

echo -e "\n=== Docker and deployment files ==="
ls -la analytics-project/ | grep -E "(docker|Docker|yml|yaml)"

Length of output: 8751

---

Let me gather more details about the README and documentation to provide a complete summary.

---

🏁 Script executed:

#!/bin/bash
# Get more architecture and README details
echo "=== Main README ==="
if [ -f "README.md" ]; then
    cat README.md
fi

echo -e "\n=== Analytics Project Documentation ==="
if [ -f "docs/analytics-project.md" ]; then
    head -30 docs/analytics-project.md
fi

echo -e "\n=== Architecture Diagram ==="
if [ -f "analytics-project/architecture-diagram.md" ]; then
    cat analytics-project/architecture-diagram.md
fi

echo -e "\n=== Complete Architecture Documentation ==="
cat analytics-project/architecture.md

echo -e "\n=== Docker Compose Services ==="
echo "Main docker-compose.yml services:"
grep -A 2 "services:" analytics-project/docker-compose.yml
grep "image:" analytics-project/docker-compose.yml

echo -e "\n=== Database Migration ==="
if [ -f "analytics-project/migrations/V1_0__scaffolding.sql" ]; then
    echo "Database schema:"
    head -20 analytics-project/migrations/V1_0__scaffolding.sql
fi

Length of output: 6455

---

@Yedidyar Here's a comprehensive summary of this PR:

📊 Analytics Platform Implementation

This PR introduces a complete distributed analytics system to the backend-rebase repository, adding 45 files that implement a high-performance, scalable analytics platform.

🏗️ Architecture Overview

The system implements a microservices architecture with the following key components:

graph TB
    Client[Client/Browser] --> API[API Gateway]
    API -->|Write| PV0[page_views_0 Queue]
    API -->|Write| PV1[page_views_1 Queue]
    API -->|Read| Analytics[Analytics Service]
    PV0 --> Agg0[Aggregator Service 0]
    PV1 --> Agg1[Aggregator Service 1]
    Agg0 --> Inc[Increments Service]
    Agg1 --> Inc
    Inc --> DB[(PostgreSQL)]

Loading

🔧 Core Services

1. API Gateway - Entry point handling 3 routes:

   • GET /report/{page} - Analytics reporting
   • POST /page-views/single - Single page view tracking
   • POST /page-views/multi - Batch page view tracking

2. Analytics Service - Query processing and reporting

3. Increments Service - Page view increment operations

4. Aggregator Services - Batch processing (1000 messages/1 minute)

5. Partitioner Service - Load balancing across queues

6. The Wolf - User management service

🛠️ Technical Implementation

• Message Queuing: RabbitMQ with partitioned queues for horizontal scaling
• Database: PostgreSQL with materialized views for optimized reads
• Containerization: Full Docker setup with docker-compose.yml
• Testing: Comprehensive integration tests using Vitest
• Logging: Winston with Logz.io integration
• Schema Validation: Zod for type-safe API contracts

📈 Key Features

• High Throughput: Handles large-scale page view tracking
• Eventual Consistency: Prioritizes availability over strict consistency
• Horizontal Scaling: Partitioned architecture supports multiple instances
• Materialization Pattern: Optimized for both writes and reads
• Real-time Analytics: Sub-second query performance
• Production Ready: Includes migrations, monitoring, and deployment configs

📦 Dependencies Added

Key new dependencies include:

• fastify - High-performance web framework
• amqplib - RabbitMQ client
• pg - PostgreSQL client
• winston - Logging framework
• zod - Schema validation

🚀 Value Proposition

This analytics platform provides:

• Scalable architecture capable of handling millions of page views
• Real-time insights with optimized query performance
• Fault tolerance through message queuing and partitioning
• Developer experience with comprehensive documentation and testing

The implementation follows enterprise-grade patterns and is ready for production deployment with proper monitoring, logging, and scaling capabilities.