Sync rootcodelabs/Global-Classifier dev with buerokratt/Global-Classifier dev

.github/copilot-instructions.md

@@ -0,0 +1,127 @@
+# Global Classifier AI Coding Instructions
+
+## Code Style Requirements
+
+**CRITICAL: Never use emojis in any generated code, comments, log messages, or documentation. Use plain text only.**
+
+## Architecture Overview
+
+Global Classifier is a machine learning platform built on the **BYK Stack** - a microservices architecture with specialized DSL-based components:
+
+- **Ruuter**: API gateway handling REST endpoints via YAML DSL configurations (`DSL/Ruuter.public/`, `DSL/Ruuter.private/`)
+- **Resql**: Database abstraction layer using SQL files as endpoints (`DSL/Resql/global-classifier/`)
+- **Data Mapper**: Template engine for dynamic content generation (`DSL/DMapper/`)
+- **TIM**: Authentication and authorization service
+- **CronManager**: Scheduled task execution
+
+## Key Development Patterns
+
+### DSL-First API Development
+APIs are defined declaratively in YAML files, not traditional controllers:
+```yaml
+# DSL/Ruuter.private/global-classifier/POST/inference/deploy.yml
+declaration:
+  call: declare
+  method: post
+  accepts: json
+  allowlist:
+    body:
+      - field: modelId
+        type: string
+```
+
+### Database Operations via Resql
+Database interactions use `.sql` files as endpoints, not ORM models:
+```sql
+-- DSL/Resql/global-classifier/POST/insert-data-models.sql
+INSERT INTO public.data_models (model_name, deployment_env, base_models)
+VALUES (:modelName, :deploymentEnv, :baseModels::jsonb)
+```
+
+### Service Configuration in constants.ini
+All service URLs are centralized in `constants.ini` using `[#SERVICE_NAME]` placeholder syntax:
+```ini
+GLOBAL_CLASSIFIER_RUUTER_PRIVATE=http://ruuter-private:8088/global-classifier
+GLOBAL_CLASSIFIER_RESQL=http://resql:8082/global-classifier
+```
+
+## Development Workflows
+
+### Environment Setup
+```bash
+# Use uv package manager (mandatory)
+uv venv && uv sync
+source .venv/bin/activate
+
+# Build required BYK stack images first
+docker build -t ruuter . # in cloned Ruuter repo
+docker build -t resql . # in cloned Resql repo
+```
+
+### Testing Requirements
+x- **Linting**: All code must pass `ruff check .` and `ruff format .`
+
+### Branch Strategy
+1. **wip** → **testing** → **dev** (three-tier workflow)
+2. All PRs target `wip` branch first
+3. Automated validation in `testing` before promoting to `dev`
+
+## Component Structure
+
+### Python Services (`src/`)
+- `training/`: ML model training scripts
+- `inference/`: Model serving (prod/testing environments)  
+- `classifier-service/`: Node.js mock service for chat classification
+- `dataset_file_handler/`: Data processing utilities
+
+### Frontend (`GUI/`)
+- React + TypeScript with Vite
+- Radix UI components and TanStack Query
+- Multi-language support via `translations/`
+
+### Experiments (`experiments/`)
+- `base_model_training/`: BERT/RoBERTa/XLM model experiments
+- `ood_detection/`: Out-of-distribution detection research
+
+## Critical Integration Points
+
+### Authentication Flow
+Routes use `.guard` files for auth checks. Private routes require cookie-based authentication validated through TIM service.
+
+### Model Deployment Pipeline
+1. Create model metadata → Resql `insert-data-models.sql`
+2. Update dataset connections → `update-datasets-connected-models.sql`  
+3. Initiate training → Call `/datamodels/train` endpoint
+4. Environment progression: undeployed → testing → production
+
+### Configuration Management
+- Docker services defined in multiple compose files (dev, inference-cpu, inference-gpu)
+- Environment-specific configurations in `config.env` and `sidecar.env`
+- Service discovery via DNS names in docker network `bykstack`
+
+## Common Anti-Patterns to Avoid
+
+- **Don't** create traditional REST controllers - use Ruuter YAML DSL
+- **Don't** write raw SQL in application code - use Resql `.sql` files
+- **Don't** hardcode service URLs - reference `constants.ini` placeholders
+- **Don't** bypass authentication guards on private routes
+- **Don't** use pip/conda - project requires `uv` package manager
+
+## Quick Reference
+
+### Adding New API Endpoint
+1. Create YAML in `DSL/Ruuter.{public|private}/global-classifier/{METHOD}/`
+2. Add SQL queries in `DSL/Resql/global-classifier/{METHOD}/`
+3. Update service constants if calling external services
+4. Add authentication guard if private endpoint
+
+### Running Services Locally
+```bash
+docker-compose up -d  # Full stack
+docker-compose -f docker-compose-dev.yml up  # Development mode
+```
+
+### Model Training/Inference
+- Training: Triggered via `/datamodels/train` POST endpoint
+- Inference: Environment-specific deployments in `src/inference/{prod|testing}/`
+- Models support: BERT, RoBERTa, XLM base architectures

.github/workflows/ci-cd-development.yml

@@ -4,18 +4,47 @@ on:
   push:
     branches:
       - dev
+  pull_request:
+    branches:
+      - dev
 
 jobs:
+  lint:
+    name: Run ESLint
+    runs-on: ubuntu-latest
+    if: github.event_name == 'pull_request'
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v3
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v3
+        with:
+          node-version: '18'
+          cache: 'npm'
+          cache-dependency-path: 'GUI/package-lock.json'
+
+      - name: Install dependencies
+        working-directory: ./GUI
+        run: npm install --legacy-peer-deps
+
+      - name: Lint code
+        working-directory: ./GUI
+        run: npx eslint "src/**/*.{ts,tsx}" --quiet
+
   deploy:
     runs-on: [self-hosted, development]
-
+    if: github.event_name == 'push'
+
     steps:
       - name: Checkout code
         uses: actions/checkout@v3
 
       - name: Stop and remove existing containers
         run: |
           docker compose down
+
       - name: Build & up Docker container 
         run: |
           docker compose up --build -d

.gitignore

@@ -1,4 +1,5 @@
 tim-db
+global-classifier-db
 .vscode/
 .idea/
 # Node modules
@@ -33,12 +34,23 @@ logs/
 test/logs/
 **/logs/
 *.log.*
-*.zip
+
 
 *.bkp
 *.dtmp
 
 # Python cache files
 __pycache__/
 .ruff_cache/
-.pytest_cache/
+.pytest_cache/
+
+models
+data
+output_datasets/
+venv
+dataset_artifacts/
+venv
+/signed.py
+/minio_presigned_urls.txt
+minio_data/
+temp_models/

.llm-contexts/README.md

@@ -0,0 +1,5 @@
+## LLM CONTEXTS
+
+---
+
+This folder contains the necessary contexts for any LLM tool to understand the contexts about Ruuter, Resql, Data Mapper, so that any AI-based coding tools that developers use like Co-Pilot have necessary context to give useful recommendations

.llm-contexts/byk-stack-summary.md

@@ -0,0 +1,28 @@
+# Bürokratt Project Overview
+
+## Goal and Vision
+
+Bürokratt is an initiative by the Estonian Ministry of Economic Affairs and Communications (MKM), with technical implementation managed by the Information System Authority (RIA).
+The core vision is to create a seamless, unified, and user-friendly channel for citizens and businesses to access Estonia's approximately 3,000 public e-services (and potentially private sector services).
+It aims to function as an interoperable network of public and private sector AI solutions, acting as a virtual assistant accessible via text and voice, 24/7.
+The goal is to make interacting with the state radically easier, moving beyond simple chatbots to a system that understands user needs and proactively offers bundled services, potentially based on life events.
+
+## Development and Architecture (Based on Codebase & ADRs)
+
+- **Open Development:** Bürokratt follows an open development model, with all code, planning, and project management happening publicly on GitHub. This facilitates collaboration with numerous development partners.
+- **Modular Architecture:** The system is designed to be highly modular, with functionalities developed as independent, replaceable services communicating via APIs (ARCH-001).
+- **DSL-Based Development:** A key principle is Domain-Specific Language (DSL)-based development. Instead of traditional programming languages for many tasks, developers work with YAML (e.g., for Ruuter configurations), SQL files (for Resql), and Handlebars (for DataMapper) to define service logic and data transformations.
+
+### Key Components:
+
+- **Ruuter:** Acts as the central router and reverse proxy, handling service requests (POST, GET, etc.), conditional logic, and templated services. Mock services are often built first using Ruuter's capabilities.
+- **Resql:** Manages all PostgreSQL database interactions. Each query resides in a separate .sql file, exposed as a REST endpoint. ADRs emphasize strict standards for SQL: no UPDATE/DELETE, standardized formatting, no SELECT *, no cross-table joins (enforcing denormalization), query parameterization, and security declarations.
+- **DataMapper (DMapper):** Uses Handlebars templates to restructure JSON outputs.
+- **TIM (TARA Integration Module):** Facilitates TARA-based authentication and JWT management.
+- **CVI:** A fork of a front-end component library for reusable React GUI elements.
+- **PostgreSQL:** The relational database used. Schema management uses Liquibase with specific rules (SQL-only definitions tracked via XML, UUIDs for all objects, use of ENUMs and TEXT, mandatory indexing, defined schemas instead of public).
+- **Liquibase:** Used for database schema version control.
+- **OpenSearch:** Increasingly used, particularly for its Query DSL capabilities. Storage ADRs indicate OpenSearch is a reference for document-oriented storage for metadata/structured text, separate from blob storage (like S3) for raw files.
+- **Rasa:** Used for chatbot functionalities like rules and stories, though intent recognition might be replaced.
+- **Centops:** It's the central orchestrator for the entire BYK stack
+- **DMR:** Stands for distributed message rooms, a message orchestration system which can recieve messages from one Buerokratt chatbot and pass it to another