feat: Enhance production readiness and scalability

CONFIGURATION.md

@@ -0,0 +1,69 @@
+# Configuration Management in FastFlight
+
+FastFlight utilizes a robust configuration system based on [Pydantic Settings](https://docs.pydantic.dev/latest/usage/settings/). This allows for type-validated settings loaded from various sources with clear precedence.
+
+## Overview
+
+Configuration for different components of FastFlight (Logging, Flight Server, FastAPI Application, Connection Bouncer) is managed through distinct Pydantic `BaseSettings` models. These models define the expected configuration parameters, their types, and default values.
+
+## Loading Mechanism
+
+Settings are loaded with the following order of precedence (highest to lowest):
+
+1.  **CLI Arguments:** Command-line arguments provided to `fastflight` CLI commands (e.g., `--port` for `start-fastapi`) will override any other source for the specific parameters they control.
+2.  **Environment Variables:** Settings can be provided as environment variables. Each setting group has a specific prefix.
+3.  **`.env` File:** If a `.env` file is present in the working directory when the application starts and the `python-dotenv` package is installed, environment variables will be loaded from this file. These are then treated as regular environment variables.
+4.  **Default Values:** If a setting is not found in any of the above sources, the default value defined in the Pydantic model is used.
+
+## Setting Groups
+
+### 1. Logging Settings
+
+Controls the application-wide logging behavior.
+**Environment Variable Prefix:** `FASTFLIGHT_LOGGING_`
+
+| Variable Suffix | Description                       | Type   | Default Value | Example Value      |
+| :-------------- | :-------------------------------- | :----- | :------------ | :----------------- |
+| `LOG_LEVEL`     | Minimum logging level to output.  | `str`  | `"INFO"`      | `"DEBUG"`, `"WARN"` |
+| `LOG_FORMAT`    | Log output format.                | `str`  | `"plain"`     | `"json"`           |
+
+### 2. Flight Server Settings
+
+Controls the behavior of the Arrow Flight server.
+**Environment Variable Prefix:** `FASTFLIGHT_SERVER_`
+
+| Variable Suffix        | Description                                  | Type   | Default Value | Example Value                  |
+| :--------------------- | :------------------------------------------- | :----- | :------------ | :----------------------------- |
+| `HOST`                 | Host address to bind the server to.          | `str`  | `"0.0.0.0"`   | `"127.0.0.1"`                  |
+| `PORT`                 | Port to bind the server to.                  | `int`  | `8815`        | `9000`                         |
+| `LOG_LEVEL`            | Logging level specific to the Flight server. | `str`  | `"INFO"`      | `"DEBUG"`                      |
+| `AUTH_TOKEN`           | Enables token authentication if set.         | `str`  | `None`        | `"your-secret-token"`          |
+| `TLS_CERT_PATH`        | Path to the server's TLS certificate file.   | `str`  | `None`        | `"/path/to/server.crt"`        |
+| `TLS_KEY_PATH`         | Path to the server's TLS private key file.   | `str`  | `None`        | `"/path/to/server.key"`        |
+
+### 3. FastAPI Application Settings
+
+Controls the behavior of the FastAPI web application.
+**Environment Variable Prefix:** `FASTFLIGHT_API_`
+
+| Variable Suffix           | Description                                                        | Type        | Default Value           | Example Value                             |
+| :------------------------ | :----------------------------------------------------------------- | :---------- | :---------------------- | :---------------------------------------- |
+| `HOST`                    | Host address for Uvicorn to bind to.                               | `str`       | `"0.0.0.0"`             | `"127.0.0.1"`                             |
+| `PORT`                    | Port for Uvicorn to bind to.                                       | `int`       | `8000`                  | `8080`                                    |
+| `LOG_LEVEL`               | Logging level for Uvicorn and FastAPI app.                         | `str`       | `"INFO"`                | `"DEBUG"`                                 |
+| `FLIGHT_SERVER_LOCATION`  | URL for the FastAPI app to connect to the Flight server.           | `str`       | `"grpc://localhost:8815"` | `"grpc+tls://flight.example.com:443"`   |
+| `VALID_API_KEYS`          | Comma-separated list of valid API keys for client authentication.  | `list[str]` | `[]` (empty list)       | `"key1,key2,anotherkey"`                  |
+| `SSL_KEYFILE`             | Path to the SSL private key file for Uvicorn (HTTPS).              | `str`       | `None`                  | `"/path/to/api.key"`                      |
+| `SSL_CERTFILE`            | Path to the SSL certificate file for Uvicorn (HTTPS).              | `str`       | `None`                  | `"/path/to/api.crt"`                      |
+| `METRICS_ENABLED`         | Enable (`True`) or disable (`False`) the `/metrics` endpoint.      | `bool`      | `True`                  | `False` (or `"false"`, `"0"`)            |
+
+*Note on `VALID_API_KEYS`: An empty string for the environment variable `FASTFLIGHT_API_VALID_API_KEYS` will result in an empty list, effectively disabling API key checks if that's the desired policy (see `SECURITY.md`).*
+
+### 4. Bouncer Settings
+
+Controls the default behavior of the `FastFlightBouncer` (client-side Flight connection pool).
+**Environment Variable Prefix:** `FASTFLIGHT_BOUNCER_`
+
+| Variable Suffix | Description                               | Type  | Default Value | Example Value |
+| :-------------- | :---------------------------------------- | :---- | :------------ | :------------ |
+| `POOL_SIZE`     | Default number of connections in the pool. | `int` | `10`          | `20`          |

docker/Dockerfile

@@ -1,11 +1,98 @@
-FROM python:3.10-buster
+# ---- Builder Stage ----
+FROM python:3.10-slim-buster AS builder
 
-ENV PYTHONPATH="/app"
+# Set working directory for virtual environment
+WORKDIR /opt/venv
 
+# Create virtual environment
+RUN python -m venv .
+ENV PATH="/opt/venv/bin:$PATH" # Activate venv
+
+# Install dependencies
 COPY requirements.txt .
+# Ensure pip is up-to-date and install wheel for potentially compiled packages
+RUN pip install --no-cache-dir --upgrade pip wheel
 RUN pip install --no-cache-dir -r requirements.txt
 
+
+# ---- Final Stage ----
+FROM python:3.10-slim-buster
+
+# Set environment variables for Python
+ENV PYTHONDONTWRITEBYTECODE 1
+ENV PYTHONUNBUFFERED 1
+
+# Install curl for health checks and any other essential OS packages
+# Also update apt-get and clean up to keep image size down
+RUN apt-get update && \
+    apt-get install -y curl --no-install-recommends && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
+
+# Create a non-root user and group
+ARG APP_USER=fastflight_user
+ARG APP_GROUP=fastflight_user
+RUN groupadd -r ${APP_GROUP} && useradd --no-log-init -r -g ${APP_GROUP} ${APP_USER}
+
+# Copy the virtual environment from the builder stage
+COPY --from=builder /opt/venv /opt/venv
+
+# Set up working directory and copy application code
 WORKDIR /app
-COPY src /app
+# Copy the src directory and any other necessary files
+COPY --chown=${APP_USER}:${APP_GROUP} src /app/src
+
+# Set up environment for the application
+ENV PATH="/opt/venv/bin:$PATH" # Activate venv for the final stage
+ENV PYTHONPATH="/app" # Application source code is in /app/src, so /app is the root for imports
+
+# Switch to the non-root user
+USER ${APP_USER}
+
+# Expose default ports
+EXPOSE 8815 # Default for Flight Server (FASTFLIGHT_SERVER_PORT)
+EXPOSE 8000 # Default for FastAPI (FASTFLIGHT_API_PORT)
+
+# Health check for FastAPI (assumes FastAPI runs on port 8000)
+# The CMD should ideally run `start-all` or ensure FastAPI is running.
+# If only Flight server runs, this health check might fail or be irrelevant.
+# This assumes the default CMD will be changed to run both, or this Dockerfile is for FastAPI.
+# For now, adding it with the assumption that the CMD will eventually run the FastAPI server.
+# TODO: Make health check ports configurable via ARG or tied to ENV if possible, though HEALTHCHECK doesn't directly expand ENV.
+HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
+  CMD curl -f http://localhost:8000/fastflight/health || exit 1
+
+# Health check for Flight Server (simple TCP check on port 8815)
+# This assumes the Flight server runs on port 8815.
+# TODO: Similar to FastAPI, port should be configurable.
+# Using a Python snippet for TCP check as netcat might not be available.
+# This will be a separate HEALTHCHECK layer if both are running.
+# Docker combines HEALTHCHECKs if multiple are defined; the last one is used.
+# To check both, a wrapper script is typically needed.
+# For this iteration, let's assume we want to check one primary service determined by CMD.
+# If CMD is `python /app/src/fastflight/server.py`, then Flight server health check is more relevant.
+# If CMD is `python -m src.fastflight.cli start-all`, then both should be healthy.
+# A single script for combined health check is better.
+
+# For now, providing a combined health check script approach.
+# Create a health check script first.
+
+# Let's defer the complex combined health check script for a moment and set a primary CMD.
+# The subtask implies refining the Dockerfile for the *Flight server* as per original CMD.
+# However, the requirement also asks for FastAPI health check.
+# This suggests the image should ideally run both via `start-all`.
+
+# Default CMD - Changed to run both services using the CLI's start-all command.
+# This makes both health checks potentially relevant.
+CMD ["python", "-m", "src.fastflight.cli", "start-all"]
+
+# Given the CMD runs `start-all`, a single HEALTHCHECK instruction
+# should ideally verify both services. A wrapper script is best for this.
+# For now, I will add a health check script and use it.
+# This script will check FastAPI first, then Flight server.
+
+COPY --chown=${APP_USER}:${APP_GROUP} docker/healthcheck.sh /app/healthcheck.sh
+RUN chmod +x /app/healthcheck.sh
 
-CMD ["python", "/app/fastflight/flight_server.py"]
+HEALTHCHECK --interval=30s --timeout=10s --start-period=15s --retries=3 \
+  CMD /app/healthcheck.sh

docker/healthcheck.sh

@@ -0,0 +1,30 @@
+#!/bin/sh
+set -eo pipefail
+
+# Default ports, can be overridden by environment variables if needed by the script
+FASTAPI_PORT="${FASTFLIGHT_API_PORT:-8000}"
+FLIGHT_PORT="${FASTFLIGHT_SERVER_PORT:-8815}"
+FASTAPI_HOST="${FASTFLIGHT_API_HOST:-localhost}" # Healthcheck runs inside the container
+FLIGHT_HOST="${FASTFLIGHT_SERVER_HOST:-localhost}" # Healthcheck runs inside the container
+
+echo "Healthcheck: Checking FastAPI server at http://${FASTAPI_HOST}:${FASTAPI_PORT}/fastflight/health"
+if curl -fsS "http://${FASTAPI_HOST}:${FASTAPI_PORT}/fastflight/health" > /dev/null; then
+  echo "Healthcheck: FastAPI server is healthy."
+else
+  echo "Healthcheck: FastAPI server failed."
+  exit 1
+fi
+
+echo "Healthcheck: Checking Flight server TCP connection at ${FLIGHT_HOST}:${FLIGHT_PORT}"
+# Use python to do a simple TCP check for the Flight server
+# This avoids needing netcat or other tools not guaranteed in slim images.
+# The python in /opt/venv/bin should be available.
+if /opt/venv/bin/python -c "import socket; s = socket.socket(socket.AF_INET, socket.SOCK_STREAM); s.settimeout(5); s.connect(('${FLIGHT_HOST}', ${FLIGHT_PORT})); s.close()"; then
+  echo "Healthcheck: Flight server TCP connection successful."
+else
+  echo "Healthcheck: Flight server TCP connection failed."
+  exit 1
+fi
+
+echo "Healthcheck: All services healthy."
+exit 0