Skip to content

ROADMAP.md

Latest commit

 

History

History
372 lines (301 loc) · 25.3 KB

ROADMAP.md

File metadata and controls

372 lines (301 loc) · 25.3 KB

Roadmap

Last updated: March 2026

This document outlines where OpenFarm is today and where it's headed. If you'd like to contribute to any of these areas, check the Contributing Guide and look for issues labeled help wanted or good first issue.

Strategic Architecture

OpenFarm is built on a 3-layer architecture where each layer has a distinct strategic role:

┌─────────────────────────────────────────────────────────────────┐
│  Layer C — Delivery Surfaces                   (Distribution)  │
│  Map UI · Reports · API · Webhooks · MCP · Mobile scouting     │
├─────────────────────────────────────────────────────────────────┤
│  Layer B — Intelligence Engine                       (Moat)    │
│  Phenology · Anomaly detection · Stress signals · Yield        │
│  Risk models · Soil-derived insights · Explainability          │
├─────────────────────────────────────────────────────────────────┤
│  Layer A — Observation Infrastructure         (Data Gravity)   │
│  Satellite · Weather · Soil · Field boundaries · Sensors       │
└─────────────────────────────────────────────────────────────────┘

Layer A — Observation Infrastructure (Data Gravity)

Collects, standardizes, and stores raw signals about every field. Source-agnostic, reproducible, and extensible.

Domain Current State Next
Satellite Sentinel-2 (NDVI, EVI, SAVI, NDWI), STAC ingestion, COG storage, 24-month backfill Landsat, Planet, SAR (Sentinel-1), cloud masking, fusion
Weather Open-Meteo (ERA5 + forecast), GDD, ET₀, water balance, drought index Additional providers, irrigation scheduling inputs
Soil SoilGrids (global 250m) + POLARIS (US 30m), 10 properties × 6 depths, texture classification, AWC, risk scoring, Rosetta PTF hydraulic properties, crop suitability, sampling zones, carbon estimation Terrain layers (DEM), sensor calibration
Boundaries FTW deep learning model, interactive review, GeoJSON/KML import Multi-model ensemble, higher-res detection
Sensors — IoT soil sensors, weather stations, device plugin framework

Layer B — Intelligence Engine (Moat)

Transforms raw observations into explainable, agronomically meaningful insights. This is the strategic differentiator — multi-signal reasoning with confidence scores and input attribution.

Tier Capabilities Dependencies
Statistical analysis Anomaly detection (z-score, CUSUM), phenology tracking, drought/water stress composite scoring, soil×weather stress indicators Existing indices + weather + soil
Composite intelligence Crop suitability (4-pillar scoring, 68 profiles), nutrient risk classification, carbon sequestration estimation, soil-aware alerts, sampling zone recommendations Tier 1 outputs
Actionable recommendations Yield forecasting, harvest timing, fertilizer/irrigation advisory Tier 2 outputs
ML classification Crop type detection, tree detection, nutrient deficiency signatures GPU + fine-tuned models
Advanced modeling Climate impact, advanced carbon modeling, sustainability reporting All prior tiers

Every insight carries: confidence score, contributing signal breakdown ("NDVI drop + rainfall deficit + sandy soil"), and historical comparison ("This field behaved similarly in 2022").

Layer C — Delivery Surfaces (Distribution)

Ensures OpenFarm is a platform others can build on, not just a tool.

Surface Current State Next
Map UI MapLibre + PMTiles, layer toggles, ECharts time series, dark/light theme, scouting + sampling zone markers with popups Field comparison, zone visualization
Reports Share links with multi-index + weather + soil summary (incl. crop suitability, carbon) PDF export, scheduled reports, seasonal summaries
API REST (/v1), JWT + RBAC, pagination, org-scoped Versioned public API with API keys, webhooks
Scouting Geotagged observations, photo upload, weather snapshots Mobile-optimized, offline sync
Integrations — MCP server for AI agents, plugin system, machinery telemetry
Exports — CSV, GeoJSON, raster tiles, PDF

Current Status

OpenFarm Milestone 11 (Soil Intelligence & Decision Support) is complete. The platform delivers end-to-end satellite-powered crop intelligence with four vegetation indices (NDVI, EVI, SAVI, NDWI), ML-powered automatic field boundary detection, daily weather data with agricultural indices, automatic 24-month historical index backfill, soil profile intelligence with derived agronomic layers, and a full intelligence engine — crop suitability scoring (68 crops, 4-pillar model), nutrient risk classification, carbon sequestration estimation, soil×weather stress monitoring, intelligent sampling zones, and soil-aware alerts — all functional and deployed. The focus now shifts to anomaly detection, phenology tracking, and broader agricultural intelligence. See Future Ideas for what's next.

Milestone 0 — Foundation ✅

  • Monorepo structure (apps/web, services/api, services/tiler)
  • Docker Compose with all services (Postgres/PostGIS, Redis, MinIO, API, Celery, TiTiler, Web)
  • Database schema — all 13 tables with UUID PKs, PostGIS geometry, soft-delete
  • JWT auth (NextAuth Google OAuth → shared HS256 JWT bridge)
  • RBAC system (owner / admin / member / viewer)
  • MapLibre base map with PMTiles + 4 style options
  • Health checks across all services
  • Structured logging (structlog + pino)

Milestone 1 — Org, Farm & Field Management ✅

  • Org CRUD — create, rename, member management, invites, audit log
  • Farm CRUD — create, edit, soft-delete, list with pagination
  • Field CRUD — draw polygon on map, edit vertices, GeoJSON import
  • Auto area calculation (hectares, geodesic)
  • Dashboard with org stats and quick actions
  • i18n support (English + Spanish)
  • Dark/light theme, responsive sidebar
  • Landing page for unauthenticated users

Milestone 2 — NDVI Monitoring ✅

  • NDVI pipeline: STAC search → band download → NDVI computation → COG → MinIO
  • Zonal statistics (mean, median, min, max, stddev, p10, p90)
  • TiTiler tile serving with JWT auth
  • NDVI tile overlay on map with floating legend
  • Time-series chart (Apache ECharts) with percentile bands
  • Alert rules: ndvi_drop (15% drop) and ndvi_threshold (below 0.3)
  • Job progress tracking with 7-step sub-status

Milestone 3 — Alerts, Scouting & Sharing ✅

  • Alerts API + UI — list with severity badges, close/acknowledge, notification badges
  • Alerts page with field/farm/status filters
  • Scouting API + UI — create/list observations, pin on map, photo upload via presigned URLs
  • Scouting observations as interactive map markers
  • Share API + UI — create/revoke share links with expiry, copy URL
  • Public report page (/share/[token]) — map, NDVI snapshot, time-series chart, alerts, scouting notes

Milestone 4 — Polish, Security & QA ✅

  • Viewer role enforcement across all routers
  • Rate limiting (slowapi) — Redis-backed, per-endpoint limits
  • Pagination consistency — all list endpoints use PaginatedResponse envelope
  • Audit log UI in settings page — event icons, search, pagination
  • Frontend structured logging (pino)
  • Celery worker health check in Docker Compose
  • Backup script (deploy/backup.sh) — automated daily pg_dump, 7-day retention
  • MinIO bucket versioning + WAL archiving/PITR documentation
  • API unit/integration tests
  • Frontend component tests
  • E2E acceptance tests

Milestone 5 — Multi-Index Vegetation Analysis ✅

  • Index registry (INDEX_REGISTRY) — pluggable formula, bands, colormap, rescale, alert defaults
  • EVI pipeline: 2.5 × (B08−B04) / (B08 + 6×B04 − 7.5×B02 + 1)
  • SAVI pipeline: ((B08−B04)/(B08+B04+L)) × (1+L) with configurable L factor (0–1)
  • NDWI pipeline: (B03−B08)/(B03+B08) for water stress detection
  • Per-index colormaps (rdylgn for NDVI/EVI/SAVI, rdbu for NDWI)
  • Per-index alert rules with configurable thresholds and drop percentages
  • Generalized job endpoint (POST /fields/{id}/jobs/index) + backward-compatible NDVI alias
  • Index selector UI on field detail page with floating map toggle
  • Multi-index job submission from single form
  • Per-index chart, legend, layer list, and map overlay
  • Share page with index toggle for multi-index reports
  • Full i18n translations (English + Spanish) for all index UI strings

Milestone 6 — Security Hardening & Code Quality ✅

  • Non-root containers (API + Tiler Dockerfiles)
  • Upload content-type whitelist (JPEG, PNG, WebP only)
  • JWT-in-tile-URL security trade-off documented in SECURITY.md
  • httpx client lifecycle via FastAPI lifespan
  • Missing FK indexes added (alerts, field_stats, jobs, scouting)
  • Shapely geometry validation with descriptive error messages
  • Atomic farm soft-delete cascade
  • Pagination offset cap (100,000)
  • Rate limits on invitation/member management endpoints
  • Shared wkb_to_geojson() utility to reduce code duplication
  • Index task map derived from registry (single source of truth)
  • SAVI L factor stored in layer params_json and displayed in UI
  • In-app changelog page with parsed Keep a Changelog rendering

Milestone 7 — Automatic Boundary Detection ✅

  • FTW (Fields of The World) deep learning model integration for field boundary detection
  • Dedicated ML processor Docker service with PyTorch, torchgeo, and model weights
  • Detection API — trigger, list, accept, discard, with org-scoped access control
  • Full-page detection UI (/farms/[id]/detect) with draw → detecting → review workflow
  • Interactive polygon drawing with MapLibre GL Draw (draw, move, edit vertices, delete)
  • Viewport preservation across phase transitions
  • Boundary review with confidence scores, bulk accept/discard, zoom-to-boundary
  • Boundary geometry editing before accepting as a field
  • 7-step progress tracking (validate → STAC search → download → prepare → inference → polygonize → store)
  • Alembic migrations for detected boundaries, nullable job field_id, updated_at
  • i18n translations (English + Spanish) for all detection UI

Milestone 8 — Weather Data Integration ✅

  • weather_daily table with 18 raw variables + 5 derived indices + metadata
  • Open-Meteo API integration (free, CC BY 4.0) for forecast + historical archive (ERA5)
  • Celery task fetch_weather_for_field — centroid-based fetch, GDD/water balance/drought index computation, upsert
  • Celery task schedule_daily_weather_fetch — Beat schedule at 08:00 UTC, batched by 50
  • Celery task backfill_weather_for_field — 90-day historical fetch on demand
  • API: GET /fields/{id}/weather with date range + optional 7-day forecast
  • API: GET /fields/{id}/weather/summary — 30-day aggregated stats
  • API: POST /fields/{id}/weather/backfill — manual trigger (202 Accepted)
  • Alert enrichment — weather_context JSONB on alerts (precip 7d, ET₀, deficit, soil moisture, GDD, drought index)
  • Scouting weather snapshot — auto-attached 7-day weather context on observation creation
  • Share report — weather_summary + weather_data (90 days) in public report endpoint
  • Weather tab — summary cards, 7-day forecast bar, temp+precip chart, ET₀+water balance chart
  • Soil moisture gauge — 5-depth horizontal bars with color coding
  • Soil temperature display — 4-depth bars (surface, 6cm, 18cm, 54cm)
  • Vapor pressure deficit (VPD) — value with zone classification (low/ideal/moderate/high)
  • NDVI + weather overlay chart — dual-axis with precip bars + ET₀ line, toggle in Indices tab
  • Alembic migrations: weather_daily table (0008), weather_context on alerts (0009), weather_snapshot on scouting (0010)
  • Config settings: Open-Meteo URLs, backfill days, batch size, GDD base temp, heat stress threshold
  • i18n translations (English + Spanish) for all weather UI strings

Milestone 9 — Historical Data Backfill ✅

  • Backfill orchestrator task — splits 24-month range into 90-day chunks, dispatches one job per (chunk × index) with staggered countdowns
  • Chunk-level and index-level deduplication — skips chunks/indices where raster data already exists
  • Alert suppression — backfill pipeline runs do not create alerts (is_backfill flag in params_json)
  • Auto-trigger on field creation — new fields automatically get 24 months of all 4 vegetation indices
  • Weekly auto-compute Celery Beat schedule (schedule_weekly_index_compute) — Monday 06:00 UTC, skips fresh fields
  • Manual backfill API: POST /fields/{id}/backfill-indices (admin/owner, rate-limited 1/min)
  • Backfill status API: GET /fields/{id}/backfill-status — pending, running, completed job counts
  • Endpoint-level deduplication — 409 Conflict if backfill already in progress
  • Sentinel job pattern — synchronous placeholder job before async dispatch to prevent race conditions
  • Batch backfill task (backfill_all_existing_fields) for retroactive backfill of all org fields
  • Backfill History button on Indices tab with active-backfill detection and progress banner
  • Config settings: index_backfill_months (24), index_backfill_chunk_days (90)
  • i18n translations (English + Spanish) for all backfill UI strings

Milestone 10 — Soil Intelligence Foundation ✅

  • soil_profiles, soil_layers, soil_field_summary database tables with UUID PKs and PostGIS integration
  • SoilGrids WCS client — 10 properties × 6 depths × 3 quantiles (mean, Q05, Q95) globally at 250m
  • POLARIS S3 client — 9 properties × 6 depths for US fields at 30m resolution
  • Automatic source routing (POLARIS for CONUS, SoilGrids globally)
  • USDA soil texture triangle classification (12 classes)
  • Available Water Capacity (AWC) computation and rootzone integration
  • Field summary aggregation with risk scoring (acidification, compaction, leaching, rooting)
  • Data quality scoring based on uncertainty spread
  • Celery task fetch_soil_for_field with retry, backoff, and partial-data handling
  • Auto-trigger on field creation
  • API: GET /fields/{id}/soil, GET /fields/{id}/soil/summary, POST /fields/{id}/soil/refresh
  • Soil tab with texture-by-depth stacked bar chart, color-coded legend, and interactive tooltips
  • Property cards: pH, Organic Carbon, CEC, Bulk Density with color-coded thresholds
  • AWC gauge, drainage class badge, risk indicators, data quality display
  • Job progress tracking for soil refresh (6-step real-time polling)
  • "Regional Estimate" disclaimer banner
  • Alembic migration 0011_add_soil_tables
  • Config settings: SoilGrids WCS URL, POLARIS bucket, fetch timeout, source priority
  • i18n translations (English + Spanish) for all soil UI strings

Milestone 11 — Soil Intelligence & Decision Support ✅

  • Soil intelligence engine (core/soil_intelligence.py) — 2,200+ lines of agronomic logic
  • Rosetta PTF integration — pedotransfer functions for estimating field capacity, wilting point, saturated conductivity from texture + bulk density
  • Derived agronomic columns — field_capacity, wilting_point, saturated_conductivity, awc_derived on soil layers
  • Refined risk scoring — waterlogging risk from poor drainage + low Ksat, updated compaction/leaching formulas
  • SOC stock calculation — depth-integrated organic carbon stock (t/ha) in field summary
  • Crop suitability assessment — 4-pillar weighted scoring (Soil 40%, Water 25%, Climate 20%, Stress 15%) with 68 crop profiles
  • Crop profiles spanning cereals, legumes, oilseeds, industrial, root/tuber, vegetables, fruits, plantations, spices
  • Weather-aware suitability — integrates annual rainfall, temperature range, drought/flood indices from weather_daily table
  • API: GET /fields/{id}/soil/crop-suitability — top 10 crops with scores, ratings, limiting factors; field crop highlighted
  • Sampling zone recommendations — within-field variability analysis for clay, SOC, pH, AWC
  • API: GET /fields/{id}/soil/sampling-zones — GeoJSON FeatureCollection with prioritized sampling points
  • Nutrient risk classification — zones classified as nutrient_loss_risk, nutrient_retentive, or nutrient_responsive
  • API: GET /fields/{id}/soil/nutrient-context — zone classification with confidence and interpretation
  • Carbon sequestration estimation — Hassink (1997) clay-dependent model with climate adjustment
  • API: GET /fields/{id}/soil/carbon-estimate — current SOC stock, saturation deficit, sequestration potential
  • Soil × weather stress indicators — combines AWC with 30-day water balance, drought index, soil moisture
  • API: GET /fields/{id}/soil/weather-stress — drought/wet/optimal status with severity score
  • Soil-aware alerts — automatic alert generation for pH, SOC, compaction, waterlogging, sand, CEC thresholds
  • soil_context JSONB column on alerts table — stores soil property values at alert creation time
  • Soil summary in share reports — texture, pH, SOC, CEC, drainage in public share page
  • Sampling zone map markers (MapLibre) — bullseye-style with priority color coding (P1=red, P2=yellow, P3=blue)
  • Scouting observation map markers — circle style with popups showing observation details
  • Interactive map popups — dark-mode-aware popup styling with CSS variables for both marker types
  • Click-to-zoom on sampling zone sidebar cards — flyTo with zoom 17
  • Tab-based marker visibility toggling — markers only visible on their respective tabs
  • Soil tab intelligence panels — crop suitability, nutrient context, carbon estimation, weather stress, sampling zones
  • Crop suitability UI — top 10 ranked list with scores, ratings, limiting factors, weather-required banner
  • Carbon estimation UI — SOC gauge, saturation %, sequestration opportunity range with disclaimer
  • Alembic migrations: 0012_add_derived_agronomic_columns, 0013_add_soil_context_to_alerts
  • i18n translations (English + Spanish) for all intelligence UI strings (~100 keys)

Future Ideas (Post-MVP)

Organized by architecture layer and ordered by dependency. Items higher in each section are prerequisites for items below.

Layer A — Observation Infrastructure

Completed:

  • Historical data backfill — ✅ Milestone 9
  • Soil data integration — ✅ Milestone 10
  • Derived agronomic layers (Rosetta PTF) — ✅ Milestone 11

Satellite & Imagery:

  • Multi-satellite support — Landsat, Planet (currently Sentinel-2 only) (prerequisite for higher-frequency monitoring)
  • Higher-frequency monitoring — daily revisit satellites (PlanetScope) for near real-time crop monitoring
  • Higher-res imagery — sub-meter commercial imagery for detailed crop monitoring (prerequisite for tree canopy analysis)
  • Drone imagery support — ingest and process high-res drone imagery (depends on higher-res pipeline)
  • Cloud masking & gap handling — improved scene quality filtering and temporal fusion
  • Custom index builder — user-defined indices from available bands with formula editor

Field & Spatial:

  • Terrain layers — elevation, slope, aspect from SRTM/Copernicus DEM
  • Multi-model boundary detection — ensemble approach for improved accuracy

Sensors & External Data:

  • Device/Sensor plugin framework — connect soil sensors, weather stations, IoT devices
  • Machinery telemetry integration — GPS tracks and operational data from farm equipment (depends on plugin framework)

Layer B — Intelligence Engine

Each tier builds on the one above it.

Completed:

  • Drought and water stress monitoring — ✅ Milestone 11 (soil×weather stress indicators)
  • Crop suitability assessment — ✅ Milestone 11 (4-pillar scoring, 68 crop profiles)
  • Nutrient risk classification — ✅ Milestone 11
  • Carbon sequestration estimation — ✅ Milestone 11
  • Soil-aware alerts — ✅ Milestone 11
  • Sampling zone recommendations — ✅ Milestone 11

Tier 1 — Statistical analysis (CPU-only, builds on existing observations):

  • Anomaly detection — z-score, CUSUM, moving average deviation from historical baseline (historical backfill ✅)
  • Phenology tracking — crop growth stages from NDVI/EVI temporal curves (historical backfill ✅)

Tier 2 — Composite intelligence (builds on Tier 1):

  • Disease/pest risk signals — risk scoring combining vegetation anomalies + weather + regional pest data (depends on anomaly detection)
  • Soil moisture estimation — enhanced modeling combining remote sensing with in-situ sensor data

Tier 3 — Actionable recommendations (builds on Tier 2):

  • Yield forecasting — predict from historical trends, weather, soil, and field data (depends on phenology + anomaly detection)
  • Harvest timing — optimal windows from crop maturity models and vegetation indices (depends on phenology)
  • Fertilizer and irrigation advisory — insights from crop health, weather forecasts, and soil data (depends on disease risk + water stress)

Tier 4 — ML classification (requires GPU + fine-tuned models):

  • Crop type detection — multi-temporal spectral classification using foundation models (see research doc)
  • Tree detection and classification — tree crop identification and health monitoring (depends on crop classification)
  • Nutrient deficiency detection — spectral signatures of common deficiencies (depends on crop classification)
  • Tree canopy analysis — cover, LAI, height estimation (depends on higher-res imagery)

Tier 5 — Advanced modeling (builds on all tiers):

  • Climate impact modeling — carbon sequestration, emissions from field data + agronomic models (depends on yield + crop classification)
  • Carbon/sustainability reporting — track and report sustainability metrics (depends on climate impact)

Layer C — Delivery Surfaces

Reports & Exports:

  • Data export — CSV, GeoJSON, PDF formats for field data, stats, and reports
  • Seasonal summaries — automated end-of-season reports
  • Field comparison — side-by-side health comparison across fields
  • Historical analytics — season-over-season trend analysis (historical backfill ✅)
  • Advanced analytics dashboard — customizable dashboards, scheduled reports (depends on field comparison)

API & Integrations:

  • Versioned public API — stable API with API keys for third-party integration (prerequisite for MCP + webhooks)
  • Webhook/notification system — email, Slack, SMS on alerts (depends on public API)
  • Plugin system — extensible processing pipelines for custom analysis
  • Model Context Protocol (MCP) server — standardized interface for AI agents (depends on public API)
  • AI agent integration — LLM-powered natural language insights and conversational interface (depends on MCP)

Platform & Auth:

  • Enhanced RBAC — field-level permissions, read-only API keys, user groups
  • Email/Microsoft & Enterprise SSO — email/password, Microsoft OAuth, SAML/OIDC
  • Advanced workflows — rule-based automation (auto-trigger on new imagery, scheduled monitoring) (depends on webhooks)

Community:

  • Supply chain / traceability integrations — link field data to downstream logistics and compliance
  • Community data sharing — opt-in anonymized data for regional insights and benchmarking

Enterprise & Scale

  • Enterprise admin controls — SSO enforcement, audit policies, usage quotas, multi-tenant admin
  • Mobile app — React Native companion for field scouting
  • Custom analytics packs per vertical — tailored modules for tree crops, forestry, viticulture, etc. (depends on analytics framework)
  • Modular ERP — lightweight farm operations management (inventory, tasks, financials)
  • Hosted offering — managed cloud version _(depends on enterprise admin controls)

How to Contribute

The MVP is complete! The highest-impact contributions right now are:

  1. Automated tests — API integration tests, frontend component tests, and E2E acceptance tests (Milestone 4 remaining items)
  2. Future Ideas — pick any item from the list above or browse open issues

See CONTRIBUTING.md for setup and guidelines.