Skip to content

Commit eaac56b

Browse files
committed
Sprint 4: RBAC hardening, rate limiting, pagination fixes, backup docs
- Enforce viewer read-only across all write endpoints (require_roles) - Add slowapi rate limiting with Redis backend (120 req/min default) - Wrap layers/stats/members in PaginatedResponse envelope - Update frontend to consume paginated responses - Add backup script (deploy/backup.sh) with 7-day retention - Document MinIO versioning and WAL archiving in DEPLOYMENT.md - Reverse NDVI layer order (latest first) - Update ROADMAP.md: all MVP milestones complete (179/182 tasks)
1 parent ca5ecb3 commit eaac56b

20 files changed

Lines changed: 448 additions & 111 deletions

File tree

DEPLOYMENT.md

Lines changed: 129 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -179,12 +179,138 @@ sudo docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --bui
179179

180180
### Database Backup
181181

182+
OpenFarm includes an automated backup script at `deploy/backup.sh`.
183+
184+
**Manual backup:**
185+
182186
```bash
183-
# Create backup
187+
# Quick one-liner
184188
sudo docker compose exec db pg_dump -U openfarm openfarm | gzip > backup_$(date +%Y%m%d).sql.gz
185189

186-
# Restore backup
187-
gunzip -c backup_20260215.sql.gz | sudo docker compose exec -T db psql -U openfarm openfarm
190+
# Using the backup script (recommended)
191+
sudo /opt/openfarm/deploy/backup.sh
192+
```
193+
194+
**Automated daily backups (cron):**
195+
196+
```bash
197+
# Add to root crontab
198+
sudo crontab -e
199+
200+
# Daily at 02:00 UTC, 7-day retention (default)
201+
0 2 * * * /opt/openfarm/deploy/backup.sh >> /var/log/openfarm-backup.log 2>&1
202+
```
203+
204+
**Configuration (environment variables):**
205+
206+
| Variable | Default | Description |
207+
|----------|---------|-------------|
208+
| `BACKUP_DIR` | `/opt/openfarm/backups` | Local backup directory |
209+
| `RETENTION_DAYS` | `7` | Days to keep local backups |
210+
| `UPLOAD_TO_MINIO` | `false` | Upload backups to MinIO/S3 |
211+
| `MINIO_ALIAS` | `local` | mc alias for MinIO |
212+
| `MINIO_BUCKET` | `openfarm` | Target bucket |
213+
214+
**Restore from backup:**
215+
216+
```bash
217+
# From custom format (.dump) — recommended
218+
sudo docker compose exec -T db pg_restore -U openfarm -d openfarm --clean < backups/openfarm_20260215_020000.dump
219+
220+
# From SQL format (.sql.gz)
221+
gunzip -c backups/openfarm_20260215.sql.gz | sudo docker compose exec -T db psql -U openfarm openfarm
222+
```
223+
224+
### MinIO Bucket Versioning
225+
226+
Enable versioning to protect against accidental object deletion/overwrite (COG rasters, photos):
227+
228+
```bash
229+
# Install MinIO client (mc) if not present
230+
curl -sSL https://dl.min.io/client/mc/release/linux-arm64/mc -o /usr/local/bin/mc && chmod +x /usr/local/bin/mc
231+
232+
# Configure mc alias
233+
mc alias set local http://localhost:9000 openfarm openfarm_dev_secret
234+
235+
# Enable versioning on the openfarm bucket
236+
mc version enable local/openfarm
237+
238+
# Verify
239+
mc version info local/openfarm
240+
# Expected: local/openfarm versioning is enabled
241+
242+
# Optional: set lifecycle rule to expire old versions after 30 days
243+
mc ilm rule add local/openfarm --noncurrent-expire-days 30
244+
```
245+
246+
**What versioning protects:**
247+
- `cogs/{org}/{field}/{date}/ndvi.tif` — NDVI raster layers (re-processable but slow)
248+
- `photos/{org}/{uuid}.{ext}` — Scouting observation photos (not recoverable)
249+
- `basemap/` — PMTiles basemap (re-downloadable)
250+
251+
### WAL Archiving (Point-in-Time Recovery)
252+
253+
For production deployments requiring point-in-time recovery (PITR), enable PostgreSQL WAL archiving.
254+
255+
**1. Create archive directory:**
256+
257+
```bash
258+
sudo mkdir -p /opt/openfarm/wal-archive
259+
sudo chown 999:999 /opt/openfarm/wal-archive # postgres container UID
260+
```
261+
262+
**2. Add PostgreSQL config overrides** — create `deploy/postgresql.conf`:
263+
264+
```ini
265+
# WAL archiving for PITR
266+
wal_level = replica
267+
archive_mode = on
268+
archive_command = 'cp %p /var/lib/postgresql/wal-archive/%f'
269+
archive_timeout = 300
270+
```
271+
272+
**3. Mount in Docker Compose** — add to `docker-compose.prod.yml` db service:
273+
274+
```yaml
275+
db:
276+
volumes:
277+
- ./deploy/postgresql.conf:/etc/postgresql/conf.d/wal.conf:ro
278+
- /opt/openfarm/wal-archive:/var/lib/postgresql/wal-archive
279+
command: >
280+
postgres
281+
-c config_file=/etc/postgresql/postgresql.conf
282+
-c include_dir=/etc/postgresql/conf.d
283+
```
284+
285+
**4. Point-in-Time Recovery procedure:**
286+
287+
```bash
288+
# Stop the application
289+
sudo docker compose down
290+
291+
# Create base backup
292+
sudo docker compose exec db pg_basebackup -U openfarm -D /tmp/basebackup -Ft -z
293+
294+
# To restore to a specific time:
295+
# 1. Replace the data directory with the base backup
296+
# 2. Create recovery.signal file
297+
# 3. Set recovery_target_time in postgresql.conf:
298+
# recovery_target_time = '2026-02-15 14:30:00 UTC'
299+
# restore_command = 'cp /var/lib/postgresql/wal-archive/%f %p'
300+
# 4. Start PostgreSQL — it will replay WAL up to the target time
301+
302+
sudo docker compose up -d
303+
```
304+
305+
**WAL archive maintenance:**
306+
307+
```bash
308+
# Check archive size
309+
du -sh /opt/openfarm/wal-archive/
310+
311+
# Prune WAL files older than the oldest base backup (manual)
312+
# Keep at minimum 7 days of WAL for PITR window
313+
find /opt/openfarm/wal-archive/ -name "*.gz" -mtime +7 -delete
188314
```
189315

190316
### Restart a Service

ROADMAP.md

Lines changed: 30 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,9 @@ This document outlines where OpenFarm is today and where it's headed. If you'd l
88

99
## Current Status
1010

11-
OpenFarm is in **early alpha**. The core platform (auth, org management, farm/field CRUD, and NDVI monitoring pipeline) is functional end-to-end.
11+
OpenFarm **Phase 1 MVP is complete**. The platform delivers end-to-end satellite-powered crop intelligence: auth, org management, farm/field CRUD, NDVI monitoring pipeline, alerts, scouting observations, shareable field health reports, and production-grade security hardening — all functional and deployed.
12+
13+
**179 of 182 tasks complete (98%).** The only remaining items are automated testing (API, frontend, E2E).
1214

1315
---
1416

@@ -21,14 +23,13 @@ OpenFarm is in **early alpha**. The core platform (auth, org management, farm/fi
2123
- [x] RBAC system (`owner` / `admin` / `member` / `viewer`)
2224
- [x] MapLibre base map with PMTiles + 4 style options
2325
- [x] Health checks across all services
24-
- [x] Structured logging (structlog)
25-
- [x] CI pipeline (ESLint, TypeScript, ruff)
26+
- [x] Structured logging (structlog + pino)
2627

2728
## Milestone 1 — Org, Farm & Field Management ✅
2829

2930
- [x] Org CRUD — create, rename, member management, invites, audit log
3031
- [x] Farm CRUD — create, edit, soft-delete, list with pagination
31-
- [x] Field CRUD — draw polygon on map, edit vertices, GeoJSON/KML import
32+
- [x] Field CRUD — draw polygon on map, edit vertices, GeoJSON import
3233
- [x] Auto area calculation (hectares, geodesic)
3334
- [x] Dashboard with org stats and quick actions
3435
- [x] i18n support (English + Spanish)
@@ -40,33 +41,31 @@ OpenFarm is in **early alpha**. The core platform (auth, org management, farm/fi
4041
- [x] NDVI pipeline: STAC search → band download → NDVI computation → COG → MinIO
4142
- [x] Zonal statistics (mean, median, min, max, stddev, p10, p90)
4243
- [x] TiTiler tile serving with JWT auth
43-
- [x] NDVI tile overlay on map
44+
- [x] NDVI tile overlay on map with floating legend
4445
- [x] Time-series chart (Apache ECharts) with percentile bands
4546
- [x] Alert rules: `ndvi_drop` (15% drop) and `ndvi_threshold` (below 0.3)
4647
- [x] Job progress tracking with 7-step sub-status
4748

48-
## Milestone 3 — Alerts, Scouting & Sharing 🔧 In Progress
49-
50-
Backend APIs are complete. Frontend UI is the remaining work.
51-
52-
- [x] Alerts API (list, filter by field/farm/status, close/reopen)
53-
- [ ] **Alerts UI** — list alerts with severity badges, close/acknowledge actions
54-
- [ ] **Alerts on farm dashboard** — summary of active alerts across fields
55-
- [x] Scouting API (CRUD observations with geotagged points, photo upload via presigned URLs)
56-
- [ ] **Scouting UI** — create/list observations, pin on map, photo upload
57-
- [x] Share API (create/revoke share links with expiry, public report endpoint)
58-
- [ ] **Share UI** — create link, copy URL, manage active links
59-
- [ ] **Public report page** (`/share/[token]`) — map, NDVI snapshot, time-series chart, alerts, scouting notes
60-
61-
## Milestone 4 — Polish, Security & QA ⬜ Planned
62-
63-
- [ ] Viewer role enforcement (read-only for viewer members)
64-
- [ ] Audit log UI in settings page
65-
- [ ] Floating NDVI legend on map
66-
- [ ] Pagination consistency across all list endpoints
67-
- [ ] Frontend structured logging (pino)
68-
- [ ] Celery worker health check in Docker Compose
69-
- [ ] Backup/restore documentation (pg_dump, MinIO versioning)
49+
## Milestone 3 — Alerts, Scouting & Sharing ✅
50+
51+
- [x] Alerts API + UI — list with severity badges, close/acknowledge, notification badges
52+
- [x] Alerts page with field/farm/status filters
53+
- [x] Scouting API + UI — create/list observations, pin on map, photo upload via presigned URLs
54+
- [x] Scouting observations as interactive map markers
55+
- [x] Share API + UI — create/revoke share links with expiry, copy URL
56+
- [x] Public report page (`/share/[token]`) — map, NDVI snapshot, time-series chart, alerts, scouting notes
57+
58+
## Milestone 4 — Polish, Security & QA ✅
59+
60+
- [x] Viewer role enforcement — write endpoints restricted to member+ across all routers
61+
- [x] Rate limiting (slowapi) — 120 req/min default, tighter limits on jobs and uploads, Redis-backed
62+
- [x] Pagination consistency — all list endpoints use `PaginatedResponse` envelope
63+
- [x] Audit log UI in settings page — event icons, search, pagination
64+
- [x] Frontend structured logging (pino)
65+
- [x] Celery worker health check in Docker Compose
66+
- [x] Backup script (`deploy/backup.sh`) — automated daily pg_dump, 7-day retention
67+
- [x] MinIO bucket versioning documentation
68+
- [x] WAL archiving / PITR documentation
7069
- [ ] API unit/integration tests
7170
- [ ] Frontend component tests
7271
- [ ] E2E acceptance tests
@@ -116,6 +115,9 @@ These are under consideration but not yet committed. Grouped by theme and roughl
116115

117116
## How to Contribute
118117

119-
Pick any unchecked item above, or browse [open issues](https://github.com/superzero11/OpenFarm/issues). Milestone 3 frontend tasks are the highest-impact contributions right now.
118+
The MVP is complete! The highest-impact contributions right now are:
119+
120+
1. **Automated tests** — API integration tests, frontend component tests, and E2E acceptance tests (Milestone 4 remaining items)
121+
2. **Future Ideas** — pick any item from the list above or browse [open issues](https://github.com/superzero11/OpenFarm/issues)
120122

121123
See [CONTRIBUTING.md](CONTRIBUTING.md) for setup and guidelines.

apps/web/src/app/[locale]/(authenticated)/settings/page.tsx

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -159,8 +159,8 @@ export default function SettingsPage() {
159159
const loadMembers = useCallback(async () => {
160160
if (!currentOrg) return;
161161
try {
162-
const list = await orgsApi.members(currentOrg.id);
163-
setMembers(list);
162+
const res = await orgsApi.members(currentOrg.id);
163+
setMembers(res.items);
164164
} catch (err) {
165165
toast.error(t("failedLoadMembers"));
166166
}

apps/web/src/components/field/ndvi-tab.tsx

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -90,15 +90,15 @@ export default function NdviTab({ fieldId, onShowLayer }: NdviTabProps) {
9090
// ── Load layers + stats ──────────────────────────
9191
const loadData = useCallback(async () => {
9292
try {
93-
const [l, s] = await Promise.all([
93+
const [layersRes, statsRes] = await Promise.all([
9494
monitoringApi.layers(fieldId),
9595
monitoringApi.stats(fieldId),
9696
]);
97-
setLayers(l);
98-
setStats(s);
97+
setLayers(layersRes.items);
98+
setStats(statsRes.items);
9999
// Auto-select latest date
100-
if (l.length > 0 && !selectedDate) {
101-
setSelectedDate(l[l.length - 1].date);
100+
if (layersRes.items.length > 0 && !selectedDate) {
101+
setSelectedDate(layersRes.items[layersRes.items.length - 1].date);
102102
}
103103
} catch {
104104
// silent — may simply have no data
@@ -373,7 +373,7 @@ export default function NdviTab({ fieldId, onShowLayer }: NdviTabProps) {
373373
</CardHeader>
374374
<CardContent className="px-3 pb-3 pt-0">
375375
<div className="space-y-1 max-h-36 overflow-y-auto">
376-
{[...layers].reverse().map((layer) => {
376+
{layers.map((layer) => {
377377
const isSelected = layer.date === selectedDate;
378378
const stat = stats.find((s) => s.date === layer.date);
379379
return (

apps/web/src/lib/api.ts

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -278,7 +278,7 @@ export const orgsApi = {
278278
get: (orgId: string) => apiFetch<OrgDetail>(`/orgs/${orgId}`, { orgId }),
279279
create: (name: string) => apiFetch<Org>("/orgs", { method: "POST", body: JSON.stringify({ name }), skipOrg: true }),
280280
update: (orgId: string, name: string) => apiFetch<Org>(`/orgs/${orgId}`, { method: "PATCH", body: JSON.stringify({ name }), orgId }),
281-
members: (orgId: string) => apiFetch<Member[]>(`/orgs/${orgId}/members`, { orgId }),
281+
members: (orgId: string) => apiFetch<Paginated<Member>>(`/orgs/${orgId}/members`, { orgId }),
282282
changeMemberRole: (orgId: string, userId: string, role: string) =>
283283
apiFetch(`/orgs/${orgId}/members/${userId}`, { method: "PATCH", body: JSON.stringify({ role }), orgId }),
284284
removeMember: (orgId: string, userId: string) =>
@@ -332,9 +332,9 @@ export const fieldsApi = {
332332

333333
export const monitoringApi = {
334334
layers: (fieldId: string, type = "NDVI", limit = 50) =>
335-
apiFetch<RasterLayer[]>(`/fields/${fieldId}/layers?type=${type}&limit=${limit}`),
335+
apiFetch<Paginated<RasterLayer>>(`/fields/${fieldId}/layers?type=${type}&limit=${limit}`),
336336
stats: (fieldId: string, type = "NDVI", limit = 200) =>
337-
apiFetch<FieldStat[]>(`/fields/${fieldId}/stats?type=${type}&limit=${limit}`),
337+
apiFetch<Paginated<FieldStat>>(`/fields/${fieldId}/stats?type=${type}&limit=${limit}`),
338338
};
339339

340340
// ── Jobs ─────────────────────────────────────────────────────────────

apps/web/tsconfig.tsbuildinfo

Lines changed: 1 addition & 1 deletion
Large diffs are not rendered by default.

0 commit comments

Comments
 (0)