From abc496026d334ef945ddc79c2c9f973c6ce64f7b Mon Sep 17 00:00:00 2001 From: Daniele Martinoli Date: Wed, 22 Oct 2025 16:35:30 +0200 Subject: [PATCH 1/7] draft --- ...HV-2287-refactor-mcpregistry-controller.md | 105 ++++++++++++++++++ 1 file changed, 105 insertions(+) create mode 100644 docs/proposals/THV-2287-refactor-mcpregistry-controller.md diff --git a/docs/proposals/THV-2287-refactor-mcpregistry-controller.md b/docs/proposals/THV-2287-refactor-mcpregistry-controller.md new file mode 100644 index 000000000..eb15d543c --- /dev/null +++ b/docs/proposals/THV-2287-refactor-mcpregistry-controller.md @@ -0,0 +1,105 @@ +# Refactor MCPRegistry controller +## Goal +The main goal is to expose all the data source management functions in the `thv-registry-api` +module, so that both the CLI and the Kubernetes users can benefit the same features. + +The impacted features, today implemented in the MCPRegistry controller, are the following: +* Data source configuration +* Periodical and on-demand sync +* Data filtering + +## Current status +### Data source configuration +The operator allows to define the data source in the `spec.source` and `spec.filter` sections of +the `MCPRegistry`: +```yaml +spec: + source: + type: configmap + configmap: + name: minimal-registry-data + syncPolicy: + interval: "30m" # Automatically sync every 30 minutes + filter: + tags: + include: + - "database" # Only database servers + - "production" # Only production-ready servers + exclude: + - "experimental" # Exclude experimental servers + - "deprecated" # Exclude deprecated servers + - "beta" # Exclude beta versions +``` +### Implementation +The implementation is defined in the following packages: +- `cmd/thv-operator/pkg/sources`: data source management functions + - Includes the implementations of the `SourceHandler` interface for the managed types + (`api`, `configmap` and `git`). + - Includes the implementation of the `StorageManager` interface using a `ConfigMap` +- `cmd/thv-operator/pkg/httpclient`: utility classes for the `api` type. +- `cmd/thv-operator/pkg/git`: utility classes for the `git` type. +- `cmd/thv-operator/pkg/sync`: interface and default implementation of the `Manager` interface to + handle the automatic or on-demand sync flow. +- `cmd/thv-operator/pkg/filtering`: the `FilterService` interface and a default implementation to + apply name and tag filtering to the original data. + +The above packages are executed in the context the the operator controller, with no interactions +with other components. + +### Interaction with Registry API server +The `MCPRegistry` controller manages the deployments of the registry API servers but does not +interact with them after they are deployed. + +### MCPRegistry status +The `MCPRegistry` status keeps track of the deployment status of the registry API and also the sync +status, both with explicit fields and the standard condition fields. +```yaml +apiStatus: + endpoint: http://thv-git-api.toolhive-system:8080 + message: Registry API is ready and serving requests + phase: Ready + readySince: "2025-10-22T08:30:38Z" +conditions: + - lastTransitionTime: "2025-10-22T08:30:38Z" + message: Registry API is ready and serving requests + reason: APIReady + status: "True" + type: APIReady +lastAppliedFilterHash: 74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b +lastManualSyncTrigger: "2025-10-22T08:38:16.286Z" +message: Registry is ready and API is serving requests +phase: Ready +storageRef: + configMapRef: + name: thv-git-registry-storage + type: configmap +syncStatus: + lastAttempt: "2025-10-22T08:30:27Z" + lastSyncHash: c013b6b36286ab438f3c08306474f2e163bee6554c8886c2c8155164479cdf53 + lastSyncTime: "2025-10-22T08:30:27Z" + message: Registry data synchronized successfully + phase: Complete + serverCount: 88 +``` +## Refactory Proposal +### High level requirements +- Both the Kubernetes and the CLI registries provide the same data source functionalities. +- The Kubernetes controller takes care deploying the registry API server as today and ensure + that its configuration matches the `MCPRegistry` specification. +- The registry API reacts to configuration changes to apply them. +- The Kubernetes controller + + +## Impacted code +### Operator +#### MCPRegistry controller +#### Integration tests + +## Impacted code +#### Registry AP Server +#### Integration tests + + +### [NEXT] New CRDs for data source specification +Move the data source management functions from the `thv-operator` module to the `thv-registry-api` +module \ No newline at end of file From ef84812be76c829d3bbe22c241d566ecd7d532fe Mon Sep 17 00:00:00 2001 From: Daniele Martinoli Date: Wed, 22 Oct 2025 16:57:59 +0200 Subject: [PATCH 2/7] draft --- .../THV-2287-refactor-mcpregistry-controller.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/docs/proposals/THV-2287-refactor-mcpregistry-controller.md b/docs/proposals/THV-2287-refactor-mcpregistry-controller.md index eb15d543c..2a11d2258 100644 --- a/docs/proposals/THV-2287-refactor-mcpregistry-controller.md +++ b/docs/proposals/THV-2287-refactor-mcpregistry-controller.md @@ -84,10 +84,11 @@ syncStatus: ## Refactory Proposal ### High level requirements - Both the Kubernetes and the CLI registries provide the same data source functionalities. -- The Kubernetes controller takes care deploying the registry API server as today and ensure - that its configuration matches the `MCPRegistry` specification. +- The Kubernetes controller takes care of: + - Deploying the registry API server as today. + - Persist the registry API server configuration and ensure it is updated with the `MCPRegistry` specification. + - Maintaining the status information about the API and Sync status using today's format. - The registry API reacts to configuration changes to apply them. -- The Kubernetes controller ## Impacted code @@ -99,6 +100,11 @@ syncStatus: #### Registry AP Server #### Integration tests +## Phase 1 +### Tasks + +## Phase 1 +### Tasks ### [NEXT] New CRDs for data source specification Move the data source management functions from the `thv-operator` module to the `thv-registry-api` From baa626b4e225876339f0c523366719fb835d1712 Mon Sep 17 00:00:00 2001 From: Daniele Martinoli Date: Thu, 23 Oct 2025 13:24:45 +0200 Subject: [PATCH 3/7] Plan for moving registry source handling logic to thv-registry-api Signed-off-by: Daniele Martinoli --- ...HV-2287-refactor-mcpregistry-controller.md | 378 ++++++++++++++---- 1 file changed, 291 insertions(+), 87 deletions(-) diff --git a/docs/proposals/THV-2287-refactor-mcpregistry-controller.md b/docs/proposals/THV-2287-refactor-mcpregistry-controller.md index 2a11d2258..c2fce7948 100644 --- a/docs/proposals/THV-2287-refactor-mcpregistry-controller.md +++ b/docs/proposals/THV-2287-refactor-mcpregistry-controller.md @@ -1,17 +1,29 @@ # Refactor MCPRegistry controller -## Goal -The main goal is to expose all the data source management functions in the `thv-registry-api` -module, so that both the CLI and the Kubernetes users can benefit the same features. +## Requirement Overview -The impacted features, today implemented in the MCPRegistry controller, are the following: -* Data source configuration -* Periodical and on-demand sync -* Data filtering +The MCPRegistry operator uses the `MCPRegistry` CRD to deploy Registry API servers pre-populated with curated MCP server definitions. The controller fetches registry data from multiple source types (`ConfigMap`, Git repositories, or external Registry APIs), applies filtering rules, and stores the result in a `ConfigMap` mounted by the Registry API server. + +Currently, these data source management capabilities exist only in the Kubernetes operator. However, ToolHive users typically start with the CLI before deploying to Kubernetes. To provide a consistent experience, we need to move data source management into `thv-registry-api` itself, making these features available in both CLI and Kubernetes environments. + +**Affected features:** +- Data source configuration (ConfigMap, Git, Registry API) +- Periodic and on-demand synchronization +- Tag and name filtering + +This proposal describes moving functionality from the `thv-operator` module ([toolhive](https://github.com/stacklok/toolhive)) to the `thv-registry-api` module ([toolhive-registry-server](https://github.com/stacklok/toolhive-registry-server)). + +### Design Constraints + +This refactoring maintains feature parity without introducing new capabilities: + +- **API stability**: The `MCPRegistry` CRD spec remains unchanged +- **Status evolution**: Status field changes will be documented if required +- **CLI extension**: New runtime flags may be added to `thv-registry-api` +- **Storage**: File-based storage will be implemented; database storage is out of scope ## Current status ### Data source configuration -The operator allows to define the data source in the `spec.source` and `spec.filter` sections of -the `MCPRegistry`: +`MCPRegistry` resources define data sources and filtering rules in the `spec.source` and `spec.filter` fields: ```yaml spec: source: @@ -19,93 +31,285 @@ spec: configmap: name: minimal-registry-data syncPolicy: - interval: "30m" # Automatically sync every 30 minutes + interval: "30m" filter: tags: - include: - - "database" # Only database servers - - "production" # Only production-ready servers - exclude: - - "experimental" # Exclude experimental servers - - "deprecated" # Exclude deprecated servers - - "beta" # Exclude beta versions + include: ["database", "production"] + exclude: ["experimental", "deprecated", "beta"] ``` -### Implementation -The implementation is defined in the following packages: -- `cmd/thv-operator/pkg/sources`: data source management functions - - Includes the implementations of the `SourceHandler` interface for the managed types - (`api`, `configmap` and `git`). - - Includes the implementation of the `StorageManager` interface using a `ConfigMap` -- `cmd/thv-operator/pkg/httpclient`: utility classes for the `api` type. -- `cmd/thv-operator/pkg/git`: utility classes for the `git` type. -- `cmd/thv-operator/pkg/sync`: interface and default implementation of the `Manager` interface to - handle the automatic or on-demand sync flow. -- `cmd/thv-operator/pkg/filtering`: the `FilterService` interface and a default implementation to - apply name and tag filtering to the original data. - -The above packages are executed in the context the the operator controller, with no interactions -with other components. - -### Interaction with Registry API server -The `MCPRegistry` controller manages the deployments of the registry API servers but does not -interact with them after they are deployed. - -### MCPRegistry status -The `MCPRegistry` status keeps track of the deployment status of the registry API and also the sync -status, both with explicit fields and the standard condition fields. +### Current Implementation +The `MCPRegistry` controller implements data source management in these operator packages: +| Package| Purpose | +|--------|---------| +|`cmd/thv-operator/pkg/sources`|`SourceHandler` interface and implementations for `ConfigMap`, Git, and Registry API sources; StorageManager implementation using `ConfigMap` | +|`cmd/thv-operator/pkg/httpclient`|HTTP client for Registry API sources| +|`cmd/thv-operator/pkg/git`|Git repository operations| +|`cmd/thv-operator/pkg/sync`|`Manager` interface for periodic and on-demand synchronization| +|`cmd/thv-operator/pkg/filtering`|`FilterService` interface and implementation for tag and name filtering| + +These packages run exclusively within the operator controller with no external dependencies. + +### Controller Responsibilities +The controller currently handles two distinct concerns: +1. Deployment management: Creates and maintains Registry API server Deployments and Services +1. Data synchronization: Fetches, filters, and stores registry data in `ConfigMap`s + +After deployment, the controller does not communicate with the Registry API server—it only manages the `ConfigMap` containing registry data. + +### Status Reporting +The MCPRegistry status tracks deployment, synchronization and storage state information. + +#### API deployment status ```yaml apiStatus: endpoint: http://thv-git-api.toolhive-system:8080 - message: Registry API is ready and serving requests phase: Ready readySince: "2025-10-22T08:30:38Z" -conditions: - - lastTransitionTime: "2025-10-22T08:30:38Z" - message: Registry API is ready and serving requests - reason: APIReady - status: "True" - type: APIReady -lastAppliedFilterHash: 74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b -lastManualSyncTrigger: "2025-10-22T08:38:16.286Z" -message: Registry is ready and API is serving requests -phase: Ready -storageRef: - configMapRef: - name: thv-git-registry-storage - type: configmap +``` +#### Synchronization status +```yaml syncStatus: - lastAttempt: "2025-10-22T08:30:27Z" - lastSyncHash: c013b6b36286ab438f3c08306474f2e163bee6554c8886c2c8155164479cdf53 - lastSyncTime: "2025-10-22T08:30:27Z" - message: Registry data synchronized successfully phase: Complete + lastSyncTime: "2025-10-22T08:30:27Z" serverCount: 88 + lastSyncHash: c013b6b36286ab... +``` +#### Storage reference +```yaml +storageRef: + type: configmap + configMapRef: + name: thv-git-registry-storage +``` +#### Standard conditions +```yaml +conditions: + - type: APIReady + status: "True" + lastTransitionTime: "2025-10-22T08:30:38Z" +``` + +## Refactoring Proposal + +### Component Responsibilities + +**Controller:** +- Deploy and manage Registry API (`Deployment`, `Service`) +- Create and update configuration `ConfigMap` from `MCPRegistry` spec +- Monitor deployment health and update `apiStatus` +- Trigger manual syncs via Registry API endpoint +- Update `syncStatus`, see [Sync Status Updates](#sync-status-updates) options + +**Registry API:** +- Watch configuration file for changes +- Execute periodic and manual synchronization +- Fetch, filter, and store registry data +- Expose sync status via HTTP endpoint + +### Configuration Management + +**Flow:** +1. Controller creates `ConfigMap` `-config` containing `MCPRegistry` spec +2. `ConfigMap` mounted at `/config/spec.yaml` in Registry API pod +3. Registry API launched with `--config /config/spec.yaml` flag +4. Registry API watches file and triggers sync on changes + +### Storage + +**File-based (Phase 1):** +- Registry API stores data at path defined by `--storage-path` flag +- Controller sets `--storage-path /data` using mounted `emptyDir` volume +- Data format: ToolHive registry JSON schema (unchanged from current) +- Persistence: Data survives pod restarts, lost on deployment deletion + +**Database-backed (Future):** Deferred to separate proposal for multi-replica support. + +### Sync Status Updates + +**Options:** + +**Option 1 - HTTP Polling (Recommended):** +- Registry API exposes `GET /api/v1/status` returning sync history, stored in the configured storage path +- Controller polls every 60 seconds and updates `syncStatus` +- Pros: Simple, no new CRDs, works for CLI and Kubernetes +- Cons: 60-second status latency + +**Option 2 - Event CRD:** +- New `SyncCompletionEvent` CRD created by Registry API on sync completion +- Controller watches events and updates `MCPRegistry.status.syncStatus` +- Pros: Real-time updates +- Cons: Requires RBAC, adds CRD complexity, Kubernetes-only + +**Option 3 - Remove syncStatus:** +- Remove `syncStatus` from `MCPRegistry` entirely +- Users query Registry API `GET /api/v1/status` directly for sync information +- Pros: Simplest implementation +- Cons: Breaking change, loses kubectl visibility + +**Recommendation:** Option 1 provides best balance of simplicity and functionality. + +**Status endpoint response:** +```json +{ + "lastSync": { + "type": "automatic", + "startTime": "2025-10-22T08:30:38Z", + "completionTime": "2025-10-22T08:32:38Z", + "hash": "c013b6b36286ab...", + "serverCount": 88, + "result": "completed", + "attemptCount": 0, + "errorMessage": null + } +} ``` -## Refactory Proposal -### High level requirements -- Both the Kubernetes and the CLI registries provide the same data source functionalities. -- The Kubernetes controller takes care of: - - Deploying the registry API server as today. - - Persist the registry API server configuration and ensure it is updated with the `MCPRegistry` specification. - - Maintaining the status information about the API and Sync status using today's format. -- The registry API reacts to configuration changes to apply them. - - -## Impacted code -### Operator -#### MCPRegistry controller -#### Integration tests - -## Impacted code -#### Registry AP Server -#### Integration tests - -## Phase 1 -### Tasks - -## Phase 1 -### Tasks - -### [NEXT] New CRDs for data source specification -Move the data source management functions from the `thv-operator` module to the `thv-registry-api` -module \ No newline at end of file + +### Automatic Sync + +- Registry API uses `time.Ticker` for periodic sync based on `syncPolicy.interval` +- On each interval: check source hash, sync only if changed +- Record result in sync history + +### Manual Sync + +- Registry API exposes `POST /api/v1/sync` endpoint +- Controller calls endpoint when `toolhive.stacklok.dev/sync-trigger` annotation changes +- Endpoint returns 202 (accepted) or 409 (sync in progress) +- Sync result recorded in history + +### Sync Failure Handling + +**Current behavior:** +- Fixed 5-minute retry interval +- Unlimited retry attempts +- `syncStatus.attemptCount` increments on failure, resets on success + +**Proposed (Phase 1):** +Match current behavior - Registry API retries every 5 minutes indefinitely. + +**Future enhancement:** +Add configurable retry policy to `syncPolicy.retryPolicy` (tracked separately). + +### Data Filtering + +- Registry API applies name and tag filters during sync processing +- Filter changes trigger immediate sync (hash-based detection) +- Move `cmd/thv-operator/pkg/filtering` package to `thv-registry-api` + +### API Status Updates + +Controller continues managing `apiStatus` with no changes: +- Monitors Deployment and Service health +- Updates phase: NotStarted, Deploying, Ready, Unhealthy, Error +- Existing `apiStatus` structure preserved + +## Implementation Plan + +### Phase 1: Move Data Source Management to Registry API + +**Goal:** Full-functioning solution with sync logic in `thv-registry-api`, maintaining backward compatibility with existing MCPRegistry resources. + +**Task Order Rationale:** Registry API must be enhanced first before operator changes, to avoid breaking existing deployments. + +#### Registry API Development + +1. **Add configuration file support** + - Implement `--config` flag in `thv-registry-api serve` command + - Add YAML configuration parsing for `source`, `syncPolicy`, and `filter` fields + - Implement file watcher for configuration changes (using `fsnotify` or similar) + - Graceful reload on configuration changes + +2. **Add file-based storage** + - Implement `--storage-path` flag for registry data location + - Add file-based storage backend matching current ConfigMap JSON format + - Atomic file writes (temp file + rename pattern) + - Load existing data on startup if present + +3. **Port source handler packages** + - Copy `cmd/thv-operator/pkg/sources` to `thv-registry-api` + - Copy `cmd/thv-operator/pkg/httpclient` to `thv-registry-api` + - Copy `cmd/thv-operator/pkg/git` to `thv-registry-api` + - Adapt packages to work without Kubernetes client dependencies + - Update import paths + +4. **Port sync manager** + - Copy `cmd/thv-operator/pkg/sync` to `thv-registry-api` + - Adapt sync logic to use file-based storage instead of ConfigMap + - Implement retry logic (5-minute interval, unlimited attempts) + - Remove Kubernetes-specific dependencies + +5. **Port filtering logic** + - Copy `cmd/thv-operator/pkg/filtering` to `thv-registry-api` + - Keep filtering interface and implementation unchanged + - Update import paths + +6. **Implement automatic sync** + - Add `time.Ticker` based periodic sync using `syncPolicy.interval` + - Implement hash-based change detection before syncing + - Store sync history in file at storage path + +7. **Add sync status endpoint** + - Implement `GET /api/v1/status` returning sync history + - Return format matching proposed JSON schema + - Read from sync history file + +8. **Add manual sync endpoint** + - Implement `POST /api/v1/sync` for manual sync triggering + - Return 202 (accepted) or 409 (sync in progress) + - Prevent concurrent syncs + +9. **Add integration tests** + - Test configuration file watching and reload + - Test all source types (ConfigMap, Git, API) + - Test filtering logic + - Test automatic and manual sync + - Test retry behavior on failures + +#### Operator Updates + +10. **Update operator to use new Registry API** + - Modify MCPRegistry controller to create configuration ConfigMap + - Update Deployment spec to mount config ConfigMap at `/config/spec.yaml` + - Update Deployment spec to mount emptyDir volume at `/data` + - Add `--config /config/spec.yaml --storage-path /data` flags to registry-api container + - Add polling logic to call `GET /api/v1/status` every 60 seconds (depends on [selected update option](#sync-status-updates)) + - Update `syncStatus` from polling results + +11. **Add manual sync trigger via API** + - Detect `toolhive.stacklok.dev/sync-trigger` annotation changes + - Call `POST /api/v1/sync` on Registry API endpoint + - Update `lastManualSyncTrigger` after successful API call + +12. **Remove sync logic from operator** + - Remove sync execution code from MCPRegistry controller (keep status update only) + - Mark `cmd/thv-operator/pkg/sources` as deprecated (keep for reference) + - Mark `cmd/thv-operator/pkg/sync` as deprecated (keep for reference) + - Mark `cmd/thv-operator/pkg/filtering` as deprecated (keep for reference) + +13. **Update operator integration tests** + - Update tests to verify controller creates config ConfigMap + - Test controller polls Registry API status endpoint + - Test manual sync via annotation + - Test backward compatibility with existing MCPRegistry resources + +14. **Update documentation** + - Document new Registry API flags and endpoints + - Update MCPRegistry CRD examples + - Add migration guide for users (if any operator upgrade steps needed) + +### Phase 2: Future Enhancements + +**Goal:** Improvements deferred from Phase 1 to keep initial scope manageable. + +1. **Configurable retry policy** + - Add `spec.syncPolicy.retryPolicy` field to MCPRegistry CRD + - Support `maxAttempts`, `retryInterval`, `backoff` configuration + - Implement in Registry API sync logic + - Update operator to pass retry config via ConfigMap + +2. **Real-time status updates (if needed)** + - Evaluate if 60-second polling latency is acceptable + - If not: Implement Option 2 (Event CRD) from proposal + - Add `SyncCompletionEvent` CRD + - Update Registry API to create events + - Update controller to watch events From f531c340bf82a7f79cf9078fa5b6e8d1040df9b4 Mon Sep 17 00:00:00 2001 From: Daniele Martinoli Date: Thu, 23 Oct 2025 13:29:48 +0200 Subject: [PATCH 4/7] renamed to match validation pattern Signed-off-by: Daniele Martinoli --- ...-controller.md => THV-2301-refactor-mcpregistry-controller.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/proposals/{THV-2287-refactor-mcpregistry-controller.md => THV-2301-refactor-mcpregistry-controller.md} (100%) diff --git a/docs/proposals/THV-2287-refactor-mcpregistry-controller.md b/docs/proposals/THV-2301-refactor-mcpregistry-controller.md similarity index 100% rename from docs/proposals/THV-2287-refactor-mcpregistry-controller.md rename to docs/proposals/THV-2301-refactor-mcpregistry-controller.md From c336698b2d79d8780262cfd9717a16fcf35bf6e2 Mon Sep 17 00:00:00 2001 From: Daniele Martinoli Date: Mon, 27 Oct 2025 10:22:36 +0100 Subject: [PATCH 5/7] reviewed API paths Signed-off-by: Daniele Martinoli --- .../THV-2301-refactor-mcpregistry-controller.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/proposals/THV-2301-refactor-mcpregistry-controller.md b/docs/proposals/THV-2301-refactor-mcpregistry-controller.md index c2fce7948..ec87742f7 100644 --- a/docs/proposals/THV-2301-refactor-mcpregistry-controller.md +++ b/docs/proposals/THV-2301-refactor-mcpregistry-controller.md @@ -129,7 +129,7 @@ conditions: **Options:** **Option 1 - HTTP Polling (Recommended):** -- Registry API exposes `GET /api/v1/status` returning sync history, stored in the configured storage path +- Registry API exposes `GET /registry/status` returning sync history, stored in the configured storage path - Controller polls every 60 seconds and updates `syncStatus` - Pros: Simple, no new CRDs, works for CLI and Kubernetes - Cons: 60-second status latency @@ -142,7 +142,7 @@ conditions: **Option 3 - Remove syncStatus:** - Remove `syncStatus` from `MCPRegistry` entirely -- Users query Registry API `GET /api/v1/status` directly for sync information +- Users query Registry API `GET /registry/status` directly for sync information - Pros: Simplest implementation - Cons: Breaking change, loses kubectl visibility @@ -172,7 +172,7 @@ conditions: ### Manual Sync -- Registry API exposes `POST /api/v1/sync` endpoint +- Registry API exposes `POST /registry/sync` endpoint - Controller calls endpoint when `toolhive.stacklok.dev/sync-trigger` annotation changes - Endpoint returns 202 (accepted) or 409 (sync in progress) - Sync result recorded in history @@ -249,12 +249,12 @@ Controller continues managing `apiStatus` with no changes: - Store sync history in file at storage path 7. **Add sync status endpoint** - - Implement `GET /api/v1/status` returning sync history + - Implement `GET /registry/status` returning sync history - Return format matching proposed JSON schema - Read from sync history file 8. **Add manual sync endpoint** - - Implement `POST /api/v1/sync` for manual sync triggering + - Implement `POST /registry/sync` for manual sync triggering - Return 202 (accepted) or 409 (sync in progress) - Prevent concurrent syncs @@ -272,12 +272,12 @@ Controller continues managing `apiStatus` with no changes: - Update Deployment spec to mount config ConfigMap at `/config/spec.yaml` - Update Deployment spec to mount emptyDir volume at `/data` - Add `--config /config/spec.yaml --storage-path /data` flags to registry-api container - - Add polling logic to call `GET /api/v1/status` every 60 seconds (depends on [selected update option](#sync-status-updates)) + - Add polling logic to call `GET /registry/status` every 60 seconds (depends on [selected update option](#sync-status-updates)) - Update `syncStatus` from polling results 11. **Add manual sync trigger via API** - Detect `toolhive.stacklok.dev/sync-trigger` annotation changes - - Call `POST /api/v1/sync` on Registry API endpoint + - Call `POST /registry/sync` on Registry API endpoint - Update `lastManualSyncTrigger` after successful API call 12. **Remove sync logic from operator** From f2fbdbe312ce4225b650b34d701d407508bdb9ae Mon Sep 17 00:00:00 2001 From: Daniele Martinoli Date: Tue, 28 Oct 2025 10:09:59 +0100 Subject: [PATCH 6/7] added option 4 for sync updates Signed-off-by: Daniele Martinoli --- .../THV-2301-refactor-mcpregistry-controller.md | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/docs/proposals/THV-2301-refactor-mcpregistry-controller.md b/docs/proposals/THV-2301-refactor-mcpregistry-controller.md index ec87742f7..740f1464e 100644 --- a/docs/proposals/THV-2301-refactor-mcpregistry-controller.md +++ b/docs/proposals/THV-2301-refactor-mcpregistry-controller.md @@ -54,7 +54,7 @@ The controller currently handles two distinct concerns: 1. Deployment management: Creates and maintains Registry API server Deployments and Services 1. Data synchronization: Fetches, filters, and stores registry data in `ConfigMap`s -After deployment, the controller does not communicate with the Registry API server—it only manages the `ConfigMap` containing registry data. +After deploying the Registry API server, the controller does not communicate with it. Instead, the controller directly fetches registry data from the configured source (Git, ConfigMap, or remote API) and stores it in a `ConfigMap`. The deployed Registry API server reads from this same `ConfigMap` to serve requests from external clients. ### Status Reporting The MCPRegistry status tracks deployment, synchronization and storage state information. @@ -128,7 +128,7 @@ conditions: **Options:** -**Option 1 - HTTP Polling (Recommended):** +**Option 1 - HTTP Polling:** - Registry API exposes `GET /registry/status` returning sync history, stored in the configured storage path - Controller polls every 60 seconds and updates `syncStatus` - Pros: Simple, no new CRDs, works for CLI and Kubernetes @@ -146,7 +146,14 @@ conditions: - Pros: Simplest implementation - Cons: Breaking change, loses kubectl visibility -**Recommendation:** Option 1 provides best balance of simplicity and functionality. +**Option 4 - Hybrid polling (Recommended):** +- Registry API exposes `GET /registry/status` returning sync history, stored in the configured storage path +- Controller polls every 60 seconds and creates an instance of the new `SyncCompletionEvent` CRD on sync completion +- Controller watches events and updates `MCPRegistry.status.syncStatus` +- Pros: Anticipate changes for the final solution (option 2) +- Cons: Adds CRD complexity, 60-second status latency + +**Recommendation:** Option 4 provides best balance of simplicity and functionality. **Status endpoint response:** ```json From 51a162e204063994c09ccee5fee5e076ecaccb1d Mon Sep 17 00:00:00 2001 From: Daniele Martinoli Date: Tue, 28 Oct 2025 10:47:50 +0100 Subject: [PATCH 7/7] changed to PUT /registry/sync Signed-off-by: Daniele Martinoli --- docs/proposals/THV-2301-refactor-mcpregistry-controller.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/proposals/THV-2301-refactor-mcpregistry-controller.md b/docs/proposals/THV-2301-refactor-mcpregistry-controller.md index 740f1464e..090527fe7 100644 --- a/docs/proposals/THV-2301-refactor-mcpregistry-controller.md +++ b/docs/proposals/THV-2301-refactor-mcpregistry-controller.md @@ -179,9 +179,9 @@ conditions: ### Manual Sync -- Registry API exposes `POST /registry/sync` endpoint +- Registry API exposes `PUT /registry/sync` endpoint - Controller calls endpoint when `toolhive.stacklok.dev/sync-trigger` annotation changes -- Endpoint returns 202 (accepted) or 409 (sync in progress) +- Endpoint returns 201 (created) or 409 (sync in progress) - Sync result recorded in history ### Sync Failure Handling