Skip to content

feat(examples,mcp): add YAML pipeline builder + local/shell MCP transports, configurable timeouts & retry overrides#21

Merged
assagman merged 7 commits intomainfrom
example-generator
Dec 16, 2025
Merged

feat(examples,mcp): add YAML pipeline builder + local/shell MCP transports, configurable timeouts & retry overrides#21
assagman merged 7 commits intomainfrom
example-generator

Conversation

@assagman
Copy link
Owner

@assagman assagman commented Dec 16, 2025

  • Add a new examples/yaml_program that lets users define DSGo pipelines declaratively in YAML (signatures, modules, tools, and pipeline steps) instead of wiring everything in Go.
  • Support tool-driven pipelines via ReAct by allowing YAML to reference:
    • filesystem tools,
    • built-in function tools (no API keys),
    • MCP-backed tools (Exa/Jina/Tavily/custom), including richer multi-stage pipeline examples.
  • Introduce in-process MCP support with a Local transport/handler so tools/servers can be invoked without network or subprocess overhead—useful for tests and embedding.
  • Add a built-in MCP “shell server” implementation to expose shell-style tools through MCP, enabling quick local automation workflows.
  • Harden MCP transports with configurable timeouts (HTTP + SSE variants) and re-export the new MCP APIs in dsgo.go so consumers can use them from the public package surface.
  • Improve robustness of retry behavior by enabling partial overrides (instead of forcing full retry config replacement), making it easier to tune a single parameter safely.

@assagman assagman self-assigned this Dec 16, 2025
@assagman assagman added the enhancement New feature or request label Dec 16, 2025
@assagman assagman changed the title new example: example generator via yaml feat(examples,mcp): add YAML pipeline builder + local/shell MCP transports, configurable timeouts & retry overrides Dec 16, 2025
Add `examples/yaml_program` demonstrating declarative DSGo
programs via YAML (signatures, modules, tools, MCP clients)
with sample pipelines for Predict/CoT/ReAct and web research flows.

Signed-off-by: assagman <ahmetsercansagman@gmail.com>
Signed-off-by: assagman <ahmetsercansagman@gmail.com>
- re-export LocalTransport/LocalHandler and ShellServer APIs in dsgo.go
- add LocalTransport for routing MCP JSON-RPC requests in-process
- introduce safe ShellServer tools (shell_run/apply_patch) with path/command validation and output limits
- add configurable HTTP/SSE transport timeouts with new constructors
- include unit tests for local transport and shell server behavior

Signed-off-by: assagman <ahmetsercansagman@gmail.com>
create dsgo programs from yaml

Signed-off-by: assagman <ahmetsercansagman@gmail.com>
…m example

- Add `examples/yaml_program` (standalone module) and document it in examples index
- Support `DSGO_CACHE_SIZE` env var and allow disabling cache via `WithCache(<=0)`
- Make HTTP retries configurable via options, honor `Retry-After`, and wire `MaxRetries` from settings into providers
- Prefix cache keys with provider name to avoid cross-provider collisions
- Harden MCP shell server argument/path validation; panic early on nil LocalTransport handler
- Improve yaml_program tools path validation and environment reporting (runtime GOOS/GOARCH/Go version/CPU count)

Signed-off-by: assagman <ahmetsercansagman@gmail.com>
…verrides

Signed-off-by: assagman <ahmetsercansagman@gmail.com>
…P support

- Split “custom tools” (function/filesystem) from MCP tools; remove YAML `tools: {type: mcp}` path
- Add `modules.<name>.mcp` config with tool filtering (`"*"` or explicit tool names)
- Support `filesystem` MCP client with optional `allowed_dirs`
- Update YAML pipeline examples/README to reflect new MCP model and reorder TODO pipeline
- Make MCP local transport constructor return error (no panic) and add nil-handler test
- Ignore/remove generated yaml_program binary from repo output

Signed-off-by: assagman <ahmetsercansagman@gmail.com>
@assagman assagman merged commit 63b8130 into main Dec 16, 2025
4 checks passed
@assagman assagman deleted the example-generator branch December 16, 2025 12:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant