feature/website tools #193
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Implement WebsiteToolsDiscovery to fetch tools from /.well-known/llms.txt - Support both array and manifest format for tool definitions - Create CreateWebMCPTools to generate dynamic tools from discovered website tools - Dispatch custom events to webpage for tool execution
…across agents and services
Contributor
|
CLA Assistant Lite bot ✅ All contributors have signed the CLA. Thank you for helping make BrowserOS better! |
Author
|
I have read the CLA Document and I hereby sign the CLA |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Greptile Overview
Greptile Summary
This PR adds website tools capability, enabling the agent to discover and use custom tools exposed by websites via /.well-known/llms.txt manifests. The implementation follows the agentic website pattern where sites can register DOM-based tools that the agent invokes through CustomEvents.
Key Changes:
WebsiteToolsDiscovery: Fetches and parsesllms.txtmanifests to discover available tools per routeCreateWebMCPTools: Converts website tool definitions into LangChain DynamicStructuredTools that dispatch CustomEventsBrowserAgent: Registers website tools dynamically on page load and re-registers when navigation occurs- Settings: Added UI toggle for enabling/disabling website MCP tools (defaults to enabled)
- Token counting: Replaced
TokenCounterclass with simple character-based estimation (length / 4)
Issues Found:
- Critical: Tool schema doesn't incorporate manifest parameters - LLM won't know what arguments are required
- Critical: Uses
toolNameinstead oftool.eventNamefor event dispatch - Major: Route matching fallback returns first route instead of null, causing wrong tools to be registered
- Performance: Repeated
getWebsiteTools()calls and tool re-registration on every execution adds overhead - Code quality: Excessive logging, direct
chrome.storage.localaccess instead of BrowserOS API
Positive aspects:
- Well-structured with proper Zod schemas for validation
- Good error handling with fallbacks
- Clean settings integration with migration
- Comprehensive prompt updates teaching the agent about tool preference order
Confidence Score: 2/5
- This PR has critical functional issues that will prevent website tools from working correctly
- Score reflects two critical bugs that break core functionality: (1) tool schema doesn't include manifest parameters so LLM can't provide correct arguments, and (2) wrong event name is dispatched. The route matching fallback issue could also cause tools to be registered for wrong pages. These must be fixed before merge.
- Pay close attention to
src/lib/tools/CreateWebMCPTools.ts(schema and eventName bugs) andsrc/lib/browser/WebsiteToolsDiscovery.ts(route fallback logic)
Important Files Changed
File Analysis
| Filename | Score | Overview |
|---|---|---|
| src/lib/browser/WebsiteToolsDiscovery.ts | 3/5 | New file implementing website tool discovery via /.well-known/llms.txt. Has route matching fallback issue and excessive logging. |
| src/lib/tools/CreateWebMCPTools.ts | 2/5 | New file creating LangChain tools from website manifests. Critical issues: doesn't use tool.eventName, schema doesn't include manifest parameters. |
| src/lib/agent/BrowserAgent.ts | 3/5 | Added website tools registration and re-registration logic. Uses chrome.storage.local directly, excessive logging, performance concerns with repeated tool registration. |
Sequence Diagram
sequenceDiagram
participant User
participant BrowserAgent
participant BrowserPage
participant WebsiteToolsDiscovery
participant Website
participant CreateWebMCPTools
participant ToolManager
participant LLM
User->>BrowserAgent: execute(task)
BrowserAgent->>BrowserAgent: _getWebsiteMCPSetting()
BrowserAgent->>chrome.storage.local: get('nxtscape-settings')
chrome.storage.local-->>BrowserAgent: settings
alt enableWebsiteMCP is true
BrowserAgent->>BrowserAgent: _registerWebsiteTools()
BrowserAgent->>BrowserPage: getCurrentPage()
BrowserPage-->>BrowserAgent: page
BrowserAgent->>BrowserPage: getWebsiteTools()
BrowserPage->>WebsiteToolsDiscovery: discoverTools(url)
WebsiteToolsDiscovery->>Website: fetch(/.well-known/llms.txt)
Website-->>WebsiteToolsDiscovery: LLM text with routes
WebsiteToolsDiscovery->>WebsiteToolsDiscovery: _getCurrentRoute()
WebsiteToolsDiscovery->>Website: fetch(manifestUrl)
Website-->>WebsiteToolsDiscovery: manifest with tools
WebsiteToolsDiscovery-->>BrowserPage: WebsiteTool[]
BrowserPage-->>BrowserAgent: WebsiteTool[]
loop for each tool
BrowserAgent->>CreateWebMCPTools: createWebsiteMCPTools(context, toolDef)
CreateWebMCPTools-->>BrowserAgent: DynamicStructuredTool
BrowserAgent->>ToolManager: register(tool)
end
end
BrowserAgent->>BrowserAgent: _getBrowserStateMessage()
BrowserAgent->>BrowserPage: getWebsiteTools()
BrowserPage->>WebsiteToolsDiscovery: discoverTools(url)
WebsiteToolsDiscovery-->>BrowserPage: WebsiteTool[]
BrowserPage-->>BrowserAgent: WebsiteTool[]
BrowserAgent->>BrowserAgent: append tools to browserStateString
BrowserAgent->>LLM: invoke with tools and state
LLM-->>BrowserAgent: AIMessage with tool_calls
BrowserAgent->>BrowserAgent: _processToolCalls()
BrowserAgent->>ToolManager: get(toolName)
ToolManager-->>BrowserAgent: tool
BrowserAgent->>CreateWebMCPTools: invoke tool function
CreateWebMCPTools->>chrome.scripting: executeScript (dispatch CustomEvent)
chrome.scripting-->>CreateWebMCPTools: event dispatched
CreateWebMCPTools-->>BrowserAgent: result
BrowserAgent->>BrowserAgent: _maybeReRegisterWebsiteTools()
BrowserAgent->>BrowserPage: getCurrentPage()
BrowserPage-->>BrowserAgent: page (new URL)
alt page changed
BrowserAgent->>BrowserAgent: _registerWebsiteTools()
BrowserAgent->>LLM: bindTools (re-bind)
end
BrowserAgent-->>User: task result
19 files reviewed, 3 comments
Comment on lines
+173
to
+177
| // Fallback: return first route as default | ||
| if (routes.length > 0) { | ||
| Logging.log('WebsiteToolsDiscovery', `No matching route for ${pathname}, using default: ${routes[0].pageId}`, 'info'); | ||
| return routes[0]; | ||
| } |
Contributor
There was a problem hiding this comment.
logic: Fallback to first route could cause issues. If the pathname doesn't match any route and the first route is for a different page, tools will be registered for the wrong context.
Suggested change
| // Fallback: return first route as default | |
| if (routes.length > 0) { | |
| Logging.log('WebsiteToolsDiscovery', `No matching route for ${pathname}, using default: ${routes[0].pageId}`, 'info'); | |
| return routes[0]; | |
| } | |
| // Fallback: No route found, return null | |
| Logging.log('WebsiteToolsDiscovery', `No matching route found for ${pathname}`, 'warning'); | |
| return null; |
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/lib/browser/WebsiteToolsDiscovery.ts
Line: 173:177
Comment:
**logic:** Fallback to first route could cause issues. If the pathname doesn't match any route and the first route is for a different page, tools will be registered for the wrong context.
```suggestion
// Fallback: No route found, return null
Logging.log('WebsiteToolsDiscovery', `No matching route found for ${pathname}`, 'warning');
return null;
```
How can I resolve this? If you propose a fix, please make it concise.
Comment on lines
+164
to
+166
| const schema = z.object({ | ||
| toolName: z.string(), | ||
| }).passthrough(); |
Contributor
There was a problem hiding this comment.
logic: passthrough() allows any additional properties, but the actual tool parameters from the manifest aren't being incorporated into the schema. The LLM won't know what parameters are required/optional for each tool.
Suggested change
| const schema = z.object({ | |
| toolName: z.string(), | |
| }).passthrough(); | |
| // Build dynamic schema from tool parameters | |
| const schemaFields: Record<string, z.ZodTypeAny> = { | |
| toolName: z.string(), | |
| }; | |
| // Add each parameter from the manifest to the schema | |
| for (const [paramName, paramDef] of Object.entries(tool.parameters)) { | |
| let paramSchema: z.ZodTypeAny; | |
| switch (paramDef.type) { | |
| case 'string': paramSchema = z.string(); break; | |
| case 'number': paramSchema = z.number(); break; | |
| case 'boolean': paramSchema = z.boolean(); break; | |
| case 'array': paramSchema = z.array(z.any()); break; | |
| case 'object': paramSchema = z.object({}).passthrough(); break; | |
| default: paramSchema = z.any(); | |
| } | |
| schemaFields[paramName] = paramDef.required ? paramSchema : paramSchema.optional(); | |
| } | |
| const schema = z.object(schemaFields); |
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/lib/tools/CreateWebMCPTools.ts
Line: 164:166
Comment:
**logic:** `passthrough()` allows any additional properties, but the actual tool parameters from the manifest aren't being incorporated into the schema. The LLM won't know what parameters are required/optional for each tool.
```suggestion
// Build dynamic schema from tool parameters
const schemaFields: Record<string, z.ZodTypeAny> = {
toolName: z.string(),
};
// Add each parameter from the manifest to the schema
for (const [paramName, paramDef] of Object.entries(tool.parameters)) {
let paramSchema: z.ZodTypeAny;
switch (paramDef.type) {
case 'string': paramSchema = z.string(); break;
case 'number': paramSchema = z.number(); break;
case 'boolean': paramSchema = z.boolean(); break;
case 'array': paramSchema = z.array(z.any()); break;
case 'object': paramSchema = z.object({}).passthrough(); break;
default: paramSchema = z.any();
}
schemaFields[paramName] = paramDef.required ? paramSchema : paramSchema.optional();
}
const schema = z.object(schemaFields);
```
How can I resolve this? If you propose a fix, please make it concise.| } | ||
| } | ||
|
|
||
| const eventName = toolName; |
Contributor
There was a problem hiding this comment.
logic: eventName is set to toolName, but the manifest defines eventName separately in the WebsiteTool schema. Should use tool.eventName instead.
Suggested change
| const eventName = toolName; | |
| const eventName = tool.eventName; |
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/lib/tools/CreateWebMCPTools.ts
Line: 63:63
Comment:
**logic:** `eventName` is set to `toolName`, but the manifest defines `eventName` separately in the `WebsiteTool` schema. Should use `tool.eventName` instead.
```suggestion
const eventName = tool.eventName;
```
How can I resolve this? If you propose a fix, please make it concise.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request includes capability for this agent, which allows the agent to detect tools if provided by a website to perform actions.
The is sample webiste which I made Sample Website, which is provides tools for the agent and agent performs actions only via these tools
Inorder for a website to be accessible: