Skip to content

Conversation

@devin-ai-integration
Copy link
Contributor

Fix: Add multimodal tools support in Agent.kickoff() methods

Summary

Fixes issue #3936 where setting multimodal=True on an Agent did not add the AddImageTool when calling agent.kickoff() or agent.kickoff_async().

Key Changes:

  • Added _build_runtime_tools() helper method that combines tools from all sources (agent tools, platform tools, MCP tools, and multimodal tools) without mutating self.tools
  • Updated both kickoff() and kickoff_async() to use this helper method
  • Fixed inconsistency where kickoff_async() was not including platform/MCP tools (now it does, matching kickoff() behavior)
  • Added tool deduplication by name to prevent duplicate tools across multiple kickoff calls
  • Added comprehensive test coverage for multimodal tool handling

Before: When multimodal=True, the AddImageTool was never added to the agent's tools during kickoff execution.

After: When multimodal=True, the AddImageTool is properly included in the tools passed to LiteAgent, and calling kickoff multiple times doesn't duplicate tools or mutate self.tools.

Review & Testing Checklist for Human

⚠️ Important: I was unable to run tests locally due to environment issues with uv.lock, so CI validation is critical.

  • Verify CI tests pass - The new tests in test_agent_multimodal_kickoff.py should all pass
  • Manual multimodal test - Create an agent with multimodal=True, call kickoff() with an image URL, and verify the AddImageTool is actually available and works correctly
  • Test multiple kickoff calls - Verify that calling kickoff() multiple times on the same agent doesn't cause tool duplication or other side effects
  • Verify kickoff_async consistency - The PR also fixes kickoff_async() to include platform/MCP tools (previously missing). Test that agents with apps or mcps configured now work correctly with kickoff_async()
  • Review deduplication logic - The deduplication is done by tool name only (line 1447-1452 in core.py). Consider if this is sufficient or if we need more sophisticated logic (e.g., handling i18n tool names, or tools with same name but different implementations)

Test Plan

from crewai import Agent

# Test 1: Verify multimodal tool is added
agent = Agent(
    role="Image captioner",
    goal="caption images",
    backstory="You are used to caption images",
    multimodal=True
)

image_url = "https://upload.wikimedia.org/wikipedia/commons/thumb/4/4d/Cat_November_2010-1a.jpg/960px-Cat_November_2010-1a.jpg"
result = agent.kickoff(f"What's in this image? {image_url}")
print(result)  # Should successfully process the image

# Test 2: Verify no duplication on multiple calls
result2 = agent.kickoff(f"Describe this image: {image_url}")
print(result2)  # Should work without errors

# Test 3: Verify async works too
import asyncio
result3 = asyncio.run(agent.kickoff_async(f"Analyze this image: {image_url}"))
print(result3)

Notes

- Add _build_runtime_tools() helper method to combine tools from multiple sources
- Fix issue #3936 where multimodal=True was not adding AddImageTool in kickoff()
- Ensure kickoff_async() includes platform, MCP, and multimodal tools for consistency
- Prevent tool duplication by deduplicating tools by name
- Avoid mutating self.tools to prevent side effects across multiple kickoff calls
- Add comprehensive tests for multimodal tool handling in both sync and async kickoff methods

Co-Authored-By: João <joao@crewai.com>
@devin-ai-integration
Copy link
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

…s_platform_and_mcp_tools

- Change mcps parameter from ['test_mcp'] to ['crewai-amp:dummy'] to pass validation
- Add assertions to verify get_platform_tools and get_mcp_tools are called
- MCP references must start with 'https://' or 'crewai-amp:' per validation rules

Co-Authored-By: João <joao@crewai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant