Add AI tools integration documentation#10713
Draft
robertomonteromiguel wants to merge 4 commits intomasterfrom
Draft
Add AI tools integration documentation#10713robertomonteromiguel wants to merge 4 commits intomasterfrom
robertomonteromiguel wants to merge 4 commits intomasterfrom
Conversation
This change introduces comprehensive AI-assisted development support: - Add AI Tools Integration Guide documenting available tools and setup - Configure MCP servers for Datadog CI Visibility integration - Add Claude CLI configuration files - Update AGENTS.md with developer tools guidance (MCP, GitHub CLI, GitLab CLI) - Update CONTRIBUTING.md with link to AI tools documentation Made-with: Cursor
robertomonteromiguel
added
comp: testing
PerfectSlayer
added
the
tag: ai generated
Contributor
PerfectSlayer
left a comment
PerfectSlayer
left a comment
There was a problem hiding this comment.
Few comment from early review.
As a side note, it seems Cursor dose not follow the PR labeling rules about AI... Nor followed the template. That's annoying and would have to have a look at enforcing it.
Comment on lines
+3
to
+6
| "enabledMcpjsonServers": [ | ||
| "datadog" | ||
| ], | ||
| "enableAllProjectMcpServers": true |
Contributor
There was a problem hiding this comment.
💭 thought: Do we want to have MCP? And more over, enabled by default?
They fill up the context pretty quick and are less effective than CLI or skills with commands.
Contributor
Author
There was a problem hiding this comment.
I tried it out and I think it provides interesting tools for exploiting CI Visibility data. By default, our pipelines are reporting to CI Visibility. This MCP doesn't expose too many tools, so I don't think it overloads the context. I think it's a good improvement.
|
|
||
| **Available MCP Servers:** | ||
|
|
||
| - **Datadog MCP Server**: Provides CI Visibility integration including pipeline event search and aggregation, test event analysis, flaky test detection, code coverage summaries, and PR insights. Pre-configured in the repository via `.mcp.json` (for Claude CLI and other compatible tools). Cursor users need to set up `.cursor/mcp.json` manually (see setup instructions below). |
Contributor
There was a problem hiding this comment.
❔ question: What the benefits of this dedicated MCP compared to the datadog market place skills? It looks duplicate.
Contributor
Author
There was a problem hiding this comment.
I don't know if any of the skills in the Datadog Claude Marketplace provide access to CI Visibility. In any case, I usually use Cursor, and I can't use Claude plugins in Cursor.
Contributor
There was a problem hiding this comment.
CI Visibility is indeed supported from our plugin market place.
|
|
||
| #### Datadog MCP Server Setup for dd-trace-java | ||
|
|
||
| The Datadog MCP server connects Cursor's/Claude AI assistant to Datadog's CI Visibility platform, letting you query pipeline events, test results, flaky tests, code coverage, and PR insights directly from the chat. |
Contributor
There was a problem hiding this comment.
❔ question: Is this configuration available for all contributors, even not from Datadog?
Contributor
Author
There was a problem hiding this comment.
I don't think anyone outside the organization can access our pipelines, let alone GitLab. But having the MCP configured by default won't affect development. If Claude or Cursor detect that the MCP isn't working, they'll try to use other mechanisms to resolve user queries.
Comment on lines
+35
to
+46
| The repository includes the MCP server configuration for Claude CLI and other compatible tools in `.mcp.json`: | ||
|
|
||
| ```json | ||
| { | ||
| "mcpServers": { | ||
| "datadog": { | ||
| "type": "http", | ||
| "url": "https://mcp.datadoghq.com/api/unstable/mcp-server/mcp?toolsets=software-delivery" | ||
| } | ||
| } | ||
| } | ||
| ``` |
Contributor
There was a problem hiding this comment.
❔ question: What's the benefit of repeating what's already in VCS?
Comment on lines
+120
to
+122
| ```bash | ||
| brew install gh | ||
| ``` |
Contributor
There was a problem hiding this comment.
📝 notes: If we start requiring some new tools, we will need to provide installation documentation for all operating systems and configuration.
Contributor
Author
There was a problem hiding this comment.
you're right, I guess the solution would be to link to the GitHub or GitLab page where the installation is detailed.
Contributor
There was a problem hiding this comment.
We detailed installation process from our documentation rather than sending to the application page.
Check https://github.com/DataDog/dd-trace-java/blob/master/BUILDING.md for example.
Contributor
Author
There was a problem hiding this comment.
I think in this case we may be talking about slightly different things. The build process is a core part of the project and usually involves project-specific configuration, so it makes sense to document it in detail.
For the GitHub and GitLab CLIs, there isn’t really any repo-specific setup required. That’s why I’m not sure it’s worth detailing the whole process here. In my view, linking to the official installation pages should be sufficient, and we can always expand the docs later if we find people are getting stuck.
Comment on lines
+95
to
+96
| - When interacting with GitLab (gitlab.ddbuild.io), ALWAYS use `glab` on the command line. | ||
| - If `glab` CLI is not installed or there are authentication or configuration problems, check the document: [AI Tools Integration Guide - GitLab CLI](docs/ai-tools-integration-guide.md#gitlab-cli-glab) |
Contributor
There was a problem hiding this comment.
❔ question: Will this work for all contributors? Including the ones not from Datadog?
Contributor
Author
There was a problem hiding this comment.
It won’t work outside the organization. I don’t see any issue with that. Even within the org, if you don’t have the CLI tool and/or the MCP configured, the AI will either suggest setting them up or tell you it can’t access the data you’re asking for.
Contributor
There was a problem hiding this comment.
I don’t see any issue with that
We're Open Source software repository. That's not very welcoming to the community 😕
| ### Handling the Pull Request and CI | ||
|
|
||
| - To create and monitor PR data, use the gh client (GitHub Tools). | ||
| - If the request relates to CI status for a branch or PR, MUST ALWAYS try the Datadog MCP first. |
Contributor
There was a problem hiding this comment.
🎯 suggestion: I would rather use the skill from the Datadog plugin (or even mine) rather than relying on an MCP.
Contributor
Author
There was a problem hiding this comment.
Among other differences, the mcp server can be used by any client, and the claude marketplace is only for claude.
Contributor
There was a problem hiding this comment.
But it's making Claude code, the recommended AI coded tool by the company so far, behaving significantly worse from experience with this repository. I would rather not have it enabled by default in such case.
What about using other tool hooks to setup their specific config for example?
That would leave the best experience for the default tool and allow others to setup their copies for specific tool if needed?
Contributor
Author
There was a problem hiding this comment.
I completely agree that we should NOT enable the MCP server by default, and instead let developers opt in if they want to use it. (I will update the PR)
That said, I still believe the MCP server itself shouldn’t negatively impact the repository context, and I’d still lean towards using standard, vendor-neutral tooling whenever possible, rather than approaches that are tightly coupled to a specific provider.
Comment on lines
+84
to
+92
| ## AI-assisted development | ||
|
|
||
| This project includes documentation and configurations to help you use AI coding assistants effectively. | ||
|
|
||
| Check [AI Tools Integration Guide](docs/ai-tools-integration-guide.md) for information about: | ||
| * Project-specific AI context and conventions | ||
| * How to use AI tools with dd-trace-java | ||
| * Best practices for AI-assisted contributions | ||
|
|
Contributor
There was a problem hiding this comment.
💭 thought: Not sure we went to advocate for AI-assisted development on the repo.
It feels to early and 3rd party contributions already require a good amount of time to get merged.
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 62 metrics, 9 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.60.0-SNAPSHOT~5e26682fbc, baseline=1.60.0-SNAPSHOT~1c44c6f9a7
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.065 s) : 0, 1064963
Total [baseline] (10.994 s) : 0, 10993671
Agent [candidate] (1.068 s) : 0, 1068496
Total [candidate] (10.97 s) : 0, 10970192
section appsec
Agent [baseline] (1.236 s) : 0, 1235871
Total [baseline] (10.989 s) : 0, 10989000
Agent [candidate] (1.238 s) : 0, 1237675
Total [candidate] (11.091 s) : 0, 11090783
section iast
Agent [baseline] (1.247 s) : 0, 1246849
Total [baseline] (11.269 s) : 0, 11269040
Agent [candidate] (1.231 s) : 0, 1230850
Total [candidate] (11.199 s) : 0, 11198655
section profiling
Agent [baseline] (1.208 s) : 0, 1208058
Total [baseline] (10.924 s) : 0, 10924101
Agent [candidate] (1.193 s) : 0, 1192776
Total [candidate] (10.942 s) : 0, 10942209
gantt
title petclinic - break down per module: candidate=1.60.0-SNAPSHOT~5e26682fbc, baseline=1.60.0-SNAPSHOT~1c44c6f9a7
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.184 ms) : 0, 1184
crashtracking [candidate] (1.189 ms) : 0, 1189
BytebuddyAgent [baseline] (627.117 ms) : 0, 627117
BytebuddyAgent [candidate] (629.375 ms) : 0, 629375
AgentMeter [baseline] (29.094 ms) : 0, 29094
AgentMeter [candidate] (29.11 ms) : 0, 29110
GlobalTracer [baseline] (257.62 ms) : 0, 257620
GlobalTracer [candidate] (258.26 ms) : 0, 258260
AppSec [baseline] (33.008 ms) : 0, 33008
AppSec [candidate] (33.128 ms) : 0, 33128
Debugger [baseline] (63.363 ms) : 0, 63363
Debugger [candidate] (64.387 ms) : 0, 64387
Remote Config [baseline] (626.295 µs) : 0, 626
Remote Config [candidate] (611.425 µs) : 0, 611
Telemetry [baseline] (12.315 ms) : 0, 12315
Telemetry [candidate] (9.983 ms) : 0, 9983
Flare Poller [baseline] (4.537 ms) : 0, 4537
Flare Poller [candidate] (6.254 ms) : 0, 6254
section appsec
crashtracking [baseline] (1.186 ms) : 0, 1186
crashtracking [candidate] (1.215 ms) : 0, 1215
BytebuddyAgent [baseline] (655.054 ms) : 0, 655054
BytebuddyAgent [candidate] (655.764 ms) : 0, 655764
AgentMeter [baseline] (11.996 ms) : 0, 11996
AgentMeter [candidate] (11.947 ms) : 0, 11947
GlobalTracer [baseline] (257.837 ms) : 0, 257837
GlobalTracer [candidate] (258.463 ms) : 0, 258463
IAST [baseline] (25.38 ms) : 0, 25380
IAST [candidate] (25.449 ms) : 0, 25449
AppSec [baseline] (167.831 ms) : 0, 167831
AppSec [candidate] (168.393 ms) : 0, 168393
Debugger [baseline] (66.868 ms) : 0, 66868
Debugger [candidate] (66.651 ms) : 0, 66651
Remote Config [baseline] (674.271 µs) : 0, 674
Remote Config [candidate] (697.553 µs) : 0, 698
Telemetry [baseline] (9.328 ms) : 0, 9328
Telemetry [candidate] (9.377 ms) : 0, 9377
Flare Poller [baseline] (3.661 ms) : 0, 3661
Flare Poller [candidate] (3.701 ms) : 0, 3701
section iast
crashtracking [baseline] (1.212 ms) : 0, 1212
crashtracking [candidate] (1.188 ms) : 0, 1188
BytebuddyAgent [baseline] (805.766 ms) : 0, 805766
BytebuddyAgent [candidate] (794.732 ms) : 0, 794732
AgentMeter [baseline] (11.865 ms) : 0, 11865
AgentMeter [candidate] (11.305 ms) : 0, 11305
GlobalTracer [baseline] (249.545 ms) : 0, 249545
GlobalTracer [candidate] (247.43 ms) : 0, 247430
IAST [baseline] (27.496 ms) : 0, 27496
IAST [candidate] (27.003 ms) : 0, 27003
AppSec [baseline] (35.699 ms) : 0, 35699
AppSec [candidate] (33.328 ms) : 0, 33328
Debugger [baseline] (66.228 ms) : 0, 66228
Debugger [candidate] (67.309 ms) : 0, 67309
Remote Config [baseline] (544.474 µs) : 0, 544
Remote Config [candidate] (531.282 µs) : 0, 531
Telemetry [baseline] (8.592 ms) : 0, 8592
Telemetry [candidate] (8.533 ms) : 0, 8533
Flare Poller [baseline] (3.473 ms) : 0, 3473
Flare Poller [candidate] (3.383 ms) : 0, 3383
section profiling
crashtracking [baseline] (1.187 ms) : 0, 1187
crashtracking [candidate] (1.173 ms) : 0, 1173
BytebuddyAgent [baseline] (693.649 ms) : 0, 693649
BytebuddyAgent [candidate] (683.267 ms) : 0, 683267
AgentMeter [baseline] (8.73 ms) : 0, 8730
AgentMeter [candidate] (8.59 ms) : 0, 8590
GlobalTracer [baseline] (219.16 ms) : 0, 219160
GlobalTracer [candidate] (216.344 ms) : 0, 216344
AppSec [baseline] (33.081 ms) : 0, 33081
AppSec [candidate] (32.541 ms) : 0, 32541
Debugger [baseline] (67.798 ms) : 0, 67798
Debugger [candidate] (67.128 ms) : 0, 67128
Remote Config [baseline] (630.92 µs) : 0, 631
Remote Config [candidate] (628.04 µs) : 0, 628
Telemetry [baseline] (8.947 ms) : 0, 8947
Telemetry [candidate] (8.988 ms) : 0, 8988
Flare Poller [baseline] (3.791 ms) : 0, 3791
Flare Poller [candidate] (3.752 ms) : 0, 3752
ProfilingAgent [baseline] (99.055 ms) : 0, 99055
ProfilingAgent [candidate] (99.547 ms) : 0, 99547
Profiling [baseline] (99.623 ms) : 0, 99623
Profiling [candidate] (100.126 ms) : 0, 100126
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.60.0-SNAPSHOT~5e26682fbc, baseline=1.60.0-SNAPSHOT~1c44c6f9a7
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.065 s) : 0, 1064689
Total [baseline] (8.737 s) : 0, 8736867
Agent [candidate] (1.067 s) : 0, 1066965
Total [candidate] (8.75 s) : 0, 8749923
section iast
Agent [baseline] (1.24 s) : 0, 1240279
Total [baseline] (9.415 s) : 0, 9414977
Agent [candidate] (1.233 s) : 0, 1233331
Total [candidate] (9.369 s) : 0, 9369311
gantt
title insecure-bank - break down per module: candidate=1.60.0-SNAPSHOT~5e26682fbc, baseline=1.60.0-SNAPSHOT~1c44c6f9a7
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.192 ms) : 0, 1192
crashtracking [candidate] (1.198 ms) : 0, 1198
BytebuddyAgent [baseline] (628.104 ms) : 0, 628104
BytebuddyAgent [candidate] (628.774 ms) : 0, 628774
AgentMeter [baseline] (29.223 ms) : 0, 29223
AgentMeter [candidate] (29.124 ms) : 0, 29124
GlobalTracer [baseline] (257.43 ms) : 0, 257430
GlobalTracer [candidate] (258.56 ms) : 0, 258560
AppSec [baseline] (33.014 ms) : 0, 33014
AppSec [candidate] (32.889 ms) : 0, 32889
Debugger [baseline] (63.564 ms) : 0, 63564
Debugger [candidate] (66.789 ms) : 0, 66789
Remote Config [baseline] (624.001 µs) : 0, 624
Remote Config [candidate] (609.447 µs) : 0, 609
Telemetry [baseline] (11.438 ms) : 0, 11438
Telemetry [candidate] (9.044 ms) : 0, 9044
Flare Poller [baseline] (3.8 ms) : 0, 3800
Flare Poller [candidate] (3.748 ms) : 0, 3748
section iast
crashtracking [baseline] (1.216 ms) : 0, 1216
crashtracking [candidate] (1.241 ms) : 0, 1241
BytebuddyAgent [baseline] (801.623 ms) : 0, 801623
BytebuddyAgent [candidate] (797.094 ms) : 0, 797094
AgentMeter [baseline] (11.549 ms) : 0, 11549
AgentMeter [candidate] (11.32 ms) : 0, 11320
GlobalTracer [baseline] (249.22 ms) : 0, 249220
GlobalTracer [candidate] (248.207 ms) : 0, 248207
AppSec [baseline] (33.515 ms) : 0, 33515
AppSec [candidate] (33.793 ms) : 0, 33793
Debugger [baseline] (66.818 ms) : 0, 66818
Debugger [candidate] (65.761 ms) : 0, 65761
Remote Config [baseline] (547.85 µs) : 0, 548
Remote Config [candidate] (544.29 µs) : 0, 544
Telemetry [baseline] (8.819 ms) : 0, 8819
Telemetry [candidate] (8.692 ms) : 0, 8692
Flare Poller [baseline] (3.474 ms) : 0, 3474
Flare Poller [candidate] (3.443 ms) : 0, 3443
IAST [baseline] (27.307 ms) : 0, 27307
IAST [candidate] (27.114 ms) : 0, 27114
LoadParameters
See matching parameters
SummaryFound 1 performance improvements and 2 performance regressions! Performance is the same for 17 metrics, 16 unstable metrics.
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.60.0-SNAPSHOT~5e26682fbc, baseline=1.60.0-SNAPSHOT~1c44c6f9a7
dateFormat X
axisFormat %s
section baseline
no_agent (1.173 ms) : 1162, 1184
. : milestone, 1173,
iast (3.074 ms) : 3033, 3115
. : milestone, 3074,
iast_FULL (5.787 ms) : 5728, 5845
. : milestone, 5787,
iast_GLOBAL (3.558 ms) : 3495, 3621
. : milestone, 3558,
profiling (2.163 ms) : 2143, 2183
. : milestone, 2163,
tracing (1.801 ms) : 1786, 1815
. : milestone, 1801,
section candidate
no_agent (1.2 ms) : 1189, 1212
. : milestone, 1200,
iast (3.288 ms) : 3245, 3332
. : milestone, 3288,
iast_FULL (5.758 ms) : 5701, 5815
. : milestone, 5758,
iast_GLOBAL (3.364 ms) : 3320, 3408
. : milestone, 3364,
profiling (2.165 ms) : 2143, 2186
. : milestone, 2165,
tracing (1.775 ms) : 1760, 1791
. : milestone, 1775,
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.60.0-SNAPSHOT~5e26682fbc, baseline=1.60.0-SNAPSHOT~1c44c6f9a7
dateFormat X
axisFormat %s
section baseline
no_agent (18.557 ms) : 18365, 18748
. : milestone, 18557,
appsec (18.563 ms) : 18375, 18751
. : milestone, 18563,
code_origins (17.442 ms) : 17270, 17614
. : milestone, 17442,
iast (17.854 ms) : 17672, 18035
. : milestone, 17854,
profiling (20.161 ms) : 19950, 20371
. : milestone, 20161,
tracing (17.541 ms) : 17367, 17715
. : milestone, 17541,
section candidate
no_agent (19.034 ms) : 18842, 19225
. : milestone, 19034,
appsec (18.543 ms) : 18353, 18733
. : milestone, 18543,
code_origins (17.508 ms) : 17335, 17681
. : milestone, 17508,
iast (17.602 ms) : 17428, 17777
. : milestone, 17602,
profiling (19.447 ms) : 19253, 19640
. : milestone, 19447,
tracing (17.531 ms) : 17358, 17704
. : milestone, 17531,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics. Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.60.0-SNAPSHOT~5e26682fbc, baseline=1.60.0-SNAPSHOT~1c44c6f9a7
dateFormat X
axisFormat %s
section baseline
no_agent (15.337 s) : 15337000, 15337000
. : milestone, 15337000,
appsec (14.888 s) : 14888000, 14888000
. : milestone, 14888000,
iast (18.364 s) : 18364000, 18364000
. : milestone, 18364000,
iast_GLOBAL (17.824 s) : 17824000, 17824000
. : milestone, 17824000,
profiling (14.697 s) : 14697000, 14697000
. : milestone, 14697000,
tracing (14.511 s) : 14511000, 14511000
. : milestone, 14511000,
section candidate
no_agent (15.424 s) : 15424000, 15424000
. : milestone, 15424000,
appsec (14.986 s) : 14986000, 14986000
. : milestone, 14986000,
iast (18.364 s) : 18364000, 18364000
. : milestone, 18364000,
iast_GLOBAL (17.968 s) : 17968000, 17968000
. : milestone, 17968000,
profiling (14.951 s) : 14951000, 14951000
. : milestone, 14951000,
tracing (14.658 s) : 14658000, 14658000
. : milestone, 14658000,
Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.60.0-SNAPSHOT~5e26682fbc, baseline=1.60.0-SNAPSHOT~1c44c6f9a7
dateFormat X
axisFormat %s
section baseline
no_agent (1.465 ms) : 1453, 1476
. : milestone, 1465,
appsec (3.797 ms) : 3577, 4016
. : milestone, 3797,
iast (2.253 ms) : 2184, 2322
. : milestone, 2253,
iast_GLOBAL (2.293 ms) : 2223, 2362
. : milestone, 2293,
profiling (2.125 ms) : 2068, 2182
. : milestone, 2125,
tracing (2.058 ms) : 2005, 2111
. : milestone, 2058,
section candidate
no_agent (1.474 ms) : 1463, 1486
. : milestone, 1474,
appsec (3.787 ms) : 3567, 4007
. : milestone, 3787,
iast (2.249 ms) : 2180, 2318
. : milestone, 2249,
iast_GLOBAL (2.298 ms) : 2228, 2367
. : milestone, 2298,
profiling (2.073 ms) : 2019, 2128
. : milestone, 2073,
tracing (2.055 ms) : 2002, 2108
. : milestone, 2055,
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces comprehensive AI-assisted development support to dd-trace-java:
docs/ai-tools-integration-guide.md) covering available AI tools: Claude CLI, MCP server integration, and CLI tool setup (gh, glab).mcp.jsonand.cursor/mcp.jsonfor Datadog CI Visibility integration.claude/settings.jsonfor Claude CLI behavior settingsChanges
New Files
docs/ai-tools-integration-guide.md- Comprehensive guide for AI tools integration.mcp.json- MCP server configuration for Claude CLI and compatible tools.claude/settings.json- Claude CLI settingsModified Files
AGENTS.md- Added Developer Tools sectionCONTRIBUTING.md- Added AI-assisted development sectionNotes
.cursor/mcp.jsonfile is intentionally not included as.cursor/is in.gitignoreMade with Cursor