Split Weekly Generation Pipeline #3430

ramsessanchez · 2025-10-22T08:38:47Z

Split Weekly generation pipeline into two separate pipelines, one to retrieve openApi updates and individually build the modules, and a second pipeline to generate the comandMetadata which requires all modules build on a single image.

The following yml files handle the following:
1.) downloading the most recent openApi docs
2.) determine which modules have been updated
3.) build the updated modules independently to validate openApi document correctness
4.) if all updated modules build successfully then a PR is created with the updated openApi documents.
Weekly-Generation.yml
Individualized-Workload-Modules.yml

The second will be triggered on merges to Dev and can also be triggered manually.
1.) All modules are built on a single image
2.) the commandMetadata is generated after all modules are built.
3.) successful completion of both results in a pull request into the Dev branch with an updated commandMetadata file.
Module-Metadata-generation.yml

Note: The second pipeline is still in testing as we can't test the pipeline fully until we have it registered as one of our pipelines in devops and add the appropriate token.

Split Weekly generation pipeline into two separate pipelines, one to retrieve openApi updates and individually build the modules. A second pipeline to take the current api docs and generate the commandMetadata when we approach a release date.

ramsessanchez · 2025-10-22T08:42:43Z

The following demonstrates ability to build modules individually:
https://microsoftgraph.visualstudio.com/Graph%20Developer%20Experiences/_build/results?buildId=200425&view=results
The following demonstrates ability to specify which modules to build and create pr:
https://microsoftgraph.visualstudio.com/Graph%20Developer%20Experiences/_build/results?buildId=200930&view=results
The following pr was made by the previous pipeline run, no chages to the openApi docs as updating the openApi doc step was intentionally skipped. #3429

Copilot

Pull Request Overview

This PR splits the weekly generation pipeline into two separate pipelines: one for OpenAPI document validation (weekly-generation.yml) and another for command metadata generation (module-metadata-generation.yml). The first pipeline downloads OpenAPI docs, identifies changed modules, builds them independently in parallel for validation, and creates a PR with updated OpenAPI documents. The second pipeline (triggered on merges to dev) builds all modules together and generates command metadata.

Key changes:

Introduced a matrix strategy to build modules independently in parallel based on detected OpenAPI changes
Created a new standalone pipeline for metadata generation that builds all modules on a single image
Added parameters to optionally specify which modules to generate for testing purposes

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
`.azure-pipelines/weekly-generation.yml`	Restructured to use matrix strategy for parallel module builds, removed metadata generation steps, added module specification parameters
`.azure-pipelines/module-metadata-generation.yml`	New pipeline for building all modules together and generating command metadata, triggered on dev branch merges
`.azure-pipelines/generation-templates/individualized-workload-modules.yml`	New template for building individual modules with module name and version parameters
`.azure-pipelines/common-templates/download-openapi-docs.yml`	Updated module change detection to include version suffix in module names

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

.azure-pipelines/common-templates/download-openapi-docs.yml

.azure-pipelines/generation-templates/individualized-workload-modules.yml

.azure-pipelines/module-metadata-generation.yml

MIchaelMainer · 2025-10-22T16:16:57Z

@ramsessanchez do we have any preliminary pipeline runs that use these updated scripts?

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

ramsessanchez · 2025-10-22T16:38:35Z

@ramsessanchez do we have any preliminary pipeline runs that use these updated scripts?

Assuming you're talking about the script in download-openApi .yml? No other pipelines use this outside of the weekly generation pipeline. The update focuses on making sure that the list of modules in the diff are listed with v1.0 or beta label, for the purpose of the module build matrix.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull Request Overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-10-28T21:04:15Z

.azure-pipelines/generation-templates/individualized-workload-modules.yml

+          targetType: inline
+          pwsh: true
+          script: |
+            . $(System.DefaultWorkingDirectory)/tools/GenerateModules.ps1 -SkipGeneration -Pack -ArtifactsLocation $(Build.ArtifactStagingDirectory)


The Pack step does not pass the -ModuleToGenerate and -ApiVersion parameters that were used in the Generate step (line 29). This will cause the Pack command to attempt packing all modules instead of just the specific module that was generated, which is inconsistent with the individualized approach.

Suggested change

. $(System.DefaultWorkingDirectory)/tools/GenerateModules.ps1 -SkipGeneration -Pack -ArtifactsLocation $(Build.ArtifactStagingDirectory)

. $(System.DefaultWorkingDirectory)/tools/GenerateModules.ps1 -SkipGeneration -Pack -ArtifactsLocation $(Build.ArtifactStagingDirectory) -ModuleToGenerate ${{ parameters.ModuleName }} -ApiVersion ${{ parameters.ModuleVersion }}

Copilot · 2025-10-28T21:04:15Z

.azure-pipelines/weekly-generation.yml

+                    moduleName    = $item
+                  }
+                } else {
+                  $name, $version = $item -split '_'


The script splits on underscore to extract name and version, but module names may contain underscores (e.g., 'Identity_SignIns'). This will incorrectly split such module names. Consider using -split '_', 2 to limit the split to a maximum of 2 parts, or use a different approach to extract the last segment as the version.

Suggested change

$name, $version = $item -split '_'

$segments = $item -split '_'

$version = $segments[-1]

$name = ($segments[0..($segments.Length - 2)] -join '_')

Copilot · 2025-10-28T21:04:15Z

.azure-pipelines/weekly-generation.yml

+                if ($item -notmatch '_') {
+                  # If '_' is not present, create two versions: v1.0 and beta
+                  $jsonOutput["${item}_v1.0"] = @{
+                    moduleVersion = 'v1.0'
+                    moduleName    = $item
+                  }
+                  $jsonOutput["${item}_beta"] = @{
+                    moduleVersion = 'beta'
+                    moduleName    = $item
+                  }
+                } else {
+                  $name, $version = $item -split '_'
+                  $jsonOutput[$item] = @{
+                    moduleVersion = $version
+                    moduleName    = $name
+                  }


The logic assumes that items without underscores should generate both v1.0 and beta versions. However, this conflicts with the updated download-openapi-docs.yml (line 78) which now always includes the version suffix. This means items will always contain '_', making this branch unreachable. Either remove this dead code or update the logic to handle the new format.

Suggested change

if ($item -notmatch '_') {

# If '_' is not present, create two versions: v1.0 and beta

$jsonOutput["${item}_v1.0"] = @{

moduleVersion = 'v1.0'

moduleName = $item

}

$jsonOutput["${item}_beta"] = @{

moduleVersion = 'beta'

moduleName = $item

}

} else {

$name, $version = $item -split '_'

$jsonOutput[$item] = @{

moduleVersion = $version

moduleName = $name

}

$name, $version = $item -split '_'

$jsonOutput[$item] = @{

moduleVersion = $version

moduleName = $name

gavinbarron · 2025-10-28T21:35:44Z

.azure-pipelines/common-templates/download-openapi-docs.yml

+            if ($_ -match 'openApiDocs\/(v1.0|beta)\/(.*)\.yml') {
+              $version = if ($matches[1] -eq 'v1.0') { 'v1.0' } else { 'beta' }
+              $moduleName = "$($matches[2])_$version"
+              if (!$ModulesWithChanges.ContainsKey($moduleName)) {
+                $ModulesWithChanges.Add($moduleName, $matches[1])
+              }


Seeing a script this complex inline in a pipeline makes me a bit nervous.
yaml is not a great authoring tool for multiline scripts.
If this were refactored into a stand-alone script that you could call then your can test it independent of a pipeline run.
Plus you could eliminate the if statment in here an use a condition on the step combined with setting the ModulesWithChanges variable to 'Skipped' by default.

Gotcha. Included the above runs as references showing that the scripts did work. Initially I was just editing the inline scripts that were already there, however as the complexity grew they definitely should have been moved to their own file. Will update with the changes.

gavinbarron · 2025-10-28T21:39:23Z

.azure-pipelines/weekly-generation.yml

+            targetType: inline
+            pwsh: true
+            script: |
+              if ($${{ parameters.SpecifyModules }}) {


Again, I'd refactor this to a script so that you can test it on your own without having to run a pipeline

ramsessanchez added 2 commits October 21, 2025 13:38

Split Weekly Generation Pipeline

fb83966

Split Weekly generation pipeline into two separate pipelines, one to retrieve openApi updates and individually build the modules. A second pipeline to take the current api docs and generate the commandMetadata when we approach a release date.

Merge branch 'dev' into ramsess/generateIndividualModules

1a288f3

ramsessanchez marked this pull request as ready for review October 22, 2025 08:39

ramsessanchez requested a review from a team as a code owner October 22, 2025 08:39

MIchaelMainer requested a review from Copilot October 22, 2025 16:11

Copilot AI reviewed Oct 22, 2025

View reviewed changes

ramsessanchez and others added 2 commits October 22, 2025 09:32

remove testing line

088d3f2

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

remove duplicate lines

68a7e8c

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

ramsessanchez and others added 2 commits October 22, 2025 09:41

update checkout step ref

c1ea55d

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update .azure-pipelines/module-metadata-generation.yml

6add689

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

ramsessanchez requested a review from Copilot October 28, 2025 21:01

Copilot AI reviewed Oct 28, 2025

View reviewed changes

gavinbarron reviewed Oct 28, 2025

View reviewed changes

	. $(System.DefaultWorkingDirectory)/tools/GenerateModules.ps1 -SkipGeneration -Pack -ArtifactsLocation $(Build.ArtifactStagingDirectory)
	. $(System.DefaultWorkingDirectory)/tools/GenerateModules.ps1 -SkipGeneration -Pack -ArtifactsLocation $(Build.ArtifactStagingDirectory) -ModuleToGenerate ${{ parameters.ModuleName }} -ApiVersion ${{ parameters.ModuleVersion }}

-                  $name, $version = $item -split '_'
+                  $segments = $item -split '_'
+                  $version = $segments[-1]
+                  $name = ($segments[0..($segments.Length - 2)] -join '_')

Split Weekly Generation Pipeline #3430

Are you sure you want to change the base?

Split Weekly Generation Pipeline #3430

Conversation

ramsessanchez commented Oct 22, 2025

Uh oh!

ramsessanchez commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MIchaelMainer commented Oct 22, 2025

Uh oh!

ramsessanchez commented Oct 22, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

gavinbarron Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

ramsessanchez Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

gavinbarron Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ramsessanchez commented Oct 22, 2025 •

edited

Loading