RFC: Automate the Project Board to monitor contributions

[matteogrs]

What is the motivation for this suggestion?

We start summarizing the process that is in place for contributing code to the O3DE repositories. In particular, we are going to identify three different phases that happen between two major releases of the engine, in order to highlight the actions that are performed by the involved actors and which of them can become critical for the entire process.

It is important to note that we don't aim to provide here a full description of the contribution process, but just a brief overview of the context in which this RFC is applied. For further details, please refer to the Contribution Guidelines and SIG-Release Process Docs.

Current process

Let's assume that the latest stable release is publicly available. The following steps are expected for enhancing the next version of the engine:

1. A User opens a new issue selecting the desired template: (1) report a bug, or (2) suggest a new feature. The need-triage and need-sig labels are assigned automatically.
2. During the weekly meetings, a Special Interest Group (SIG) Chair/Co-Chair removes the need-* labels and assign the issue to the correct SIG(s), by adding the relative sig/<name> and triage/accepted labels.
3. Eventually, a Contributor picks up and solves the issue. The solution is contributed back to the main repository by opening a Pull Request (PR) toward the development branch.
4. Two Maintainers review and accept the PR, which can be merged into the codebase only if it passes all tests on he Automated Review (AR).

When a date for the next major release is picked:

1. The Release Manager creates a Project Board to list the important governance tasks, and a Milestone to track bugs.
2. A stabilization/<version> branch is created from development for each repository that is part of the release.
3. Contributors focus on testing the code that is part of the stabilization branch, in order to find any minor / major / critical bug.
4. When an error is found, it is reported following the above point (1). It is very important that the Contributor remembers to fill the Found in Branch section in the issue template, in order to distinguish between this phase and the previous one.
5. The Release Manager adds the issue to the Milestone to track its progress.
6. The issue is fixed following the standard process, exception that PR is created against stabilization rather than development.

When the release date is approaching:

1. An Exception Board is used to filter which PR can still be accepted into the stabilization branch, according to the severity of the bug.
2. The Release Manager creates a Feature List that describes all the changes since the last major release, using the PRs that were created over the past months. All SIG Chairs/Co-Chairs can provide additional details about them. Finally, the Documentation Manager convert it to a full Release Notes for the site.
3. The Release Manager makes a determination on the stability of the code, according to the bugs that are still open.
4. If everything is fine, the stabilized code is merged into the main branch and released. The stabilization branch is then cancelled.

Please remember that the first phase is always active, even while the other two ones are running. Periodic synchronizations between branches are done by cherry-picking changes back from stabilization to development.

Critical points

1. Provisioning of data requires actors to perform a manual operation on issues or PRs, such as writing a valid text, assigning the correct label or milestone. When it is missed due to other duties, information about that contribution may get lost.
2. Retrieving planned / implemented features requires inspecting every PRs, since information is manly included in their text fields.
3. Monitoring issues and PRs that are related to a specific version requires to generate queries that combine multiple fields, such as labels, creation date, etc.

Suggestion design description:

In the past months, GitHub had launched new features that we can leverage to automate parts of the issue management. They could be used to address the above critical points, introducing only a minimal additional effort for Contributors (see chapter about disadvantages).

1. The issue template for bugs is enhanced to use issue forms, which provides a validation on input data rather than the classical markdown text. Common widgets (e.g. dropdown, checkbox) are used to restrict the supported values, so that it is simpler to automate them. In particular:

   • Set the existing Found in Branch field to a set of multiple radiobuttons to distinguish between the affected engine versions and the valid ones. Since O3DE doesn't have a Long-Term Support (LTS), the following versions are suggested:

     • <latest_stable_number> (stable)
     • <candidate_stabilization_number> (stabilization)
     • development

     For each of them, the following values are defined:

     • yes
     • no
     • unknown

2. Existing labels for code repositories have to be revisited in order to describe the information that are currently available only in the text fields of issues and PRs (e.g. descriptions, titles). In particular:

   • Add a new set for describing the affected versions:
     • branch/stable
     • branch/stabilization
     • branch/development
   • Add a new set for describing the versions that still need to be double-checked for a reported bug:
     • need-check/stable
     • need-check/stabilization
     • need-check/development
   • Add a new set for marking PRs that were merged in a branch, but they still need to be ported to another branch by a cherry-pick:
     • need-sync/to-development: merged in stabilization, but it needs need to be ported to development;
     • need-sync/to-stabilization: merged in development, but it needs to be ported to stabilization
   • Add a new set for keeping track of which PRs were ported between branches in order to keep them up-to-date with the latest fixes:
     • sync/to-development: it was ported from stabilization to development;
     • sync/to-stabilization: it was ported from development to stabilization
       This label is mutually exclusive with the previos set need-sync/*. In particular, it has to replace it when the porting operation is completed.
   • Split the feature/<name> set into two hierarchical ones:
     • category/<category_name> - Describe one main area of the engine (e.g. audio, gameplay, graphics, physics, network)
     • subcategory/<category_name>/<feature_name> - Describe one specific feature inside a main area (e.g. physics/rigid-body, physics/kinematics-body)
       Due to the amount of features that need to be identified, the full list of new labels will be provided at a second time during the approval process of this RFC.
   • Deprecate kind/bug, kind/feature, and others in the kind/* set in favor of the new Types field. The new UI interface for issues already provides a dedicated filter for this new property, which will allow to reduce the number of labels that are displayed near each issue title.

3. New GitHub Actions are activated to help assigning the correct labels without requiring any manual effort. In particular, the following Actions are good candidates for the following user-cases, since they have a permissive license and are already available:

   • Advanced Issue Labeler - Assign one or more branch/* labels to issues according to which versions are set to Yes in the Found in Branch field. In the same way, add one or more need-check/* labels when the value is set to Unknown.
   • Pull Request Labeler - Assign one branch/* label to PRs according to the target branch name. For all features that are self-contained in the codebase, it may be feasible to apply category/* and subcategory/* according to the changed file paths. Otherwise, the Maintainer has to set the correct values manually when reviewing the PR.

4. The Project Board for stabilization (YY.MM Release) is extended to include also the bug issues, rather than only the governance tasks. The loading can be automated using an auto-add workflow that looks for the branch/stabilization label on the new created issues, and automatically add them to the project when it is found.

5. Due to the amount of items that will be displayed. the previous Project Board is split into five dedicated vies that match the different phases of the release process, as they are described in Major Release Process - v1.3 (May 25, 2023):

   • Planning
   • Stabilization
   • Pre-Release
   • Release Day
   • Post-Release

   Each view is shown as dedicated Board with three columns (Todo, In Progress and Done), excepting for the stabilization one that uses List to better display issues as table rows. The closed status can be automated using built-in automations that is provided by GitHub Projects.

6. The Milestone is converted to collect PRs rather than issues. This approach allows to separate the proposal (project board) from the actual implementation (milestone), tracking whether a fix is really part of the final release or not. In fact, after that a bug is found (opening an issue), the following scenarios may happen:

   • The issue is not taken by anyone.
   • The PR with the fix is open, but it isn't reviewed yet.
   • The PR is approved, but it is still waiting to pass AR.
   • The PR is merged, but the related issue wasn't closed.

   Since we would count only merged PRs in the Milestone, it will be safe to assume that all the planned fixes have successfully reached the stabilization branch when the progression value reaches 100%. Otherwise, there might still be some pending PRs that were created but are waiting to be merged.

7. A new Project Board for development (Next Release) is created alongside the existing one. It contains only the second view that is described at point (5) (i.e Stabilization, which is now renamed to Development). Its purpose is to collect all the issues that are open against the development branch at any time. Moreover, it can also receive those issues that are rejected from the stabilization branch, due to time constraints, lower severity, etc. When a new major release is announced:

   • This project board is renamed to YY.MM + 1 Release
   • The required views for governance are created in this project board, with all the required tasks.
   • The auto-add workflow for issues is switched from branch/development to branch/stabilization, in order to collect only new bug issues. Note that all the issues that were previously open during the development phase are still there after the switch.
   • A new project board Next Release is created from scratch to start collecting again new issues from development.
   • The Release Manager and SIG Chair/Co-Chairs can postpone the resolution of some bugs by transferring issues between the first board and the second one.

What are the advantages of the suggestion?

The Project Board becomes a single-source-of-truth to monitor the status of the O3DE project. Each actor can easily filter data according to their needs:

• Contributors can see which features are under development, and are planned for the current release or the next one.
• Maintainers can monitor incoming bugs and assign different priorities.
• The Release Manager can extract information for the Feature List, and track the progression of the stabilization process.

What are the disadvantages of the suggestion?

Most of the heavy lifting for labeling issues and PRs can be automated with GitHub Actions, but it is still required that labels for category/subcategory are applied by Maintainers manually. In fact, Users would not have the permissions to add labels on their own PRs, and the process cannot be automated any further since YAML syntax isn't currently supported in PR.

What is the strategy for adoption?

Due to the amount of existing issues (3k+ open / 4k+ closed), the new process will be applied only to new issues and PRs. This means that the new process can be fully effective only at the second major release after that it is activated.

The documentation at SIG-Release Process Docs will be updated in order to describe the new process.

[matteogrs]

• Rename version/* labels to branch/*, since they are more related to the branch where the issue was found. It also free the version prefix for future uses (e.g. version/<number>).
• Convert Found in Branch from checkboxes to multiple groups of radios with three states (yes, no, unknown). Users report bugs for the version that they are currently using (e.g. development), but it is unlikely that they can try also on another one (e.g. stabilization). Thus, it is important to track if the double-check has still to be done (adding the new labels need-check/*) or not.

[matteogrs]

• Split synchronization label into sync/to-development and sync/to-stabilization, since there are some cases in which a fix is merged into development, but it is also applicable for stabilization. For instance, if the stabilization branch is cut while a contributor is working on a long feature with multiple PRs, it may happen that the first PRs are part of the upcoming release but the latest PRs are lost. Thus, they have to manually cherry-picked from development to stabilization.
• Add need-sync/* to mark which PRs are eligible for porting (cherry-picking) from a branch to another one. The label has to be switched to sync/* when the operation is completed.

[matteogrs]

The following labels have been created in o3de/o3de repository, in order to start testing the new process manually

• branch/main
• branch/development
• branch/stabilization
• need-check/main
• need-check/development
• need-check/stabilization
• need-sync/to-development
• need-sync/to-stabilization
• sync/to-development
• sync/to-stabilization

The YY.MM Release Board is updated as follows:

• Status field is splitted into Phase and Status.
• New dedicated views are created for each phase. A common view is still used to show all governance tasks using Status as columns and grouped by Pahse.

[jhanca-robotecai]

A few comments/questions about the implementation:

1. Shouldn't we start implementing the idea with the automation and then add labels manually to the old issues/PRs? You added the labels manually, now the maintainers/developers/users will have to do the same for each new PR/issue. Are we ready for this additional effort and is it worth it?
2. Is the sync/* useful? Getting the list of PRs that got into the stabilization takes no time. I would still use git instead of the label when doing the sync, as I do not want to miss any PR due to invalid labels (each commit requires verification before the cherry-pick anyway).

git log --oneline development..stabilization/25050

e7c4c700f8 Fix a few ASan and other corruption errors found. (#18819)
722a7de027 Fixes a buffer overrun ASan error in LUA Script Properties (#18807)
5855f6b946 Fixes compilation if test modules are disabled. (#18786)
1371bd7784 Fixes containers (vectors, maps, etc) not saving state or undos when modified in the inspector (#18802)
693243153c (Stabilization) BUGFIX: Fixes a crash when editing overrides on a prefab instance. (#18795)
b99901dc02 Fix lua editor connectivty with ringbuffer error message (#18782)
b534babb0c Fixes intermittent test failures (#18760) (#18797)
543cb0e030 BUGFIX - Crash on CTRL+G if SC is open. (#18791)
8d26995eb4 [Track View] Cherry-Pick recent Track View Changes from development to stabilization/25050. (#18788)
fd6a5a4c6a Fix lua script initialization properties (fixes #10076) (#18771)
12e90d8799 ScriptCanvas: Fix crash on SC editor close (#18734) (#18758)
9b8d468f64 Fixes a issue #18720 (vulkan malloc crash) and compile error (#18736)

3. Are you sure labels branch/* will help? We still have some issues from 2021 in our backlog. Keeping in mind we do point releases, I am afraid branch/main will become meaningless quickly.
4. need-check/* labels duplicate the information from branch/* (an issue found in A always needs a confirmation in B and C)

[matteogrs]

Thanks @jhanca-robotecai for the feedback, here my answers to your questions:

1. It has been chosen to start implementing this RFC incrementally, in order to minimize the impacts on the upcoming release. In particular:

   • the automation requires some development taks, which can be in conflict with the effort that is already planned for the stabilization phase;
   • maintainers are now responsible for the entire triage process, and have a prior experience to handle troubles. The automation aims to simplify parts of their tasks, but it may also add a new variable to the equation. If something get wrong in the implementation, we may risk to lose issues, time, etc. which is not desired in this phase, but it can be postponed to a quieter one (June-July).

   Moreover, after the initial proposal, I had to make some additional refinements to address some corner cases that happen during the last weeks. I'm expecting that something else can happen in the next month, so I would like to have a more stable version of the automated process before starting to write its code.

   Regarding the additional effort, now I'm applying myself the process manually on a regular basis (~2 times / week), in order to have a PoC that can be showcased at the next meetings and see if there is still any pitfall. At the moment it is not required for maintainers to use the new labels (users won't have permissions), but they are welcome to opt-in if they want!

2. The sync/* labels are mainly intended for filtering the history when generating the Release Notes, and have to be applied after that the PR is merged into the destination branch.

   I give you a real example to explain better the scenario:
   Fix lua script initialization properties (fixes #10076) o3de#18771 was fixed in stabilization as part of the upcoming release. So, it will be included in the Release Notes in May. However, it has also to be brought back to development by cherry-picking with (Cherry pick from Stab) Fix lua script initialization properties o3de#18779. This last PR has not to be listed in the Release Notes in October, because it was already published in the previous one.

   At the moment, this information is contained only in the PR description (or title, sometimes) which means that you have to inspect all of them to rebuild the full history of changes. As you pointed out, there is no guarantee that the labels will be correctly added, but at least should provide a first-level filter if it is there.

   Please note that the same can happen in the opposite direction (see Fixes intermittent test failures o3de#18760 and (Stabilization) Cherry-pick from dev - Fixes intermittent test failures o3de#18797).

3. The branch/* labels are required to trigger the auto-add workflow in the Project Board. Now they are useless since it is disabled, but I started anyway to add the labels in order to see how they look. I have to admit that I'm not happy so far, because there might bee too many labels on a single issue (for instance, a bug that is affecting ALL the versions. See 24.09 & 23.10 Releases - Directional light cascaded shadows corruption at far distance on Nvidia GPU (regression) o3de#18776). I think that it could be a cleaner solution using a custom-made GitHub Action, but it would require to add further complexity to the automation process (both for development and maintenance).

4. The branch/* and need-check/* labels are complementary. Users usually report an issue only for their current version: development or the latest main. If we are in the stabilization phase, it is really important to know if the bug is also affecting the stabilization branch. But we cannot require users to check again on that version. Thus, the issue can be marked with the pair branch/development and need-check/stabilization, in order to know that it MIGHT affect stabilization and we need someone that is willing to double-check that. If it happens, the need-check/stabilization can be removed, and the issue can now have:

   • branch/development and branch/stabilization, if the bug can be reproduced in both versions;
   • only branch/development, if it is confirmed that the stabilization is safe.

   Using only branch/* doesn't allow to know if an issue was already tested against another branch or not.

   An alternative solution can be replacing need-check/* with its negative not-branch/*, in order to confirm explicitly when a branch is not affected. If an issue doesn't have a branch/* or a not-branch/*, then it can be inferred that it has still to be tested.

   This latter approach would required much less labels, since there are fewer issues that are double-checked on multiple versions rather than the single-version ones. This is especially true for need-check/main, which may become useless since development is generally very different than stable. However, a need-check/* label can be easily used by new contributors that are looking for easy tasks to start contributing to the project (similar to the good-first-issue).

[jhanca-robotecai]

Thank you for the discussion. It is much more clear now. Please understand me correctly: this discussion has two goals in mind. First, reduce the amount of manual work. Secondly, I believe the increasing number of labels makes them less handy. This is also the reason why we should start with the automation.

Ad 1.
I did not want to put that much manual work on anyone, including you. Thank you for volunteering!

Ad 2.
I do not get how this label could help in generating the release notes. Generating release notes is always manual and there is no escape from that.

The last full release was in October 2024, with a feature freeze in May. This means the release notes must include all changes from the development branch between May 2024 and February 2025 + some of the bug fixes (if not related to the new features) from the stabilization. The number of PRs is relatively high (549), hence I cannot imagine the label being any helpful.

git rev-list --count $(git merge-base development main)..$(git merge-base development stabilization/25050) 
537

git rev-list --count $(git merge-base development stabilization/25050)..stabilization/25050
12

If you make release notes for the point releases, then the number of commits is very low, usually less than 15. Going manually through all such commits is not a big deal:

git rev-list --count 2409.0..2409.1
12

git rev-list --count 2409.1..2409.2
13

Ad. 3
Sorry, incorrect vocabulary on my side. E.g. with the release 2409.0 and 2409.1 within less than a month, the label branch/main would get inaccurate (or outdated) before the issue is solved.

[jhanca-robotecai]

2. As you pointed out, there is no guarantee that the labels will be correctly added, but at least should provide a first-level filter if it is there.

Git history is the only valid source of the truth. If I were doing the release notes, I would go through the history (not the labels, which might have been set incorrectly).