Skip to content

Feature/community tested command#1230

Open
sudorishabh wants to merge 13 commits intoRocketChat:developfrom
sudorishabh:feature/community-tested-command
Open

Feature/community tested command#1230
sudorishabh wants to merge 13 commits intoRocketChat:developfrom
sudorishabh:feature/community-tested-command

Conversation

@sudorishabh
Copy link

ci: add /tested command to apply community-tested label on PRs

Adds a GitHub Actions workflow that lets community members comment /tested on a PR to signal they have verified it locally. Once two unique non-author contributors do so, the community-tested label is automatically applied.

Acceptance Criteria fulfillment

  • Users can comment /tested on a PR
  • Only non-authors are counted, and each user is counted once
  • community-tested is applied after at least 2 unique users
  • Temporary feedback messages are shown and do not persist

Fixes #1226

What changed

One new file: .github/workflows/pr-community-tested.yml

Design decisions

  • Exact match over substring: The script requires body.trim() === "/tested" rather than includes("/tested"), so comments like /tested-on-staging or prose referencing the command are never counted by mistake.
  • Reactions instead of bot comments: Feedback is delivered via emoji reactions on the user's comment, which are non-persistent and add zero thread noise. There is no temporary comment that needs a sleep-and-delete pattern.
  • Safe error handling on label creation: Label creation uses an explicit try/catch that only suppresses 404 (label already exists) and re-throws everything else, so real failures (permissions, network) are visible in the workflow logs.
  • Bot exclusion: Any login ending in [bot] is excluded from the tester set so automated tools cannot contribute to the count.
  • Open PR guard: The workflow skips cleanly when the PR is closed or merged.

How to test

  1. Open any PR on the repo (draft is fine).
  2. As a non-author collaborator, comment exactly /tested on the Conversation tab. You should see a 👍 reaction appear and the Actions summary show 1/2 - waiting.
  3. Have a second different collaborator (not the author) do the same. The summary shows 2/2 - label applied and the community-tested label appears on the PR.
  4. Have the PR author comment /tested - they get a 👀 reaction and the count does not change.
  5. Have the first contributor comment /tested again -still 👍, but the count stays at 2 (no double-counting).

Video/Screenshots

PR Test Details

Note: The PR will be ready for live testing at https://rocketchat.github.io/EmbeddedChat/pulls/pr-<pr_number> after approval. Contributors are requested to replace <pr_number> with the actual PR number.

sudorishabh and others added 7 commits March 21, 2026 15:50
Adds .github/workflows/pr-community-tested.yml with the issue_comment
trigger (created only), PR-only job guard, and least-privilege permissions
(issues: write, pull-requests: write). The script step is a placeholder
for the counting and labeling logic added in subsequent commits.
…g Fetches the PR author and all conversation comments on each /tested trigger. Builds a deduplicated Set of non-author, non-bot logins whose comment body is exactly '/tested'. Applies the community-tested label once 2 unique testers are recorded; addLabels is idempotent so re-running never causes issues.
…ummary Reacts with +1 (👍) on a valid /tested comment and eyes (👀) when the comment is ignored (author self-test or not exact match). Writes a progress table to the Actions step summary so maintainers can see tester status without any new PR thread noise.
…ned Before applying the community-tested label, the workflow now checks whether it exists and creates it (color #0075ca, with description) if not. This removes any need for manual repo setup — the first qualifying PR triggers label creation automatically.
… Compare trimmed comment bodies case-insensitively when counting testers and when deciding +1 vs eyes reactions, so /Tested and /TESTED match the command once the workflow has started.
@Spiral-Memory
Copy link
Collaborator

There's a minor change planned. Instead of any 2 people, we'll go something like this:

Only a person with at least 1 merged PR in EmbeddedChat, and who is not the author of the current PR, can run this command.

Then it will mark it as community-tested and ask to upload a demo video for the testing.

After that, as many people write /tested, it will count them and show a confidence score of how many people have tested it.

@sudorishabh sudorishabh marked this pull request as ready for review March 21, 2026 12:37
@sudorishabh
Copy link
Author

working on this.

@sudorishabh sudorishabh marked this pull request as draft March 21, 2026 12:43
@sudorishabh sudorishabh marked this pull request as ready for review March 21, 2026 14:47
@sudorishabh
Copy link
Author

Please review it, @Spiral-Memory

@sudorishabh
Copy link
Author

Improved pr-community-tested workflow

Previously, any 2 people could mark a PR as tested. Now, only contributors who already have a merged PR in the repo can do it (and not the PR author themselves). If someone tries and doesn't qualify, the bot tells them why.

The first person to run /tested triggers the label and gets asked to share a demo video. After that, each additional tester bumps up a confidence score -Low, Medium, or High - shown in a single comment that updates in place rather than spamming the thread.

Also fixed a subtle bug where someone who had commented /tested before they were eligible could still get counted later. The bot now keeps its own verified list so only people who actually passed the check at the time they commented are counted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Auto-Apply community-tested Label Based on /tested Comments from Multiple Users

2 participants