From 7e59a96608fd3a2e816da9e4e7742fc4804a369e Mon Sep 17 00:00:00 2001 From: Colin Walters Date: Fri, 19 Dec 2025 12:53:39 -0500 Subject: [PATCH 1/2] common/tasks: New dir, add AI-driven forge review workflow Add a tasks directory designed primarily for AI agents to execute. These are called "skills" in Claude Code and "commands" in OpenCode, but they're simply structured markdown files. The first task is perform-forge-review.md, which defines an AI-augmented human-approved code review workflow. The key design principle is that the AI builds a review in a local JSONL file, which the human can inspect and edit before submission. The review is submitted as a pending/draft review, allowing the human to make final edits in the forge UI before publishing. Assisted-by: OpenCode (Claude Sonnet 4) Signed-off-by: Colin Walters --- common/AGENTS.md | 9 + common/agents/tasks/README.md | 14 + common/agents/tasks/perform-forge-review.md | 414 ++++++++++++++++++ .../scripts/forge-review-append-comment.sh | 154 +++++++ .../scripts/forge-review-submit-forgejo.sh | 123 ++++++ .../scripts/forge-review-submit-github.sh | 102 +++++ .../scripts/forge-review-submit-gitlab.sh | 165 +++++++ 7 files changed, 981 insertions(+) create mode 100644 common/agents/tasks/README.md create mode 100644 common/agents/tasks/perform-forge-review.md create mode 100755 common/agents/tasks/scripts/forge-review-append-comment.sh create mode 100755 common/agents/tasks/scripts/forge-review-submit-forgejo.sh create mode 100755 common/agents/tasks/scripts/forge-review-submit-github.sh create mode 100755 common/agents/tasks/scripts/forge-review-submit-gitlab.sh diff --git a/common/AGENTS.md b/common/AGENTS.md index 98ff247..d0ceade 100644 --- a/common/AGENTS.md +++ b/common/AGENTS.md @@ -34,6 +34,15 @@ looking for any other issues). If you are performing a review of other's code, the same principles apply. +## Tasks + +The [agents/tasks/](agents/tasks/) directory contains reusable task definitions that you +can execute. These are roughly equivalent to OpenCode "commands" or Claude Code +"skills". + +If the user gives an instruction of the form "run task ..." or similar +then you should read in agents/tasks/README.md and find the relevant task and continue. + ## Follow other guidelines Look at the project README.md and look for guidelines diff --git a/common/agents/tasks/README.md b/common/agents/tasks/README.md new file mode 100644 index 0000000..1c1defe --- /dev/null +++ b/common/agents/tasks/README.md @@ -0,0 +1,14 @@ +# Tasks + +Reusable task definitions for AI agents. See [AGENTS.md](../../AGENTS.md) +for how to execute these tasks. + +Each `.md` file uses YAML frontmatter (`name`, `description`) followed +by markdown instructions — compatible with Claude Code skills and +OpenCode commands. + +## Available Tasks + +- **[perform-forge-review](perform-forge-review.md)** — Create AI-assisted code + reviews on GitHub, GitLab, or Forgejo. Builds review comments in a local JSONL + file for human inspection before submitting as a pending/draft review. diff --git a/common/agents/tasks/perform-forge-review.md b/common/agents/tasks/perform-forge-review.md new file mode 100644 index 0000000..774a711 --- /dev/null +++ b/common/agents/tasks/perform-forge-review.md @@ -0,0 +1,414 @@ +--- +name: perform-forge-review +description: Create AI-assisted code reviews on GitHub, GitLab, or Forgejo. Use when asked to review a PR/MR, analyze code changes, or provide review feedback. +--- + +# Perform Forge Review + +This task describes how to create and manage code reviews on GitHub, GitLab, +and Forgejo/Gitea with human oversight. + +## Overview + +The recommended workflow: + +1. AI analyzes the PR diff and builds comments in a local JSONL file +2. Submit the review as pending/draft (not visible to others yet) +3. Human reviews in the forge UI, can add `@ai:` tasks for follow-up +4. Human submits when satisfied + +Optionally after step 1, the JSONL review state can be passed off +into another security context that has write access (or at least +the ability to create draft reviews). + +## Attribution Convention + +**Review header:** +``` +< your text here > + +--- +Assisted-by: ToolName (ModelName) + +
+Comments prefixed with "AI:" are unedited AI output. +Executed from bootc-dev/infra task. +
+``` + +**Comment prefixes:** +- `AI: ` — unedited AI output +- `@ai: ` — human question/task for AI to process +- No prefix — human has reviewed/edited + +**Comment types:** + +- `*Important*:` — should be resolved before merge +- (no marker) — normal suggestion, can be addressed post-merge or ignored +- `(low)` — minor nit, feel free to ignore +- `**Question**:` — clarification needed (can combine with priority) + +Examples: +- `AI: *Important*: This will panic on empty input` +- `AI: Consider using iterators here` +- `AI: (low) Rename to follow naming convention` +- `AI: **Question**: Is this intentional?` + +**Filtering by priority:** + +If user instructions specify a priority filter (e.g., "important only"), create +inline comments only for that priority level. Summarize the most relevant +normal/low priority items in a `
` section in the review body: + +```markdown +
+Additional suggestions (normal/low priority) + +- `src/lib.rs:42` — Consider using iterators +- `src/main.rs:15` — Minor: rename to follow convention +
+``` + +Avoid inline comments for purely positive observations (e.g., "Good approach here"). +These create noise and require manual resolution. If positive aspects are worth +noting, briefly mention them in the review body. + +**Review body content:** + +Do not summarize or re-describe the PR changes — the commit messages already +contain that information. The review body should only contain: +- Attribution header (required) +- Positive observations worth highlighting (optional, brief) +- Concerns not tied to specific lines (optional) +- Notes about missing context in the PR description (if any) +- Collapsed lower-priority items when filtering (see above) + +--- + +## Workflow + +**Note**: If you already have a pending review on this PR, you cannot create +another. Check for existing pending reviews first (see API Reference) and +either add comments to it or delete it before proceeding. + +### Step 1: Check Out the PR + +Check out the PR branch locally. This lets you read files directly to get +accurate line numbers (diff line numbers are error-prone): + +```bash +# GitHub +gh pr checkout PR_NUMBER + +# GitLab +glab mr checkout MR_IID + +# Forgejo (using forgejo-cli, or fall back to git fetch) +forgejo-cli pr checkout PR_INDEX +# or: git fetch origin pull/PR_INDEX/head:pr-PR_INDEX && git checkout pr-PR_INDEX +``` + +### Step 2: See the Code + +After checkout, determine the merge base (fork point) and review the changes: + +```bash +# Find the merge base with the target branch (usually main) +MERGE_BASE=$(git merge-base HEAD main) + +# View commit history since fork point +git log --oneline $MERGE_BASE..HEAD + +# View the combined diff of all changes +git diff $MERGE_BASE..HEAD + +# Or view each commit's diff separately +git log -p $MERGE_BASE..HEAD +``` + +Review commit-by-commit to understand the logical structure of the changes. +Pay attention to commit messages — they should explain the "why" behind each +change. For larger PRs, reviewing each commit separately often provides better +context than a single combined diff. Note any commits where the message doesn't +match the code changes or where the reasoning is unclear. + +### Step 3: Build the Review + +The scripts in this task are located in the `scripts/` subdirectory relative to this +file (i.e., `common/agents/tasks/scripts/` from the repo root, or wherever `common/` +is synced in your project). + +Use the `forge-review-append-comment.sh` script to add comments. It validates that +your comment targets the correct line by requiring a matching text fragment: + +```bash +scripts/forge-review-append-comment.sh \ + --file src/lib.rs \ + --line 42 \ + --match "fn process_data" \ + --body "AI: *Important*: Missing error handling for empty input" \ + --review-file .git/review-123.jsonl +``` + +This prevents comments from being attached to wrong lines if the file has changed. +The script will error if the match text is not found on the specified line. + +Use a PR-specific review file (e.g., `.git/review-123.jsonl`) to avoid conflicts +when reviewing multiple PRs. + +After adding comments, validate the review before submitting: + +```bash +scripts/forge-review-submit-github.sh --dry-run owner repo 123 .git/review-123.jsonl +# Output: Review validated: 5 pending comment(s) for owner/repo#123 +``` + +### Step 4: Submit the Review + +Submit the review using the appropriate script for your forge: + +#### GitHub + +```bash +scripts/forge-review-submit-github.sh owner repo 123 .git/review-123.jsonl +``` + +Requires: `gh` CLI configured with authentication. + +#### GitLab + +```bash +export GITLAB_TOKEN="glpat-xxxx" +scripts/forge-review-submit-gitlab.sh 12345 67 .git/review-67.jsonl +``` + +For GitLab renames, use `--old-path` when appending comments. + +#### Forgejo + +```bash +export FORGEJO_TOKEN="xxxx" +export FORGEJO_URL="https://codeberg.org" +scripts/forge-review-submit-forgejo.sh owner repo 123 .git/review-123.jsonl +``` + +--- + +## Processing @ai: Tasks + +When a human adds `@ai: ` to a comment, the AI should: + +1. Check for existing pending review (see API Reference below) +2. Find comments containing `@ai:` +3. Read the question/task and relevant code context +4. Generate a response +5. Update the comment, appending: + +```markdown +@ai: Is this error handling correct? + +--- +**AI Response**: No, the error is being silently ignored. Consider... +``` + +The human can then: +- Edit to remove the `@ai:` prefix if satisfied +- Add follow-up `@ai:` tasks +- Delete the comment if not useful +- Submit when done + +--- + +## Platform Comparison + +| Feature | GitHub | GitLab | Forgejo | +|---------|--------|--------|---------| +| Pending/Draft Reviews | ✅ | ✅ | ✅ | +| Edit Pending Comments | ✅ (GraphQL) | ✅ (REST) | ❌ | +| Delete Pending Comments | ✅ | ✅ | ✅ | +| Add to Pending Review | ✅ | ✅ | ✅ | +| Inline Comments | ✅ | ✅ | ✅ | +| Submit with State | ✅ | ✅* | ✅ | +| CLI Support | gh | glab | tea | + +*GitLab handles APPROVE separately via Approvals API. + +--- + +## API Reference + +Direct API calls for advanced operations (editing, deleting, submitting). + +### GitHub + +#### Check for Existing Pending Review + +```bash +gh api graphql -f query=' +{ + repository(owner: "OWNER", name: "REPO") { + pullRequest(number: PR_NUMBER) { + reviews(last: 10, states: [PENDING]) { + nodes { + id + databaseId + body + comments(first: 100) { + nodes { id body path line } + } + } + } + } + } +}' +``` + +#### Edit a Pending Comment + +REST returns 404 for pending comments. Use GraphQL: + +```bash +gh api graphql -f query=' +mutation { + updatePullRequestReviewComment(input: { + pullRequestReviewCommentId: "PRRC_xxxxx", + body: "Updated comment text" + }) { + pullRequestReviewComment { id body } + } +}' +``` + +#### Add Comment to Existing Pending Review + +```bash +gh api graphql -f query=' +mutation { + addPullRequestReviewThread(input: { + pullRequestReviewId: "PRR_xxxxx", + body: "AI: New comment", + path: "src/lib.rs", + line: 50, + side: RIGHT + }) { + thread { id } + } +}' +``` + +#### Delete a Pending Comment + +```bash +gh api graphql -f query=' +mutation { + deletePullRequestReviewComment(input: { + id: "PRRC_xxxxx" + }) { + pullRequestReviewComment { id } + } +}' +``` + +#### Submit the Review + +```bash +gh api repos/OWNER/REPO/pulls/PR_NUMBER/reviews/REVIEW_ID/events \ + -X POST -f event="COMMENT" # or REQUEST_CHANGES or APPROVE +``` + +### GitLab + +#### Check for Existing Draft Notes + +```bash +curl --header "PRIVATE-TOKEN: $TOKEN" \ + "$GITLAB_URL/api/v4/projects/$PROJECT_ID/merge_requests/$MR_IID/draft_notes" +``` + +#### Edit a Draft Note + +```bash +curl --request PUT \ + --header "PRIVATE-TOKEN: $TOKEN" \ + --form-string "note=Updated comment text" \ + "$GITLAB_URL/api/v4/projects/$PROJECT_ID/merge_requests/$MR_IID/draft_notes/$NOTE_ID" +``` + +#### Delete a Draft Note + +```bash +curl --request DELETE \ + --header "PRIVATE-TOKEN: $TOKEN" \ + "$GITLAB_URL/api/v4/projects/$PROJECT_ID/merge_requests/$MR_IID/draft_notes/$NOTE_ID" +``` + +#### Submit Review (Publish All Drafts) + +```bash +curl --request POST \ + --header "PRIVATE-TOKEN: $TOKEN" \ + "$GITLAB_URL/api/v4/projects/$PROJECT_ID/merge_requests/$MR_IID/draft_notes/bulk_publish" +``` + +#### Approve MR (Separate from Review) + +```bash +curl --request POST \ + --header "PRIVATE-TOKEN: $TOKEN" \ + "$GITLAB_URL/api/v4/projects/$PROJECT_ID/merge_requests/$MR_IID/approve" +``` + +### Forgejo + +#### List Reviews (Find Pending) + +```bash +curl -H "Authorization: token $TOKEN" \ + "$FORGEJO_URL/api/v1/repos/$OWNER/$REPO/pulls/$PR_INDEX/reviews" +``` + +Look for reviews with `state: "PENDING"`. + +#### Add Comment to Existing Pending Review + +```bash +curl -X POST \ + -H "Authorization: token $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"path": "src/lib.rs", "new_position": 50, "body": "AI: Comment"}' \ + "$FORGEJO_URL/api/v1/repos/$OWNER/$REPO/pulls/$PR_INDEX/reviews/$REVIEW_ID/comments" +``` + +#### Delete a Comment + +```bash +curl -X DELETE \ + -H "Authorization: token $TOKEN" \ + "$FORGEJO_URL/api/v1/repos/$OWNER/$REPO/pulls/$PR_INDEX/reviews/$REVIEW_ID/comments/$COMMENT_ID" +``` + +#### Submit the Review + +```bash +curl -X POST \ + -H "Authorization: token $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"event": "COMMENT", "body": "Review complete"}' \ + "$FORGEJO_URL/api/v1/repos/$OWNER/$REPO/pulls/$PR_INDEX/reviews/$REVIEW_ID" +``` + +Valid event values: `APPROVE`, `REQUEST_CHANGES`, `COMMENT` + +**Note**: Forgejo does not support editing review comments via API. +Workaround: Delete and recreate the comment. + +--- + +## Notes + +- Always get the diff first to understand line positioning +- Node IDs (GitHub) come from GraphQL queries +- Project IDs (GitLab) can be numeric or URL-encoded paths +- Pending reviews are only visible to their author +- The JSONL workflow enables future sandboxed execution where the agent + runs with read-only access and a separate privileged process submits the review diff --git a/common/agents/tasks/scripts/forge-review-append-comment.sh b/common/agents/tasks/scripts/forge-review-append-comment.sh new file mode 100755 index 0000000..a5704a8 --- /dev/null +++ b/common/agents/tasks/scripts/forge-review-append-comment.sh @@ -0,0 +1,154 @@ +#!/bin/bash +# Append a review comment to a JSONL file with line content validation +# +# Usage: forge-review-append-comment.sh --file PATH --line NUM --match TEXT --body COMMENT [--review-file FILE] +# +# Arguments: +# --file PATH File path relative to repo root (e.g., src/lib.rs) +# --line NUM Line number in the file +# --match TEXT Text that must appear on that line (for validation) +# --body COMMENT The review comment body +# --review-file FILE Output JSONL file (default: .git/review-{inferred}.jsonl) +# --old-path PATH Original file path for renames (GitLab only) +# +# The script verifies that --match text appears on the specified line before +# appending. This prevents comments from being attached to wrong lines when +# the file has changed. +# +# Example: +# forge-review-append-comment.sh \ +# --file src/lib.rs \ +# --line 42 \ +# --match "fn process_data" \ +# --body "AI: *Important*: Missing error handling for empty input" + +set -euo pipefail + +usage() { + cat >&2 <&2 + usage + ;; + esac +done + +# Validate required arguments +if [[ -z "$FILE" ]]; then + echo "Error: --file is required" >&2 + usage +fi +if [[ -z "$LINE" ]]; then + echo "Error: --line is required" >&2 + usage +fi +if [[ -z "$MATCH" ]]; then + echo "Error: --match is required" >&2 + usage +fi +if [[ -z "$BODY" ]]; then + echo "Error: --body is required" >&2 + usage +fi + +# Validate line is a number +if ! [[ "$LINE" =~ ^[0-9]+$ ]]; then + echo "Error: --line must be a positive integer, got: $LINE" >&2 + exit 1 +fi + +# Check file exists +if [[ ! -f "$FILE" ]]; then + echo "Error: File not found: $FILE" >&2 + exit 1 +fi + +# Extract the actual line content +ACTUAL_LINE=$(sed -n "${LINE}p" "$FILE") + +if [[ -z "$ACTUAL_LINE" ]]; then + echo "Error: Line $LINE does not exist in $FILE" >&2 + exit 1 +fi + +# Check if match text appears on the line +if [[ "$ACTUAL_LINE" != *"$MATCH"* ]]; then + echo "Error: Match text not found on line $LINE" >&2 + echo " Expected to find: $MATCH" >&2 + echo " Actual line content: $ACTUAL_LINE" >&2 + exit 1 +fi + +# Default review file +if [[ -z "$REVIEW_FILE" ]]; then + REVIEW_FILE=".git/review.jsonl" +fi + +# Ensure parent directory exists +mkdir -p "$(dirname "$REVIEW_FILE")" + +# Build the JSON object +if [[ -n "$OLD_PATH" ]]; then + # Include old_path for GitLab renames + jq -n --arg path "$FILE" --argjson line "$LINE" --arg body "$BODY" --arg old_path "$OLD_PATH" \ + '{path: $path, line: $line, body: $body, old_path: $old_path}' >> "$REVIEW_FILE" +else + jq -n --arg path "$FILE" --argjson line "$LINE" --arg body "$BODY" \ + '{path: $path, line: $line, body: $body}' >> "$REVIEW_FILE" +fi + +echo "Added comment for $FILE:$LINE" diff --git a/common/agents/tasks/scripts/forge-review-submit-forgejo.sh b/common/agents/tasks/scripts/forge-review-submit-forgejo.sh new file mode 100755 index 0000000..8143fc9 --- /dev/null +++ b/common/agents/tasks/scripts/forge-review-submit-forgejo.sh @@ -0,0 +1,123 @@ +#!/bin/bash +# Submit a forge review to Forgejo/Gitea +# +# STATUS: DRAFT/UNTESTED - This script has not been tested against a real Forgejo/Gitea instance. +# +# Usage: forge-review-submit-forgejo.sh [--dry-run] OWNER REPO PR_INDEX [REVIEW_FILE] [REVIEW_BODY] +# +# Options: +# --dry-run Validate the review file without submitting +# +# Arguments: +# OWNER Repository owner +# REPO Repository name +# PR_INDEX Pull request index number +# REVIEW_FILE Path to JSONL file with comments (default: .git/.forge-review.jsonl) +# REVIEW_BODY Review body text (default: standard attribution header) +# +# Environment: +# FORGEJO_TOKEN or GITEA_TOKEN - Forgejo/Gitea authentication token (required) +# FORGEJO_URL or GITEA_URL - Instance URL (required, e.g., https://codeberg.org) +# +# The JSONL file should contain one JSON object per line: +# {"path": "src/lib.rs", "line": 42, "body": "AI: Comment text"} +# +# On success, removes the review file. On failure, preserves it. + +set -euo pipefail + +usage() { + echo "Usage: $0 [--dry-run] OWNER REPO PR_INDEX [REVIEW_FILE] [REVIEW_BODY]" >&2 + echo "" >&2 + echo "Options:" >&2 + echo " --dry-run Validate the review file without submitting" >&2 + echo "" >&2 + echo "Environment variables:" >&2 + echo " FORGEJO_TOKEN or GITEA_TOKEN - Authentication token (required)" >&2 + echo " FORGEJO_URL or GITEA_URL - Instance URL (required)" >&2 + exit 1 +} + +DRY_RUN=0 +if [[ "${1:-}" == "--dry-run" ]]; then + DRY_RUN=1 + shift +fi + +if [[ $# -lt 3 ]]; then + usage +fi + +OWNER="$1" +REPO="$2" +PR_INDEX="$3" +REVIEW_FILE="${4:-.git/.forge-review.jsonl}" +REVIEW_BODY="${5:-Assisted-by: OpenCode (Claude Sonnet 4) + +AI-generated review based on REVIEW.md guidelines. +Comments prefixed with \"AI:\" are unedited AI output.}" + +if [[ ! -f "$REVIEW_FILE" ]]; then + echo "Error: Review file not found: $REVIEW_FILE" >&2 + exit 1 +fi + +# Validate JSONL syntax and count comments +if ! COMMENTS_RAW=$(jq -s '. // []' < "$REVIEW_FILE" 2>&1); then + echo "Error: Invalid JSONL syntax in $REVIEW_FILE" >&2 + echo "$COMMENTS_RAW" >&2 + exit 1 +fi + +COMMENT_COUNT=$(echo "$COMMENTS_RAW" | jq 'length') + +# Validate each comment has required fields +INVALID=$(echo "$COMMENTS_RAW" | jq '[.[] | select(.path == null or .body == null)] | length') +if [[ "$INVALID" -gt 0 ]]; then + echo "Error: $INVALID comment(s) missing required fields (path, body)" >&2 + exit 1 +fi + +if [[ $DRY_RUN -eq 1 ]]; then + echo "Review validated: $COMMENT_COUNT pending comment(s) for $OWNER/$REPO#$PR_INDEX" + exit 0 +fi + +TOKEN="${FORGEJO_TOKEN:-${GITEA_TOKEN:-}}" +BASE_URL="${FORGEJO_URL:-${GITEA_URL:-}}" + +if [[ -z "$TOKEN" ]]; then + echo "Error: FORGEJO_TOKEN or GITEA_TOKEN environment variable required" >&2 + exit 1 +fi + +if [[ -z "$BASE_URL" ]]; then + echo "Error: FORGEJO_URL or GITEA_URL environment variable required" >&2 + exit 1 +fi + +# Strip trailing slash from URL +BASE_URL="${BASE_URL%/}" + +# Transform JSONL to Forgejo format (line -> new_position) +COMMENTS=$(echo "$COMMENTS_RAW" | jq 'map({path, new_position: .line, body})') + +# Create the review (pending by default - omit event param) +RESULT=$(jq -n \ + --arg body "$REVIEW_BODY" \ + --argjson comments "$COMMENTS" \ + '{body: $body, comments: $comments}' | \ +curl -s -X POST \ + -H "Authorization: token $TOKEN" \ + -H "Content-Type: application/json" \ + -d @- \ + "$BASE_URL/api/v1/repos/$OWNER/$REPO/pulls/$PR_INDEX/reviews" 2>&1) + +if echo "$RESULT" | jq -e '.id' > /dev/null 2>&1; then + echo "Review created successfully" + rm "$REVIEW_FILE" +else + echo "Failed to create review: $RESULT" >&2 + echo "Review file preserved at $REVIEW_FILE" >&2 + exit 1 +fi diff --git a/common/agents/tasks/scripts/forge-review-submit-github.sh b/common/agents/tasks/scripts/forge-review-submit-github.sh new file mode 100755 index 0000000..7eef55b --- /dev/null +++ b/common/agents/tasks/scripts/forge-review-submit-github.sh @@ -0,0 +1,102 @@ +#!/bin/bash +# Submit a forge review to GitHub +# +# Usage: forge-review-submit-github.sh [--dry-run] OWNER REPO PR_NUMBER [REVIEW_FILE] [REVIEW_BODY] +# +# Options: +# --dry-run Validate the review file without submitting +# +# Arguments: +# OWNER Repository owner +# REPO Repository name +# PR_NUMBER Pull request number +# REVIEW_FILE Path to JSONL file with comments (default: .git/.forge-review.jsonl) +# REVIEW_BODY Review body text (default: standard attribution header) +# +# Environment: +# GH_TOKEN or GITHUB_TOKEN - GitHub authentication token (optional if gh is configured) +# +# The JSONL file should contain one JSON object per line: +# {"path": "src/lib.rs", "line": 42, "body": "AI: Comment text"} +# +# On success, removes the review file. On failure, preserves it. + +set -euo pipefail + +usage() { + echo "Usage: $0 [--dry-run] OWNER REPO PR_NUMBER [REVIEW_FILE] [REVIEW_BODY]" >&2 + echo "" >&2 + echo "Options:" >&2 + echo " --dry-run Validate the review file without submitting" >&2 + exit 1 +} + +DRY_RUN=0 +if [[ "${1:-}" == "--dry-run" ]]; then + DRY_RUN=1 + shift +fi + +if [[ $# -lt 3 ]]; then + usage +fi + +OWNER="$1" +REPO="$2" +PR_NUMBER="$3" +REVIEW_FILE="${4:-.git/.forge-review.jsonl}" +REVIEW_BODY="${5:-Assisted-by: OpenCode (Claude Sonnet 4) + +AI-generated review based on REVIEW.md guidelines. +Comments prefixed with \"AI:\" are unedited AI output.}" + +if [[ ! -f "$REVIEW_FILE" ]]; then + echo "Error: Review file not found: $REVIEW_FILE" >&2 + exit 1 +fi + +# Validate JSONL syntax and count comments +if ! COMMENTS=$(jq -s '. // []' < "$REVIEW_FILE" 2>&1); then + echo "Error: Invalid JSONL syntax in $REVIEW_FILE" >&2 + echo "$COMMENTS" >&2 + exit 1 +fi + +COMMENT_COUNT=$(echo "$COMMENTS" | jq 'length') + +# Validate each comment has required fields +INVALID=$(echo "$COMMENTS" | jq '[.[] | select(.path == null or .body == null)] | length') +if [[ "$INVALID" -gt 0 ]]; then + echo "Error: $INVALID comment(s) missing required fields (path, body)" >&2 + exit 1 +fi + +if [[ $DRY_RUN -eq 1 ]]; then + echo "Review validated: $COMMENT_COUNT pending comment(s) for $OWNER/$REPO#$PR_NUMBER" + exit 0 +fi + +# Check gh CLI is available +if ! command -v gh >/dev/null 2>&1; then + echo "Error: gh CLI not found. Install from https://cli.github.com/" >&2 + exit 1 +fi + +# Create the review (pending by default - omit event param) +RESULT=$(jq -n \ + --arg body "$REVIEW_BODY" \ + --argjson comments "$COMMENTS" \ + '{body: $body, comments: $comments}' | \ +gh api "repos/$OWNER/$REPO/pulls/$PR_NUMBER/reviews" \ + -X POST --input - 2>&1) || true + +# Check result and clean up +if echo "$RESULT" | jq -e '.id' > /dev/null 2>&1; then + echo "Review created: $(echo "$RESULT" | jq -r '.html_url // .id')" + rm "$REVIEW_FILE" +else + echo "Failed to create review: $RESULT" >&2 + echo "Review file preserved at $REVIEW_FILE" >&2 + # Common error: "User can only have one pending review per pull request" + exit 1 +fi diff --git a/common/agents/tasks/scripts/forge-review-submit-gitlab.sh b/common/agents/tasks/scripts/forge-review-submit-gitlab.sh new file mode 100755 index 0000000..5838faf --- /dev/null +++ b/common/agents/tasks/scripts/forge-review-submit-gitlab.sh @@ -0,0 +1,165 @@ +#!/bin/bash +# Submit a forge review to GitLab +# +# STATUS: DRAFT/UNTESTED - This script has not been tested against a real GitLab instance. +# +# Usage: forge-review-submit-gitlab.sh [--dry-run] PROJECT_ID MR_IID [REVIEW_FILE] [REVIEW_BODY] +# +# Options: +# --dry-run Validate the review file without submitting +# +# Arguments: +# PROJECT_ID GitLab project ID (numeric or URL-encoded path) +# MR_IID Merge request IID +# REVIEW_FILE Path to JSONL file with comments (default: .git/.forge-review.jsonl) +# REVIEW_BODY Review body text (default: standard attribution header) +# +# Environment: +# GITLAB_TOKEN or PRIVATE_TOKEN - GitLab authentication token (required) +# GITLAB_URL - GitLab instance URL (default: https://gitlab.com) +# +# The JSONL file should contain one JSON object per line: +# {"path": "src/lib.rs", "line": 42, "body": "AI: Comment text", "old_path": "src/lib.rs"} +# +# Note: old_path is optional, defaults to path if not specified. +# +# On success, removes the review file. On failure, preserves it. + +set -euo pipefail + +usage() { + echo "Usage: $0 [--dry-run] PROJECT_ID MR_IID [REVIEW_FILE] [REVIEW_BODY]" >&2 + echo "" >&2 + echo "Options:" >&2 + echo " --dry-run Validate the review file without submitting" >&2 + echo "" >&2 + echo "Environment variables:" >&2 + echo " GITLAB_TOKEN or PRIVATE_TOKEN - GitLab authentication token (required)" >&2 + echo " GITLAB_URL - GitLab instance URL (default: https://gitlab.com)" >&2 + exit 1 +} + +DRY_RUN=0 +if [[ "${1:-}" == "--dry-run" ]]; then + DRY_RUN=1 + shift +fi + +if [[ $# -lt 2 ]]; then + usage +fi + +PROJECT_ID="$1" +MR_IID="$2" +REVIEW_FILE="${3:-.git/.forge-review.jsonl}" +REVIEW_BODY="${4:-Assisted-by: OpenCode (Claude Sonnet 4) + +AI-generated review based on REVIEW.md guidelines. +Comments prefixed with \"AI:\" are unedited AI output.}" + +TOKEN="${GITLAB_TOKEN:-${PRIVATE_TOKEN:-}}" +GITLAB_URL="${GITLAB_URL:-https://gitlab.com}" +# Strip trailing slash from URL +GITLAB_URL="${GITLAB_URL%/}" + +if [[ ! -f "$REVIEW_FILE" ]]; then + echo "Error: Review file not found: $REVIEW_FILE" >&2 + exit 1 +fi + +# Validate JSONL syntax and count comments +if ! COMMENTS=$(jq -s '. // []' < "$REVIEW_FILE" 2>&1); then + echo "Error: Invalid JSONL syntax in $REVIEW_FILE" >&2 + echo "$COMMENTS" >&2 + exit 1 +fi + +COMMENT_COUNT=$(echo "$COMMENTS" | jq 'length') + +# Validate each comment has required fields +INVALID=$(echo "$COMMENTS" | jq '[.[] | select(.path == null or .body == null)] | length') +if [[ "$INVALID" -gt 0 ]]; then + echo "Error: $INVALID comment(s) missing required fields (path, body)" >&2 + exit 1 +fi + +if [[ $DRY_RUN -eq 1 ]]; then + echo "Review validated: $COMMENT_COUNT pending comment(s) for GitLab project $PROJECT_ID MR !$MR_IID" + exit 0 +fi + +if [[ -z "$TOKEN" ]]; then + echo "Error: GITLAB_TOKEN or PRIVATE_TOKEN environment variable required" >&2 + exit 1 +fi + +# Get MR version info first +VERSION_RESPONSE=$(curl -s --header "PRIVATE-TOKEN: $TOKEN" \ + "$GITLAB_URL/api/v4/projects/$PROJECT_ID/merge_requests/$MR_IID/versions" 2>&1) + +VERSION_INFO=$(echo "$VERSION_RESPONSE" | jq '.[0]' 2>/dev/null) + +if [[ -z "$VERSION_INFO" || "$VERSION_INFO" == "null" ]]; then + echo "Error: Failed to get MR version info: $VERSION_RESPONSE" >&2 + echo "Review file preserved at $REVIEW_FILE" >&2 + exit 1 +fi + +BASE_SHA=$(echo "$VERSION_INFO" | jq -r '.base_commit_sha') +HEAD_SHA=$(echo "$VERSION_INFO" | jq -r '.head_commit_sha') +START_SHA=$(echo "$VERSION_INFO" | jq -r '.start_commit_sha') + +# Validate we got valid SHAs +if [[ "$BASE_SHA" == "null" || -z "$BASE_SHA" ]]; then + echo "Error: Invalid version info - missing base_commit_sha" >&2 + echo "Review file preserved at $REVIEW_FILE" >&2 + exit 1 +fi + +# Create draft notes from JSONL +FAILED=0 +while IFS= read -r line; do + [[ -z "$line" ]] && continue + + path=$(echo "$line" | jq -r '.path') + line_num=$(echo "$line" | jq -r '.line') + body=$(echo "$line" | jq -r '.body') + old_path=$(echo "$line" | jq -r '.old_path // .path') + + RESULT=$(curl -s --request POST \ + --header "PRIVATE-TOKEN: $TOKEN" \ + --form-string "note=$body" \ + --form-string "position[position_type]=text" \ + --form-string "position[base_sha]=$BASE_SHA" \ + --form-string "position[head_sha]=$HEAD_SHA" \ + --form-string "position[start_sha]=$START_SHA" \ + --form-string "position[old_path]=$old_path" \ + --form-string "position[new_path]=$path" \ + --form-string "position[new_line]=$line_num" \ + "$GITLAB_URL/api/v4/projects/$PROJECT_ID/merge_requests/$MR_IID/draft_notes" 2>&1) + + if ! echo "$RESULT" | jq -e '.id' > /dev/null 2>&1; then + echo "Failed to create draft note for $path:$line_num: $RESULT" >&2 + FAILED=1 + break + fi +done < "$REVIEW_FILE" + +if [[ $FAILED -eq 1 ]]; then + echo "Review file preserved at $REVIEW_FILE" >&2 + exit 1 +fi + +# Post the main review body as a note +NOTE_RESULT=$(curl -s --request POST \ + --header "PRIVATE-TOKEN: $TOKEN" \ + --form-string "body=$REVIEW_BODY" \ + "$GITLAB_URL/api/v4/projects/$PROJECT_ID/merge_requests/$MR_IID/notes" 2>&1) + +if ! echo "$NOTE_RESULT" | jq -e '.id' > /dev/null 2>&1; then + echo "Warning: Failed to post review summary note: $NOTE_RESULT" >&2 + echo "Draft comments were created successfully" >&2 +fi + +echo "Review created successfully" +rm "$REVIEW_FILE" From 6656a551cc2362551b4dcc8e0474dc2c0b57db96 Mon Sep 17 00:00:00 2001 From: Colin Walters Date: Fri, 19 Dec 2025 16:15:06 -0500 Subject: [PATCH 2/2] common/tasks: Add diff-quiz task for verifying code understanding Add a new diff-quiz task that generates quizzes to verify human understanding of code changes. This is particularly relevant when AI tools assist in generating code, helping ensure developers understand what they're submitting. Features: - Three difficulty levels (easy, medium, hard) - Markdown or interactive bash script output formats - Separate grading step with constructive feedback - Early exit support for efficient verification Assisted-by: OpenCode (claude-sonnet-4-20250514) Signed-off-by: Colin Walters --- common/agents/tasks/README.md | 4 + common/agents/tasks/diff-quiz.md | 325 +++++++++++++++++++++++++++++++ 2 files changed, 329 insertions(+) create mode 100644 common/agents/tasks/diff-quiz.md diff --git a/common/agents/tasks/README.md b/common/agents/tasks/README.md index 1c1defe..5ba0315 100644 --- a/common/agents/tasks/README.md +++ b/common/agents/tasks/README.md @@ -9,6 +9,10 @@ OpenCode commands. ## Available Tasks +- **[diff-quiz](diff-quiz.md)** — Generate a quiz to verify human understanding + of code changes. Helps ensure that developers using AI tools understand the + code they're submitting. Supports easy, medium, and hard difficulty levels. + - **[perform-forge-review](perform-forge-review.md)** — Create AI-assisted code reviews on GitHub, GitLab, or Forgejo. Builds review comments in a local JSONL file for human inspection before submitting as a pending/draft review. diff --git a/common/agents/tasks/diff-quiz.md b/common/agents/tasks/diff-quiz.md new file mode 100644 index 0000000..7dd2879 --- /dev/null +++ b/common/agents/tasks/diff-quiz.md @@ -0,0 +1,325 @@ +--- +name: diff-quiz +description: Generate a quiz to verify human understanding of code changes. Use when reviewing PRs or commits to ensure the submitter understands what they're proposing. +--- + +# Diff Quiz + +This task generates a quiz based on a git diff to help verify that a human +submitter has meaningful understanding of code they're proposing — particularly +relevant when agentic AI tools assisted in generating the code. + +## Purpose + +When developers use AI tools to generate code, there's a risk of submitting +changes without fully understanding them. This quiz helps: + +- Verify baseline competency in the relevant programming language(s) +- Confirm understanding of the specific changes being proposed +- Ensure the submitter could maintain and debug the code in the future + +The quiz is designed to be educational, not adversarial. It should help +identify knowledge gaps that could lead to problems down the road. + +## Difficulty Levels + +### Easy + +Basic verification that the submitter is familiar with the project and has +general programming competency. Questions at this level do NOT require deep +understanding of the specific diff. + +Example question types: +- What programming language(s) is this project written in? +- What is the purpose of this project (based on README or documentation)? +- What build system or package manager does this project use? +- Name one external dependency this project uses and what it's for +- What testing framework does this project use? +- What does `` do? (e.g., "What does `?` do in Rust?") + +### Medium + +Verify baseline understanding of both the programming language and the specific +changes in the diff. The submitter should be able to explain what the code does. + +Example question types: +- Explain in your own words what this function/method does +- What error conditions does this code handle? +- Why might the author have chosen `` over ``? +- What would happen if `` were passed to this function? +- This code uses ``. What is the purpose of the ``? +- What tests would you write to verify this change works correctly? +- Walk through what happens when `` occurs +- What existing code does this change interact with? + +### Hard + +Verify deep understanding — the submitter should in theory be able to have +written this patch themselves. Questions probe implementation details, +architectural decisions, and broader project knowledge. + +Example question types: +- Write pseudocode for how you would implement `` +- This change affects ``. What other parts of the codebase might + need to be updated as a result? +- What are the performance implications of this approach? +- Describe an alternative implementation and explain the tradeoffs +- How does this change interact with ``? +- What edge cases are NOT handled by this implementation? +- If this code fails in production, how would you debug it? +- Why is `` necessary? What would break without it? +- Explain the memory/ownership model used here (for languages like Rust/C++) + +## Workflow + +### Step 1: Identify the Diff + +Determine the commit range to quiz on: + +```bash +# For a PR, find the merge base +MERGE_BASE=$(git merge-base HEAD main) +git log --oneline $MERGE_BASE..HEAD +git diff $MERGE_BASE..HEAD + +# For a specific commit +git show + +# For a range of commits +git diff .. +``` + +### Step 2: Analyze the Changes + +Before generating questions, understand: + +1. **Languages involved** — What programming languages are being modified? +2. **Scope of changes** — How many files? How many lines? New feature vs bugfix? +3. **Complexity** — Simple refactor or complex algorithmic changes? +4. **Project context** — What part of the system is being modified? + +### Step 3: Generate Questions + +Generate questions appropriate to the requested difficulty level. Guidelines: + +- **Easy**: 4-6 questions, should take 2-5 minutes to answer +- **Medium**: 6-9 questions, should take 10-15 minutes to answer +- **Hard**: 8-12 questions, should take 20-30 minutes to answer + +**Question ordering**: Put the most important questions first. The quiz should +front-load questions that best verify understanding, so users who demonstrate +competency early can stop without answering everything. + +For each question: +- State the question clearly +- If referencing specific code, include the relevant snippet or file:line +- Indicate what type of answer is expected (short answer, explanation, pseudocode) + +**IMPORTANT**: Do NOT include grading notes, expected answers, or hints in the +quiz output. The quiz is for the human to answer, and grading happens in a +separate step. + +## Output Formats + +### Format 1: Markdown (default) + +Present the quiz in markdown for the human to read and answer conversationally: + +```markdown +# Diff Quiz: [Brief description of changes] + +**Difficulty:** [Easy/Medium/Hard] +**Estimated time:** [X minutes] +**Commits:** [commit range or PR number] + +--- + +## Questions + +### Question 1 +[Question text] + +**Expected answer type:** [Short answer / Explanation / Pseudocode / etc.] + +### Question 2 +... +``` + +### Format 2: Bash Script (`--script` or "as a script") + +Generate a self-contained bash script that: +1. Displays each question interactively +2. Prompts the user for their answer +3. Appends all answers to a `.answers.txt` file for later grading +4. **Asks "Continue? [Y/n]" every 2-3 questions** — allowing early exit if + the user has demonstrated sufficient understanding + +The script should be saved to a file like `diff-quiz-.sh`. + +```bash +#!/bin/bash +# Diff Quiz: [Brief description] +# Difficulty: [Easy/Medium/Hard] +# Commit: [commit hash] +# Generated: [date] + +set -e + +ANSWERS_FILE=".answers-$(date +%Y%m%d-%H%M%S).txt" + +cat << 'EOF' +╔════════════════════════════════════════════════════════════════╗ +║ DIFF QUIZ ║ +║────────────────────────────────────────────────────────────────║ +║ Difficulty: [Easy/Medium/Hard] ║ +║ Estimated time: [X] minutes ║ +║ Commit: [hash] ║ +║ ║ +║ Answer each question. Your responses will be saved to: ║ +║ [answers file] ║ +║ ║ +║ For multi-line answers, type your response and press ║ +║ Enter twice (empty line) to submit. ║ +╚════════════════════════════════════════════════════════════════╝ +EOF + +echo "Answers file: $ANSWERS_FILE" +echo "" + +# Header for answers file +cat << EOF > "$ANSWERS_FILE" +# Diff Quiz Answers +# Difficulty: [Easy/Medium/Hard] +# Commit: [hash] +# Date: $(date -Iseconds) +# ───────────────────────────────────────────────────────────────── + +EOF + +read_answer() { + local answer="" + local line + while IFS= read -r line; do + [[ -z "$line" ]] && break + answer+="$line"$'\n' + done + echo "${answer%$'\n'}" # Remove trailing newline +} + +# Question 1 +echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" +echo "Question 1 of N" +echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" +echo "" +cat << 'EOF' +[Question text here] + +Expected answer type: [Short answer / Explanation / etc.] +EOF +echo "" +echo "Your answer (empty line to submit):" +answer=$(read_answer) +cat << EOF >> "$ANSWERS_FILE" +## Question 1 +[Question text here] + +### Answer +$answer + +EOF +echo "" + +# ... repeat for each question ... + +# After every 2-3 questions, prompt to continue: +echo "" +read -p "Continue to more questions? [Y/n] " -n 1 -r +echo "" +if [[ $REPLY =~ ^[Nn]$ ]]; then + echo "Stopping early. Your answers so far have been saved." + # jump to completion message +fi + +# ... continue with remaining questions ... + +echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" +echo "Quiz complete! Your answers have been saved to: $ANSWERS_FILE" +echo "" +echo "To have your answers graded, run:" +echo " [agent command] grade diff-quiz $ANSWERS_FILE" +echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" +``` + +Make the script executable: `chmod +x diff-quiz-.sh` + +## Grading + +Grading is a **separate step** performed after the human completes the quiz. + +When asked to grade a quiz (e.g., "grade diff-quiz .answers.txt"): + +1. Read the answers file +2. Re-analyze the original commit/diff to understand the correct answers +3. For each answer, evaluate: + - **Correctness** — Is the answer factually accurate? + - **Completeness** — Did they address all parts of the question? + - **Depth** — Does the answer show genuine understanding vs. surface-level recall? +4. Provide feedback for each question: + - What was good about the answer + - What was missing or incorrect + - Additional context that might help their understanding +5. Give an overall assessment: + - **Pass** — Demonstrates sufficient understanding for the difficulty level + - **Partial** — Some gaps, may want to review specific areas + - **Needs Review** — Significant gaps suggest the code should be reviewed more carefully + +Grading should be constructive and educational, not punitive. + +## Usage Examples + +### Example 1: Quick sanity check before merge + +``` +User: Run diff quiz on this PR, easy difficulty + +Agent: [Generates 3-5 basic questions about the project and language] +``` + +### Example 2: Generate a script for async completion + +``` +User: Generate a medium diff-quiz script for commit abc123 + +Agent: [Creates diff-quiz-abc123.sh that the user can run on their own time] +``` + +### Example 3: Grade completed answers + +``` +User: Grade diff-quiz .answers-20240115-143022.txt + +Agent: [Reads answers, evaluates against the commit, provides feedback] +``` + +### Example 4: Full workflow + +``` +User: Generate hard diff-quiz for this PR as a script +[User runs the script, answers questions] +User: Grade the quiz: .answers-20240115-143022.txt +Agent: [Provides detailed feedback and pass/fail assessment] +``` + +## Notes + +- The quiz should be fair — questions should be answerable by someone who + genuinely wrote or deeply reviewed the code +- Avoid gotcha questions or obscure trivia +- For Easy level, questions should be passable by any competent developer + familiar with the project, even if they didn't write this specific code +- For Hard level, it's acceptable if the submitter needs to look things up, + but they should know *what* to look up and understand the answers +- Consider the context — a typo fix doesn't need a Hard quiz +- Questions should probe understanding, not memorization +- **Never include answers or grading notes in the quiz itself** — this defeats + the purpose of verifying understanding