Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 25 additions & 41 deletions .claude/skills/triage-ci-flake/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,11 @@ Systematic workflow for triaging and fixing test failures in CI, especially flak
**YOU MUST EXECUTE THESE COMMANDS. Reading code or analyzing logs does NOT count as reproduction.**

1. **Extract** suite name, test name, and error from CI logs
2. **EXECUTE**: Kill port 3000 to avoid conflicts
3. **EXECUTE**: `pnpm dev $SUITE_NAME` (use run_in_background=true)
4. **EXECUTE**: Wait for server to be ready (check with curl or sleep)
5. **EXECUTE**: Run the specific failing test with Playwright directly (npx playwright test test/TEST_SUITE_NAME/e2e.spec.ts:31:3 --headed -g "TEST_DESCRIPTION_TARGET_GOES_HERE")
6. **If test passes**, **EXECUTE**: `pnpm prepare-run-test-against-prod`
7. **EXECUTE**: `pnpm dev:prod $SUITE_NAME` and run test again
2. **EXECUTE**: `pnpm dev $SUITE_NAME` (use run_in_background=true) — portless handles port assignment
3. **EXECUTE**: Wait for server to be ready (check with curl to $PORTLESS_URL)
4. **EXECUTE**: Run the specific failing test with Playwright directly (npx playwright test test/TEST_SUITE_NAME/e2e.spec.ts:31:3 --headed -g "TEST_DESCRIPTION_TARGET_GOES_HERE")
5. **If test passes**, **EXECUTE**: `pnpm prepare-run-test-against-prod`
6. **EXECUTE**: Stop dev server, then `pnpm dev:prod $SUITE_NAME` and run test again

**Only after EXECUTING these commands and seeing their output** can you proceed to analysis and fixes.

Expand Down Expand Up @@ -89,30 +88,24 @@ From CI logs or GitHub Actions URL, identify:

**SERVER MANAGEMENT RULES:**

1. **ALWAYS kill all servers before starting a new one**
2. **NEVER assume ports are free**
3. **ALWAYS wait for server ready confirmation before running tests**
Portless handles port assignment automatically — no need to guess or free ports. If a stale dev server from another suite is running, stop it first (Ctrl+C the running `pnpm dev`, or `pkill -f "test/dev.ts"` if the process is orphaned) before starting a new one. Do NOT kill port 1355 — that's the portless proxy daemon which should stay running.

```bash
# ========================================
# STEP 2A: STOP ALL SERVERS
# ========================================
lsof -ti:3000 | xargs kill -9 2>/dev/null || echo "Port 3000 clear"

# ========================================
# STEP 2B: START DEV SERVER
# STEP 2A: START DEV SERVER
# ========================================
# Start dev server with the suite (in background with run_in_background=true)
# Portless auto-assigns a port and serves at $PORTLESS_URL
pnpm dev $SUITE_NAME

# ========================================
# STEP 2C: WAIT FOR SERVER READY
# STEP 2B: WAIT FOR SERVER READY
# ========================================
# Wait for server to be ready (REQUIRED - do not skip)
until curl -s http://localhost:3000/admin > /dev/null 2>&1; do sleep 1; done && echo "Server ready"
until curl -s $PORTLESS_URL/admin > /dev/null 2>&1; do sleep 1; done && echo "Server ready"

# ========================================
# STEP 2D: RUN SPECIFIC TEST
# STEP 2C: RUN SPECIFIC TEST
# ========================================
# Run ONLY the specific failing test using Playwright directly
# For E2E tests (DO NOT use pnpm test:e2e as it spawns its own server):
Expand All @@ -131,36 +124,30 @@ pnpm test:int $SUITE_NAME -t "exact test name"

If test passed with dev code, the issue is likely in bundled/production code.

**IMPORTANT: You MUST stop the dev server before starting prod server.**
**IMPORTANT: Stop the dev server (Ctrl+C) before starting prod server.**

```bash
# ========================================
# STEP 3A: STOP ALL SERVERS (INCLUDING DEV SERVER FROM STEP 2)
# ========================================
lsof -ti:3000 | xargs kill -9 2>/dev/null || echo "Port 3000 clear"

# ========================================
# STEP 3B: BUILD AND PACK FOR PROD
# STEP 3A: BUILD AND PACK FOR PROD
# ========================================
# Build all packages and pack them (this takes time - be patient)
pnpm prepare-run-test-against-prod

# ========================================
# STEP 3C: START PROD SERVER
# STEP 3B: START PROD SERVER
# ========================================
# Start prod dev server (in background with run_in_background=true)
# Portless reuses the same URL: $PORTLESS_URL
pnpm dev:prod $SUITE_NAME

# ========================================
# STEP 3D: WAIT FOR SERVER READY
# STEP 3C: WAIT FOR SERVER READY
# ========================================
# Wait for server to be ready (REQUIRED - do not skip)
until curl -s http://localhost:3000/admin > /dev/null 2>&1; do sleep 1; done && echo "Server ready"
until curl -s http://payload-monorepo.localhost:1355/admin > /dev/null 2>&1; do sleep 1; done && echo "Server ready"

# ========================================
# STEP 3E: RUN SPECIFIC TEST
# STEP 3D: RUN SPECIFIC TEST
# ========================================
# Run the specific test again using Playwright directly
pnpm exec playwright test test/$SUITE_NAME/e2e.spec.ts -g "exact test name"
# OR for integration tests:
pnpm test:int $SUITE_NAME -t "exact test name"
Expand Down Expand Up @@ -237,7 +224,7 @@ When fixing e2e tests, be aware of these eslint rules:
After fixing:

```bash
# Ensure dev server is running on port 3000
# Ensure dev server is running via portless (pnpm dev $SUITE_NAME)
# Run test multiple times to confirm stability
for i in {1..10}; do
pnpm exec playwright test test/$SUITE_NAME/e2e.spec.ts -g "exact test name" || break
Expand All @@ -247,10 +234,10 @@ done
pnpm exec playwright test test/$SUITE_NAME/e2e.spec.ts

# If you modified bundled code, test with prod build
lsof -ti:3000 | xargs kill -9 2>/dev/null
# Stop dev server first (Ctrl+C), then:
pnpm prepare-run-test-against-prod
pnpm dev:prod $SUITE_NAME
until curl -s http://localhost:3000/admin > /dev/null; do sleep 1; done
until curl -s http://payload-monorepo.localhost:1355/admin > /dev/null; do sleep 1; done
pnpm exec playwright test test/$SUITE_NAME/e2e.spec.ts
```

Expand Down Expand Up @@ -317,23 +304,20 @@ Agent:
- Error: "Error loading language: en" TypeError: network error

2. Executing reproduction workflow:
[Uses Bash tool to kill ports]
$ lsof -ti:3000 | xargs kill -9 2>/dev/null || echo "Port 3000 clear"

[Uses Bash tool with run_in_background=true]
$ pnpm dev i18n
# Portless auto-assigns port, serves at http://payload-monorepo.localhost:1355

[Waits for server to start with curl check]
$ until curl -s http://localhost:3000/admin > /dev/null; do sleep 1; done && echo "Server ready"
$ until curl -s http://payload-monorepo.localhost:1355/admin > /dev/null; do sleep 1; done && echo "Server ready"

[Uses Bash tool to run Playwright directly]
$ pnpm exec playwright test test/i18n/e2e.spec.ts -g "ensure translations update correctly when switching language"

Output shows: Test PASSED locally

3. Test passed with dev code, trying bundled code:
[Kills ports again]
$ lsof -ti:3000 | xargs kill -9 2>/dev/null
[Stops dev server (Ctrl+C)]

[Uses Bash tool]
$ pnpm prepare-run-test-against-prod
Expand All @@ -342,7 +326,7 @@ Agent:
$ pnpm dev:prod i18n

[Waits for server]
$ until curl -s http://localhost:3000/admin > /dev/null; do sleep 1; done
$ until curl -s http://payload-monorepo.localhost:1355/admin > /dev/null; do sleep 1; done

[Uses Bash tool]
$ pnpm exec playwright test test/i18n/e2e.spec.ts -g "ensure translations update correctly when switching language"
Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ Screenshots are saved to `.playwright-mcp/` and displayed inline.

**Usage flow:**

1. Ensure dev server is running on `localhost:3000`
1. Ensure dev server is running (URL is derived from folder name, e.g. `http://<folder-name>.localhost:1355` — check the startup log)
2. Call `browser_navigate` to open a page
3. Call `browser_snapshot` to get element refs
4. Use refs to interact with `browser_click`, `browser_fill_form`, etc.
Expand Down
28 changes: 27 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ This project includes:
**Prerequisites:**

- The dev server must be running (`pnpm dev`) before using the Playwright MCP
- The AI will navigate to `localhost:3000` to interact with the app
- The AI will navigate to the dev server URL (derived from folder name, e.g. `http://<folder-name>.localhost:1355`) to interact with the app

Without MCP, the AI cannot interactively browse and test the running application. It would be limited to writing test code without being able to verify the UI directly.

Expand Down Expand Up @@ -173,6 +173,32 @@ MONGODB_URL=mongodb://127.0.0.1/payloadtests # Point this to your locally instal
POSTGRES_URL=postgres://127.0.0.1:5432/payloadtests # Point this to your locally installed PostgreSQL database
```

### Running Multiple Clones in Parallel

If you work on multiple branches simultaneously (or run multiple AI agents on different clones), each clone automatically gets its own identity based on its folder name. This gives each clone a unique dev server URL and database name.

**How it works:**

- Folder `payload-feature-x` → URL `http://payload-feature-x.localhost:1355`, DB name `payload_feature_x`
- Folder `payload-bugfix-y` → URL `http://payload-bugfix-y.localhost:1355`, DB name `payload_bugfix_y`

All clones share the same Docker containers (started with `pnpm docker:start` once), but connect to separate databases within them. MongoDB and PostgreSQL auto-create databases on first connection.

**Recommended folder naming:** Use descriptive names starting with a letter (e.g. `payload-feature-x`, `payload-my-branch`). Avoid generic names like `test` or `app` which could shadow system databases — the dev server will warn you if this happens.

**To run two clones in parallel:**

1. Start Docker once: `pnpm docker:start` (from any clone)
2. Run `pnpm dev` in each clone directory — each gets its own URL and database automatically

**Overrides:**

| Variable | Purpose | Example |
|---|---|---|
| `PAYLOAD_CLONE_ID` | Override the clone identity (hostname) | `PAYLOAD_CLONE_ID=my-name pnpm dev` |
| `PAYLOAD_DB_NAME` | Override the database name | `PAYLOAD_DB_NAME=custom_db pnpm dev` |
| `PORTLESS=0` | Bypass portless entirely (use `localhost:3000`) | `PORTLESS=0 pnpm dev` |

### Running the e2e and int tests

You can run the entire test suite using `pnpm test`. If you wish to only run e2e tests, you can use `pnpm test:e2e`. If you wish to only run int tests, you can use `pnpm test:int`.
Expand Down
2 changes: 1 addition & 1 deletion ISSUE_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,4 +66,4 @@ Once they are installed you can open the `testing` tab in vscode sidebar and dri

#### Notes

- It is recommended to add the test credentials (located in `test/credentials.ts`) to your autofill for `localhost:3000/admin` as this will be required on every nodemon restart. The default credentials are `dev@payloadcms.com` as email and `test` as password.
- It is recommended to add the test credentials (located in `test/credentials.ts`) to your autofill for your dev server URL (e.g. `http://<folder-name>.localhost:1355/admin`) as this will be required on every nodemon restart. The default credentials are `dev@payloadcms.com` as email and `test` as password.
Loading
Loading