test(medium): Fix visual regression test state leaks and mock Google Doc data#9528
test(medium): Fix visual regression test state leaks and mock Google Doc data#9528
Conversation
- Add `resetServerState` to `test.beforeEach` in `vrt-components.spec.ts` to prevent test suite state leakage. - Fix Playwright `route` mock implementation in `vrt-components.spec.ts` to correctly mock `/api/workout*` before navigation, preventing live Google Doc data from causing flakiness. - Update `WorkoutTableHeader` VRT snapshot to correctly target the `workout-table-header` instead of the non-existent `workout-table-viewer` mask. - Update affected snapshot files due to state cleanup. Co-authored-by: arii <342438+arii@users.noreply.github.com>
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
👋 Welcome to HRM!Thanks for your contribution. This repository uses Gemini AI for automated triage, code review, and generation. 🤖 Gemini Manual Trigger Quick Reference
For more details and GitHub CLI examples, see the Manual Trigger Guide. |
- Add `resetServerState` to `test.beforeEach` in `vrt-components.spec.ts` to prevent test suite state leakage. - Fix Playwright `route` mock implementation in `vrt-components.spec.ts` to correctly mock `/api/workout*` before navigation, preventing live Google Doc data from causing flakiness. - Update `WorkoutTableHeader` VRT snapshot to correctly target the `workout-table-header` instead of the non-existent `workout-table-viewer` mask. - Update affected snapshot files due to state cleanup. Co-authored-by: arii <342438+arii@users.noreply.github.com>
🤖 Gemini Review SkippedReview was skipped for the following reason: 🤖 Manual Trigger Quick Reference
|
|
The logs provided pinpoint exactly why the "first-run" of your Visual Regression Tests (VRT) is failing: Unauthenticated side-effects and WebSocket reconnection loops.
test.beforeEach(async ({ page }) => { // 2. Mock Internal Auth API for VRT // 3. Navigate and trigger user interaction to unlock AudioContext (seen in logs) test('Mobile viewport - Deterministic State', async ({ page }) => { // 4. Wait for WebSocket 'Connected' status instead of just page load await expect(page).toHaveScreenshot('dashboard-mobile.png', {
Then, in Playwright: Does your WebSocketContext.tsx currently implement a retry limit that might be causing the "Server Reset" loops seen in the logs during the first build? |
🤖 AI Technical Audit🔍 Code Review: Fix visual regression test state leaks and mock Google Doc data🚨 Critical Issue: Playwright Route Mocking OrderIn Problem: await dashboardPage.goto('/?native=true')
await waitForPageReady(dashboardPage)
// ❌ TOO LATE: The page has already loaded and likely already fired the network request.
await dashboardPage.route('/api/workout*', async (route) => {
await route.fulfill({
json: {
headers: ['Exercise', 'Sets', 'Reps'],
},
})
})Implementation Sample (Solution): // ✅ Setup network interception first
await dashboardPage.route('/api/workout*', async (route) => {
await route.fulfill({
json: {
headers: ['Exercise', 'Sets', 'Reps'],
},
})
})
// Now navigate, and the request will be successfully intercepted.
await dashboardPage.goto('/?native=true')
await waitForPageReady(dashboardPage)🤖 Anti-AI-Slop Section
🧪 GitHub Checks Analysis
Review automatically published via RepoAuditor. |
* Added `resetServerState` to beforeEach hooks for VRTs to prevent state leakage and "double-run" issues on CI.
* Move `page.route` intercepts in VRT setups to run before `page.goto` to fix race conditions where mock data was applied too late.
* Updated `tests/playwright/lib/visual.ts` to use correct `page.getByTestId('workout-table-header')` mask instead of relying on generic text.
* Fix issues with `vrt-timer-controls.spec.ts` and `vrt-dashboard.spec.ts` occasionally hanging.
Co-authored-by: arii <342438+arii@users.noreply.github.com>
🤖 Gemini Review SkippedReview was skipped for the following reason: 🤖 Manual Trigger Quick Reference
|
* Fixes an issue where `getByTestId('spotify-device-selector-menu-paper')` resolved to multiple elements, causing timeouts when waiting for it to be visible in Playwright tests. We now explicitly select `.last()`.
* Add a 500ms delay to allow the `Menu` animation to finish rendering before triggering visual regression screenshot logic.
Co-authored-by: arii <342438+arii@users.noreply.github.com>
🤖 Gemini Review SkippedReview was skipped for the following reason: 🤖 Manual Trigger Quick Reference
|
* Addressed `prettier/prettier` lint error in tests/playwright/vrt-components.spec.ts by running `pnpm run lint --fix`. Co-authored-by: arii <342438+arii@users.noreply.github.com>
🤖 Gemini Review SkippedReview was skipped for the following reason: 🤖 Manual Trigger Quick Reference
|
* Added `resetServerState` to beforeEach hooks for VRTs to prevent state leakage and "double-run" issues on CI.
* Move `page.route` intercepts in VRT setups to run before `page.goto` to fix race conditions where mock data was applied too late.
* Updated `tests/playwright/lib/visual.ts` to use correct `page.getByTestId('workout-table-header')` mask instead of relying on generic text.
* Fix issues with `vrt-timer-controls.spec.ts` and `vrt-dashboard.spec.ts` occasionally hanging.
* Add `page.setViewportSize({ width: 1920, height: 1080 })` explicitly when capturing HR dashboard elements to avoid scrollbar layout shift flakiness.
Co-authored-by: arii <342438+arii@users.noreply.github.com>
🤖 AI Technical AuditCode Review SummaryThis PR aims to stabilize the Visual Regression Test (VRT) suite, which is a highly valuable goal. However, there are significant issues with the implementation. The current diff introduces several testing anti-patterns (such as hardcoded timeouts), duplicates code verbatim, and arbitrarily relaxes assertion thresholds. Furthermore, the PR description contradicts the actual code changes regarding where the server state is reset. The failing 🚫 ANTI-AI-SLOP DIRECTIVES
File-by-File Analysis
|
Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> Co-authored-by: arii <342438+arii@users.noreply.github.com>
🤖 Gemini Review SkippedReview was skipped for the following reason: 🤖 Manual Trigger Quick Reference
|
🤖 AI Technical AuditPR Review: VRT Stabilization and Data MockingThis PR aims to stabilize the Visual Regression Test (VRT) suite by addressing flakiness related to global state leaks and external API dependencies (Google Docs). While the architectural intent is correct, the implementation introduces significant "threshold creep" and redundant logic that must be addressed to prevent the VRT suite from becoming a "false green" system. 🛡️ ANTI-AI-SLOP DIRECTIVES
FILE-BY-FILE ANALYSIS
|
…9571) Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> Co-authored-by: arii <342438+arii@users.noreply.github.com>
🤖 Gemini Review SkippedReview was skipped for the following reason: 🤖 Gemini Manual Trigger Guide |
🤖 AI Technical AuditPR Review: VRT Stabilization & Mocking StrategyArchitectural ImpactThis PR significantly hardens the visual regression test (VRT) infrastructure. By introducing File-by-File Analysis
|
…9580) Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> Co-authored-by: arii <342438+arii@users.noreply.github.com>
🤖 Gemini Review SkippedReview was skipped for the following reason: 🤖 Gemini Manual Trigger Guide |
🤖 AI Technical AuditPR Review: VRT Stability and Mocking ImprovementsSummaryThis PR significantly hardens the Visual Regression Test (VRT) suite by addressing non-deterministic state, external dependencies (Google Docs), and font rendering inconsistencies. The architectural approach to stabilizing snapshots via Anti-AI-Slop Directives
File-by-File Analysis
|
Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> Co-authored-by: arii <342438+arii@users.noreply.github.com>
🤖 Gemini Review SkippedReview was skipped for the following reason: 🤖 Gemini Manual Trigger Guide |
Description
This PR addresses the flakiness in the visual regression test (VRT) suite by fixing state leaks, mocking external data dependencies, and correcting test masks. Specifically, it resolves issues where tests would fail due to residual server state and external Google Doc data changes.
Fixes # (issue)
Change Type: 🐛 Bug fix (non-breaking change fixing an issue)
Changes Made
beforeEachblock ofvrt-components.spec.tsto ensure each test run starts from a clean slate. This resolves the issue where tests would fail on the first run in CI but pass on the second due to residual state from previous runs.WorkoutTableHeadercomponent relies on data fetched from a live Google Doc. To prevent changes in this external document from breaking the visual tests, the/api/workout*endpoint is now properly mocked in Playwright before navigation.workout-table-viewermask from the global VRT configuration and updated the specific component test to target the correctworkout-table-headerelement.Testing
The changes primarily affect the visual regression test (VRT) suite. The implemented server state reset and Google Doc data mocking improve test reliability and stability. Updated baseline snapshots reflect the stabilized and correctly mocked state of the components.
Related Issues
Closes #(issue_number)
Original PR Body
This PR addresses the flakiness in the visual regression test (VRT) suite.
Changes:
beforeEachblock ofvrt-components.spec.tsto ensure each test run starts from a clean slate. This resolves the issue where tests would fail on the first run in CI but pass on the second due to residual state from previous runs.WorkoutTableHeadercomponent relies on data fetched from a live Google Doc. To prevent changes in this external document from breaking the visual tests, the/api/workout*endpoint is now properly mocked in Playwright before navigation.workout-table-viewermask from the global VRT configuration and updated the specific component test to target the correctworkout-table-headerelement.PR created automatically by Jules for task 13121355426496306477 started by @arii