From 6a7e06d0d4e1a6d91c5be44836d1ae10bf5aa778 Mon Sep 17 00:00:00 2001 From: Subhash Khileri Date: Tue, 7 Apr 2026 13:42:10 +0530 Subject: [PATCH] chore(docs) : 1.1.24 docs update --- CLAUDE.md | 284 ++++++++++++++++++ docs/.vitepress/config.ts | 6 +- docs/CLAUDE.md | 180 ++++++++--- docs/api/deployment/rhdh-deployment.md | 13 +- docs/api/utils/plugin-metadata.md | 179 +++-------- docs/changelog.md | 2 +- .../configuration/environment-variables.md | 4 +- .../core-concepts/playwright-fixtures.md | 19 +- docs/guide/deployment/rhdh-deployment.md | 5 +- docs/guide/utilities/index.md | 22 ++ docs/guide/utilities/plugin-metadata.md | 103 ++++--- .../reference/environment-variables.md | 29 +- docs/overlay/reference/run-e2e.md | 248 +++++++++++++++ .../test-structure/configuration-files.md | 135 +++++++++ .../test-structure/directory-layout.md | 33 ++ docs/overlay/tutorials/ci-pipeline.md | 56 +++- docs/overlay/tutorials/running-locally.md | 15 + 17 files changed, 1105 insertions(+), 228 deletions(-) create mode 100644 CLAUDE.md create mode 100644 docs/overlay/reference/run-e2e.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..b9e3df8 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,284 @@ +# CLAUDE.md + +This file provides guidance to AI coding tools when working with the `@red-hat-developer-hub/e2e-test-utils` package. + +## Package Purpose + +A shared npm package (`@red-hat-developer-hub/e2e-test-utils`) that provides everything needed to write and run E2E tests for Red Hat Developer Hub (RHDH) plugins. It handles: + +- Deploying RHDH to OpenShift (Helm or Operator) +- Deploying Keycloak for authentication +- Playwright test fixtures, helpers, and page objects +- Dynamic plugin configuration with metadata-based OCI resolution +- Config file merging (common defaults + auth + user overrides) +- Kubernetes API operations +- Per-project namespace teardown in CI + +Published to npm under `@red-hat-developer-hub` scope. Primary consumers are overlay workspaces in `rhdh-plugin-export-overlays`. + +## Project Structure + +``` +src/ +├── deployment/ +│ ├── rhdh/ # RHDHDeployment class — core deployment orchestration +│ │ ├── deployment.ts # Main class (configure, deploy, waitUntilReady, teardown) +│ │ ├── types.ts # DeploymentOptions, DeploymentConfig +│ │ ├── constants.ts # Config paths, auth providers, chart URLs +│ │ └── config/ # YAML templates (common/, auth/, helm/, operator/) +│ ├── keycloak/ # KeycloakHelper — Keycloak Helm deployment + OIDC setup +│ └── orchestrator/ # Workflow orchestrator installer +├── playwright/ +│ ├── fixtures/test.ts # Custom fixtures: rhdh, uiHelper, loginHelper, baseURL +│ ├── base-config.ts # Base Playwright config (reporters, timeouts, video/trace) +│ ├── global-setup.ts # Pre-test: check binaries, detect cluster, deploy Keycloak +│ ├── run-once.ts # Cross-worker one-time execution with file locking +│ ├── teardown-reporter.ts # Per-project namespace cleanup (CI only) +│ ├── teardown-namespaces.ts # Custom namespace registry for teardown +│ ├── helpers/ # LoginHelper, UIhelper, APIHelper, RbacApiHelper, etc. +│ ├── pages/ # Page objects: CatalogPage, HomePage, ExtensionsPage, etc. +│ └── page-objects/ # Shared element selectors/constants +├── utils/ +│ ├── plugin-metadata.ts # Plugin discovery, OCI resolution, PR/nightly modes +│ ├── workspace-paths.ts # Path resolution from Playwright testDir (not CWD) +│ ├── kubernetes-client.ts # KubernetesClientHelper (K8s API wrapper) +│ ├── merge-yamls.ts # Deep merge with array strategies (replace, concat, byKey) +│ ├── bash.ts # zx shell wrapper ($, runQuietUnlessFailure) +│ └── common.ts # envsubst, requireEnv helpers +└── eslint/ + └── base.config.ts # createEslintConfig() factory for consumers +``` + +## Package Exports + +| Import Path | What It Provides | +| --------------------- | ------------------------------------------------------------------------------ | +| `./test` | `test`, `expect` — Playwright fixtures with RHDH-specific fixtures | +| `./playwright-config` | `defineConfig`, `baseConfig` — base Playwright configuration | +| `./rhdh` | `RHDHDeployment` — deployment orchestration class | +| `./utils` | `$`, `KubernetesClientHelper`, `WorkspacePaths`, `envsubst`, `mergeYamlFiles` | +| `./helpers` | `UIhelper`, `LoginHelper`, `APIHelper`, `RbacApiHelper`, `AccessibilityHelper` | +| `./pages` | `CatalogPage`, `HomePage`, `ExtensionsPage`, etc. | +| `./keycloak` | `KeycloakHelper` — Keycloak deployment and OIDC configuration | +| `./teardown` | `registerTeardownNamespace` — custom namespace cleanup registration | +| `./orchestrator` | `installOrchestrator` — workflow orchestrator deployment | +| `./eslint` | `createEslintConfig` — ESLint config factory | +| `./tsconfig` | Base TypeScript configuration (JSON, not code) | + +## Build System + +```bash +yarn build # Clean + tsc + copy config YAML files + copy shell scripts to dist/ +yarn test # node --test "dist/**/*.test.js" (must build first) +yarn check # typecheck + lint + prettier +``` + +- **TypeScript**: ES2022 target, ESNext modules, strict mode +- **Module system**: ESM (`"type": "module"`) +- **Build output**: `dist/` (JS + .d.ts declarations + config YAMLs) +- **Config files**: Copied as-is from `src/deployment/*/config/` to `dist/` during build +- **Package manager**: Yarn v3.8.7, Node >= 22.18.0 + +## Key Architectural Concepts + +### Configuration Merging (3-level cascade) + +``` +Package defaults (config/common/) + ↓ deep merge +Auth-specific (config/auth/{keycloak|guest|github}/) + ↓ deep merge +User config (workspace's tests/config/*.yaml) + ↓ += Final merged config +``` + +Array merge uses "replace" strategy by default. Plugin arrays use `byKey: "package"` with normalized keys (strips trailing `-dynamic`). + +### Plugin Metadata Resolution + +`RHDHDeployment.deploy()` internally calls `processPluginsForDeployment()` which operates in two modes: + +| Mode | Detection | Behavior | +| ------------- | ---------------------------------------------------------- | --------------------------------------------------------------------------------------------- | +| **PR check** | `GIT_PR_NUMBER` is set | Injects metadata configs + resolves to PR-built OCI URLs (`pr_{number}__{version}`) | +| **Nightly** | `E2E_NIGHTLY_MODE=true` or `JOB_NAME` contains `periodic-` | Resolves to released OCI refs from `spec.dynamicArtifact` in metadata; skips config injection | +| **Local dev** | Neither set | Uses local paths as-is; injects metadata configs | + +Priority: `GIT_PR_NUMBER` (forces PR mode) > `E2E_NIGHTLY_MODE` > `JOB_NAME` + +When no `dynamic-plugins.yaml` exists in the workspace, plugins are auto-generated from `metadata/*.yaml` files. + +### WorkspacePaths + +Static utility that resolves config file paths from `test.info().project.testDir` (Playwright-provided absolute path) instead of `process.cwd()`. This is critical because tests can run from two contexts: + +- Workspace level: `cd workspaces/tech-radar/e2e-tests && yarn test` +- Repo root: `./run-e2e.sh -w tech-radar` + +The worker fixture also does `process.chdir(e2eRoot)` as a complementary safety net. + +### Playwright Fixtures + +Worker fixture creates `RHDHDeployment(projectName)` from the Playwright project name (which becomes the K8s namespace) and sets CWD to the workspace e2e-tests directory. Test-scoped fixtures (`uiHelper`, `loginHelper`, `baseURL`) are created fresh per test. See the [consumer code example](#typical-test-structure) for usage. + +### runOnce — Cross-Worker Deduplication + +`deploy()` uses `runOnce()` internally to execute exactly once per test run, even when Playwright restarts workers after test failures. Uses file-based flags with `proper-lockfile` in `/tmp/playwright-once-{ppid}/`. + +### Teardown Reporter + +Custom Playwright reporter that deletes namespaces per-project (not per-suite) as tests complete. Only active when `CI=true`. Needed because: + +- `afterAll` hooks fire on worker restart (before retries can run) +- Worker fixture teardown has the same problem +- `globalTeardown` has no visibility into which projects ran + +### Global Setup + +Runs once before all tests: + +1. Checks required binaries (`oc`, `kubectl`, `helm`) +2. Auto-detects cluster domain (`K8S_CLUSTER_ROUTER_BASE`) +3. Deploys Keycloak (unless `SKIP_KEYCLOAK_DEPLOYMENT=true`) + +### RHDHDeployment.deploy() Flow + +1. Merges config files (common + auth + user) +2. Injects plugin metadata into dynamic plugins config (PR/local mode) +3. Applies ConfigMaps (app-config, dynamic-plugins) +4. Applies Secrets (with `envsubst` for environment variable substitution) +5. Installs RHDH via Helm or Operator +6. Waits for readiness: pod `Ready=True` + route HTTP health check +7. Sets `RHDH_BASE_URL` environment variable + +Helm upgrades perform `scaleDownAndRestart()` to avoid `MigrationLocked` errors. Fresh installs skip this. + +## Key Environment Variables + +| Variable | Purpose | Default | +| ------------------------------------- | ------------------------------------------------ | -------- | +| `RHDH_VERSION` | RHDH version to deploy | `"next"` | +| `INSTALLATION_METHOD` | `"helm"` or `"operator"` | `"helm"` | +| `GIT_PR_NUMBER` | PR number — triggers PR-mode OCI resolution | - | +| `E2E_NIGHTLY_MODE` | `"true"` or `"1"` — nightly mode | - | +| `JOB_NAME` | CI job name; `periodic-` prefix triggers nightly | - | +| `SKIP_KEYCLOAK_DEPLOYMENT` | Skip Keycloak in global setup | - | +| `K8S_CLUSTER_ROUTER_BASE` | Cluster domain (auto-detected) | - | +| `CI` | Enables `forbidOnly`, teardown reporter | - | +| `RHDH_SKIP_PLUGIN_METADATA_INJECTION` | Skip metadata config injection in PR mode | - | +| `CATALOG_INDEX_IMAGE` | Override catalog index image | - | + +## Consumer Perspective (Overlay Workspaces) + +Changes to this package affect ~67 overlay workspaces. Understanding how they use the package is critical before modifying APIs. + +### Typical Test Structure + +```typescript +import { test, expect } from "@red-hat-developer-hub/e2e-test-utils/test"; +import { $ } from "@red-hat-developer-hub/e2e-test-utils/utils"; + +test.describe("My Plugin", () => { + test.beforeAll(async ({ rhdh }) => { + // 1. Pre-deployment setup (external services, K8s resources) + await $`bash ${setupScript} ${rhdh.deploymentConfig.namespace}`; + process.env.SERVICE_URL = await rhdh.k8sClient.getRouteLocation( + namespace, + "my-svc", + ); + + // 2. Configure + deploy RHDH + await rhdh.configure({ auth: "keycloak" }); + await rhdh.deploy(); + }); + + test.beforeEach(async ({ loginHelper }) => { + await loginHelper.loginAsKeycloakUser(); + }); + + test("verify feature", async ({ uiHelper }) => { + await uiHelper.openSidebar("My Plugin"); + await uiHelper.verifyHeading("Expected Title"); + }); +}); +``` + +### Consumer Gotchas + +- **Config paths are workspace-relative** — `appConfig: "tests/config/app-config.yaml"` resolves from the workspace's `e2e-tests/` directory via `WorkspacePaths`, not from `process.cwd()`. +- **`$` shell is for setup, not assertions** — used in `beforeAll` for deployment scripts. Use `$({ stdio: "pipe" })` to capture output. +- **`rhdh.configure()` must precede `rhdh.deploy()`** — every workspace follows this order. +- **`defineConfig` only needs `projects`** — base config handles reporters, timeouts, video/trace, global setup. Consumers just specify project names (which become K8s namespaces). + +### What Consumers Use vs What's Internal + +| Consumer-facing (public API) | Internal (used by deploy()) | +| ----------------------------------------------- | ------------------------------- | +| `test`, `expect` fixtures | `processPluginsForDeployment()` | +| `rhdh.configure()`, `rhdh.deploy()` | `isNightlyJob()` | +| `UIhelper`, `LoginHelper`, `APIHelper` | `generatePluginsFromMetadata()` | +| `$`, `WorkspacePaths`, `KubernetesClientHelper` | `injectMetadataConfig()` | +| `defineConfig` from `./playwright-config` | `resolvePluginPackages()` | +| `registerTeardownNamespace` from `./teardown` | `disablePluginWrappers()` | + +## Testing + +Tests use Node.js built-in `node:test` module (not Playwright): + +```bash +yarn build && yarn test +``` + +Test files: + +- `src/deployment/rhdh/deployment.test.ts` — plugin merge behavior +- `src/utils/merge-yamls.test.ts` — YAML merge strategies +- `src/utils/tests/plugin-metadata.*.test.ts` — metadata resolution (PR, nightly, fixtures) + +## Naming Conventions + +- **Class**: `UIhelper` (capital U, lowercase h) — matches source code +- **Fixture**: `uiHelper` (camelCase) — used in test fixtures +- **Method names**: Follow source exactly (e.g., `verifyTextinCard` not `verifyTextInCard`) + +## Documentation + +VitePress docs in `docs/` — standalone package with its own `package.json`. See `docs/CLAUDE.md` for documentation-specific guidance. + +```bash +cd docs && yarn install && yarn dev # http://localhost:5173 +``` + +Deployed to GitHub Pages via `.github/workflows/deploy-docs.yml` on pushes to `main`. + +## Common Tasks + +### Adding a New Export Path + +1. Add the entry to `package.json` `exports` with `types` and `default` +2. Create the source file and its `index.ts` barrel export +3. Document in the VitePress docs (guide + API sections) + +### Adding a New Auth Provider + +1. Create config files in `src/deployment/rhdh/config/auth//` +2. Add the provider to the `AUTH_PROVIDERS` constant in `constants.ts` +3. Update the `auth` type in `types.ts` + +### Modifying Config Merging + +Config merge order is defined in `deployment.ts` `_buildDeploymentConfig()`. Array merge strategies are in `merge-yamls.ts`. Plugin-specific merging uses `getNormalizedPluginMergeKey()` to deduplicate entries with/without `-dynamic` suffix. + +### Updating Plugin Metadata Logic + +All in `src/utils/plugin-metadata.ts`. Key functions: + +- `isNightlyJob()` — mode detection +- `processPluginsForDeployment()` — unified entry point for PR + nightly +- `generatePluginsFromMetadata()` — auto-generate from metadata files +- `resolvePluginPackages()` — OCI URL resolution +- `injectMetadataConfig()` — merge metadata configs into plugin entries +- `disablePluginWrappers()` — disable local wrappers when using OCI images + +Test thoroughly — changes affect all overlay workspaces. Run `yarn build && yarn test` to verify. diff --git a/docs/.vitepress/config.ts b/docs/.vitepress/config.ts index 76bef3e..5cfec6b 100644 --- a/docs/.vitepress/config.ts +++ b/docs/.vitepress/config.ts @@ -33,7 +33,7 @@ export default defineConfig({ { text: "Examples", link: "/examples/" }, { text: "Overlay Testing", link: "/overlay/" }, { - text: "v1.1.22", + text: "v1.1.24", items: [{ text: "Changelog", link: "/changelog" }], }, ], @@ -395,6 +395,10 @@ export default defineConfig({ text: "Environment Variables", link: "/overlay/reference/environment-variables", }, + { + text: "Unified Test Runner (run-e2e.sh)", + link: "/overlay/reference/run-e2e", + }, { text: "Local OCI Testing", link: "/overlay/reference/local-oci-testing", diff --git a/docs/CLAUDE.md b/docs/CLAUDE.md index 3a341cf..0882ece 100644 --- a/docs/CLAUDE.md +++ b/docs/CLAUDE.md @@ -2,33 +2,51 @@ This file contains essential information for maintaining and extending the VitePress documentation for `@red-hat-developer-hub/e2e-test-utils`. +For package-level architecture, source code structure, and build system details, see the root `CLAUDE.md`. + ## Documentation Structure +The docs serve two audiences with distinct sections: + ``` docs/ ├── .vitepress/ -│ └── config.ts # VitePress configuration (nav, sidebar, theme) -├── snippets/ # Reusable code snippets (included via ) -├── guide/ # Conceptual documentation -│ ├── core-concepts/ # Architecture, fixtures, patterns -│ ├── deployment/ # RHDH, Keycloak, Helm, Operator -│ ├── helpers/ # UIhelper, LoginHelper, APIHelper -│ ├── page-objects/ # CatalogPage, HomePage, etc. -│ ├── utilities/ # Kubernetes client, bash, YAML merging -│ └── configuration/ # Config files, ESLint, TypeScript -├── api/ # API reference (method signatures, types) -│ ├── deployment/ # RHDHDeployment, KeycloakHelper -│ ├── playwright/ # Fixtures, base config, global setup -│ ├── helpers/ # Helper class APIs -│ ├── pages/ # Page object APIs -│ ├── utils/ # Utility APIs -│ └── eslint/ # ESLint config API -├── tutorials/ # Step-by-step learning guides -├── examples/ # Copy-paste ready code examples -├── index.md # Home page -└── changelog.md # Version history +│ └── config.ts # VitePress configuration (nav, sidebar, theme) +├── snippets/ # Reusable code snippets (included via ) +├── guide/ # Package documentation (for any consumer) +│ ├── core-concepts/ # Architecture, fixtures, patterns, global setup +│ ├── deployment/ # RHDH, Keycloak, Helm, Operator +│ ├── helpers/ # UIhelper, LoginHelper, APIHelper +│ ├── page-objects/ # CatalogPage, HomePage, etc. +│ ├── utilities/ # Kubernetes client, bash, YAML merging, plugin metadata +│ └── configuration/ # Config files, ESLint, TypeScript, env vars +├── api/ # API reference (method signatures, types) +│ ├── deployment/ # RHDHDeployment, KeycloakHelper +│ ├── playwright/ # Fixtures, base config, global setup +│ ├── helpers/ # Helper class APIs +│ ├── pages/ # Page object APIs +│ ├── utils/ # Utility APIs (including plugin metadata) +│ └── eslint/ # ESLint config API +├── overlay/ # Overlay-specific docs (for rhdh-plugin-export-overlays) +│ ├── test-structure/ # Directory layout, config files, spec files +│ ├── tutorials/ # New workspace, running locally, CI pipeline, secrets +│ ├── examples/ # Tech radar, basic plugin +│ └── reference/ # Env vars, run-e2e.sh, OCI testing, scripts, patterns +├── tutorials/ # Step-by-step learning guides (package-level) +├── examples/ # Copy-paste ready code examples +├── index.md # Home page +└── changelog.md # Version history ``` +### Two Documentation Audiences + +| Section | Audience | Scope | +|---------|----------|-------| +| **Guide + API + Tutorials + Examples** | Any project using `@red-hat-developer-hub/e2e-test-utils` | Package features, API, patterns | +| **Overlay** (`/overlay/`) | Contributors writing tests in `rhdh-plugin-export-overlays` | Overlay-specific workflows, CI, run-e2e.sh | + +Overlay pages include a `::: tip Overlay Documentation` banner to indicate scope. + ## Key Conventions ### Naming Conventions @@ -78,10 +96,10 @@ Always add "Related Pages" section at the end of documentation pages: ```markdown ## Related Pages -- [Page Name](/path/to/page.md) - Brief description +- [Page Name](/path/to/page) - Brief description ``` -### Guide vs API vs Tutorial vs Example +### Guide vs API vs Tutorial vs Example vs Overlay | Section | Purpose | Content Style | |---------|---------|---------------| @@ -89,6 +107,38 @@ Always add "Related Pages" section at the end of documentation pages: | **API** | Reference | Method signatures, parameter tables, brief examples | | **Tutorial** | Teach step-by-step | Sequential steps, explanations, learning-focused | | **Example** | Copy-paste code | Minimal prose, complete working code | +| **Overlay** | Overlay-specific workflows | Mix of reference and tutorial, always with `::: tip` banner | + +## Important Architectural Concepts to Document Correctly + +These are frequently misunderstood. When writing or updating docs, ensure accuracy: + +### Plugin Metadata Is Internal +Plugin metadata functions (`processPluginsForDeployment`, `isNightlyJob`, `generatePluginsFromMetadata`) are NOT part of the public `./utils` export. They are internal to `RHDHDeployment.deploy()`. Document them as automatic behavior, not as functions consumers call. + +### Three Execution Modes +| Mode | Detection | Plugin Resolution | +|------|-----------|-------------------| +| PR check | `GIT_PR_NUMBER` set | PR-built OCI images + metadata config injection | +| Nightly | `E2E_NIGHTLY_MODE=true` | Released OCI refs from metadata; no config injection | +| Local dev | Neither set | Local paths as-is; metadata config injection | + +Priority: `GIT_PR_NUMBER` > `E2E_NIGHTLY_MODE` > `JOB_NAME` + +### WorkspacePaths +Resolves config paths from `test.info().project.testDir` (not CWD). This enables tests to work from both workspace level (`yarn test`) and repo root (`run-e2e.sh`). The worker fixture also does `process.chdir(e2eRoot)` as a complementary measure. + +### Teardown Reporter (Not afterAll) +Namespace cleanup uses a custom Playwright Reporter, not `afterAll` hooks or fixture teardown. This is because `afterAll` and fixture cleanup fire on worker restart (before retries), which would delete the namespace while tests still need it. + +### Configuration Merging +3-level cascade: package defaults -> auth-specific -> user overrides. Arrays use "replace" strategy by default. Plugin arrays use `byKey: "package"` with normalized keys (strips `-dynamic` suffix). + +### deploy() Has Built-in Protection +`rhdh.deploy()` uses `runOnce()` internally — it executes exactly once per test run, even across worker restarts. No wrapping needed unless there's other expensive setup (use `test.runOnce("key", fn)` for that). + +### E2E_NIGHTLY_MODE Accepts Both Values +The code checks `=== "true" || === "1"`. Document both. ## VitePress Configuration @@ -105,10 +155,22 @@ Always add "Related Pages" section at the end of documentation pages: ``` ### Sidebar Structure -- `collapsed: false` - Section expanded by default -- `collapsed: true` - Section collapsed by default +The sidebar has 5 top-level sections, each keyed by URL prefix: +- `/guide/` — Package guide (7 subsections) +- `/api/` — API reference (6 subsections) +- `/tutorials/` — Package tutorials +- `/examples/` — Code examples +- `/overlay/` — Overlay testing (5 subsections) + +Settings: +- `collapsed: false` — Section expanded by default +- `collapsed: true` — Section collapsed by default - Items without `collapsed` are always visible +### Version Updates +1. Update version in `docs/.vitepress/config.ts` nav dropdown text +2. Update `docs/changelog.md` with changes + ## Source Code Locations When documenting, reference these source files: @@ -116,34 +178,57 @@ When documenting, reference these source files: | Component | Source Path | |-----------|-------------| | UIhelper | `src/playwright/helpers/ui-helper.ts` | -| LoginHelper | `src/playwright/helpers/login-helper.ts` | +| LoginHelper | `src/playwright/helpers/common.ts` | | APIHelper | `src/playwright/helpers/api-helper.ts` | -| RHDHDeployment | `src/deployment/rhdh/rhdh-deployment.ts` | -| KeycloakHelper | `src/deployment/keycloak/keycloak-helper.ts` | +| AuthApiHelper | `src/playwright/helpers/auth-api-helper.ts` | +| RbacApiHelper | `src/playwright/helpers/rbac-api-helper.ts` | +| AccessibilityHelper | `src/playwright/helpers/accessibility.ts` | +| RHDHDeployment | `src/deployment/rhdh/deployment.ts` | +| Deployment Types | `src/deployment/rhdh/types.ts` | +| Deployment Constants | `src/deployment/rhdh/constants.ts` | +| KeycloakHelper | `src/deployment/keycloak/deployment.ts` | | Page Objects | `src/playwright/pages/*.ts` | | Fixtures | `src/playwright/fixtures/test.ts` | | Base Config | `src/playwright/base-config.ts` | | Global Setup | `src/playwright/global-setup.ts` | +| runOnce | `src/playwright/run-once.ts` | | Teardown Reporter | `src/playwright/teardown-reporter.ts` | | Teardown Namespaces | `src/playwright/teardown-namespaces.ts` | +| Plugin Metadata | `src/utils/plugin-metadata.ts` | +| WorkspacePaths | `src/utils/workspace-paths.ts` | +| KubernetesClientHelper | `src/utils/kubernetes-client.ts` | +| YAML Merging | `src/utils/merge-yamls.ts` | +| Bash ($) | `src/utils/bash.ts` | +| envsubst | `src/utils/common.ts` | +| Config Templates | `src/deployment/rhdh/config/{common,auth,helm,operator}/` | ## Common Tasks ### Update API Documentation 1. Read the source file to get accurate method signatures -2. Update both Guide (`/guide/helpers/`) and API (`/api/helpers/`) sections +2. Update both Guide (`/guide/`) and API (`/api/`) sections 3. Ensure method names match source exactly +4. Check that documented functions are actually exported (verify via `src/*/index.ts`) ### Add New Feature Documentation 1. Add to Guide section with detailed explanation 2. Add to API section with method signatures 3. Add example to Examples section if applicable -4. Update Related Pages in relevant files -5. Add to sidebar in `config.ts` - -### Update Version -1. Update version in `docs/.vitepress/config.ts` nav dropdown -2. Update `docs/changelog.md` with changes +4. If overlay-specific, add to Overlay section with `::: tip` banner +5. Update Related Pages in relevant files +6. Add to sidebar in `config.ts` + +### Add Overlay Documentation +1. Create page in appropriate `overlay/` subdirectory +2. Add `::: tip Overlay Documentation` banner at top +3. Add sidebar entry in `config.ts` under the `/overlay/` section +4. Cross-reference from related guide/API pages where relevant + +### Update Environment Variables +Environment variables are documented in multiple places — keep them in sync: +- `docs/guide/configuration/environment-variables.md` — Package-level env vars +- `docs/overlay/reference/environment-variables.md` — Overlay-specific env vars (superset) +- Individual feature pages that reference specific vars ## Standalone Documentation @@ -197,23 +282,18 @@ The workflow: Base URL is configured as `/rhdh-e2e-test-utils/` in `config.ts`. -## Known Issues / TODOs - -- [ ] Consider adding search indexing for Algolia (currently using local search) -- [ ] Add API documentation for remaining utility functions -- [ ] Consider adding interactive code examples with StackBlitz - ## Package Exports Reference | Export | Import Path | Description | |--------|-------------|-------------| -| Test fixtures | `@red-hat-developer-hub/e2e-test-utils/test` | Main test API | -| Playwright config | `@red-hat-developer-hub/e2e-test-utils/playwright-config` | Base config | -| RHDH deployment | `@red-hat-developer-hub/e2e-test-utils/rhdh` | RHDHDeployment | -| Keycloak | `@red-hat-developer-hub/e2e-test-utils/keycloak` | KeycloakHelper | -| Helpers | `@red-hat-developer-hub/e2e-test-utils/helpers` | UIhelper, LoginHelper, etc. | -| Page objects | `@red-hat-developer-hub/e2e-test-utils/pages` | CatalogPage, HomePage, etc. | -| Utilities | `@red-hat-developer-hub/e2e-test-utils/utils` | KubernetesClientHelper, etc. | -| Teardown | `@red-hat-developer-hub/e2e-test-utils/teardown` | Custom namespace teardown registration | -| ESLint | `@red-hat-developer-hub/e2e-test-utils/eslint` | ESLint config | -| TypeScript | `@red-hat-developer-hub/e2e-test-utils/tsconfig` | TSConfig base | +| Test fixtures | `@red-hat-developer-hub/e2e-test-utils/test` | `test`, `expect`, fixtures | +| Playwright config | `@red-hat-developer-hub/e2e-test-utils/playwright-config` | `defineConfig`, `baseConfig` | +| RHDH deployment | `@red-hat-developer-hub/e2e-test-utils/rhdh` | `RHDHDeployment` | +| Keycloak | `@red-hat-developer-hub/e2e-test-utils/keycloak` | `KeycloakHelper` | +| Helpers | `@red-hat-developer-hub/e2e-test-utils/helpers` | `UIhelper`, `LoginHelper`, etc. | +| Page objects | `@red-hat-developer-hub/e2e-test-utils/pages` | `CatalogPage`, `HomePage`, etc. | +| Utilities | `@red-hat-developer-hub/e2e-test-utils/utils` | `$`, `KubernetesClientHelper`, `WorkspacePaths`, etc. | +| Teardown | `@red-hat-developer-hub/e2e-test-utils/teardown` | `registerTeardownNamespace` | +| Orchestrator | `@red-hat-developer-hub/e2e-test-utils/orchestrator` | `installOrchestrator` | +| ESLint | `@red-hat-developer-hub/e2e-test-utils/eslint` | `createEslintConfig` | +| TypeScript | `@red-hat-developer-hub/e2e-test-utils/tsconfig` | Base TSConfig (JSON) | diff --git a/docs/api/deployment/rhdh-deployment.md b/docs/api/deployment/rhdh-deployment.md index 4e7f772..085dfd8 100644 --- a/docs/api/deployment/rhdh-deployment.md +++ b/docs/api/deployment/rhdh-deployment.md @@ -105,7 +105,12 @@ await rhdh.deploy({ timeout: null }); async waitUntilReady(timeout?: number): Promise ``` -Wait for RHDH deployment to be ready. +Wait for RHDH deployment to be ready. Performs two checks: + +1. **Pod readiness** — Waits for all pods to have `Ready=True`, with early failure detection for `CrashLoopBackOff`, `ImagePullBackOff`, etc. +2. **Route readiness** — HTTP health check against the RHDH route URL using Playwright's `request.newContext({ ignoreHTTPSErrors: true })`. This closes the gap between pod `Ready=True` and the OpenShift Router/HAProxy actually serving traffic. + +The remaining timeout after pod readiness is used for the route check (minimum 30 seconds). | Parameter | Type | Default | Description | |-----------|------|---------|-------------| @@ -133,7 +138,11 @@ await rhdh.rolloutRestart(); async scaleDownAndRestart(): Promise ``` -Scale down to 0, then back to 1 replica. +Scale down to 0, wait for pod termination, then scale back to 1 replica. Prevents `MigrationLocked` errors when two backstage instances try to run database migrations simultaneously. + +::: info +Called automatically during `deploy()` only on **helm upgrades** (when an existing deployment is detected). Skipped on fresh installs to avoid unnecessary pod cycling. +::: ```typescript await rhdh.scaleDownAndRestart(); diff --git a/docs/api/utils/plugin-metadata.md b/docs/api/utils/plugin-metadata.md index 2d2e084..59714b2 100644 --- a/docs/api/utils/plugin-metadata.md +++ b/docs/api/utils/plugin-metadata.md @@ -1,49 +1,13 @@ # Plugin Metadata -Utilities for loading and injecting plugin metadata from Package CRD files into dynamic plugins configuration. +Utilities for loading and processing plugin metadata from Package CRD files. These functions are used **internally** by `RHDHDeployment.deploy()` — most consumers don't need to call them directly. -## Import - -```typescript -import { - shouldInjectPluginMetadata, - extractPluginName, - getNormalizedPluginMergeKey, - getMetadataDirectory, - parseAllMetadataFiles, - injectMetadataConfig, - generateDynamicPluginsConfigFromMetadata, - loadAndInjectPluginMetadata, -} from "@red-hat-developer-hub/e2e-test-utils/utils"; -``` +::: info +Plugin metadata handling is fully automatic during `rhdh.deploy()`. The functions documented here are exported from the module for advanced use cases and testing, but are not part of the `./utils` public export path. They are imported internally by the deployment layer. +::: ## Functions -### shouldInjectPluginMetadata() - -Checks if plugin metadata handling should be enabled. - -```typescript -function shouldInjectPluginMetadata(): boolean -``` - -**Returns:** `true` if metadata handling is enabled, `false` otherwise. - -**Behavior:** -- Returns `false` if `RHDH_SKIP_PLUGIN_METADATA_INJECTION` environment variable is set -- Returns `false` if `JOB_NAME` contains `periodic-` (nightly/periodic builds) -- Returns `true` otherwise (default for local dev and PR builds) - -**Example:** - -```typescript -if (shouldInjectPluginMetadata()) { - // Load and inject metadata -} -``` - ---- - ### extractPluginName() Extracts the plugin name from a package path or OCI reference. @@ -110,6 +74,24 @@ getNormalizedPluginMergeKey({ --- +### isNightlyJob() + +Determines whether the current execution is a nightly/periodic job. Controls whether metadata config injection is enabled and which OCI resolution strategy is used. + +```typescript +function isNightlyJob(): boolean +``` + +**Returns:** `true` if running in nightly mode, `false` for PR/local mode. + +**Priority order:** +1. If `GIT_PR_NUMBER` is set → returns `false` (PR mode takes precedence) +2. If `E2E_NIGHTLY_MODE` is `"true"` or `"1"` → returns `true` +3. If `JOB_NAME` contains `periodic-` → returns `true` +4. Otherwise → returns `false` + +--- + ### getMetadataDirectory() Gets the metadata directory path. @@ -125,15 +107,6 @@ function getMetadataDirectory(metadataPath?: string): string | null **Returns:** The resolved metadata directory path, or `null` if it doesn't exist. -**Example:** - -```typescript -const metadataDir = getMetadataDirectory(); -if (metadataDir) { - console.log(`Found metadata at: ${metadataDir}`); -} -``` - --- ### parseAllMetadataFiles() @@ -153,47 +126,34 @@ async function parseAllMetadataFiles( **Returns:** Map of plugin name to [`PluginMetadata`](#pluginmetadata). -**Example:** - -```typescript -const metadataDir = getMetadataDirectory(); -if (metadataDir) { - const metadataMap = await parseAllMetadataFiles(metadataDir); - console.log(`Found ${metadataMap.size} plugins`); -} -``` - --- -### injectMetadataConfig() +### generatePluginsFromMetadata() -Injects plugin configurations from metadata into a dynamic plugins config. +Auto-generates plugin entries from workspace metadata files when no user-provided `dynamic-plugins.yaml` exists. Each plugin is enabled by default. ```typescript -function injectMetadataConfig( - dynamicPluginsConfig: DynamicPluginsConfig, - metadataMap: Map -): DynamicPluginsConfig +async function generatePluginsFromMetadata( + metadataPath?: string +): Promise ``` **Parameters:** -| Parameter | Type | Description | -|-----------|------|-------------| -| `dynamicPluginsConfig` | [`DynamicPluginsConfig`](#dynamicpluginsconfig) | The config to augment | -| `metadataMap` | `Map` | Map of plugin metadata | - -**Returns:** Augmented configuration with injected pluginConfigs. +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `metadataPath` | `string` | `"../metadata"` | Path to metadata directory | -**Merge Behavior:** Metadata config serves as the base, user-provided pluginConfig overrides it. +**Returns:** Dynamic plugins configuration with all plugins enabled (`disabled: false`). --- -### generateDynamicPluginsConfigFromMetadata() +### processPluginsForDeployment() -Generates a complete dynamic-plugins configuration from metadata files. +Unified entry point for both PR and nightly plugin resolution flows. Called automatically by `RHDHDeployment.deploy()`. ```typescript -async function generateDynamicPluginsConfigFromMetadata( +async function processPluginsForDeployment( + config: DynamicPluginsConfig, metadataPath?: string ): Promise ``` @@ -201,72 +161,32 @@ async function generateDynamicPluginsConfigFromMetadata( **Parameters:** | Parameter | Type | Default | Description | |-----------|------|---------|-------------| +| `config` | [`DynamicPluginsConfig`](#dynamicpluginsconfig) | - | The plugins config to process | | `metadataPath` | `string` | `"../metadata"` | Path to metadata directory | -**Returns:** Complete dynamic plugins configuration with all plugins enabled. +**Returns:** Processed configuration with resolved OCI references. **Behavior:** -- Returns `{ plugins: [] }` if [`shouldInjectPluginMetadata()`](#shouldinjectpluginmetadata) returns `false` -- Throws error if metadata directory not found -- Throws error if no valid metadata files found -- All generated plugins have `disabled: false` - -**PR Build Behavior (when `GIT_PR_NUMBER` is set):** -- Replaces local plugin paths with OCI URLs -- Fetches plugin versions from source repo's `package.json` -- Throws error if `source.json` or `plugins-list.yaml` not found -- Throws error if version fetching fails - -**Example:** - -```typescript -const config = await generateDynamicPluginsConfigFromMetadata(); -console.log(`Generated config with ${config.plugins?.length} plugins`); -``` - -**Example Output (PR Build):** - -```yaml -plugins: - - package: oci://ghcr.io/redhat-developer/rhdh-plugin-export-overlays/my-plugin:pr_1234__1.0.0 - disabled: false - pluginConfig: - # ... from metadata -``` +- **PR mode** (`!isNightlyJob()`): Injects `appConfigExamples` from metadata as base config, then resolves packages to OCI URLs (PR-specific if `GIT_PR_NUMBER` set, metadata refs otherwise) +- **Nightly mode** (`isNightlyJob()`): Resolves packages to OCI refs from metadata only (no config injection) +- Respects `RHDH_SKIP_PLUGIN_METADATA_INJECTION` to skip config injection --- -### loadAndInjectPluginMetadata() +### disablePluginWrappers() -Main function to load and inject plugin metadata for PR builds. +Creates a dynamic plugins config that disables wrapper plugins. Used during PR builds when wrapper plugins would conflict with PR-built OCI images. ```typescript -async function loadAndInjectPluginMetadata( - dynamicPluginsConfig: DynamicPluginsConfig, - metadataPath?: string -): Promise +function disablePluginWrappers(plugins: string[]): DynamicPluginsConfig ``` **Parameters:** -| Parameter | Type | Default | Description | -|-----------|------|---------|-------------| -| `dynamicPluginsConfig` | [`DynamicPluginsConfig`](#dynamicpluginsconfig) | - | The config to augment | -| `metadataPath` | `string` | `"../metadata"` | Path to metadata directory | - -**Returns:** Augmented configuration with metadata (for PR) or unchanged (for nightly). - -**Behavior:** -- Returns config unchanged if [`shouldInjectPluginMetadata()`](#shouldinjectpluginmetadata) returns `false` -- Throws error if metadata directory not found -- Throws error if no valid metadata files found -- Only injects metadata for plugins already in the config - -**Example:** +| Parameter | Type | Description | +|-----------|------|-------------| +| `plugins` | `string[]` | Plugin names to disable (e.g., `["backstage-community-plugin-tech-radar"]`) | -```typescript -const config = { plugins: [{ package: "./dynamic-plugins/dist/my-plugin", disabled: false }] }; -const augmented = await loadAndInjectPluginMetadata(config); -``` +**Returns:** Config with each plugin set to `disabled: true` using local wrapper paths. ## Types @@ -274,7 +194,7 @@ const augmented = await loadAndInjectPluginMetadata(config); ```typescript interface PluginMetadata { - /** The dynamic artifact path (e.g., ./dynamic-plugins/dist/plugin-name) */ + /** The dynamic artifact path (e.g., oci://ghcr.io/.../plugin-name:1.0.0) */ packagePath: string; /** The plugin configuration from appConfigExamples[0].content */ pluginConfig: Record; @@ -318,5 +238,6 @@ Default metadata directory path relative to the e2e-tests directory. ## See Also -- [Configuration Files](/guide/configuration/config-files#plugin-metadata-injection) - How metadata injection works during deployment +- [Plugin Metadata Guide](/guide/utilities/plugin-metadata) - How metadata handling works during deployment +- [Configuration Files](/guide/configuration/config-files#plugin-metadata-injection) - Configuration merge behavior - [Environment Variables](/guide/configuration/environment-variables#plugin-metadata-variables) - Variables that control metadata handling diff --git a/docs/changelog.md b/docs/changelog.md index a62bbd3..72b73f0 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -191,7 +191,7 @@ All notable changes to this project will be documented in this file. - **Plugin metadata auto-generation**: When `dynamic-plugins.yaml` doesn't exist, configuration is automatically generated from `metadata/*.yaml` files - **OCI URL generation for PR builds**: When `GIT_PR_NUMBER` is set, local plugin paths are replaced with OCI URLs (e.g., `oci://ghcr.io/redhat-developer/rhdh-plugin-export-overlays/my-plugin:pr_1234__1.0.0`) - Plugin metadata injection into existing `dynamic-plugins.yaml` configurations -- New utilities: `shouldInjectPluginMetadata()`, `generateDynamicPluginsConfigFromMetadata()`, `loadAndInjectPluginMetadata()`, `extractPluginName()` +- New utilities: `extractPluginName()`, `generatePluginsFromMetadata()`, `processPluginsForDeployment()`, `isNightlyJob()` - **Early pod failure detection**: `waitForPodsWithFailureDetection()` in KubernetesClientHelper detects CrashLoopBackOff, ImagePullBackOff, init container failures, etc. within seconds instead of waiting for full timeout ### Changed diff --git a/docs/guide/configuration/environment-variables.md b/docs/guide/configuration/environment-variables.md index 7c9b5b7..4cc304f 100644 --- a/docs/guide/configuration/environment-variables.md +++ b/docs/guide/configuration/environment-variables.md @@ -43,8 +43,10 @@ These control automatic plugin configuration injection from metadata files: | Variable | Description | Effect | |----------|-------------|--------| | `GIT_PR_NUMBER` | PR number (set by OpenShift CI) | Enables OCI URL generation for PR builds | -| `RHDH_SKIP_PLUGIN_METADATA_INJECTION` | When set (any value), disables metadata injection | Opt-out | +| `E2E_NIGHTLY_MODE` | When `"true"`, activates nightly mode | Uses released OCI refs, skips metadata injection | +| `RHDH_SKIP_PLUGIN_METADATA_INJECTION` | When `"true"`, disables metadata injection | Opt-out | | `JOB_NAME` | CI job name (set by OpenShift CI/Prow) | If contains `periodic-`, injection is disabled | +| `JOB_MODE` | CI-only: `nightly` or `pr-check` (set by step registry) | Informational | ### OCI URL Generation diff --git a/docs/guide/core-concepts/playwright-fixtures.md b/docs/guide/core-concepts/playwright-fixtures.md index 58c5685..accda7a 100644 --- a/docs/guide/core-concepts/playwright-fixtures.md +++ b/docs/guide/core-concepts/playwright-fixtures.md @@ -276,11 +276,24 @@ await test.runOnce("full-setup", async () => { ## Namespace Cleanup (Teardown) -In CI environments (`CI` environment variable is set), namespaces are automatically deleted after all tests complete. This is handled by a built-in **teardown reporter** that: +In CI environments (`CI=true`), namespaces are automatically deleted after tests complete. + +::: info Why a Reporter? +Neither `afterAll` hooks nor worker fixture cleanup can safely handle namespace deletion: + +- **`afterAll` hook**: Runs inside a worker process. When a test fails and Playwright restarts the worker for retries, the old worker's `afterAll` fires and deletes the namespace — before the retry worker can use it. +- **Worker fixture teardown**: Same problem — the fixture's teardown callback runs when the worker process exits, which happens on every worker restart (not just at the end of the suite). +- **`globalTeardown`**: Runs after all tests but has no visibility into which projects ran or which namespaces were created. + +A **reporter** solves all of these: it runs in the main Playwright process (survives worker restarts), tracks per-project test completion including retries, and only cleans up after a project's last test is truly done. +::: + +This is handled by a built-in **teardown reporter** that: 1. Runs in the main Playwright process (survives worker restarts) -2. Waits for **all tests** to finish -3. Deletes the namespace matching the project name +2. Tracks test completion **per project** — deletes each project's namespace as soon as all its tests finish, freeing cluster resources early instead of waiting for the entire suite +3. Handles retries correctly — only counts a test as done when it passes, is skipped, or exhausts all retries +4. Provides fallback cleanup in `onEnd` for interrupted runs (e.g., `maxFailures` reached) ### Default Behavior diff --git a/docs/guide/deployment/rhdh-deployment.md b/docs/guide/deployment/rhdh-deployment.md index 1c742b8..985152e 100644 --- a/docs/guide/deployment/rhdh-deployment.md +++ b/docs/guide/deployment/rhdh-deployment.md @@ -143,7 +143,10 @@ Any other value will throw an error during deployment. ### `waitUntilReady(timeout?)` -Wait for the RHDH deployment to be ready: +Wait for the RHDH deployment to be ready. Performs two-phase readiness check: + +1. **Pod readiness** — Waits for all pods to have `Ready=True` with early failure detection (CrashLoopBackOff, ImagePullBackOff, etc.) +2. **Route readiness** — HTTP health check against the RHDH route, closing the gap between pod readiness and the OpenShift Router actually serving traffic ```typescript // Default timeout: 500 seconds (~8 minutes) diff --git a/docs/guide/utilities/index.md b/docs/guide/utilities/index.md index 966102c..add044f 100644 --- a/docs/guide/utilities/index.md +++ b/docs/guide/utilities/index.md @@ -11,6 +11,7 @@ The package provides utility functions for common operations in E2E testing. | [YAML Merging](/guide/utilities/yaml-merging) | Merge YAML files | | [envsubst](/guide/utilities/environment-substitution) | Environment variable substitution | | [Plugin Metadata](/guide/utilities/plugin-metadata) | Plugin metadata injection | +| [WorkspacePaths](#workspacepaths) | Workspace config file path resolution | ## Importing Utilities @@ -18,6 +19,7 @@ The package provides utility functions for common operations in E2E testing. import { $, KubernetesClientHelper, + WorkspacePaths, envsubst, mergeYamlFiles, mergeYamlFilesToFile, @@ -76,3 +78,23 @@ const merged = mergeYamlFiles([ "override-config.yaml", ]); ``` + +### WorkspacePaths + +Static utility that resolves workspace config file paths relative to the `e2e-tests/` directory. Used internally by `RHDHDeployment` to locate configuration files. + +```typescript +import { WorkspacePaths } from "@red-hat-developer-hub/e2e-test-utils/utils"; + +WorkspacePaths.e2eRoot // /abs/path/workspaces/xyz/e2e-tests +WorkspacePaths.workspaceRoot // /abs/path/workspaces/xyz +WorkspacePaths.metadataDir // /abs/path/workspaces/xyz/metadata +WorkspacePaths.configDir // /abs/path/workspaces/xyz/e2e-tests/tests/config +WorkspacePaths.appConfig // .../tests/config/app-config-rhdh.yaml +WorkspacePaths.secrets // .../tests/config/rhdh-secrets.yaml +WorkspacePaths.dynamicPlugins // .../tests/config/dynamic-plugins.yaml +WorkspacePaths.valueFile // .../tests/config/value_file.yaml +WorkspacePaths.subscription // .../tests/config/subscription.yaml +``` + +Derives paths from Playwright's `test.info().project.testDir`, so it works correctly whether Playwright runs from the workspace root or the repo root. diff --git a/docs/guide/utilities/plugin-metadata.md b/docs/guide/utilities/plugin-metadata.md index 5f06f95..ea3d81e 100644 --- a/docs/guide/utilities/plugin-metadata.md +++ b/docs/guide/utilities/plugin-metadata.md @@ -28,54 +28,54 @@ Metadata handling is **enabled by default** for: - PR builds in CI Metadata handling is **disabled** when: -- `RHDH_SKIP_PLUGIN_METADATA_INJECTION` is set +- `RHDH_SKIP_PLUGIN_METADATA_INJECTION` is set to `true` +- `E2E_NIGHTLY_MODE` is set to `true` - `JOB_NAME` contains `periodic-` (nightly builds) -## Basic Usage +::: info Priority +The `isNightlyJob()` function checks in this order: +1. If `GIT_PR_NUMBER` is set → PR mode (returns `false`, metadata injection enabled) +2. If `E2E_NIGHTLY_MODE` is `"true"` or `"1"` → nightly mode (returns `true`) +3. If `JOB_NAME` contains `periodic-` → nightly mode (returns `true`) +4. Otherwise → PR/local mode (returns `false`) -### Check If Enabled +PR mode always takes precedence, preventing broken combinations of PR images with nightly config. +::: -```typescript -import { shouldInjectPluginMetadata } from "@red-hat-developer-hub/e2e-test-utils/utils"; +## Usage -if (shouldInjectPluginMetadata()) { - console.log("Metadata handling is enabled"); -} +Plugin metadata handling is **fully automatic** — it runs inside `rhdh.deploy()` with no additional code required. You don't need to import or call any metadata functions directly. + +```typescript +test.beforeAll(async ({ rhdh }) => { + await rhdh.configure({ auth: "keycloak" }); + await rhdh.deploy(); // Metadata is loaded, injected, and resolved automatically +}); ``` -### Auto-Generate Configuration +### What Happens During deploy() -When your `dynamic-plugins.yaml` doesn't exist, generate a complete config from all metadata files: +1. **Config assembly**: + - If `dynamic-plugins.yaml` **exists**: merged with package defaults + auth config + - If `dynamic-plugins.yaml` **doesn't exist**: auto-generated from all `metadata/*.yaml` files, then merged with defaults/auth (deduplicated by normalized plugin name — OCI wins over local `-dynamic` paths) -```typescript -import { generateDynamicPluginsConfigFromMetadata } from "@red-hat-developer-hub/e2e-test-utils/utils"; +2. **Metadata injection** (PR/local mode only, skipped in nightly): + - `appConfigExamples` from metadata merged as base config + - User-provided `pluginConfig` overrides metadata values -const config = await generateDynamicPluginsConfigFromMetadata(); -// All plugins from metadata/*.yaml are enabled by default -``` - -### Inject Into Existing Configuration +3. **Package resolution** (both modes) — per plugin, in priority order: + | Condition | Result | + |-----------|--------| + | Plugin in workspace build + `GIT_PR_NUMBER` set | PR OCI URL: `pr_{number}__{version}` | + | Plugin has metadata with OCI `dynamicArtifact` | Metadata's OCI ref (preserves original registry) | + | No metadata match (cross-workspace plugins, npm packages) | Kept as-is | -When you have a `dynamic-plugins.yaml`, inject metadata for listed plugins: +4. **Wrapper disabling** (PR builds only, when `GIT_PR_NUMBER` set): + - Appends `disabled: true` entries for wrapper plugins listed in `disableWrappers` -```typescript -import { loadAndInjectPluginMetadata } from "@red-hat-developer-hub/e2e-test-utils/utils"; - -const existingConfig = { - plugins: [ - { - package: "./dynamic-plugins/dist/my-plugin", - disabled: false, - pluginConfig: { - // Your overrides here - }, - }, - ], -}; - -const augmented = await loadAndInjectPluginMetadata(existingConfig); -// Metadata is merged as base, your pluginConfig overrides it -``` +::: info Multiple OCI Registries +Plugin OCI references use the **actual registry** from each plugin's `spec.dynamicArtifact` — not a single hardcoded registry. Plugins can come from `ghcr.io`, `quay.io/rhdh`, `registry.access.redhat.com/rhdh`, or other registries. +::: ## Extract Plugin Name @@ -167,13 +167,44 @@ For OCI URL generation, your workspace must have these files (generated by CI): For PR builds, OCI URL generation is strict - deployment will fail if required files are missing or version fetching fails. This ensures PR builds don't silently fall back to local paths. ::: +## Mode Comparison + +The system operates in three modes based on environment variables: + +| | **PR Check** | **Nightly** | **Local Dev** | +|---|---|---|---| +| **Trigger** | `GIT_PR_NUMBER` set | `E2E_NIGHTLY_MODE=true` | No env vars | +| **Config injection** | Yes — `appConfigExamples` merged | Skipped | Yes | +| **OCI resolution** | PR tags (`pr_{n}__{v}`) for workspace plugins, metadata refs for others | Metadata refs for all | Metadata refs for all | +| **Wrapper disabling** | Yes (`disableWrappers`) | No | No | +| **Cross-workspace plugins** | Kept as-is | Kept as-is | Kept as-is | + +### Why Metadata Refs (Not `{{inherit}}`) + +Metadata files are the most accurate source for latest published plugin versions. The daily `update-plugins-repo-refs` workflow keeps them current. By contrast, many OCI plugins (~49) are not in the catalog index (DPDY), and some that are have older versions. Using metadata ensures nightly tests run against the latest published artifacts. + +### processPluginsForDeployment + +This is the unified entry point for both PR and nightly plugin resolution flows. It is called automatically during `deploy()`. + +``` +Step 1: Inject metadata configs (PR/local mode only) + → deepMerge(metadata.appConfigExamples, user.pluginConfig) + → Skipped when: isNightlyJob() OR RHDH_SKIP_PLUGIN_METADATA_INJECTION="true" + +Step 2: Resolve packages to OCI (both modes) + → Per plugin: PR OCI URL > metadata OCI ref > passthrough +``` + ## Environment Variables | Variable | Effect | |----------|--------| | `GIT_PR_NUMBER` | Enables OCI URL generation for PR builds | +| `E2E_NIGHTLY_MODE` | When `true`, activates nightly mode (uses released OCI refs) | | `RHDH_SKIP_PLUGIN_METADATA_INJECTION` | Disables all metadata handling | | `JOB_NAME` | If contains `periodic-`, disables metadata handling | +| `JOB_MODE` | CI-only: `nightly` or `pr-check` (set by step registry) | See [Environment Variables](/guide/configuration/environment-variables#plugin-metadata-variables) for details. diff --git a/docs/overlay/reference/environment-variables.md b/docs/overlay/reference/environment-variables.md index 089d37d..aa3e640 100644 --- a/docs/overlay/reference/environment-variables.md +++ b/docs/overlay/reference/environment-variables.md @@ -50,6 +50,7 @@ Set `SKIP_KEYCLOAK_DEPLOYMENT=true` when using guest authentication and you don' | Variable | Description | Default | Required | |----------|-------------|---------|----------| | `CI` | Set automatically in CI environments | `false` | No | +| `JOB_MODE` | Set by CI step registry: `nightly` or `pr-check` | - | No | ## Plugin Metadata Variables @@ -58,7 +59,8 @@ These control automatic plugin configuration generation from metadata files: | Variable | Description | Effect | |----------|-------------|--------| | `GIT_PR_NUMBER` | PR number | Enables OCI URL generation using that PR's built images | -| `RHDH_SKIP_PLUGIN_METADATA_INJECTION` | When set (any value), disables metadata injection | Opt-out for all metadata handling | +| `E2E_NIGHTLY_MODE` | When `true`, activates nightly mode | Uses released OCI refs from metadata, skips config injection | +| `RHDH_SKIP_PLUGIN_METADATA_INJECTION` | When `true`, disables metadata injection | Opt-out for all metadata handling | | `JOB_NAME` | CI job name (set by OpenShift CI/Prow) | If contains `periodic-`, injection is disabled | ### When to Use These Variables @@ -67,7 +69,7 @@ These control automatic plugin configuration generation from metadata files: |----------|------------------| | PR builds in CI | `GIT_PR_NUMBER` is set automatically | | Test PR builds locally | Set `GIT_PR_NUMBER` manually to use PR's OCI images | -| Nightly/periodic builds | `JOB_NAME` contains `periodic-` (auto-detected) | +| Nightly/periodic builds | `E2E_NIGHTLY_MODE=true` or `JOB_NAME` contains `periodic-` (auto-detected in CI) | | Manual opt-out | Set `RHDH_SKIP_PLUGIN_METADATA_INJECTION=true` | ### Metadata Handling Behavior @@ -77,9 +79,14 @@ These control automatic plugin configuration generation from metadata files: - PR builds in CI **Disabled automatically** when: -- `RHDH_SKIP_PLUGIN_METADATA_INJECTION` is set (any value) +- `RHDH_SKIP_PLUGIN_METADATA_INJECTION` is set to `true` +- `E2E_NIGHTLY_MODE` is set to `true` - `JOB_NAME` contains `periodic-` (nightly builds) +::: info Priority +When `GIT_PR_NUMBER` is set, PR mode always takes precedence over nightly mode. This prevents broken combinations of PR images with nightly configuration. +::: + ### OCI URL Generation When `GIT_PR_NUMBER` is set (in CI or locally): @@ -110,6 +117,22 @@ Example transformation: See [Configuration Files - PR Builds](/overlay/test-structure/configuration-files#pr-builds-and-oci-images) for details. +## Test Runner Variables + +These are used by `run-e2e.sh` (the [unified test runner](/overlay/reference/run-e2e)): + +| Variable | Description | Default | +|----------|-------------|---------| +| `E2E_TEST_UTILS_PATH` | Absolute path to a local `e2e-test-utils` build | - | +| `E2E_TEST_UTILS_VERSION` | Pin `@red-hat-developer-hub/e2e-test-utils` npm version | `latest` (nightly), empty otherwise | +| `PLAYWRIGHT_VERSION` | Pin `@playwright/test` version | `1.57.0` | + +::: tip Version Pinning +`E2E_TEST_UTILS_PATH` takes precedence over `E2E_TEST_UTILS_VERSION`. If neither is set, the version in each workspace's `package.json` is used. + +In nightly mode (`E2E_NIGHTLY_MODE=true`), `E2E_TEST_UTILS_VERSION` defaults to `latest`. +::: + ## Setting Variables ### In .env File (Local Development) diff --git a/docs/overlay/reference/run-e2e.md b/docs/overlay/reference/run-e2e.md new file mode 100644 index 0000000..e68b8df --- /dev/null +++ b/docs/overlay/reference/run-e2e.md @@ -0,0 +1,248 @@ +# Unified Test Runner (run-e2e.sh) + +::: tip Overlay Documentation +This page covers writing tests within rhdh-plugin-export-overlays. +For using @red-hat-developer-hub/e2e-test-utils in external projects, see the [Guide](/guide/). +::: + +The `run-e2e.sh` script orchestrates E2E test execution across multiple workspaces. It handles workspace discovery, dependency installation, Playwright configuration generation, and parallel test execution. + +## Usage + +```bash +# Run all workspace tests +./run-e2e.sh + +# List discovered projects (dry run) +./run-e2e.sh --list + +# Run specific workspaces +./run-e2e.sh -w tech-radar +./run-e2e.sh -w backstage -w quickstart + +# Control parallelism +./run-e2e.sh --workers=4 + +# Run specific project +./run-e2e.sh --project=acr + +# Filter tests by name +./run-e2e.sh --grep="Quick" + +# Combine flags +./run-e2e.sh -w backstage --workers=2 + +# List projects in a workspace +./run-e2e.sh -w backstage --list + +# Use a local build of e2e-test-utils +E2E_TEST_UTILS_PATH=/path/to/rhdh-e2e-test-utils ./run-e2e.sh -w tech-radar + +# Pin a specific npm version of e2e-test-utils +E2E_TEST_UTILS_VERSION=1.1.24 ./run-e2e.sh -w tech-radar +``` + +## Workspace Selection + +Use the `-w` or `--workspace` flag to filter which workspaces to test. Without this flag, all workspaces with E2E tests are included. + +A workspace is discovered when it has a `workspaces//e2e-tests/` directory containing both `package.json` and `playwright.config.ts`. + +```bash +# Single workspace +./run-e2e.sh -w tech-radar + +# Multiple workspaces +./run-e2e.sh -w tech-radar -w keycloak -w github-actions +``` + +## Environment Variables + +### RHDH Deployment + +| Variable | Description | Default | +|----------|-------------|---------| +| `RHDH_VERSION` | RHDH version to deploy (e.g., `1.10`, `next`) | `1.10` | +| `INSTALLATION_METHOD` | Deployment method: `helm` or `operator` | `helm` | +| `SKIP_KEYCLOAK_DEPLOYMENT` | Set `true` to skip Keycloak deployment | - | +| `CATALOG_INDEX_IMAGE` | Override the default catalog index image baked into the RHDH chart | - | + +### Test Framework + +| Variable | Description | Default | +|----------|-------------|---------| +| `CI` | Enables CI mode (forbidOnly, namespace teardown) | `true` | +| `PLAYWRIGHT_VERSION` | Pin `@playwright/test` version | `1.57.0` | +| `E2E_TEST_UTILS_PATH` | Absolute path to a local `e2e-test-utils` build (for testing unpublished changes) | - | +| `E2E_TEST_UTILS_VERSION` | Pin `@red-hat-developer-hub/e2e-test-utils` npm version | `latest` (nightly), empty otherwise | + +### Plugin Resolution + +| Variable | Description | Default | +|----------|-------------|---------| +| `E2E_NIGHTLY_MODE` | When `true`, uses released OCI images from metadata; defaults `E2E_TEST_UTILS_VERSION` to `latest` | `false` | +| `GIT_PR_NUMBER` | PR number for OCI URL generation (uses PR-built images) | - | +| `JOB_NAME` | CI job name; if contains `periodic-`, disables metadata injection | - | + +## How It Works + +The script performs these steps in order: + +### 1. Validate Prerequisites + +Checks for `node`, `yarn`, `jq`, and verifies cluster login (`oc whoami`). Cluster login is skipped for `--list` mode. + +### 2. Discover Workspaces + +Scans `workspaces/*/e2e-tests/` for directories containing both `package.json` and `playwright.config.ts`. Applies `-w` filter if provided. + +### 3. Generate Root `package.json` + +Creates a root `package.json` with: +- **Yarn workspaces** pointing to selected `workspaces/*/e2e-tests` directories +- **Resolutions** to pin `@playwright/test` and optionally `@red-hat-developer-hub/e2e-test-utils` + +```json +{ + "workspaces": ["workspaces/tech-radar/e2e-tests", "workspaces/keycloak/e2e-tests"], + "resolutions": { + "@playwright/test": "1.57.0", + "@red-hat-developer-hub/e2e-test-utils": "1.1.24" + } +} +``` + +### 4. Install Dependencies + +Cleans all `node_modules` and `yarn.lock` to ensure fresh resolution, then runs `yarn install`. + +### 5. Generate Root `playwright.config.ts` + +Extracts `projects: [...]` blocks from each workspace's `playwright.config.ts` using `sed` text processing. Injects `testDir` into each project pointing to the workspace's `tests/` directory. + +::: info Why sed Instead of Import? +Importing workspace configs would execute their top-level code (e.g., `dotenv.config()`, `process.env` mutations), which can pollute the environment for other workspaces. Text extraction avoids this. +::: + +The generated config imports `baseConfig` from the test utils package, which provides reporters, timeouts, video/screenshot/trace settings, and global setup. + +### 6. Run Tests + +Runs `npx playwright test` with any additional arguments passed through. All arguments not recognized as `-w`/`--workspace` are forwarded directly to Playwright. + +### 7. Display Summary + +Parses `playwright-report/results.json` and displays: +- Duration, passed/failed/flaky/skipped counts +- Overall status (PASSED/FAILED) +- Report file location + +## Version Pinning + +There are two ways to control which version of `@red-hat-developer-hub/e2e-test-utils` is used: + +### Local Build (Development) + +Use `E2E_TEST_UTILS_PATH` to point to a local checkout. The script builds it before installing: + +```bash +E2E_TEST_UTILS_PATH=/home/user/rhdh-e2e-test-utils ./run-e2e.sh -w tech-radar +``` + +### Specific npm Version + +Use `E2E_TEST_UTILS_VERSION` to pin a published version: + +```bash +E2E_TEST_UTILS_VERSION=1.1.24 ./run-e2e.sh -w tech-radar +``` + +::: info Nightly Default +When `E2E_NIGHTLY_MODE=true`, `E2E_TEST_UTILS_VERSION` defaults to `latest` to pick up the most recent published version. This ensures nightly runs always use the latest test utilities. +::: + +### Priority + +`E2E_TEST_UTILS_PATH` takes precedence over `E2E_TEST_UTILS_VERSION`. If neither is set, the version declared in each workspace's `package.json` is used (after resolution). + +## List Mode + +Use `--list` to preview discovered test projects without running them: + +```bash +./run-e2e.sh --list +./run-e2e.sh -w backstage --list +``` + +This generates a lightweight config that skips `globalSetup` and `teardown` reporters, so no cluster connection is needed. + +## Playwright Arguments + +All arguments not recognized as `-w`/`--workspace` are forwarded directly to Playwright: + +```bash +# Control workers +./run-e2e.sh --workers=1 + +# Run specific project +./run-e2e.sh --project=tech-radar + +# Filter by test name +./run-e2e.sh --grep="catalog" + +# Combine multiple flags +./run-e2e.sh -w backstage --workers=2 --retries=1 --project=backstage-kubernetes +``` + +## Generated Files + +The script generates these temporary files in the repository root: + +| File | Purpose | +|------|---------| +| `package.json` | Root workspace config with resolutions | +| `.yarnrc.yml` | Yarn node-modules linker config | +| `playwright.config.ts` | Combined Playwright config with all workspace projects | +| `playwright.list.config.ts` | Lightweight config for `--list` mode (when used) | + +These files are generated fresh on each run. + +## Path Resolution + +Running tests from the repo root changes the working directory relative to each workspace. The package handles this transparently: + +- **WorkspacePaths** resolves all config file paths (app-config, secrets, dynamic-plugins, metadata) from Playwright's `test.info().project.testDir` — an absolute path — instead of `process.cwd()`. This works correctly regardless of where the process was launched. +- **Worker fixture** sets `process.chdir()` to the workspace's `e2e-tests/` directory when the test worker starts, so relative paths in shell scripts and `fs` calls also resolve correctly. + +No changes to test code are needed. The same spec files work both with `yarn test` from the workspace and `./run-e2e.sh` from the repo root. See [Path Resolution](/overlay/test-structure/directory-layout#path-resolution-workspacepaths) for details. + +## Design Decisions + +### Why Single Root Playwright (Not Per-Workspace Parallel) + +Two strategies were evaluated for running all workspace tests in CI: + +| | Single root Playwright | Per-workspace shell parallel | +|---|---|---| +| **Parallelism** | Worker-level — Playwright auto-balances across all projects | Workspace-level — a large workspace bottlenecks while small ones sit idle | +| **Keycloak** | `globalSetup` runs once, no races | Multiple processes deploy simultaneously, causing races | +| **Reporting** | Single report with traces/screenshots/videos | Blob merge step needed, adds a failure point | +| **Dependency validation** | Yarn resolutions validates upgrades across all workspaces in one run | No way to test a dependency upgrade across all workspaces at once | +| **CLI** | Standard Playwright flags work (`--project`, `--grep`, `--shard`) | Flags must be forwarded per-process | + +The single root approach requires [WorkspacePaths](#path-resolution) to resolve config paths correctly, but this change is backward-compatible and benefits all execution modes. + +### Why Yarn Workspaces Is Required + +Playwright errors out if `@playwright/test` is loaded from more than one file path in a process. With separate `node_modules` per workspace, each workspace resolves the package from a different path. Yarn workspaces hoists all dependencies to a single root `node_modules`, ensuring one copy. This is the primary reason yarn workspaces is used — not just for convenience. + +::: info No Impact on Existing Workflows +Nothing is committed to the repo. `package.json`, `playwright.config.ts`, and `.yarnrc.yml` are generated at runtime. Workspace `yarn.lock` files are bypassed (fresh resolution at root), which is fine for nightly — PR checks still use `--immutable` per workspace. Fresh resolution in nightly actually catches dependency regressions early. +::: + +## Related Pages + +- [Running Tests Locally](/overlay/tutorials/running-locally) - Individual workspace testing +- [CI/CD Pipeline](/overlay/tutorials/ci-pipeline) - How run-e2e.sh is used in OpenShift CI +- [Environment Variables](/overlay/reference/environment-variables) - All supported variables +- [Package.json Scripts](/overlay/reference/scripts) - Workspace-level test scripts diff --git a/docs/overlay/test-structure/configuration-files.md b/docs/overlay/test-structure/configuration-files.md index 3dfe583..752b46f 100644 --- a/docs/overlay/test-structure/configuration-files.md +++ b/docs/overlay/test-structure/configuration-files.md @@ -351,6 +351,25 @@ In CI, these are generated automatically. For local testing with `GIT_PR_NUMBER` OCI URL generation is strict - deployment will fail if required files are missing or version fetching fails. This ensures builds don't silently fall back to local paths. ::: +## Nightly Builds and OCI Resolution + +In nightly mode (`E2E_NIGHTLY_MODE=true`), plugin packages are resolved to **released** OCI refs from workspace metadata (`spec.dynamicArtifact`) instead of PR-built images. Metadata config injection is skipped — only package resolution happens. + +### How Each Plugin Type Is Resolved + +| Plugin Type | Example | Resolution | +|-------------|---------|------------| +| Workspace plugin with metadata | `./dynamic-plugins/dist/plugin-tech-radar` | Metadata OCI ref (e.g., `oci://ghcr.io/.../plugin-tech-radar:bs_1.45.3__1.13.0`) | +| Cross-workspace plugin (no metadata) | `./dynamic-plugins/dist/backstage-plugin-kubernetes-backend-dynamic` | Kept as-is | +| npm package | `@red-hat-developer-hub/backstage-plugin-global-header-test@0.0.2` | Kept as-is (with integrity hash preserved) | +| Existing OCI reference | `oci://quay.io/.../plugin-name:tag` | Kept as-is | + +::: info Multiple Registries +Plugin OCI refs use the **actual registry** from each plugin's metadata — plugins may come from `ghcr.io`, `quay.io/rhdh`, `registry.access.redhat.com/rhdh`, or other registries. The system does not assume a single registry. +::: + +See [Plugin Metadata - Mode Comparison](/guide/utilities/plugin-metadata#mode-comparison) for the full comparison of PR check, nightly, and local dev modes. + ## Configuration Merging `@red-hat-developer-hub/e2e-test-utils` merges your configuration with defaults in this order: @@ -460,8 +479,124 @@ app: title: RHDH Test Instance ``` +## Real-World Workspace Patterns + +These examples show how different workspaces use `dynamic-plugins.yaml` and how the package resolves their plugins. + +### Auto-Generated (No dynamic-plugins.yaml) + +**Workspaces:** tech-radar, quickstart, acr + +When no `dynamic-plugins.yaml` exists, the package auto-generates entries from all `metadata/*.yaml` files: + +```yaml +# Auto-generated at deploy time: +plugins: + - package: oci://ghcr.io/.../backstage-community-plugin-tech-radar:bs_1.45.3__1.13.0 + disabled: false + # pluginConfig injected from metadata appConfigExamples (PR/local mode) +``` + +No configuration files needed — metadata provides everything. + +### Cross-Workspace Plugins + +**Workspace:** argocd + +When your workspace needs plugins from another workspace (e.g., Kubernetes backend for ArgoCD): + +```yaml +plugins: + # Workspace plugin — resolved to metadata OCI ref (or PR tag) + - package: oci://ghcr.io/.../backstage-community-plugin-argocd:bs_1.45.3__2.4.3!backstage-community-plugin-argocd + pluginConfig: + dynamicPlugins: + frontend: { ... } + + # Cross-workspace plugin — no metadata match, kept as-is + - package: ./dynamic-plugins/dist/backstage-plugin-kubernetes-backend-dynamic +``` + +Cross-workspace plugins have no metadata in the current workspace, so they pass through unchanged in all modes. + +### OCI Aliases + +**Workspace:** redhat-resource-optimization + +Some plugins share a single OCI image with multiple plugins distinguished by aliases (the `!alias` suffix): + +```yaml +plugins: + - package: oci://quay.io/redhat-resource-optimization/dynamic-plugins:1.3.2!red-hat-developer-hub-plugin-redhat-resource-optimization +``` + +The alias after `!` tells RHDH which plugin to extract from the shared image. + +### Disabled Wrapper Plugins + +**Workspace:** scorecard + +When your workspace uses an OCI image for a plugin that also has a local wrapper enabled by default: + +```yaml +plugins: + # Workspace plugin — resolved to metadata ref + - package: oci://ghcr.io/.../red-hat-developer-hub-backstage-plugin-scorecard:tag!alias + + # Cross-workspace OCI — kept as-is + - package: oci://ghcr.io/.../red-hat-developer-hub-backstage-plugin-dynamic-home-page:tag!alias + + # Disable the local wrapper to avoid conflicts + - package: ./dynamic-plugins/dist/red-hat-developer-hub-backstage-plugin-dynamic-home-page + disabled: true +``` + +### Different OCI Registries + +Plugins can come from different registries. The package preserves the original registry from each plugin's metadata: + +```yaml +# ghcr.io (community plugins) +- package: oci://ghcr.io/redhat-developer/rhdh-plugin-export-overlays/backstage-community-plugin-tech-radar:bs_1.45.3__1.13.0 + +# quay.io (Red Hat plugins) +- package: oci://quay.io/rhdh/red-hat-developer-hub-backstage-plugin-scaffolder-relation-processor@sha256:abc123 + +# registry.access.redhat.com (certified plugins) +- package: oci://registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator@sha256:f40d39fb +``` + +### npm Packages + +**Workspace:** global-header + +For plugins published to npm instead of OCI: + +```yaml +plugins: + - package: "@red-hat-developer-hub/backstage-plugin-global-header-test@0.0.2" + integrity: "sha512-ABC123..." +``` + +npm packages with integrity hashes pass through unchanged in all modes — no metadata resolution. + +### All Local Paths + +**Workspace:** topology + +Some workspaces use only local paths (no OCI references in their config): + +```yaml +plugins: + - package: ./dynamic-plugins/dist/backstage-community-plugin-topology + - package: ./dynamic-plugins/dist/backstage-plugin-kubernetes-backend-dynamic +``` + +In PR mode, `pluginConfig` from metadata is injected. In nightly mode, local paths are resolved to metadata OCI refs if metadata exists. Local paths with no metadata match stay unchanged. + ## Related Pages - [Directory Layout](./directory-layout) - Where config files go - [Spec Files](./spec-files) - Using config in tests - [Environment Variables](/overlay/reference/environment-variables) - All supported variables +- [Plugin Metadata](/guide/utilities/plugin-metadata) - How metadata resolution works diff --git a/docs/overlay/test-structure/directory-layout.md b/docs/overlay/test-structure/directory-layout.md index ecd2e6c..a1afc5c 100644 --- a/docs/overlay/test-structure/directory-layout.md +++ b/docs/overlay/test-structure/directory-layout.md @@ -206,8 +206,41 @@ deploy_external_service "$1" | Deployment scripts | `deploy-.sh` | `deploy-customization-provider.sh` | | Config files | Standard names | `app-config-rhdh.yaml` | +## Path Resolution (WorkspacePaths) + +Tests in this repo can run from two different working directories: + +| Context | CWD | How | +|---------|-----|-----| +| Individual workspace | `workspaces//e2e-tests/` | `cd workspaces/tech-radar/e2e-tests && yarn test` | +| Repo root (unified runner) | Repository root | `./run-e2e.sh -w tech-radar` | + +Configuration files live under `tests/config/` relative to each workspace's `e2e-tests/` directory. If paths were resolved from `process.cwd()`, they would break when running from the repo root. + +### How It Works + +The `WorkspacePaths` utility resolves all config file paths from Playwright's `test.info().project.testDir` — an **absolute path** set by Playwright itself — instead of relying on CWD. This makes path resolution correct regardless of where the process was launched. + +``` +test.info().project.testDir + → /abs/path/workspaces/tech-radar/e2e-tests/tests + → e2eRoot: /abs/path/workspaces/tech-radar/e2e-tests + → configDir: /abs/path/workspaces/tech-radar/e2e-tests/tests/config + → appConfig: /abs/path/workspaces/tech-radar/e2e-tests/tests/config/app-config-rhdh.yaml + → metadataDir: /abs/path/workspaces/tech-radar/metadata +``` + +### Complementary CWD Fix + +The worker fixture also sets `process.chdir(e2eRoot)` and `$.cwd = e2eRoot` when the test worker starts. This ensures that any code using relative paths (e.g., shell scripts, `fs.readFileSync`) also resolves correctly. WorkspacePaths handles the framework-level paths; the CWD change covers everything else. + +::: info No Action Required +WorkspacePaths is used internally by `RHDHDeployment`. You don't need to use it directly unless you're resolving custom file paths in your test setup. See [WorkspacePaths API](/guide/utilities/#workspacepaths) for available properties. +::: + ## Related Pages - [Configuration Files](./configuration-files) - Detailed YAML configuration - [Spec Files](./spec-files) - Writing test specifications - [Pre-requisite Services](/overlay/tutorials/custom-deployment) - Deploy dependencies before RHDH +- [Unified Test Runner](/overlay/reference/run-e2e) - Running tests from repo root diff --git a/docs/overlay/tutorials/ci-pipeline.md b/docs/overlay/tutorials/ci-pipeline.md index af188ba..29fd474 100644 --- a/docs/overlay/tutorials/ci-pipeline.md +++ b/docs/overlay/tutorials/ci-pipeline.md @@ -42,6 +42,57 @@ The `e2e-ocp-helm` job runs Helm-based E2E tests on an OpenShift cluster. artifacts/e2e-ocp-helm/redhat-developer-rhdh-plugin-export-overlays-ocp-helm/artifacts/playwright-report/index.html ``` +## e2e-ocp-helm-nightly Job + +The `e2e-ocp-helm-nightly` job runs E2E tests against **released** OCI images across all workspaces, verifying that published plugins work correctly with the current RHDH version. + +### When It Runs + +- **Daily cron**: 4 AM UTC +- **Manually** via PR comment: `/test e2e-ocp-helm-nightly` +- **Rehearse** via release repo PRs + +### How It Differs from PR Check + +| Aspect | PR Check (`e2e-ocp-helm`) | Nightly (`e2e-ocp-helm-nightly`) | +|--------|--------------------------|----------------------------------| +| **JOB_MODE** | `pr-check` | `nightly` | +| **GIT_PR_NUMBER** | Set (PR number) | Not exported | +| **OCI Images** | PR-built (`pr_` tags) for workspace plugins | Released (from metadata `spec.dynamicArtifact`) | +| **Metadata injection** | Yes — `appConfigExamples` merged | Skipped | +| **Wrapper disabling** | Yes (`disableWrappers`) | No | +| **Workspaces Tested** | Changed workspace only | All workspaces | +| **E2E_NIGHTLY_MODE** | Not set | `true` | + +::: info Why Released OCI Images? +Nightly tests use OCI refs from workspace metadata (`spec.dynamicArtifact`) because metadata files are the most accurate source for latest published plugin versions — updated daily by the `update-plugins-repo-refs` workflow. Many plugins are not in the catalog index, and those that are may have older versions. +::: + +### JOB_MODE + +The CI step registry determines `JOB_MODE` early and uses it to drive all downstream behavior: + +- **`nightly`**: Periodic cron jobs, or presubmit jobs with `nightly` in the name +- **`pr-check`**: All other presubmit jobs + +`GIT_PR_NUMBER` is only parsed and exported in `pr-check` mode, preventing nightly jobs from accidentally resolving PR-specific OCI images. + +### Where to Find the Playwright Report + +Same as the pr-check job: + +``` +artifacts/e2e-ocp-helm-nightly/redhat-developer-rhdh-plugin-export-overlays-ocp-helm/artifacts/playwright-report/index.html +``` + +### Rehearsing the Nightly Job + +To test nightly job changes via the release repo: + +1. Open a PR in the release repo with your step registry changes +2. Comment `/test e2e-ocp-helm-nightly` on the PR +3. The rehearse job runs with `JOB_MODE=nightly`, testing all workspaces + ## PR OCI Image Builds When testing a PR, the plugins need to be built into OCI images that RHDH can use. This is handled through the `/publish` command. @@ -118,8 +169,11 @@ The following environment variables are available during CI execution: | `RHDH_VERSION` | RHDH version to deploy | | `INSTALLATION_METHOD` | `helm` or `operator` | | `CI` | Set to `true` in CI environment | -| `GIT_PR_NUMBER` | PR number (enables OCI URL generation) | +| `GIT_PR_NUMBER` | PR number (enables OCI URL generation) — pr-check only | | `JOB_NAME` | CI job name (if contains `periodic-`, disables metadata) | +| `JOB_MODE` | `nightly` or `pr-check` — set by step registry | +| `E2E_NIGHTLY_MODE` | `true` for nightly jobs | +| `E2E_TEST_UTILS_VERSION` | Pinned e2e-test-utils version (nightly only) | | `VAULT_*` | All Vault secrets with this prefix | ### Plugin Metadata Variables diff --git a/docs/overlay/tutorials/running-locally.md b/docs/overlay/tutorials/running-locally.md index 5470689..9296e0c 100644 --- a/docs/overlay/tutorials/running-locally.md +++ b/docs/overlay/tutorials/running-locally.md @@ -39,6 +39,21 @@ npx playwright install ## Running Tests +::: tip Multi-Workspace Testing +To run tests across multiple workspaces simultaneously, use the [`run-e2e.sh`](/overlay/reference/run-e2e) unified test runner instead of individual workspace commands: + +```bash +# Run all workspaces +./run-e2e.sh + +# Run specific workspaces +./run-e2e.sh -w tech-radar -w keycloak + +# Use local e2e-test-utils build +E2E_TEST_UTILS_PATH=/path/to/rhdh-e2e-test-utils ./run-e2e.sh -w tech-radar +``` +::: + ### Run All Tests ```bash