feat(poc): replace Docker-based smoke tests with native Node.js harness by rostalan · Pull Request #2231 · redhat-developer/rhdh-plugin-export-overlays

rostalan · 2026-04-09T12:53:03Z

Boot a minimal Backstage backend directly on the runner using createBackend() + dynamicPluginsFeatureLoader, probe /api/<pluginId> routes, and report results as structured JSON. Includes core bundled plugins (catalog, auth, permission, scaffolder, events, search, proxy) so dynamic plugins resolve their dependencies correctly.

Frontend plugins are also validated via two layers:

Static bundle checks — verifies dist-scalprum/ exists and contains JavaScript files after OCI download
Loaded-plugin probe — an inline backend plugin queries dynamicPluginsServiceRef to confirm frontend plugins were registered without errors

Results are merged into a single report with per-plugin status (pass, fail-bundle, fail-load, warn, skip) and the process exits non-zero on any failure.

Made-with: Cursor

openshift-ci · 2026-04-09T12:53:08Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

rostalan · 2026-04-09T12:54:43Z

/publish

github-actions · 2026-04-09T12:55:11Z

PR action (/publish) cancelled: PR doesn't touch only 1 workspace.

github-actions · 2026-04-09T12:55:36Z

⚠️
Smoke test workflow skipped: PR doesn't touch exactly one workspace.

github-actions · 2026-04-09T19:03:53Z

The file versions.json could not be synced from branch main into this because your PR is from a fork.

You should update the versions.json file with the following content:

{
    "backstage": "1.48.3",
    "node": "22.22.0",
    "cli": "1.10.4",
    "cliPackage": "@red-hat-developer-hub/cli"
}

gustavolira · 2026-04-10T16:03:49Z

+}
+
+// ---------------------------------------------------------------------------
+// OCI Download  (mirrors install-dynamic-plugins.py §663-715)


The comment itself acknowledges this mirrors install-dynamic-plugins.py. Should we just call the Python script directly as a pre-step instead of reimplementing it in JS?

The overlay repo currently gets the script from the RHDH container image — by removing Docker, we lose access to it. But we could copy the script here (with a version marker) as a short-term solution. That would:

Eliminate ~60 lines of JS OCI code

Inherit all security protections for free (zip bomb, symlink traversal, integrity checks)

Leverage the 130KB+ test suite that already covers this logic

Longer-term options: publish as a pip package or container tool so any repo can consume it.

For context, there's also a parallel POC in RHDH core (redhat-developer/rhdh#4523) that extends the same Python script with parallel downloads (ThreadPoolExecutor) achieving ~34% speedup. Reusing it here would get that performance benefit too.

for now, I added a step that downloads install-dynamic-plugins script and its requirements and uses it directly. Not sure if the would benefit from the parallel downloads, since it only runs on a single workspace, so only a couple OCIs at a time.

gustavolira · 2026-04-10T16:03:49Z

+
+function extractPlugin(tarFile, pluginPath, dest) {
+  mkdirSync(join(dest, pluginPath), { recursive: true });
+  execSync(`tar xf "${tarFile}" -C "${dest}" "${pluginPath}/"`, {


This extracts tar contents with a bare execSync('tar xf ...') — no validation of archive members before extraction. The Python script in RHDH core (install-dynamic-plugins.py) checks for:

Zip bomb: member.size > MAX_ENTRY_SIZE (20MB default per entry)

Symlink traversal: os.path.realpath() validation against the destination directory

Hardlink traversal: same check

Device files/FIFOs: rejected entirely

Safe tar filter: tar.extractall(..., filter='tar')

Even in CI, a crafted OCI layer could exploit path traversal via symlinks. If we keep the JS implementation, all of these checks need to be ported.

replaced by python script.

gustavolira · 2026-04-10T16:03:49Z

+    if (existsSync(p)) process.argv.push("--config", p);
+  }
+
+  const { createBackend } = await import("@backstage/backend-defaults");


createBackend() is the production API. Backstage provides startTestBackend() from @backstage/backend-test-utils specifically for this use case — it handles lifecycle management, automatic cleanup, built-in in-memory SQLite, and exposes mock services. It would also reduce the manual plugin registration below (~20 lines of backend.add() calls).

redhat-developer/rhdh#4523 uses startTestBackend() for the same plugin loadability validation and it works well.

This was meant to keep the environment somewhat real, but I guess it makes sense to have this as a simple "plugin loads" check rather than anything more complicated.

gustavolira · 2026-04-10T16:03:49Z

+// Config generation
+// ---------------------------------------------------------------------------
+
+function deepMerge(src, dst) {


This reimplements config deep-merge that already exists in install-dynamic-plugins.py (the merge() function). The Python version also detects key conflicts and raises errors — this one silently overwrites. If we use the Python script as a pre-step, this function becomes unnecessary since the script already generates app-config.dynamic-plugins.yaml with all plugin configs merged.

replaced with python script

gustavolira · 2026-04-10T16:03:49Z

+  );
+
+  backend.add(
+    createBackendPlugin({


This probe plugin concept is great — using dynamicPluginsServiceRef to verify frontend plugins actually loaded at runtime is more thorough than filesystem-only checks. Worth keeping regardless of the OCI download approach. The sanity check POC in RHDH core (redhat-developer/rhdh#4523) currently only validates bundles via filesystem (dist-scalprum/plugin-manifest.json); this runtime probe approach would complement it nicely.

gustavolira · 2026-04-10T16:03:49Z

+  });
+}
+
+async function downloadPlugins(plugins, dest) {


Downloads are sequential here. The install-dynamic-plugins-fast.py in redhat-developer/rhdh#4523 uses ThreadPoolExecutor for parallel OCI downloads with a shared image cache (one download per unique image, not per plugin), achieving ~34% speedup. Another reason to reuse the existing script rather than maintaining a separate implementation.

replaced with the basic python script, same reason as above.

github-actions · 2026-04-14T10:57:53Z

The file versions.json could not be synced from branch main into this because your PR is from a fork.

You should update the versions.json file with the following content:

{
    "backstage": "1.49.4",
    "node": "22.22.0",
    "cli": "1.10.4",
    "cliPackage": "@red-hat-developer-hub/cli"
}

rostalan · 2026-04-15T07:49:36Z

/publish

github-actions · 2026-04-15T07:49:58Z

PR action (/publish) cancelled: PR doesn't touch only 1 workspace.

rostalan · 2026-04-15T07:59:11Z

/publish

github-actions · 2026-04-15T07:59:32Z

PR action (/publish) cancelled: PR doesn't touch only 1 workspace.

rostalan · 2026-04-15T08:15:24Z

/publish

github-actions · 2026-04-15T08:15:48Z

PR action (/publish) cancelled: PR doesn't touch only 1 workspace.

jonkoops · 2026-04-15T11:10:35Z

-      - name: Log in to GitHub Container Registry
-        uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
+      - name: Setup Node.js
+        uses: actions/setup-node@v4


This should be pinned to a hash and match the other workflows:

Suggested change

uses: actions/setup-node@v4

uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0

jonkoops · 2026-04-15T11:11:11Z

-          registry: ghcr.io
-          username: ${{ github.actor }}
-          password: ${{ secrets.GITHUB_TOKEN }}
+          node-version: "20"


There are two LTS releases after this one that are currently active, let's not target older Node.js versions if we can avoid it.

Suggested change

node-version: "20"

node-version: "24"

jonkoops · 2026-04-15T11:11:44Z

-          echo "RHDH did not become ready in time."
-          exit 1
+          curl -fsSL \
+            "https://raw.githubusercontent.com/redhat-developer/rhdh/3efb9cc140ff/scripts/install-dynamic-plugins/install-dynamic-plugins.py" \


Is this supposed to be pinned to a specific commit?

this will get replaced with the ts version after redhat-developer/rhdh#4574 gets merged anyway...

jonkoops · 2026-04-15T11:13:27Z

+  "type": "module",
+  "description": "Lightweight smoke test harness for RHDH dynamic plugins — no Docker required",
+  "scripts": {
+    "test": "node smoke-test.mjs",


Looks like this script is not triggered in the CI, but rather executed manually, let's use npm test for consistency.

jonkoops · 2026-04-15T11:14:26Z

+    "@backstage/backend-dynamic-feature-service": "0.7.9",
+    "@backstage/backend-plugin-api": "1.7.0",
+    "@backstage/backend-test-utils": "1.4.0",
+    "@backstage/cli-node": "0.2.18",
+    "@backstage/config": "1.3.6",
+    "@backstage/config-loader": "1.10.8",
+    "@backstage/errors": "1.2.7",
+    "@backstage/types": "1.2.2",
+    "js-yaml": "4.1.0"


Let's not rely on pinned versions here, but use caret ranges ^ and a lockfile.

jonkoops · 2026-04-15T11:15:24Z

There is no need to use .mjs, you can use .js and Node will be able to detect it, also because the type is already set in the package.

jonkoops · 2026-04-15T11:16:38Z

Node.js can run TypeScript these days, so using .ts would also work, and make your code type-safe.

jonkoops · 2026-04-15T11:17:14Z

+// CLI
+// ---------------------------------------------------------------------------
+
+function parseArgs(argv) {


Node.js has built-in support for parsing CLI arguments, there is no need to implement this yourself.

jonkoops · 2026-04-15T11:18:28Z

+  return args;
+}
+
+function loadEnvFile(filePath) {


Node.js has built-in support for parsing .env files, so there is no need to implement this yourself.

jonkoops · 2026-04-15T12:37:35Z

+// Config helpers
+// ---------------------------------------------------------------------------
+
+function deepMerge(src, dst) {


The custom deepMerge and loadConfigs can be removed entirely — Backstage's config loader already handles deep merging of multiple --config files. Just push the config paths onto process.argv before calling startTestBackend() and let the default rootConfig service pick them up. This is the same approach the old Docker-based smoke test used (chaining --config flags on the node packages/backend command).

This also removes the need for mockServices.rootConfig.factory({ data }).

Boot a minimal Backstage backend directly on the runner using createBackend() + dynamicPluginsFeatureLoader, probe /api/<pluginId> routes, and report results as structured JSON. Includes core bundled plugins (catalog, auth, permission, scaffolder, events, search, proxy) so dynamic plugins resolve their dependencies correctly. Made-with: Cursor

…aded-plugin probe Add two-layer frontend plugin validation to the smoke test harness: - Layer 1: static validation of dist-scalprum/ after OCI download - Layer 2: inline backend probe plugin querying dynamicPluginsServiceRef Made-with: Cursor

…to install-dynamic-plugins.py Replace createBackend() with startTestBackend() for a minimal, focused smoke test that only validates dynamic plugin loading. Remove 23 static plugin dependencies and the in-process OCI download logic in favor of the upstream install-dynamic-plugins.py script fetched at CI time. Made-with: Cursor Signed-of-by: rlan@redhat.com

…d smoke-test file

sonarqubecloud · 2026-04-16T08:37:22Z

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

openshift-ci bot added the do-not-merge/work-in-progress label Apr 9, 2026

github-actions bot added the non-workspace-changes PR changes files outside workspace directories label Apr 9, 2026

rostalan changed the title ~~Replace Docker-based smoke tests with native Node.js harness~~ feat(poc): replace Docker-based smoke tests with native Node.js harness Apr 9, 2026

openshift-ci bot added the needs-rebase label Apr 9, 2026

gustavolira reviewed Apr 10, 2026

View reviewed changes

gustavolira mentioned this pull request Apr 13, 2026

feat(install-dynamic-plugins): port from Python to TypeScript/Node.js redhat-developer/rhdh#4574

Open

6 tasks

rostalan force-pushed the smoke-test-poc branch from a88d461 to a827513 Compare April 13, 2026 16:16

rostalan force-pushed the smoke-test-poc branch from a827513 to b23aead Compare April 15, 2026 07:48

openshift-ci bot removed the needs-rebase label Apr 15, 2026

rostalan marked this pull request as ready for review April 15, 2026 07:49

rostalan requested review from a team, gashcrumb and kadel as code owners April 15, 2026 07:49

openshift-ci bot removed the do-not-merge/work-in-progress label Apr 15, 2026

rostalan mentioned this pull request Apr 15, 2026

chore(tests) updated the rhdh and plugin error detection #1973

Closed

rostalan force-pushed the smoke-test-poc branch from eec5ff0 to 855f65d Compare April 15, 2026 12:22

jonkoops reviewed Apr 15, 2026

View reviewed changes

rostalan added 7 commits April 16, 2026 10:36

address sonarqube vulnerability

8357cf4

address sonarqube analysis

c0424d9

refactor complex reportAndWrite

3b83f1d

bumped node, switched to npm test, pinned action to commit, refactore…

b085b24

…d smoke-test file

changed test file to .ts, addressed errors, fixed linter errors

c503446

rostalan force-pushed the smoke-test-poc branch from cf2f1a7 to c503446 Compare April 16, 2026 08:36

rostalan requested review from gustavolira and jonkoops April 16, 2026 08:37

	uses: actions/setup-node@v4
	uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0

Conversation

rostalan commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci bot commented Apr 9, 2026

Uh oh!

rostalan commented Apr 9, 2026

Uh oh!

github-actions bot commented Apr 9, 2026

Uh oh!

github-actions bot commented Apr 9, 2026

Uh oh!

github-actions bot commented Apr 9, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rostalan Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 14, 2026

Uh oh!

rostalan commented Apr 15, 2026

Uh oh!

github-actions bot commented Apr 15, 2026

Uh oh!

rostalan commented Apr 15, 2026

Uh oh!

github-actions bot commented Apr 15, 2026

Uh oh!

rostalan commented Apr 15, 2026

Uh oh!

github-actions bot commented Apr 15, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Apr 16, 2026

Quality Gate passed

Uh oh!

Reviewers

Assignees

Labels

rostalan commented Apr 9, 2026 •

edited

Loading

rostalan Apr 13, 2026 •

edited

Loading