feat(install-dynamic-plugins): port from Python to TypeScript/Node.js#4574
Open
gustavolira wants to merge 20 commits intoredhat-developer:mainfrom
Open
feat(install-dynamic-plugins): port from Python to TypeScript/Node.js#4574gustavolira wants to merge 20 commits intoredhat-developer:mainfrom
gustavolira wants to merge 20 commits intoredhat-developer:mainfrom
Conversation
Code Review by Qodo
1.
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Review Summary by QodoPort install-dynamic-plugins from Python to TypeScript/Node.js with performance improvements
WalkthroughsDescription• **Complete rewrite**: Replaces Python-based install-dynamic-plugins.py with a TypeScript/Node.js
implementation (18 modules, 105 Jest tests)
• **Performance improvements**: Incorporates parallel OCI downloads, shared image cache, and
streaming SHA verification from the install-dynamic-plugins-fast.py POC
• **New package structure**: scripts/install-dynamic-plugins/ with TypeScript sources, bundled to
single dist/install-dynamic-plugins.cjs (~412 KB)
• **Runtime contract unchanged**: Same dynamic-plugins.yaml input schema, same output format, same
lock-file behavior, same {{inherit}} semantics
• **Resource-conscious concurrency**: Respects cgroup CPU limits with default workers `max(1,
min(floor(cpus/2), 6)), configurable via DYNAMIC_PLUGINS_WORKERS`
• **Memory-efficient**: Streaming tar extraction and SHA verification; typical 10-plugin run uses
20–80 MB peak RSS
• **Security parity**: Path traversal prevention, zip-bomb detection, symlink validation, device
file rejection, SRI integrity verification, registry fallback
• **CI updated**: Replaced pytest with npm run tsc && npm test plus bundle freshness
verification
• **Container image**: Updated Containerfile to copy .cjs bundle instead of .py; wrapper shell
script execs node instead of python
• **Deleted**: install-dynamic-plugins.py (1288 LOC), test_install-dynamic-plugins.py (3065
LOC), pytest.ini
• **Documentation**: Comprehensive README, updated user docs and CI references
Diagramflowchart LR
A["Python Script<br/>install-dynamic-plugins.py"] -->|"Replaced by"| B["TypeScript/Node.js<br/>18 modules + 105 tests"]
B -->|"Bundled to"| C["dist/install-dynamic-plugins.cjs<br/>~412 KB"]
C -->|"Copied in"| D["Container Image<br/>Containerfile"]
E["Parallel OCI<br/>Downloads"] -->|"Incorporated"| B
F["Shared Image<br/>Cache"] -->|"Incorporated"| B
G["Streaming SHA<br/>Verification"] -->|"Incorporated"| B
H["pytest"] -->|"Replaced by"| I["npm test<br/>Jest 105 tests"]
File Changes1. scripts/install-dynamic-plugins/src/index.ts
|
Contributor
|
The container image build workflow finished with status: |
Contributor
|
my browser cannot even load the 11,5k-line file 😅 |
Member
Author
@rostalan No need to review that file, it's the auto-generated esbuild bundle of src/. Just pushed a commit marking it as linguist-generated so GitHub collapses it in the diff. Only src/ and tests/ need review; CI verifies the bundle is up-to-date on every push. |
Contributor
|
The container image build workflow finished with status: |
gustavolira
added a commit
to gustavolira/rhdh
that referenced
this pull request
Apr 13, 2026
… feedback Addresses review feedback on PR redhat-developer#4574: - Extract shared helpers into `src/util.ts`: `fileExists`, `isInside`, `isPlainObject`, `isAllowedEntryType`, `markAsFresh`. Removes 4x duplication across `index.ts`, `catalog-index.ts`, `installer-oci.ts`, `merger.ts`, and `tar-extract.ts`. - Unify `catalog-index.ts` tar filter with `tar-extract.ts`: oversized entries now throw `InstallException` instead of being silently dropped; `OldFile` and `ContiguousFile` are accepted (were previously excluded by mistake). Uses the `isAllowedEntryType` helper. - Extract `markAsFresh(installed, pluginPath)` helper used by both installers to drop stale hash entries after a successful install. - `installer-npm.ts`: use `npm pack --json` instead of parsing the last line of text stdout (warnings on stdout would shift the filename). Also simplify the integrity-check flow — one gate that throws on missing hash, then one verify call (was two overlapping conditionals). - Split `Plugin` → `PluginSpec` (YAML schema) + `Plugin` (internal, with `_level`/`plugin_hash`/`version`). Makes it explicit which fields originate from user YAML vs runtime state. - `lock-file.ts`: add `DYNAMIC_PLUGINS_LOCK_TIMEOUT_MS` (default 10 min) so a stale lock from a `kill -9`'d process no longer wedges the init container forever. New test covers the timeout path. - Drop the broken `lint:check` script — it had `|| true` silencing every lint error and there is no ESLint config in the package. - `README.md`: remove stale reference to non-existent `cli.ts`, document the new lock-timeout env var, mention `util.ts`. Tests: 115 (was 114). Bundle rebuilt (413.4 KB).
Contributor
|
The container image build workflow finished with status: |
Contributor
|
The container image build workflow finished with status: |
gustavolira
added a commit
to gustavolira/rhdh
that referenced
this pull request
Apr 13, 2026
Resolves the 21 issues flagged by SonarQube on PR redhat-developer#4574: Prototype pollution (CodeQL, merger.ts) - `deepMerge` now assigns via `Object.defineProperty` (bypasses the `__proto__` setter on `Object.prototype`) in addition to the existing `FORBIDDEN_KEYS` guard. CodeQL recognizes this pattern. Redundant type assertions - `index.ts:180`: drop `pc as Record<string, unknown>` — use the `isPlainObject` type guard already imported from `util.ts`. - `installer-npm.ts:37`, `installer-oci.ts:35`: replace `(plugin.pluginConfig ?? {}) as Record<string, unknown>` with a typed local variable. - `installer-oci.ts:41,51,71,78`: drop `as string` casts by restructuring the `isAlreadyInstalled` helper with proper `undefined` checks. - `merger.ts:136-140`: replace `.slice(-1)[0] as string` with `.at(-1) ?? ''`. - `merger.ts:215`: `ReadonlyArray<keyof Plugin | string>` collapses to `ReadonlyArray<string>`. Cognitive complexity reductions - `installOciPlugin` (17 → ~10): extract `resolvePullPolicy` and `isAlreadyInstalled` helpers. - `mergeOciPlugin` (20 → ~12): extract `resolveInherit`. - `npmPluginKey` (16 → ~7): extract `tryParseAlias`, `isGitLikeSpec`, `stripRefSuffix`. - `ociPluginKey`: extract `autoDetectPluginPath`. Modern JS / readability (es2015-es2022) - `integrity.ts`: `charCodeAt` → `codePointAt` (es2015). - `oci-key.ts`: use `String.raw` for the regex pieces containing `\s`, `\d`, `\]`, `\\` instead of escaped string literals (es2015). - `oci-key.ts:escape`: `.replace(/.../g, ...)` → `.replaceAll(...)` (es2021). - `plugin-hash.ts`: pass an explicit code-point comparator to `sort` so deterministic-hash behavior is spelled out. `localeCompare` is NOT used — it varies per-locale and would break hash stability. All 115 tests still pass. Bundle rebuilt (415.1 KB).
Contributor
|
The container image build workflow finished with status: |
| * gives CodeQL the pattern it recognizes for prototype-pollution safety. | ||
| */ | ||
| function safeSet(dst: Record<string, unknown>, key: string, value: unknown): void { | ||
| Object.defineProperty(dst, key, { |
Member
Author
|
/review |
PR Reviewer Guide 🔍(Review updated until commit ee984d5)Here are some key observations to aid the review process:
|
Contributor
|
The container image build workflow finished with status: |
gustavolira
added a commit
to gustavolira/rhdh
that referenced
this pull request
Apr 13, 2026
… feedback Addresses review feedback on PR redhat-developer#4574: - Extract shared helpers into `src/util.ts`: `fileExists`, `isInside`, `isPlainObject`, `isAllowedEntryType`, `markAsFresh`. Removes 4x duplication across `index.ts`, `catalog-index.ts`, `installer-oci.ts`, `merger.ts`, and `tar-extract.ts`. - Unify `catalog-index.ts` tar filter with `tar-extract.ts`: oversized entries now throw `InstallException` instead of being silently dropped; `OldFile` and `ContiguousFile` are accepted (were previously excluded by mistake). Uses the `isAllowedEntryType` helper. - Extract `markAsFresh(installed, pluginPath)` helper used by both installers to drop stale hash entries after a successful install. - `installer-npm.ts`: use `npm pack --json` instead of parsing the last line of text stdout (warnings on stdout would shift the filename). Also simplify the integrity-check flow — one gate that throws on missing hash, then one verify call (was two overlapping conditionals). - Split `Plugin` → `PluginSpec` (YAML schema) + `Plugin` (internal, with `_level`/`plugin_hash`/`version`). Makes it explicit which fields originate from user YAML vs runtime state. - `lock-file.ts`: add `DYNAMIC_PLUGINS_LOCK_TIMEOUT_MS` (default 10 min) so a stale lock from a `kill -9`'d process no longer wedges the init container forever. New test covers the timeout path. - Drop the broken `lint:check` script — it had `|| true` silencing every lint error and there is no ESLint config in the package. - `README.md`: remove stale reference to non-existent `cli.ts`, document the new lock-timeout env var, mention `util.ts`. Tests: 115 (was 114). Bundle rebuilt (413.4 KB).
gustavolira
force-pushed
the
feat/install-dynamic-plugins-ts
branch
from
b76ee6a to
dd84f75
Compare
gustavolira
added a commit
to gustavolira/rhdh
that referenced
this pull request
Apr 13, 2026
Resolves the 21 issues flagged by SonarQube on PR redhat-developer#4574: Prototype pollution (CodeQL, merger.ts) - `deepMerge` now assigns via `Object.defineProperty` (bypasses the `__proto__` setter on `Object.prototype`) in addition to the existing `FORBIDDEN_KEYS` guard. CodeQL recognizes this pattern. Redundant type assertions - `index.ts:180`: drop `pc as Record<string, unknown>` — use the `isPlainObject` type guard already imported from `util.ts`. - `installer-npm.ts:37`, `installer-oci.ts:35`: replace `(plugin.pluginConfig ?? {}) as Record<string, unknown>` with a typed local variable. - `installer-oci.ts:41,51,71,78`: drop `as string` casts by restructuring the `isAlreadyInstalled` helper with proper `undefined` checks. - `merger.ts:136-140`: replace `.slice(-1)[0] as string` with `.at(-1) ?? ''`. - `merger.ts:215`: `ReadonlyArray<keyof Plugin | string>` collapses to `ReadonlyArray<string>`. Cognitive complexity reductions - `installOciPlugin` (17 → ~10): extract `resolvePullPolicy` and `isAlreadyInstalled` helpers. - `mergeOciPlugin` (20 → ~12): extract `resolveInherit`. - `npmPluginKey` (16 → ~7): extract `tryParseAlias`, `isGitLikeSpec`, `stripRefSuffix`. - `ociPluginKey`: extract `autoDetectPluginPath`. Modern JS / readability (es2015-es2022) - `integrity.ts`: `charCodeAt` → `codePointAt` (es2015). - `oci-key.ts`: use `String.raw` for the regex pieces containing `\s`, `\d`, `\]`, `\\` instead of escaped string literals (es2015). - `oci-key.ts:escape`: `.replace(/.../g, ...)` → `.replaceAll(...)` (es2021). - `plugin-hash.ts`: pass an explicit code-point comparator to `sort` so deterministic-hash behavior is spelled out. `localeCompare` is NOT used — it varies per-locale and would break hash stability. All 115 tests still pass. Bundle rebuilt (415.1 KB).
Contributor
|
The container image build workflow finished with status: |
Member
Author
|
/review |
|
Persistent review updated to latest commit ee984d5 |
Three fixes + regression tests from automated PR review: 1. `MAX_ENTRY_SIZE` NaN fallback (Qodo bug #1) `Number(process.env.MAX_ENTRY_SIZE)` returned NaN for non-numeric values, which silently disabled the zip-bomb / oversized-entry size check (every `stat.size > NaN` evaluates false). Extracted to a `parseMaxEntrySize()` helper that uses `Number.parseInt` + validates the result is a finite integer >= 1, else falls back to the default. 2. Boundary-safe OCI plugin-path filter (Qodo bug #2) `extractOciPlugin` used `filePath.startsWith(pluginPath)` which also matched sibling directories with the same prefix (e.g. requesting `plugin-one` would also extract `plugin-one-evil/`). Now requires either exact match or a path-separator boundary. 3. Prototype-pollution guard in `deepMerge` (CodeQL finding) User-supplied YAML could contain `__proto__`, `constructor`, or `prototype` keys. `deepMerge` blindly copied them via `dst[key] =`, allowing prototype-chain pollution. These keys are now skipped in both `deepMerge` and `copyPluginFields`. Added regression tests: prefix-collision extraction, prototype-pollution merge, 7 tests for `parseMaxEntrySize` covering NaN/empty/negative/zero inputs. Total: 114 tests (was 105). Bundle rebuilt via `npm run build`.
… feedback Addresses review feedback on PR redhat-developer#4574: - Extract shared helpers into `src/util.ts`: `fileExists`, `isInside`, `isPlainObject`, `isAllowedEntryType`, `markAsFresh`. Removes 4x duplication across `index.ts`, `catalog-index.ts`, `installer-oci.ts`, `merger.ts`, and `tar-extract.ts`. - Unify `catalog-index.ts` tar filter with `tar-extract.ts`: oversized entries now throw `InstallException` instead of being silently dropped; `OldFile` and `ContiguousFile` are accepted (were previously excluded by mistake). Uses the `isAllowedEntryType` helper. - Extract `markAsFresh(installed, pluginPath)` helper used by both installers to drop stale hash entries after a successful install. - `installer-npm.ts`: use `npm pack --json` instead of parsing the last line of text stdout (warnings on stdout would shift the filename). Also simplify the integrity-check flow — one gate that throws on missing hash, then one verify call (was two overlapping conditionals). - Split `Plugin` → `PluginSpec` (YAML schema) + `Plugin` (internal, with `_level`/`plugin_hash`/`version`). Makes it explicit which fields originate from user YAML vs runtime state. - `lock-file.ts`: add `DYNAMIC_PLUGINS_LOCK_TIMEOUT_MS` (default 10 min) so a stale lock from a `kill -9`'d process no longer wedges the init container forever. New test covers the timeout path. - Drop the broken `lint:check` script — it had `|| true` silencing every lint error and there is no ESLint config in the package. - `README.md`: remove stale reference to non-existent `cli.ts`, document the new lock-timeout env var, mention `util.ts`. Tests: 115 (was 114). Bundle rebuilt (413.4 KB).
Sonar typescript:S5852 flagged `/^[A-Za-z0-9+/]+={0,2}$/` and `/=+$/` in
`integrity.ts` as ReDoS hotspots (super-linear backtracking risk). In
practice these are O(n) because the character classes are flat, but
Sonar is conservative about any `+`/`*` quantifier.
Replaces both regexes with `isBase64Shape()` (char-code scan, one pass)
and `stripTrailingEquals()` (char-code trim). Same linear complexity,
no regex engine involved, Sonar-clean.
Resolves the 21 issues flagged by SonarQube on PR redhat-developer#4574: Prototype pollution (CodeQL, merger.ts) - `deepMerge` now assigns via `Object.defineProperty` (bypasses the `__proto__` setter on `Object.prototype`) in addition to the existing `FORBIDDEN_KEYS` guard. CodeQL recognizes this pattern. Redundant type assertions - `index.ts:180`: drop `pc as Record<string, unknown>` — use the `isPlainObject` type guard already imported from `util.ts`. - `installer-npm.ts:37`, `installer-oci.ts:35`: replace `(plugin.pluginConfig ?? {}) as Record<string, unknown>` with a typed local variable. - `installer-oci.ts:41,51,71,78`: drop `as string` casts by restructuring the `isAlreadyInstalled` helper with proper `undefined` checks. - `merger.ts:136-140`: replace `.slice(-1)[0] as string` with `.at(-1) ?? ''`. - `merger.ts:215`: `ReadonlyArray<keyof Plugin | string>` collapses to `ReadonlyArray<string>`. Cognitive complexity reductions - `installOciPlugin` (17 → ~10): extract `resolvePullPolicy` and `isAlreadyInstalled` helpers. - `mergeOciPlugin` (20 → ~12): extract `resolveInherit`. - `npmPluginKey` (16 → ~7): extract `tryParseAlias`, `isGitLikeSpec`, `stripRefSuffix`. - `ociPluginKey`: extract `autoDetectPluginPath`. Modern JS / readability (es2015-es2022) - `integrity.ts`: `charCodeAt` → `codePointAt` (es2015). - `oci-key.ts`: use `String.raw` for the regex pieces containing `\s`, `\d`, `\]`, `\\` instead of escaped string literals (es2015). - `oci-key.ts:escape`: `.replace(/.../g, ...)` → `.replaceAll(...)` (es2021). - `plugin-hash.ts`: pass an explicit code-point comparator to `sort` so deterministic-hash behavior is spelled out. `localeCompare` is NOT used — it varies per-locale and would break hash stability. All 115 tests still pass. Bundle rebuilt (415.1 KB).
- `merger.ts isEqual` (complexity 20 → ~6): extract `isArrayEqual` and `isObjectEqual` helpers. Each branch now dispatches to a single- purpose function so the main `isEqual` is a flat type-dispatch instead of nested loops. - `plugin-hash.ts compareCodePoint`: replace the nested ternary `a < b ? -1 : a > b ? 1 : 0` with three early returns. Same behaviour, no nested conditional. 115 tests still pass.
…d context
The root `.dockerignore` has `**/dist` which also filters out the
committed esbuild bundle at
`scripts/install-dynamic-plugins/dist/install-dynamic-plugins.cjs`.
The runtime stage of `build/containerfiles/Containerfile` copies that
file, so the Hermetic Build Image job fails with:
no items matching glob
".../scripts/install-dynamic-plugins/dist/install-dynamic-plugins.cjs"
copied (1 filtered out using .dockerignore)
Add negation patterns so the bundle (and only the bundle) is
re-included in the build context, matching the `.yarn/releases/yarn-*.cjs`
pattern the repo already uses for committed bundled entry points.
…tions Two focus-area suggestions from the Qodo reviewer guide on redhat-developer#4574: 1. Path handling (index.ts) - Resolve `dynamic-plugins.yaml` to an absolute path at startup and log it — the cwd-dependency is now explicit in the operator log. - Resolve `includes[]` entries relative to the config file's directory (was implicitly relative to cwd). Absolute include paths are preserved untouched so existing deployments that pass a full path still work. 2. Segment-based path-traversal check (tar-extract.ts) - Replace the coarse `pluginPath.includes('..')` filter with a segment-based scan: split on `/` and `\\` and reject any segment that is empty, `.`, or `..`. Absolute paths are still rejected up front. - Benign filenames that happen to contain `..` inside a segment (e.g., `my..plugin/`) are now accepted; path-traversal via `foo/../bar`, leading `.`/`..` segments, or empty segments (`foo//bar`) is still blocked. Added 3 new Jest cases covering: - `foo/../bar` rejected - `./plugin-one` rejected - `foo//bar` rejected - `my..plugin` accepted (regression guard for the stricter check) Total tests: 117 (was 115). Bundle rebuilt.
…exists + skip parallel for no-ops Three perf/correctness fixes targeting the no-op "already-installed" restart path that Stan Lewis benchmarked at 17.77s vs 13.72s for Python in redhat-developer#4574 (comment): 1. plugin-hash: byte-compatible with Python (correctness) The previous TS implementation used different field names and JSON separators than Python, so install hashes never matched across the migration — switching from Python to TS triggered a full reinstall of every plugin on the first run. Fixes: - Rename internal field `_level` → `last_modified_level` (Python's name) - Keep `last_modified_level` in the hash (Python keeps it, we used to strip) - Rename `_local` → `_local_package_info`, plus inner field renames (`_pj` → `_package_json`, `_pj_mtime` → `_package_json_mtime`, `_mtime` → `_directory_mtime`, `_missing` → `_not_found`, `_err` → `_error`) - Convert `mtimeMs` to seconds (Python's `os.path.getmtime` returns float seconds) - `stableStringify` now emits Python-style `, ` and `: ` separators so the serialized JSON is byte-identical to `json.dumps(..., sort_keys=True)` Two regression tests pin the hash to known Python-computed SHA256 values; if these break, cross-compat is silently lost. 2. Skopeo.exists: cache forks across calls (perf) `exists()` was the only Skopeo wrapper without an in-memory cache — `image-resolver.ts` calls it per OCI plugin to probe the RHDH registry. With N plugins from the same image, that's N redundant `skopeo inspect --no-tags` forks each restart. Now memoized like `inspect()` / `inspectRaw()` (promise cache for in-flight dedup). For a 30-plugin / 1-image catalog: 30 forks → 1. 3. installOci: skip parallel pool for definitely-no-op plugins (perf) The IF_NOT_PRESENT no-op path was already a fast hash lookup, but it went through `mapConcurrent` + Semaphore + Promise wrapping for every plugin. Added a synchronous pre-pass that filters out "hash present + not forced + pull-policy ≠ Always" plugins before spinning up the worker pool. Saves the entire Promise machinery for the common restart case where every plugin is steady-state. ALWAYS-pull plugins (`:latest!` or explicit `pullPolicy: Always`) still go through the regular install path because they need a `skopeo inspect` to compare local vs remote digest — that's where Fix #2's exists-cache delivers the win. 124 tests now (was 117): +5 Python-compat hash assertions, +2 Skopeo exists-cache assertions. Bundle 418.1 KB (was 415.7).
…trees Mirrors the Python-side fix in redhat-developer#4610 (RHDHBUGS-2139): deep-merging two Backstage `HumanDuration` objects silently combined sibling keys, so a default `{minutes: 60}` plus a user override `{seconds: 30}` produced `PT60M30S` instead of replacing one with the other. Subtrees ending in `schedule.frequency`, `schedule.timeout`, or `schedule.initialDelay` are now replaced wholesale; everything else continues to deep-merge as before. The constant list lives in both implementations and must stay in sync. Also picks up two related improvements from the Python fix: - Explicit collision when `dst[key]` is a scalar but `src[key]` is a dict (was previously masked by safeSet overwriting the scalar). - Splits the dict / scalar branches into `mergeDictValue` / `mergeScalarValue` helpers — keeps `deepMerge` itself well below the Sonar cognitive-complexity threshold. 8 new Jest tests: - Per-subtree replacement (frequency, timeout, initialDelay) - The full RHDHBUGS-2139 repro path (catalog.providers.keycloakOrg.default.schedule.frequency) - Sibling merge inside `schedule` still works (timeout vs frequency) - A bare `frequency` key not under `schedule` still deep-merges - Scalar-vs-dict collision throws with full path - Nested-collision error includes the dotted path Total tests: 132. Bundle 418.5 KB.
Six small cleanups from @jonkoops's review on redhat-developer#4574: - pr.yaml: rename CI step to "Test install-dynamic-plugins" (drop the TypeScript suffix — the implementation language isn't part of the contract being tested). - pr.yaml: switch CI install from `npm install` to `npm ci` (faster, deterministic, fails on lockfile/manifest drift). - package.json: replace `main` with `bin` — this is a CLI entry point, not a programmatic API, so `bin` is the appropriate field. - package.json: switch dependencies from pinned to caret ranges so yarn/npm can dedupe transitive resolutions; the lockfile still guarantees reproducibility. - .prettierrc.js: trim to only the three options that diverge from Prettier 3 defaults (printWidth, singleQuote, arrowParens). Matches RHDH conventions, drops noise. - tsconfig.json: drop options that match TS 5.x defaults (`lib`, `noImplicitAny` (covered by strict), `forceConsistentCasingInFileNames`, `resolveJsonModule` (we don't import JSON), `outDir`/`declaration`/`sourceMap` (no emit needed — esbuild bundles), and add `noEmit: true` so the `tsc` script no longer needs the `--noEmit` flag. 132 tests still pass, bundle rebuilt. Comments not addressed in this commit (out of scope or established design): - ESM vs CJS bundle output — CJS keeps faster startup and matches the `.yarn/releases/yarn-*.cjs` precedent in this repo. - `npm install --production` + ship node_modules instead of bundling — hermetic Konflux builds disable network during the image build, so the bundle is the simplest path; happy to discuss alternatives. - Vitest in place of Jest — significant test-framework swap that's worth its own PR.
Per @jonkoops's review feedback. Vitest already exists in the repo (via `backstage-cli package test` in 3 plugins), is the modern direction of the Backstage CLI test runner, and works natively with TypeScript + ESM without the ts-jest transform layer. Changes: - Replace `jest` + `ts-jest` + `@types/jest` with `vitest@^4.1.4` + `@vitest/coverage-v8@^4.1.4` (matches the version pinned by the three RHDH plugins already on Vitest). - Replace `jest.config.cjs` with `vitest.config.ts` — globals enabled so existing `describe` / `it` / `expect` / `beforeEach` / `afterEach` keep working with no per-file imports. - Drop the `moduleNameMapper` workaround for `.js` imports — Vitest uses Vite's resolver which understands NodeNext-style imports natively. - `finalize-install.test.ts` (the only file using a Jest-specific API): swap `jest.spyOn` for `vi.spyOn` and `jest.SpyInstance` for the `MockInstance` type imported from vitest. - `tsconfig.json`: swap the `"jest"` types entry for `"vitest/globals"`. Includes the previously-untracked `finalize-install.test.ts` (3 tests covering `finalizeInstall`'s error-path skip-and-preserve behaviour) because it is real coverage for code that actually ships. Stats: - node_modules count: 59 packages (was 294 with Jest+ts-jest) - All 132 tests still pass; suite runs in ~2.6s - Bundle unchanged (test runner is dev-only)
Per @jonkoops's review feedback. The previous `prettier --check src __tests__` invocation missed the package's config files and docs (`vitest.config.ts`, `esbuild.config.mjs`, `.prettierrc.js`, `README.md`). Widen to `.` plus a `.prettierignore` so node_modules, the committed bundle, coverage reports, and the npm lockfile stay out of formatting. Reformatted in this commit (auto-applied): - README.md - esbuild.config.mjs - src/merger.ts (one array line) - src/installer-npm.ts (one nested expression) - __tests__/merger.test.ts (one nested expression) - __tests__/skopeo.test.ts (whitespace) 132 tests still pass.
gustavolira
force-pushed
the
feat/install-dynamic-plugins-ts
branch
from
76ab6eb to
22df76d
Compare
Contributor
|
The container image build workflow finished with status: |
…eSet CodeQL keeps flagging `safeSet` because it analyzes the function in isolation and does not see the `FORBIDDEN_KEYS` check at the call site in `deepMerge`. `Object.defineProperty` is already prototype-safe (it bypasses the `__proto__` setter and writes an own descriptor), but the analyzer wants the guard at the point of write. Add the `FORBIDDEN_KEYS.has(key)` check inside `safeSet` itself. Belt-and-suspenders: still safe even if a future caller forgets the guard, and the static analysis is now happy. 132 tests still pass.
Contributor
|
The container image build workflow finished with status: |
…ve complexity Sonar flagged `runInstaller` at cognitive complexity 24 (limit 15). The function was doing too many things in one body: env-var-driven catalog extraction, config file load with two short-circuit cases, include path resolution + DPDY substitution, plugin merge across two levels, hash computation, and the install orchestration. Pull the discrete phases into well-named helpers so `runInstaller` reads as a top-level outline: - `maybeExtractCatalogIndex` — `CATALOG_INDEX_IMAGE` extraction - `loadDynamicPluginsConfig` — read + parse + the two short-circuit cases - `loadAllPlugins` — merge across includes + main file + hash computation - `resolveIncludes` — include-path resolution + DPDY placeholder swap `runInstaller` itself is now a flat sequence with one early-return; well under the 15-point threshold. No behaviour change. 132 tests still pass.
Contributor
|
The container image build workflow finished with status: |
Member
Author
|
@jonkoops tyvm for the review, addressed pretty much all of them, left inline comments on the two I didn't apply |
Stan Lewis benchmarked the full-install case (empty dynamic-plugins-root) at TS 71.4s vs Python 63.5s. The gap is `npm pack` + tar extraction — node-tar is pure JS while Python's `tarfile` is C-backed, so per-package extraction is slower. The earlier perf fixes (skopeo cache, no-op pre-pass) targeted the restart path; they don't help when every plugin has to be downloaded. NPM installs were sequential by design to avoid registry throttling. Add `getNpmWorkers()` (default 3, capped lower than OCI's 6 because the public NPM registry and the shared `~/.npm/_cacache` don't tolerate heavy concurrency) and route NPM installs through `mapConcurrent` instead of a serial loop. Also apply the same `definitelyNoOp` pre-pass that OCI gained, so the steady-state restart path stays just-as-fast. Override with `DYNAMIC_PLUGINS_NPM_WORKERS=1` to restore the original sequential behaviour if a deployment needs to. Local benchmark (30 local packages, ~100KB each, fresh dynamic-plugins-root, 3 runs averaged): TS sequential (NPM_WORKERS=1) 8.57s wall (9% slower than Python) TS parallel (default 3) 3.89s wall (~2x faster than Python) Python (always sequential) 7.80s wall (baseline) 136 tests pass (+4 for getNpmWorkers).
…ts on npm pack Two small follow-ups stacked on the parallel-NPM commit: 1. Block lifecycle hooks during `npm pack` Add `--ignore-scripts` to the `npm pack` invocation. Dynamic plugins are not expected to ship build steps that need to run at install time, and skipping the `preinstall`/`prepack`/`prepare` hooks removes a code-execution-on-install attack surface (a malicious or compromised package could otherwise run arbitrary code in the init container as part of being unpacked). Also shaves a fork+exec per package off the wall clock. 2. Minify the production bundle (with external sourcemap) esbuild now emits a minified `dist/install-dynamic-plugins.cjs` (~221 KB, was ~422 KB) plus a separate `dist/install-dynamic-plugins.cjs.map` for debugging. The sourcemap is not committed (`.gitignore` already restricts the tracked dist/ to just the .cjs) and is not copied into the runtime container by the Containerfile, so there's no production bloat. Cold-start parse cost drops accordingly — most relevant for init container startup which runs once per pod restart. Local benchmark, 30 local NPM packages, fresh dynamic-plugins-root, 5 runs averaged: TS sequential (NPM_WORKERS=1) 8.54s wall (was 8.57s) TS parallel (default 3) 3.82s wall (was 3.99s, ~4% faster) Python (sequential) 8.40s wall Difference is small in steady-state hot-cache benches; the real win is the security hardening from #1 and the cold-start parse-cost reduction from #2 (init container starts on a cold OS page cache). 136 tests still pass.
…ll cleanups Eight items from the local self-review on redhat-developer#4574: 1. Extract `runInstallPipeline` so `installOci` and `installNpm` no longer share ~45 lines of copy-paste body. The pre-pass loop, the `mapConcurrent` invocation, and the outcome-handling loop now live in one place; `installOci` / `installNpm` are thin shims that pass in the worker count, the per-plugin install function, and the label. 2. New `mergeConfigSafely` helper that wraps the `isPlainObject + try/catch deepMerge` pattern that appeared in four different places. Returns false on conflict so the caller can skip the "Installed" success log. 3. `definitelyNoOp` is now a type guard (`plugin is Plugin & { plugin_hash: string }`) instead of a plain `boolean` predicate. The `as string` cast on `installed.delete` inside the consequent is gone. 4. `definitelyNoOp` uses `PullPolicy.ALWAYS` / `PullPolicy.IF_NOT_PRESENT` instead of the bare `'Always'` / `'IfNotPresent'` strings — matches what the rest of the codebase does and would survive a future enum value change. 5. Drop the dead inner short-circuit in `installNpmPlugin`. The pre-pass in `runInstallPipeline` already filters definitely-no-op plugins, so the `installed.has(hash) && !force && pullPolicy !== ALWAYS` block was unreachable under the normal call path. Updated the doc comment to reflect the bounded-parallel reality (no longer "kept sequential to avoid registry throttling"). 6. New `LATEST_TAG_MARKER` constant in `types.ts` for the `:latest!` suffix that was hard-coded in `installer-oci.ts` and `index.ts`. Single source of truth if the OCI tag convention ever shifts. 7. `isAlreadyInstalled` (oci): drop the unused `_plugin` parameter and include the package name in the "already installed" / "digest unchanged" log lines so an operator can correlate the message with the right plugin in an init-container log. 8. The "already installed" log format is now consistent across both the pre-pass (already had it) and the OCI inner check (used to be nameless): `\t==> <package>: already installed, skipping`. Net diff: -77 lines in `index.ts`, no behaviour change. 136 tests pass.
Contributor
|
The container image build workflow finished with status: |
…ognitive complexity Sonar flagged `runInstallPipeline` at cognitive complexity 24 (limit 15) right after the previous consolidation commit moved the per-category loops into one place. Pull the two largest blocks out as their own helpers: - `partitionDefinitelyNoOp(plugins, installed, globalConfig, errors)` — the synchronous pre-pass that drops definitely-no-op plugins and returns the remaining `needsWork` list. - `applyInstallOutcomes(results, globalConfig, errors)` — the loop that records errors, merges `pluginConfig` into the global config, and logs success lines. `runInstallPipeline` is now just the four-step outline (early-exit on empty list, partition, log + run, drain outcomes); cognitive complexity falls well under 15 with no behaviour change. 136 tests still pass.
|
Contributor
|
The container image build workflow finished with status: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the Python-based
install-dynamic-plugins.pyinit-container script with a TypeScript/Node.js implementation, incorporating all improvements from theinstall-dynamic-plugins-fast.pyPOC (#4523): parallel OCI downloads, shared image cache, streaming SHA verification.Motivation: The Python script was a longstanding duplication target for export overlays (see rhdh-plugin-export-overlays#2231). A Node.js implementation removes the Python dependency from the init-container runtime, enables reuse across RHDH core and overlays, and adopts the faster parallel architecture natively.
What changed
scripts/install-dynamic-plugins/— 18 TypeScript modules + 9 Jest test files (105 tests)dist/install-dynamic-plugins.cjs(~412 KB), committed and verified fresh in CI (same pattern as.yarn/releases/yarn-*.cjs).cjsbundle instead of.py; wrapper shell script execsnodeinstead ofpythonnpm run tsc && npm test+ bundle freshness check (replacespytest)install-dynamic-plugins.py(1288 LOC),test_install-dynamic-plugins.py(3065 LOC),pytest.iniKey design decisions
Runtime contract is unchanged
Same
dynamic-plugins.yamlinput schema, sameapp-config.dynamic-plugins.yamloutput, samedynamic-plugin-config.hash/dynamic-plugin-image.hashfiles, same lock-file behaviour, same{{inherit}}semantics and OCI path auto-detection.Resource-conscious concurrency
availableParallelism()respects cgroup CPU limits (init containers often get 0.5 CPU)max(1, min(floor(cpus/2), 6))— cap avoids exhausting registry/networkDYNAMIC_PLUGINS_WORKERS=<n>Memory: streaming everywhere
node-tarstreams extraction — no full-archive read into RAMnode:cryptopipeline for SHA integrity — chunks through the hashSecurity parity with Python
.., absolute)tar-extract.tsMAX_ENTRY_SIZE)tar-extract.ts,catalog-index.tstar-extract.tstar-extract.tspackage/prefix enforced for NPM tarballstar-extract.tssha256/sha384/sha512)integrity.tsregistry.access.redhat.com/rhdh→quay.io/rhdhimage-resolver.tsTest plan
npm run tscpasses (strict mode,noUncheckedIndexedAccess)npm test— 105 Jest tests pass (npm-key, oci-key, integrity, tar-extract, merger, concurrency, lock-file, image-resolver, plugin-hash)npm run buildproduces freshdist/install-dynamic-plugins.cjs.cjs(CI will verify)fast.pybaseline (~2:42 for full catalog)Compatibility
skopeois still installed for OCI inspectioninstall-dynamic-plugins.shwrapper contract unchanged (./install-dynamic-plugins.sh /dynamic-plugins-root)Related