Introduce secondary report format to enable caching of plugin results

[BioPhoton]

User story

As a developer I want to be able to cache as much as possible with Nx and leverage affected

Why Commit Info Breaks Caching
To get reliable caching in tools like Nx, task inputs must stay stable across commits — unless the output truly depends on them.

Example ESLint:

ESLint can be cached effectively when its inputs are:

• The ESLint version
• Source files of the project
• Dependency .d.ts files

In this setup, even if you're on a different commit or branch, ESLint won’t re-run unless something relevant changes. This allows maximum cache reuse.

Note

Using Commit SHA as Input

If you include the commit SHA in:

• Task inputs
• Generated files
• Plugin outputs

Then every commit creates a unique input hash, and ESLint (or any task) loses its cache — even when the code hasn’t changed.

Suggestion

Legend:

• 🐳 Owned by Nx — Nx or other external caching solutions
• 🔌 Owned by plugin creator — depends on code changes
• ☑️ Owned by CodePushup — depends on code changes
• 🔧 Code-related — depends on code changes
• ⛓️ Commit-related — depends on commit (includes code changes)
• ⚡ Fast — fast enough to optionally skip caching
• 💾 Cached — slow or expensive, should be cached
• 🚫 No Cache — intentionally avoids caching

New CLI options for command collect

Option	Type	Default	Description
--persist.report	boolean	true	Will not produce report.json. Normally in combination with --cache
--cache.write	boolean	false	If true it will generate a {persist.outputDir}/<plugin-slug>/audit-outputs.json file containing the audit output data of a plugin
--cache.read	boolean	false	If true it will try to read the plugins audit outputs for file system. If not available it will run the plugin to get the audit outputs on the fly. If

Collect Flow - Generation Phase

Command: npx @code-pushup/cli collect --cache.write --no-report --onlyPlugins eslint

1. For each plugin listed in --onlyPlugins:

   IF --cache.write is given it will write the audit outputs data to {persist.outputDir}/<plugin-slug>/audit-outputs.json (💾☑️).
   This is done in executePlugin after the execution.

2. After the plugins are run

IF --no-report is set, no final report persist.outputDir/report.json is created. (☑️)

Collect Flow - Aggregation Phase

Command: npx @code-pushup/cli collect --cache.read

IF --cache.read is given the CLI checks for existing plugin audit outputs under persist.outputDir/<plugin-slug>/audit-outputs.json and if given it will take the data instead of executing the plugin runner.
This is done in executePlugin before the execution.

ELSE:

IF generateArtifactsCommand is defined
→ Run the command e.g., npx nx run-many ... to produce the raw artifacts (🐳)

Optional Generate plugin artifacts (💾 🔧 🔌)

Generate the plugin report and persist it to persist.outputDir/<plugin-slug>/audit-outputs.json (💾 🔧 ☑️)
In executePlugin we save the plugin report as AuditOutputs.

2. After all plugins are processed, the CLI will combine reports into the final report persist.outputDir/report.json (⛓️ ⚡ 🚫 🔧 ☑️)

Code Pushup Config

{
    plugins: [
      await coveragePlugin({
        generateArtifactsCommand: 'npx nx run-many --project lib-a --targets unit-test int-test',
        reports: ['packages/lib-a/coverage/unit-test/lcov.info', 'packages/lib-a/coverage/int-test/lcov.info'],
      }),
     await nxPerfPlugin({
        audits: ['project-graph-time'],
      }),
    ],
}

Targets

{
  "name": "lib-a",
  "targets": {
     "int-test": {
       "cache": true,
       "outputs": ["{options.coverage.reportsDirectory}"],
       "executor": "@nx/vite:test",
       "options": {
         "configFile": "packages/lib-a/vitest.int.config.ts",
         "coverage.reportsDirectory": "{projectRoot}/coverage/int-test"
       }
    },
     "unit-test": {
       "cache": true,
       "outputs": ["{options.coverage.reportsDirectory}"],
       "executor": "@nx/vite:test",
       "options": {
         "configFile": "packages/lib-a/vitest.unit.config.ts",
         "coverage.reportsDirectory": "{projectRoot}/coverage/unit-test"
       }
    },
    "code-pushup:coverage": {
       "dependsOn": ["unit-test", "int-test"],
       "cache": true,
       "inputs": ["{coverage-speciftic-globs}"],
       "outputs": ["{options.outputPath}"],
       "executor": "@code-pushup/nx-plugin:cli",
       "options": {
         "report": false,
         "cache.write": true,
         "onlyPlugins": "coverage",
         "persist.outputDir": "{projectRoot}/.code-pushup/coverage", 
         "outputPath": "{projectRoot}/.code-pushup/coverage/<plugin-slug>/plugin-report.json",
       }
    },
    "//":"This target could be completely skipped as we d not cache anything here",
   "code-pushup:nx-perf": {
       "cache": false,
       "executor": "@code-pushup/nx-plugin:cli",
       "options": {
         "report": false,
         "onlyPlugins": "nx-perf"
         "persist.outputDir": "{projectRoot}/.code-pushup/nx-perf" 
       }
    },
   "code-pushup": {
       "dependsOn": ["code-pushup:coverage", "code-pushup:nx-perf"],
       "cache": false,
       "outputs": ["{options.outputPath}"],
       "executor": "@code-pushup/nx-plugin:cli",
       "options": {
         "cache.read": true,
         "persist.outputDir": "{projectRoot}/.code-pushup",
         "outputPath": "{projectRoot}/.code-pushup/report.*",
       }
    }
  }
}

Nx Task Graph

graph TD
  A[lib-a:code-pushup ⚡ ⛓️ 🚫] --> B[lib-a:code-pushup:coverage 💾 🔧 ☑️]
  A --> E[lib-a:code-pushup:nx-perf 🚫 🔌]
  B --> C[lib-a:unit-test 💾 🔧 🐳]
  B --> D[lib-a:int-test 💾 🔧 🐳]

Loading

Acceptance criteria

• New CLI options --report - feat: add report option to cli #1058
  • New report format {persist.outputDir}/<plugin-slug>/runner-output.json
• New CLI options --cache.write and --cache.read feat: add audit output caching for execute plugin #1057
  • Collect looks for plugin reports first and only runs if they are missing
• CLI respects report and caching options feat: add caching options to cli #1059

Implementation details

No response

[beaussan]

The only thing I would say would be to introduce a dedicated persist option, instead of reusing persist.outputDir. Otherwise code-pushup can't know where plugin cached outputs (audits, issues, etc) are located

[BioPhoton]

The only thing I would say would be to introduce a dedicated persist option, instead of reusing persist.outputDir. Otherwise code-pushup can't know where plugin cached outputs (audits, issues, etc) are located

We could make it owned by the CLI and use one of the patterns:

• <persist.outputDir>/<plugin-slug>-plugin-report.json
• <persist.outputDir>/<plugin-slug>/plugin-report.json
• <persist.outputDir>/plugin-report.<plugin-slug>.json
• ...

Also the output would be this structure: https://github.com/code-pushup/cli/blob/main/packages/models/docs/models-reference.md#pluginreport

[beaussan]

Generate the plugin report and persist it to persist.outputDir//plugin-report.json (💾 🔧 ☑️)
In executePlugin we save the plugin report as PluginReport before we return in.

I think this step should only be run when running explicitly a plugin directly, otherwise this could lead to weird state where things are not re ran properly. A bit like the new ci setup, Default is safe, but slow. If you want fast, you have to deal with caching per plugin if you need so, but beware that if you don't cleenup properly, or not cache properly, your data will be stale

[matejchalk]

To cache plugins, we can only cache the runner output, not the whole PluginReport. All the plugin metadata is loaded when the code-pushup.config.ts is imported, only the runner is executed by the CLI, the AuditOutput[] is then combined with all the metadata to create the PluginReport. You need to load the config file before you even know which plugins exist, so how could you cache the metadata? I don't think you need to anyway, the slow part is always in the runner.

[BioPhoton]

Changed type. ☝️

[BioPhoton]

Added CLI options