Multiple URLs in Lighthouse plugin

[matejchalk]

User story

As a user, I would like to run Lighthouse on multiple pages in my app, to get a more holistic report of my app's performance, accessibility, etc. But the Lighthouse plugin currently only supports a single URL.

Proposal

We can easily accept an array of URLs and orchestrate multiple Lighthouse reports. The more tricky question is how to process the results from multiple pages. Lighthouse CI creates separate Lighthouse reports per page and doesn't attempt to aggregate them in any way.

The most reasonable solution is to generate the same audits and groups for each page - slugs would be performance-1, accessibility-1, performance-2, etc (display titles would include the source URL for readability). These could then be combined into a single category using multiple refs (e.g. performance category will reference groups performance-1, performance-2, etc.) - we could provide a helper function for that.

import lighthousePlugin, { mergeLighthouseCategoriesFromUrls } from '@code-pushup/lighthouse-plugin';

const lhPlugin = lighthousePlugin([
  'https://example.com',
  'https://example.com/about',
  'https://example.com/contact'
]);

export default {
  plugins: [
    lhPlugin,
    // ... other plugins ...
  ],
  categories: [
     ...mergeLighthouseCategories(lhPlugin),
     // ... other categories ...
  ],
};

Since it's likely that some pages in an application will be more likely than others, we could enable users to each URL their own weight to reflect this in the score calculation.

lighthousePlugin({
  'https://example.com': 2,
  'https://example.com/about': 1
  'https://example.com/contact': 1,
});

Acceptance criteria

• support array of URLs in lighthousePlugin function
• generate distinct audits and groups for each URL
• run lighthouse for each URL sequentially
• provide helper function which configures combined categories for each URL's report
• enable user to provide custom weights per URL (optional, default will be uniform weight)
• single URL usage stays the same (simpler to get started)

[BioPhoton]

Looking at the problem we should consider looking at

• https://github.com/push-based/user-flow?tab=readme-ov-file#basic-user-flows
• https://github.com/push-based/user-flow/blob/main/packages/cli/docs/writing-basic-user-flows.md

Light house tackled this problem in a standardised data structure.

All in all I believe the problem would deserve more discussions.

[matejchalk]

Looking at the problem we should consider looking at

• https://github.com/push-based/user-flow?tab=readme-ov-file#basic-user-flows
• https://github.com/push-based/user-flow/blob/main/packages/cli/docs/writing-basic-user-flows.md

Light house tackled this problem in a standardised data structure.

All in all I believe the problem would deserve more discussions.

I don't understand what you mean about @push-based/user-flow. Based on the docs, the collect command also only accepts a single URL (-t/--url), so I don't see how it solves the problem of multiple URLs 😕 Or is there something I'm missing?

I researched how lighthouse-ci handles multiple URLs. See their collect implementation - it's a simple sequential loop, each URL gets its own JSON report. They don't have any algorithm to merge reports from different URLs together. But integrating it with Code PushUp's weighted average scoring formula is quite an elegant way to aggregate the scores, I believe.

[hanna-skryl]

@matejchalk

But integrating it with Code PushUp's weighted average scoring formula is quite an elegant way to aggregate the scores, I believe.

I've implemented the core support for handling multiple URLs (string | string[]) (#1026), but I'm currently stuck on the URL weighting feature and would appreciate some guidance on how to approach weight distribution.

Here’s what I have accomplished so far:

• Multi-URL support (string | string[])
• Sequential Lighthouse execution
• URL-specific audits and groups (e.g., performance-1, performance-2, etc.)
• Category merging using mergeLighthouseCategories(plugin, userCategories) helper

I have also added a second argument to the helper function, which allows for user-defined categories where specific weights can be set. In my current implementation, when a user defines a category with a specific weight, that weight is preserved for all groups within that category.

However, I am unsure how the weighted URLs should function when the user assigns values in the plugin input. Should these URL weights override the weights set for category refs? If that is the case, propagating the URL weights from the plugin input to the categories helper will be challenging.

Could you clarify the intended behavior for URL weights and how they should interact with the category merging system?

[matejchalk]

@hanna-skryl

Let's take this input as an example:

lighthousePlugin({
  'https://example.com': 2,
  'https://example.com/about': 1
  'https://example.com/contact': 1,
});

The plugin would generate the following audits:

[
  {
    slug: 'largest-contentful-paint-1',
    title: 'Largest Contentful Paint (https://example.com)',
    // description, docsUrl
  },
  {
    slug: 'cumulative-layout-shift-1',
    title: 'Cumulative Layout Shift (https://example.com)',
    // description, docsUrl
  },
  // ... other audits for https://example.com ...

  {
    slug: 'largest-contentful-paint-2',
    title: 'Largest Contentful Paint (https://example.com/about)',
    // description, docsUrl
  },
  {
    slug: 'cumulative-layout-shift-2',
    title: 'Cumulative Layout Shift (https://example.com/about)',
    // description, docsUrl
  },
  // ... other audits for https://example.com/about ...

  {
    slug: 'largest-contentful-paint-3',
    title: 'Largest Contentful Paint (https://example.com/contact)',
    // description, docsUrl
  },
  {
    slug: 'cumulative-layout-shift-3',
    title: 'Cumulative Layout Shift (https://example.com/contact)',
    // description, docsUrl
  },
  // ... other audits for https://example.com/contact ...
]

And the following groups:

[
  {
    slug: 'performance-1',
    title: 'Performance (https://example.com)',
    // description, docsUrl
    refs: [
      { slug: 'largest-contentful-paint-1', weight: 25 },
      { slug: 'cumulative-layout-shift-1', weight: 25 },
      // etc.
    ],
  },
  // ... other Lighthouse categories for https://example.com ...

  {
    slug: 'performance-2',
    title: 'Performance (https://example.com/about)',
    // description, docsUrl
    refs: [
      { slug: 'largest-contentful-paint-2', weight: 25 },
      { slug: 'cumulative-layout-shift-2', weight: 25 },
      // etc.
    ],
  },
  // ... other Lighthouse categories for https://example.com/about ...

  {
    slug: 'performance-3',
    title: 'Performance (https://example.com/contact)',
    // description, docsUrl
    refs: [
      { slug: 'largest-contentful-paint-3', weight: 25 },
      { slug: 'cumulative-layout-shift-3', weight: 25 },
      // etc.
    ],
  },
  // ... other Lighthouse categories for https://example.com/contact ...
]

Then the plugin helper would then combine these results for different URLs into aggregated categories:

[
  {
    slug: 'performance',
    title: 'Performance',
    // description, docsUrl
    refs: [
      { plugin: 'lighthouse', type: 'group', slug: 'performance-1', weight: 2 },
      { plugin: 'lighthouse', type: 'group', slug: 'performance-2', weight: 1 },
      { plugin: 'lighthouse', type: 'group', slug: 'performance-3', weight: 1 },
    ],
  },
  // ... other aggregated categories ...
]

Does that help?

(The -1, -2, -3 suffixes are just a suggestion, BTW. I'm open to having something more descriptive in the slug - e.g. the URL path could be useful, it's likely all URLs will be from the same domain.)