Add certification test plans for Ubuntu 26 (New) by stanley31huang · Pull Request #2304 · canonical/checkbox

stanley31huang · 2026-01-26T06:30:20Z

WARNING: This modifies com.canonical.certification::sru-server

Description

This goal for this PR is create certification test plans for PC platform, IoT platform running server image and desktop image.

there are some key points for those changes in this PR:

We are developing component-level test plans to manage the scope for Ubuntu 26 more effectively. This also allows us to run targeted, minimal tests on different SKUs sharing the same platform.
We are leveraging the same test scope for both PC and IoT Desktop images. We are attempting to mirror this coverage on IoT Server images, but currently, several test cases are not applicable or available for that specific environment.

Resolved issues

Documentation

Tests

- adding ARM vfp tests into generic cpu-cert-automated test plan - create a new media card test plan as some interfaces are no longer avaialable in recent IoT platform

adding IoT cert test plan for server/desktop image

codecov · 2026-01-26T06:31:28Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 55.01%. Comparing base (38b29e2) to head (ad07a14).
⚠️ Report is 12 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2304      +/-   ##
==========================================
+ Coverage   54.88%   55.01%   +0.13%     
==========================================
  Files         412      413       +1     
  Lines       44246    44393     +147     
  Branches     8154     8183      +29     
==========================================
+ Hits        24284    24423     +139     
+ Misses      19132    19131       -1     
- Partials      830      839       +9

Flag	Coverage Δ
checkbox-ng	`71.63% <ø> (ø)`
checkbox-support	`69.55% <ø> (ø)`
provider-base	`32.26% <ø> (+0.54%)`	⬆️
provider-certification-client	`57.14% <ø> (ø)`
provider-certification-server	`57.14% <ø> (ø)`
provider-genio	`96.90% <ø> (ø)`
provider-gpgpu	`93.14% <ø> (ø)`
provider-iiotg	`100.00% <ø> (ø)`
provider-resource	`39.57% <ø> (ø)`
provider-sru	`97.97% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

fix test plan check issue

Hook25

Pure grammar/consistency pass for now

I think you forgot to commit these:

unable to find nested part: com.canonical.certification::after-suspend-camera-automated
unable to find nested part: com.canonical.certification::after-suspend-fingerprint-cert-Automated

Why are we still using regex inclusions? do we have any "blocker" to using only template ids?
Why are there quite a few "empty" testplans around?
Why is the list of exclusions so long on many even though the testplans are new?
I have prepared a diff from the previous coverage. Please give it a look and see if all changes make sense, some are stautus changes (non-blocker -> blocker), some seem like addition or removals. Do comment on them.

Caveat, these dont include regex inclusions, as they arent supported by expand. If you want to play around with them use this

function diff_tp(){
  tp1=$1
  tp2=$2
  expand_tp1=$(checkbox-cli expand --format=json $tp1 | jq -r '.[] | (.["template-id"] // .id) + " "  + .["certification-status"]?' | sort)
  expand_tp2=$(checkbox-cli expand --format=json $tp2 | jq -r '.[] | (.["template-id"] // .id) + " "  + .["certification-status"]?' | sort)
  diff <(echo "$expand_tp1") <(echo "$expand_tp2")
}

Diffs: (< is 26.04, > is 24.04)

client-cert-desktop.diff.txt

client-iot-desktop.diff.txt

client-iot-server.diff.txt

providers/base/units/keys/jobs.pxu

providers/base/units/led/jobs.pxu

Hook25 · 2026-01-27T13:30:36Z

providers/base/units/mediacard/test-plan.pxu

+    mediacard/storage-preinserted-.*
+bootstrap_include:
+    removable_partition


This was empty before, do we intend changing it? fine by me, weird it was empty before

This is a missing job under the automated part, we should add it in this test plan.

Hook25 · 2026-01-27T13:32:51Z

providers/base/units/memory/test-plan.pxu

+include:
+


is this supposed to be empty? whats the purpose of an empty testplan?

Yes, they are empty on purpose!
It’s a placeholder strategy for the 26.04 cycle.

If we link these Component Test Plans now (even while empty), we never have to touch the 'Big Test Plan' like IOT/Server/Desktop etc. again just to add a job. We can just update the Component Plan later, and it automatically reflects in the main plan. It saves us from having to review or might missing to add the Big Plan every single time we add a test case.

Hook25 · 2026-01-27T13:34:34Z

providers/base/units/power-management/test-plan.pxu

+include:
+


We are making this empty and shifting all under a new testplan?

The both of jobs (suspend-on-battery-power and suspend-on-AC-power) are related the suspend function test, it's more clearly to add both in the after suspend test plan.

Hook25 · 2026-01-27T13:36:38Z

providers/base/units/thunderbolt/test-plan.pxu

+include:



Yes, they are empty on purpose!
It’s a placeholder strategy for the 26.04 cycle.

If we link these Component Test Plans now (even while empty), we never have to touch the 'Big Test Plan' like IOT/Server/Desktop etc. again just to add a job. We can just update the Component Plan later, and it automatically reflects in the main plan. It saves us from having to review or might missing to add the Big Plan every single time we add a test case.

Hook25 · 2026-01-27T13:36:59Z

providers/base/units/touchpad/test-plan.pxu

+_name: Touchpad tests after suspend (Manual)
+_description:
+ Touchpad tests after suspend (Manual)
+include:


Minor, whats going on with tabs here?

Actually i has modified a lot for this, this one is lost, I will modify it

Hook25 · 2026-01-27T13:37:47Z

providers/base/units/usb/test-plan.pxu

+unit: test plan
+_name: USB Type-C tests (Automated)
+_description: USB Type-C tests (Automated)
+include:


Yes, they are empty on purpose!
It’s a placeholder strategy for the 26.04 cycle.

If we link these Component Test Plans now (even while empty), we never have to touch the 'Big Test Plan' like IOT/Server/Desktop etc. again just to add a job. We can just update the Component Plan later, and it automatically reflects in the main plan. It saves us from having to review or might missing to add the Big Plan every single time we add a test case.

clairlin53 · 2026-02-03T06:26:56Z

Pure grammar/consistency pass for now

I think you forgot to commit these:

unable to find nested part: com.canonical.certification::after-suspend-camera-automated
unable to find nested part: com.canonical.certification::after-suspend-fingerprint-cert-Automated

Ans: I've updated, thanks for pointing out.

2. Why are we still using regex inclusions? do we have any "blocker" to using only template ids?

I just copied and pasted the original job definition. There is no blocker, but I suggest we do that cleanup in a follow-up PR.

3. Why are there quite a few "empty" testplans around?

Ans:
Yes, they are empty on purpose!
It’s a placeholder strategy for the 26.04 cycle.

If we link these Component Test Plans now (even while empty), we never have to touch the 'Big Test Plan' like IOT/Server/Desktop etc. again just to add a job. We can just update the Component Plan later, and it automatically reflects in the main plan. It saves us from having to review or might missing to add the Big Plan every single time we add a test case.

4. Why is the list of exclusions so long on many even though the testplans are new?

Ans: Need @stanley31huang or @rickwu666666 to answer the question.

5. I have prepared a diff from the previous coverage. Please give it a look and see if all changes make sense, some are stautus changes (non-blocker -> blocker), some seem like addition or removals. Do comment on them.

Ans: I've modified and updated in the google spreadsheets for reference.

Caveat, these dont include regex inclusions, as they arent supported by expand. If you want to play around with them use this
function diff_tp(){
  tp1=$1
  tp2=$2
  expand_tp1=$(checkbox-cli expand --format=json $tp1 | jq -r '.[] | (.["template-id"] // .id) + " "  + .["certification-status"]?' | sort)
  expand_tp2=$(checkbox-cli expand --format=json $tp2 | jq -r '.[] | (.["template-id"] // .id) + " "  + .["certification-status"]?' | sort)
  diff <(echo "$expand_tp1") <(echo "$expand_tp2")
}
Diffs: (< is 26.04, > is 24.04)

client-cert-desktop.diff.txt

client-iot-desktop.diff.txt

client-iot-server.diff.txt

I also renamed desktop to base per @pieqq's suggestion.

rickwu666666 · 2026-02-03T06:40:44Z

Why is the list of exclusions so long on many even though the testplans are new?

I think these jobs don't suit the IoT project. ideally, we plan to manage these jobs via manifest.

clairlin53 and others added 3 commits January 26, 2026 14:09

add_desktop26_testplan

8b91cd2

update CPU and media card test plans

d662b45

- adding ARM vfp tests into generic cpu-cert-automated test plan - create a new media card test plan as some interfaces are no longer avaialable in recent IoT platform

add IoT cert test plan for server/desktop image

05c1596

adding IoT cert test plan for server/desktop image

fix test plan check issue

66e40dc

fix test plan check issue

Hook25 requested changes Jan 27, 2026

View reviewed changes

Rename desktop naming and fix typos, missing tp

ad07a14

clairlin53 requested a review from Hook25 February 5, 2026 02:27

		include:

		include:

		include:

Conversation

stanley31huang commented Jan 26, 2026 • edited by clairlin53 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

WARNING: This modifies com.canonical.certification::sru-server

Description

Resolved issues

Documentation

Tests

Uh oh!

codecov bot commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Hook25 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

clairlin53 commented Feb 3, 2026

Uh oh!

rickwu666666 commented Feb 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

stanley31huang commented Jan 26, 2026 •

edited by clairlin53

Loading

codecov bot commented Jan 26, 2026 •

edited

Loading