🌱 Skip E2E tests for documentation and YAML-only changes #5005

Gautam3994 · 2025-08-11T22:41:43Z

Issue: Proposal: Speed up E2E tests by skipping suites when changes are irrelevant - #5004

Problem

Currently, all E2E tests run even if a PR only modifies documentation (*.md, *.markdown) or YAML (*.yml, *.yaml) files. This results in unnecessary CI resource usage and slower feedback cycles.

Goal

Skip all E2E tests if the PR contains only documentation or YAML changes (case-insensitive).
Run alpha E2E suites (like alphagenerate and alphaupdate) only if changes are within pkg/cli/alpha/.
Keep the decision logic atomic and per-suite, so each E2E suite can decide locally and consistently whether to run — both in CI and local development.

Proposed Solution

Introduce a Go helper utility in test/e2e/utils that:
- Retrieves the changed files list either from CI environment variables or by running git diff.
- Detects if changes are only docs or YAML files.
- Checks if changes match any required path prefixes for a given suite.
Modify each E2E suite to call this helper in its BeforeSuite hook and Skip itself when appropriate.

This improves CI efficiency by avoiding needless E2E runs, speeding up feedback without sacrificing test coverage.

linux-foundation-easycla · 2025-08-11T22:41:47Z

The committers listed above are authorized under a signed CLA.

✅ login: Gautam3994 / name: Gautam Kumar (345cc41)

k8s-ci-robot · 2025-08-11T22:41:51Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Gautam3994
Once this PR has been reviewed and has the lgtm label, please assign varshaprasad96 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2025-08-11T22:41:52Z

Welcome @Gautam3994!

It looks like this is your first PR to kubernetes-sigs/kubebuilder 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/kubebuilder has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2025-08-11T22:41:53Z

Hi @Gautam3994. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

test/e2e/alphaupdate/e2e_suite_test.go

camilamacedo86 · 2025-08-13T09:56:26Z

/ok-to-test

camilamacedo86

Thank you a lot for the nice work 🥇

We have an error here to solve see the job: https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kubebuilder/5005/pull-kubebuilder-test/1955569142404747264

make[1]: Leaving directory '/home/prow/go/src/sigs.k8s.io/kubebuilder/docs/book/src/getting-started/testdata/project'
./test/check-license.sh
Checking for license header...
./test/check-license.sh: line 29: file: command not found
./test/check-license.sh: line 29: file: command not found
./test/check-license.sh: line 29: file: command not found
./test/check-license.sh: line 29: file: command not found
./test/check-license.sh: line 29: file: command not found
./test/check-license.sh: line 29: file: command not found
./test/check-license.sh: line 29: file: command not found
./test/check-license.sh: line 29: file: command not found
./test/check-license.sh: line 29: file: command not found
./test/check-license.sh: line 29: file: command not found
./test/check-license.sh: line 29: file: command not found
./test/check-license.sh: line 29: file: command not found
./test/check-license.sh: line 29: file: command not found
./test/check-license.sh: line 29: file: command not found
./test/check-license.sh: line 29: file: command not found
./test/check-license.sh: line 29: file: command not found

Gautam3994 · 2025-08-14T22:42:57Z

@camilamacedo86 I noticed the missing file command in the build environment. I realize the guidelines recommend one issue per PR — should I open a separate PR for this fix, or is it okay to include it here since it’s a small change?

Gautam3994 · 2025-08-18T17:39:15Z

Hi @camilamacedo86 just checking in — would you prefer I open a separate PR for the missing file command fix, or should I keep it in this one?

Gautam3994 · 2025-08-27T23:22:09Z

@camilamacedo86 just checking in please let me know if anything else is required from my end

camilamacedo86 · 2025-08-27T23:53:33Z

Hi @Gautam3994

I had no time to review this one properly yet
But you are right we should have another PR to fix the issue with the shell
If you want to do that will be amazing 🚀

we need ensure that this changes will work fine and we will not skip when we should test
Give me a little time to review this one as soon as possible. Thank you a lot !!!

vitorfloriano · 2025-09-14T01:55:50Z

test/e2e/utils/suite_filter.go

+	pattern := `(?i)(^docs/|\.md$|\.markdown$|^\.github/|` +
+		`(OWNERS|OWNERS_ALIASES|SECURITY_CONTACTS|LICENSE)(\.md)?$|\.ya?ml$)`


@Gautam3994 Sometimes we change workflows (which are .yaml files located in the .github dir) that have config that could possibly break the app and we would want to validate that in a full E2E test. Would this logic exclude those files?

k8s-ci-robot · 2025-10-26T20:48:44Z

@Gautam3994: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kubebuilder-e2e-k8s-1-34-1	`345cc41`	link	true	`/test pull-kubebuilder-e2e-k8s-1-34-1`
pull-kubebuilder-e2e-k8s-1-34-0	`345cc41`	link	true	`/test pull-kubebuilder-e2e-k8s-1-34-0`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Copilot

Pull Request Overview

This PR introduces a conditional test execution mechanism to skip E2E test suites when changes are limited to documentation and YAML files, improving CI efficiency. The implementation adds a reusable utility for detecting file changes and determining whether tests should run based on configurable criteria.

Key changes:

New suite_filter.go utility that detects changed files from CI environment variables or git diff
Integration with alpha E2E test suites (alphagenerate and alphaupdate) to skip execution when changes are outside relevant directories
Support for both CI and local development workflows with automatic fallback logic

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
test/e2e/utils/suite_filter.go	Core utility implementing change detection logic with support for regex/prefix matching and docs/YAML filtering
test/e2e/alphaupdate/e2e_suite_test.go	Adds BeforeSuite hook to skip suite when changes are outside `pkg/cli/alpha/` or `test/e2e/alphaupdate/`
test/e2e/alphagenerate/e2e_suite_test.go	Adds BeforeSuite hook to skip suite when changes are outside `pkg/cli/alpha/` or `test/e2e/alphagenerate/`

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-10-27T07:58:26Z

test/e2e/alphaupdate/e2e_suite_test.go

+
+var _ = BeforeSuite(func() {
+	run, why, _ := utils.ShouldRun(utils.Options{
+		RepoRoot:           ".",


The RepoRoot is hardcoded to . which assumes the test is always run from the repository root. If tests are executed from a different directory, this will fail. Consider using a relative path calculation or documenting this assumption clearly.

Copilot · 2025-10-27T07:58:26Z

test/e2e/alphagenerate/e2e_suite_test.go

+
+var _ = BeforeSuite(func() {
+	run, why, _ := utils.ShouldRun(utils.Options{
+		RepoRoot:           ".",


The RepoRoot is hardcoded to . which assumes the test is always run from the repository root. If tests are executed from a different directory, this will fail. Consider using a relative path calculation or documenting this assumption clearly.

Copilot · 2025-10-27T07:58:26Z

test/e2e/utils/suite_filter.go

+	return false, "no files matched include prefixes", nil
+}
+
+func onlyDocsOrYAML(files []string) bool {


The complex regex pattern for detecting docs/YAML files lacks explanation. Add a comment documenting what file types are considered documentation-only (e.g., docs/, *.md, .github/, OWNERS files, .yml/.yaml).

Suggested change

func onlyDocsOrYAML(files []string) bool {

func onlyDocsOrYAML(files []string) bool {

// Documentation-only files are considered to be:

// - Any file under the docs/ directory

// - Markdown files (*.md, *.markdown)

// - Any file under the .github/ directory

// - Special project files: OWNERS, OWNERS_ALIASES, SECURITY_CONTACTS, LICENSE (optionally with .md extension)

// - YAML files (*.yml, *.yaml)

// The following regex matches all of the above (case-insensitive):

k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Aug 11, 2025

k8s-ci-robot requested review from Kavinjsir and camilamacedo86 August 11, 2025 22:41

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 11, 2025

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Aug 11, 2025

camilamacedo86 reviewed Aug 11, 2025

View reviewed changes

test/e2e/alphaupdate/e2e_suite_test.go Outdated Show resolved Hide resolved

Gautam3994 force-pushed the ci/skip-e2e-for-docs-and-yaml branch from 724f35d to 7338b6a Compare August 12, 2025 20:12

Gautam3994 requested a review from camilamacedo86 August 12, 2025 20:13

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 13, 2025

camilamacedo86 requested changes Aug 13, 2025

View reviewed changes

Gautam3994 force-pushed the ci/skip-e2e-for-docs-and-yaml branch from 7338b6a to c1bb6af Compare August 13, 2025 23:44

ci: skip E2E tests for docs and YAML-only changes

345cc41

Gautam3994 force-pushed the ci/skip-e2e-for-docs-and-yaml branch from c1bb6af to 345cc41 Compare August 14, 2025 20:01

Gautam3994 requested a review from camilamacedo86 August 15, 2025 12:25

vitorfloriano reviewed Sep 14, 2025

View reviewed changes

camilamacedo86 requested a review from Copilot October 27, 2025 07:57

Copilot AI reviewed Oct 27, 2025

View reviewed changes

		pattern := `(?i)(^docs/\|\.md$\|\.markdown$\|^\.github/\|` +
		`(OWNERS\|OWNERS_ALIASES\|SECURITY_CONTACTS\|LICENSE)(\.md)?$\|\.ya?ml$)`

-func onlyDocsOrYAML(files []string) bool {
+func onlyDocsOrYAML(files []string) bool {
+	// Documentation-only files are considered to be:
+	//   - Any file under the docs/ directory
+	//   - Markdown files (*.md, *.markdown)
+	//   - Any file under the .github/ directory
+	//   - Special project files: OWNERS, OWNERS_ALIASES, SECURITY_CONTACTS, LICENSE (optionally with .md extension)
+	//   - YAML files (*.yml, *.yaml)
+	// The following regex matches all of the above (case-insensitive):

Uh oh!

🌱 Skip E2E tests for documentation and YAML-only changes #5005

Are you sure you want to change the base?

🌱 Skip E2E tests for documentation and YAML-only changes #5005

Conversation

Gautam3994 commented Aug 11, 2025

Problem

Goal

Proposed Solution

Uh oh!

linux-foundation-easycla bot commented Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Aug 11, 2025

Uh oh!

k8s-ci-robot commented Aug 11, 2025

Uh oh!

k8s-ci-robot commented Aug 11, 2025

Uh oh!

Uh oh!

camilamacedo86 commented Aug 13, 2025

Uh oh!

camilamacedo86 left a comment

Choose a reason for hiding this comment

Uh oh!

Gautam3994 commented Aug 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Gautam3994 commented Aug 18, 2025

Uh oh!

Gautam3994 commented Aug 27, 2025

Uh oh!

camilamacedo86 commented Aug 27, 2025

Uh oh!

vitorfloriano Sep 14, 2025

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented Oct 26, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

linux-foundation-easycla bot commented Aug 11, 2025 •

edited

Loading

Gautam3994 commented Aug 14, 2025 •

edited

Loading