[TE-5134] Add --tag-filters option to filter tests by markers by gchan · Pull Request #426 · buildkite/test-engine-client

gchan · 2026-01-23T00:57:02Z

Description

Add a new --tag-filters CLI flag that allows filtering tests by execution
tags. This is useful for selectively running subsets of tests based on
key-value tag expressions.

Only Pytest is currently supported, using the --tag-filters flag
provided by the Python Test Collector plugin (v1.2.0+) [1]. Tests must be
decorated with @pytest.mark.execution_tag("key", "value") markers to be
filtered by tags.

Tag filtering only works when server-side test filtering is disabled (i.e.,
when all files are returned), ensuring that all tests are available for
tag-based selection.

Example usage:
bktec run --tag-filters "speed:slow"

BUILDKITE_TEST_ENGINE_TAG_FILTERS="speed:fast" bktec run

Please don't confuse tag filtering with the existing filtering
functionality which relies on server-side 'filtering' to return a list of
slow files.

Context

[1] buildkite/test-collector-python#83

https://linear.app/buildkite/issue/TE-5134/add-tag-filters-option-to-bktec

#418

Changes

Add --tag-filters flag and BUILDKITE_TEST_ENGINE_TAG_FILTERS env var
Add TagFilters field to Config struct
Update Pytest runner to pass tag filters to the Python Test Collector plugin via --tag-filters flag
Add version check to ensure buildkite-test-collector v1.2.0+ is installed when using tag filters
Add validation to ensure tag filtering only works with pytest
Ensure all pytest files are available for tag filtering (not just slow/skipped files)
Add comprehensive integration tests for --tag-filters option with pytest markers
Add uv to .tool-versions for local testing of unreleased python_test_collector changes

gchan · 2026-01-23T02:33:22Z

internal/api/filter_tests.go

Clarifying comments to not confuse filter_tests with tag filtering. This class is actually fetching test files, not really filtering down the tests/files to execute.

Add a new --tag-filters CLI flag that allows filtering tests by execution tags. This is useful for selectively running subsets of tests. Currently only Pytest is supported via markers [1]. When specified, the tag filter is passed to pytest via the -m flag (e.g., pytest -m "test_execution"). We will replace the default marker filter with a custom filter option implemented in the Python test collector [2] Changes: - Add --tag-filters flag and BUILDKITE_TEST_ENGINE_TAG_FILTERS env var - Add TagFilters field to Config struct - Update Pytest runner to use -m flag when tag filters are specified. (This will be replaced with a custom filter) - Add test coverage for marker-based filtering for pytest Example usage: bktec run --tag-filters "slow" BUILDKITE_TEST_ENGINE_TAG_FILTERS="not integration" bktec run [1] https://docs.pytest.org/en/7.1.x/example/markers.html [2] buildkite/test-collector-python#83

…ality

…w or skipped files

…markers

…ecution_tag markers Replace pytest's built-in `-m` marker flag with the `--tag-filters` flag from the Python Test Collector plugin (v1.2.0+). This change enables proper key-value tag filtering for test execution. Update test data markers from `@pytest.mark.test_execution("key", "value")` to `@pytest.mark.execution_tag("key", "value")` to match the plugin's expected format. Add documentation explaining the --tag-filters feature and its requirements.

…thon_test_collector changes The uv tool is required to install branch versions of the python_test_collector for local testing of the --tag-filters functionality. This allows developers to test unreleased collector changes by installing from a local path with `uv pip install -e /path/to/python_test_collector` in a uv-managed virtual environment, providing a consistent and reproducible testing workflow. Quick cheat sheet: uv venv source .venv/bin/activate uv pip install -e /path/to/python_test_collector go test ...

…tag-filters support

gchan · 2026-01-26T04:09:43Z

The build will pass when we release a new version of the Python Test Collector buildkite/test-collector-python#83

malclocke · 2026-01-26T21:08:34Z

internal/config/validate.go

 		}
 	}

+	if c.TagFilters != "" && c.TestRunner != "pytest" {


Just a note, I have a wip branch to implement a SupportedFeatures() interface on each runner, e.g.:

func (p Pytest) SupportedFeatures() SupportedFeatures { return SupportedFeatures{ SplitByFile: true, SplitByExample: false, FilterTestFiles: true, AutoRetry: true, Mute: true, Skip: false, } }

Main motivations being that a) this (anti) pattern of switching by runner name is starting to proliferate, b) to give us a way to support config validation like you have implemented here, and c) to give us a way to build the feature support matrix in the README.md from the code base.

Long winded way of saying this is 👍 but I may sweep back to update it some time soonish 😅

Sounds good to me!

malclocke · 2026-01-26T21:13:12Z

internal/command/run_test.go

 			TestCommand: "rspec",
 		},
 	})
-


🤔 why these whitespace removals?

It seems the linter in my editor is doing this! I'll see if I can track down which one

gofumpt maybe? If so we should discuss whether or not we want gofumpt on this project and reformat everything wholesale in one commit if we do.

Yep it was gofumpt it was the default formatter in my editor config.

We had an internal discussion about using gofumpt wholesale across all BK Go projects last year and the consensus was mostly in agreement.
https://buildkite-corp.slack.com/archives/C05Q8CCJEUC/p1763597169525229

malclocke · 2026-01-26T21:26:11Z

internal/runner/pytest.go

+// getPythonPackageVersion retrieves the version of a Python package using importlib.metadata.
+// The pkgName should use hyphens (e.g., "buildkite-test-collector") as that's the package name in metadata.
+func getPythonPackageVersion(pkgName string) (string, error) {
+	pythonCmd := exec.Command("python", "-c", "import importlib.metadata, sys; print(importlib.metadata.version(sys.argv[1]))", pkgName)


Just a note. I think this will be OK but there may be configurations where somehow the test suite runs in a different venv to the environment bktec is running in and this check fails.

I think it's unlikely, but it would be a blocker and we would need to cut a new bktec release to fix. So we could maybe consider logging an error rather than failing outright here.

Logging an error and ignoring the --filter-tags option seems like the safer option. I'll merge this as is and look at refactoring the code in a separate PR.

https://linear.app/buildkite/issue/TE-5148/log-when-bktec-detects-tag-filter-usage-with-invalid-python-test

…support

gchan · 2026-01-27T01:18:22Z

I've added a commit to bump the version of the Python Test Collector. And as expected, the CI build now passes. :)

nprizal · 2026-01-28T20:33:12Z

internal/command/request_param.go

+// that are slow or contain skipped tests. These files are then split into examples
+// The remaining files are sent as is.
+//
+// If tag filtering is enabled, all files are split into examples to support filtering.


gchan commented Jan 23, 2026

View reviewed changes

gchan force-pushed the te-5134-add-tag-filters-option-to-bktec branch 2 times, most recently from 508fb29 to f5ea4ff Compare January 23, 2026 04:27

gchan added 4 commits January 26, 2026 10:47

[TE-5134] Clarify the purpose of the existing 'filter tests' function…

f773ced

…ality

[TE-5134] Enable all pytest files to be filtered by tag, not just slo…

45f3cd1

…w or skipped files

[TE-5134] Validate config to ensure tag filtering only works with pytest

f8631b6

gchan force-pushed the te-5134-add-tag-filters-option-to-bktec branch from f5ea4ff to 6e8faf5 Compare January 25, 2026 22:46

gchan added 3 commits January 26, 2026 14:33

[TE-5134] Add integration tests for --tag-filters option with pytest …

275be6d

…markers

gchan mentioned this pull request Jan 26, 2026

Add --tag-filters option to filter tests using execution_tag marker buildkite/test-collector-python#83

Merged

gchan force-pushed the te-5134-add-tag-filters-option-to-bktec branch 2 times, most recently from 541d1a2 to 2a5ab29 Compare January 26, 2026 03:59

[TE-5134] Add version check for buildkite-test-collector to ensure --…

1a7169a

…tag-filters support

gchan force-pushed the te-5134-add-tag-filters-option-to-bktec branch from 2a5ab29 to 1a7169a Compare January 26, 2026 04:08

gchan marked this pull request as ready for review January 26, 2026 04:10

gchan requested a review from a team as a code owner January 26, 2026 04:10

malclocke reviewed Jan 26, 2026

View reviewed changes

malclocke approved these changes Jan 26, 2026

View reviewed changes

[TE-5134] Bump buildkite-test-collector to >=1.3.0 for --tag-filters …

fb712ca

…support

gchan merged commit 65b5995 into main Jan 27, 2026
1 check passed

gchan deleted the te-5134-add-tag-filters-option-to-bktec branch January 27, 2026 01:18

nprizal reviewed Jan 28, 2026

View reviewed changes

Conversation

gchan commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Context

Changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gchan commented Jan 26, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gchan commented Jan 27, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gchan commented Jan 23, 2026 •

edited

Loading