More flexible --test option #359

mattias-p · 2023-11-23T08:16:31Z

Purpose

This PR implements flexible command line overriding of which test cases to run from the command line.

Context

This is an extension to the improvements in Extend --test option to allow passing only a testcase #333.
Replaces Better performance when running specified test. Function to exclude tests #130. The point of that PR is to allow users to exclude test cases from the command line. This PR includes the same functionality using a different command line syntax.
Leverages Ability to run single testcase via test_zone() zonemaster-engine#1312 to properly handle the Basic test module (c.f. Better performance when running specified test. Function to exclude tests #130 (comment)).
Handles the interaction of --test and --profile in accordance with Better performance when running specified test. Function to exclude tests #130 (comment).
For what it's worth, the idea for this design came to me in a discussion about Refactoring of function Zonemaster::Engine::Test::run_one() zonemaster-engine#1294.

Changes

The set of values accepted by --test is extended.
Which test cases to include in the run is now controlled by setting the test_cases profile property and running Zonemaster::Engine->test_zone().
When --test=basic or --test=basic02 is specified, and basic02 considers the domain too broken to test, a CANNOT_CONTINUE message is now emitted, whereas it wasn't emitted before this PR.

Reviewing

Look especially for lacking test coverage.

Testing

Unit tests should pass.

matsduf · 2023-11-23T08:34:02Z

Before this PR CLI would call either test_zone, test_module or test_method in Zonemaster::Engine
to run the test. After this PR CLI always calls test_zone and uses the test_cases profile property to
control which tests are run. This means that the Basic tests are always included even when a single test case
is specified.

Since the semantics of specifying a single test case or single test module is changed so that the Basic tests
are included, an argument could be made that this is a breaking change, but I believe an argument could also
be made that it's still just an addition.

That is a bad change. When working with test zones you want to be able to run just the specific test case and nothing else.

Secondly, with that change, if Basic fails completely no other test case is run, and that is not what we want when testing and creating test zones.

I am against a change that makes it impossible to run just the selected test case.

mattias-p · 2023-11-23T09:05:36Z

All I'm hearing is "no" and "bad". Is there anything about this proposal that you do like?

Also, I'm trying to understand your objection here. AFAICT you're saying that when a user requests that a single test case is run on a domain that doesn't fulfill the requirements posed by Basic, the requested test case still needs to run. Is that correct? And if so, could you explain why this is important, because it's not obvious to me.

matsduf · 2023-11-23T12:20:50Z

All I'm hearing is "no" and "bad". Is there anything about this proposal that you do like?

It would be good to be able to exclude test cases without creating a new profile. It would be good to be able to select two or more test cases and they are run without overlap.

Sometimes it is probably good that Basic is run before a selected test.

Also, I'm trying to understand your objection here. AFAICT you're saying that when a user requests that a single test case is run on a domain that doesn't fulfill the requirements posed by Basic, the requested test case still needs to run. Is that correct? And if so, could you explain why this is important, because it's not obvious to me.

I want to see what happens in that test case for the test zone even when there are errors. And I want minimal output without the extra output from Basic when testing test zones. Sometimes I have to run with --level debug which gives many lines, which will be anymore if Basic is run.

Give the possibility to run a test without adding Basic.

If Basic is not in the profile, why should Basic still be run? The magic could rather be to put the Basic test cases in the profile first, if there are any.

mattias-p · 2023-11-23T15:19:11Z

I want to see what happens in that test case for the test zone even when there are errors. And I want minimal output without the extra output from Basic when testing test zones. Sometimes I have to run with --level debug which gives many lines, which will be anymore if Basic is run.

If Basic is not in the profile, why should Basic still be run? The magic could rather be to put the Basic test cases in the profile first, if there are any.

Ok. Yeah, I see how this could be very valuable in development contexts. And I also don't particularly like that Basic is forced.

I have an idea of how we could simplify things in Engine in a way that supports this. I think I'll make a draft PR there too.

I won't make any more updates to the code here before Engine is in a better shape to support it. But if anyone has more feedback on the direction of this PR, please let me know. Maybe it'll affect the updates in Engine.

matsduf · 2024-11-11T09:36:19Z

@mattias-p, is this still relevant or can it be closed now?

mattias-p · 2024-11-11T13:48:24Z

This conflicts with #371 as well as some of its documentation improvement followups. I plan to rebase this PR once those things have been merged. It seems somewhat unreasonable to get this merged for 2024.2 so I postponed it.

mattias-p · 2024-11-15T09:49:11Z

I rebased this onto latest develop and fixed some typos added some unit tests. I also tightened some validations for sanity reasons, but that shouldn't have any effect on zonemaster-cli behavior.

lib/Zonemaster/CLI/TestCaseSet.pm

script/zonemaster-cli

mattias-p · 2025-03-14T20:42:25Z

Can you please give a more explicit description of the module? I still do not understand it.

I read through the module documentation again and I agree there was some room for improvement. So I worked through it again.

mattias-p · 2025-05-26T15:12:06Z

I've rebased onto current develop and also improved the documentation.

Please review again.

matsduf

This fine, but I have some comments on the documentation to make is clearer. The functions work well.

script/zonemaster-cli

mattias-p · 2025-06-05T06:31:41Z

@matsduf, I applied your suggestions. Please review again.

marc-vanderwal

Looks fine.

I’ve run the test suite through Devel::Cover and found a single line in TestCaseSet.pm that wasn’t covered by the test suite. Otherwise it’s 100% covered.

marc-vanderwal · 2025-06-06T09:30:06Z

lib/Zonemaster/CLI/TestCaseSet.pm

+        }
+    }
+    else {
+        croak "Unrecognized operator '$op'";


This line is the only thing not covered by the existing unit tests.

marc-vanderwal · 2025-06-06T09:30:52Z

t/test_case_set.t

+        } ## end for my $case ( @cases )
+    };    ## end 'new' => sub
+
+    subtest 'apply_modifier' => sub {


There should be a test case checking whether there is a croak if passing an illegal modifier.

Co-authored-by: Mats Dufberg <mats.dufberg@iis.se>

mattias-p · 2025-06-10T08:10:44Z

I've rebased and added a unit test for invalid operators. The unit test is slightly messy, but I didn't know how to do it more cleanly. Please review again.

marc-vanderwal · 2025-06-11T06:51:00Z

Successfully release tested on Rocky Linux 9.

Unit tests pass, and I did a few test runs with varying --test command line arguments, using the new expression syntax this PR introduces. It seems to work as intended.

mattias-p changed the base branch from master to develop November 23, 2023 08:16

mattias-p mentioned this pull request Nov 23, 2023

Extend --test option to allow passing only a testcase #333

Merged

matsduf mentioned this pull request Dec 4, 2023

Better performance when running specified test. Function to exclude tests #130

Closed

matsduf added this to the v2024.1 milestone Dec 4, 2023

mattias-p mentioned this pull request Dec 8, 2023

Ability to run single testcase via test_zone() zonemaster/zonemaster-engine#1312

Merged

mattias-p modified the milestones: v2024.1, v2024.2 Jun 17, 2024

mattias-p force-pushed the test-option branch from 87a3d76 to 3e3e077 Compare August 2, 2024 09:11

mattias-p modified the milestones: v2024.2, v2025.1 Nov 11, 2024

mattias-p force-pushed the test-option branch 2 times, most recently from ecf71b3 to b316e12 Compare November 12, 2024 17:25

matsduf modified the milestones: v2025.1, v2024.2 Nov 13, 2024

mattias-p force-pushed the test-option branch 2 times, most recently from 5ebbad5 to 7690a3c Compare November 14, 2024 14:02

mattias-p added the V-Major Versioning: The change gives an update of major in version. label Nov 14, 2024

mattias-p modified the milestones: v2025.1, v2024.2 Nov 14, 2024

mattias-p marked this pull request as ready for review November 14, 2024 15:15

mattias-p force-pushed the test-option branch from 7690a3c to b536acf Compare November 15, 2024 09:45

matsduf reviewed Mar 14, 2025

View reviewed changes

lib/Zonemaster/CLI/TestCaseSet.pm Outdated Show resolved Hide resolved

lib/Zonemaster/CLI/TestCaseSet.pm Outdated Show resolved Hide resolved

matsduf reviewed Mar 14, 2025

View reviewed changes

script/zonemaster-cli Outdated Show resolved Hide resolved

mattias-p force-pushed the test-option branch from f17fe16 to ca9e7a4 Compare May 26, 2025 15:09

mattias-p requested a review from matsduf May 26, 2025 15:12

mattias-p force-pushed the test-option branch from ca9e7a4 to 37b8f83 Compare June 4, 2025 10:29

matsduf reviewed Jun 4, 2025

View reviewed changes

script/zonemaster-cli Outdated Show resolved Hide resolved

script/zonemaster-cli Outdated Show resolved Hide resolved

script/zonemaster-cli Outdated Show resolved Hide resolved

script/zonemaster-cli Outdated Show resolved Hide resolved

matsduf previously approved these changes Jun 5, 2025

View reviewed changes

marc-vanderwal reviewed Jun 6, 2025

View reviewed changes

mattias-p and others added 11 commits June 10, 2025 10:08

Tidy

1b37345

Expression language for specifying tests to run

20821ca

Update script/zonemaster-cli

d1bcb4e

Co-authored-by: Mats Dufberg <mats.dufberg@iis.se>

Updates after review comments

6992747

Update lib/Zonemaster/CLI/TestCaseSet.pm

bdddff7

Co-authored-by: Mats Dufberg <mats.dufberg@iis.se>

Update lib/Zonemaster/CLI/TestCaseSet.pm

16e7525

Co-authored-by: Mats Dufberg <mats.dufberg@iis.se>

Reworked documentation and naming in TestCaseSet

7c38015

Further tweaks to documentation

204e50c

Apply suggestions from code review

ebe0d08

Co-authored-by: Mats Dufberg <mats.dufberg@iis.se>

Update script/zonemaster-cli

5293577

Co-authored-by: Mats Dufberg <mats.dufberg@iis.se>

Add test for invalid operator

db24ac9

mattias-p dismissed matsduf’s stale review via db24ac9 June 10, 2025 08:08

mattias-p force-pushed the test-option branch from 7d8756b to db24ac9 Compare June 10, 2025 08:08

marc-vanderwal approved these changes Jun 10, 2025

View reviewed changes

matsduf approved these changes Jun 10, 2025

View reviewed changes

mattias-p merged commit 3973afb into zonemaster:develop Jun 10, 2025
3 checks passed

marc-vanderwal added the S-ReleaseTested Status: The PR has been successfully tested in release testing label Jun 11, 2025

More flexible --test option #359

More flexible --test option #359

Uh oh!

Conversation

mattias-p commented Nov 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Context

Changes

Reviewing

Testing

Uh oh!

matsduf commented Nov 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mattias-p commented Nov 23, 2023

Uh oh!

matsduf commented Nov 23, 2023

Uh oh!

mattias-p commented Nov 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

matsduf commented Nov 11, 2024

Uh oh!

mattias-p commented Nov 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mattias-p commented Nov 15, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mattias-p commented Mar 14, 2025

Uh oh!

mattias-p commented May 26, 2025

Uh oh!

matsduf left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mattias-p commented Jun 5, 2025

Uh oh!

marc-vanderwal left a comment

Choose a reason for hiding this comment

Uh oh!

marc-vanderwal Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

marc-vanderwal Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

mattias-p commented Jun 10, 2025

Uh oh!

Uh oh!

marc-vanderwal commented Jun 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mattias-p commented Nov 23, 2023 •

edited

Loading

matsduf commented Nov 23, 2023 •

edited

Loading

mattias-p commented Nov 23, 2023 •

edited

Loading

mattias-p commented Nov 11, 2024 •

edited

Loading