test: aac add test coverage for `ParseError` variants #6631

federico-stacks · 2025-10-28T10:36:15Z

Description

This PR adds consensus test coverage for the ParseError enum in preparation for the upcoming refactor of that enum. The PR is currently a draft to gather early feedback on test organization and macro usage.

The PR is currently a draft to gather early feedback about test organization and macro usage::

Although this is not to be intended as the final structure, I’ve started isolating ParseError related tests in a dedicate module parse_tests.rs
To support this, I've exported the contract_deploy_consensus_test! and contract_call_consensus_test! macros opting for:
- scoped export, to prevent visibilitty at the crate root level
- impoort consistency, so macros can be used without additional imports (i.e., resolving internal macro dependencies using absolute namespaces).
Futhermore I've added clarity_types crate as test dependency for documentation purposes (doc referencing Parse error variants). This is just a preliminary attempt, and I open to remove it if it's not beneficial (I'm also trying understanding this)
- Currently, it doesn’t work properly with the macro, causing IDE warnings and broken hyperlinks (would function correctly with a standard test function)
- I also noticed that with the macro we loose the Outline view (is empty)
  
  So, apart the doc thing, I'm wondering if in general would be convenient having "native" test functions while keeping the macro just for configuring the test body. Something like this:
```
#[test]
fn test_parse_error_lexer() {
    contract_deploy_consensus_test!(
       parse_error__import_trait_bad_signature,
       contract_name: "my-contract",
       contract_code: &{"(use-trait)"},
   );
}
```

Applicable issues

fixes AAC Testing: Add test coverage for ParseError enum #6627

Additional info (benefits, drawbacks, caveats)

Checklist

Test coverage for new or modified code paths
Changelog is updated
Required documentation changes (e.g., docs/rpc/openapi.yaml and rpc-endpoints.md for v2 endpoints, event-dispatcher.md for new events)
New clarity functions have corresponding PR in clarity-benchmarking repo

…ks-network#6627

…tacks-network#6627

stackslib/src/chainstate/tests/consensus.rs

jacinta-stacks · 2025-10-28T15:08:49Z

stackslib/src/chainstate/tests/parse_tests.rs

+
+/// ParserError: [`ParseErrors::Lexer`]
+/// Caused by: unknown symbol
+/// Outcome: block accepted.


I like your comments. Clear, concise, easy to follow/reason about.

jacinta-stacks · 2025-10-28T15:13:21Z

I actually think it might be slightly easier to reason about the test name/be able to find tests if we did something like

#[test]
fn test_parse_error_lexer() {
    contract_deploy_consensus_test!(
       function_name!(),
       contract_name: "my-contract",
       contract_code: &{"(use-trait)"},
   );
}

since when I initially looked at contract_deploy_consensus_test! macro it took me a bit to realize it was a #[test]. But I am not strongly one way or the other. Whichever makes it easier for you to reuse these functions in your test, go for it. If you need help fixing your errors, let me know. I will give it a look.

federico-stacks · 2025-10-28T16:21:25Z

I actually think it might be slightly easier to reason about the test name/be able to find tests if we did something like
#[test]
fn test_parse_error_lexer() {
    contract_deploy_consensus_test!(
       function_name!(),
       contract_name: "my-contract",
       contract_code: &{"(use-trait)"},
   );
}
since when I initially looked at contract_deploy_consensus_test! macro it took me a bit to realize it was a #[test]. But I am not strongly one way or the other. Whichever makes it easier for you to reuse these functions in your test, go for it. If you need help fixing your errors, let me know. I will give it a look.

Considering we have a macro, we should be able to use function_name!() internally.
Otherwise, if we also expose the function_name() as in your example, we could even replace the macro with a function.

…k#6627

federico-stacks · 2025-10-30T10:55:13Z

Based on the feedback, I've updated consensus macros as proposed in the PR description.
It seems to work nicely (with documentation, but also I added a should_panic test for unreachable error variant)

By the way, adding more tests exposed a new issue related to the vm_error field, basically this test:

#[test]
fn test_circular_reference() {
    contract_deploy_consensus_test!(
        contract_name: "my-contract",
        contract_code: &{"
            (define-constant my-a my-b)
            (define-constant my-b my-a)
        "},
    );
}

produce a non-deterministic vm_error description. Here the possible output:

detected interdependent functions (my-a, my-b)
detected interdependent functions (my-b, my-a)

where the element listed in the paranthesis can have a different order and the flip/flop between them at each run.

codecov · 2025-10-30T11:19:31Z

Codecov Report

❌ Patch coverage is 87.39496% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.38%. Comparing base (f8d02cd) to head (298959a).

Files with missing lines	Patch %	Lines
stackslib/src/chainstate/tests/parse_tests.rs	85.98%	15 Missing ⚠️

❌ Your project check has failed because the head coverage (74.38%) is below the target coverage (80.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #6631      +/-   ##
===========================================
+ Coverage    64.85%   74.38%   +9.52%     
===========================================
  Files          574      575       +1     
  Lines       355039   355145     +106     
===========================================
+ Hits        230278   264188   +33910     
+ Misses      124761    90957   -33804

Files with missing lines	Coverage Δ
stackslib/src/chainstate/tests/consensus.rs	`95.00% <100.00%> (+9.78%)`	⬆️
stackslib/src/chainstate/tests/mod.rs	`70.08% <ø> (ø)`
stackslib/src/chainstate/tests/parse_tests.rs	`85.98% <85.98%> (ø)`

... and 373 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f8d02cd...298959a. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

francesco-stacks

Good changes! Definitely fine with the named tests.

PS:I am sure you already noticed, but some unit tests are currently failing

francesco-stacks · 2025-10-30T12:09:29Z

stackslib/src/chainstate/tests/parse_tests.rs

+use clarity::vm::ast::stack_depth_checker::AST_CALL_STACK_DEPTH_BUFFER;
+use clarity::vm::MAX_CALL_STACK_DEPTH;
+#[allow(unused_imports)] // Just used for documentation purpose
+use clarity_types::{errors::ParseErrors, ContractName};


nit: maybe we can directly import it from use clarity::vm::ast::errors::ParseErrors and avoid adding clarity_types directly?

you're right! the IDE didn't show auto-completition option to gather these symbols from clarity crate. Damn it :)

crc: 2ac068b

francesco-stacks · 2025-10-30T12:23:42Z

...tests/snapshots/blockstack_lib__chainstate__tests__parse_tests__trait_reference_unknown.snap

+    transactions: [
+      ExpectedTransactionOutput(
+        tx: "SmartContract(name: my-contract-Epoch3_3-Clarity4, code_body: [..], clarity_version: Some(Clarity4))",
+        vm_error: "Some(use of undeclared trait <my-trait>) [NON-CONSENSUS BREAKING]",


damn, that's a pity that vm_error contains the "message" of the error and not the error variant itself

francesco-stacks · 2025-10-30T12:40:59Z

stackslib/src/chainstate/tests/parse_tests.rs

+        contract_code: &{"
+            (define-data-var my-int int 340282366920938463463374607431768211455)   
+        "},


nit, you can remove &{ in case you pass a string slice

Suggested change

contract_code: &{"

(define-data-var my-int int 340282366920938463463374607431768211455)

"},

contract_code: "(define-data-var my-int int 340282366920938463463374607431768211455)",

even multiline strings will work fine

crc: 730fc6d

stackslib/src/chainstate/tests/parse_tests.rs

federico-stacks · 2025-10-30T16:54:11Z

Good changes! Definitely fine with the named tests.

PS:I am sure you already noticed, but some unit tests are currently failing

Yeah! This is due the vm_error for the test_circular_reference being not deterministic as mentioned in this comment: #6631 (comment)

…tacks-network#6627

…twork#6627

federico-stacks · 2025-10-31T12:17:19Z

By the way, adding more tests exposed a new issue related to the vm_error field, basically this test:
#[test]
fn test_circular_reference() {
    contract_deploy_consensus_test!(
        contract_name: "my-contract",
        contract_code: &{"
            (define-constant my-a my-b)
            (define-constant my-b my-a)
        "},
    );
}
produce a non-deterministic vm_error description. Here the possible output:

detected interdependent functions (my-a, my-b)

detected interdependent functions (my-b, my-a)

where the element listed in the paranthesis can have a different order and the flip/flop between them at each run.

I opted to manage this issue, forcing the CircularRefercence being emitted with a determinitic list.
see commit: a81db5d

federico-stacks added 3 commits October 28, 2025 10:53

chore: make aac macros usable from other modules, stacks-network#6627

0db522b

chore: aac add clarity_types as test dependency for doc purpose, stac…

1eead86

…ks-network#6627

test: aac add consensus coverage for ParseError variants subset (11), s…

c01f81e

…tacks-network#6627

federico-stacks self-assigned this Oct 28, 2025

federico-stacks added aac Avoiding Accidental Consensus aac-testing Avoiding Accidental Consensus Testing Specific Task labels Oct 28, 2025

federico-stacks linked an issue Oct 28, 2025 that may be closed by this pull request

AAC Testing: Add test coverage for ParseError enum #6627

Open

federico-stacks requested review from francesco-stacks and jacinta-stacks October 28, 2025 10:36

jacinta-stacks reviewed Oct 28, 2025

View reviewed changes

stackslib/src/chainstate/tests/consensus.rs Outdated Show resolved Hide resolved

jacinta-stacks reviewed Oct 28, 2025

View reviewed changes

federico-stacks added 2 commits October 30, 2025 10:29

refactor: aac consensus macro just produce a test body, stacks-networ…

fe6a8d5

…k#6627

merge: address conflict with develop, stacks-network#6627

298959a

francesco-stacks reviewed Oct 30, 2025

View reviewed changes

federico-stacks added 3 commits October 30, 2025 18:05

crc: remove clarity_types dependency and use clarity crate aliases, s…

2ac068b

…tacks-network#6627

crc: clean contract code string literal, stacks-network#6627

730fc6d

chore: aac fix ParseError::CircularReference indeterminism, stacks-ne…

a81db5d

…twork#6627

federico-stacks force-pushed the chore/aac-parse-error-test branch from 27b1d56 to a81db5d Compare October 31, 2025 12:10

francesco-stacks mentioned this pull request Oct 31, 2025

chore: add consensus tests for MemoryBalanceExceeded #6642

Open

4 tasks

test: aac add test coverage for ParseError variants #6631

Are you sure you want to change the base?

test: aac add test coverage for ParseError variants #6631

Conversation

federico-stacks commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Applicable issues

Additional info (benefits, drawbacks, caveats)

Checklist

Uh oh!

Uh oh!

jacinta-stacks Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

jacinta-stacks commented Oct 28, 2025

Uh oh!

federico-stacks commented Oct 28, 2025

Uh oh!

federico-stacks commented Oct 30, 2025

Uh oh!

codecov bot commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

francesco-stacks left a comment

Choose a reason for hiding this comment

Uh oh!

francesco-stacks Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

federico-stacks Oct 31, 2025

Choose a reason for hiding this comment

Uh oh!

francesco-stacks Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

francesco-stacks Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

federico-stacks Oct 31, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

federico-stacks commented Oct 30, 2025

Uh oh!

federico-stacks commented Oct 31, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

test: aac add test coverage for `ParseError` variants #6631

test: aac add test coverage for `ParseError` variants #6631

federico-stacks commented Oct 28, 2025 •

edited

Loading

codecov bot commented Oct 30, 2025 •

edited

Loading