mark tests as flaky

[cdietrich]

see #3644

[cdietrich]

@LorenzoBettini an not sure if this will help if the problem might be in the
@BeforeClass

[github-actions]

Test Results

  8 076 files  + 4    8 076 suites  +4   3h 46m 12s ⏱️ - 22m 40s
 43 241 tests ± 0   42 651 ✅  -  5    584 💤 ±0  6 ❌ +5 
212 508 runs  +23  209 581 ✅ +15  2 921 💤 +3  6 ❌ +5 

For more details on these failures, see this check.

Results for commit dedcd6b. ± Comparison against base commit 088392b.

♻️ This comment has been updated with latest results.

[LorenzoBettini]

@LorenzoBettini an not sure if this will help if the problem might be in the @BeforeClass

If it's in @BeforeClass I think our @Flaky annotation would not help since it won't re-execute that method, is that right @szarnekow ?

Maybe, instead, rerun count of surefire also keeps that into consideration...

But the method with @BeforeClass doesn't throw or anything, does it?

[cdietrich]

i just saw we still have the issue here
https://ci.eclipse.org/xtext/job/xtext/job/cd-mark-flaky/

[LorenzoBettini]

This https://github.com/eclipse-xtext/xtext/blob/main/org.eclipse.xtext.common.types.eclipse.tests/tests/org/eclipse/xtext/common/types/access/xtext/AbstractConstructorScopeTest.java#L37 already performs a waitForBuild, which in turns wait for the JDT index.

However, https://github.com/eclipse-xtext/xtext/blob/main/org.eclipse.xtext.common.types.eclipse.tests/tests/org/eclipse/xtext/common/types/access/jdt/MockJavaProjectProvider.java#L69 shows that the waitForBuild is done only once, when creating the project.

In wonder whether a @before always performing it might help.

Moreover, there are two versions of AbstractConstructorScopeTest.
This one does a setup https://github.com/eclipse-xtext/xtext/blob/main/org.eclipse.xtext.common.types.eclipse.tests/tests/org/eclipse/xtext/common/types/access/xtext/AbstractConstructorScopeTest.java#L37
This one doesn't https://github.com/eclipse-xtext/xtext/blob/main/org.eclipse.xtext.common.types.tests/tests/org/eclipse/xtext/common/types/access/xtext/AbstractConstructorScopeTest.java#L36

[cdietrich]

my assumtion is the 2nd runs without eclipse