Flaky tests

[cdietrich]

the number of flaky tests, with latests targets, seems to be more flaky recently. we need to figure out whats happening (or disabling the tests ?!?)

[LorenzoBettini]

In the past, I did my best to put as many "synchronization" points as possible.
That helped a lot, but not completely.
I would avoid re-adding the rerun count in Maven, as that could make the reports harder to inspect, and I don't think it's yet supported by the GitHub Actions we use.

What we could do is take note of them and use our @Flaky annotations.

I gave up trying to synchronize things because it was really hard to get it right; in particular, it's really hard to understand the cause of the problem.

[cdietrich]

org.eclipse.xtext.common.types.access.xtext.ui.JdtBasedConstructorScopeTest.testGetContents_03
org.eclipse.xtext.common.types.access.xtext.ui.JdtBasedConstructorScopeTest.testGetContents_04
org.eclipse.xtext.common.types.access.xtext.ui.JdtBasedSimpleTypeScopeTest.testGetContents_01
org.eclipse.xtext.common.types.access.xtext.ui.JdtBasedSimpleTypeScopeTest.testGetContents_03
org.eclipse.xtext.common.types.xtext.ui.ContentAssistTest.testSubtypeBlockingQueue_01
org.eclipse.xtext.common.types.xtext.ui.ContentAssistTest.testSubtypeBlockingQueue_02
org.eclipse.xtext.common.types.xtext.ui.ContentAssistTest.testSubtypeBlockingQueue_03
org.eclipse.xtext.common.types.xtext.ui.ContentAssistTest.testCustomArrayList_01
org.eclipse.xtext.common.types.xtext.ui.ContentAssistTest.testCustomArrayList_02
org.eclipse.xtext.common.types.xtext.ui.ContentAssistTest.testCustomArrayList_03
org.eclipse.xtext.common.types.xtext.ui.ContentAssistTest.testSubtypeHashSet_01
org.eclipse.xtext.common.types.xtext.ui.ContentAssistTest.testSubtypeHashSet_02
org.eclipse.xtext.common.types.xtext.ui.ContentAssistTest.testSubtypeArrayList_01
org.eclipse.xtext.common.types.xtext.ui.ContentAssistTest.testSubtypeArrayList_02
org.eclipse.xtext.common.types.xtext.ui.ContentAssistTest.testDefaultBlockingQueue_01
org.eclipse.xtext.common.types.xtext.ui.ContentAssistTest.testDefaultBlockingQueue_02
org.eclipse.xtext.common.types.xtext.ui.ContentAssistTest.testDefaultArrayList_01
org.eclipse.xtext.common.types.xtext.ui.ContentAssistTest.testDefaultArrayList_02
org.eclipse.xtext.common.types.xtext.ui.ContentAssistTest.testMap_01
org.eclipse.xtext.common.types.xtext.ui.ContentAssistTest.testMap_02
org.eclipse.xtext.common.types.xtext.ui.ContentAssistTest.testMap_03
org.eclipse.xtext.common.types.xtext.ui.ContentAssistTest.testCustomBlockingQueue_01
org.eclipse.xtext.common.types.xtext.ui.ContentAssistTest.testCustomBlockingQueue_02
org.eclipse.xtext.common.types.xtext.ui.ContentAssistTest.testCustomBlockingQueue_03

[cdietrich]

@LorenzoBettini from your past changes. maybe we need to spread some more
IResourcesSetupUtil.waitForJdtIndex(); ?

[LorenzoBettini]

@LorenzoBettini from your past changes. maybe we need to spread some more IResourcesSetupUtil.waitForJdtIndex(); ?

I'm quite sure I did not add a wait for JDT index in tests that was not based on Xbase.
Instead, as you said yesterday, some content assist implementations still use the JDT index even if their language is not based on Xbase; so that might help.
If those tests are based on our waitForBuild, though, I seem to remember that that method already waits for JDT index.

[cdietrich]

Failures in the Windows build are:

org.eclipse.xtext.xtext.ui.editor.contentassist.XtextContentAssistTest.testCompletionOnDatatypeReference_01 -- Time elapsed: 0.070 s <<< FAILURE!
org.junit.ComparisonFailure:
expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>
at org.junit.Assert.assertEquals(Assert.java:117)
at org.junit.Assert.assertEquals(Assert.java:146)
at org.eclipse.xtext.ui.testing.ContentAssistProcessorTestBuilder.assertTextAtCursorPosition(ContentAssistProcessorTestBuilder.java:357)
at org.eclipse.xtext.ui.testing.ContentAssistProcessorTestBuilder.assertTextAtCursorPosition(ContentAssistProcessorTestBuilder.java:335)
at org.eclipse.xtext.xtext.ui.editor.contentassist.XtextContentAssistTest.testCompletionOnDatatypeReference_01(XtextContentAssistTest.java:129)

org.eclipse.xtext.xtext.ui.editor.contentassist.XtextContentAssistTest.testCompletionOnDatatypeReference_02 -- Time elapsed: 0.050 s <<< FAILURE!
org.junit.ComparisonFailure:
expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>
at org.junit.Assert.assertEquals(Assert.java:117)
at org.junit.Assert.assertEquals(Assert.java:146)
at org.eclipse.xtext.ui.testing.ContentAssistProcessorTestBuilder.assertTextAtCursorPosition(ContentAssistProcessorTestBuilder.java:357)
at org.eclipse.xtext.ui.testing.ContentAssistProcessorTestBuilder.assertTextAtCursorPosition(ContentAssistProcessorTestBuilder.java:335)
at org.eclipse.xtext.xtext.ui.editor.contentassist.XtextContentAssistTest.testCompletionOnDatatypeReference_02(XtextContentAssistTest.java:140)

org.eclipse.xtext.xtext.ui.editor.contentassist.XtextContentAssistTest.testCompletionOnDatatypeReference_03 -- Time elapsed: 0.065 s <<< FAILURE!
org.junit.ComparisonFailure:
expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>
at org.junit.Assert.assertEquals(Assert.java:117)
at org.junit.Assert.assertEquals(Assert.java:146)
at org.eclipse.xtext.ui.testing.ContentAssistProcessorTestBuilder.assertTextAtCursorPosition(ContentAssistProcessorTestBuilder.java:357)
at org.eclipse.xtext.ui.testing.ContentAssistProcessorTestBuilder.assertTextAtCursorPosition(ContentAssistProcessorTestBuilder.java:335)
at org.eclipse.xtext.xtext.ui.editor.contentassist.XtextContentAssistTest.testCompletionOnDatatypeReference_03(XtextContentAssistTest.java:148)

org.eclipse.xtext.xtext.ui.editor.contentassist.XtextContentAssistTest.testCompletionOnDatatypeReference_04 -- Time elapsed: 0.066 s <<< FAILURE!
java.lang.AssertionError: No such proposal: Terminals - org.eclipse.xtext.common.Terminals Found: [@, @deprecated, @exported, @Final, @OverRide]
at org.junit.Assert.fail(Assert.java:89)
at org.eclipse.xtext.ui.testing.ContentAssistProcessorTestBuilder.assertProposalDisplayedAtCursor(ContentAssistProcessorTestBuilder.java:303)
at org.eclipse.xtext.xtext.ui.editor.contentassist.XtextContentAssistTest.testCompletionOnDatatypeReference_04(XtextContentAssistTest.java:154)

org.eclipse.xtext.xtext.ui.editor.contentassist.XtextContentAssistTest.testBug463215_01 -- Time elapsed: 0.048 s <<< FAILURE!
org.junit.ComparisonFailure: expected:<[org.eclipse.xtext.common.Terminals]> but was:<[]>
at org.junit.Assert.assertEquals(Assert.java:117)
at org.junit.Assert.assertEquals(Assert.java:146)
at org.eclipse.xtext.ui.testing.ContentAssistProcessorTestBuilder.assertTextAtCursorPosition(ContentAssistProcessorTestBuilder.java:357)
at org.eclipse.xtext.ui.testing.ContentAssistProcessorTestBuilder.assertTextAtCursorPosition(ContentAssistProcessorTestBuilder.java:335)
at org.eclipse.xtext.xtext.ui.editor.contentassist.XtextContentAssistTest.testBug463215_01(XtextContentAssistTest.java:107)

org.eclipse.xtext.xtext.ui.editor.contentassist.XtextContentAssistTest.testBug463215_02 -- Time elapsed: 0.076 s <<< FAILURE!
org.junit.ComparisonFailure:
expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>
at org.junit.Assert.assertEquals(Assert.java:117)
at org.junit.Assert.assertEquals(Assert.java:146)
at org.eclipse.xtext.ui.testing.ContentAssistProcessorTestBuilder.assertTextAtCursorPosition(ContentAssistProcessorTestBuilder.java:357)
at org.eclipse.xtext.ui.testing.ContentAssistProcessorTestBuilder.assertTextAtCursorPosition(ContentAssistProcessorTestBuilder.java:335)
at org.eclipse.xtext.xtext.ui.editor.contentassist.XtextContentAssistTest.testBug463215_02(XtextContentAssistTest.java:118)

[cdietrich]

@LorenzoBettini XtextContentAssistTest also seems to have a waitForBuild only....

[LorenzoBettini]

@LorenzoBettini XtextContentAssistTest also seems to have a waitForBuild only....

But our waitForBuild in resource utils internally calls wait for java index.

[cdietrich]

so we have no clue what the problem is

[cdietrich]

at least i can reproduce by running test in a big suite of itself cd-grammar-rep

wiredly it fails for all suites from2nd
so mabye there is a improper clear/testorder problem

when nulling project it does not happen
so i wonder if there is sth that does not proper setup project in some cases

[cdietrich]

argg there also seem to be a thread leak
[331.164s][warning][os,thread] Failed to start the native thread for java.lang.Thread "pool-4866-thread-1"
[331.266s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 2048k, guardsize: 16k, detached.
[331.266s][warning][os,thread] Failed to start the native thread for java.lang.Thread "pool-4867-thread-1"
[331.351s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 2048k, guardsize: 16k, detached.
[331.351s][warning][os,thread] Failed to start the native thread for java.lang.Thread "pool-4868-thread-1"

[LorenzoBettini]

I seem to understand that nulling project means it builds the workspace before each test, right? So I'd try to add a wait for build before each test.

When you say you reproduce it locally, do you mean in windows?

Maybe we're running out of threads because we don't wait for existing threads finishing their jobs?

[cdietrich]

no i mean mac
in that case run 12 of 21 failed