[AUTOCUT] Gradle Check Flaky Test Report for IngestFromKafkaIT

[opensearch-ci-bot]

Flaky Test Report for IngestFromKafkaIT

Noticed the IngestFromKafkaIT has some flaky, failing tests that failed during post-merge actions and timer trigerred run on main branch.

Details

Git Reference	Merged Pull Request	Build Details	Test Name
main	null	33 62752 62902 63631 64397 66445 67850 69104 69182 69208 69571 73021 73130 74937 75061	org.opensearch.plugin.kafka.IngestFromKafkaIT.testAllActiveOffsetBasedLag org.opensearch.plugin.kafka.IngestFromKafkaIT.testKafkaIngestion_RewindByOffset org.opensearch.plugin.kafka.IngestFromKafkaIT.testAllActiveIngestion org.opensearch.plugin.kafka.IngestFromKafkaIT.testUpdateWithoutIDField org.opensearch.plugin.kafka.IngestFromKafkaIT.testAllActiveIngestionBatchStartPointerOnReplicaPromotion org.opensearch.plugin.kafka.IngestFromKafkaIT.testDynamicConfigUpdateOnNoMessages org.opensearch.plugin.kafka.IngestFromKafkaIT.testKafkaIngestion org.opensearch.plugin.kafka.IngestFromKafkaIT.testMessageOperationTypes
8ffa576	20940	73372	org.opensearch.plugin.kafka.IngestFromKafkaIT.testDynamicUpdateKafkaParams
08ed9ee	19480	64831	org.opensearch.plugin.kafka.IngestFromKafkaIT.testUpdateWithoutIDField
b9c5bc7	19184	63789	org.opensearch.plugin.kafka.IngestFromKafkaIT.testCloseIndex
e25f332	18031	56911	org.opensearch.plugin.kafka.IngestFromKafkaIT.testCloseIndex
c472314	17640	54929	org.opensearch.plugin.kafka.IngestFromKafkaIT.testCloseIndex
852011a	16021	53136	org.opensearch.plugin.kafka.IngestFromKafkaIT.classMethod
20a58eb	17194	52946	org.opensearch.plugin.kafka.IngestFromKafkaIT.classMethod
d601c37	17121	52942	org.opensearch.plugin.kafka.IngestFromKafkaIT.classMethod

The other pull requests, besides those involved in post-merge actions, that contain failing tests with the IngestFromKafkaIT class are:

• 16958
• 18333
• 17631
• 20526
• 18332
• 18693
• 17752
• 17240
• 17241
• 17638
• 18249
• 19046
• 19320
• 19635
• 20858
• 14912
• 17101
• 17132
• 17252
• 17273
• 17359
• 17365
• 17538
• 17565
• 17575
• 17771
• 17922
• 17925
• 18036
• 18106
• 18193
• 18255
• 18451
• 18454
• 18509
• 18530
• 18683
• 18827
• 19009
• 19335
• 19489
• 19540
• 19769
• 19878
• 20816
• 20874
• 20929
• 20936
• 20949
• 20982
• 20987
• 21047
• 21055
• 21128
• 21145
• 21178
• 21183
• 21245
• 21253
• 21270
• 21289

For more details on the failed tests refer to OpenSearch Gradle Check Metrics dashboard.

[andrross]

@yupeng9 Unfortunately it looks like there are still cases where the thread leak failure happens :(

[yupeng9]

ack, can take on this issue and investigate

[andrross]

Another failure here: https://build.ci.opensearch.org/job/gradle-check/56900/consoleText

REPRODUCE WITH: ./gradlew ':plugins:ingestion-kafka:internalClusterTest' --tests "org.opensearch.plugin.kafka.IngestFromKafkaIT.testPluginsAreInstalled" -Dtests.seed=5D3C7ADEF6C7F804 -Dtests.security.manager=false -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=uz-Arab -Dtests.timezone=Africa/Timbuktu -Druntime.java=21

IngestFromKafkaIT > testPluginsAreInstalled FAILED
    org.awaitility.core.ConditionTimeoutException: Condition org.opensearch.plugin.kafka.KafkaUtils$$Lambda/0x000071e77c743558 was not fulfilled within 3 seconds.
        at __randomizedtesting.SeedInfo.seed([5D3C7ADEF6C7F804:2A88A93E0A1AC475]:0)
        at app//org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
        at app//org.awaitility.core.CallableCondition.await(CallableCondition.java:78)
        at app//org.awaitility.core.CallableCondition.await(CallableCondition.java:26)
        at app//org.awaitility.core.ConditionFactory.until(ConditionFactory.java:985)
        at app//org.awaitility.core.ConditionFactory.until(ConditionFactory.java:954)
        at app//org.opensearch.plugin.kafka.KafkaUtils.createTopic(KafkaUtils.java:56)
        at app//org.opensearch.plugin.kafka.KafkaIngestionBaseIT.setupKafka(KafkaIngestionBaseIT.java:79)
        at app//org.opensearch.plugin.kafka.KafkaIngestionBaseIT.setup(KafkaIngestionBaseIT.java:63)
        at java.****@21.0.6/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.****@21.0.6/java.lang.reflect.Method.invoke(Method.java:580)
        at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:972)
        at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at app//org.opensearch.test.OpenSearchTestClusterRule$1.evaluate(OpenSearchTestClusterRule.java:369)
        at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
        at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
        at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.****@21.0.6/java.lang.Thread.run(Thread.java:1583)

        Caused by:
        java.util.concurrent.TimeoutException
            at java.****/java.util.concurrent.FutureTask.get(FutureTask.java:204)
            at org.awaitility.core.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:101)
            at org.awaitility.core.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:81)
            at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:103)
            ... 45 more