[BUG] Performance Analyzer fails to start in OpenSearch 2.19.2+ and 3.0.0+ due to missing RCA dependencies

[DdangJin]

Describe the bug

Performance Analyzer fails to start in OpenSearch versions 2.19.2 and above, as well as in 3.x versions.
(Note: It works correctly in version 2.19.1).

The root cause appears to be that the startup script attempts to execute org.opensearch.performanceanalyzer.PerformanceAnalyzerApp, which relies on the RCA (Root Cause Analysis) library. However, the RCA artifacts seem to be missing or unloaded in these specific versions.

Specific Observations:

1. Regression in 2.19.2+:

   • Performance Analyzer works fine in 2.19.1.
   • In 2.19.2, the RCA components appear to be missing, causing the crash.
     • OPENSEARCH_HOME/performance-analyzer-rca/ directory (which contains PerformanceAnalyzerApp) exists in version 2.19.1 but is completely missing in version 2.19.2.
   • According to Pull Request opensearch-project/OpenSearch#598 (Comment), RCA was intended to be deprecated in 2.19 and removed in 3.0. Therefore, it should still be present and functional in the 2.19.x series (including 2.19.2).

2. Broken in 3.x:

   • In version 3.0 and above, RCA is indeed removed as planned. However, the Performance Analyzer application still attempts to load the RCA class (PerformanceAnalyzerApp), leading to a ClassNotFoundException. The startup logic has not been updated to handle this removal.

3. Unclear Migration Path:

   • While RCA is deprecated (see Issue opensearch-project/OpenSearch#591 and Issue opensearch-project/OpenSearch#585) in favor of an OpenTelemetry (OTel) based approach, there is currently no clear documentation or configuration method available to enable this new OTel-based transmission.

4. Documentation & Config Issues:

   • Documentation for version 3.3.2 includes examples using port 9600, but this is impossible to reproduce as the service cannot be started.
     • https://docs.opensearch.org/latest/monitoring-your-cluster/pa/index/
   • Starting from version 3.0.0, the opensearch-performance-analyzer directory is not automatically created in the OPENSEARCH_CONFIG_PATH.
     • releated: [BUG] File Not Found Exception for opensearch-performance-analyzer plugin. #857 (comment)

Related component

Plugins

To Reproduce

1. Download OpenSearch distribution for version 2.19.2 (or later 2.x) OR 3.0.0 (or later).
2. Attempt to start the Performance Analyzer.
3. Observe the following error in the logs:

performance-analyzer-agent-cli: Error: Could not find or load main class org.opensearch.performanceanalyzer.PerformanceAnalyzerApp
performance-analyzer-agent-cli: Caused by: java.lang.ClassNotFoundException: org.opensearch.performanceanalyzer.PerformanceAnalyzerApp

Expected behavior

• For 2.19.x (specifically 2.19.2+): The RCA components should still be present (as they are only deprecated), and Performance Analyzer should start correctly, just as it does in 2.19.1.
• For 3.x: Since RCA is removed, the startup mechanism should be updated to function without invoking the RCA classes.
• OTel Transition: If the architecture has shifted to an OpenTelemetry-based collector, the distribution should reflect this change, and the Performance Analyzer should be able to start using the new mechanism without errors.

Additional Details

Plugins

• opensearch-performance-analyzer

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: Ubuntu 24.04
• Version: 2.19.2-4, 3.0.0-3.3.2

Additional context
Add any other context about the problem here.

• Related issues reporting similar behavior:
  • [BUG] FileNotFoundException When OpenSearch Starts OpenSearch#19147
  • [BUG] Performance Analyzer doesn't work in 3.2.0 docker container OpenSearch#19983
  • Missing performance-analyzer.properties in OpenSearch 3.1.0 clean install #832
  • [BUG] File Not Found Exception for opensearch-performance-analyzer plugin. #857
  • [BUG] performance-analyzer still requires performance-analyzer-rca which is deprecated. #884

[andrross]

@opensearch-project/admin Can this be transferred to the performance analyzer repo?

[krisfreedain]

@khushbr @ansjcy @Gaganjuneja & maintainers - can you take a look?

Catch All Triage - 1 2