From 8c268b5dc91be38ba6a0665865af716a61862547 Mon Sep 17 00:00:00 2001 From: Neil Maartens Date: Fri, 24 Oct 2025 14:01:51 -0500 Subject: [PATCH] significant updates to content/layout and bulleting to support clean HTML conversion of content and to align with writing standards. --- README.md | 4 +- .../analytics/caat_environment_watch_setup.md | 67 +- docs/create_kibana_users.md | 17 +- docs/datagrid_audit_setup.md | 87 +- ...elasticsearch_pre_installation_overview.md | 10 +- docs/elasticsearch_setup_development.md | 803 ++++++++---------- docs/elasticsearch_setup_production.md | 2 +- .../post-install-verification.md | 15 +- .../alert-overview.md | 77 +- .../elasticsearch-cluster-health.md | 59 +- .../monitoring-agents.md | 94 +- docs/environment_watch_product_overview.md | 10 +- docs/environment_watch_troubleshooting.md | 24 +- docs/environment_watch_upgrade.md | 166 ++-- ...all_environment_watch_monitoring_agents.md | 89 +- docs/rabbitmq/rabbitmq_integration.md | 72 +- docs/relativity_alerts_installation.md | 23 +- docs/relativity_server_cli_setup.md | 147 ++-- docs/troubleshooting/apm-server.md | 259 +++--- docs/troubleshooting/elasticsearch.md | 323 +++---- docs/troubleshooting/kibana.md | 388 ++++----- .../monitoring-agent-and-otel-collector.md | 381 ++++----- .../pre-requisite-troubleshooting.md | 342 ++++---- docs/troubleshooting/relativity-server-cli.md | 118 ++- .../relativity_alerts_troubleshooting.md | 44 +- 25 files changed, 1564 insertions(+), 2057 deletions(-) diff --git a/README.md b/README.md index ee99c37..c8866a5 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ This release bundle contains packages that are required to set up or upgrade two | --------------------------------- | --------------------------------------------------------- | | Relativity Server | Server 2024 Patch 2 | | Elastic Stack | 8.17.3 is required for Environment Watch.
7.17.x can be used but for Data Grid Audit only. | -| Network Ports | Ensure required ports are open for Elastic Stack communication. [Refer to the port diagram](/docs/environment-watch/port-diagram.md) for full network configuration details. | +| Network Ports | Ensure required ports are open for Elastic Stack communication. [Refer to the port diagram](docs/environment-watch/port-diagram.md) for full network configuration details. | ## Getting Started @@ -24,4 +24,4 @@ This release bundle contains packages that are required to set up or upgrade two - **Upgrade**: [Click here for upgrade instructions](docs/environment_watch_upgrade.md). -- **Troubleshooting**: [Click here for the troubleshooting guide](/docs/environment_watch_troubleshooting.md). +- **Troubleshooting**: [Click here for the troubleshooting guide](docs/environment_watch_troubleshooting.md). diff --git a/docs/analytics/caat_environment_watch_setup.md b/docs/analytics/caat_environment_watch_setup.md index dc3afe4..6e4dfaa 100644 --- a/docs/analytics/caat_environment_watch_setup.md +++ b/docs/analytics/caat_environment_watch_setup.md @@ -3,62 +3,47 @@ This guide provides step-by-step instructions for installing/updating Relativity Analytics Engine (CAAT) along with the OpenTelemetry Java Agent. This enables distributed tracing and observability with minimal code changes. -For other Other integrations, refer to the [Environment Watch Install other Integrations Guide](environment_watch_install_other_integrations.md). +For other Other integrations, refer to the [Environment Watch Install other Integrations Guide](../environment_watch_install_other_integrations.md). ---- - -## 1. Why Use `opentelemetry-javaagent`? +## Why use opentelemetry-javaagent? - **Automatic Instrumentation:** The OpenTelemetry Java Agent automatically instruments supported libraries and frameworks, capturing telemetry data (traces, metrics, logs) without code changes. - **Observability:** It provides deep visibility into application performance, dependencies, and bottlenecks. - **Standardization:** OpenTelemetry is a vendor-neutral standard, ensuring compatibility with various observability backends (e.g., Azure Monitor, Jaeger, Zipkin). ---- +## Why Auto-Instrumentation is safe -## 2. How to install/update CAAT +- **Non-Intrusive:** The agent works by attaching to the JVM at startup, using bytecode manipulation. It does not require code changes or recompilation. +- **Widely Adopted:** OpenTelemetry Java Agent is used in production by many organizations and is actively maintained. +- **Configurable:** Instrumentation can be enabled/disabled for specific libraries, and the agent can be removed by simply omitting the `-javaagent` flag. +- **No Persistent Changes:** The agent does not persistently modify your application or its dependencies. - 1. Copy CAAT Bundle to your server - 2. Unzip the folder - 3. Replace response-file.properties with master copy - 4. Open Powershell as an administrator - 5. Run .\Install.cmd - 6. Once the installation is done, start “Relativity Analytics Engine” and check if the page is loading. - 7. Open ElasticSearch and search for service.name: “relsvr_caat” in metrics-* data view +## How to install/update CAAT +1. Copy CAAT Bundle to your server +2. Unzip the folder +3. Replace response-file.properties with master copy +4. Open Powershell as an administrator +5. Run `.\Install.cmd` +6. Once the installation is done, start **Relativity Analytics Engine** and check if the page is loading. +7. Open ElasticSearch and search for service.name: “relsvr_caat” in metrics-* data view ---- -## 3. What's updated +## What's updated -- Startup.cmd file is updated to include the OpenTelemetry Java Agent. This references the `opentelemetry-javaagent.jar` file, which is used to instrument the CAAT service for telemetry data collection. +- The Startup.cmd file is updated to include the OpenTelemetry Java Agent. This references the `opentelemetry-javaagent.jar` file, which is used to instrument the CAAT service for telemetry data collection. - Check for these lines within `startup.cmd`: -``` --javaagent:C:\\CAAT\\bin\\opentelemetry-javaagent.jar --Dotel.service.name="relsvr.caat" --Dotel.metrics.exporter=otlp --Dotel.exporter.otlp.endpoint="http://localhost:4318" --Dotel.instrumentation.runtime-metrics.enabled=true -``` ---- - -## 4. Why Auto-Instrumentation is Safe - -- **Non-Intrusive:** The agent works by attaching to the JVM at startup, using bytecode manipulation. It does not require code changes or recompilation. -- **Widely Adopted:** OpenTelemetry Java Agent is used in production by many organizations and is actively maintained. -- **Configurable:** Instrumentation can be enabled/disabled for specific libraries, and the agent can be removed by simply omitting the `-javaagent` flag. -- **No Persistent Changes:** The agent does not persistently modify your application or its dependencies. + ``` + -javaagent:C:\\CAAT\\bin\\opentelemetry-javaagent.jar + -Dotel.service.name="relsvr.caat" + -Dotel.metrics.exporter=otlp + -Dotel.exporter.otlp.endpoint="http://localhost:4318" + -Dotel.instrumentation.runtime-metrics.enabled=true + ``` ---- - -## 5. How to Verify the Changes +## How to verify the changes 1. **Check Application Logs:** On startup, the agent logs its initialization. Look for lines mentioning `opentelemetry-javaagent` in the CAAT logs. 2. **Verify Telemetry Export:** Confirm that traces and metrics are being sent to your configured backend (e.g., OpenTelemetry Collector, Jaeger, Azure Monitor). 3. **Check Service Name:** Ensure the service appears as `caat-analytics-engine` (or your configured name) in your observability backend. -4. **Disable to Compare:** Temporarily remove the `-javaagent` flag and confirm that telemetry stops, verifying the agent is responsible for the data. - ---- - -> [!NOTE] -> This documentation uses the active document as context because you have the checkmark checked. -> You can include additional context using **#** references. Typing **#** opens a completion list of available context. +4. **Disable to Compare:** Temporarily remove the `-javaagent` flag and confirm that telemetry stops, verifying the agent is responsible for the data. \ No newline at end of file diff --git a/docs/create_kibana_users.md b/docs/create_kibana_users.md index 114eaa4..a0c3d4c 100644 --- a/docs/create_kibana_users.md +++ b/docs/create_kibana_users.md @@ -1,4 +1,4 @@ -# Step 7: Create Kibana Users and Assign Roles +# Create Kibana Users and Assign Roles ![Setup stage](../resources/createuserssetupstage.png) @@ -13,18 +13,17 @@ To create a Kibana user and assign the custom Kibana role: 2. Navigate to **Stack Management** > **Security** > **Users**. 3. Click **Create user**. 4. Fill out the following: -- **Username**: A unique login name (e.g., `alerts_dashboard_user`).
-- **Password**: Set a strong password.
-- **Full name / Email address**: Optional but recommended.
+ - **Username**: A unique login name (e.g., `alerts_dashboard_user`).
+ - **Password**: Set a strong password.
+ - **Full name / Email address**: Optional but recommended.
5. Under **Roles**, search for and assign the `relativity_dashboard_user` role. 6. Click **Create user** to save. ![](../resources/custom_kibana_role.png) - -> [!WARNING] -> **Important:** Users must log in with this account to access the Kibana dashboards and saved searches provided by Environment Watch. ---- +> [!WARNING] +> Users must log in with this account to access the Kibana dashboards and saved searches provided by Environment Watch. + ### Congratulations! -You have reached the end of the setup process. If you are still running into any issues, please refer to the [Troubleshooting Guides](/docs/environment_watch_troubleshooting.md) for further assistance. +You have reached the end of the setup process. If you are still running into any issues, please refer to the [Troubleshooting Guides](environment_watch_troubleshooting.md) for further assistance. diff --git a/docs/datagrid_audit_setup.md b/docs/datagrid_audit_setup.md index e20f008..935472e 100644 --- a/docs/datagrid_audit_setup.md +++ b/docs/datagrid_audit_setup.md @@ -3,17 +3,18 @@ ![Setup Stage](../resources/enable_environmentwatch.png) > [!NOTE] -> This section applies to Datagrid Only and the Audit setup via CLI should be run only once per environment. +> This section applies to Datagrid Only. + +After installing the required Elastic components for Data Grid Audit, the integration between Elastic and Relativity is configured by running the Relativity Server CLI on the Primary SQL Server. + + > Please review the following important information before proceeding: > * **For Existing Data Grid Audit Customers:** You must be on Elasticsearch 7.17 when initially running this setup. After the setup is complete, you can upgrade to Elasticsearch 8.17.3. > * Before upgrading to Elasticsearch 8.17.3, the `ESIndexCreationSetting` may need to be updated. For details, refer to the [Instance setting Details](https://help.relativity.com/Server2024/Content/System_Guides/Instance_Setting_Guide/Instance_setting_descriptions.htm#ESIndexCreationSettings). > * Always verify the minimum required Elasticsearch version in your specific release bundle, as it may differ from the versions mentioned here. -After installing the required Elastic components for Data Grid Audit, the integration between Elastic and Relativity is configured by running the Relativity Server CLI. It is recommended to run the CLI from the Primary SQL Server. - ### Prerequisites - 1. Install the mapper-size plugin on all nodes in the Elasticsearch cluster (instructions available [here](https://www.elastic.co/guide/en/elasticsearch/plugins/current/mapper-size.html)). The Elasticsearch service must also be restarted after installing the plugin. 2. The Server-bundle zip file has been downloaded and extracted to `C:\Server.Bundle.x.y.z' @@ -22,61 +23,59 @@ After installing the required Elastic components for Data Grid Audit, the integr -#### Set up instructions +### Set up instructions Follow these steps to set up Data Grid Audit using the Relativity Server CLI. All setup will occur on the SQL Primary server. -**Step 1** - Open elevated command prompt/powershell. Run below command. Select Datagrid - -``` -C:\Server.Bundle.x.y.z\relsvr.exe setup - -Relativity Server CLI - 24.0.1196 -Copyright (c) 2025, Relativity ODA LLC +1. Open elevated command prompt/powershell. Run below command. Select **Datagrid** + ``` + C:\Server.Bundle.x.y.z\relsvr.exe setup + Relativity Server CLI - 24.0.1196 + Copyright (c) 2025, Relativity ODA LLC -What would you like to setup? -> DataGrid - Environment watch - Exit -``` + What would you like to setup? + > DataGrid + Environment watch + Exit + ``` -**Step 2** - Enter the required Relativity and Elasticsearch parameters. +2. Enter the required Relativity and Elasticsearch parameters. -``` -Confirm you would like to perform the 'DataGrid' setup [y/n] (y): y + ``` + Confirm you would like to perform the 'DataGrid' setup [y/n] (y): y -Existing settings do not exist -Enter the Relativity admin username (relativity.admin@kcura.com): relativity.admin@kcura.com -Enter the Relativity admin password: ********* -Enter the Relativity instance url (https://emttest/Relativity): https://emttest/Relativity -Relativity instance is verified -Enter the Elasticsearch admin username (elastic): elastic -Enter the Elasticsearch admin password: ********* -Enter the Elasticsearch cluster endpoint URL (https://emttest:9200): https://emttest:9200 + Existing settings do not exist + Enter the Relativity admin username (relativity.admin@kcura.com): relativity.admin@kcura.com + Enter the Relativity admin password: ********* + Enter the Relativity instance url (https://emttest/Relativity): https://emttest/Relativity + Relativity instance is verified + Enter the Elasticsearch admin username (elastic): elastic + Enter the Elasticsearch admin password: ********* + Enter the Elasticsearch cluster endpoint URL (https://emttest:9200): https://emttest:9200 -``` + ``` -**Step 3** - Wait for Setup to Complete. +3. Wait for Setup to Complete. -``` -Elasticsearch cluster endpoint URL is verified -Elasticsearch plugin verified + ``` + Elasticsearch cluster endpoint URL is verified + Elasticsearch plugin verified -API Key creation and validation completed ------------------------- 100% -Relativity instance setting validation completed ------------------ 100% -Relativity secret store updated ----------------------------------- 100% -Elastic Stack settings validation completed ----------------------- 100% -Relativity toggles validation completed --------------------------- 100% + API Key creation and validation completed ------------------------- 100% + Relativity instance setting validation completed ------------------ 100% + Relativity secret store updated ----------------------------------- 100% + Elastic Stack settings validation completed ----------------------- 100% + Relativity toggles validation completed --------------------------- 100% -The Relativity Data Grid setup has been completed. Please restart all Relativity services, including "kCura Edds Agent Manager," "kCura Edds Web Processing Manager," and "kCura Service Host Manager" on each server contained within this Relativity instance to complete the setup. -``` -If the setup completes successfully, Datagrid is now configured for the environment. + The Relativity Data Grid setup has been completed. Please restart all Relativity services, including "kCura Edds Agent Manager," "kCura Edds Web Processing Manager," and "kCura Service Host Manager" on each server contained within this Relativity instance to complete the setup. + ``` + + If the setup completes successfully, Datagrid is now configured for the environment. -**Step 4** - Restart the Relativity services on all machines for the changes to take effect. +4. Restart the Relativity services on all machines for the changes to take effect. -**Step 5** - Verify Audit Dashboard -Navigate to the Audit tab in the Relativity environment and confirm that the dashboard and its data are loading correctly. +5. Verify Audit Dashboard - navigate to the Audit tab in the Relativity environment and confirm that the dashboard and its data are loading correctly. ## Next diff --git a/docs/elasticsearch_pre_installation_overview.md b/docs/elasticsearch_pre_installation_overview.md index 53cac32..8d5ca38 100644 --- a/docs/elasticsearch_pre_installation_overview.md +++ b/docs/elasticsearch_pre_installation_overview.md @@ -127,9 +127,7 @@ The environment size is defined by the number of Web, Agent, and Worker servers | Medium | 2-4 | 5-9 | 2-9 | | Large | 5+ | 10+ | 10+ | -
-Each Elasticsearch server should have at least 4 vCPU and 32 GB RAM. -
+> Each Elasticsearch server should have at least 4 vCPU and 32 GB RAM. #### Environment Size – Development @@ -230,8 +228,8 @@ Each Elasticsearch server should have at least 4 vCPU and 32 GB RAMFor a large environment, Elasticsearch is scaled horizontally by adding more nodes to the cluster(s). -
+> [!NOTE] +> For a large environment, Elasticsearch is scaled horizontally by adding more nodes to the cluster(s). | Elastic Stack Component | Server Count | | ----------------------------------------- | ---------------------- | @@ -286,7 +284,7 @@ If you have used Elasticsearch for the optional Data Grid Audit feature on Relat 3. **Verify the minimum supported version of Elastic** - All Relativity products have been certified with Elastic Stack 8.17.x, though Data Grid Audit alone is supported on 7.17.x. If you intend to use a single cluster for both, you must install the same version of Elasticsearch on all nodes in the cluster, and that version must be the higher of the minimum versions for Environment Watch and Data Grid Audit, if different. -4. **At least the minimum Relativity major version and patch** specified in the Environment Watch bundle you intend to deploy is installed on all servers in the environment. See the [System Requirements](/README.md) for the minimum version required. +4. **At least the minimum Relativity major version and patch** specified in the Environment Watch bundle you intend to deploy is installed on all servers in the environment. See the [System Requirements](../README.md) for the minimum version required. 5. **At least the minimum supported version of Windows Server** for the major version of Relativity installed in your environment is installed on each target server in your cluster (see [here](https://help.relativity.com/Server2024/Content/System_Guides/Workstation_Configuration/User_hardware_and_software_requirements.htm)). diff --git a/docs/elasticsearch_setup_development.md b/docs/elasticsearch_setup_development.md index 49314d9..ce77490 100644 --- a/docs/elasticsearch_setup_development.md +++ b/docs/elasticsearch_setup_development.md @@ -6,513 +6,424 @@ > This step is required for both Environment Watch and Data Grid Audit ## How to Unblock Downloaded Files - -> [!NOTE] If you download a .zip or other file from the internet, Windows may block the file and prevent it from running correctly. To unblock a file: -- Right-click the downloaded file and select **Properties**. -- In the **General** tab, check the box for **Unblock** at the bottom (if present). -- Click **Apply** and then **OK**. - -![Unblock file screenshot](../resources/troubleshooting-images/unblocked.png) - -## Step 1: Download and Install Elasticsearch 8.17.3 on One Server - - -**1.1 Download Elasticsearch 8.17.3** - -- Visit [Elastic’s official download page](https://www.elastic.co/downloads/elasticsearch). -- Download the 8.17.3 Windows .zip version. - -- Before extracting, see [How to Unblock Downloaded Files](#how-to-unblock-downloaded-files). - -- Extract the files to `C:\elastic` - -**1.2 Install and Configure Elasticsearch 8.17.3** - -- Open an elevated PowerShell and run the following command to start Elasticsearch and perform the auto installation steps: - - ``` - C:\elastic\elasticsearch-8.17.3\bin\elasticsearch.bat - ``` - - -> [!IMPORTANT] - When starting Elasticsearch for the first time, security features are enabled and configured by default: - - Authentication and authorization are enabled, and a password is generated for the elastic built-in superuser. - - Certificates and keys for TLS are generated for the transport and HTTP layer, and TLS is enabled and configured with these keys and certificates. - - An enrollment token is generated for Kibana, which is valid for 30 minutes. - -- Save the token for future reference. Once the enrollment token is displayed, you need to stop Elasticsearch so you can proceed with the next steps. To do this, return to the PowerShell window where Elasticsearch is running and press `Ctrl+C` on your keyboard. This will safely terminate the process. The enrollment token will look similar to: - - ``` - Enrollment token for Kibana: - eyJ2ZXIiOiI4LjE3LjMiLCJ... - ``` - -> [!TIP] -> To stop Elasticsearch after you have copied the enrollment token, click inside the PowerShell window and press `Ctrl` and `C` at the same time. This will end the running process. - -- Open an elevated PowerShell and run the following command to install Elasticsearch as a Windows service: - - ``` - C:\elastic\elasticsearch-8.17.3\bin\elasticsearch-service.bat install - ``` - - The output will look similar to: - - ``` - Installing service : "elasticsearch-service-x64" - Using JAVA_HOME (64-bit): C:\Program Files\Java\jdk-17 - The service 'elasticsearch-service-x64' has been installed. - ``` - -**1.3 Run Elasticsearch as a Windows Service** - -- Open an elevated PowerShell and run the following command to start the Elasticsearch service: - - ``` - C:\elastic\elasticsearch-8.17.3\bin\elasticsearch-service.bat start - ``` - - The output will look similar to: - - ``` - Starting service : "elasticsearch-service-x64" - The service 'elasticsearch-service-x64' has been started. - ``` +1. Right-click the downloaded file and select **Properties**. +2. In the **General** tab, check the box for **Unblock** at the bottom (if present). +3. Click **Apply** and then **OK**. + + ![Unblock file screenshot](../resources/troubleshooting-images/unblocked.png) + +### Step 1: Download and Install Elasticsearch 8.17.3 on one server +1. Download Elasticsearch 8.17.3 + 1. Visit [Elastic’s official download page](https://www.elastic.co/downloads/elasticsearch). + 2. Download the 8.17.3 Windows .zip version. + 3. Before extracting, see [How to Unblock Downloaded Files](#how-to-unblock-downloaded-files). + 4. Extract the files to `C:\elastic` +2. Install and Configure Elasticsearch 8.17.3 + 1. Open an elevated PowerShell and run the following command to start Elasticsearch and perform the auto installation steps: + ``` + C:\elastic\elasticsearch-8.17.3\bin\elasticsearch.bat + ``` + + + > When starting Elasticsearch for the first time, security features are enabled and configured by default: + > - Authentication and authorization are enabled, and a password is generated for the elastic built-in superuser. + > - Certificates and keys for TLS are generated for the transport and HTTP layer, and TLS is enabled and configured with these keys and certificates. + > - An enrollment token is generated for Kibana, which is valid for 30 minutes. + + 2. Save the token for future reference. Once the enrollment token is displayed, you need to stop Elasticsearch so you can proceed with the next steps. To do this, return to the PowerShell window where Elasticsearch is running and press `Ctrl+C` on your keyboard. This will safely terminate the process. The enrollment token will look similar to: + ``` + Enrollment token for Kibana: + eyJ2ZXIiOiI4LjE3LjMiLCJ... + ``` + > [!NOTE] + > To stop Elasticsearch after you have copied the enrollment token, click inside the PowerShell window and press `Ctrl` and `C` at the same time. This will end the running process. + + 3. Open an elevated PowerShell and run the following command to install Elasticsearch as a Windows service: + ``` + C:\elastic\elasticsearch-8.17.3\bin\elasticsearch-service.bat install + ``` + The output will look similar to: + ``` + Installing service : "elasticsearch-service-x64" + Using JAVA_HOME (64-bit): C:\Program Files\Java\jdk-17 + The service 'elasticsearch-service-x64' has been installed. + ``` +3. Run Elasticsearch as a Windows Service + 1. Open an elevated PowerShell and run the following command to start the Elasticsearch service: + ``` + C:\elastic\elasticsearch-8.17.3\bin\elasticsearch-service.bat start + ``` + The output will look similar to: + ``` + Starting service : "elasticsearch-service-x64" + The service 'elasticsearch-service-x64' has been started. + ``` +4. Enable Stack Monitoring + 1. Navigate to the Elasticsearch configuration folder (e.g., `C:\elastic\elasticsearch-8.17.3\config`) and open the **elasticsearch.yml** file. + 2. Add the following line to enable Stack Monitoring: + ``` + xpack.monitoring.collection.enabled: true + ``` + 3. Save the changes and restart the Elasticsearch service by opening an elevated PowerShell and running the following command: + ``` + Restart-Service -Name "elasticsearch-service-x64" + ``` +5. Reset the Elastic (Admin) User Password + 1. The following command resets the password for the `elastic` user, which is the default superuser (admin) account in Elasticsearch. This account is required for logging in to Kibana and for performing administrative tasks such as managing users, roles, and system settings. + 2. Navigate to ElasticSearch's bin folder(`C:\elastic\elasticsearch-8.17.3\bin`) + 3. Open an elevated PowerShell and run the following command: + ``` + .\elasticsearch-reset-password -u elastic + ``` + 4. When prompted, press 'Y' to confirm and reset the password + + ![elastic-reset-password](../resources/troubleshooting-images/elastic-reset-password.png) + + 5. When you run this command, a new password will be generated and displayed in the console output. + > [!IMPORTANT] + > The password is shown only once and cannot be retrieved later. Immediately record and securely store the password according to your organization’s credential management and security policies. You will need this password for future authentication to Elasticsearch and Kibana. + +6. Install the 'mapper-size' plugin + 1. Open an elevated PowerShell and run the following command to install the 'mapper-size' plugin: + ``` + C:\elastic\elasticsearch-8.17.3\bin\elasticsearch-plugin install mapper-size + ``` + 2. To verify the 'mapper-size' plugin is installed, run: + ``` + C:\elastic\elasticsearch-8.17.3\bin\elasticsearch-plugin list + ``` + 3. Restart the Elasticsearch Service. To restart the Elasticsearch service, run the following in an elevated PowerShell session: + ``` + Restart-Service -Name "elasticsearch-service-x64" + ``` + The output will look similar to: + ``` + WARNING: Waiting for service 'Elasticsearch 8.17.3 (elasticsearch-service-x64) (elasticsearch-service-x64)' to stop... + ``` + +7. Verify Elasticsearch Server + 1. To verify Elasticsearch is running, open an elevated Command Prompt and run the following command (replace ``, ``, and `` with your actual values): + ``` + curl -u : -k https://:9200 + ``` + 2. The response should show basic cluster information in JSON format if the server is running and accessible. + +
+ Sample JSON response + + ``` + { + "name" : "emttest", + "cluster_name" : "elasticsearch", + "cluster_uuid" : "q5VtYDCQT2iNHU9dOdqomw", + "version" : { + "number" : "8.17.3", + "build_flavor" : "default", + "build_type" : "zip", + "build_hash" : "a091390de485bd4b127884f7e565c0cad59b10d2", + "build_date" : "2025-02-28T10:07:26.089129809Z", + "build_snapshot" : false, + "lucene_version" : "9.12.0", + "minimum_wire_compatibility_version" : "7.17.0", + "minimum_index_compatibility_version" : "7.0.0" + }, + "tagline" : "You Know, for Search" + } + ``` + +
+ +### Step 2: Install and Configure Kibana + +1. Download Kibana 8.17.3 + 1. Download and extract the 8.17.3 Windows .zip version of Kibana from [Elastic’s official Kibana download page](https://www.elastic.co/downloads/kibana). + 2. Before extracting, see [How to Unblock Downloaded Files](#how-to-unblock-downloaded-files). + +2. Start Kibana from the command line + 1. Navigate to Kibana's `bin` folder (e.g., `C:\elastic\kibana\bin`). + 2. Open an elevated PowerShell and run the following command: + ``` + C:\Kibana\kibana-8.17.3\bin\kibana.bat + ``` + 3. If successful, you should see output indicating that the Kibana server has started and is listening on port 5601. Look for lines similar to: + ``` + [INFO][server][http] http server running at http://localhost:5601 + ... + kibana has not been configured + Go to https://localhost:5601/?code=xyz to get started + ``` +3. Enroll Kibana + 1. In your terminal, click the generated link to open Kibana in your browser. + 2. In your browser, paste the enrollment token that was generated in the terminal when you started Elasticsearch, then click the Configure Elastic button to connect your Kibana instance with Elasticsearch. + [See where the enrollment token is generated.](#enrollment-token-generation) + 3. If the token has expired, generate a new one by running the following command in the Elasticsearch's bin folder (e.g., `C:\elastic\elasticsearch-8.17.3\bin`). + ``` + C:\elastic\elasticsearch-8.17.3\bin\elasticsearch-create-enrollment-token --scope kibana + ``` +
+ Sample output + eyJ2ZXIiOiI4LjE0LjAiLCJhZHIiOlsiMTAuMC4yLjI6OTIwMCJdLCJmZ3IiOiI4ZGE1MWZkYTExZmM1ZDAwNDBhZWZlNTJlNmRiYzQ5ZTM2NmYxYTkyOGIwY2NiMzExOGY0MWFjZTczODNkZDliIiwia2V5IjoiOGFfc1BKZ0Jra09qNlh6dngycS06bG5sWkNEMnpSbFNiZjZZclpRSHF6dyJ9 +
+ 4. Log in to Kibana as the `elastic` user with the password that was generated when you started Elasticsearch. + 5. See the screenshot below for the login screen: + + ![](../resources/elasticsearch_setup_003.png) + +4. Generate Kibana encryption keys + > [!NOTE] + > Skipping the steps below will cause the Relativity Server CLI to fail. + + 1. Open an elevated PowerShell and run the following command: + ``` + C:\Kibana\kibana-8.17.3\bin\kibana-encryption-keys generate + ``` + + 2. If successful, you will see output showing the generated encryption keys. For example: + +
+ Sample output + + ``` + xpack.encryptedSavedObjects.encryptionKey: "" + xpack.reporting.encryptionKey: "" + xpack.security.encryptionKey: "" + ``` + +
+ + 3. Copy the generated encryption keys and paste them at the end of your `kibana.yml` file (e.g., `C:\Kibana\kibana-8.17.3\config\kibana.yml`). The configuration should look similar to: + +
+ Sample kibana.yml configuration + + ```yaml + xpack.encryptedSavedObjects.encryptionKey: "" + xpack.reporting.encryptionKey: "" + xpack.security.encryptionKey: "" + ``` + +
+ + 4. Restart the Kibana service, by opening an elevated PowerShell and run the following command: + ``` + C:\Kibana\kibana-8.17.3\bin\kibana.bat + ``` + + 5. To verify success, check the terminal output for lines indicating that Kibana has started successfully. You can also refer to the screenshots below: -**1.4 Enable Stack Monitoring** + ![Kibana restart verification](../resources/troubleshooting-images/kibanarerun.png) -- Navigate to the Elasticsearch configuration folder (e.g., `C:\elastic\elasticsearch-8.17.3\config`) and open the `elasticsearch.yml` file. -- Add the following line to enable Stack Monitoring: + 6. After Kibana has restarted, open a browser and go to `http://:5601`. + 7. Log in using the `elastic` username and the password you generated earlier. This verifies that Kibana is running and your credentials are working. - ``` - xpack.monitoring.collection.enabled: true - ``` + ![Kibana restart](../resources/troubleshooting-images/kibanaloginpageresponse.png) -- Save the changes and restart the Elasticsearch service: + 8. For more details, refer to the official documentation: https://www.elastic.co/guide/en/kibana/current/kibana-encryption-keys.html - - Open an elevated PowerShell and run the following command: - ``` - Restart-Service -Name "elasticsearch-service-x64" - ``` +5. Create Kibana Windows Service + > [!IMPORTANT] + > **Running Kibana as a Windows Service is Optional** + > Environment Watch does NOT require Kibana to run as a Windows service, nor does it require the use of NSSM. NSSM is a commonly used open-source tool to help run applications as services, but it is not mandatory. You can run Kibana manually from the command line if you prefer, and this will work perfectly for development and most production scenarios. + > Only use NSSM if you want Kibana to start automatically as a service on Windows. If you do not wish to use NSSM, simply run `kibana.bat` manually. + > + > ``` + > C:\Kibana\kibana-8.17.3\bin\kibana.bat + > ``` + + 1. Download the latest NSSM executable from https://nssm.cc/download and place it in the C drive (e.g., `C:\nssm-2.24`). + > [!NOTE] + > Kibana does not install as a Windows service by default. We recommend using NSSM — a commonly used open-source tool—to run Kibana as a Windows service. -**1.5 Reset the Elastic (Admin) User Password** + 2. Open an elevated PowerShell and run the following command: -- The following command resets the password for the `elastic` user, which is the default superuser (admin) account in Elasticsearch. This account is required for logging in to Kibana and for performing administrative tasks such as managing users, roles, and system settings. + ``` + C:\nssm-2.24\win64\nssm.exe install kibana + ``` + + This will open a popup to create a Windows service for Kibana. - - Navigate to ElasticSearch's bin folder(`C:\elastic\elasticsearch-8.17.3\bin`) - - Open an elevated PowerShell and run the following command: + 3. In the Application tab, enter the path to `kibana.bat` and its folder as shown below: + + ![Kibana service application tab](../resources/troubleshooting-images/kibanaservice-applicationtab.png) - ``` - .\elasticsearch-reset-password -u elastic - ``` - - When prompted, press 'Y' to confirm and reset the password - ![elastic-reset-password](../resources/troubleshooting-images/elastic-reset-password.png) + > **Editing Kibana Service Properties:** + > If you accidentally install the Kibana service before completing your configuration (for example, by pressing Return too early in the NSSM dialog), you can easily edit the service properties afterward. This allows you to update the application path, log file settings, or other options without reinstalling the service. + > To edit the service properties, open an elevated PowerShell and run the following command: `C:\nssm-2.24\win64\nssm.exe edit kibana` -- When you run this command, a new password will be generated and displayed in the console output. -> [!NOTE] - The password is shown only once and cannot be retrieved later. Immediately record and securely store the password according to your organization’s credential management and security policies. You will need this password for future authentication to Elasticsearch and Kibana. - -**1.6 Install the 'mapper-size' plugin** - -- Open an elevated PowerShell and run the following command to install the 'mapper-size' plugin: - ``` - C:\elastic\elasticsearch-8.17.3\bin\elasticsearch-plugin install mapper-size - ``` -- To verify the 'mapper-size' plugin is installed, run: - ``` - C:\elastic\elasticsearch-8.17.3\bin\elasticsearch-plugin list - ``` - -- Restart the Elasticsearch Service: - - To restart the Elasticsearch service, run the following in an elevated PowerShell session: - - ``` - Restart-Service -Name "elasticsearch-service-x64" - ``` - - The output will look similar to: - - ``` - WARNING: Waiting for service 'Elasticsearch 8.17.3 (elasticsearch-service-x64) (elasticsearch-service-x64)' to stop... - ``` - -**1.7 Verify Elasticsearch Server** - -- To verify Elasticsearch is running, open an elevated Command Prompt and run the following command (replace ``, ``, and `` with your actual values): - - ``` - curl -u : -k https://:9200 - ``` - -- The response should show basic cluster information in JSON format if the server is running and accessible. - -
- Sample JSON response - - ``` - { - "name" : "emttest", - "cluster_name" : "elasticsearch", - "cluster_uuid" : "q5VtYDCQT2iNHU9dOdqomw", - "version" : { - "number" : "8.17.3", - "build_flavor" : "default", - "build_type" : "zip", - "build_hash" : "a091390de485bd4b127884f7e565c0cad59b10d2", - "build_date" : "2025-02-28T10:07:26.089129809Z", - "build_snapshot" : false, - "lucene_version" : "9.12.0", - "minimum_wire_compatibility_version" : "7.17.0", - "minimum_index_compatibility_version" : "7.0.0" - }, - "tagline" : "You Know, for Search" - } - ``` -
- -## Step 2: Install and Configure Kibana - -**2.1 Download Kibana 8.17.3** + 4. In the I/O tab, enter the full path of a log file where the service logs will be stored. For example, create a folder in the Kibana directory (e.g., `C:\Kibana\kibana-8.17.3\service_logs`) and a blank log file (e.g., `C:\Kibana\kibana-8.17.3\service_logs\kibana_service.log`). -- Download and extract the 8.17.3 Windows .zip version of Kibana from [Elastic’s official Kibana download page](https://www.elastic.co/downloads/kibana). - -- Before extracting, see [How to Unblock Downloaded Files](#how-to-unblock-downloaded-files). - -**2.2 Start Kibana from the command line** + 5. Copy the full log file path into the stdout and stderr sections: -- Navigate to Kibana's `bin` folder (e.g., `C:\elastic\kibana\bin`). -- Open an elevated PowerShell and run the following command: - - ``` - C:\Kibana\kibana-8.17.3\bin\kibana.bat - ``` - -- If successful, you should see output indicating that the Kibana server has started and is listening on port 5601. Look for lines similar to: - - ``` - [INFO][server][http] http server running at http://localhost:5601 - ... - kibana has not been configured - Go to https://localhost:5601/?code=xyz to get started - ``` - -**2.3 Enroll Kibana** + ![Kibana service I/O tab](../resources/troubleshooting-images/kibanaservice-io-tab.png) -- In your terminal, click the generated link to open Kibana in your browser. -- In your browser, paste the enrollment token that was generated in the terminal when you started Elasticsearch, then click the Configure Elastic button to connect your Kibana instance with Elasticsearch. - - [See where the enrollment token is generated.](#enrollment-token-generation) -- If the token has expired, generate a new one by running the following command in the Elasticsearch's bin folder (e.g., `C:\elastic\elasticsearch-8.17.3\bin`). - - ``` - C:\elastic\elasticsearch-8.17.3\bin\elasticsearch-create-enrollment-token --scope kibana - ``` -
- Sample output - eyJ2ZXIiOiI4LjE0LjAiLCJhZHIiOlsiMTAuMC4yLjI6OTIwMCJdLCJmZ3IiOiI4ZGE1MWZkYTExZmM1ZDAwNDBhZWZlNTJlNmRiYzQ5ZTM2NmYxYTkyOGIwY2NiMzExOGY0MWFjZTczODNkZDliIiwia2V5IjoiOGFfc1BKZ0Jra09qNlh6dngycS06bG5sWkNEMnpSbFNiZjZZclpRSHF6dyJ9 -
- -- Log in to Kibana as the `elastic` user with the password that was generated when you started Elasticsearch. -- See the screenshot below for the login screen: - - ![](/resources/elasticsearch_setup_003.png) - -**2.4 Generate Kibana encryption keys** - -> [!IMPORTANT] - Skipping the steps below will cause the Relativity Server CLI to fail. - -- Open an elevated PowerShell and run the following command: - - ``` - C:\Kibana\kibana-8.17.3\bin\kibana-encryption-keys generate - ``` - - - If successful, you will see output showing the generated encryption keys. For example: - -
- Sample output - - ``` - xpack.encryptedSavedObjects.encryptionKey: "" - xpack.reporting.encryptionKey: "" - xpack.security.encryptionKey: "" - ``` -
- - - Copy the generated encryption keys and paste them at the end of your `kibana.yml` file (e.g., `C:\Kibana\kibana-8.17.3\config\kibana.yml`). The configuration should look similar to: - -
- Sample kibana.yml configuration - - ```yaml - xpack.encryptedSavedObjects.encryptionKey: "" - xpack.reporting.encryptionKey: "" - xpack.security.encryptionKey: "" - ``` -
- - - Restart the Kibana service: - - Open an elevated PowerShell and run the following command: - - ``` - C:\Kibana\kibana-8.17.3\bin\kibana.bat - ``` - - - To verify success, check the terminal output for lines indicating that Kibana has started successfully. You can also refer to the screenshots below: - - ![Kibana restart verification](../resources/troubleshooting-images/kibanarerun.png) - -> [!IMPORTANT] -After Kibana has restarted, open a browser and go to `http://:5601`. -Log in using the `elastic` username and the password you generated earlier. -This verifies that Kibana is running and your credentials are working. - - ![Kibana restart](../resources/troubleshooting-images/kibanaloginpageresponse.png) - - - For more details, refer to the official documentation: https://www.elastic.co/guide/en/kibana/current/kibana-encryption-keys.html - -**2.5 Create Kibana Windows Service** -> [!IMPORTANT] -> **Running Kibana as a Windows Service is Optional** -> Environment Watch does NOT require Kibana to run as a Windows service, nor does it require the use of NSSM. NSSM is a commonly used open-source tool to help run applications as services, but it is not mandatory. You can run Kibana manually from the command line if you prefer, and this will work perfectly for development and most production scenarios. -> -> Only use NSSM if you want Kibana to start automatically as a service on Windows. If you do not wish to use NSSM, simply run `kibana.bat` manually. - - ``` - C:\Kibana\kibana-8.17.3\bin\kibana.bat - ``` + 6. In the File rotation tab, check all boxes and enter `10485760` bytes so a new log file is generated for every 10 MB of logs: -- Download the latest NSSM executable from https://nssm.cc/download and place it in the C drive (e.g., `C:\nssm-2.24`). - -> [!NOTE] -Kibana does not install as a Windows service by default. We recommend using NSSM — a commonly used open-source tool—to run Kibana as a Windows service. - -- Open an elevated PowerShell and run the following command: - - ``` - C:\nssm-2.24\win64\nssm.exe install kibana - ``` - - This will open a popup to create a Windows service for Kibana. - -- In the Application tab, enter the path to `kibana.bat` and its folder as shown below: - - ![Kibana service application tab](../resources/troubleshooting-images/kibanaservice-applicationtab.png) - - -> [!TIP] -> **Editing Kibana Service Properties:** -> If you accidentally install the Kibana service before completing your configuration (for example, by pressing Return too early in the NSSM dialog), you can easily edit the service properties afterward. This allows you to update the application path, log file settings, or other options without reinstalling the service. - -
-To edit the service properties - -- Open an elevated PowerShell and run the following command: - - ``` - C:\nssm-2.24\win64\nssm.exe edit kibana - ``` - -
- -- In the I/O tab, enter the full path of a log file where the service logs will be stored. For example, create a folder in the Kibana directory (e.g., `C:\Kibana\kibana-8.17.3\service_logs`) and a blank log file (e.g., `C:\Kibana\kibana-8.17.3\service_logs\kibana_service.log`). - -- Copy the full log file path into the stdout and stderr sections: - - ![Kibana service I/O tab](../resources/troubleshooting-images/kibanaservice-io-tab.png) - -- In the File rotation tab, check all boxes and enter `10485760` bytes so a new log file is generated for every 10 MB of logs: - - ![Kibana service file rotation tab](../resources/troubleshooting-images/kibanaservice-filerotationtab.png) - -- Click the Install service button to create the Windows service for Kibana. - -- Go to the Services app in Windows, search for the `kibana` service, right click, and start the service. - -- Right click on the service and open Properties to change the startup type to Automatic, so Kibana runs automatically on system startup. - -- Verify that Kibana is running by opening it in your browser. - -> [!NOTE] - It is normal for Kibana to take 1-5 minutes to become accessible after starting the service, depending on your system. Please be patient while it starts up. - -**2.6 Verify Kibana Server** -- Open a browser and go to `http://:5601`. -- Log in using the `elastic` credential to verify successful access. - - ![Kibana Login Page](../resources/troubleshooting-images/kibanaloginpage.png) - ![Kibana Login Page Response](../resources/troubleshooting-images/kibanaloginpageresponse.png) - -## Step 3: Install and Configure APM Server - -**3.1 Prerequisites to setup APM Server** - -- Elastic and Kibana should be configured and services should be up and running. - -**3.2 Download APM Server 8.17.3** - -- Visit [Elastic’s APM Server page](https://www.elastic.co/downloads/apm). -- Download and extract the 8.17.3 Windows .zip file. -- Before extracting, see [How to Unblock Downloaded Files](#how-to-unblock-downloaded-files). -- Extract the files to `C:\`. - - -**3.3 Configure APM Server (`C:\apm-server-8.17.3-windows-x86_64\apm-server.yml`)** - -- An API key is required for configuring both APM and Beats. To create an API key: - - - Log in to Kibana (`http://:5601`) using the `elastic` credential. + ![Kibana service file rotation tab](../resources/troubleshooting-images/kibanaservice-filerotationtab.png) + + 7. Click the Install service button to create the Windows service for Kibana. - - Use the global search at the top of Kibana to search for "API keys" and select it from the results. + 8. Go to the Services app in Windows, search for the `kibana` service, right click, and start the service. - - Click the **Create API key** button. + 9. Right click on the service and open **Properties** to change the startup type to **Automatic**, so Kibana runs automatically on system startup. - - Enter a name for your API key (for example, specify if it will be used for Beats or APM). + 10. Verify that Kibana is running by opening it in your browser. + > [!NOTE] + > It is normal for Kibana to take 1-5 minutes to become accessible after starting the service, depending on your system. Please be patient while it starts up. - - Click the **Privileges** dropdown and select **Beats** to automatically apply the recommended permissions for Beats. +6. Verify Kibana Server + 1. Open a browser and go to `http://:5601`. + 2. Log in using the `elastic` credential to verify successful access. - - Click **Create API key**. - - - Copy and securely save the generated `id` and `api_key` values. + ![Kibana Login Page](../resources/troubleshooting-images/kibanaloginpage.png) + ![Kibana Login Page Response](../resources/troubleshooting-images/kibanaloginpageresponse.png) - ![create-apikey](../resources/troubleshooting-images/create-apikey.png) +### Step 3: Install and Configure APM Server -> [!NOTE] -Copy and save `id` and `api_key` values immediately and store them securely according to your organization’s credential management and security policies. +1. Prerequisites to setup APM Server + - Elastic and Kibana should be configured and services should be up and running. -- Navigate to the apm-server folder (e.g., `C:\apm-server-8.17.3-windows-x86_64`) and open the `apm-server.yml` file using a text editor. +2. Download APM Server 8.17.3 + 1. Visit [Elastic’s APM Server page](https://www.elastic.co/downloads/apm). + 2. Download and extract the 8.17.3 Windows .zip file. + 3. Before extracting, see [How to Unblock Downloaded Files](#how-to-unblock-downloaded-files). + 4. Extract the files to `C:\`. -- Open the `apm-server.yml` file in a text editor and update it to match the following sample configuration. +3. Configure APM Server (`C:\apm-server-8.17.3-windows-x86_64\apm-server.yml`) -- Replace the placeholder values (``, ``, ``, ``) as needed for your environment. + 1. An API key is required for configuring both APM and Beats. To create an API key: + 1. Log in to Kibana (`http://:5601`) using the `elastic` credential. + 2. Use the global search at the top of Kibana to search for "API keys" and select it from the results. + 3. Click the **Create API key** button. + 4. Enter a name for your API key (for example, specify if it will be used for Beats or APM). + 5. Click the **Privileges** dropdown and select **Beats** to automatically apply the recommended permissions for Beats. + 6. Click **Create API key**. + 7. Copy and securely save the generated `id` and `api_key` values. + + ![create-apikey](../resources/troubleshooting-images/create-apikey.png) -- Below is a sample configuration + > Copy and save `id` and `api_key` values immediately and store them securely according to your organization’s credential management and security policies. - ```yaml - apm-server: - host: ":8200" + 2. Navigate to the apm-server folder (e.g., `C:\apm-server-8.17.3-windows-x86_64`) and open the `apm-server.yml` file using a text editor. - output.elasticsearch: - hosts: [":9200"] - protocol: "https" - api_key: ":" - ssl.enabled: true - ssl.verification_mode: none + 3. Update the `apm-server.yml` file to match the following sample configuration. Replace the placeholder values (``, ``, ``, ``) as needed for your environment. Below is a sample configuration: - instrumentation: - enabled: true - environment: production - hosts: - - "http://:8200" - ``` + ```yaml + apm-server: + host: ":8200" + + output.elasticsearch: + hosts: [":9200"] + protocol: "https" + api_key: ":" + ssl.enabled: true + ssl.verification_mode: none + + instrumentation: + enabled: true + environment: production + hosts: + - "http://:8200" + ``` -**3.4 Execute required scripts to install APM Server as a Windows service** - -- Open an elevated PowerShell. -- Run the following command to install the APM Server as a Windows service: - - ``` - PowerShell.exe -ExecutionPolicy UnRestricted -File C:\apm-server-8.17.3-windows-x86_64\install-service.ps1 - ``` - - The output will look similar to: - - ``` - Installing service apm-server... - Service "apm-server" has been successfully installed. - ``` +4. Execute required scripts to install APM Server as a Windows service + 1. Open an elevated PowerShell. + 2. Run the following command to install the APM Server as a Windows service: + + ``` + PowerShell.exe -ExecutionPolicy UnRestricted -File C:\apm-server-8.17.3-windows-x86_64\install-service.ps1 + ``` + The output will look similar to: + + ``` + Installing service apm-server... + Service "apm-server" has been successfully installed. + ``` -**3.5 Start the APM Server service** - -- Open an elevated PowerShell and run the following command: - - ``` - Start-Service -Name "apm-server" - ``` +5. Start the APM Server service + 1. Open an elevated PowerShell and run the following command: + + ``` + Start-Service -Name "apm-server" + ``` -- Once the instrumentation is set, you can verify it in Kibana as shown below: + 2. Once the instrumentation is set, you can verify it in Kibana as shown below: - ![verify-instrumentation](../resources/troubleshooting-images/verify-instrumentation.png) + ![verify-instrumentation](../resources/troubleshooting-images/verify-instrumentation.png) -**3.6 Verify APM Server** +6. Verify APM Server -- Open an elevated Command Prompt and run the following command (replace `` with your actual value): + 1. Open an elevated Command Prompt and run the following command (replace `` with your actual value): - ``` - curl -k http://:8200 - ``` + ``` + curl -k http://:8200 + ``` - The response should indicate `publish_ready` is `true` and will look similar to: + The response should indicate `publish_ready` is `true` and will look similar to: - ``` - { - "build_date": "2025-02-27T18:17:35Z", - "build_sha": "f6b917b725e1a22af433e5b52c5c6f0ff9164adf", - "publish_ready": true, - "version": "8.17.3" - } - ``` + ``` + { + "build_date": "2025-02-27T18:17:35Z", + "build_sha": "f6b917b725e1a22af433e5b52c5c6f0ff9164adf", + "publish_ready": true, + "version": "8.17.3" + } + ``` -## Step 4: Additional Setup and Verification +### Step 4: Additional Setup and Verification +1. Add Elastic APM Integration Package + > [!IMPORTANT] + > Skipping the steps below will cause the Relativity Server CLI to fail. -**4.1 Add Elastic APM Integration Package** - -> [!IMPORTANT] - Skipping the steps below will cause the Relativity Server CLI to fail. - - - Login to Kibana and select the Elastic APM under Integration, or in the search bar type "Elastic APM" and select it under Integration. + 1. Login to Kibana and select the Elastic APM under Integration, or in the search bar type "Elastic APM" and select it under Integration. - ![alt text](../resources/troubleshooting-images/apm-integration.png) + ![alt text](../resources/troubleshooting-images/apm-integration.png) - - In the top right, select the Add Elastic APM button. + 2. In the top right, select the Add Elastic APM button. - ![alt text](../resources/troubleshooting-images/add-apm-integration.png) + ![alt text](../resources/troubleshooting-images/add-apm-integration.png) - - Add an Integration name and for server configuration. Update apm hostname and apm url. - Ex: Host: :8200 - URL: http://:8200 + 3. Add an Integration name and for server configuration. Update apm hostname and apm url. + Ex: Host: :8200 + URL: http://:8200 - - Click on Save and Continue. + 4. Click on **Save** and **Continue**. - ![alt text](../resources/troubleshooting-images/apm-integration-host-name.png) + ![alt text](../resources/troubleshooting-images/apm-integration-host-name.png) - - Select "Add Elastic Agent later" button as Agent is not required for the initial setups. + 5. Select **Add Elastic Agent later** button as Agent is not required for the initial setups. - ![alt text](../resources/troubleshooting-images/agent-button.png) + ![alt text](../resources/troubleshooting-images/agent-button.png) -**4.2 Verify APM Data View** +2. Verify APM Data View + Before proceeding with EW CLI, check if the APM Data View is created in Kibana: -Before proceeding with EW CLI, check if the APM Data View is created in Kibana: - -- Open a browser and go to http://:5601 -- Log in using elastic credentials -- Navigate to `Discover`, or type `Discover` in the search bar. -- Confirm the APM Data View is present: + 1. Open a browser and go to http://:5601 + 2. Log in using elastic credentials + 3. Navigate to **Discover**, or type `Discover` in the search bar. + 4. Confirm the APM Data View is present: - ![dataview](../resources/troubleshooting-images/dataview.png) + ![dataview](../resources/troubleshooting-images/dataview.png) -**4.3 Verify Cluster Health** +3. Verify Cluster Health -- Open an elevated Command Prompt and run the following command (replace `username`, `password`, and `{hostname_or_ip}` with your actual values): + 1. Open an elevated Command Prompt and run the following command (replace `username`, `password`, and `hostname_or_ip` with your actual values): - ``` - curl -u : -k https://:9200/_cat/health - ``` + ``` + curl -u : -k https://:9200/_cat/health + ``` -- You should see a response similar to: + 2. You should see a response similar to: - ``` - 1690219200 10:00:00 elasticsearch green 1 1 0 0 0 0 0 0 - 100.0% - ``` + ``` + 1690219200 10:00:00 elasticsearch green 1 1 0 0 0 0 0 0 - 100.0% + ``` -- The word `green` in the response means the cluster is healthy. The word `yellow` in the response means the cluster is partially healthy. If you see `red`, investigate further. + 3. The word `green` in the response means the cluster is healthy. The word `yellow` in the response means the cluster is partially healthy. If you see `red`, investigate further. ## Next Step diff --git a/docs/elasticsearch_setup_production.md b/docs/elasticsearch_setup_production.md index f5d6beb..98f836f 100644 --- a/docs/elasticsearch_setup_production.md +++ b/docs/elasticsearch_setup_production.md @@ -1,3 +1,3 @@ -# 🚧 # Install Elasticsearch, Kibana and APM Server - Production Environment 🚧 +# Install Elasticsearch, Kibana and APM Server - Production Environment This page is currently under construction. Please check back soon! \ No newline at end of file diff --git a/docs/environment-watch/post-install-verification.md b/docs/environment-watch/post-install-verification.md index acb02eb..8dab9d2 100644 --- a/docs/environment-watch/post-install-verification.md +++ b/docs/environment-watch/post-install-verification.md @@ -7,19 +7,19 @@ ## Prerequisites -> [!IMPORTANT] -> After installation, wait 10–15 minutes before starting the verification process. This allows time for: -> - All system services to fully initialize and become available -> - Data collection agents to begin exporting telemetry to Elasticsearch -> - Dashboard visualizations to populate with accurate status information -> - Health indicators to show accurate statuses + +After installation, wait 10–15 minutes before starting the verification process. This allows time for: +- All system services to fully initialize and become available +- Data collection agents to begin exporting telemetry to Elasticsearch +- Dashboard visualizations to populate with accurate status information +- Health indicators to show accurate statuses ## Overview This documentation outlines the step-by-step procedures for verifying that the entire Relativity Environment Watch system is functioning properly after installation. The verification process confirms that the Elastic Stack cluster is healthy, all monitoring agents are exporting telemetry to Elasticsearch, and the integration with Relativity (e.g. Relativity Alerts) is working correctly. -## Verification Steps +## Verification steps ### 1. Elastic Cluster Health This section guides through verifying the health and proper functioning of the Elasticsearch cluster. @@ -39,7 +39,6 @@ This section covers how to ensure that the alerting mechanism is working as expe > [!NOTE] > All Kibana dashboards are designed and optimized for **1920x1080** screen resolution to ensure optimal viewing experience and proper layout formatting. ---- ## Next Step diff --git a/docs/environment-watch/post-install-verification/alert-overview.md b/docs/environment-watch/post-install-verification/alert-overview.md index 1a7f463..6b2dfe3 100644 --- a/docs/environment-watch/post-install-verification/alert-overview.md +++ b/docs/environment-watch/post-install-verification/alert-overview.md @@ -1,131 +1,98 @@ # Post-Install Verification for Alerts ![Post-Install Verification Banner](../../../resources/post-install-verification-images/Post-installation-verification.svg) -## Table of Contents +## Verify that the dashboard exists -- [Post-Install Verification for Alerts](#post-install-verification-for-alerts) - - [Verify Dashboard Exists](#verify-dashboard-exists) - - [Verify Dashboard Tag](#verify-dashboard-tag) - - [Verify Health Indicators Are Displayed](#verify-health-indicators-are-displayed) - - [Verify Individual Health Indicator Status](#verify-individual-health-indicator-status) - - [Verify Dashboard in a Time Range](#verify-dashboard-in-a-time-range) - ---- -> [!NOTE] -> The below Health Indicators will be available from September 8th 2025 release. -## Verify Dashboard Exists - -**Description:** -Ensure that the `[Relativity] Alerts Overview` dashboard is successfully installed and visible in Kibana. +Ensure that the **[Relativity] Alerts Overview** dashboard is successfully installed and visible in Kibana. **Steps:** -1. Open **Microsoft Edge**. + +1. Open Microsoft Edge. 2. Navigate to Kibana. 3. In Kibana, go to **Dashboards**. 4. Use the search bar to search for `Alerts Overview`. -5. Select `[Relativity] Alerts Overview` from the results. +5. Select **[Relativity] Alerts Overview** from the results. -
-Expected Result +**Expected Result:** -- `[Relativity] Alerts Overview` appears in the dashboard list. +- **[Relativity] Alerts Overview** appears in the dashboard list. - The dashboard is accessible without errors. -
**Screenshot:** ![Screenshot: Dashboard listed in Kibana](../../../resources/post-install-verification-images/alerts-overview/dashboard-listed.png) ---- -## Verify Dashboard Tag +## Verify the dashboard tag -**Description:** Confirm that the correct tag is applied to the dashboard for proper categorization. **Steps:** + 1. In Kibana, go to **Dashboards**. -2. Search for `Alerts Overview` and select `[Relativity] Alerts Overview`. -3. Verify that the tag `Relativity Environment Watch` is displayed. +2. Search for `Alerts Overview` and select **[Relativity] Alerts Overview**. +3. Verify that the tag **Relativity Environment Watch** is displayed. -
-Expected Result +**Expected Result:** -- The dashboard includes the `Relativity Environment Watch` tag. -
+- The dashboard includes the **Relativity Environment Watch** tag. **Screenshot:** ![Screenshot: Dashboard tag](../../../resources/post-install-verification-images/alerts-overview/dashboard-tag.png) ---- -## Verify Health Indicators Are Displayed +## Verify that health indicators are displayed -**Description:** Ensure that the health indicators section is visible at the top of the dashboard and follows the correct layout. **Steps:** -1. Click into `[Relativity] Alerts Overview` dashboard. +1. Click into **[Relativity] Alerts Overview** dashboard. 2. Confirm that the health indicators appear at the top of the main content area. 3. Confirm all tiles include a **title**, **subtitle ("Healthy")**, and **color status**. -
-Expected Result - +**Expected Result:** - Health indicators are present and aligned at the top. - All indicators include: - Title (e.g., "Agents", "Monitoring") - Subtitle: *Healthy* - Color: Green or Red based on alert state. -
**Screenshot:** ![Screenshot: Health indicators at top](../../../resources/post-install-verification-images/alerts-overview/health-indicators-overview.png) ---- -## Verify Individual Health Indicator Status +## Verify individual health indicator status -**Description:** Verify the status and formatting of each health indicator tile, based on alert-driven logic and the color coding conventions. **Steps:** -1. Open the `[Relativity] Alerts Overview` dashboard. +1. Open the **[Relativity] Alerts Overview** dashboard. 2. Locate each of the following indicators and confirm the title, subtitle, and color. -
-Expected Result - +**Expected Result:** - **Green** = No active alerts (Healthy) - **Red** = Active alerts present (Unhealthy) -
-> [!NOTE] > All health indicators should display either Green or Red status after the initial wait period. If any health indicators aren't displaying correctly after waiting the full 10-15 minutes, verify that all related services are running properly and data collection is functioning correctly. **Example Screenshot:** ![All Health Indicators](../../../resources/post-install-verification-images/alerts-overview/all-health-indicators.png) ---- -## Verify Dashboard in a Time Range +## Verify dashboard in a time range -**Description:** Ensure that the dashboard is using a custom 15-minute time range as required for health indicators. **Steps:** 1. On the dashboard, navigate to one of the alerts. 2. Locate the time filter at the top right. 3. Click on the time range selector. -4. Select **"Apply custom time range"**. +4. Select **Apply custom time range**. 5. Set the range to the **last 15 minutes**. 6. Apply changes. -
-Expected Result - +**Expected Result:** - The time range reflects the last 15 minutes. - Health indicators update dynamically based on this range. -
**Screenshot:** ![Screenshot: Time range 15 minutes](../../../resources/post-install-verification-images/alerts-overview/time-range-15-minutes.png) diff --git a/docs/environment-watch/post-install-verification/elasticsearch-cluster-health.md b/docs/environment-watch/post-install-verification/elasticsearch-cluster-health.md index 1e61396..4324ab7 100644 --- a/docs/environment-watch/post-install-verification/elasticsearch-cluster-health.md +++ b/docs/environment-watch/post-install-verification/elasticsearch-cluster-health.md @@ -1,43 +1,25 @@ # Post-Install Verification for Elastic Cluster Health ![Post-Install Verification Banner](../../../resources/post-install-verification-images/Post-installation-verification.svg) -## Table of Contents +## Verify that the dashboard exists -- [Post-Install Verification for Elastic Cluster Health](#post-install-verification-for-elastic-cluster-health) - - [Verify Dashboard Exists](#verify-dashboard-exists) - - [Verify Cluster Health Summary](#verify-cluster-health-summary) - - [Verify Node Metrics](#verify-node-metrics) - - [Verify Index Statistics](#verify-index-statistics) - - [Verify Disk and Storage Utilization](#verify-disk-and-storage-utilization) - ---- -> [!NOTE] -> The below Cluster Dashboard will be available starting from September 8th 2025 release. -## Verify Dashboard Exists - -**Description:** Navigate to the Elasticsearch cluster health monitoring dashboard. **Steps:** 1. Login to Kibana. 2. Navigate to **Dashboards** -2. Use the search bar to search for "Monitoring Agents" and open the dashboard +2. Use the search bar to search for `Monitoring Agents` and open the dashboard 3. Click on **Cluster Overview**. 4. Alternatively, visit this direct link: http://:5601/app/monitoring# -
-Expected Result - +**Expected Result:** - Successfully access the Elasticsearch monitoring dashboard. - Cluster health panels and metrics are visible. - Dashboard loads without errors. -
---- -## Verify Cluster Health Summary +## Verify cluster health summary -**Description:** Confirm cluster health status, total nodes, shards, indices, and document count are displayed and accurate. **Steps:** @@ -50,23 +32,18 @@ Confirm cluster health status, total nodes, shards, indices, and document count - Total documents. - Data size. -
-Expected Result - +**Expected Result:** - Cluster health is **Green (Healthy)**. - Nodes, shards, indices, and documents display current, non-zero values. - No unassigned shards. - Data size is displayed accurately. -
**Screenshot:** ![Screenshot: Cluster Health Summary](../../../resources/post-install-verification-images/elasticsearch-cluster-health/cluster-health-summary.png) ---- -## Verify Node Metrics +## Verify node metrics -**Description:** Ensure node-level metrics such as CPU usage, JVM heap usage, and disk space are reported per node. **Steps:** @@ -78,22 +55,17 @@ Ensure node-level metrics such as CPU usage, JVM heap usage, and disk space are - Disk free space. - Load Average (may be unavailable, verify if data present). -
-Expected Result - +**Expected Result:** - All nodes listed. - CPU, JVM heap %, and disk free space values present. - Load average may show as N/A if unsupported but should be monitored for future inclusion. -
**Screenshot:** ![Screenshot: Node Metrics](../../../resources/post-install-verification-images/elasticsearch-cluster-health/node-metrics.png) ---- -## Verify Index Statistics +## Verify index statistics -**Description:** Validate index-level metrics including document counts, data size, indexing rate, and search rate. **Steps:** @@ -106,36 +78,27 @@ Validate index-level metrics including document counts, data size, indexing rate - Unassigned shards (should be zero). 3. Check for any alerts or warnings on indices. -
-Expected Result - +**Expected Result:** - Per-index document counts and data sizes are populated. - Indexing and search rates update regularly. - No unassigned shards. - Alerts show clear or no issues on indices. -
**Screenshot:** ![Screenshot: Index Statistics](../../../resources/post-install-verification-images/elasticsearch-cluster-health/index-stats.png) ---- +## Verify disk and storage utilization -## Verify Disk and Storage Utilization - -**Description:** Ensure disk free space and usage metrics are visible for each node. **Steps:** 1. Check disk free space shown per node in the **Node Metrics** panel. 2. Confirm reported disk free space aligns with expectations. -
-Expected Result - +**Expected Result:** - Disk free space values displayed for all nodes. - No fields marked as N/A for disk metrics. -
**Screenshot:** ![Screenshot: Disk and Storage Utilization](../../../resources/post-install-verification-images/elasticsearch-cluster-health/disk-storage-utilization.png) diff --git a/docs/environment-watch/post-install-verification/monitoring-agents.md b/docs/environment-watch/post-install-verification/monitoring-agents.md index f4ed35b..be9a4c4 100644 --- a/docs/environment-watch/post-install-verification/monitoring-agents.md +++ b/docs/environment-watch/post-install-verification/monitoring-agents.md @@ -1,64 +1,42 @@ # Post-Install Verification for Monitoring Agents ![Post-Install Verification Banner](../../../resources/post-install-verification-images/Post-installation-verification.svg) -## Table of Contents +## Verify Monitoring Agent dashboard exists -- [Verify Monitoring Agent Dashboard Exists](#verify-monitoring-agent-dashboard-exists) -- [Verify Monitoring Agents Dashboard Tags](#verify-monitoring-agents-dashboard-tags) -- [Verify Dashboard Filters Populate with Data](#verify-dashboard-filters-populate-with-data) -- [Verify Data Population by Changing Filters and Time Ranges](#verify-data-population-by-changing-filters-and-time-ranges) -- [Verify Hosts and Agent Versions](#verify-hosts-and-agent-versions) -- [Verify Data in Discover](#verify-data-in-discover) - ---- - -## Verify Monitoring Agent Dashboard Exists - -**Description:** Ensure the Monitoring Agent dashboard is present. **Steps:** 1. Login to Kibana. 2. Navigate to **Dashboards** -3. Use the search bar to search for "Monitoring Agents". +3. Use the search bar to search for `Monitoring Agents`. -
-Expected Result - -* "Monitoring Agent" is listed. -
+**Expected Result:** +- **Monitoring Agent** is listed. **Screenshot:** ![Screenshot: Dashboard exists](../../../resources/post-install-verification-images/monitoring-agents/dashboard-exists.png) ---- -## Verify Monitoring Agents Dashboard Tags +## Verify Monitoring Agents dashboard tags -**Description:** Ensure the correct tags are assigned to the dashboard. **Steps:** 1. Login to Kibana. 2. Navigate to **Dashboards**. -3. Use the search bar to search for "Monitoring Agents" and open the dashboard. - -
-Expected Result +3. Use the search bar to search for `Monitoring Agents` and open the dashboard. -* Tags: - * `Relativity Environment Watch` - * `FeatureDomain: Monitoring` -
+**Expected Result:** +- Tags: + - **Relativity Environment Watch** + - **FeatureDomain: Monitoring** **Screenshot:** ![Screenshot: Tags](../../../resources/post-install-verification-images/monitoring-agents/dashboard-tags.png) ---- -## Verify Dashboard Filters Populate with Data +## Verify dashboard filters populate with data -**Description:** Ensure filter dropdowns are populated with available data. **Steps:** @@ -66,70 +44,53 @@ Ensure filter dropdowns are populated with available data. 2. Navigate to **Observability → Dashboard**. 3. Click on the dashboard and open each filter dropdown. -
-Expected Result - -* Filter dropdowns show available values. -
+**Expected Result:** +- Filter dropdowns show available values. -> [!NOTE] > Filters may take a few minutes to populate after initial data collection begins. **Screenshot:** ![Screenshot: Filter dropdown populated](../../../resources/post-install-verification-images/monitoring-agents/filter-dropdown-populated.png) ---- -## Verify Data Population by Changing Filters and Time Ranges +## Verify data population by changing filters and time ranges -**Description:** Ensure the dashboard data is loading correctly. **Steps:** 1. Login to Kibana. -2. Open the "Monitoring Agent" dashboard. +2. Open the **Monitoring Agent** dashboard. 3. To change the time range, use the time filter at the top right, select the desired range (e.g., 15 minutes, 1 hour, 12 hours, 24 hours), and apply changes. 4. Adjust filters to see if data updates accordingly. -
-Expected Result - +**Expected Result:** * All panels are populated with data. -
**Screenshot:** ![Screenshot: Populated data](../../../resources/post-install-verification-images/monitoring-agents/data-populated.png) ---- -## Verify Hosts and Agent Versions +## Verify Hosts and Agent versions -**Description:** Ensure hosts and agent versions are correctly displayed. **Steps:** 1. Login to Kibana. 2. Open the "Monitoring Agent" dashboard. -
-Expected Result - -* Host column lists multiple hosts. -* Agent Version is the same for all hosts. -* Last Modified Date column shows timestamp to verify installation time. -
+**Expected Result:** +- Host column lists multiple hosts. +- Agent Version is the same for all hosts. +- Last Modified Date column shows timestamp to verify installation time. -> [!NOTE] > Verify that the agent version is consistent across all nodes to ensure uniform monitoring capabilities. **Screenshot:** ![Screenshot: Hosts and versions](../../../resources/post-install-verification-images/monitoring-agents/hosts-agent-versions.png) ---- -## Verify Data in Discover +## Verify data in Discover -**Description:** Ensure dashboard data is reflected in Discover. **Steps:** @@ -137,14 +98,9 @@ Ensure dashboard data is reflected in Discover. 2. Open the dashboard and hover over the top right corner of the Monitoring Agents table to reveal the **Explore in Discover** option. 3. Click **Explore in Discover**. -
-Expected Result - -* Data is visible in Discover. -
+**Expected Result:** +- Data is visible in Discover. **Screenshot:** ![Screenshot: Discover view 01](../../../resources/post-install-verification-images/monitoring-agents/explore-discover-view.png) -![Screenshot: Discover view 02](../../../resources/post-install-verification-images/monitoring-agents/discover-view.png) ---- - +![Screenshot: Discover view 02](../../../resources/post-install-verification-images/monitoring-agents/discover-view.png) \ No newline at end of file diff --git a/docs/environment_watch_product_overview.md b/docs/environment_watch_product_overview.md index c8b6908..bd385ec 100644 --- a/docs/environment_watch_product_overview.md +++ b/docs/environment_watch_product_overview.md @@ -112,7 +112,7 @@ Starting with Server 2024, the Environment Watch architecture relies on the Elas #### High-Level Architecture ![](../resources/environment_watch_product_overview_003.png) -> [!Note] +> [!NOTE] > Ensure inbound/outbound ports are open for this Elastic stack component. [See the port diagram for network requirements.](environment-watch/port-diagram.md) @@ -164,7 +164,7 @@ These defaults can be customized in Elasticsearch to meet your organization's da Although alerts can be viewed within Kibana, Environment Watch uses a new Relativity Alerts application to integrate the alerts directly within the main Relativity UI. A new Alert Manager agent periodically uses a Kibana REST API to fetch alert details and update the "bell" header with the number of active alerts. Additional details about the Relativity Alerts application are provided below. -> [!Note] +> [!NOTE] > Relativity Alerts reflect the state of Kibana alerts within the Relativity UI. A Relativity agent periodically synchronizes the RDOs with the corresponding Kibana alerts, updating their state and trigger information. > > - **Kibana alerts** define rules in Elastic. @@ -176,7 +176,7 @@ Although alerts can be viewed within Kibana, Environment Watch uses a new Relati Environment Watch is enabled by the technology components: | Component | Type | Used for | Setup | -| --- | --- | --- | --- | +| - | - | - | - | | Elasticsearch | Third-party software (Elastic) | NoSQL document storage | Installed and configured by customer on own infrastructure | | Kibana | Third-party software (Elastic) | Alert and dashboard creation and management. | Installed and configured by customer on own infrastructure | | APM Server | Third-party software (Elastic) | Receives telemetry data from monitored hosts. | Installed and configured by customer on own infrastructure | @@ -251,7 +251,7 @@ When you have Relativity Alerts installed and Environment Watch is fully configu - **Alert State Updated** – When did the alert last change from inactive to active or active to inactive? - **Dashboard or Tab Jump Link** – Alerts are "generalized" in order to reduce noise and minimize the number of overall alerts. To see important details about an alert when active, Relativity will guide you to a Relativity tab or Kibana dashboard to further explore the potential issue. For example, when the ‘One ore more agents are disabled’ alert is active, the jump link will take you to the Agents tab to see which agents are actually disabled. Or, if the ‘Memory is exceeding 96% on at least one host’ alert is active, the jump link will take you directly into a Kibana dashboard to see which hosts are currently triggering the memory alert. -**Note** - Users will need Elastic credentials to log in to Kibana to see dashboards. The Elastic installation guide includes information about a Kibana role that is automatically created by the Relativity Server CLI that we recommend using for your Kibana users that need dashboard view-access. +> Users will need Elastic credentials to log in to Kibana to see dashboards. The Elastic installation guide includes information about a Kibana role that is automatically created by the Relativity Server CLI that we recommend using for your Kibana users that need dashboard view-access. - **Alert Enabled for Instance** – If set to No, Relativity will stop querying Kibana for updated alert state information - **Feature Domain** – Alerts are grouped by feature domain. You can see a list of all feature domains [here](https://github.com/relativityone/server-relativity-docs/blob/main/environment-watch/feature-domains.md). @@ -298,4 +298,4 @@ Any user can pause or turn off in-app notifications for individual alerts that t The Logs & Dashboards tab provides links to all Kibana dashboards and saved searches that Relativity has created and packaged as part of Environment Watch. -![](../resources/environment_watch_product_overview_015.png) +![](../resources/environment_watch_product_overview_015.png) \ No newline at end of file diff --git a/docs/environment_watch_troubleshooting.md b/docs/environment_watch_troubleshooting.md index 22dc97b..0434434 100644 --- a/docs/environment_watch_troubleshooting.md +++ b/docs/environment_watch_troubleshooting.md @@ -4,20 +4,10 @@ This document provides quick reference links to detailed troubleshooting guides ## Component Troubleshooting Guides -### [Pre-requisite Troubleshooting](troubleshooting/pre-requisite-troubleshooting.md) - -### [Elasticsearch Troubleshooting](troubleshooting/elasticsearch.md) - -### [Kibana Troubleshooting](troubleshooting/kibana.md) - -### [APM Server Troubleshooting](troubleshooting/apm-server.md) - -### [Environment Watch Monitoring Agent and Open Telemetry Collector Troubleshooting](troubleshooting/monitoring-agent-and-otel-collector.md) - -### [Relativity Server CLI Troubleshooting](troubleshooting/relativity-server-cli.md) - -### [Relativity Alerts Troubleshooting](troubleshooting/relativity_alerts_troubleshooting.md) - - - - +- [Pre-requisite Troubleshooting](troubleshooting/pre-requisite-troubleshooting.md) +- [Elasticsearch Troubleshooting](troubleshooting/elasticsearch.md) +- [Kibana Troubleshooting](troubleshooting/kibana.md) +- [APM Server Troubleshooting](troubleshooting/apm-server.md) +- [Environment Watch Monitoring Agent and Open Telemetry Collector Troubleshooting](troubleshooting/monitoring-agent-and-otel-collector.md) +- [Relativity Server CLI Troubleshooting](troubleshooting/relativity-server-cli.md) +- [Relativity Alerts Troubleshooting](troubleshooting/relativity_alerts_troubleshooting.md) \ No newline at end of file diff --git a/docs/environment_watch_upgrade.md b/docs/environment_watch_upgrade.md index b867766..8322da7 100644 --- a/docs/environment_watch_upgrade.md +++ b/docs/environment_watch_upgrade.md @@ -11,67 +11,68 @@ This section guides you through upgrading Environment Watch using the Relativity ### Prerequisites -1. The new Server-bundle zip file has been downloaded and extracted to `C:\Server.Bundle.x.y.z` +- The new Server-bundle zip file has been downloaded and extracted to `C:\Server.Bundle.x.y.z` ### Upgrade Instructions -**Step 1** - Open elevated command prompt/powershell. Run the below command. Select Environment Watch. - -``` -C:\Server.Bundle.x.y.z\relsvr.exe setup - -Relativity Server CLI - 24.0.1196 -Copyright (c) 2025, Relativity ODA LLC - -What would you like to setup? - DataGrid -> Environment watch - Exit -``` - -**Step 2** - Enter 'y' to proceed. -``` -Confirm you would like to perform the 'Environment Watch' setup [y/n] (y): y -``` - -**Step 3** - The CLI will detect that setup has already been performed. Select the **Upgrade** option to proceed. - -``` -Setup is already performed. Select an option to proceed: -> Upgrade - Rerun Setup - Exit -``` - -**Step 4** - The CLI will retrieve the existing settings and perform the upgrade. Please wait for the process to finish. - -``` -Retrieved existing settings -The Relativity Environment Watch Kibana Upgrade setup is executing in silent or unattended mode. -API Key creation and validation completed ------------------------- 100% -OAuth2 client exists ---------------------------------------------- 100% -Relativity secret store updated ----------------------------------- 100% -Relativity logging updated ---------------------------------------- 100% -Relativity toggles updated ---------------------------------------- 100% -Relativity AppDomain monitoring enabled --------------------------- 100% -APM settings updated ---------------------------------------------- 100% -Elasticsearch indexes updated ------------------------------------- 100% -Kibana Tag imported ----------------------------------------------- 100% -Kibana IndexPattern imported -------------------------------------- 100% -Kibana SavedSearch imported --------------------------------------- 100% -Kibana Dashboard imported ----------------------------------------- 100% -Kibana Alert imported --------------------------------------------- 100% -Kibana Custom Role created. --------------------------------------- 100% -Relativity installation SQL record updated ------------------------ 100% - -The Environment Watch setup has been completed. The Relativity Environment Watch installer package should now be installed within each server contained within this Relativity instance. As each server is setup for monitoring, restart all Relativity services within the machine including "kCura Edds Agent Manager," "kCura Edds Web Processing Manager," and "kCura Service Host Manager" to begin sending telemetry to Elasticsearch. -``` +1. Open elevated command prompt/powershell. Run the below command. Select **Environment Watch**. + + ``` + C:\Server.Bundle.x.y.z\relsvr.exe setup + + Relativity Server CLI - 24.0.1196 + Copyright (c) 2025, Relativity ODA LLC + + What would you like to setup? + DataGrid + > Environment watch + Exit + ``` + +2. Enter 'y' to proceed. + + ``` + Confirm you would like to perform the 'Environment Watch' setup [y/n] (y): y + ``` + +3. The CLI will detect that setup has already been performed. Select the **Upgrade** option to proceed. + + ``` + Setup is already performed. Select an option to proceed: + > Upgrade + Rerun Setup + Exit + ``` + +4. The CLI will retrieve the existing settings and perform the upgrade. Please wait for the process to finish. + + ``` + Retrieved existing settings + The Relativity Environment Watch Kibana Upgrade setup is executing in silent or unattended mode. + API Key creation and validation completed ------------------------- 100% + OAuth2 client exists ---------------------------------------------- 100% + Relativity secret store updated ----------------------------------- 100% + Relativity logging updated ---------------------------------------- 100% + Relativity toggles updated ---------------------------------------- 100% + Relativity AppDomain monitoring enabled --------------------------- 100% + APM settings updated ---------------------------------------------- 100% + Elasticsearch indexes updated ------------------------------------- 100% + Kibana Tag imported ----------------------------------------------- 100% + Kibana IndexPattern imported -------------------------------------- 100% + Kibana SavedSearch imported --------------------------------------- 100% + Kibana Dashboard imported ----------------------------------------- 100% + Kibana Alert imported --------------------------------------------- 100% + Kibana Custom Role created. --------------------------------------- 100% + Relativity installation SQL record updated ------------------------ 100% + + The Environment Watch setup has been completed. The Relativity Environment Watch installer package should now be installed within each server contained within this Relativity instance. As each server is setup for monitoring, restart all Relativity services within the machine including "kCura Edds Agent Manager," "kCura Edds Web Processing Manager," and "kCura Service Host Manager" to begin sending telemetry to Elasticsearch. + ``` Successful completion indicates that Environment Watch is configured for the new version. Refer to the [Troubleshooting Guide](troubleshooting/relativity-server-cli.md) if you encounter any issues. ---- + ## Upgrade Environment Watch Installer @@ -82,43 +83,44 @@ This document outlines the steps to upgrade the Environment Watch monitoring age ### Prerequisites -1. The Environment Watch CLI upgrade has been completed. -2. The new Server-bundle zip file has been downloaded and extracted. - -### Step 1: Install the Upgraded Monitoring Agent +- The Environment Watch CLI upgrade has been completed. +- The new Server-bundle zip file has been downloaded and extracted. -1. From the new extracted Server Bundle, run `Relativity.EnvironmentWatch.Installer.xx.x.xxxx.exe`. -2. Accept the license terms and click **Install**. -3. Once the installation is complete, click **Close**. +1. Install the Upgraded Monitoring Agent -### Step 2: Verify the Installation + 1. From the new extracted Server Bundle, run `Relativity.EnvironmentWatch.Installer.xx.x.xxxx.exe`. + 2. Accept the license terms and click **Install**. + 3. Once the installation is complete, click **Close**. -1. Open **Services** and confirm the **Relativity Environment Watch** service is running. -2. Open **Task Manager** and verify the following processes are running: - * `rel-envwatch-service.exe` - * `rel-infrawatch-agent.exe` - * `otelcol-relativity.exe` -3. Check for new log files in `C:\ProgramData\Relativity\EnvironmentWatch\Services\InfraWatchAgent\Logs`. -4. Verify that the `otelcol-config-auto-generated.yaml` configuration file is updated in: - `C:\ProgramData\Relativity\EnvironmentWatch\Services\InfraWatchAgent\` +2. Verify the Installation -### Step 3: Verify Metrics in Kibana + 1. Open **Services** and confirm the **Relativity Environment Watch** service is running. + 2. Open **Task Manager** and verify the following processes are running: + * `rel-envwatch-service.exe` + * `rel-infrawatch-agent.exe` + * `otelcol-relativity.exe` + 3. Check for new log files in `C:\ProgramData\Relativity\EnvironmentWatch\Services\InfraWatchAgent\Logs`. + 4. Verify that the `otelcol-config-auto-generated.yaml` configuration file is updated in + `C:\ProgramData\Relativity\EnvironmentWatch\Services\InfraWatchAgent\` -1. In Kibana, navigate to **Dashboards** and open the **[Relativity] Host Infrastructure Overview** dashboard. -2. Confirm that CPU, RAM, and Disk metrics are visible for the upgraded host. - ![Host metrics visible in Kibana](../resources/Installer_hostmetric.png) -3. Navigate to the **[Relativity] Monitoring Agent** dashboard to verify: - - The monitoring agent version is updated to the new version. - - Host information and other important fields are properly reflected. - - Agent status and health metrics are showing current data. +3. Verify Metrics in Kibana -### Step 4: Repeat Installation for Other Servers + 1. In Kibana, navigate to **Dashboards** and open the **[Relativity] Host Infrastructure Overview** dashboard. + 2. Confirm that CPU, RAM, and Disk metrics are visible for the upgraded host. + + ![Host metrics visible in Kibana](../resources/Installer_hostmetric.png) + 3. Navigate to the **[Relativity] Monitoring Agent** dashboard to verify: + - The monitoring agent version is updated to the new version. + - Host information and other important fields are properly reflected. + - Agent status and health metrics are showing current data. +4. Repeat Installation for other servers. After verifying the first upgrade, repeat these steps for the remaining servers in the following order: -1. SQL Distributed Servers -2. Web Servers -3. Agent Servers -4. Other Servers (e.g., File Share, Analytics, Worker) + + 1. SQL Distributed Servers + 2. Web Servers + 3. Agent Servers + 4. Other Servers (e.g., File Share, Analytics, Worker) > [!NOTE] > For troubleshooting, refer to the installer logs in the `%TEMP%` directory or see the [Monitoring Agent Troubleshooting Guide](troubleshooting/monitoring-agent-and-otel-collector.md). diff --git a/docs/install_environment_watch_monitoring_agents.md b/docs/install_environment_watch_monitoring_agents.md index 078b701..e609d6b 100644 --- a/docs/install_environment_watch_monitoring_agents.md +++ b/docs/install_environment_watch_monitoring_agents.md @@ -10,53 +10,54 @@ This document outlines the steps to install the Environment Watch monitoring age ### Prerequisites -1. Elastic Stack Certificates are installed on all Servers -2. Access to Elastic Stack, Primary SQL Server, and Secret Store (Whitelisted for Secret Store access. Please see [here](https://help.relativity.com/Server2024/Content/System_Guides/Secret_Store/Secret_Store.htm#Configuringclients) for information on whitelisting.) -3. The Server-bundle zip file has been downloaded and extracted to `C:\Server.Bundle.x.y.z' -4. Environment Watch has been set up using the Relativity Server CLI -5. The Environment Watch monitoring agent must be installed on a supported Windows Server OS including Windows Server 2022, Windows Server 2019, Windows Server 2016. Please see [here](https://help.relativity.com/Server2024/Content/Installing_and_Upgrading/System_requirements/Compatibility_matrix.htm#Relativitysystemrequirementsmatrix) for information on Windows Server compatibility for Relativity Server. - - -### Step 1: Install the Monitoring Agent - -1. From the extracted Server Bundle, run `Relativity.EnvironmentWatch.Installer.xx.x.xxxx.exe`. -2. If prompted, enter the Relativity Service Account credentials as: - - **Username:** `domain\Username` - - **Password:** +- Elastic Stack Certificates are installed on all Servers +- Access to Elastic Stack, Primary SQL Server, and Secret Store (Whitelisted for Secret Store access. Please see [here](https://help.relativity.com/Server2024/Content/System_Guides/Secret_Store/Secret_Store.htm#Configuringclients) for information on whitelisting.) +- The Server-bundle zip file has been downloaded and extracted to `C:\Server.Bundle.x.y.z' +- Environment Watch has been set up using the Relativity Server CLI +- The Environment Watch monitoring agent must be installed on a supported Windows Server OS including Windows Server 2022, Windows Server 2019, Windows Server 2016. Please see [here](https://help.relativity.com/Server2024/Content/Installing_and_Upgrading/System_requirements/Compatibility_matrix.htm#Relativitysystemrequirementsmatrix) for information on Windows Server compatibility for Relativity Server. + +### Steps + +1. Install the Monitoring Agent + 1. From the extracted Server Bundle, run `Relativity.EnvironmentWatch.Installer.xx.x.xxxx.exe`. + 2. If prompted, enter the Relativity Service Account credentials as: + - **Username:** `domain\Username` + - **Password:** {password} - This step may be skipped if other Relativity products are already installed. -3. (Optional) To specify a custom installation path, click **Options**, browse to your desired directory, and click **OK**. -4. Accept the license terms and click **Install**. -5. Once the installation is complete, click **Close**. - -### Step 2: Verify the Installation - -1. Open **Services** and confirm the **Relativity Environment Watch** service is running. -2. Open **Task Manager** and verify the following processes are running: - * `rel-envwatch-service.exe` - * `rel-infrawatch-agent.exe` - * `otelcol-relativity.exe` -3. Check for new log files in `C:\ProgramData\Relativity\EnvironmentWatch\Services\InfraWatchAgent\Logs`. -4. Verify that the `otelcol-config-auto-generated.yaml` configuration file is generated in: - `C:\ProgramData\Relativity\EnvironmentWatch\Services\InfraWatchAgent\` - -### Step 3: Verify Metrics in Kibana - -1. In Kibana, navigate to **Dashboards** and open the **[Relativity] Host Infrastructure Overview** dashboard. -2. Confirm that CPU, RAM, and Disk metrics are visible for the newly added host. - ![Host metrics visible in Kibana](../resources/Installer_hostmetric.png) -3. Navigate to the **[Relativity] Monitoring Agent** dashboard to verify: - - The installed monitoring agent version is displayed correctly - - Host information and other important fields are properly reflected - - Agent status and health metrics are showing current data - -### Step 4: Repeat Installation + This step may be skipped if other Relativity products are already installed. + 3. (Optional) To specify a custom installation path, click **Options**, browse to your desired directory, and click **OK**. + 4. Accept the license terms and click **Install**. + 5. Once the installation is complete, click **Close**. + +2. Verify the Installation + + 1. Open **Services** and confirm the **Relativity Environment Watch** service is running. + 2. Open **Task Manager** and verify the following processes are running: + * `rel-envwatch-service.exe` + * `rel-infrawatch-agent.exe` + * `otelcol-relativity.exe` + 3. Check for new log files in `C:\ProgramData\Relativity\EnvironmentWatch\Services\InfraWatchAgent\Logs`. + 4. Verify that the `otelcol-config-auto-generated.yaml` configuration file is generated in: + `C:\ProgramData\Relativity\EnvironmentWatch\Services\InfraWatchAgent\` + +3. Verify Metrics in Kibana + + 1. In Kibana, navigate to **Dashboards** and open the **[Relativity] Host Infrastructure Overview** dashboard. + 2. Confirm that CPU, RAM, and Disk metrics are visible for the newly added host. + + ![Host metrics visible in Kibana](../resources/Installer_hostmetric.png) + 3. Navigate to the **[Relativity] Monitoring Agent** dashboard to verify that: + - The installed monitoring agent version is displayed correctly + - Host information and other important fields are properly reflected + - Agent status and health metrics are showing current data +4. Repeat Installation After verifying the first installation, repeat these steps for the remaining servers in the following order: -1. SQL Distributed Servers -2. Web Servers -3. Agent Servers -4. Other Servers (e.g., File Share, Analytics, Worker) + 1. SQL Distributed Servers + 2. Web Servers + 3. Agent Servers + 4. Other Servers (e.g., File Share, Analytics, Worker) + > [!NOTE] > For troubleshooting, refer to the installer logs in the `%TEMP%` directory or see the [Monitoring Agent Troubleshooting Guide](troubleshooting/monitoring-agent-and-otel-collector.md). diff --git a/docs/rabbitmq/rabbitmq_integration.md b/docs/rabbitmq/rabbitmq_integration.md index 171e9b6..79ae323 100644 --- a/docs/rabbitmq/rabbitmq_integration.md +++ b/docs/rabbitmq/rabbitmq_integration.md @@ -3,9 +3,9 @@ This guide provides information on integrating RabbitMQ with Environment Watch. -### RabbitMQ Management Plugin +## RabbitMQ Management Plugin -## Prerequisites +### Prerequisites - RabbitMQ server installed and running - Administrative access to the RabbitMQ server @@ -13,7 +13,7 @@ This guide provides information on integrating RabbitMQ with Environment Watch. - Default admin user credentials (default: `guest`/`guest` for localhost only) - For production: Create a dedicated admin user for API access -## Enable the Management Plugin +### Enable the Management Plugin ```powershell # Enable the management plugin @@ -21,9 +21,9 @@ rabbitmq-plugins enable rabbitmq_management ``` -**Note:** Node restart is not required after plugin activation. +> Node restart is not required after plugin activation. -## Verify Plugin Status +### Verify plugin status ```powershell # Check if the plugin is enabled @@ -32,9 +32,7 @@ rabbitmq-plugins list # Check listening ports rabbitmq-diagnostics -s listeners ``` - -
-Expected output +**Expected Output:** ``` Listing plugins with pattern ".*" ... @@ -78,27 +76,24 @@ Listing plugins with pattern ".*" ... [ ] rabbitmq_web_stomp 3.13.7 [ ] rabbitmq_web_stomp_examples 3.13.7 ``` -**Note:** You should see `[E ] rabbitmq_management` indicating the plugin is enabled. -
. +> You should see `[E ] rabbitmq_management` indicating the plugin is enabled. ## Access the Management UI The management UI can be accessed at: `http://:15672/` ## User Management with curl Commands -> [!NOTE] + > For best compatibility on Windows, execute the following curl commands in Command Prompt (`cmd.exe`), not PowerShell. This ensures proper handling of double quotes and JSON payloads. ### Create Administrative User - +**Create admin user with full access** ``` # Create admin user with full access curl -i -u : -H "content-type:application/json" -X PUT http://:15672/api/users/admin_user -d "{\"password\":\"\",\"tags\":\"administrator\"}" ``` -
-Expected output - +**Expected Output:** ``` HTTP/1.1 201 Created content-length: 0 @@ -107,16 +102,13 @@ date: Fri, 29 Aug 2025 07:56:44 GMT server: Cowboy vary: accept, accept-encoding, origin ``` -
- +**Grant full permissions on default virtual host** ``` # Grant full permissions on default virtual host curl -i -u : -H "content-type:application/json" -X PUT http://:15672/api/permissions/%2F/admin_user -d "{\"configure\":\".*\",\"write\":\".*\",\"read\":\".*\"}" ``` -
-Expected output - +**Expected Output:** ``` HTTP/1.1 201 Created content-length: 0 @@ -125,18 +117,16 @@ date: Fri, 29 Aug 2025 08:08:00 GMT server: Cowboy vary: accept, accept-encoding, origin ``` -
### Create Monitoring User (Read-Only) +**Create monitoring user** ``` # Create monitoring user curl -i -u : -H "content-type:application/json" -X PUT http://:15672/api/users/monitoring_user -d "{\"password\":\"\",\"tags\":\"monitoring\"}" ``` -
-Expected output - +**Expected Output:** ``` HTTP/1.1 201 Created content-length: 0 @@ -145,16 +135,14 @@ date: Fri, 29 Aug 2025 08:08:32 GMT server: Cowboy vary: accept, accept-encoding, origin ``` -
+**Grant empty permissions** ``` # Grant empty permissions (monitoring access without resource permissions) curl -i -u : -H "content-type:application/json" -X PUT http://:15672/api/permissions/%2F/monitoring_user -d "{\"configure\":\"^$\",\"write\":\"^$\",\"read\":\"^$\"}" ``` -
-Expected output - +**Expected Output:** ``` HTTP/1.1 201 Created content-length: 0 @@ -163,18 +151,16 @@ date: Fri, 29 Aug 2025 08:09:01 GMT server: Cowboy vary: accept, accept-encoding, origin ``` -
### Create Management User (Basic Access) +**Create management user** ``` # Create management user curl -i -u : -H "content-type:application/json" -X PUT http://:15672/api/users/mgmt_user -d "{\"password\":\"\",\"tags\":\"management\"}" ``` -
-Expected output - +**Expected Output:** ``` HTTP/1.1 201 Created content-length: 0 @@ -183,16 +169,13 @@ date: Fri, 29 Aug 2025 08:09:33 GMT server: Cowboy vary: accept, accept-encoding, origin ``` -
+**Grant specific permissions** ``` # Grant specific permissions curl -i -u : -H "content-type:application/json" -X PUT http://:15672/api/permissions/%2F/mgmt_user -d "{\"configure\":\"^specific_queue.*\",\"write\":\"^specific_queue.*\",\"read\":\"^specific_queue.*\"}" ``` - -
-Expected output - +**Expected Output:** ``` HTTP/1.1 201 Created content-length: 0 @@ -201,18 +184,16 @@ date: Fri, 29 Aug 2025 08:10:08 GMT server: Cowboy vary: accept, accept-encoding, origin ``` -
## Verify User Setup +**List all users** ``` # List all users curl -i -u : http://:15672/api/users ``` -
-Expected output - +**Expected Output:** ``` HTTP/1.1 200 OK cache-control: no-cache @@ -225,16 +206,14 @@ vary: accept, accept-encoding, origin [{"hashing_algorithm":"rabbit_password_hashing_sha256","limits":{},"name":"admin","password_hash":"R1rPwt+kCnp/dBAl38a97mFvzxBPHVKSsrMbhcwCOjgBmHsC","tags":["administrator"]},{"hashing_algorithm":"rabbit_password_hashing_sha256","limits":{},"name":"admin_user","password_hash":"xxDctOHpGvwpwX4dHNtSfoo47crYYLL7b6LKnPlKbBQOmbOa","tags":["administrator"]},{"hashing_algorithm":"rabbit_password_hashing_sha256","limits":{},"name":"guest","password_hash":"LPKa7nr7nwIeUa4lGWtjG5Og6fi1JPFWbnjXVL3d2ReTCYsj","tags":["administrator"]},{"hashing_algorithm":"rabbit_password_hashing_sha256","limits":{},"name":"mgmt_user","password_hash":"K1tIRGiroPcXAw80MzKksRf4yBwcvAXBsnW/GgMsNrQB1GL1","tags":["management"]},{"hashing_algorithm":"rabbit_password_hashing_sha256","limits":{},"name":"monitoring_user","password_hash":"hZLpi5Ql5y+k28FVL0UdcMUPbMM0y9o/ZXtbZQCjFtjhhNM+","tags":["monitoring"]}] ``` -
+**Check specific user permissions** ``` # Check specific user permissions curl -i -u : http://:15672/api/permissions/%2F/admin_user ``` -
-Expected output - +**Expected Output:** ``` HTTP/1.1 200 OK cache-control: no-cache @@ -247,7 +226,6 @@ vary: accept, accept-encoding, origin {"user":"admin_user","vhost":"/","configure":".*","write":".*","read":".*"} ``` -
## User Permission Tags @@ -257,4 +235,4 @@ vary: accept, accept-encoding, origin | **monitoring** | Read-only | View all data, no modifications | | **management** | Basic access | Manage own virtual hosts only | -**Note:** Virtual host `/` is encoded as `%2F` in URLs. +> Virtual host `/` is encoded as `%2F` in URLs. diff --git a/docs/relativity_alerts_installation.md b/docs/relativity_alerts_installation.md index 9ac10da..c7ce4f2 100644 --- a/docs/relativity_alerts_installation.md +++ b/docs/relativity_alerts_installation.md @@ -17,13 +17,13 @@ The Relativity Alerts application is a new application as part of the Relativity To install the Relativity Alerts application on your Relativity instance: 1. Log in to Relativity as a System Administrator. 2. Navigate to the Application Library tab. -3. Click Upload Application.
+3. Click **Upload Application**
![relativity_alerts_installation_001](../resources/relativity_alerts_installation_001.png) -4. Click Select File.
+4. Click **Select File**.
![relativity_alerts_installation_002](../resources/relativity_alerts_installation_002.png) 5. Navigate to and select your local copy of Relativity Alerts RAP file which was downloaded from the latest [Environment Watch Bundle](https://github.com/relativitydev/server-bundle-release/releases), and then click Open.
![relativity_alerts_installation_003](../resources/relativity_alerts_installation_003.png) -6. Click Save to upload the file to the Application Library.
+6. Click **Save** to upload the file to the Application Library.
![relativity_alerts_installation_004](../resources/relativity_alerts_installation_004.png) ![relativity_alerts_installation_005](../resources/relativity_alerts_installation_005.png) ![relativity_alerts_installation_006](../resources/relativity_alerts_installation_006.png) @@ -31,11 +31,11 @@ To install the Relativity Alerts application on your Relativity instance: A list of common install issues and their resolutions are available at the [troubleshooting guide](../docs/troubleshooting/relativity_alerts_troubleshooting.md). ## Verification -1. Navigate to the Alerts tab, you should see all alerts available in Kibana.
+1. Navigate to the **Alerts** tab, you should see all alerts available in Kibana.
![relativity_alerts_verification_001](../resources/relativity_alerts_verification_001.png) -2. Navigate to "Logs & Dashboards" tab, you should see all Environment Watch dashboards and saved searches available in Kibana.
+2. Navigate to **Logs & Dashboards** tab, you should see all Environment Watch dashboards and saved searches available in Kibana.
![relativity_alerts_verification_002](../resources/relativity_alerts_verification_002.png) -3. When on "Logs & Dashboards" tab, find "Alerts Overview" dashboard and click the link. A new web browser tab will be created and you should be redirected to "Alerts Overview" dashboard in Kibana. If you have not logged in into Kibana yet, you will be prompted with a Kibana login page. Please login with your Kibana user account with privileges.
+3. When on **Logs & Dashboards** tab, find **Alerts Overview88 dashboard and click the link. A new web browser tab will be created and you should be redirected to "Alerts Overview" dashboard in Kibana. If you have not logged in into Kibana yet, you will be prompted with a Kibana login page. Please login with your Kibana user account with privileges.
![relativity_alerts_verification_003_001](../resources/relativity_alerts_verification_003_001.png) ![relativity_alerts_verification_003_002](../resources/relativity_alerts_verification_003_002.png) ![relativity_alerts_verification_003_003](../resources/relativity_alerts_verification_003_003.png) @@ -43,27 +43,26 @@ A list of common install issues and their resolutions are available at the [trou a. Go to the Agents tab
- b. Select "File Deletion Manager" agent and disable it.
+ b. Select **File Deletion Manager** agent and disable it.
![relativity_alerts_verification_004_001](../resources/relativity_alerts_verification_004_001.png) c. Within the next 30-60 seconds, the bell icon, in the top-right corner, will indicate that there is a new Alert.
![relativity_alerts_verification_004_002](../resources/relativity_alerts_verification_004_002.png) - d. Click on the bell icon, and you should be able to see the active 'One or more agents are disabled' alert in the alert flyout.
+ d. Click on the bell icon, and you should be able to see the active **One or more agents are disabled** alert in the alert flyout.
![relativity_alerts_verification_004_003](../resources/relativity_alerts_verification_004_003.png) - e. Click on the question mark icon next to the 'One or more agents are disabled' alert, and you will be redirected to the resolution guidance for this alert in a new tab of your web browser.
+ e. Click on the question mark icon next to the **One or more agents are disabled** alert, and you will be redirected to the resolution guidance for this alert in a new tab of your web browser.
![relativity_alerts_verification_004_004](../resources/relativity_alerts_verification_004_004.png)
![relativity_alerts_verification_004_005](../resources/relativity_alerts_verification_004_005.png) f. Return to the previous tab in Relativity, then click on the bell icon in the top-right corner. When the alert flyout opens, click on the 'One or more agents are disabled' alert, and you will be redirected to a filtered list of disabled agents in the Agents tab within Relativity.
![relativity_alerts_verification_004_006](../resources/relativity_alerts_verification_004_006.png) - g. While on Agents tab, please remember to enable the "File Deletion Manager" agent. - + g. While on Agents tab, please remember to enable the **File Deletion Manager** agent. If any of the previous steps have failed, or any errors were displayed, please see [troubleshooting guide](../docs/troubleshooting/relativity_alerts_troubleshooting.md) for more information. ## Next Step -[Click here for the next step](/docs/create_kibana_users.md) +[Click here for the next step](create_kibana_users.md) diff --git a/docs/relativity_server_cli_setup.md b/docs/relativity_server_cli_setup.md index a824d57..4e0e7e5 100644 --- a/docs/relativity_server_cli_setup.md +++ b/docs/relativity_server_cli_setup.md @@ -1,88 +1,87 @@ -# Enable Environment Watch +# Set up Environment Watch using the Relativity Server CLI ![Setup Stage](../resources/enable_environmentwatch.png) > [!NOTE] -> This step is required for Environment Watch. The Environment Watch setup via CLI should be run only **once** per environment. The setup performs tasks such as importing and updating Kibana resources including alerts, dashboards, saved searches and defect fixes. - -# Set up Environment Watch using the Relativity Server CLI +> This step is required for Environment Watch. -It is recommended to run the CLI from the Primary SQL Server. +> It is recommended to run the CLI from the Primary SQL Server. ### Prerequisites -1. Elastic Stack Certificates are installed on all Servers -2. Verify that the Elastic Stack services (Elasticsearch, Kibana, and APM) are running -3. Access to Elastic Stack, Primary SQL Server, and Secret Store (Whitelisted for Secret Store access. Please see [here](https://help.relativity.com/Server2024/Content/System_Guides/Secret_Store/Secret_Store.htm#Configuringclients) for information on whitelisting.) -4. The Server-bundle zip file has been downloaded and extracted to `C:\Server.Bundle.x.y.z' +- Elastic Stack Certificates are installed on all Servers +- Verify that the Elastic Stack services (Elasticsearch, Kibana, and APM) are running +- Access to Elastic Stack, Primary SQL Server, and Secret Store (Whitelisted for Secret Store access. Please see [here](https://help.relativity.com/Server2024/Content/System_Guides/Secret_Store/Secret_Store.htm#Configuringclients) for information on whitelisting.) +- The Server-bundle zip file has been downloaded and extracted to `C:\Server.Bundle.x.y.z' ### Set up instructions -**Step 1** - Open elevated command prompt/powershell. Run the below command. Select Environment Watch - -``` -C:\Server.Bundle.x.y.z\relsvr.exe setup - -Relativity Server CLI - 24.0.1196 -Copyright (c) 2025, Relativity ODA LLC - -What would you like to setup? - DataGrid -> Environment watch - Exit -``` - -**Step 2** - Confirm the Environment Watch setup. -``` -Confirm you would like to perform the 'Environment Watch' setup [y/n] (y): y -``` - -**Step 3** - Enter the required Relativity parameters. - -``` -Existing settings do not exist -Enter the Relativity admin username (relativity.admin@kcura.com): relativity.admin@kcura.com -Enter the Relativity admin password: ********* -Enter the Relativity instance url (https://emttest/Relativity): https://emttest/Relativity -Relativity instance is verified -``` - -**Step 4** - Enter the required Elastic Stack parameters. - -``` -Enter the Elasticsearch admin username (elastic): elastic -Enter the Elasticsearch admin password: ********* -Enter the Elasticsearch cluster endpoint URL (https://emttest:9200): https://emttest:9200 -Elasticsearch cluster endpoint URL is verified -Enter the Elasticsearch APM Server endpoint URL (http://emttest:8200): http://emttest:8200 -Enter the Elasticsearch Kibana endpoint URL (http://emttest:5601): http://emttest:5601 -``` - -**Step 5** - The CLI will now apply the necessary configurations. Please wait for the process to finish. - -``` -Elasticsearch Kibana URL is verified -API Key creation and validation completed ------------------------- 100% -OAuth2 client exists ---------------------------------------------- 100% -Relativity secret store updated ----------------------------------- 100% -Relativity logging updated ---------------------------------------- 100% -Relativity toggles updated ---------------------------------------- 100% -Relativity AppDomain monitoring enabled --------------------------- 100% -APM settings updated ---------------------------------------------- 100% -Elasticsearch indexes updated ------------------------------------- 100% -Kibana Tag imported ----------------------------------------------- 100% -Kibana IndexPattern imported -------------------------------------- 100% -Kibana SavedSearch imported --------------------------------------- 100% -Kibana Dashboard imported ----------------------------------------- 100% -Kibana Alert imported --------------------------------------------- 100% -Kibana Custom Role created. --------------------------------------- 100% -Relativity installation SQL record updated ------------------------ 100% - -The Environment Watch setup has been completed. The Relativity Environment Watch installer package should now be installed within each server contained within this Relativity instance. As each server is setup for monitoring, restart all Relativity services within the machine including "kCura Edds Agent Manager," "kCura Edds Web Processing Manager," and "kCura Service Host Manager" to begin sending telemetry to Elasticsearch. - -``` - -Successful completion indicates that Environment Watch is configured. The setup process will automatically retry three times on parameter entry errors before exiting. If it exits, the process must be restarted.. +1. Open elevated command prompt/powershell. Run the below command. Select Environment Watch + + ``` + C:\Server.Bundle.x.y.z\relsvr.exe setup + + Relativity Server CLI - 24.0.1196 + Copyright (c) 2025, Relativity ODA LLC + + What would you like to setup? + DataGrid + > Environment watch + Exit + ``` + +2. Confirm the Environment Watch setup. + + ``` + Confirm you would like to perform the 'Environment Watch' setup [y/n] (y): y + ``` + +2. Enter the required Relativity parameters. + + ``` + Existing settings do not exist + Enter the Relativity admin username (relativity.admin@kcura.com): relativity.admin@kcura.com + Enter the Relativity admin password: ********* + Enter the Relativity instance url (https://emttest/Relativity): https://emttest/Relativity + Relativity instance is verified + ``` + +4. Enter the required Elastic Stack parameters. + + ``` + Enter the Elasticsearch admin username (elastic): elastic + Enter the Elasticsearch admin password: ********* + Enter the Elasticsearch cluster endpoint URL (https://emttest:9200): https://emttest:9200 + Elasticsearch cluster endpoint URL is verified + Enter the Elasticsearch APM Server endpoint URL (http://emttest:8200): http://emttest:8200 + Enter the Elasticsearch Kibana endpoint URL (http://emttest:5601): http://emttest:5601 + ``` + +5. The CLI will now apply the necessary configurations. Please wait for the process to finish. + + ``` + Elasticsearch Kibana URL is verified + API Key creation and validation completed ------------------------- 100% + OAuth2 client exists ---------------------------------------------- 100% + Relativity secret store updated ----------------------------------- 100% + Relativity logging updated ---------------------------------------- 100% + Relativity toggles updated ---------------------------------------- 100% + Relativity AppDomain monitoring enabled --------------------------- 100% + APM settings updated ---------------------------------------------- 100% + Elasticsearch indexes updated ------------------------------------- 100% + Kibana Tag imported ----------------------------------------------- 100% + Kibana IndexPattern imported -------------------------------------- 100% + Kibana SavedSearch imported --------------------------------------- 100% + Kibana Dashboard imported ----------------------------------------- 100% + Kibana Alert imported --------------------------------------------- 100% + Kibana Custom Role created. --------------------------------------- 100% + Relativity installation SQL record updated ------------------------ 100% + + The Environment Watch setup has been completed. The Relativity Environment Watch installer package should now be installed within each server contained within this Relativity instance. As each server is setup for monitoring, restart all Relativity services within the machine including "kCura Edds Agent Manager," "kCura Edds Web Processing Manager," and "kCura Service Host Manager" to begin sending telemetry to Elasticsearch. + + ``` + +6. Successful completion indicates that Environment Watch is configured. The setup process will automatically retry three times on parameter entry errors before exiting. If it exits, the process must be restarted.. Refer to the [Troubleshooting Guide](troubleshooting/relativity-server-cli.md) if you encounter any issues. diff --git a/docs/troubleshooting/apm-server.md b/docs/troubleshooting/apm-server.md index ed97254..49c7dd5 100644 --- a/docs/troubleshooting/apm-server.md +++ b/docs/troubleshooting/apm-server.md @@ -5,21 +5,9 @@ This document provides troubleshooting guidance for common APM Server issues enc > [!NOTE] > This guide assumes a default APM Server installation path of `C:\elastic\apm-server`. Adjust paths according to your actual installation directory. -## Table of Contents +## Windows Service issues - - [1. Windows Service Issues](#1-windows-service-issues) - - [1.1 APM Server Service Not Starting](#11-apm-server-service-not-starting) - - [1.2 Service Crashes or Stops Unexpectedly](#12-service-crashes-or-stops-unexpectedly) - - [1.3 Permission and Access Issues](#13-permission-and-access-issues) - - [2. Memory Issues](#2-memory-issues) - - [3. Self-Instrumentation](#3-self-instrumentation) - - [4. Service Verification](#4-service-verification) - ---- - -## 1. Windows Service Issues - -### 1.1 APM Server Service Not Starting +### APM Server service not starting **Symptoms:** - APM Server service fails to start @@ -28,55 +16,46 @@ This document provides troubleshooting guidance for common APM Server issues enc **Troubleshooting Steps:** -**Check APM Server Status:** +1. Check APM Server Status: ```powershell Get-Service -Name apm-server ``` -
- Expected response + Expected Response: ``` Status Name DisplayName ------ ---- ----------- Running apm-server Elastic APM Server ``` -
-**Verify Service Configuration:** +2. Verify Service Configuration: ```powershell (Get-CimInstance Win32_Service -Filter "Name = 'apm-server'").StartName ``` -
- Expected response + Expected Response: ``` LocalSystem ``` -
-**Check APM Server Logs:** - 1. Navigate to `C:\Program Files\apm-server\logs\` - 2. Review the latest log files (`apm-server.log`) for error messages - 3. Look for configuration errors or connection issues with Elasticsearch +3. Check APM Server Logs: + 1. Navigate to `C:\Program Files\apm-server\logs\` + 2. Review the latest log files (`apm-server.log`) for error messages + 3. Look for configuration errors or connection issues with Elasticsearch -> [!NOTE] -> For Elasticsearch connection issues, see [Elasticsearch Troubleshooting](elasticsearch.md) + > For Elasticsearch connection issues, see [Elasticsearch Troubleshooting](elasticsearch.md) -**Verify Configuration File:** +3. Verify Configuration File: ```powershell C:\elastic\apm-server\apm-server.exe test config -c "C:\elastic\apm-server\apm-server.yml" ``` -
- Expected response - + + Expected Response: ``` Config OK ``` -
- ---- -### 1.2 Service Crashes or Stops Unexpectedly +### Service crashes or stops unexpectedly **Symptoms:** - APM Server service starts but stops after a short period @@ -85,19 +64,17 @@ This document provides troubleshooting guidance for common APM Server issues enc **Troubleshooting Steps:** -* **Check APM Server Logs:** - See above. +1. Check APM Server Logs (See steps above) -* **Review APM Server Configuration:** - - Check `apm-server.yml` file in `C:\elastic\apm-server\` - - Verify Elasticsearch connection settings (see [Elasticsearch Troubleshooting](elasticsearch.md) for detailed troubleshooting) - - Common configuration issues: - - **TLS**: Ensure correct protocol (`http` vs `https`) - - **Hostname**: Verify correct Elasticsearch server hostname - - **Port**: Confirm correct Elasticsearch port (usually 9200) +2. Review APM Server Configuration: + 1. Check `apm-server.yml` file in `C:\elastic\apm-server\` + 2. Verify Elasticsearch connection settings (see [Elasticsearch Troubleshooting](elasticsearch.md) for detailed troubleshooting) + 3. Common configuration issues: + - **TLS**: Ensure correct protocol (`http` vs `https`) + - **Hostname**: Verify correct Elasticsearch server hostname + - **Port**: Confirm correct Elasticsearch port (usually 9200) -> [!NOTE] -> API keys are the preferred authentication method and expire by default in 6 months. Consider switching from username/password to API key authentication. For API key creation, see [Kibana Troubleshooting](kibana.md). + > API keys are the preferred authentication method and expire by default in 6 months. Consider switching from username/password to API key authentication. For API key creation, see [Kibana Troubleshooting](kibana.md). ```yaml output.elasticsearch: @@ -107,19 +84,19 @@ This document provides troubleshooting guidance for common APM Server issues enc # username: "" # password: "" ``` -> [!NOTE] -> This section in `apm-server.yml` configures how APM Server connects to your Elasticsearch cluster. -> - `hosts`: The URL(s) of your Elasticsearch node(s). -> - `api_key`: The recommended authentication method. -> - `username`/`password`: Legacy authentication (not recommended; use API keys instead). -> For instructions on creating an API key, see [Kibana Troubleshooting](kibana.md). -* **To verify the connection, run:** + > This section in `apm-server.yml` configures how APM Server connects to your Elasticsearch cluster. + > - `hosts`: The URL(s) of your Elasticsearch node(s). + > - `api_key`: The recommended authentication method. + > - `username`/`password`: Legacy authentication (not recommended; use API keys instead). + > For instructions on creating an API key, see [Kibana Troubleshooting](kibana.md). + +3. To verify the connection, run: ```powershell C:\elastic\apm-server\apm-server.exe test output -c "C:\elastic\apm-server\apm-server.yml" ``` -
- Expected output for successful connection + + Expected output for successful connection: ``` elasticsearch: https://:9200... @@ -137,14 +114,11 @@ This document provides troubleshooting guidance for common APM Server issues enc talk to server... OK version: 8.17.3 ``` -
-> [!NOTE] -> To verify Elasticsearch connectivity, see [Elasticsearch Troubleshooting](elasticsearch.md). +4. To verify Elasticsearch connectivity, see [Elasticsearch Troubleshooting](elasticsearch.md). ---- -### 1.3 Permission and Access Issues +### Permission and access issues **Symptoms:** - Access denied errors when starting service @@ -153,15 +127,14 @@ This document provides troubleshooting guidance for common APM Server issues enc **Troubleshooting Steps:** -* The APM Server Windows service runs under Local System account by default. -* Verify access to `C:\elastic\apm-server\` directory. -* Check write permissions to `C:\Program Files\apm-server\logs\` directory. +1. The APM Server Windows service runs under Local System account by default. +2. Verify access to `C:\elastic\apm-server\` directory. +3. Check write permissions to `C:\Program Files\apm-server\logs\` directory. ---- -## 2. Memory Issues +## Memory issues -### 2.1 High Memory Usage +### High memory usage **Symptoms:** - APM Server consuming excessive memory @@ -170,62 +143,54 @@ This document provides troubleshooting guidance for common APM Server issues enc **Troubleshooting Steps:** -* **Check APM Server Logs:** - - Look for memory-related error messages or stack traces. - - Example error message: - ``` - fatal error: out of memory - ``` - -* **Review APM Server Configuration:** - - Check `apm-server.yml` for memory-related settings. - - Common settings to review: - - `apm-server.memory.limit`: Maximum memory APM Server can use. - - `apm-server.memory.queue`: Size of the memory queue for incoming events. - - - Example configuration: - ```yaml - apm-server: - memory: - limit: 512mb - queue: 1000 - ``` - -* **Monitor System Resources:** - - Use Task Manager or Resource Monitor to check APM Server memory usage. - - Compare with configured limits in `apm-server.yml`. - -* **Adjust Memory Settings:** - - If APM Server is using too much memory, consider adjusting the following settings in `apm-server.yml`: - - `apm-server.memory.limit`: Decrease the maximum memory limit. - - `apm-server.memory.queue`: Decrease the memory queue size. - - - Example: - ```yaml - apm-server: - memory: - limit: 256mb - queue: 500 - ``` - -* **Restart APM Server:** - - After making changes, restart the APM Server service: - ```powershell - Restart-Service apm-server - ``` - -
- Expected response - - ``` - WARNING: Waiting for service 'apm-server - (apm-server)' to stop... - ``` -
- ---- - -## 3. Self-Instrumentation +1. Check APM Server Logs: + 1. Look for memory-related error messages or stack traces. Example error message: + ``` + fatal error: out of memory + ``` + +2. Review APM Server Configuration: + 1. Check `apm-server.yml` for memory-related settings. + 2. Common settings to review: + - `apm-server.memory.limit`: Maximum memory APM Server can use. + - `apm-server.memory.queue`: Size of the memory queue for incoming events. + Example configuration: + ```yaml + apm-server: + memory: + limit: 512mb + queue: 1000 + ``` + +3. Monitor System Resources: + 1. Use Task Manager or Resource Monitor to check APM Server memory usage. + 2. Compare with configured limits in `apm-server.yml`. + +4. Adjust Memory Settings: + 1. If APM Server is using too much memory, consider adjusting the following settings in `apm-server.yml`: + - `apm-server.memory.limit`: Decrease the maximum memory limit. + - `apm-server.memory.queue`: Decrease the memory queue size. + - Example: + ```yaml + apm-server: + memory: + limit: 256mb + queue: 500 + ``` + +5. Restart APM Server: + 1. After making changes, restart the APM Server service: + ```powershell + Restart-Service apm-server + ``` + + Expected response: + ``` + WARNING: Waiting for service 'apm-server + (apm-server)' to stop... + ``` + +## Self-Instrumentation **Symptoms:** - Need to monitor APM Server itself for performance and health metrics @@ -233,33 +198,29 @@ This document provides troubleshooting guidance for common APM Server issues enc **Troubleshooting Steps:** -* **Enable Self-Instrumentation:** - - Self-instrumentation allows APM Server to monitor its own performance. - - This feature is configured in the `apm-server.yml` file. - - Refer to Step 3 in the [Install Elasticsearch, Kibana and APM Server - Development Environment](../elasticsearch_setup_development.md) guide for complete configuration details. - -* **Verify Self-Instrumentation:** - - After configuration, restart the APM Server service: - ```powershell - Restart-Service apm-server - ``` - -
- Expected response - - ``` - WARNING: Waiting for service 'apm-server - (apm-server)' to stop... - ``` -
+1. Enable Self-Instrumentation: + 1. Self-instrumentation allows APM Server to monitor its own performance. + 2. This feature is configured in the `apm-server.yml` file. + 3. Refer to Step 3 in the [Install Elasticsearch, Kibana and APM Server - Development Environment](../elasticsearch_setup_development.md) guide for complete configuration details. +2. Verify Self-Instrumentation: + 1. After configuration, restart the APM Server service: + + ```powershell + Restart-Service apm-server + ``` + + Expected response: + ``` + WARNING: Waiting for service 'apm-server + (apm-server)' to stop... + ``` - - Check Kibana to verify that APM Server metrics are being collected +3. Check Kibana to verify that APM Server metrics are being collected ---- -## 4. Service Verification +## Service verification -### 4.1 Verifying APM Server Health +### Verifying APM Server Health **Symptoms:** - Need to confirm APM Server is operating correctly @@ -267,13 +228,12 @@ This document provides troubleshooting guidance for common APM Server issues enc **Troubleshooting Steps:** -* **Check APM Server Status:** +1. Check APM Server Status: ```powershell curl.exe -k "http://:8200/" ``` -
- Expected output + Expected reponse: ```json { "build_date": "...", @@ -282,5 +242,4 @@ This document provides troubleshooting guidance for common APM Server issues enc "version": "8.17.3" } ``` -
diff --git a/docs/troubleshooting/elasticsearch.md b/docs/troubleshooting/elasticsearch.md index b3e766a..26f2295 100644 --- a/docs/troubleshooting/elasticsearch.md +++ b/docs/troubleshooting/elasticsearch.md @@ -5,24 +5,8 @@ This document provides troubleshooting guidance for common Elasticsearch issues > [!NOTE] > This guide assumes a default Elasticsearch installation path of `C:\elastic\elasticsearch`. Adjust paths according to your actual installation directory. -## Table of Contents +## Windows service issues -- [Elasticsearch Troubleshooting](#elasticsearch-troubleshooting) - - [1. Windows Service Issues](#1-windows-service-issues) - - [1.1 Elasticsearch Service Not Starting](#11-elasticsearch-service-not-starting) - - [1.2 Service Crashes or Stops Unexpectedly](#12-service-crashes-or-stops-unexpectedly) - - [2. Memory Issues](#2-memory-issues) - - [2.1 Insufficient Memory Allocation](#21-insufficient-memory-allocation) - - [3. Authentication Issues](#3-authentication-issues) - - [3.1 Username/Password Authentication Problems](#31-usernamepassword-authentication-problems) - - - [4. Service Verification](#4-service-verification) - - [4.1 Verifying Elasticsearch Health](#41-verifying-elasticsearch-health) - ---- - - -## 1. Windows Service Issues > [!IMPORTANT] > Before troubleshooting Elasticsearch, verify that all required Elastic Stack services are running. If any of these are not running, other troubleshooting steps may be irrelevant. @@ -35,8 +19,7 @@ This document provides troubleshooting guidance for common Elasticsearch issues > ```powershell > Get-Service -Name elasticsearch, kibana, apm-server | Format-Table -AutoSize > ``` ->
-> Expected output +> Expected output: > > ``` > Status Name DisplayName @@ -45,13 +28,12 @@ This document provides troubleshooting guidance for common Elasticsearch issues > Running kibana kibana > Running apm-server apm-server > ``` ->
> > - The `kibana` Windows service may not exist if Kibana was not installed as a Windows service (e.g., via NSSM). NSSM is not required; Kibana can be run manually or as a scheduled task. Only check for the `kibana` service if you have installed it as a service. > - If Kibana is not in running state, [click here for Kibana troubleshooting](../troubleshooting/kibana.md). > - If APM is not in running state, [click here for APM troubleshooting](apm-server.md) -### 1.1 Elasticsearch Service Not Starting +### Elasticsearch service not starting **Symptoms:** - Elasticsearch service fails to start @@ -60,7 +42,7 @@ This document provides troubleshooting guidance for common Elasticsearch issues **Troubleshooting Steps:** -* **Check Service Status:** +1. Check Service Status: ```powershell Get-Service -Name elasticsearch-service-x64 | Select-Object Status, StartType, Name ``` @@ -68,17 +50,15 @@ This document provides troubleshooting guidance for common Elasticsearch issues ```powershell Get-Service -Name elasticsearch | Select-Object Status, StartType, Name ``` -
- Expected output - + + Expected output: ``` Status StartType Name ------ --------- ---- Running Automatic elasticsearch ``` -
-* **Verify Service Configuration:** +2. Verify Service Configuration: ```powershell (Get-CimInstance Win32_Service -Filter "Name = 'elasticsearch-service-x64'").StartName ``` @@ -86,29 +66,24 @@ This document provides troubleshooting guidance for common Elasticsearch issues ```powershell (Get-CimInstance Win32_Service -Filter "Name = 'elasticsearch'").StartName ``` -
- Expected output - + + Expected output: ``` LocalSystem ``` -
- -* **Check Elasticsearch Logs:** - - Navigate to the log directory (default: `C:\elastic\elasticsearch-8.17.3\logs\`). - - Review the Elasticsearch log file (`elasticsearch.log`) for error messages. - - Check the slow logs and garbage collection logs if present. - - For every error in the Elasticsearch log, provide troubleshooting for that specific error. -> [!TIP] -> For detailed logging information, refer to the [official Elasticsearch logging documentation](https://www.elastic.co/guide/en/elasticsearch/reference/8.17/logging.html) +3. Check Elasticsearch Logs: + 1. Navigate to the log directory (default: `C:\elastic\elasticsearch-8.17.3\logs\`). + 2. Review the Elasticsearch log file (`elasticsearch.log`) for error messages. + 3. Check the slow logs and garbage collection logs if present. + 4. For every error in the Elasticsearch log, provide troubleshooting for that specific error. -> [!NOTE] -> - Elasticsearch includes a bundled Java runtime, so a separate Java installation is not required. -> - If the `JAVA_HOME` environment variable is defined, Elasticsearch will use the specified Java version instead of the bundled one. -> - If you want to use a specific Java version, ensure `JAVA_HOME` is set correctly. + > - For detailed logging information, refer to the [official Elasticsearch logging documentation](https://www.elastic.co/guide/en/elasticsearch/reference/8.17/logging.html) + > - Elasticsearch includes a bundled Java runtime, so a separate Java installation is not required. + > - If the `JAVA_HOME` environment variable is defined, Elasticsearch will use the specified Java version instead of the bundled one. + > - If you want to use a specific Java version, ensure `JAVA_HOME` is set correctly. -* **Start Service Manually:** +4. Start Service Manually: ```powershell Start-Service elasticsearch-service-x64 ``` @@ -116,15 +91,13 @@ This document provides troubleshooting guidance for common Elasticsearch issues ```powershell Start-Service elasticsearch ``` -
- Expected output - + + Expected output: ``` (No output if successful. Service status will be "Running" after execution.) ``` -
-### 1.2 Service Crashes or Stops Unexpectedly +### Service crashes or stops unexpectedly **Symptoms:** - Elasticsearch service starts but stops after a short period @@ -132,39 +105,33 @@ This document provides troubleshooting guidance for common Elasticsearch issues **Troubleshooting Steps:** -* **Check Elasticsearch Logs:** - - Navigate to `C:\elastic\elasticsearch\logs\` - - Review the Elasticsearch log file (`elasticsearch.log`) for errors - - For every error in the Elasticsearch log, provide troubleshooting for that specific error. -> [!NOTE] -> Always check the latest error in the Elasticsearch log and troubleshoot accordingly. This approach should be followed everywhere. - -* **Verify Disk Space:** - - Ensure sufficient disk space on data and log directories (minimum 15% free) - - Verify data and log files are on separate drives from the Operating System drive. If you store Elasticsearch data on a network share, ensure the share is accessible and has sufficient free space. Some environments may not use mapped drives. - ```powershell - # Check disk space - Get-WmiObject -Class Win32_LogicalDisk | Select-Object DeviceID, @{Name="FreeSpace(%)";Expression={[math]::Round(($_.FreeSpace/$_.Size)*100,2)}} - ``` -
- Expected output - - ``` - DeviceID FreeSpace(%) - -------- ------------ - C: 22.15 - D: 48.92 - ``` -
- -> [!NOTE] -> For port-related issues, see the [Port Configuration Troubleshooting](port-configuration-troubleshooting.md) guide. - ---- - -## 2. Memory Issues - -### 2.1 Insufficient Memory Allocation +1. Check Elasticsearch Logs: + 1. Navigate to `C:\elastic\elasticsearch\logs\` + 2. Review the Elasticsearch log file (`elasticsearch.log`) for errors + 3. For every error in the Elasticsearch log, provide troubleshooting for that specific error. + > Always check the latest error in the Elasticsearch log and troubleshoot accordingly. This approach should be followed everywhere. + +2. Verify Disk Space: + 1. Ensure sufficient disk space on data and log directories (minimum 15% free) + 2. Verify data and log files are on separate drives from the Operating System drive. If you store Elasticsearch data on a network share, ensure the share is accessible and has sufficient free space. Some environments may not use mapped drives. + ```powershell + # Check disk space + Get-WmiObject -Class Win32_LogicalDisk | Select-Object DeviceID, @{Name="FreeSpace(%)";Expression={[math]::Round(($_.FreeSpace/$_.Size)*100,2)}} + ``` + + Expected output: + ``` + DeviceID FreeSpace(%) + -------- ------------ + C: 22.15 + D: 48.92 + ``` + +> For port-related issues, see the [Port Configuration Troubleshooting](pre-requisite-troubleshooting.md) guide. + +## Memory issues + +### Insufficient memory allocation **Symptoms:** - OutOfMemoryError in Elasticsearch logs @@ -173,45 +140,41 @@ This document provides troubleshooting guidance for common Elasticsearch issues **Troubleshooting Steps:** -* **Check Current Memory Usage:** +1. Check Current Memory Usage: ```powershell Get-WmiObject -Class Win32_PhysicalMemory | Measure-Object -Property Capacity -Sum ``` -
- Expected output - + + Expected output: ``` Count : 2 Average : Sum : 34359738368 Property : Capacity ``` -
-> [!NOTE] -> `Sum` is the total RAM in bytes (e.g., 34359738368 bytes = 32 GB). - -* **Review JVM Heap Settings:** - - Edit `C:\elastic\elasticsearch\config\jvm.options` file: - - If the file does not exist, create it. - ``` - # Recommended: Set Xms and Xmx to same value - # Example for system with 8GB+ RAM: - -Xms4g - -Xmx4g - ``` -> [!IMPORTANT] -> Set heap to 50% of available RAM, maximum 32GB. Monitor current memory usage before making changes. -* **Check for Memory Leaks:** - - Monitor heap usage over time - - Look for continuously increasing memory consumption - - Review application logs for memory-related warnings + > `Sum` is the total RAM in bytes (e.g., 34359738368 bytes = 32 GB). + +2. Review JVM Heap Settings: + 1. Edit `C:\elastic\elasticsearch\config\jvm.options` file. (If the file does not exist, create it.) + ``` + # Recommended: Set Xms and Xmx to same value + # Example for system with 8GB+ RAM: + -Xms4g + -Xmx4g + ``` + + > Set heap to 50% of available RAM, maximum 32GB. Monitor current memory usage before making changes. ---- +3. Check for Memory Leaks: + 1. Monitor heap usage over time + 2. Look for continuously increasing memory consumption + 3. Review application logs for memory-related warnings -## 3. Authentication Issues -### 3.1 Username/Password Authentication Problems +## Authentication issues + +### Username/Password authentication issues **Symptoms:** - Login failures @@ -220,13 +183,12 @@ This document provides troubleshooting guidance for common Elasticsearch issues **Troubleshooting Steps:** -* **Verify User Exists:** +1. Verify User Exists: ```powershell curl.exe -k -X GET "https://:9200/_security/user/" -u : ``` -
- Expected output + Expected output: ```json { "": { @@ -238,34 +200,30 @@ This document provides troubleshooting guidance for common Elasticsearch issues } } ``` -
-* **Reset Password:** +2. Reset Password: ```powershell C:\elastic\elasticsearch\bin\elasticsearch-reset-password.bat -u ``` -
- Expected output - + + Expected output: ``` Password for the [] user successfully reset. - ``` -
+ ``` -* **Verify Security Configuration:** - - Check `C:\elastic\elasticsearch\config\elasticsearch.yml`: - ```yaml - xpack.security.enabled: true - ``` -> [!NOTE] -> Also verify that the URL you are using is `https://:9200/` +3. Verify Security Configuration: + 1. Check `C:\elastic\elasticsearch\config\elasticsearch.yml`: + ```yaml + xpack.security.enabled: true + ``` + > Also verify that the URL you are using is `https://:9200/` ---- -## 4. Service Verification -### 4.1 Verifying Elasticsearch Health +## Service verification + +### Verifying Elasticsearch health **Symptoms:** - Uncertainty about cluster status @@ -274,70 +232,63 @@ This document provides troubleshooting guidance for common Elasticsearch issues **Troubleshooting Steps:** -* **Check Cluster Health:** - ```powershell - curl.exe -k -u : -X GET "https://:9200/_cluster/health?pretty" - ``` -
- Expected response for healthy cluster - - ```json - { - "cluster_name": "elasticsearch", - "status": "green", - "timed_out": false, - "number_of_nodes": 3, - "number_of_data_nodes": 3 - } - ``` -
- -* **Verify Node Status:** - ```powershell - curl.exe -k -u : -X GET "https://:9200/_cat/nodes?v" - ``` -
- Expected output +1. Check Cluster Health: + ```powershell + curl.exe -k -u : -X GET "https://:9200/_cluster/health?pretty" + ``` - ``` - ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name - 14 95 28 cdfhilmrstw * - ``` -
+ Expected response for healthy cluster: : + + ```json + { + "cluster_name": "elasticsearch", + "status": "green", + "timed_out": false, + "number_of_nodes": 3, + "number_of_data_nodes": 3 + } + ``` + +2. Verify Node Status: + ```powershell + curl.exe -k -u : -X GET "https://:9200/_cat/nodes?v" + ``` -* **Check Index Health:** - ```powershell - curl.exe -k -u : -X GET "https://:9200/_cat/indices?v" - ``` -
- Expected output + Expected output: + ``` + ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name + 14 95 28 cdfhilmrstw * + ``` - ``` - health status index uuid pri rep docs.count docs.deleted store.size pri.store.size - green open myindex 1a2b3c4d5e6f7g8h9i0j 1 1 1000 0 1.2mb 600kb - ``` -
+3. Check Index Health: + ```powershell + curl.exe -k -u : -X GET "https://:9200/_cat/indices?v" + ``` + + Expected output: + ``` + health status index uuid pri rep docs.count docs.deleted store.size pri.store.size + green open myindex 1a2b3c4d5e6f7g8h9i0j 1 1 1000 0 1.2mb 600kb + ``` -* **Monitor Performance:** - ```powershell - curl.exe -k -u : -X GET "https://:9200/_nodes/stats?pretty" +4. Monitor Performance: + ```powershell + curl.exe -k -u : -X GET "https://:9200/_nodes/stats?pretty" + ``` + + Expected output: + ```json + { + "nodes": { + "node_id": { + "name": "node-1", + "host": "10.0.0.1", + ... + } + } + } ``` -
- Expected output - ```json - { - "nodes": { - "node_id": { - "name": "node-1", - "host": "10.0.0.1", - ... - } - } - } - ``` -
---- diff --git a/docs/troubleshooting/kibana.md b/docs/troubleshooting/kibana.md index 26d5657..1017846 100644 --- a/docs/troubleshooting/kibana.md +++ b/docs/troubleshooting/kibana.md @@ -5,23 +5,8 @@ This document provides troubleshooting guidance for common Kibana issues encount > [!NOTE] > This guide assumes a default Kibana installation path of `C:\elastic\kibana`. Adjust paths according to your actual installation directory. -## Table of Contents - -- [1. Windows Service Issues](#1-windows-service-issues) - - [1.1 Kibana Service Not Starting](#11-kibana-service-not-starting) - - [1.2 Service Crashes or Stops Unexpectedly](#12-service-crashes-or-stops-unexpectedly) -- [2. Authentication Issues](#2-authentication-issues) - - [2.1 Username/Password Authentication Issues](#21-usernamepassword-authentication-issues) -- [3. Memory Issues](#3-memory-issues) - - [3.1 Insufficient Memory Allocation](#31-insufficient-memory-allocation) -- [4. Kibana Encryption Keys Configuration](#4-kibana-encryption-keys-configuration) - - [4.1 Missing or Invalid Encryption Keys](#41-missing-or-invalid-encryption-keys) -- [5. Service Verification](#5-service-verification) - - [5.1 Verifying Kibana Health and Status](#51-verifying-kibana-health-and-status) -- [6. Additional Diagnostic Commands](#6-additional-diagnostic-commands) - - [6.1 Configuration Validation](#61-configuration-validation) -- [7. Installation Issues](#7-installation-issues) - - [7.1 Long Path Issues During Extraction](#71-long-path-issues-during-extraction) +## Windows service issues + --- @@ -31,7 +16,7 @@ This document provides troubleshooting guidance for common Kibana issues encount > [!IMPORTANT] > The following troubleshooting steps apply **only if Kibana was set up as a Windows service** (e.g., using NSSM or a similar tool). If you did not install Kibana as a Windows service, you must ensure that `kibana.bat` is running in a command prompt or as a scheduled task. -### 1.1 Kibana Service Not Starting +### Kibana service not starting **Symptoms:** - Kibana service fails to start @@ -39,145 +24,130 @@ This document provides troubleshooting guidance for common Kibana issues encount **Troubleshooting Steps:** -* **Check Service Status and Configuration:** - ```powershell - Get-Service -Name kibana - ``` -
- Expected output - - ``` - Status Name DisplayName - ------ ---- ----------- - Running kibana Kibana - ``` -
- - ```powershell - Get-CimInstance -ClassName Win32_Service -Filter "Name='kibana'" | Select-Object Name, State, StartMode, StartName - ``` - - Verify the service is running under Local System account (default configuration). - -* **Check Kibana Logs:** - - Navigate to `C:\elastic\kibana\logs\` - - Review the latest log files (`kibana.log`) for error messages. - - Look for configuration errors or Elasticsearch connection issues (for example: `Unable to connect to Elasticsearch at https://:9200`). - -* **Start Service Manually:** - ```powershell - Start-Service kibana - ``` -
- Expected output - - ``` - (No output if successful. Service status will be "Running" after execution.) - ``` -
- -### 1.2 Service Crashes or Stops Unexpectedly - -**Symptoms:** -- Kibana service starts but stops after a short period -- Service status shows "Stopped" unexpectedly -- Users lose access to Kibana interface - -**Troubleshooting Steps:** - -* **Check Kibana Logs:** - - Navigate to `C:\elastic\kibana\logs\` - - Review the latest log files (`kibana.log`) for crash details - - Look for memory issues or connection failures - -* **Review Kibana Configuration:** - - Check `C:\elastic\kibana\config\kibana.yml` file - - Verify Elasticsearch connection settings - - Ensure all required configuration parameters are present - -* **Verify Elasticsearch Connectivity:** - ```powershell - curl.exe -k -u : -X GET "https://:9200/" - ``` -
- Expected output - - ```json - { - "name" : "EMTTEST", - "cluster_name" : "elasticsearch", - "cluster_uuid" : "PwBZoINKQjGZ53WH4gFfBg", - "version" : { - "number" : "8.17.3", - "build_flavor" : "default", - "build_type" : "zip", - "build_hash" : "a091390de485bd4b127884f7e565c0cad59b10d2", - "build_date" : "2025-02-28T10:07:26.089129809Z", - "build_snapshot" : false, - "lucene_version" : "9.12.0", - "minimum_wire_compatibility_version" : "7.17.0", - "minimum_index_compatibility_version" : "7.0.0" - }, - "tagline" : "You Know, for Search" - } - ``` -
+1. Check Service Status and Configuration: + ```powershell + Get-Service -Name kibana + ``` + + Expected output: + ``` + Status Name DisplayName + ------ ---- ----------- + Running kibana Kibana + ``` -* **Check Memory Usage:** - - Monitor Kibana process memory consumption using Task Manager or Resource Monitor. - - Verify sufficient system memory is available. + ```powershell + Get-CimInstance -ClassName Win32_Service -Filter "Name='kibana'" | Select-Object Name, State, StartMode, StartName + ``` +2. Verify the service is running under Local System account (default configuration). ---- +3. Check Kibana Logs: + 1. Navigate to `C:\elastic\kibana\logs\` + 2. Review the latest log files (`kibana.log`) for error messages. + 3. Look for configuration errors or Elasticsearch connection issues (for example: `Unable to connect to Elasticsearch at https://:9200`). -## 2. Authentication Issues +4. Start Service Manually: + ```powershell + Start-Service kibana + ``` + + Expected output + ``` + (No output if successful. Service status will be "Running" after execution.) + ``` -### 2.1 Username/Password Authentication Issues +### Service crashes or stops unexpectedly **Symptoms:** -- Login failures at Kibana interface -- "Invalid username or password" errors -- Users cannot access Kibana dashboards +- Kibana service starts but stops after a short period +- Service status shows "Stopped" unexpectedly +- Users lose access to Kibana interface **Troubleshooting Steps:** -* **Verify User Configuration:** - - Ask the user to log in using the `elastic` username credentials. - - Check `C:\elastic\kibana\config\kibana.yml`: - ```yaml - elasticsearch.username: "" - elasticsearch.password: "" - ``` +1. Check Kibana Logs: + 1. Navigate to `C:\elastic\kibana\logs\` + 2. Review the latest log files (`kibana.log`) for crash details + 3. Look for memory issues or connection failures + +2. Review Kibana Configuration: + 1. Check `C:\elastic\kibana\config\kibana.yml` file + 2. Verify Elasticsearch connection settings + 3. Ensure all required configuration parameters are present -* **Test Elasticsearch Credentials Independently:** - - Use the `elastic` username and password from `kibana.yml` to verify connectivity to Elasticsearch: +3. Verify Elasticsearch Connectivity: ```powershell curl.exe -k -u : -X GET "https://:9200/" ``` -
- Expected output + + Expected output: ```json { "name" : "EMTTEST", "cluster_name" : "elasticsearch", - ... + "cluster_uuid" : "PwBZoINKQjGZ53WH4gFfBg", + "version" : { + "number" : "8.17.3", + "build_flavor" : "default", + "build_type" : "zip", + "build_hash" : "a091390de485bd4b127884f7e565c0cad59b10d2", + "build_date" : "2025-02-28T10:07:26.089129809Z", + "build_snapshot" : false, + "lucene_version" : "9.12.0", + "minimum_wire_compatibility_version" : "7.17.0", + "minimum_index_compatibility_version" : "7.0.0" + }, + "tagline" : "You Know, for Search" } ``` -
+4. Check Memory Usage: + 1. Monitor Kibana process memory consumption using Task Manager or Resource Monitor. + 2. Verify sufficient system memory is available. +## Authentication issues ---- +### Username/Password authentication issues +**Symptoms:** +- Login failures at Kibana interface +- "Invalid username or password" errors +- Users cannot access Kibana dashboards + +**Troubleshooting Steps:** +1. Verify User Configuration: + 1. Ask the user to log in using the `elastic` username credentials. + 2. Check `C:\elastic\kibana\config\kibana.yml`: + ```yaml + elasticsearch.username: "" + elasticsearch.password: "" + ``` + +2. Test Elasticsearch Credentials Independently: + 1. Use the `elastic` username and password from `kibana.yml` to verify connectivity to Elasticsearch: + ```powershell + curl.exe -k -u : -X GET "https://:9200/" + ``` + + Expected output: + + ```json + { + "name" : "EMTTEST", + "cluster_name" : "elasticsearch", + ... + } + ``` > [!NOTE] -> For port-related issues, see the [Port Configuration Troubleshooting](port-configuration-troubleshooting.md) guide. +> For port-related issues, see the [Port Configuration Troubleshooting](pre-requisite-troubleshooting.md) guide. ---- -## 3. Memory Issues +## Memory issues -### 3.1 Insufficient Memory Allocation +### Insufficient memory allocation **Symptoms:** - Kibana becomes unresponsive @@ -186,34 +156,33 @@ This document provides troubleshooting guidance for common Kibana issues encount **Troubleshooting Steps:** -* **Check Current Memory Usage:** - ```powershell - Get-Process -Name node | Where-Object {$_.ProcessName -eq "node"} | Select-Object WorkingSet, VirtualMemorySize - ``` -
- Expected output +1. Check Current Memory Usage: + ```powershell + Get-Process -Name node | Where-Object {$_.ProcessName -eq "node"} | Select-Object WorkingSet, VirtualMemorySize + ``` + + Expected output: + ``` + WorkingSet VirtualMemorySize + ---------- ---------------- + 12345678 23456789 + ``` - ``` - WorkingSet VirtualMemorySize - ---------- ---------------- - 12345678 23456789 - ``` -
+2. Review Out of Memory Errors in Logs: * **Review Out of Memory Errors in Logs:** - Check for "out of memory" or "heap" errors in `C:\elastic\kibana\logs\kibana.log`. > [!NOTE] > Out of memory errors typically indicate insufficient system memory or improper Node.js heap settings. Review the error details in the log for specific causes and recommended actions. -* **Verify Disk Space:** - - Insufficient disk space can also cause memory-related failures. - - For disk space troubleshooting steps, see [Verify Disk Space](elasticsearch.md#verify-disk-space). +3. Verify Disk Space: + 1. Insufficient disk space can also cause memory-related failures. + 2. For disk space troubleshooting steps, see [Verify Disk Space](elasticsearch.md#verify-disk-space). ---- -## 4. Kibana Encryption Keys Configuration +## Kibana encryption keys configuration -### 4.1 Missing or Invalid Encryption Keys +### Missing or invalid encryption keys **Symptoms:** - Kibana fails to start with encryption-related errors @@ -222,29 +191,26 @@ This document provides troubleshooting guidance for common Kibana issues encount **Troubleshooting Steps:** -* **Generate Encryption Keys:** - ```powershell - cd C:\elastic\kibana\bin - .\kibana-encryption-keys.bat generate - ``` -
- Expected output +1. Generate Encryption Keys: + ```powershell + cd C:\elastic\kibana\bin + .\kibana-encryption-keys.bat generate + ``` - ``` - xpack.encryptedSavedObjects.encryptionKey: "" - xpack.reporting.encryptionKey: "" - xpack.security.encryptionKey: "" - ``` -
+ Expected output: + ``` + xpack.encryptedSavedObjects.encryptionKey: "" + xpack.reporting.encryptionKey: "" + xpack.security.encryptionKey: "" + ``` -* **Configure Required Encryption Keys in kibana.yml:** - - Copy the generated lines above and paste them into your `C:\elastic\kibana\config\kibana.yml` file. +2. Configure Required Encryption Keys in kibana.yml: + 1. Copy the generated lines above and paste them into your `C:\elastic\kibana\config\kibana.yml` file. ---- -## 5. Service Verification +## Service verification -### 5.1 Verifying Kibana Health and Status +### Verifying Kibana health and status **Symptoms:** - Need to confirm Kibana is operating correctly @@ -253,54 +219,46 @@ This document provides troubleshooting guidance for common Kibana issues encount **Troubleshooting Steps:** -* **Check Kibana Status:** - ```powershell - curl.exe -k -u : -X GET "http://:5601/api/status" - ``` -
- Expected output - - ```json - { - "name": "kibana", - "version": { - "number": "8.x.x" - }, - "status": { - "overall": { - "level": "available" - } - } - } - ``` -
- ---- - -## 6. Additional Diagnostic Commands - -### 6.1 Configuration Validation +1. Check Kibana Status: + ```powershell + curl.exe -k -u : -X GET "http://:5601/api/status" + ``` + + Expected output: + ```json + { + "name": "kibana", + "version": { + "number": "8.x.x" + }, + "status": { + "overall": { + "level": "available" + } + } + } + ``` +## Additional diagnostic commands -**Validate YAML syntax** - ```powershell - C:\elastic\kibana\bin\kibana.bat - ``` -
- Expected output +### Configuration validation - ``` - # There is no built-in --validate-config option in Kibana 8.x. - # Configuration is validated when Kibana starts. - # If there are errors in kibana.yml, they will be shown in the console output and Kibana will fail to start. - ``` -
+1. Validate YAML syntax + ```powershell + C:\elastic\kibana\bin\kibana.bat + ``` + + Expected output: ---- + ``` + # There is no built-in --validate-config option in Kibana 8.x. + # Configuration is validated when Kibana starts. + # If there are errors in kibana.yml, they will be shown in the console output and Kibana will fail to start. + ``` -## 7. Installation Issues +## Installation issues -### 7.1 Long Path Issues During Extraction +### Long path issues during extraction **Symptoms:** - Errors when extracting the `kibana-8.xx.x-windows-x86_64.zip` file. @@ -308,20 +266,18 @@ This document provides troubleshooting guidance for common Kibana issues encount **Troubleshooting Steps:** -1. **Enable Long Paths in Windows:** +1. Enable Long Paths in Windows: Windows must be configured to support long file paths. This can be enabled via the Local Group Policy Editor. + 1. Run `gpedit.msc` to open the Local Group Policy Editor. - a. Run `gpedit.msc` to open the Local Group Policy Editor. - - ![gpedit.msc](../../resources/troubleshooting-images/gpedit.png) - - b. Navigate to `Computer Configuration` → `Administrative Templates` → `System` → `Filesystem`. + ![gpedit.msc](../../resources/troubleshooting-images/gpedit.png) + 2. Navigate to `Computer Configuration` → `Administrative Templates` → `System` → `Filesystem`. - ![Filesystem settings in Group Policy Editor](../../resources/troubleshooting-images/administrativetemplate.png) + ![Filesystem settings in Group Policy Editor](../../resources/troubleshooting-images/administrativetemplate.png) - c. Double-click on `Enable Win32 long paths` and set it to **Enabled**. + 3. Double-click on `Enable Win32 long paths` and set it to **Enabled**. - ![Enable long paths policy](../../resources/troubleshooting-images/kibanaextract.png) + ![Enable long paths policy](../../resources/troubleshooting-images/kibanaextract.png) -2. **Re-extract the Kibana zip file:** - After enabling long path support, attempt to extract the `kibana-8.xx.x-windows-x86_64.zip` file again. +2. Re-extract the Kibana zip file: + 1. After enabling long path support, attempt to extract the `kibana-8.xx.x-windows-x86_64.zip` file again. \ No newline at end of file diff --git a/docs/troubleshooting/monitoring-agent-and-otel-collector.md b/docs/troubleshooting/monitoring-agent-and-otel-collector.md index fe63c60..21aedef 100644 --- a/docs/troubleshooting/monitoring-agent-and-otel-collector.md +++ b/docs/troubleshooting/monitoring-agent-and-otel-collector.md @@ -6,163 +6,128 @@ This document provides a stepwise troubleshooting guide for the Relativity Envir > The Relativity Environment Watch Windows service is responsible for auto-configuring and managing the Open Telemetry Collector on each server. There are no expectations for the user to configure the collector directly. -## Table of Contents - -1. [Verify the Elastic Stack Servers are Running](#1-verify-the-elastic-stack-servers-are-running) -2. [Verify the Monitoring Agent Hosts are Present and Sending Metrics](#2-verify-the-monitoring-agent-hosts-are-present-and-sending-metrics) -3. [Verify the Environment Watch Service and Open Telemetry Collector](#3-verify-the-environment-watch-service-and-open-telemetry-collector) -4. [Verify the Open Telemetry Collector Logs](#4-verify-the-open-telemetry-collector-logs) -5. [Additional Pre-requisite Access Checks](#5-additional-pre-requisite-access-checks) - - 5.1. [BCP Share Access Verification](#51-bcp-share-access-verification) - - 5.2. [Kepler and Web (SSL Certificate) Verification](#52-kepler-and-web-ssl-certificate-verification) - - 5.3. [Relativity Service Account Verification](#53-relativity-service-account-verification) -6. [Installer and Service Errors](#6-installer-and-service-errors) - - 6.1. [User Not in Local Security Policy](#61-user-not-in-local-security-policy) - - 6.2. [Invalid Secrets](#62-invalid-secrets) - ---- - - -## 1. Verify the Elastic Stack Servers are Running +## Verify the Elastic Stack Servers are running First, ensure that the core Elastic Stack components (Elasticsearch, Kibana, and APM Server) are running and accessible. If you are not seeing any data in dashboards, this strongly suggests a problem with the Elastic Stack itself. - **Check Service Accessibility:** -- **Elasticsearch:** - ```powershell - curl.exe -k -u : -X GET "https://:9200/" - ``` -
- Expected output - - A JSON response with cluster details indicates success. -
- -- **Kibana:** - ```powershell - curl.exe -k -u : -X GET "http://:5601/api/status" - ``` -
- Expected output +1. Elasticsearch: + ```powershell + curl.exe -k -u : -X GET "https://:9200/" + ``` - A JSON response with `"level": "available"` indicates success. -
- -- **APM Server:** - ```powershell - curl.exe -k "http://:8200/" - ``` -
- Expected output + Expected output: + A JSON response with cluster details indicates success. + +2. Kibana: + ```powershell + curl.exe -k -u : -X GET "http://:5601/api/status" + ``` - A JSON response with version details indicates success. -
+ Expected output: + A JSON response with `"level": "available"` indicates success. + +3. APM Server: + ```powershell + curl.exe -k "http://:8200/" + ``` + Expected output: + A JSON response with version details indicates success. -**If a service is not running, refer to its specific troubleshooting guide:** -- [Elasticsearch Troubleshooting](elasticsearch.md) -- [Kibana Troubleshooting](kibana.md) -- [APM Server Troubleshooting](apm-server.md) +4. If a service is not running, refer to its specific troubleshooting guide: + - [Elasticsearch Troubleshooting](elasticsearch.md) + - [Kibana Troubleshooting](kibana.md) + - [APM Server Troubleshooting](apm-server.md) - For port-related issues, see the [Pre-requisite Troubleshooting](pre-requisite-troubleshooting.md) guide. -## 2. Verify the Monitoring Agent Hosts are Present and Sending Metrics +## Verify the Monitoring Agent Hosts are Present and Sending Metrics If the Elastic Stack is running, check that your monitoring agent hosts are present in Kibana and are sending metrics. If hosts are missing or not updating, the issue may be with the agent or host configuration. -**How to check:** - -1. **Monitoring Agents Dashboard** - - Open Kibana and navigate to the **Monitoring Agents** dashboard. - - Confirm the "Last Check-In" field is updating for each monitored server. - - **Expected:** The "Last Check-In" timestamp updates regularly. - - Confirm the **Agent Version** column value is displayed for all hosts and shows the same version for every host. - - **Expected:** The "Agent Version" column value should be the same for all hosts and visible for each host. - ![Monitoring_Agent](../../resources/troubleshooting-images/Monitoring_Agent.png) - -2. **Host Infrastructure Overview** - - Open Kibana and navigate to the **Host Infrastructure Overview** dashboard. - - Confirm CPU Utilization, RAM Utilization and Disk Utilization column values are present for each host. - - **Expected result:** CPU Utilization, RAM Utilization and Disk Utilization column values are populated for the host. - ![Host_Infrastructure](../../resources/troubleshooting-images/Host_Infrastructure.png) - -3. **Discover Query** - - Open Kibana, go to **Discover**. - - Select the APM data view. - - Run the following query: - ``` - service.name: "relsvr_infrawatch_agent" and host.hostname: "" - ``` - - **Expected result:** You should see logs, data, and traces from the Open Telemetry Collector for the specified host. - ![Discover](../../resources/troubleshooting-images/EW_Discover.png) - ---- - - -## 3. Verify the Environment Watch Service and Open Telemetry Collector +1. Monitoring Agents Dashboard + 1. Open Kibana and navigate to the **Monitoring Agents** dashboard. + 2. Confirm the "Last Check-In" field is updating for each monitored server. + **Expected:** The "Last Check-In" timestamp updates regularly. + 4. Confirm the **Agent Version** column value is displayed for all hosts and shows the same version for every host. + **Expected:** The "Agent Version" column value should be the same for all hosts and visible for each host. + + ![Monitoring_Agent](../../resources/troubleshooting-images/Monitoring_Agent.png) + +2. Host Infrastructure Overview + 1. Open Kibana and navigate to the **Host Infrastructure Overview** dashboard. + 2. Confirm CPU Utilization, RAM Utilization and Disk Utilization column values are present for each host. + 3. Expected result: CPU Utilization, RAM Utilization and Disk Utilization column values are populated for the host. + + ![Host_Infrastructure](../../resources/troubleshooting-images/Host_Infrastructure.png) + +3. Discover Query + 1. Open Kibana, go to **Discover**. + 2. Select the APM data view. + 3. Run the following query: + ``` + service.name: "relsvr_infrawatch_agent" and host.hostname: "" + ``` + + Expected result: + You should see logs, data, and traces from the Open Telemetry Collector for the specified host. + + ![Discover](../../resources/troubleshooting-images/EW_Discover.png) + + +## Verify the Environment Watch Service and Open Telemetry Collector If a specific host is not reporting, check that the Environment Watch Windows service is running on that host. This service is responsible for managing the Open Telemetry Collector. Also verify that the Open Telemetry Collector process is running, its port is listening, logs are being generated, and the auto-generated YAML file exists. -**How to check:** - 1. Open PowerShell and run the following to check the service status: ```powershell Get-Service 'Relativity Environment Watch' ``` -
- Expected output - + + Expected output: ``` Status Name Display Name ------ ---- --------- Running Relativity Envi... Relativity Environment Watch ``` -
- If the service is not running, restart it: + + +2. If the service is not running, restart it: ```powershell Restart-Service -Name "Relativity Environment Watch" ``` -
- Expected output - + + Expected output: No output if successful. Service status will be "Running" after execution. -
-2. Verify logs are being generated: - - Check the directory: - `C:\ProgramData\Relativity\EnvironmentWatch\Services\InfraWatchAgent\Logs` - - Ensure files like `otelcol-relativity-stderr.log` and `otelcol-relativity-stdout.log` are present and updating. -
- Expected output - - Log files are present and their timestamps are updating as new data is written. -
-3. Open Task Manager and look for `otelcol-relativity.exe` under the Processes tab. - - Alternatively, use PowerShell: - ```powershell - Get-Process -Name otelcol-relativity - ``` -
- Expected output - ``` +3. Verify logs are being generated: + 1. Check the directory: + `C:\ProgramData\Relativity\EnvironmentWatch\Services\InfraWatchAgent\Logs` + 2. Ensure files like `otelcol-relativity-stderr.log` and `otelcol-relativity-stdout.log` are present and updating. + Expected output: + Log files are present and their timestamps are updating as new data is written. + +4. Open Task Manager and look for `otelcol-relativity.exe` under the Processes tab. Alternatively, use PowerShell: + ```powershell + Get-Process -Name otelcol-relativity + ``` + + Expected output: + ``` Handles NPM(K) PM(K) WS(K) CPU(s) Id SI ProcessName ------- ------ ----- ----- ------ -- -- ----------- ... ... ... ... ... ... ... otelcol-relativity ``` *(If not running, no output.)* -
-4. Check port status in the [Port Configuration Troubleshooting](port-configuration-troubleshooting.md) guide. -> [!NOTE] -> When running, both `rel-envwatch-service` and `otelcol-relativity` processes are present. When stopped, neither process is present. Port 4318 is listening only when service is running. - ---- +5. Check port status in the [Port Configuration Troubleshooting](pre-requisite-troubleshooting.md) guide. + > When running, both `rel-envwatch-service` and `otelcol-relativity` processes are present. When stopped, neither process is present. Port 4318 is listening only when service is running. -## 4. Verify the Open Telemetry Collector Logs +## Verify the Open Telemetry Collector Logs If the service and collector are running but data is still missing, check the logs for errors or misconfiguration. @@ -174,117 +139,106 @@ If the service and collector are running but data is still missing, check the lo > [!WARNING] > The Windows Event Log for `Relativity.EnvironmentWatch` uses a very small default size (1 MB). This can quickly fill up and cause important log entries (like "Everything is ready") to be lost. Increase the log size to at least 500 MB in Event Viewer to ensure you do not miss critical events. -**How to check:** - -> [!NOTE] > The below log files will be available in v2025.08.27 EW BUNDLE RELEASE. -* All logs are written to: - - `C:\ProgramData\Relativity\EnvironmentWatch\Services\InfraWatchAgent\Logs\otelcol-relativity-stderr.log` (errors) - - `C:\ProgramData\Relativity\EnvironmentWatch\Services\InfraWatchAgent\Logs\otelcol-relativity-stdout.log` (general logs) -* All information and error log entries are also written to the Windows event log: - - `Relativity.EnvironmentWatch` -* Use PowerShell to check error logs: - ```powershell - Get-EventLog Relativity.EnvironmentWatch | Where { $_.EntryType -eq "Error" } - ``` -
- Expected output - - ``` - Index Time EntryType Source InstanceID Message - ----- ---- --------- ------ ---------- ------- - 123 7/29/2025 Error Relativity.EnvironmentWatch ... [Error details] - ``` - (No output if no errors are present.) -
- -* Use PowerShell to check information logs: - ```powershell - Get-EventLog Relativity.EnvironmentWatch | Where { $_.EntryType -eq "Information" } - ``` -
- Expected output - - ``` - Index Time EntryType Source InstanceID Message - ----- ---- --------- ------ ---------- ------- - 124 7/29/2025 Information Relativity.EnvironmentWatch ... [Info details] - ``` -
- -* Use PowerShell to check for "Everything is ready" message: - ```powershell - Get-EventLog Relativity.EnvironmentWatch | Where { $_.Message -like "*Everything is ready*" } - ``` -
- Expected output - - ``` - Index Time EntryType Source InstanceID Message - ----- ---- --------- ------ ---------- ------- - 123 7/29/2025 Information Relativity.EnvironmentWatch ... Everything is ready - ``` -
-* You can also check the Event Viewer logs for `Relativity.EnvironmentWatch`: - - Open **Event Viewer** (eventvwr.msc) - - Navigate to **Application and Service Logs** > **Relativity.EnvironmentWatch** - - Review any Error or Warning entries for troubleshooting details + +1. All logs are written to: + - `C:\ProgramData\Relativity\EnvironmentWatch\Services\InfraWatchAgent\Logs\otelcol-relativity-stderr.log` (errors) + - `C:\ProgramData\Relativity\EnvironmentWatch\Services\InfraWatchAgent\Logs\otelcol-relativity-stdout.log` (general logs) +2. All information and error log entries are also written to the Windows event log: + - `Relativity.EnvironmentWatch` + 1. Open **Event Viewer** (eventvwr.msc) + 2. Navigate to **Application and Service Logs** > **Relativity.EnvironmentWatch** + 3. Review any Error or Warning entries for troubleshooting details +3. Use PowerShell to check error logs: + ```powershell + Get-EventLog Relativity.EnvironmentWatch | Where { $_.EntryType -eq "Error" } + ``` - ![EventViewer_RelativityEnvironmentWatch](../../resources/troubleshooting-images/EventViewer_RelativityEnvironmentWatch.png) + Expected output: + ``` + Index Time EntryType Source InstanceID Message + ----- ---- --------- ------ ---------- ------- + 123 7/29/2025 Error Relativity.EnvironmentWatch ... [Error details] + ``` + (No output if no errors are present.) + +4. Use PowerShell to check information logs: + ```powershell + Get-EventLog Relativity.EnvironmentWatch | Where { $_.EntryType -eq "Information" } + ``` + + Expected output: + ``` + Index Time EntryType Source InstanceID Message + ----- ---- --------- ------ ---------- ------- + 124 7/29/2025 Information Relativity.EnvironmentWatch ... [Info details] + ``` + +5. Use PowerShell to check for "Everything is ready" message: + ```powershell + Get-EventLog Relativity.EnvironmentWatch | Where { $_.Message -like "*Everything is ready*" } + ``` ---- + Expected output: + ``` + Index Time EntryType Source InstanceID Message + ----- ---- --------- ------ ---------- ------- + 123 7/29/2025 Information Relativity.EnvironmentWatch ... Everything is ready + ``` ---- +6. You can also check the Event Viewer logs for `Relativity.EnvironmentWatch`: + 1. Open **Event Viewer** (eventvwr.msc) + 2. Navigate to **Application and Service Logs** > **Relativity.EnvironmentWatch** + 3. Review any Error or Warning entries for troubleshooting details + + ![EventViewer_RelativityEnvironmentWatch](../../resources/troubleshooting-images/EventViewer_RelativityEnvironmentWatch.png) + -## 5. Additional Pre-requisite Access Checks +## Additional Pre-requisite Access Checks If the above steps do not resolve the issue, verify the following access and configuration requirements: -### 5.1. BCP Share Access Verification -> [!NOTE] -> If you are not logged in as the Relativity Service Account, use the commands below. +### BCP Share Access Verification -```powershell -$cred = Get-Credential # use the Relativity Service Account -Start-Process "powershell.exe" -Credential $cred -ArgumentList '-NoExit' -# In the new PowerShell session window, run: -# Example: Replace with your actual BCP share path -Test-Path "\\bcp-share\relativity-data" -``` -
-Expected output +1. If you are not logged in as the Relativity Service Account, use the commands below. + ```powershell + $cred = Get-Credential # use the Relativity Service Account + Start-Process "powershell.exe" -Credential $cred -ArgumentList '-NoExit' + ``` -True +2. In the new PowerShell session window, run: + ``` + # Example: Replace with your actual BCP share path + Test-Path "\\bcp-share\relativity-data" + ``` -*(If the path is accessible; otherwise, False.)* -
+ Expected output: + True + *(If the path is accessible; otherwise, False.)* > [!NOTE] > For troubleshooting steps related to the Relativity Secret Store, please refer to the [Pre-requisite Troubleshooting](pre-requisite-troubleshooting.md) guide. -### 5.2. Kepler and Web (SSL Certificate) Verification -- The required web certificate must be installed on the server. For certificate troubleshooting, see the [Pre-requisite Troubleshooting](pre-requisite-troubleshooting.md) guide. -- Verify Kepler API status: - ```powershell - curl.exe -k -u : -X GET "https:///relativity.rest/api/relativity-infrawatch-services/v1/infrawatch-manager/getkeplerstatusasync" - ``` -
- Expected Result +### Kepler and Web (SSL Certificate) Verification +1. The required web certificate must be installed on the server. For certificate troubleshooting, see the [Pre-requisite Troubleshooting](pre-requisite-troubleshooting.md) guide. +2. Verify Kepler API status: + ```powershell + curl.exe -k -u : -X GET "https:///relativity.rest/api/relativity-infrawatch-services/v1/infrawatch-manager/getkeplerstatusasync" + ``` + + Expected Result: + JSON response with `"status": "OK"`. - JSON response with `"status": "OK"`. -
+### Relativity Service Account Verification -### 5.3. Relativity Service Account Verification -> [!NOTE] -> For service account requirements and troubleshooting, see [Environment_Watch_Installer](../environment_watch_installation.md) +For service account requirements and troubleshooting, see [Environment_Watch_Installer](../environment_watch_installation.md) ---- -## 6. Installer and Service Errors +## Installer and Service Errors This section covers issues related to the Environment Watch installer and the underlying Windows services it manages. -### 6.1. User Not in Local Security Policy +### User Not in Local Security Policy **Symptoms:** - The product installation fails with an error indicating the user is not added to the Local Security Policy. @@ -292,24 +246,23 @@ This section covers issues related to the Environment Watch installer and the un ![User not in Local Security Policy](../../resources/troubleshooting-images/user-not-added-in-local-security.png) **Troubleshooting Steps:** -1. **Add User to Local Security Policy:** - - Open the **Local Security Policy** editor (`secpol.msc`). - - Navigate to `Local Policies` -> `User Rights Assignment`. - - Ensure the user account running the installer has the necessary permissions, such as "Log on as a service". +1. Add User to Local Security Policy: + 1. Open the **Local Security Policy** editor (`secpol.msc`). + 2. Navigate to `Local Policies` -> `User Rights Assignment`. + 3. Ensure the user account running the installer has the necessary permissions, such as "Log on as a service". - ![User added to Local Security Policy](../../resources/troubleshooting-images/useraddedtolocalsecurity.png) + ![User added to Local Security Policy](../../resources/troubleshooting-images/useraddedtolocalsecurity.png) -### 6.2. Invalid Secrets +### Invalid Secrets **Symptoms:** - The installation fails with an error message "one or more secrets are invalid". - ![One or more secrets are invalid](../../resources/troubleshooting-images/one-or-more-secrets-invalid.png) + ![One or more secrets are invalid](../../resources/troubleshooting-images/one-or-more-secrets-invalid.png) **Troubleshooting Steps:** -1. **Verify CLI Setup:** Ensure that the one-time setup using the `relsvr.exe setup` command was executed successfully and completed without errors. This step is required to generate the necessary secrets. +1. Verify CLI Setup: Ensure that the one-time setup using the `relsvr.exe setup` command was executed successfully and completed without errors. This step is required to generate the necessary secrets. ---- For additional troubleshooting, refer to the main documentation: [Environment Watch Installer](../environment_watch_installation.md) diff --git a/docs/troubleshooting/pre-requisite-troubleshooting.md b/docs/troubleshooting/pre-requisite-troubleshooting.md index b59c37b..651d34b 100644 --- a/docs/troubleshooting/pre-requisite-troubleshooting.md +++ b/docs/troubleshooting/pre-requisite-troubleshooting.md @@ -2,14 +2,6 @@ This document provides troubleshooting steps for common pre-requisites like port configuration and Secret Store access. -## Table of Contents - -- [Port Configuration](#port-configuration-troubleshooting) -- [Secret Store](#secret-store-troubleshooting) -- [Certificate](#certificate-troubleshooting) - ---- - ## Port Configuration Troubleshooting ### Default Port Reference @@ -23,163 +15,144 @@ The following table summarizes the default ports used by the Elastic Stack and E | APM Server | 8200 | HTTP/HTTPS | ✅ | | APM agent data ingestion | | OTEL Collector | 4318 | HTTP | | | OTLP data reception (HTTP) for local traffic (localhost). This deployment uses the agent model, with a collector on each server. See the [OpenTelemetry agent documentation](https://opentelemetry.io/docs/collector/deployment/agent/) for more details. | ---- + ### Elasticsearch Port Issues -#### Symptoms: +**Symptoms:** - Elasticsearch fails to bind to default ports. - "Address already in use" errors in logs. - Cannot access Elasticsearch via HTTP/HTTPS. -#### Troubleshooting Steps: - -* **Check if Ports are in Use:** - Verify that ports 9200 and 9300 are listening. - ```powershell - netstat -an | findstr ":9200" - netstat -an | findstr ":9300" - ``` -
- Expected output +**Troubleshooting Steps:** - ``` - TCP 0.0.0.0:9200 0.0.0.0:0 LISTENING - TCP 0.0.0.0:9300 0.0.0.0:0 LISTENING - ``` -
+1. Check if Ports are in use: + Verify that ports 9200 and 9300 are listening. + ```powershell + netstat -an | findstr ":9200" + netstat -an | findstr ":9300" + ``` -* **Identify Conflicting Processes:** - If a port is in use by another application, identify the process. - ```powershell - Get-NetTCPConnection -LocalPort 9200 -State Listen - Get-NetTCPConnection -LocalPort 9300 -State Listen - ``` + Expected output: + ``` + TCP 0.0.0.0:9200 0.0.0.0:0 LISTENING + TCP 0.0.0.0:9300 0.0.0.0:0 LISTENING + ``` -* **Test Elasticsearch Connectivity:** - ```powershell - curl.exe -k -u : -X GET "https://:9200/" - ``` +2. Identify Conflicting Processes. If a port is in use by another application, identify the process. + ```powershell + Get-NetTCPConnection -LocalPort 9200 -State Listen + Get-NetTCPConnection -LocalPort 9300 -State Listen + ``` -* **Verify Network Binding:** - Check `C:\elastic\elasticsearch\config\elasticsearch.yml` configuration: - ```yaml - network.host: 0.0.0.0 # For all interfaces - ``` +3. Test Elasticsearch Connectivity: + ```powershell + curl.exe -k -u : -X GET "https://:9200/" + ``` ---- +4. Verify Network Binding: + Check `C:\elastic\elasticsearch\config\elasticsearch.yml` configuration: + ```yaml + network.host: 0.0.0.0 # For all interfaces + ``` ### Kibana Port Issues -#### Symptoms: +**Symptoms:** - Kibana fails to bind to the default port. - "EADDRINUSE" errors in logs. - Cannot access Kibana web interface. -#### Troubleshooting Steps: - -* **Check if Port is in Use:** - ```powershell - netstat -an | findstr ":5601" - ``` -
- Expected output +**Troubleshooting Steps:** - ``` - TCP 0.0.0.0:5601 0.0.0.0:0 LISTENING - ``` -
+1. Check if Port is in Use: + ```powershell + netstat -an | findstr ":5601" + ``` + + Expected output: + ``` + TCP 0.0.0.0:5601 0.0.0.0:0 LISTENING + ``` -* **Test Kibana Connectivity:** - ```powershell - (curl.exe -s -k -u : -X GET "http://:5601/api/status" | ConvertFrom-Json).status.overall | ConvertTo-Json -Depth 10 - ``` +2. Test Kibana Connectivity: + ```powershell + (curl.exe -s -k -u : -X GET "http://:5601/api/status" | ConvertFrom-Json).status.overall | ConvertTo-Json -Depth 10 + ``` -* **Verify Network Binding:** - Check `C:\elastic\kibana\config\kibana.yml` configuration: - ```yaml - server.host: "0.0.0.0" # For all interfaces - ``` +3. Verify Network Binding. Check `C:\elastic\kibana\config\kibana.yml` configuration: + ```yaml + server.host: "0.0.0.0" # For all interfaces + ``` ---- ### APM Server Port Issues -#### Symptoms: +**Symptoms:** - APM Server fails to bind to the default port. - "Address already in use" errors in logs. - APM agents cannot connect to the server. -#### Troubleshooting Steps: - -* **Check if Port is in Use:** - ```powershell - netstat -an | findstr ":8200" - ``` -
- Expected output - - ``` - TCP 0.0.0.0:8200 0.0.0.0:0 LISTENING - ``` -
- -* **Test APM Server Connectivity:** - ```powershell - curl.exe -k "http://:8200/" - ``` -
- Expected output - - ```json - { - "build_date": "...", - "build_sha": "...", - "publish_ready": true, - "version": "8.17.3" - } - ``` -
+**Troubleshooting Steps:** -* **Verify Network Binding:** - Check `C:\elastic\apm-server\apm-server.yml` configuration: - ```yaml - host: "0.0.0.0:8200" - ``` +1. Check if Port is in Use: + ```powershell + netstat -an | findstr ":8200" + ``` + + Expected output: + ``` + TCP 0.0.0.0:8200 0.0.0.0:0 LISTENING + ``` + +2. Test APM Server Connectivity: + ```powershell + curl.exe -k "http://:8200/" + ``` + + Expected output: + + ```json + { + "build_date": "...", + "build_sha": "...", + "publish_ready": true, + "version": "8.17.3" + } + ``` ---- +3. Verify Network Binding. Check `C:\elastic\apm-server\apm-server.yml` configuration: + ```yaml + host: "0.0.0.0:8200" + ``` ### OpenTelemetry Collector Port Issues -#### Symptoms: +**Symptoms:** - The `otelcol-relativity.exe` process is running, but no data is being sent. - Port 4318 is not in a listening state. -#### Troubleshooting Steps: +**Troubleshooting Steps:** -* **Check if Port is in Use:** - This port is used by the OpenTelemetry Collector to receive data. The `Relativity Environment Watch` service must be running. - ```powershell - netstat -an | findstr ":4318" - ``` -
- Expected output - - ``` - TCP 0.0.0.0:4318 0.0.0.0:0 LISTENING - ``` -
+1. Check if Port is in Use. This port is used by the OpenTelemetry Collector to receive data. The `Relativity Environment Watch` service must be running. + ```powershell + netstat -an | findstr ":4318" + ``` + + Expected output: + ``` + TCP 0.0.0.0:4318 0.0.0.0:0 LISTENING + ``` - You can also use `Get-NetTCPConnection`: - ```powershell - Get-NetTCPConnection -LocalPort 4318 -State Listen - ``` + You can also use `Get-NetTCPConnection`: + ```powershell + Get-NetTCPConnection -LocalPort 4318 -State Listen + ``` ---- ### General Port Troubleshooting -#### Firewall Rules: +#### Firewall Rules Ensure that Windows Firewall or any other network security software is not blocking the required ports. You may need to create inbound rules to allow traffic on these ports. **Example for Kibana (port 5601):** @@ -187,23 +160,21 @@ The following table summarizes the default ports used by the Elastic Stack and E New-NetFirewallRule -DisplayName "Kibana Web Interface" -Direction Inbound -Protocol TCP -LocalPort 5601 -Action Allow ``` -#### Network Connectivity: +#### Network Connectivity Use `Test-NetConnection` to verify that a remote server can reach the port. ```powershell Test-NetConnection -ComputerName -Port ``` -
- Expected output - + + Expected output: ``` ComputerName : RemoteAddress : RemotePort : TcpTestSucceeded : True ``` -
---- + ## Secret Store Troubleshooting @@ -217,8 +188,7 @@ Verify that the Secret Store host is reachable on port 443. Test-NetConnection -ComputerName -Port 443 ``` -
-Expected output +Expected output: ``` ComputerName : @@ -226,7 +196,6 @@ RemoteAddress : RemotePort : 443 TcpTestSucceeded : True ``` -
#### API Access Test @@ -248,8 +217,8 @@ TcpTestSucceeded : True To check the seal status of the Secret Store, run the following script in an elevated PowerShell ISE. -1. Replace `` with the thumbprint you copied. -2. Replace `` with the URL you copied. +- Replace `` with the thumbprint you copied. +- Replace `` with the URL you copied. ```powershell $thumbprint = "" @@ -270,8 +239,7 @@ $response = Invoke-RestMethod -Uri "$url/v1/sys/seal-status" -Certificate $cert $response | ConvertTo-Json ``` -
-Expected output (for a healthy, unsealed store) +Expected output (for a healthy, unsealed store): ```json { @@ -290,120 +258,116 @@ $response | ConvertTo-Json "storage_type": "raft" } ``` -
## Certificate Troubleshooting ### SSL/TLS Certificate Issues -#### Symptoms: +**Symptoms:** - SSL handshake failures - "certificate verify failed" errors - Unable to establish secure connections - Browser shows "not secure" warning for Elasticsearch URL -#### Troubleshooting Steps: +**Troubleshooting Steps:** -* **Verify Secure URL** - - The master node domain name URL should be secure for Elasticsearch node servers, agent servers, and web servers. - - The data node domain name URL should be secured for Elasticsearch node servers. +1. Verify Secure URL + 1. The master node domain name URL should be secure for Elasticsearch node servers, agent servers, and web servers. + 2. The data node domain name URL should be secured for Elasticsearch node servers. -* **Install SSL Certificate in Trusted Store** +2. Install SSL Certificate in Trusted Store If your browser shows a "not secure" warning when accessing the Elasticsearch URL, you may need to install the certificate into your trusted store. - a. In your browser, view the certificate details and export the root certificate authority (CA) certificate. Save it to a local directory. + 1. In your browser, view the certificate details and export the root certificate authority (CA) certificate. Save it to a local directory. - b. Double-click the downloaded certificate file and click **Install Certificate**. + 2. Double-click the downloaded certificate file and click **Install Certificate**. - ![Install Certificate](../../resources/troubleshooting-images/installcertificate.png) + ![Install Certificate](../../resources/troubleshooting-images/installcertificate.png) - c. Select **Local Machine** and click **Next**. + 3. Select **Local Machine** and click **Next**. - ![Select Local Machine](../../resources/troubleshooting-images/localmachine.png) + ![Select Local Machine](../../resources/troubleshooting-images/localmachine.png) - d. Select **Place all certificates in the following store**, click **Browse**, and select **Trusted Root Certification Authorities**. Click **OK**, then **Next**, and **Finish**. + 4. Select **Place all certificates in the following store**, click **Browse**, and select **Trusted Root Certification Authorities**. Click **OK**, then **Next**, and **Finish**. - e. To confirm, open the Microsoft Management Console (MMC): + 5. To confirm, open the Microsoft Management Console (MMC): i. Run `mmc.exe`. ii. Go to **File > Add/Remove Snap-in...**. iii. Select **Certificates** and click **Add**. - ![Add/Remove Snap-in](../../resources/troubleshooting-images/Add-removesnipin.png) + ![Add/Remove Snap-in](../../resources/troubleshooting-images/Add-removesnipin.png) - ![Add Certificates Snap-in](../../resources/troubleshooting-images/addcerts.png) + ![Add Certificates Snap-in](../../resources/troubleshooting-images/addcerts.png) - f. Choose **Computer account** and click **Next**, then **Finish**, and **OK**. + 6. Choose **Computer account** and click **Next**, then **Finish**, and **OK**. - ![Select Computer Account](../../resources/troubleshooting-images/clickcomputeraccount.png) - - g. Expand **Certificates (Local Computer) > Trusted Root Certification Authorities > Certificates** and verify your certificate is listed. + ![Select Computer Account](../../resources/troubleshooting-images/clickcomputeraccount.png) - h. Close your browser and reopen the Elasticsearch URL. It should now show as secure. + 7. Expand **Certificates (Local Computer) > Trusted Root Certification Authorities > Certificates** and verify your certificate is listed. - ![Secure Connection](../../resources/troubleshooting-images/sslenabled.png) + 8. Close your browser and reopen the Elasticsearch URL. It should now show as secure. -* **Verify Certificate Path in `elasticsearch.yml`** + ![Secure Connection](../../resources/troubleshooting-images/sslenabled.png) - Ensure the `elasticsearch.yml` file points to the correct certificate files. +3. Verify Certificate Path in `elasticsearch.yml`. + 1. Ensure the `elasticsearch.yml` file points to the correct certificate files. - - Check `C:\elastic\elasticsearch\config\elasticsearch.yml`: - ```yaml - xpack.security.transport.ssl: - keystore.path: certs/transport.p12 - truststore.path: certs/transport.p12 - ``` + 2. Check `C:\elastic\elasticsearch\config\elasticsearch.yml`: + ```yaml + xpack.security.transport.ssl: + keystore.path: certs/transport.p12 + truststore.path: certs/transport.p12 + ``` -* **Check Elasticsearch Logs for SSL Errors** - - Navigate to `C:\elastic\elasticsearch\logs\`. - - Review the `elasticsearch.log` file for any SSL-related errors. - - For every error in the Elasticsearch log, provide troubleshooting for that specific error. +4. Check Elasticsearch Logs for SSL Errors + 1. Navigate to `C:\elastic\elasticsearch\logs\`. + 2. Review the `elasticsearch.log` file for any SSL-related errors. + 3. For every error in the Elasticsearch log, provide troubleshooting for that specific error. ---- ### TLS Version Mismatch -#### Symptoms: +**Symptoms:** - **Connection Failure**: During installation of Environment Watch Installer, an error pop-up may appear with a message like: > The HTTP request submitted to the server `https://:9200/` failed because of an unexpected error. Verify the server is accessible and URL is correct. Check the logs for more details or refer to the following troubleshooting guide. - **Log Errors**: Logs on the application's server indicate a failure to establish a secure connection or mention outdated TLS versions (like TLSv1.0 or TLSv1.1). - "SSLHandshakeException: client requested protocol TLSv1 is not enabled or supported in server context" -#### Cause: +**Cause:** The machine's .NET Framework is not configured to use strong cryptography, preventing it from negotiating a secure connection with modern servers that require TLS 1.2 or higher. By default, some .NET applications may attempt to use older, insecure TLS versions. Default TLS version supported by Elasticsearch is TLSv1.2 and TLSv1.3, hence causing issue -#### Troubleshooting Steps: +**Troubleshooting Steps:** To resolve this, the .NET Framework on the machine must be configured to use the system's default security protocols, which allows it to use newer versions like TLS 1.2/1.3. 1. **Verify TLS Settings**: - - Navigate to **Control Panel > Network and Internet > Internet Options > Advanced**, ensure that **Use TLS 1.2** and **Use TLS 1.3** are checked. + 1. Navigate to **Control Panel > Network and Internet > Internet Options > Advanced**, ensure that **Use TLS 1.2** and **Use TLS 1.3** are checked. 2. **Open Registry Editor**: - - Press `Win + R`, type `regedit`, and press Enter. + 1. Press `Win + R`, type `regedit`, and press Enter. 3. **Navigate to .NET Framework Registry Keys**: A new value will need to be added in two locations. - - **For 64-bit applications:** - ``` - HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v4.0.30319 - ``` - - **For 32-bit applications running on a 64-bit machine:** - ``` - HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NETFramework\v4.0.30319 - ``` + 1. **For 64-bit applications:** + ``` + HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v4.0.30319 + ``` + 2. **For 32-bit applications running on a 64-bit machine:** + ``` + HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NETFramework\v4.0.30319 + ``` 4. **Create the `SchUseStrongCrypto` Value**: - - In each of the keys mentioned above, right-click in the right-hand pane and select **New > DWORD (32-bit) Value**. - - Name the new value `SchUseStrongCrypto`. - - Double-click the new value and set its **Value data** to `1`. Click **OK**. + 1. In each of the keys mentioned above, right-click in the right-hand pane and select **New > DWORD (32-bit) Value**. + 2. Name the new value `SchUseStrongCrypto`. + 3. Double-click the new value and set its **Value data** to `1`. Click **OK**. - > **Note:** This registry key forces .NET 4.x applications to use strong cryptography, enabling support for newer TLS versions. + > This registry key forces .NET 4.x applications to use strong cryptography, enabling support for newer TLS versions. -5. **Reboot the System**: - - A system reboot is required for the changes to take effect. +5. **Reboot the System**. A system reboot is required for the changes to take effect. 6. **Verify the Fix**: - - After rebooting, re-run the Environment Watch installer or restart the application. The connection to Elasticsearch should now succeed. \ No newline at end of file + 1. After rebooting, re-run the Environment Watch installer or restart the application. The connection to Elasticsearch should now succeed. \ No newline at end of file diff --git a/docs/troubleshooting/relativity-server-cli.md b/docs/troubleshooting/relativity-server-cli.md index 2a40b74..8e46546 100644 --- a/docs/troubleshooting/relativity-server-cli.md +++ b/docs/troubleshooting/relativity-server-cli.md @@ -4,45 +4,33 @@ This document provides troubleshooting guidance for common Relativity Server CLI ## Prerequisite Access -> [!IMPORTANT] -> Before running the CLI, you must have access to all of the following: -> - **Relativity Admin account** -> - **Secret Store** -> - **Kepler (SSL certificate)** -> - **Elasticsearch** -> - **Kibana** -> - **APM Server** -> -> For verification steps for all prerequisites, see [Additional Pre-requisite Access Checks](monitoring-agent-and-otel-collector.md#additional-pre-requisite-access-checks). -> [!NOTE] -> This guide assumes the Relativity Server bundle was extracted to `C:\server-bundle` or a similar directory chosen by the user. - -## Table of Contents - -1. [APM Integration and Data View](#1-apm-integration-and-data-view) -2. [Kibana Encryption Keys Issues](#2-kibana-encryption-keys-issues) -3. [Common CLI Errors](#3-common-cli-errors) - ---- +Before running the CLI, you must have access to all of the following: +- **Relativity Admin account** +- **Secret Store** +- **Kepler (SSL certificate)** +- **Elasticsearch** +- **Kibana** +- **APM Server** +For verification steps for all prerequisites, see [Additional Pre-requisite Access Checks](monitoring-agent-and-otel-collector.md#additional-pre-requisite-access-checks). +> This guide assumes the Relativity Server bundle was extracted to `C:\server-bundle` or a similar directory chosen by the user. -## 1. APM Integration and Data View +## APM Integration and Data View -### 1.1 Elastic APM Integration Package +### Elastic APM Integration Package The Elastic APM integration package must be added and configured in Kibana before running the CLI setup. **Troubleshooting Steps:** 1. **Verify if APM Integration is already installed:** - - Login to Kibana and navigate to **Management** > **Integrations**. - - Search for "Elastic APM" in the search bar. - - Check if "Elastic APM" appears under **Installed integrations**. + 1. Login to Kibana and navigate to **Management** > **Integrations**. + 2. Search for "Elastic APM" in the search bar. + 3. Check if "Elastic APM" appears under **Installed integrations**. + 4. If APM integration is not installed, follow the detailed setup instructions in the [Elastic APM Integration Setup Guide](../elasticsearch_setup_development.md#step-4-additional-setup-and-verification). - - If APM integration is not installed, follow the detailed setup instructions in the [Elastic APM Integration Setup Guide](../elasticsearch_setup_development.md#step-4-additional-setup-and-verification). - -![Installed_Integrations](../../resources/troubleshooting-images/installed_integrations.png) + ![Installed_Integrations](../../resources/troubleshooting-images/installed_integrations.png) > [!NOTE] > If you encounter errors such as "Package not found" or installation timeouts during APM integration package installation, refer to the official [Elastic APM Integration Setup Guide](../elasticsearch_setup_development.md#elastic-apm-integration-package). @@ -51,9 +39,8 @@ To verify connectivity, always use the following format for verification command ```powershell curl.exe -k -u : -X GET "https://:9200/" ``` -
-Expected Output +Expected Output: ```json { "name" : "EMTTEST", @@ -73,96 +60,90 @@ curl.exe -k -u : -X GET "https://:9200/" "tagline" : "You Know, for Search" } ``` -
-### 1.2 Data View +### Data View #### Self Instrumentation Data View -Self-instrumentation allows you to monitor the CLI's own metrics, traces, and logs. +Self-instrumentation allows you to monitor the CLI's own metrics, traces, and logs. See [Self-Instrumentation](apm-server.md#self-instrumentation) for setup and troubleshooting instructions. -See [Self-Instrumentation](apm-server.md#self-instrumentation) for setup and troubleshooting instructions. -> [!NOTE] > Without the self-instrumentation Data View, you may not see CLI self-monitoring data in Kibana dashboards. -* To check if the APM Data View is created in Kibana: +To check if the APM Data View is created in Kibana: + +1. Open a browser and go to `http://:5601` +2. Log in using elastic credentials +3. Confirm the APM Data View is present: -* Open a browser and go to `http://:5601` -* Log in using elastic credentials -* Confirm the APM Data View is present: + ![dataview](../../resources/troubleshooting-images/dataview.png) - ![dataview](../../resources/troubleshooting-images/dataview.png) ---- -## 2. Kibana Encryption Keys Issues +## Kibana Encryption Keys Issues -* Kibana encryption keys must be added to `C:\elastic\kibana\config\kibana.yml` before running CLI setup. -**If encryption keys are missing or invalid, the CLI will display errors such as:** +Kibana encryption keys must be added to `C:\elastic\kibana\config\kibana.yml` before running CLI setup. If encryption keys are missing or invalid, the CLI will display errors such as: ``` [ERROR] Missing required Kibana encryption key: xpack.encryptedSavedObjects.encryptionKey [ERROR] Missing required Kibana encryption key: xpack.reporting.encryptionKey [ERROR] Missing required Kibana encryption key: xpack.security.encryptionKey ``` -> [!NOTE] -> If you encounter encryption key validation errors or warnings in the CLI, follow the instructions in [Kibana Encryption Keys Configuration](kibana.md#5-kibana-encryption-keys-configuration). ---- +If you encounter encryption key validation errors or warnings in the CLI, follow the instructions in [Kibana Encryption Keys Configuration](kibana.md#5-kibana-encryption-keys-configuration). -## 3. Common CLI Errors +## Common CLI Errors This section covers common errors encountered during the Environment Watch and Data Grid Audit setup workflows. -### 3.1 Unauthorized Access +### Unauthorized Access **Symptoms:** - The CLI returns an "Unauthorized" error for Relativity or Elasticsearch credentials. - ![Relativity Unauthorized Error](../../resources/EWRelativityUnauthorized.png) - ![Elasticsearch Unauthorized Error](../../resources/EWElasticUnauthorized.png) - ![DataGrid Unauthorized Error](../../resources/Issue1-Unauthorized.png) + ![Relativity Unauthorized Error](../../resources/EWRelativityUnauthorized.png) + ![Elasticsearch Unauthorized Error](../../resources/EWElasticUnauthorized.png) + ![DataGrid Unauthorized Error](../../resources/Issue1-Unauthorized.png) **Troubleshooting Steps:** 1. **Verify Relativity Credentials:** Ensure the Relativity admin username and password are correct. 2. **Verify Elasticsearch Credentials:** Ensure the Elasticsearch admin username and password are correct. -### 3.2 Incorrect Server URLs +### Incorrect Server URLs **Symptoms:** - The CLI returns an error indicating that a server URL is incorrect. - ![Incorrect Relativity URL](../../resources/EWRelativityUrlIncorrect.png) - ![Incorrect Elasticsearch URL](../../resources/EWElasticUrlIncorrect.png) - ![Incorrect APM URL](../../resources/EWAPMUrlIncorrect.png) - ![Incorrect Kibana URL](../../resources/EWKibanaUrlIncorrect.png) + ![Incorrect Relativity URL](../../resources/EWRelativityUrlIncorrect.png) + ![Incorrect Elasticsearch URL](../../resources/EWElasticUrlIncorrect.png) + ![Incorrect APM URL](../../resources/EWAPMUrlIncorrect.png) + ![Incorrect Kibana URL](../../resources/EWKibanaUrlIncorrect.png) **Troubleshooting Steps:** 1. **Verify URLs:** Check the URLs for Relativity, Elasticsearch, APM, and Kibana to ensure they are correct and accessible. -### 3.3 Incorrect Elasticsearch Server Credentials +### Incorrect Elasticsearch Server Credentials **Symptoms:** - The CLI specifically flags Elasticsearch credentials as incorrect. - ![Invalid Elasticsearch Credentials](../../resources/troubleshooting-images/invalidelasticcreds.png) - ![DataGrid Elasticsearch Credentials Error](../../resources/Issue2-ElasticUrlCredentialsWrong.png) + ![Invalid Elasticsearch Credentials](../../resources/troubleshooting-images/invalidelasticcreds.png) + ![DataGrid Elasticsearch Credentials Error](../../resources/Issue2-ElasticUrlCredentialsWrong.png) **Troubleshooting Steps:** 1. **Verify Elasticsearch Credentials and URL:** Double-check the Elasticsearch admin username, password, and server URL. -### 3.4 Retry Limit Reached +### Retry Limit Reached **Symptoms:** - The CLI exits after multiple failed attempts to enter correct parameters. - ![Relativity Max Attempts Reached](../../resources/EWRelativityMaxAttempts.png) - ![Elasticsearch Max Attempts Reached](../../resources/EWElasticMaxAttempts.png) - ![APM Max Attempts Reached](../../resources/EWAPMMaxAttempts.png) - ![Kibana Max Attempts Reached](../../resources/EWKibanaMaxAttempts.png) - ![DataGrid Retry Limit Reached](../../resources/Issue3-RetryLimit-Reached.png) + ![Relativity Max Attempts Reached](../../resources/EWRelativityMaxAttempts.png) + ![Elasticsearch Max Attempts Reached](../../resources/EWElasticMaxAttempts.png) + ![APM Max Attempts Reached](../../resources/EWAPMMaxAttempts.png) + ![Kibana Max Attempts Reached](../../resources/EWKibanaMaxAttempts.png) + ![DataGrid Retry Limit Reached](../../resources/Issue3-RetryLimit-Reached.png) **Troubleshooting Steps:** 1. **Restart the CLI:** The maximum number of attempts has been reached. Rerun the setup command: @@ -170,7 +151,4 @@ This section covers common errors encountered during the Environment Watch and D relsvr.exe setup ``` ---- - -For full setup instructions, see [Relativity_Server_CLI Setup](../relativity_server_cli_setup.md). - +For full setup instructions, see [Relativity_Server_CLI Setup](../relativity_server_cli_setup.md). \ No newline at end of file diff --git a/docs/troubleshooting/relativity_alerts_troubleshooting.md b/docs/troubleshooting/relativity_alerts_troubleshooting.md index d5bd69c..a0adbf7 100644 --- a/docs/troubleshooting/relativity_alerts_troubleshooting.md +++ b/docs/troubleshooting/relativity_alerts_troubleshooting.md @@ -4,7 +4,7 @@ This guide will help you resolve common issues that may arise during the install ## During Installation -#### Issue 1: "Relativity Alert object data is unavailable or stale because Environment Watch is not currently enabled in this instance. Please see {doc page} for instructions on how to deploy and enable Environment Watch." error is displayed. +#### Issue 1: "Relativity Alert object data is unavailable or stale because Environment Watch is not currently enabled in this instance. Please see ... for instructions on how to deploy and enable Environment Watch." error is displayed. ![ew-toggle-disabled-exception.png](../../resources/relativity-alerts-troubleshooting-images/ew-toggle-disabled-exception.png) @@ -14,7 +14,7 @@ This guide will help you resolve common issues that may arise during the install **Troubleshooting Steps:** -1. Please execute the Relativity Server CLI first time setup again to resolve this issue. +1. Run the Relativity Server CLI first time setup again to resolve this issue. #### Issue 2: "This operation cannot be completed because an HTTP request to the Elasticsearch server failed. Please try again. If the issue persists, ensure that the Elasticsearch server is accessible and the configuration is correct." error is displayed. @@ -38,8 +38,7 @@ This guide will help you resolve common issues that may arise during the install ![alert-agent-doesnot-exist2.png](../../resources/relativity-alerts-troubleshooting-images/alert-agent-doesnot-exist2.png) **Causes:** -1. You might be missing the "Alert Manager" agent. - +- You might be missing the "Alert Manager" agent. **Troubleshooting Steps:** @@ -58,21 +57,21 @@ Please follow below steps to verify and create a new agent if needed: a. Navigate to the Agents tab in relativity. - b. Click on the "New Agent" button. + b. Click on the **New Agent** button. - c. In the Agent Type dropdown, choose "Alert Manager". + c. In the Agent Type dropdown, choose **Alert Manager**. d. Enter 30 as the Run Interval (in seconds). e. Choose an active Agent Server that is assigned to one of the existing Resource Pools. - f. Click on Save + f. Click on **Save** -![alert-agent-type.png](../../resources/relativity-alerts-troubleshooting-images/alert-agent-type.png) -![testvm-hopper-instance.png](../../resources/relativity-alerts-troubleshooting-images/testvm-hopper-instance.png) -![verifyfields.png](../../resources/relativity-alerts-troubleshooting-images/verifyfields.png) + ![alert-agent-type.png](../../resources/relativity-alerts-troubleshooting-images/alert-agent-type.png) + ![testvm-hopper-instance.png](../../resources/relativity-alerts-troubleshooting-images/testvm-hopper-instance.png) + ![verifyfields.png](../../resources/relativity-alerts-troubleshooting-images/verifyfields.png) -
Once these steps are completed, the "Alert Manager" agent should now be active and functioning correctly. +Once these steps are completed, the "Alert Manager" agent should now be active and functioning correctly. #### Issue 4: "The Alert Manager agent is unable to retrieve current alert state information from Kibana. Please ensure that the Alert Manager agent is enabled. To see accurate alert state information please navigate to Kibana." error is displayed. @@ -93,20 +92,20 @@ Please follow below steps to verify and enable the agent if needed: b. Filter the Name field to check 'Alert Manager' agent is present. - c. Click on 'Alert Manager' agent and check if the agent is disabled, if so, proceed to enable the agent by following step 2 below. + c. Click on **Alert Manager** agent and check if the agent is disabled, if so, proceed to enable the agent by following step 2 below. -2. Enable the 'Alert Manager' agent +2. Enable the **Alert Manager** agent - a. Navigate to the Agents tab in Relativity. + a. Navigate to the **Agents** tab in Relativity. b. Filter the Name field to check 'Alert Manager' agent is present. - c. Click on 'Alert Manager' agent and click on 'Enable Agent'
- - ![alert-agent-enabled.png](../../resources/relativity-alerts-troubleshooting-images/alert-agent-enabled.png)
+ c. Click on **Alert Manager** agent and click on **Enable Agent** + + ![alert-agent-enabled.png](../../resources/relativity-alerts-troubleshooting-images/alert-agent-enabled.png)
-
Once these steps are completed, the "Alert Manager" agent should now be Enabled and functioning correctly. +Once these steps are completed, the "Alert Manager" agent should now be Enabled and functioning correctly. #### Issue 5: "The Alert Manager agent interval is currently set to more than 15 minutes. If you want alert state information to be retrieved from Kibana more frequently, please update the agent interval." warning is displayed. @@ -132,8 +131,9 @@ Your "Alert Manager" agent might have exceeded the interval. Please follow below b. Filter the Name field to check 'Alert Manager' agent is present. - c. Click on 'Alert Manager' agent and Update Agent interval to a value less then 900 seconds (15 minutes). Recommended value is 30 (default).
- ![verify-alert-agent-interval.png](../../resources/relativity-alerts-troubleshooting-images/verify-alert-agent-interval.png) + c. Click on 'Alert Manager' agent and Update Agent interval to a value less then 900 seconds (15 minutes). Recommended value is 30 (default). + + ![verify-alert-agent-interval.png](../../resources/relativity-alerts-troubleshooting-images/verify-alert-agent-interval.png) #### Issue 6: "Unable to retrieve alert state information from Kibana. Please ensure that Elasticsearch and Kibana are functional." error is displayed. @@ -163,11 +163,11 @@ Your "Alert Manager" agent might have exceeded the interval. Please follow below 1. Please verify that user has the System Administrator permissions to perform the required actions. #### Issue 8: "Relativity Alert object data is unavailable or stale because Environment Watch is not currently enabled in this instance. Please see {doc page} for instructions on how to deploy and enable Environment Watch" error is displayed. + ![ew-toggle-disabled-error.png](../../resources/relativity-alerts-troubleshooting-images/ew-toggle-disabled-error.png) **Causes:** 1. Relativity Server CLI First-Time Setup not executed properly. **Troubleshooting Steps:** -1. Please execute the Relativity Server CLI first time setup again to resolve this issue. - +1. Please execute the Relativity Server CLI first time setup again to resolve this issue. \ No newline at end of file