Maintaining Elasticsearch in File Access Manager

[#1]

• Prerequisites
• Installing Elasticsearch
• Checking the Logs
• Elasticsearch status reporting
• Moving Elasticsearch
• Next Steps
• Reindexing Elasticsearch
• Uninstalling Elasticsearch
• Troubleshooting
• Steps to update JRE on the server running Elasticsearch service

Important: Elasticsearch is a indexing service and data store that stores all the events collected from the Activity Monitors, and enables fast searching for retrieval and reporting purpose. Elasticsearch is a heavy-weight tool and typically requires a DEDICATED server to run.

It is recommended that you install Elasticsearch in a location that is accessible by the Event Manager, Business Website, and Central Permissions Collection/Central Data Classification services with almost equal network latency.

Prerequisites

Before installing or moving a service, the File Access Manager Server Installer must be installed on each of the servers involved (for example, in case of a move, on both the server a service is currently on, and the server it will be moved to).

Installing Elasticsearch

Begin by setting up the Elasticsearch installation configuration on the core services server:

1. Launch the File Access Manager Server Installer on the core services server.
2. Choose Use an existing IdentityIQ File Access Manager Database.
3. Enter the database credentials and click Next.
4. On the Action Select page, choose Create/Edit installation configuration and click Next.
5. If the server where you want to install the service is not in the server list, add it by entering the details of the server. For new servers, provide the Server Local Name (short NETBIOS name) and Server FQDN (Fully Qualified Domain Name). Then click Add to add this server to the Server List.
6. Click Next.
7. Check File Access Manager Elasticsearch and click on the drop-down list to choose the server to install it on.
8. Click on the option to Define manual credentials, and enter a User Name and Password. These credentials can be used later to connect to Elasticsearch independently.
9. Enter the full database path for the Elasticsearch database.
10. Click Next.
11. Choose Save Configuration only and click Next.
12. Click Finish to exit the server installer and open the Installation Summary window.
13. Check the installer logs as described in the Checking the Logs section below.

Continue by installing Elasticsearch on the server where you want it to be located:

1. On the server where Elasticsearch will be installed, install the File Access Manager Server Installer if it is not present already.
2. Launch the File Access Manager Server Installer.
3. Choose Use an existing IdentityIQ File Access Manager Database
4. Enter the database credentials to connect to the same database you created or connected to in the section above.
5. On the Action Select page, choose Perform Current Server's Installation Tasks and click Next.
6. The next panel lists the services the installer will install on this server. Verify that Elasticsearch is listed in the configuration and click Next.
7. Click Next, then click Finish to complete the installation and open the Installation Summary window.
8. Check the installer logs as described in the Checking the Logs section above.

Checking the Logs

1. In the the Installation Summary window, check the Open Installation Log box, then click Finish. This opens the installation log.
2. In the installation log, search for the term ERROR (using all capital letters) to see if any errors occurred during installation of the Elasticsearch service.
3. If you encounter errors, you can try to troubleshoot the error, or contact SailPoint Support for further assistance. If you contact Support, you will need to send the install log to analyze the issue.

Elasticsearch status reporting

File Access Manager Scheduled Task Handler service is responsible to poll for the health of Elasticsearch. This service sends out a query to Elasticsearch. In order to report the health of Elasticsearch in the Health Center and the Services Dashboard on the FAM business website, the server hosting the Scheduled Task Handler service should be able to communicate with the server hosting the Elasticsearch service on port 9200.

Moving Elasticsearch

Services may sometimes need to be moved due to architecture or hardware changes in your environment.

1. In order to move Elasticsearch from the server it is running currently to a new server, follow the instructions to uninstall Elasticsearch from the current server and install Elasticsearch on the new server.

Next Steps

1. Since Elasticsearch has been installed on a new server, you will not have access to the older activities especially if you have Elasticsearch database present on the old server. In order to obtain access to those older activities, you will have to copy the indices from the Elasticsearch Database on the old server to the new server. 
2. Indices in the Elasticsearch database are stored <ElasticDB location>\nodes\0\indices. Copy all the folders in this location to the Elasticsearch database on the new server.

Reindexing Elasticsearch

If you have a retention policy and you have purged activities from Elasticsearch or your Elasticsearch server is no longer available and you have installed Elasticsearch again to address the issue, in order bring in all the activities into Elasticsearch for querying and reporting, you will have to reindex Elasticsearch from the File Access Manager database.

All recorded activities are stored in Elasticsearch and the File Access Manager database. The re-index task takes all activities from the database and restores them to Elasticsearch.

To perform a re-index:

1. Open the File Access Manager Admin Client.
2. Click on the Health Center.
3. Click on Actions > Events Maintenance > Re-index Events.

This will launch a re-index task. The amount of time it takes for the task to complete depends on the number of events being re-indexed from the database back into Elasticsearch. Contact SailPoint Support if you need further assistance.

Uninstalling Elasticsearch

Elasticsearch service will not be available in the File Access Manager server installer uninstall options. Follow the below steps if are intending to uninstall Elasticsearch.

1. Login to the server where Elasticsearch service is installed.
2. Open a command prompt window as an administrator and run the command. "X:\Program Files\SailPoint\elasticsearch-5.1.1\bin\elasticsearch-service.bat" remove
3. Open the services snap-in (services.msc) and confirm that the elasticsearch service (File Access Manager Elasticsearch 5.1.1) is removed. If you still see the service, open a CMD terminal as an administrator and execute the command sc.exe delete elasticsearch-service-x64
4. To uninstall the WatchDog service, launch the File Access Manager Server Installer.
5. Choose Use an existing IdentityIQ File Access Manager Database.
6. Enter the database credentials and click Next.
7. In the Action Select page, select Uninstall File Access Manager features from the current server and click Next.
8. Note: This screen wouldn't show any installed services on the server. 
9. Click Next to start the uninstall process for the WatchDog service.
10. When the progress bar shows Finished, click Next to open the Installation Summary window. Check the installer logs as described in the Checking the Logs section above.
11. Rename/move/delete the folder where Elasticsearch was previously installed.
12. Rename/move/delete the Elasticsearch database folder.
13. If present, delete the WatchDog service installation folder.
14. If present, delete the row representing the WatchDog service from the [whiteops].[install_service], [whiteops].[installed_service] tables. To perform this operation, connect to the FAM database and run the query. Select * from [whiteops].[installed_service] where id=20
15. Ensure that the value in the status_enum_id column is set to 5. If not, run the query to update the values appropriately.

    BEGIN TRAN

    UPDATE [whiteops].[installed_service]

    SET

    [status_enum_id] = 5, [installed_server_id]=NULL, [version] = NULL

    WHERE [id] = 20

16. If the query has been executed successfully and you see one row updated, execute the below command to commit the transaction.

    COMMIT TRAN

17. If present, delete the row representing the watchdog service from the [whiteops].[installed_service] and [whiteops].[install_service] tables.
18. Delete the registry entries (if any) related to the watchdog service.

Troubleshooting

Issue: Elasticsearch Java Heap - Out of Memory
Resolution: Sometimes out of memory (Java heap) issues are encountered when large volumes of events are coming into Elasticsearch. This can be mitigated with the following steps:

1. Go to the X:\Program Files\SailPoint\elasticsearch-5.1.1\bin folder and run the following command:
   elasticsearch-service.bat manager
2. This opens a configuration dialog box. Go to the Java tab.
3. In the Java Options box, look for -Xms3g and -Xmx3g . If these exist, change them to the new required memory value (for example,  -Xms8g and -Xmx8g). These are the minimum and maximum values of memory Java can utilize with Elasticsearch. The minimum and maximum values should be the same.
4. Important: Set the Initial memory pool and Maximum memory pool values to the same values as above. (for example, 8096MB and 8096MB to match the 8g in the Java Options box from the example above.)
5. At a maximum, never set these two values above half of the total system RAM. Elasticsearch should never take more than half of the system RAM.
6. Restart the Elasticsearch service.

Viewing the Elasticsearch dashboard:

Navigate in a browser to the following URIs and log in with the credentials defined during the install:

https://esserver:9200/_cluster/health
https://esserver:9200/_cat/indices/

Both of these URIs provide information related to the health of Elasticsearch. Specifically, look for Green statuses as an indication of health.

Steps to update JRE on the server running Elasticsearch service

1. Connect to the server hosting the Elasticsearch service.
2. Open a CMD terminal window as an administrator and browse to the location where Elasticsearch is installed. For instance, X:\Program Files\SailPoint\elasticsearch-5.1.1\bin
3. Execute the command elasticsearch-service.bat manager
4. This will open the manager window for Elasticsearch.
5. Click on the tab named "Java" and note the JRE that Elasticsearch is using to run on the server. By default, this should be "default"
6. Go to the services snap-in and stop the File Access Manager Elasticsearch 5.1.1 service
7. Install the latest version of JRE on the server. (At the time of writing these steps, the latest version of JRE is 1.8.0_261)
8. When prompted, choose to uninstall the old version of JRE on the server.
9. Confirm that the JAVA_HOME environment variable is pointing to the path of latest JRE.
10. Open a new CMD terminal window as an administrator and browse to the location where Elasticsearch is installed.
11. Execute the command elasticsearch-service.bat manager. If this opens the Elasticsearch manager window, that indicates that Elasticsearch is using the latest version of JRE.
12. Confirm that the File Access Manager Elasticsearch 5.1.1 service is running.
13. Open the File Access Manager admin client and navigate to the Health Center.
14. Click on the Infrastructure tab and confirm that the File Access Manager Elasticsearch service is GREEN.
15. In the File Access Manager business website, run a sample query through Forensics --> Activities to make sure you can see some activities retrieved from Elasticsearch.

[julian_anderson]

In addition, you can have a look at the article below.

Elasticsearch DB Full or Almost There:https://community.sailpoint.com/t5/IdentityIQ-Wiki/Elasticsearch-DB-Full-or-Almost-There/ta-p/75676