Hello,
There are days in the morning when we start working, we realize that the servers are down…
When we go to syslog to see what happened, we see the following error:
ERROR ServerThread sailpoint.server.HeartbeatService:274 - Host: ui01, deactivating server: tk01
Hello @sergiooliveira ,
I suspect that what is happening is that your application server is crashing, and then the Heartbeat service is removing it from the cluster to prevent task failures which may be caused by a failing partition on that server.
The Heartbeat service has an object for each server node in the cluster, and it is updated periodically with a heartbeat from each node. If a node repeatedly reports heartbeat failures, then the service will remove that node to prevent failures.
For instance, let’s say you have a long running partitioned identity refresh task which is running, and 1 of the application server instances crashes which hosted one of the identity refresh partitions. The Heartbeat service would detect the failed node and remove it. After this happens, the Reanimator service would route that partition to a good node, so that the work in the task can be completed.
It would be best to get the IIQ logs from the servers which have failed, and see if you can determine any failures of the application server prior to the server being deactivated. Usually we would see this type of issue with OutOfMemory exceptions, because the server will usually crash shortly after getting the OOM exception.
Just as a thought, you could technically disable the Heartbeat service, but I would suspect you will still find the servers terminated whether Heartbeat is active or not. Just go to Administrator Console, Environment, select the gear icon to the right of a server, and flip the toggle for Heartbeat on the Services tab.