Failure of activity monitoring services to reach Event Manager
Just started having this error with all of our activity monitoring services this past weekend after scheduled windows updates and a reboot. Anyone seen this? No changes in our SSL cert setup
2022-03-22 13:56:51.3514,12,Error,SailPoint.Fam.Infrastructure.Grpc.Common.GrpcHealthClient.checkHealthForHost - Unable to check health for host 'https://SERVERNAME.fqdn:8001/' Grpc.Core.RpcException: Status(StatusCode=Internal, Detail="Error starting gRPC call: The SSL connection could not be established, see inner exception.")
at Grpc.Net.Client.Internal.HttpClientCallInvoker.BlockingUnaryCall[TRequest,TResponse](Method`2 method, String host, CallOptions options, TRequest request)
at Grpc.Core.Interceptors.InterceptingCallInvoker.<BlockingUnaryCall>b__3_0[TRequest,TResponse](TRequest req, ClientInterceptorContext`2 ctx)
at Grpc.Core.ClientBase.ClientBaseConfiguration.ClientBaseConfigurationInterceptor.BlockingUnaryCall[TRequest,TResponse](TRequest request, ClientInterceptorContext`2 context, BlockingUnaryCallContinuation`2 continuation)
at Grpc.Core.Interceptors.InterceptingCallInvoker.BlockingUnaryCall[TRequest,TResponse](Method`2 method, String host, CallOptions options, TRequest request)
at Grpc.Health.V1.Health.HealthClient.Check(HealthCheckRequest request, CallOptions options)
at Grpc.Health.V1.Health.HealthClient.Check(HealthCheckRequest request, Metadata headers, Nullable`1 deadline, CancellationToken cancellationToken)
at SailPoint.Fam.Infrastructure.Grpc.Common.GrpcHealthClient.checkHealthForHost(Uri serviceAddress)
Hi @jobowyer ,
Apologies for the delay in response. Has this been resolved? Based on the title of this post - you already have the idea of whats going on - the error message is indicating a communication issue between the activity monitor and the event manager. If not resolved, it may be helpful to open a Support ticket. It may be something we need to look into around the windows updates.
This was resolved with assistance from Expert Services. The issue was an incorrect certificate being referenced within the backend database for the Event Manager service.
you don't happen to know which table, do you?
I've been having these "Unable to check health for host" errors ever since we went to 8.2.
I could be having a different issue, but I would love to search the certificate reference... if I had an idea how/where
You can alter the expected certification by using the FAMCertificateManager.exe utility, pointed at SERVICE_ID of the service (you have to retrieve SERVICE_ID from the database, I can't remember what table but it's the one that ties the string description of a service to the actual ID). Below are the notes I added to my company's knowledge base after Expert Services assisted us on this:
- FAM services will only communicate with other FAM services if the key validCertHashes contains the thumbprint of the other FAM service certificate, which ever cert that service is deciding to use
- change cert being used by FAM service via ServerInstaller\Tools\FAMCertificateManager\FAMCertificateManager.exe SERVICE_ID (gained from db) -existingCertificate
- this generates UI prompt to pick the expected certificate (must match thumbprint)