IdentityIQ metrics: Prometheus + Grafana

[#1]

• Introduction
• Prometheus
• Grafana
• Summary

Introduction

Hi Sailors,

I want to show you a simple example how to use Prometheus + Grafana for controlling IdentityIQ metrics data. But why we need to use 3d party applications for metrics if IdentityIQ has this functionality. Usually, IdentityIQ is not only one application and if there are more than 10 applications which should be controlled/monitored we need something else. There are a lot of applications which can do it, but in this article we will use Prometheus as time-series database, Micrometer as java framework and Grafana as monitoring UI. All code is available on GitHub. So, lets write some awesome code.

Prometheus

This DB has one big advantage for using in IdentityIQ: pull-mode of getting metrics. It means: no need to store properties for Prometheus in IdentityIQ and no need to push/save data from IdentityIQ.

To implement integration Prometheus and IdentityIQ we need:

1. Create REST end-point for metrics
2. Configure Prometheus for getting metrics

I created custom REST resource:

/**
 * External identity iq controller for prometheus metrics
 */
@Slf4j
@Path("metrics")
public class ExternalMetricPrometheusRestController extends BaseResource {
 
    /**
     * Name for metrics of total time for getting all statistics
     */
    private static final String GETTING_METRICS_TIMER_NAME = "get_metrics_timer_millis";
 
    /**
     * Host name tag value
     */
    private static final String TAG_HOST_NAME = "host_name";
 
    /**
     * Active task metric name
     */
    private static final String ACTIVE_TASK_METRIC_NAME = "active_tasks";
    /**
     * Active task name tag value
     */
    private static final String TAG_ACTIVE_TASK_NAME = "active_task_name";
 
    /**
     * Get all metrics for prometheus database. There are 3 steps for getting metrics:
     * 1 - clear all current meters for prometheus registry
     * 2 - getting current host service statistic
     * 3 - pass all attributes from statistics to prometheus registry
     */
    @GET
    @Produces("text/plain;version=0.0.4;charset=utf-8")
    @Path("prometheus")
    public String getPrometheusMetrics() throws GeneralException {
        log.debug("Create metrics and start timer for getting metrics");
        StopWatch gettingMetrics = new StopWatch();
        gettingMetrics.start();
        PrometheusMeterRegistry prometheusMeterRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
        try {
            saveSystemMetrics(prometheusMeterRegistry);
            saveTasksMetrics(prometheusMeterRegistry);
        } finally {
            gettingMetrics.stop();
            log.debug("Total time of getting metrics:[{}] mills", gettingMetrics.getTotalTimeMillis());
            saveHostGougeMetric(GETTING_METRICS_TIMER_NAME, gettingMetrics.getTotalTimeMillis(),
                    prometheusMeterRegistry);
        }
        return prometheusMeterRegistry.scrape();
    }
 
    /**
     * Save statistics for task: execution time
     *
     * @param prometheusMeterRegistry - micrometer register for prometheus
     * @throws GeneralException error getting current sailpoint context
     */
    private void saveTasksMetrics(PrometheusMeterRegistry prometheusMeterRegistry) throws GeneralException {
        log.debug("Save tasks metrics");
        log.debug("Build query options only for active tasks");
        QueryOptions queryOptions = new QueryOptions();
        queryOptions.add(Filter.not(Filter.notnull("completed")));
        queryOptions.addOrdering("name", true);
        for (TaskResult taskResult : SailPointFactory.getCurrentContext().getObjects(TaskResult.class, queryOptions)) {
            prometheusMeterRegistry.gauge(
                    ACTIVE_TASK_METRIC_NAME,
                    Collections.singletonList(Tag.of(TAG_ACTIVE_TASK_NAME, taskResult.getName())),
                    Util.computeDifferenceMilli(taskResult.getLaunched(), new Date()) / Util.MILLI_IN_SECOND);
        }
    }
 
    /**
     * Save only system metrics from {@link MonitoringUtil}
     * - cpu usage (percentage)
     * - memory usage (Mb)
     * - memory usage (percentage)
     * - request threads count
     * - task threads count
     * - data base response time (millis)
     *
     * @param prometheusMeterRegistry - micrometer register for prometheus
     * @throws GeneralException error getting current sailpoint context
     */
    private void saveSystemMetrics(PrometheusMeterRegistry prometheusMeterRegistry) throws GeneralException {
        log.debug("Save system metrics");
        saveHostGougeMetric(Server.ATT_CPU_USAGE, MonitoringUtil.getCpuUsage().floatValue(), prometheusMeterRegistry);
        saveHostGougeMetric(Server.ATT_MEMORY_USAGE, MonitoringUtil.getUsedMemory(), prometheusMeterRegistry);
        saveHostGougeMetric(Server.ATT_MEMORY_USAGE_PERCENTAGE, MonitoringUtil.getMemoryUsagePercentage().floatValue(),
                prometheusMeterRegistry);
        saveHostGougeMetric(Server.ARG_MAX_REQUEST_THREADS, MonitoringUtil.getRequestProcessorThreads(),
                prometheusMeterRegistry);
        saveHostGougeMetric(Server.ATT_TASK_THREADS, MonitoringUtil.getQuartzThreads(), prometheusMeterRegistry);
        saveHostGougeMetric(Server.ATT_DATABASE_RESPONSE_TIME,
                MonitoringUtil.getDbResponseTime(SailPointFactory.getCurrentContext()), prometheusMeterRegistry);
        saveHostGougeMetric(Server.ATT_DATABASE_RESPONSE_TIME,
                MonitoringUtil.getDbResponseTime(SailPointFactory.getCurrentContext()), prometheusMeterRegistry);
    }
 
    /**
     * Save passed value as gauge to metric registry with hostName tags
     *
     * @param metricName              - metric name to check and save
     * @param metricValue             - metric value for passed name
     * @param prometheusMeterRegistry - micrometer register for prometheus
     */
    private void saveHostGougeMetric(String metricName, Number metricValue,
                                     PrometheusMeterRegistry prometheusMeterRegistry) {
        log.debug("Save metric name:[{}], value:[{}]", metricName, metricValue);
        if (metricValue != null) {
            prometheusMeterRegistry
                    .gauge(
                            metricName,
                            Collections.singletonList(Tag.of(TAG_HOST_NAME, Util.getHostName())),
                            metricValue);
        }
    }
}

This resource was added to external jersey servlet: no need authorization, but SailPointContext is available. This controller saves 2 types of metrics:

1. System metrics:
   1. CPU usage
   2. Memory usage
   3. Requests and tasks threads
   4. Database response time
2. Tasks metrics: active tasks execution time

For all of this metrics gauge is used: store current value. System metrics have common tag: host_name for separating this metrics in grafana. All tasks metrics are stored in one metric with name: active_tasks with unique tag: active_task_name where value for it equals current active task name.

IMPORTANT: Add all necessary libraries for micrometer and micrometer-prometheus to identityiq

Now, IdentityIQ is ready to receive requests by url external/rest/metrics/prometheus.

The final step is: configuring Prometheus to get metrics from IdentityIQ. We need to add a new job and set several properties to prometheus.yml:

1. metrics_path
2. targets

- job_name: 'IdentityIQ'
    metrics_path: /identityiq/external/rest/metrics/prometheus
    static_configs:
    - targets: ['kubakhov:8080', 'kubakhov:8081']

After these steps Prometheus can get metrics from IdentityIQ.

Grafana

First of all add datasource for Prometheus in Grafana, than create Dashboard using it. Simple dashboard example: IdentityIQ Grafana dashboard

This is a simple dashboard, but it can give us a lot of information in one page: system metrics and tasks. No need to switch between pages (Tasks, Environment).

Summary

Using these 2 applications gives us:

1. One place for controlling state/status/health-check for all application, not only IdentiytIQ
2. Adding custom metrics for different part of IdentityIQ (Rules metrics, Applications, System logs, Events, etc.)
3. Flexible dashboard (different area for UI servers, tasks, handling specific metrics, etc.)
4. Alerts
5. Quick response to emergency situations
6. Cool visualization
7. Default functionality of IdentityIQ does not depend on our metrics customization.

[kayleenolan]

is there any additional information on where to place these additional classes within the IIQ structure?  

[Rakesh007]

Dear @darylclaude_medina , 

Please provide the compiled java class, if possible.

And please elaborate more on the how-to setup in IIQ.

Thanks in advance.