Activity monitoring with IdentityIQ

22 Kudos

Overview
What is it and what does it do?
Activities
Configuration
Configuring activity data sources
Transformation rule
Correlation rule
FieldMap
Activity targets tab
JDBC
Log file
RACF audit log
Windows event log
Enabling activity monitoring for a user
Enabling activity monitoring for a role
Activity aggregation
More reading

Overview

IdentityIQ is a popular and powerful product for controlling access rights. What is less known, is that it also allows for monitoring the use of these rights. This feature is called Activity Monitoring. Activities could be actions on the monitored applications like logging in, logging out or creating, modifying and deleting data. The actions found can be used for auditing purposes, could trigger policy violations or could even be used for alerting the security team of suspicious events.

My blog post explains the possibilities and why you should use activity monitoring. This article is a more in depth, technical version of the story about activity monitoring.

What is it and what does it do?

With activity monitoring IdentityIQ can watch additional data sources, related to connected applications, and look for events. These event can later be analyzed as part of routine inspections or policy evaluations.

A data source for activity monitoring is most often some kind of log file or other event storage. IdentityIQ will aggregate data from such a data source, parse or convert data, try to correlate it to a user and store the event for later analysis. Each event is stored as a separate object (sailpoint.object.ApplicationActivity).

Out of the box IdentityIQ supports four types of activity data sources:

JDBC Collector: read activity data from a database table or view.
Log File: extract information from a log file, like a web server log or a system log. The required data is extracted using regular expressions.
RACF Audit Log Collector: similar to the log file collector, but with a pre-defined format.
Windows EventLog Collector: get the activity data from Windows Event Logs, through an IQService application.

If necessary, it is possible to create a custom collector, based on sailpoint.activity.AbstractActivityCollector.

Activities

IdentityIQ has a number of activity types – or actions – defined. The actual interpretation of these actions is left to the implementer and also depends on the applications monitored. The types defined are end user actions. The available actions are defined as the Enum-type sailpoint.object.ApplicationActivity.Action. Possible values are:

Accept
Authenticate
Complete
Create
Delete
Execute
Grant
Load
Lock
Login
Logout
Read
Start
Stop
Update

Another Enum-type for activities is sailpoint.object.ApplicationActivity.Result, which can have the following values:

Failure
Success

Apart from an action, every activity must also include a time stamp of the event, user information for correlation of the activity to an identity and a target. The time stamp is important to be able to find out when an activity event has taken place. This allows the security team to further investigate activities if necessary, but also allow setting up policy rules that watch for “normal” actions at “abnormal” times, like logging in to a critical system outside office hours.

The user information, usually an account name or id, is used to connect activities to a specific user.

The target is a required, but free-format text field contains the name of the system, application or component on which the activity was performed. For example this could be a host name for an SSH login, a database schema and/or table for a database query or a distinguished name for an LDAP operation.

The most important attributes for an ApplicationActivity object are summarized in IdentityIQ Object Model and Usage (IdentityIQ 6.0). Information about the functions of the object are available in the javadocs that come with IdentityIQ under /doc/javadoc/.

Configuration

Several aspects of IdentityIQ are involved in the activity monitoring and may need to be configured. The following configuration actions will be described here:

Configuration of activity data sources in the application,
Configuration of an activity aggregation task,
Enabling activity monitoring for a user,
Enabling activity monitoring for a role,
Configuration of policies.

Configuring activity data sources

Activity monitoring is related to applications defined in IdentityIQ. The application configuration page for each application has a special tab for setting up the activity monitoring.

Multiple data sources can be defined per application. This could be useful if there are multiple log files to be monitored or if it is difficult to extract all types of events using a single data source from a file. To create a new data source, click the button New Activity Data Source ....

A new screen is opened to configure a data source.

First select one of the available types of data sources:

JDBC Collector: read activity data from a database table or view.
Log File: extract information from a log file, like a web server log or a system log. The required data is extracted using regular expressions.
RACF Audit Log Collector: similar to the log file collector, but with a pre-defined format.
Windows EventLog Collector: get the activity data from Windows Event Logs, through an IQService application.

The selected type of data source will determine the inputs for the transformation rule and the options available to configure the data source.

Transformation rule

The transformation rule is optional and can be used to modify the data that has already been parsed into an ApplicationActivity object. The rule can be used to read the attributes of the object and transform them into the correct values.

Correlation rule

The correlation rule is a required rule of the rule type "ActivityCorrelation". The rule is used to find the account or identity to which the event belongs. The rule is quite similar to a correlation rule used during account aggregation or manager correlation, but has more output types.

The return value of the rule is a map with one or two values. These values can be:

linkIdentity: the nativeIdentity attribute of a link. The system will search all Link objects for the application with the specified value for nativeIdentity and correlate to the found Identity object.
linkName: the display name of a link. The system will search all Link objects for the application with the specified value for the attribute selected as the display name and correlate to the found Identity object if a unique Link is found.
linkAttributeName and linkAttributeValue: all links for the application with the specified combination of the attribute name and value are searched for. These attributes must be defined as searchable under System Setup > Account Mappings.
identityName: the activity is directly correlated to the Identity object with the specified name.
identity: the activity is directly correlated to the specified Identity object.
identityAttributeName and identityAttributeValue: all identities with the specified combination of the attribute name and value are searched for. These attributes must be defined as searchable under System Setup > Identity Mappings.

FieldMap

Some collector types (JDBC and Log File) require a field map to be configured in the XML definition of the ActvityDataSource object. This field map is like an attribute mapping converting attribute names from the source to IdentityIQ names and can define simple translations of data.

An example field map looks like:

      <entry key="fieldMap">
        <value>
          <List>
            <ActivityFieldMap dateFormat="yyyyMMddHHmmss" field="SP_TimeStamp" source="time" transformationType="DateConverter"/>
            <ActivityFieldMap field="SP_NativeUserId" source="user" transformationType="None"/>
            <ActivityFieldMap field="SP_Action" source="action" transformationType="None"/>
            <ActivityFieldMap field="SP_Target" source="target" transformationType="None"/>
          </List>
        </value>
      </entry>

The field attribute of the ActivityFieldMap specifies the IdentityIQ name of the field. Possible names are specified in the enum sailpoint.object.ActivityFieldMap.ActivityField:

SP_Action (required)
SP_Info
SP_NativeUserId (required)
SP_Result
SP_Target (required)
SP_TimeStamp (required)

The source attribute specifies the data source name of the field. These names depend on the source definition.

The transformation attribute specifies how input values are transformed into values accepted by IdentityIQ. Possible type are specified in the enum sailpoint.object.ActivityFieldMap.TransformationType:

None: no conversion is necessary, or conversion is handled by the transformation rule (see above).
DateConverter: a date or timestamp is converted using a format specified in the dateFormat attribute. This format is based on the standard Java formatting for the SimpleDateFormat class.
Rule: a rule is referenced which will perform the conversion of the field value.
Script: a script is included inline, which will perform the conversion of the field value.

The field map shown above assumes that all fields are string values. In case of a JDBC Activity Data Source (see example below) the field time is not. It is a database timestamp, which by the database abstraction layer is represented as a java.sql.Timestamp, which is in turn a subclass of java.util.Date. Actually no conversion is needed, but when a source attribute is specified, IdentityIQ expects the value to be a string type. This requires a script or rule to return the correct value.

A script or rule for this conversion has the following in addition to the BeanShell defaults:

context: a SailPointContext object
log: Log4JLogger
rowColumns: a java.util.HashMap containing the column data of the row read from the database

Since the time field already is a Date type it can simply be returned. The easiest way to do that is using a script. The resulting field map is as follows.

      <entry key="fieldMap">
        <value>
          <List>
            <ActivityFieldMap field="SP_TimeStamp" transformationType="Script">
              <Script language="beanshell">
                <Source>return rowColumns.get("time");</Source>
              </Script>
            </ActivityFieldMap>
            <ActivityFieldMap field="SP_NativeUserId" source="user" transformationType="None"/>
            <ActivityFieldMap field="SP_Action" source="action" transformationType="None"/>
            <ActivityFieldMap field="SP_Target" source="target" transformationType="None"/>
          </List>
        </value>
      </entry>

For more complex conversions a rule may be useful and can be referred from the field map.

   <entry key="fieldMap">

        <value>

          <List>

            <ActivityFieldMap field="SP_TimeStamp" transformationType="Rule">

              <RuleRef>

                <Reference class="sailpoint.object.Rule" name="getMyTimeStamp">

              </RuleRef>

            </ActivityFieldMap>

            <ActivityFieldMap field="SP_NativeUserId" source="user" transformationType="None"/>

            <ActivityFieldMap field="SP_Action" source="action" transformationType="None"/>

            <ActivityFieldMap field="SP_Target" source="target" transformationType="None"/>

          </List>

        </value>

      </entry>

Activity targets tab

All types of activity data sources have a tab called "Activity Targets". Here a number of activity target names can be specified. The names listed here can later be used in the advanced analytics.

A data source can have more targets than specified in this list, but only the ones listed in these tabs can be used for searching in the advanced analytics using activity categories. To create a category, go to Define > Activity Target Categories. Here create a new category or select an existing category and add the desired activity targets to this category.

The categories and targets can now be used in advanced analytics:

Other targets not listed in the Activity Targets tab can also be used for registration and searching of activities. For searching they need to be specified explicitly using a free format field.

JDBC

The configuration for a JDBC based activity data source consists of three tabs. The first tab contains the JDBC Connection Settings.

Fill in the database user name, if required a password, the JDBC URL and the fully qualified Java class name of the JDBC driver.

The second tab is used to specify the query settings.

The SQL Statement is required and must specify the search query to find the activity data. The result of this query on the MySQL command line client could look like:

mysql>

The date is formatted as yyyy-MM-dd HH:mm:ss. That, however, is just a text representation of what will be read as an object of java.sql.TimeStamp. This results in a field map as presented here:

     <entry key="fieldMap">
        <value>
          <List>

            <ActivityFieldMap field="SP_TimeStamp" transformationType="Script">

              <Script language="beanshell">

                <Source>return rowColumns.get("time");</Source>

              </Script>

            </ActivityFieldMap>

            <ActivityFieldMap field="SP_NativeUserId" source="user" transformationType="None"/>
            <ActivityFieldMap field="SP_Action" source="action" transformationType="None"/>
            <ActivityFieldMap field="SP_Target" source="target" transformationType="None"/>
          </List>
        </value>
      </entry>

Optionally a condition builder and position builder rule can be specified to allow IdentityIQ to more specifically select activities and to skip events already processed. Examples of these rules are included in Rules in IdentityIQ.pdf.

Unfortunately no edit buttons are provided (up to 6.1p1). Therefor the Rule objects must be created manually in the debug interface or imported from an XML file.

<?xml version='1.0' encoding='UTF-8'?>
<!DOCTYPE Rule PUBLIC "sailpoint.dtd" "sailpoint.dtd">
<Rule language="beanshell" name="MySQL Example DateTime Position" type="ActivityPositionBuilder">
<Source>import java.util.Map;
import java.util.HashMap;
Map returnMap = new HashMap();
String lastTimeStamp = row.getString("time");
returnMap.put("lastTimeStamp", lastTimeStamp);
return returnMap;</Source>
</Rule>

Similarly, an adapted version of the condition builder rule presented in Rules in IdentityIQ looks as follows.

<?xml version='1.0' encoding='UTF-8'?>

<!DOCTYPE Rule PUBLIC "sailpoint.dtd" "sailpoint.dtd">

<Source>String condition = "";

String lastTimeStamp = (String)config.get("lastTimeStamp");

if ( lastTimeStamp != null ) {

condition = "time > '" + lastTimeStamp + "'";

}

return condition;</Source>

</Rule>

Log file

A log file data source has four tabs. The first tab determines the Transport Settings, or how the log file is accessed. This can be "local" for a locally stored file.

No further configuration is needed on this tab.

The second option is FTP.

A user, password and host to be used for FTP must be supplied.

The last option is SCP, secure copy:

A user name, a hostname and either a password or a private key in "PEM" format must be supplied.

The second tab contains the settings for the file itself.

First of all the location of the file must be specified, either the local path if the selected transport is "local" or the remote path if "ftp" or "scp" is selected as transport mechanism. If the file contains a header that should be ignored, the value for Lines to Skip should be set to a non-zero value: the number of header lines. The option Filter Nulls will tell IdentityIQ, if enabled, to skip any non-matching lines. If Multi-lined Data is selected, the regular expression should check for expressions stretching multiple lines.

Finally the Regular Expression must be entered. The expression is a Java regular expression (see Pattern (Java Platform SE 7 )). The regular expression must match an entire line or multiple lines, start with ^ and end with $. A good way to figure out the correct expression can be by using a regular expression tester, like RegexPlanet: online regular expression testing for Java.

The third tab contains a required list of the fields resulting from the regular expression. The fields must be specified in the exact same order as the results of the regular expression. In case of complex expressions with nested "groups", place holder fields that are not used in the actual activity data may be needed.

The fourth tab is the Activity Targets tab explained above.

RACF audit log

The RACF Audit Log is similar to the Log File type of activity data source, but much easier to configure since the file type has a fixed format. The configuration consists of only three tabs. The first tab determines the transport mechanism and is identical to the Transport Settings for the Log File type. Only the second tab called Log File Settings is unique for the RACF Audit Log Collector.

A file name is required: a local file path of the path on a remote server must be filled in, depending on the selected transport settings. The option Lines to Skip tells IdentityIQ to skip one or more lines before parsing the log file. Filter Nulls tells IdentityIQ to skip lines that cannot be parsed.

Windows event log

The Windows Event Log type of activity data sources only has two tabs, one of which is specific to this type.

All settings are required: a valid Windows or domain user name and password, the host and port of the IQService used to access the event logs, the server where the logs reside, a query and the block size (number of items to retrieve on each aggregation). The query is a "SQL like" WMI query specifying which events should be retrieved for the activity data.

Enabling activity monitoring for a user

For activity data to be stored with an identity, the activity monitoring must be enabled for this identity. This can be enabled by looking up the identity under Define > Identities and checking the monitored applications under the Activity tab for the identity.

From now on, newly discovered activities will be shown in the Identity's Activity tab.

Enabling activity monitoring for a role

Since enabling activity monitoring for each user individually can be a lot of effort, it is also possible to enable activity monitoring on specific roles. This would be best on a detected role.

As soon as the identity has received this role and the Refresh Identity Cube task has been run, the activity monitoring should be enabled for the application(s) related to the role. When activity monitoring is enabled by a role, it can no longer be enabled or disabled on the identity cube.

Activity aggregation

In order to read the activity data, a special aggregation task must be setup. For each data source a separate task must be used. Go to Monitor > Tasks and create a new Activity Aggregation task.

A setup screen is presented to configure the task.

Only a single data source can be selected per task. The option "Enable storage of the last activity position scanned on the data source." allows IdentityIQ to search for new events using the position and condition builder rules. If this option is not enabled, all available data will be read.

When the option "Store uncorrelated activities so they can be re-scanned and correlated at a later time." is selected all valid activities will be stored even though they cannot be correlated to a known identity. This option may especially be useful if you aggregate events more often than your source application, while IdentityIQ is not authoritative for (all of the) users in that application. In such cases there is a risk that new users may be active and generating activity before IdentityIQ knows about the user. Enabling this option will store the activity data for yet unknown users.

Depending on how important it is to monitor activity on a specific data source, the aggregation task can now be scheduled like any other task, to run periodically.

Activity monitoring with IdentityIQ