cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

SailPoint IdentityNow AuditEvent Add-on for Splunk - Installation and User Guide

SailPoint IdentityNow AuditEvent Add-on for Splunk - Installation and User Guide

SailPoint's IdentityNow AuditEvent Add-on has been certified by Splunk and is designed to provide customers the ability to extract audit information from one of their IdentityNow tenant using Splunk Enterprise or Splunk Cloud.

Using IdentityNow's AuditEvents API, we can solve a number of problems with this add-on. Some examples include:

  • Surface and gain insights into the brute force password attempts IdentityNow has blocked
  • Correlate IdentityNow user activity with other system events to identify coordinated attacks
  • Evaluate the timing of login attempts from different geographies to identify problems

For more information about the /search API used by the add-on, see https://developer.sailpoint.com/idn/api/v3

NOTE: This add-on is intended to make it even easier to bring IdentityNow user activity and governance events into Splunk to improve insights from your security incident and event monitoring solution. But if you already have a Splunk integration that's working for you, you don't need to make any changes or switch to this option.

NOTE: The Splunk add-on does not interact with the virtual appliance and is an add-on for your existing Splunk deployment. Please refer to your Splunk Administrator for network permissions/needs.

 

 

Architecture

IdentityNow

Authentication

The Splunk Add-on for IdentityNow authenticates to the IDN tenant twice, once to be issued a JSON Web Token (JWT) by the API gateway, and again using this JWT when retrieving the actual AuditEvent records from IDN. The initial request requires that the add-on be configured with a Client ID,Client Secret issued by the IDN tenant, where the 'ClientID' and 'Client Secret' are attained by creating a Personal Access Token in IdentityNow (Best Practices - IdentityNow REST API Authentication).  It is recommended that when using the 'client_credentials' mechanism with a Personal Access Token, that a service account is created in IdentityNow for this process. These credentials are saved on the 'Data Input' in Splunk, which allows a single installation of the add-on to be used across all tenants owned by an organization.  The credentials are then utilized to retrieve a JSON web token (JWT) from the IdentityNow tenant. If the JWT is not issued due to error, the script will exit. Once a valid JWT is issued by IdentityNow, the add-on script will then make a POST request to /v3/search/events, using 'Bearer' authentication and this JWT token.

 

AuditEvents

AuditEvents in IdentityNow represent "things of interest" that occur during the normal day to day operations of IdentityNow. These include events such as an admin creating/deleting applications, successful authentications, provisioning failures, etc.​ IdentityNow has built-in reporting capabilities to review these events, but the /search API can be used to extract these events for further examination externally. Currently, each event extracted via the API has the following JSON format:

 

 

 

 

 

 

 

 

 

{
        "org": "org_name",
        "pod": "stg01-uswest2",
        "created": "2019-09-13T23:29:37.097Z",
        "id": "ffd69f6f-c3bc-4dbf-89cf-f4d7f91834bb",
        "action": "AUTHENTICATION-103",
        "type": "AUTH",
        "actor": {
            "name": "Adam.Kennedy"
        },
        "target": {
            "name": "Adam.Kennedy"
        },
        "stack": "oathkeeper",
        "trackingNumber": "748e1adb8fa94cda8f5b054e869c24cd",
        "ipAddress": "207.189.160.209",
        "details": "748e1adb8fa94cda8f5b054e869c24cd",
        "attributes": {
            "pod": "stg01-uswest2",
            "org": "org_name",
            "sourceName": "SailPoint",
            "info": "LOGIN_SUCCESS"
        },
        "objects": [
            "AUTHENTICATION"
        ],
        "operation": "REQUEST",
        "status": "PASSED",
        "technicalName": "AUTHENTICATION_REQUEST_PASSED",
        "name": "Request Authentication Passed",
        "synced": "2019-09-13T23:29:38.428Z"
    },

 

 

 

 

 

 

 

 

 

Splunk Enterprise/Splunk Cloud

Events

An event is a single piece of data in Splunk software, similar to a record in a log file or other data input. When data is indexed, it is divided into individual events. Each event is given a timestamp, host, source, and source type. Often, a single event corresponds to a single line in your inputs, but some inputs (for example, XML logs) have multi-line events, and some inputs have multiple events on a single line. When you run a successful search, you get back events. Similar events can be categorized together with event types. The following image shows what an IdentityNow audit event looks like as a Splunk event:

splunk_result_set.png

 

SourceTypes

Source type is a default field that identifies the data structure of an event. A source type determines how Splunk Enterprise formats the data during the indexing process. Splunk Enterprise comes with a large set of predefined source types, and it assigns a source type to your data. You can override this assignment by assigning an existing source type or creating a custom source type. This add-on creates a custom source type 'sailpoint_identitynow'. The Splunk indexer identifies and adds the source type field when it indexes the data. As a result, each indexed event has a source type field. A Splunk admin can use the source type field in searches to find all data of a certain type (as opposed to all data from a certain source). The 'sailpoint_identitynow' source type is straightforward, as seen here:

IDN_Splunk_SourceType_Details.png

 

Data Input

A Splunk deployment typically has three processing tiers: data input, indexing, and search management. A specific input consumes a raw data stream from its source (broken into 64k blocks) and annotates each block with some additional metadata (host, source, and source type). Splunk does not look at the contents of the data stream at this point, so the metadata is consistent across all data in a single stream. After raw stream input, the next thing that occurs is the data is parsed into individual events. The IDN add-on creates the events as part of the included scripts. Single data-input exists for the given sourcetype with the ability for the data input to specify execution interval. The IDN add-on defaults the data input interval to 300 seconds (5 minutes).

Screen Shot 2021-05-13 at 11.30.35 AM.png

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Screen Shot 2021-05-14 at 10.24.09 AM.png

This input executes a Python script to make HTTP requests to the correct IdentityNow api gateway endpoints, and gather the audit events. In order for this to work, IdentityNow AuditEvent input much be configured to supply the organization name, a client id, a client secret, a username and a password. Details on the setup and configuration of the data inputs can be found in the 'Setup' section of this document.

 

Add-on

An add-on is type of app that runs on the Splunk platform and provides specific capabilities to other apps, such as getting data in, mapping data, or providing saved searches and macros. An add-on is not typically run as a standalone app. Instead, an add-on is a reusable component that supports other apps across a number of different use cases. Add-ons can be installed via SplunkBase in the case of Splunk Enterprise and Splunk Cloud, or directly via the local filesystem for Splunk Enterprise. The IdentityNow add-on should be installed via SplunkBase.

 

Requirements

SailPoint IdentityNow

In order to stream AuditEvent data to a Splunk Enterprise, or Splunk Cloud deployment, an active IdentityNow tenant must exist. Configuration requires that the implementer also have the organization name of their IdentityNow API gateway supporting version 3 of the IDN API's. A client id and client secret generated by requesting a Personal Access Token from IdentityNow are required by the /search API in IdentityNow. Instructions for generating these values can be found in the 'Installation' section of this document.

 

Splunk

The add-on must be installed in a fully licensed deployment of either Splunk Enterprise, or Splunk Cloud.

 

Installation

Full installation of the IdentityNow AuditEvent add-on requires minimal configuration in both IdentityNow, and Splunk.

 

Generating a Personal Access Token (PAT) in IdentityNow

What is a Personal Access Token?

The easiest way to think of a personal access token, is a special OAuth 2.0 client credential tied to a specific user, which as such it gives the same context as the user, without having to give out that user's credentials.

Currently, the way to generate a personal access token, is via the REST API. In order to create a personal access token we'll refer to this documentation: Create Personal Access Token

Personal Access Token creation is a one-time generated token. It can then be used in configurations, in lieu of a password, to operate in a user's context.

SCOPE: Kindly select sp:search:read as shown in below screenshot when creating personal access token.

Screenshot 2023-09-11 at 17.16.24.png

 

More information on scope assignment can be found at: https://developer.sailpoint.com/idn/api/authorization#assigning-scopes-with-the-api 

An example call might look like this:

curl -X POST \
  https://example.api.identitynow.com/beta/personal-access-tokens \
  -H 'Authorization: Bearer eyJh...dJ94' \
  -H 'Content-Type: application/json' \
  -H 'cache-control: no-cache' \
  -d '{
	"name": "My Integration"
}'  

 

If successful, a JSON object representing the created Personal Access Token object will be given as a response. For example:

  • HTTP 201 Created
  • Content-Type: application/json
{
    "id": "86f1dc6fe8f54414950454cbb11278fa",
    "name": "My Integration",
    "secret": "d2be94f1fb61d0a1c63fa78ba941776e21ac6b674808db24d6426839ed758091",
    "created": "2019-11-11T21:04:38.203Z",
    "owner": {
        "type": "IDENTITY",
        "id": "ff80818155fe8c080155fe8d925b0316",
        "name": "slpt.services"
    }
}

 

Note: This is the only time the Personal Access Tokens' secret attribute will be displayed. Take note for future use.

Once we have the Personal Access Token's id and secret value, we can leverage those in a standard OAuth 2.0 client credential grant flow.

*Important* - The next step in the process is to use these values to retrieve an access token, and the use this token to retrieve a Personal Access Token - refer to Best Practices - IdentityNow REST API Authentication for additional details. The Personal Access Token 'id' and 'secret' will be used in the Splunk app in the 'client id' and 'client secret' fields.

 

Downloading and installing the add-on from SplunkBase

1. Go to SailPoint IdentityNow AuditEvent Add-on | Splunkbase

2. Click Download, license agreements will be displayed.

IDN_Splunk_Install.png

3. Verify that the SailPoint IdentityNow AuditEvent Add-on now appears on the main Splunk Dashboard.

IDN_Splunk_Verify_Download.png

 

Downloading and Installing from SplunkCloud App Browser

1. From within the Splunk instance, navigate to the App Browser by either:

  • Clicking the + icon on the main screen.

IDN_Splunk_plus_sign.png

  • Clicking the Apps dropdown and selecting Find More Apps.

IDN_Splunk_Find_More_Apps.png

2. On the Browse More Apps page, type identitynow into the search box.

IDN_Splunk_addon_search.png

3. Click the Install button next to the SailPoint IdentityNow AuditEvent Addon.

IDN_Splunk_install_button.png

4. Verify that the SailPoint IdentityNow AuditEvent Add-on now appears on the main Splunk Dashboard.

IDN_Splunk_Verify_Download.png

 

Special Installation Considerations

The IdentityNow AuditEvent add-on is designed to be used for data collection. It serves to purpose of creating a pre-defined source type for Splunk administrators to easily pull audit data from an IdentityNow tenant into Splunk. As with any Splunk add-on, there are deployment limitations that should be considered based on your organizations specific Splunk architecture.

 

Installation on Search Head Clusters

Per Splunk's documentation, while add-ons can technically be installed onto hosts designated as 'Search Head's' for search time functionality, it is not recommended that data inputs be configured on 'Search Heads'. Please refer to Splunk's documentation for additional details. Splunk Documentation - About installing Splunk add-ons - Splunk Documentation

Splunk documentation also outlines modifications that are required to install add-ons successfully on search head clusters:

"Before deploying an add-on to a search head cluster, check the documentation of the add-on to ensure it is supported on search head clusters. The add-on documentation might also include pre-deployment steps that you must perform in order to avoid validation errors. The files that cause the validation errors are all irrelevant to search head functionality, but the validation errors prevent deployment.

Unless otherwise instructed in the documentation for the specific add-on, before deploying an add-on to a search head cluster, make the following changes to the add-on package:

  1. Remove the eventgen.conf files and all files in the samples folder.
  2. Remove the inputs.conf and inputs.conf.spec files, if the add-on contains them. Exception: If you are collecting data locally from the machines running your search head nodes, keep these files.
  3. Remove the database.conf file, if the add-on contains one."

Link to documentation - Install an add-on in a distributed Splunk Enterprise deployment - Splunk Documentation

 

The IdentityNow AuditEvent add-on contains both inputs.conf, and inputs.conf.spec files in its build directory. In environments where manual installation of add-ons is not available (unassisted Splunk Cloud), the manual removal of these files to pass validation for search head cluster installation is not possible. Because of these limitations, deploying the SailPoint IdentityNow AuditEvent add-on is currently not supported on Search Head Clusters.

 

Installation via Deployment Server

Splunk documentation notes:

"If the add-on uses modular or scripted inputs to collect data from remote sources, using a deployment server to deploy the configured add-on to multiple forwarders acting as data collectors causes duplication of data. This limitation applies to third-party deployment solutions as well as the deployment server.

You can safely use a deployment server to deploy unconfigured add-ons."

Link - About installing Splunk add-ons - Splunk Documentation

 

Installation on Heavy Forwarders

Installation on Splunk heavy forwarders should be possible, but has not been tested. Installation should be done manually, due to the limitation of installation of configured data collectors with the deployment server.

 

Setup

Add-on settings

Once the add-on has been installed, fill in the add-on's global settings information which is used for data input.

Screen Shot 2021-05-13 at 11.37.28 AM.png

  1. Enter the organization name associated with the IdentityNow tenant.
  2. Enter the Client ID ('id' field from the Personal Access Token) that was generated in section 'Generating a Personal Access Token (PAT) in IdentityNow' of the 'Installation' instructions in this guide.
  3. Enter the Client Secret ('secret' field from the Personal Access Token) that was generated in section 'Generating a Personal Access Token (PAT) in IdentityNow' of the 'Installation' instructions in this guide.
  4. Click 'Save'.

 

Creating IdentityNow data input

 To define a custom data input:

1. Within the Splunk UI, go to Settings > Data > Data Inputs

IDN_Splunk_Create_DataInput1.png

2. Locate the SailPoint IdentityNow AuditEvent entry in the 'Local inputs' list. Click Add new.

Screen Shot 2021-05-13 at 11.30.35 AM.png

3. The add-on configurations interface will now be displayed.

Screen Shot 2021-05-13 at 11.35.46 AM.png

  1. Type the name of the data input. Use something descriptive, such as "IdentityNow Production."
  2. Enter the organization name associated with the IdentityNow tenant.
  3. Click the More settings checkbox, and then type in the new interval time (in seconds) for the execution of the collection.

 

Setting the interval time appropriately

The data type 'SailPoint IdentityNow AuditEvent' installed by the add-on should have its initial interval set to 300 seconds, or 5 minutes. This interval determines how frequently to run the audit event data collection from an IdentityNow tenant. Intervals can be tuned appropriately for each instantiation of the data type. It is recommended that individual organizations tune this value based on organizational needs. The more frequently the data collection scripts execute, the more 'real-time' the collection is, but at the expense of potentially putting heavier load on the IdentityNow api gateway and the Splunk application. To modify the interval on data input creation, click the More settings checkbox, and then type in the new interval time (in seconds) for the execution of the collection.

idn_splunk_interval_default.png

This can also be modified later through the Splunk UI, by navigating to Settings > Data > Data Inputs > SailPoint Identitynow AuditEvent, and the left-clicking on the name of the data input for which the interval needs to be adjusted. Again, the interval will be located under the More settings checkbox.

 

Proxy Settings

As of version 1.0.5, proxy settings are available in the configuration of the add-on. To enable proxy communications, simply

1. Click the 'SailPoint IdentityNow AuditEvent Add-on' tile on the Splunk main UI screen

2. Click the 'Configuration' tab at the top of the page

Screen Shot 2021-05-13 at 11.38.26 AM.png

3. On the 'Proxy' tab, enter the details required for your proxy server

Screen Shot 2021-05-13 at 11.38.26 AM.png

 

User Guide

Searching on Source Type

Once the add-on has been installed, and a new data input has been created, it is time to start searching. A simple way to test this, is from the Splunk Search and Reporting app, type sourcetype="sailpoint_identitynow". If everything has been configured correctly, this should start returning results.

NOTE: As of version 2.0.0 of the add-on, the add-on operates on a 60 minute delay to ensure that new AuditEvents are available to the IdentityNow Search API.

NOTE: As of version 2.0.0, if the Splunk add-on loses its connection to IdentityNow (manually disabled, credentials are removed, etc) for longer than a 24 hour period, it will begin collecting events from the time of reactivation/reconnection. For administrators, this means that if the add-on collection is suspended longer than 24 hours, audit events for this time period will not be available in Splunk. These events will still be retrievable via the IdentityNow API's.

IDN_Splunk_Searching_on_sourcetype.png

 

Advanced Search

Advanced searches can be performed by specifying one of the many fields present in an IdentityNow AuditEvent object. The below example shows a search that specifies the action "SESSION-1", which when translated, would mean show me all the audit events that indicate a user session being created (user login to IdentityNow).

SplunkIDN_SearchOnSourceType.png

 

Dashboard

The SailPoint IdentityNow AuditEvent Add-on for Splunk provides a source types with in Splunk®. The above configured source type will be are used to collect Audit events from IdentityNow.  Users can configure this source type to collect events to Splunk® and populate a custom dashboard visualizing different types and details for these events.

Initial Dashboard before configuring the source type

Screen Shot 2020-10-14 at 11.45.26 PM.png

 

  • The Dashboard displays data in different visualization format.
  • Dashboard data is updated every 5 minutes.
  • The time picker is set to last 30 days by default.
  • Time window can be modified, and the dashboard results update automatically by clicking “Submit” button.
  • The dynamic selectable filters help to get filtered results. Filtration updates the data on the dashboard.
  • Each visualization is linked to search.

Screen Shot 2020-10-13 at 10.42.45 PM.png

 

  • Pie chart shows the Error Type. On hover shows the percentage and count for the slice.

Screen Shot 2020-10-13 at 10.43.26 PM.png

 

  • The bar chart displays the count for event status on top of each column. X-axis displays the count and Y-axis shows the status.

Screen Shot 2020-10-13 at 10.43.38 PM.png

  • The bar chart displays the count for type of operations. X-axis shows each operation t and Y-axis shows the count.

Screen Shot 2020-10-13 at 10.44.01 PM.png

  • Table below shows events received from the source type. It consists of paginator and column sorting functionality.Screen Shot 2020-10-13 at 10.44.40 PM.png

 

Index Setting:

If index is set to anything other than default, please edit the script "input_module_sailpoint_identitynow_auditevent.py" and set index=helper.get_output_index() in the below code:

event = helper.new_event(data=data, time=None, host=tenant["url"], index=None, source=helper.get_input_type(), sourcetype=helper.get_sourcetype(), done=True, unbroken=True)

 

Related Links

 

Version 2 Updates - 12/2/2019

Version 2.0.0 of the SailPoint IdentityNow AuditEvent Add-on for Splunk contains several enhancements:

  • Utilizes the new IdentityNow /search API's
  • Limited look-back (24 hours) to improve performance
  • 60-minute staggered ingestion of AuditEvents
  • Modified format of AuditEvent data
  • Additional authentication configuration fields

 

1/15/2020

Version 2.0.1+ of the SailPoint IdentityNow AuditEvent Add-on for Splunk contains a change to the the authentication flow:

  • Utilizes Personal Access Token information

 

10/14/2020

Version 2.0.3 of the SailPoint IdentityNow AuditEvent Add-on for Splunk contains a custom dashboard visualizing different types and details for these events.

 

01/08/2021

Version 2.0.4

 

04/12/2021

Version 2.0.5: Bug fix related to proxy settings.

 

04/27/2021

Version 2.0.6:  Fixed reported cloud-vetting issues.

 

10/06/2021

Version 2.0.7:  Compatible with JQuery v3.5 and above.

 

11/04/2021

Version 2.0.8:  Added missing .xml files for input and configuration.

 

03/24/2021

Version 2.0.9: Updated /search Beta API to V3 API

 

08/17/2023

Version 2.0.10: Updated the add-on to be fully compatible with Python 3.7 and Splunk Cloud.

 

11/09/2023

Version 2.0.12: Updated the addon to utilize x-www-form-urlencoded and pass the client secret and client id in the request body for OAuth2 authentication.

Comments

https://community.sailpoint.com/t5/IdentityNow-Wiki/Best-Practices-Creating-an-IdentityNow-Password-... This link is not present. Can you please update the correct link?

This document says about creating the personal token using the client credentials. But when I try to create the personal token using the client credential, I get I "access_denied" error.  How do I create the personal token without attaching it to any identity?

@adam_creaney @dhara_shah 

Good afternoon Prajna - the PAT creation process requires a user context (attaching to an Identity). For more information, please see: https://community.sailpoint.com/t5/IdentityNow-Wiki/Best-Practice-Using-Personal-Access-Tokens-in-Id...

Hi @adam_creaney , 

Thank you for updating the document.

Hello !

 

I've used the code from the Splunk connector to create a script to fetch the events using the AuditEvent API.

This yields a lot of items with a lot of different "action", "type", "attributes", "status" values, etc. Is there somewhere a list of the items that one may get from this API, and their meaning, in each case? This would be extremely useful and I couldn't find it on the API documentation.

Thanks a lot!

Robin

Hello!

I have integrated the Splunk with IDN, and found out only a few event logs the Splunk could receive. For example, I created source and uploaded delimited file these days but Splunk could not receive these types of events. May I know is it normal that the Splunk will not be able to receive these event logs?

 

@adam_creaney

I am using Splunk Cloud, we attempted to have this add-on added to our Splunk Cloud instance but Splunk responded to us with the notice below.

According to my Splunk Admin, our Splunk Cloud instance does not have access to the App Browser and instead requires us to have Splunk support install modules for us.

Any help that can be offered on how we can get this working?

This is a preliminary report. More issues may be found upon further review.
Thank you for your app install request. Your app did not meet security and functionality requirements for Splunk Cloud for the following reasons:

  • When using addon-builder to build an app, if a secret is part of the input schema, it will be properly encrypted and stored when operating inside the app. However, when editing an input using the system user interface(settings->data input-><input name>) with existing inputs disabled, the secrets will be stored in plain text. To avoid this secret disclosure, you can either make the secret field a global parameter(seehttps://docs.splunk.com/Documentation/AddonBuilder/2.2.0/UserGuide/ConfigureDataCollectionAdvanced) if there is only one secret for all the modular inputs, or you can add the secret/apikey/access_token field to the global account if there is a need to store multiple api keys. File: Line Number: 1

What are the minimum account privileges for the Splunk Personal Access token? Can one be generated that is not a full administrator since it is performing read-only activities?

Splunk recently released Add-on Builder v4.0.0. If you built your Splunkbase solution with a prior release of Add-on Builder, please update your app(s) as soon as possible, and no later than August 31, 2021, using this latest version so that your Splunkbase app(s) will be compatible with the next major releases of Splunk Cloud Platform and Splunk Enterprise.

As we move to newer versions of Python and jQuery in the platform, we have updated Add-on Builder to support them. That means that the current release of your application(s) based on the prior Add-on Builder release will not be compatible with the new Splunk releases coming later this year.

You can find more information about this latest Add-On Builder release in the Splunk Add-on Builder Use Guide. To update your add-on, you will need to export your current project, then import it into this new Add-on Builder release. You can find more information in the “Import and export an add-on project” section of the user guide.

If you have any questions, please reach out to us in the Splunk Community.

Thanks in advance for your support.

Team Splunk

I noticed that the Splunk documentation on this site says that this should support multiple environment(s) - looking at the code in the python scripts though it looks like it doesn't?

After reviewing the Splunk Plugin code (the Python code which Splunk uses to read data from SailPoint), I noticed the following bits of information:

Splunk/etc/apps/Splunk_TA_sailpoint is the plugin directory where the plugin derives its files.
Splunk/etc/apps/Splunk_TA_sailpoint/bin/input_module_sailpoint_identityiq_auditevents.py – this is the file in question that caught my attention.
The way the code appears to operate is that it defines a SINGLE file for where it stores the epoch date following the logic outlined below:
 
1)  Initially there is file check (audit_events_checkpoint.txt) to see if the file exists
2)  If Python doesn’t find it attempts to created it
3)  If it fails again it creates the folder structure and then add the file
4)  After the first three steps Python opens the file
5)  Python then reads the files and pull the first value in (the Unix/Epoch time stamp)
6)  It then uses this as part of its query outbound.
 
#Read the timestamp from the checkpoint file, and create the checkpoint file if necessary
    #The checkpoint file contains the epoch datetime of the 'created' date of the last event seen in the previous execution of the script. 
    checkpoint_file = os.path.join(os.environ['SPLUNK_HOME'], 'etc', 'apps', 'Splunk_TA_sailpoint', 'tmp', "audit_events_checkpoint.txt")
    try:
        file = open(checkpoint_file, 'r')
    except IOError:
        try:
            file = open(checkpoint_file, 'w')
        except IOError:
            os.makedirs(os.path.dirname(checkpoint_file))
            file = open(checkpoint_file, 'w')
            
    with open(checkpoint_file, 'r') as f:
         checkpoint_time = f.readlines()
     
    #current epoch time in milliseconds 
    # new_checkpoint_time = int((datetime.datetime.utcnow() - datetime.datetime(1970, 1, 1)).total_seconds() *1000)
    
    if len(checkpoint_time) == 1:
        checkpoint_time =int(checkpoint_time[0])
    else:
        checkpoint_time = 1562055000000
        helper.log_info("No checkpoint time available. Setting it to default value.")
    
    #Standard query params, checkpoint_time value is set from what was saved in the checkpoint file
    queryparams= {
         "startTime" : checkpoint_time,
         "startIndex" : 1,
         "count" : 100
    }
 
1)  Jumping down to the next references we find that the JSON object that is pulled in is used to create the new timestamp that the system will use on the next request.
2)  It then takes this value and writes it to file which it will reuse the next time a call is made.
 
#Iterate the audit events array and create Splunk events for each one
    invalid_response = isListEmpty(auditEvents)
    if not invalid_response:
        for auditEvent in auditEvents:
 
            data = json.dumps(auditEvent)
            event = helper.new_event(data=data, time=None, host=None, index=helper.get_output_index(), source=helper.get_input_type(), sourcetype=helper.get_sourcetype(), done=True, unbroken=True)
            ew.write_event(event)
 
        #Get the created date of the last audit event in the run and save it as a checkpoint key in the checkpoint file
        list_of_created_date = extract_element_from_json(results, ["auditEvents", "created"])
 
        new_checkpoint_created_date = list_of_created_date[-1]
        helper.log_info("DEBUG New checkpoint date \n{}".format(new_checkpoint_created_date))
 
    #Write new checkpoint key to the checkpoint file
        with open(checkpoint_file, 'r+') as f:
            f.seek(0)
            f.write(str(new_checkpoint_created_date))
            f.truncate()
 
So here is my thought as to what is happening: When we enter information into Splunk for the connectors to EACH environment (we have a total of 6), the checkpoint_file is being over-written. I would also assume that each connected env calls the same timestamp because they all appear to be pulling that information from the same file. Do we miss a configuration, or is this a coding gap?

What is the recommended approach to source the VA's logs into Splunk?

Same issue as @robertm here. Despite configuring for two inputs we only get logs from the original IDN environment that was configured. 

Question, does anyone know if the splunk provides more details then what the Sailpoint audit logs show. So for example if someone changes the Source Config Query, or Source Config, will it says what has been changed at that level of detail, or will it just show the same, like my sample screenshot below.Example.png

Version history
Revision #:
33 of 34
Last update:
‎Nov 09, 2023 01:29 PM
Updated by: