IdentityNow Account Filtering during Account Aggregation

[#1]

Overview

Often there is a need to filter accounts on a source during an account aggregation process.

While some source connectors offer an ability to filter accounts natively, other source connectors may not, usually due to technical limitations - such as a lack of filtering abilities in APIs the connector is calling.

Luckily, IdentityNow does have an ability to filter accounts on its side, as part of the aggregation process. This is configured by setting a filterString property on the source configuration. Once configured, the aggregation process matches accounts as they are aggregated against the filter string. Accounts which match the filter string will be filtered. Accounts which do not match, will be sent to IdentityNow as normal.

The trade-offs are summarized here:

Advantages

• Ability to filter accounts during an account aggregation, despite connectivity.
• Applies to all account aggregations and delta aggregations.
  
  

Disadvantages

• This offers no aggregation performance improvements. This still forces an aggregation to run in its entriety, and filters the accounts as they are iterated back to IdentityNow.

  If performance improvements are desired, we recommend leveraging filtering on the connector, where applicable. Not all source connectors have filtering abilities, as described above.
  
  

• Excessive filtering can produce aggregation timeouts. This is because IdentityNow (in the cloud) never receives any accounts (due to filtering) and keeps waiting for a response from the virtual appliance. The more accounts which are filtered, the greater chance for an aggregation timeout.

Note: For dedicated connectors, filterString will not work on the following attribute names unless they are marked as Account Name in the source.

• displayName
• identity
• instance

We don’t recommend changing the default Account Name of any dedicated connectors as it may have some unforeseen issues.

Configuration

In order to configure this, you need to modify the source configurations to add in a filterString property. This can be done with a simple partial update to the source, using the REST APIs.

As an example, we may want to filter out accounts which belong to employees. If employee was denoted by the "type" attribute on the account, we might configure our filterString to look like this:

Attribute	Value
filterString	( type != "Employee" )

Different Filters for Different Objects - The filterString applies to all objects which are aggregated. If you want to get more specific to accounts or groups which are filtered, you can use account.filterString or group.filterString to denote specific filters for those particular objects. This document assumes filterString but you can also use the others as well.

Any account which matches this criteria will be filtered. The value you see is filter string syntax. The filter string reference guide follows this section. More complex values can be implemented, such as groupings, logical operators, etc.

The configuration of this is done via the IdentityNow Source Partial Update API. The details are as follows:

Request

PATCH /beta/sources/{id}

 

HTTP Headers

Key	Value	Description
authorization	Bearer {token}	This is the JWT OAuth token.
content-type	application/json-patch+json	This is needed for PATCH operations.

 

Path Parameters

• id - The ID of the source. e.g. 2c9180835d191a86015d27ac132112ae

 

Query Parameters

• (Not applicable)

 

Request Body

JSON Patch syntax representing the change:

content-type: application/json-patch+json

[
  {
    "op": "add",
    "path": "/connectorAttributes/filterString",
    "value": "( type != \"Employee\" )"
  }
]

Example

curl -X PATCH \
  https://example.api.identitynow.com/beta/sources/2c9180835d191a86015d27ac132112ae \
  -H 'Authorization: Bearer eyJ...BRM' \
  -H 'Content-Type: application/json-patch+json' \
  -H 'cache-control: no-cache' \
  -d '[
  {
	"op": "add",
	"path": "/connectorAttributes/filterString",
	"value": "( type != \"Employee\" )"
  }
]'

 

Response

Response Codes

HTTP Code	HTTP Status	Description
200	OK	Returned if the request was successfully processed.
401	Unauthorized	Returned if there is no authorization header, or if the JWT token is expired.
403	Forbidden	Returned if the user you are running as, doesn't have access to this end-point.
429	Too Many Requests	Returned in response to too many requests in a given period of time - rate limited. The Retry-After header in the response includes how long to wait before trying again.
500	Internal Server Error	Returned if there is an unexpected error.

 

Response Body

content-type: application/json

A modified source object.

Filter Reference

This is a filter reference to show how various account filters might be constructed. The expressions genreally follow the form:

{property} {operation} {value}

Here are additional rules for parsing correctly:

• String literals should have double-quotes.

  • e.g. firstname == "Neil"

• True / false values are treated as boolean literals

  • e.g. inactive != false

• Digits are treated as numbers

  • e.g. age < 18

• The string value 'null' (no quotes) is treated as null

  • e.g. name != null

• Everything else is assumed to be the property name

  • e.g. email == contactAddress

 

Composite Filters

Filters can also be grouped and used together as composite filters:

Composite Filter	Pattern	Example
Grouping	( {expression} )	!( type == "Employee" ) || (type == "Contractor" )
AND	( {expression} && {expression} )	( type == "Employee" && location == "Austin" )
OR	( {expression} || {expression} )	( type == "Employee" || type == "Contractor" )
NOT	!( {expression} )	!( company == "SailPoint" )

 

Operations

Note: Any comparison operator can be prepended with an 'i' to signify a case-insensitive comparison (eg - i==, i!=, etc...).

Operation	Pattern	Example
Equals	{property} == {value}	firstname == "Neil"
Not Equals	{property} != {value}	lastname != "Smith"
Less Than	{property} < {value}	age < 18
Greater Than	{property} > value	age > 18
Less Than, Equals	{property} <= {value}	age <= 18
Greater Than, Equals	{property} >= value	age >= 18
Is Null	{property}.isNull()	email.isNull()
Not Null	{property}.notNull()	company.notNull()
Is Empty	{property}.isEmpty()	Groups.isEmpty()
Like, Exact	{property} == {value}	firstname == "Neil"
Like, Start	{property}.startsWith( {value} )	lastname.startsWith( "Mc" )
Like, Start (Ignoring Case)	{property}.startsWithIgnoreCase( {value} )	lastname.startsWithIgnoreCase( "Mc" )
Like, End	{property}.endsWith( {value} )	email.endsWith( "@sailpoint.com" )
Like, End (Ignoring Case)	{property}.endsWithIgnoreCase( {value} )	email.endsWithIgnoreCase( "@sailpoint.com" )
Like, Anywhere	{property}.contains( {value} )	email.contains( "sail" )
Like, Anywhere (Ignoring Case)	{property}.containsIgnoreCase( {value} )	email.containsIgnoreCase( "sail" )
Contains All	{property}.containsAll({ {value}, {value}, {value}, ... })	Groups.containsAll( { "A", "B", "C" } )
Contains All (Ignoring Case)	{property}.containsAllIgnoreCase({ {value}, {value}, {value}, ... })	Groups.containsAllIgnoreCase( { "A", "B", "C" } )

[jnavarretejr]

Is there other ways to update and aggregate a single identity from a source or is account filtering the best approach? For example, if i wanted to update a delimited file connected source can I use the IdentityNow API to add or update a single identity instead of aggregating the entire source?

[vkoldenh]

this filter also seems to work on entitlement aggregation, at least on the Azure AD source. Is that intended or should I create a ticket?

[charlieliujcn]

what if the attribute name has a space at middle? For instance, in AWS direct connect source, it has an attribute, named as "Access Keys". We would like to set up the filter so we aggregate only AWS users that have "Access Keys"

Tried to set filter like, 

Access Keys.isNull()

but it doesn't work. If we put "Access Keys".isNull(), then "Access Keys" is treated as string literal.

[chrisp]

Does the attribute have to be part of the Account Schema for the Source?

[hrathod007]

Yes @chrisp , Based on my testing it looks like attribute must be part of account schema 

[glh6136]

Trying to use this with Azure to only pull in hybrid accounts. This is the filter that was setup and Azure connector still pulling everything in. 

"value": "(dirSyncEnabled == true && userType == \"member\")"

[spopoff]

Hello,

@neil_mcglennon   @piyush_khandelwal  @lisa_ivy 

To fully understood the flow with a custom openconnector:

1 - I create a filter in a Java "fashion" (looks like in Java)
2 - I update the Source Configuration of the openconnector
3 - When an aggregation starts, the system get the filterString attribute
 a - parse the content (if error what append ?)
 b - Convert Java like filter as pojo Filter object
 c - Launch the iterate method with this pojo (see iterate(Filter filter) in openconnector.Connector)
4 - When iterate method is launch, the connector parse and validate the pojo
5 - Transforms the Filter pojo to something that filter data

Is that correct ?

Or the system launch the Aggregation (the pojo Filter is null) and when iterate on the result apply the filter on it  ?

Best regards,

Stéphane POPOFF

[lfioramo]

Hello can anyone explain if it works correctly?

here some tests:

( lastname.startsWithIgnoreCase( "value1" ) || jobtitle.startsWithIgnoreCase( "value2" ) || lastname.startsWithIgnoreCase( "value3" ) || lastname.startsWithIgnoreCase( "value4" ) )  ---> no effect

( LastName != \"value1\" && !(lastname.startsWithIgnoreCase( \"value2\" )) ) --> no effect

( LastName != \"value1\" && lastname.startsWithIgnoreCase( \"value1\" ) ) --> no effect

( LastName != \"value1\" || !(lastname.startsWithIgnoreCase( \"value2\" )) ) --> no effect

(lastname.startsWithIgnoreCase( \"value1\" )) --> no effect, returns all

!(lastname.startsWithIgnoreCase( \"value1\" )) --> returns 0

( LastName != \"value1\" && LastName != \"value2\" ) --> return 2, the only that works

!(lastname.startsWith( \"value1\" )) --> return 0

(lastname.startsWith( \"value1\" )) --> no effect, returns all

In particular how to have the .startsWith functioning?

Thanks

[Istvan]

Just a few observations from my attempts on using this to get accounts only for a specific manager and his team:

• in case integer attributes type matters
  ManagerId=="123456" --> no effect
  ManagerId==123456 --> filters out the team, see next point
• reminder that you define which accounts to EXCLUDE
  ManagerId==123456 --> does not work
  ManagerId!=123456 --> works
  (PersonId != 123456 || ManagerId!=123456) --> does not work, see previous point
• with grouped composite filters you need all brackets
  !( PersonId==123456 || ManagerId==123456 ) --> does not work
  !((PersonId==123456 || ManagerId=123456 )) --> works
• we did not run encounter any timeout issues when applying this filter to  get ~10 out of ~16000 accounts in our sandbox  

I hope this info might help others succeed :wink:

[lfioramo]

Hi, I'm back on this feature. How to query attributes with complex name such as "Password Deactivated"?

Thanks

[Amogha_Kaisare]

Hi,

If the value of the filter string contains a space in it then operations "==" and "!="  do not work to filter out the accounts. In such cases containsIgnoreCase operator has to be used to match the value.

eg: For below filterString will filter out all the accounts which has value "WM EMPLOYEE" for the "User Groups" attribute.

"value": "((User Groups.containsIgnoreCase(\"WM EMPLOYEE\")))"

[rahul-bhosale2]

Hi,

I'm trying to add filter for Workday source but it's not working as expected.

We are trying to filter accounts based on specific worker type and also want to exclude few users from the filtration.

I tried multiple ways by changing the sequence and syntax but not getting expected result. 

Below are some tried filters.

(EMPLOYEE_TYPE == \"Contractors\" && (FILENUMBER != \"458321\" || FILENUMBER != \"5693246\" || FILENUMBER != \"783692\"))

(EMPLOYEE_TYPE == \"Contractors\" && ((FILENUMBER != \"458321\") || (FILENUMBER != \"5693246\") || (FILENUMBER != \"783692\")))

It works only if I add one condition of FILENUMBER

(EMPLOYEE_TYPE == \"Contractors\" && FILENUMBER != \"458321\")

Requirement is to filter all contractors except three users.

[KirtiSikarwar]

Wanted to know if the user filter is case-sensitive eg: will pull in both _admin as well as _Admin accounts

[rgopinath]

Hi,

endsWith condition in user filter settings for Azure AD connector is not taking effect and returns all records.

"userPrincipalName.endsWithIgnoreCase(\"Value1\") || userPrincipalName.endsWithIgnoreCase(\"Value2\") || userPrincipalName.endsWithIgnoreCase(\"Value3\")"

Any thoughts ?

[gvscdeep]

Hi, 

We are trying to perform group filtering on a SAP based system where we have to get the "Roles" (which is mapped to Name attribute in group schema) that start with certain characters. We are trying to use something like below but exactly after 3 minutes, we get an aggregation error. When we tried something different and simple as well, we are facing the same issue. 

"group.filterString": !("Name.startsWith(\"AB_\"))"

We tried a simpler one as well like "group.filterString": "Name != \"AB_ALL_INFO\"",

Error: Exception during aggregation of Object Type group on Application SAP System: DE8 [source]. Reason: java.lang.RuntimeException: An error occurred while aggregating Application SAP System: DE8 [source]

Anyone facing/faced a similar issue?

[benjamin_moya_synetis]

@gvscdeep Did you find a solution ? We currently have the same issue.

[gvscdeep]

Not yet @benjamin_moya_synetis. For now we resorted to just using the account aggregation as that is giving us the required entitlements for now. But we will have to find a solution down the line.  

[yuyanglee]

@gvscdeep you can use this for SAP role filter:

"group.filterString": "!(Name == \"SAP_BC_ENDUSER\")",

If you want to use start with it should be:

"group.filterString": "!(Name.startsWith(\"SAP_BC\"))",

[rsh3ll]

Does the filtering (filterString) work on SaaS sources as well?

[fmunerato]

@rsh3ll as far as I know, the SaaS conectors do not support filterString, SaaS connectors only suport filtering if specified in the UI configuration of the SaaS Source Connector.

[shruthi_m]

is there any way to findout the data that are excluded by this filter?