Often there is a need to filter accounts on a source during an account aggregation process.
While some source connectors offer an ability to filter accounts natively, other source connectors may not, usually due to technical limitations - such as a lack of filtering abilities in APIs the connector is calling.
Luckily, IdentityNow does have an ability to filter accounts on its side, as part of the aggregation process. This is configured by setting a filterString
property on the source configuration. Once configured, the aggregation process matches accounts as they are aggregated against the filter string. Accounts which match the filter string will be filtered. Accounts which do not match, will be sent to IdentityNow as normal.
The trade-offs are summarized here:
This offers no aggregation performance improvements. This still forces an aggregation to run in its entriety, and filters the accounts as they are iterated back to IdentityNow.
If performance improvements are desired, we recommend leveraging filtering on the connector, where applicable. Not all source connectors have filtering abilities, as described above.Excessive filtering can produce aggregation timeouts. This is because IdentityNow (in the cloud) never receives any accounts (due to filtering) and keeps waiting for a response from the virtual appliance. The more accounts which are filtered, the greater chance for an aggregation timeout.
Note: For dedicated connectors, filterString
will not work on the following attribute names unless they are marked as Account Name in the source.
We don’t recommend changing the default Account Name of any dedicated connectors as it may have some unforeseen issues.
In order to configure this, you need to modify the source configurations to add in a filterString
property. This can be done with a simple partial update to the source, using the REST APIs.
As an example, we may want to filter out accounts which belong to employees. If employee was denoted by the "type" attribute on the account, we might configure our filterString
to look like this:
Attribute | Value |
filterString | ( type != "Employee" ) |
Different Filters for Different Objects - The filterString
applies to all objects which are aggregated. If you want to get more specific to accounts or groups which are filtered, you can use account.filterString
or group.filterString
to denote specific filters for those particular objects. This document assumes filterString
but you can also use the others as well.
Any account which matches this criteria will be filtered. The value you see is filter string syntax. The filter string reference guide follows this section. More complex values can be implemented, such as groupings, logical operators, etc.
The configuration of this is done via the IdentityNow Source Partial Update API. The details are as follows:
PATCH /beta/sources/{id}
Key | Value | Description |
authorization | Bearer {token} | This is the JWT OAuth token. |
content-type | application/json-patch+json | This is needed for PATCH operations. |
JSON Patch syntax representing the change:
content-type: application/json-patch+json
[
{
"op": "add",
"path": "/connectorAttributes/filterString",
"value": "( type != \"Employee\" )"
}
]
curl -X PATCH \
https://example.api.identitynow.com/beta/sources/2c9180835d191a86015d27ac132112ae \
-H 'Authorization: Bearer eyJ...BRM' \
-H 'Content-Type: application/json-patch+json' \
-H 'cache-control: no-cache' \
-d '[
{
"op": "add",
"path": "/connectorAttributes/filterString",
"value": "( type != \"Employee\" )"
}
]'
HTTP Code | HTTP Status | Description |
200 | OK | Returned if the request was successfully processed. |
401 | Unauthorized | Returned if there is no authorization header, or if the JWT token is expired. |
403 | Forbidden | Returned if the user you are running as, doesn't have access to this end-point. |
429 | Too Many Requests | Returned in response to too many requests in a given period of time - rate limited. The Retry-After header in the response includes how long to wait before trying again. |
500 | Internal Server Error | Returned if there is an unexpected error. |
content-type: application/json
A modified source object.
This is a filter reference to show how various account filters might be constructed. The expressions genreally follow the form:
{property} {operation} {value}
Here are additional rules for parsing correctly:
String literals should have double-quotes.
firstname == "Neil"
True / false values are treated as boolean literals
inactive != false
Digits are treated as numbers
age < 18
The string value 'null' (no quotes) is treated as null
name != null
Everything else is assumed to be the property name
email == contactAddress
Filters can also be grouped and used together as composite filters:
Composite Filter | Pattern | Example |
Grouping | ( {expression} ) |
!( type == "Employee" ) || (type == "Contractor" ) |
AND | ( {expression} && {expression} ) |
( type == "Employee" && location == "Austin" ) |
OR | ( {expression} || {expression} ) |
( type == "Employee" || type == "Contractor" ) |
NOT | !( {expression} ) |
!( company == "SailPoint" ) |
Note: Any comparison operator can be prepended with an 'i' to signify a case-insensitive comparison (eg - i==, i!=, etc...).
Operation | Pattern | Example |
Equals | {property} == {value} |
firstname == "Neil" |
Not Equals | {property} != {value} |
lastname != "Smith" |
Less Than | {property} < {value} |
age < 18 |
Greater Than | {property} > value |
age > 18 |
Less Than, Equals | {property} <= {value} |
age <= 18 |
Greater Than, Equals | {property} >= value |
age >= 18 |
Is Null | {property}.isNull() |
email.isNull() |
Not Null | {property}.notNull() |
company.notNull() |
Is Empty | {property}.isEmpty() |
Groups.isEmpty() |
Like, Exact | {property} == {value} |
firstname == "Neil" |
Like, Start | {property}.startsWith( {value} ) |
lastname.startsWith( "Mc" ) |
Like, Start (Ignoring Case) |
{property}.startsWithIgnoreCase( {value} ) |
lastname.startsWithIgnoreCase( "Mc" ) |
Like, End | {property}.endsWith( {value} ) |
email.endsWith( "@sailpoint.com" ) |
Like, End (Ignoring Case) |
{property}.endsWithIgnoreCase( {value} ) |
email.endsWithIgnoreCase( "@sailpoint.com" ) |
Like, Anywhere | {property}.contains( {value} ) |
email.contains( "sail" ) |
Like, Anywhere (Ignoring Case) | {property}.containsIgnoreCase( {value} ) |
email.containsIgnoreCase( "sail" ) |
Contains All | {property}.containsAll({ {value}, {value}, {value}, ... }) |
Groups.containsAll( { "A", "B", "C" } ) |
Contains All (Ignoring Case) |
{property}.containsAllIgnoreCase({ {value}, {value}, {value}, ... }) |
Groups.containsAllIgnoreCase( { "A", "B", "C" } ) |
Is there other ways to update and aggregate a single identity from a source or is account filtering the best approach? For example, if i wanted to update a delimited file connected source can I use the IdentityNow API to add or update a single identity instead of aggregating the entire source?
this filter also seems to work on entitlement aggregation, at least on the Azure AD source. Is that intended or should I create a ticket?
what if the attribute name has a space at middle? For instance, in AWS direct connect source, it has an attribute, named as "Access Keys". We would like to set up the filter so we aggregate only AWS users that have "Access Keys"
Tried to set filter like,
Access Keys.isNull()
but it doesn't work. If we put "Access Keys".isNull(), then "Access Keys" is treated as string literal.
Does the attribute have to be part of the Account Schema for the Source?
Yes @chrisp , Based on my testing it looks like attribute must be part of account schema
Hello,
@neil_mcglennon @piyush_khandelwal @lisa_ivy
To fully understood the flow with a custom openconnector:
1 - I create a filter in a Java "fashion" (looks like in Java)
2 - I update the Source Configuration of the openconnector
3 - When an aggregation starts, the system get the filterString attribute
a - parse the content (if error what append ?)
b - Convert Java like filter as pojo Filter object
c - Launch the iterate method with this pojo (see iterate(Filter filter) in openconnector.Connector)
4 - When iterate method is launch, the connector parse and validate the pojo
5 - Transforms the Filter pojo to something that filter data
Is that correct ?
Or the system launch the Aggregation (the pojo Filter is null) and when iterate on the result apply the filter on it ?
Best regards,
Stéphane POPOFF
Hello can anyone explain if it works correctly?
here some tests:
( lastname.startsWithIgnoreCase( "value1" ) || jobtitle.startsWithIgnoreCase( "value2" ) || lastname.startsWithIgnoreCase( "value3" ) || lastname.startsWithIgnoreCase( "value4" ) ) ---> no effect
( LastName != \"value1\" && !(lastname.startsWithIgnoreCase( \"value2\" )) ) --> no effect
( LastName != \"value1\" && lastname.startsWithIgnoreCase( \"value1\" ) ) --> no effect
( LastName != \"value1\" || !(lastname.startsWithIgnoreCase( \"value2\" )) ) --> no effect
(lastname.startsWithIgnoreCase( \"value1\" )) --> no effect, returns all
!(lastname.startsWithIgnoreCase( \"value1\" )) --> returns 0
( LastName != \"value1\" && LastName != \"value2\" ) --> return 2, the only that works
!(lastname.startsWith( \"value1\" )) --> return 0
(lastname.startsWith( \"value1\" )) --> no effect, returns all
In particular how to have the .startsWith functioning?
Thanks
Just a few observations from my attempts on using this to get accounts only for a specific manager and his team:
I hope this info might help others succeed :wink:
Hi, I'm back on this feature. How to query attributes with complex name such as "Password Deactivated"?
Thanks
Hi,
If the value of the filter string contains a space in it then operations "==" and "!=" do not work to filter out the accounts. In such cases containsIgnoreCase operator has to be used to match the value.
eg: For below filterString will filter out all the accounts which has value "WM EMPLOYEE" for the "User Groups" attribute.
"value": "((User Groups.containsIgnoreCase(\"WM EMPLOYEE\")))"
Hi,
I'm trying to add filter for Workday source but it's not working as expected.
We are trying to filter accounts based on specific worker type and also want to exclude few users from the filtration.
I tried multiple ways by changing the sequence and syntax but not getting expected result.
Below are some tried filters.
(EMPLOYEE_TYPE == \"Contractors\" && (FILENUMBER != \"458321\" || FILENUMBER != \"5693246\" || FILENUMBER != \"783692\"))
(EMPLOYEE_TYPE == \"Contractors\" && ((FILENUMBER != \"458321\") || (FILENUMBER != \"5693246\") || (FILENUMBER != \"783692\")))
It works only if I add one condition of FILENUMBER
(EMPLOYEE_TYPE == \"Contractors\" && FILENUMBER != \"458321\")
Requirement is to filter all contractors except three users.
Wanted to know if the user filter is case-sensitive eg: will pull in both _admin as well as _Admin accounts
Hi,
endsWith condition in user filter settings for Azure AD connector is not taking effect and returns all records.
"userPrincipalName.endsWithIgnoreCase(\"Value1\") || userPrincipalName.endsWithIgnoreCase(\"Value2\") || userPrincipalName.endsWithIgnoreCase(\"Value3\")"
Any thoughts ?
Hi,
We are trying to perform group filtering on a SAP based system where we have to get the "Roles" (which is mapped to Name attribute in group schema) that start with certain characters. We are trying to use something like below but exactly after 3 minutes, we get an aggregation error. When we tried something different and simple as well, we are facing the same issue.
"group.filterString": !("Name.startsWith(\"AB_\"))"
We tried a simpler one as well like "group.filterString": "Name != \"AB_ALL_INFO\"",
Error: Exception during aggregation of Object Type group on Application SAP System: DE8 [source]. Reason: java.lang.RuntimeException: An error occurred while aggregating Application SAP System: DE8 [source]
Anyone facing/faced a similar issue?
@gvscdeep Did you find a solution ? We currently have the same issue.
Not yet @benjamin_moya_synetis. For now we resorted to just using the account aggregation as that is giving us the required entitlements for now. But we will have to find a solution down the line.
@gvscdeep you can use this for SAP role filter:
Does the filtering (filterString) work on SaaS sources as well?
@rsh3ll as far as I know, the SaaS conectors do not support filterString, SaaS connectors only suport filtering if specified in the UI configuration of the SaaS Source Connector.
is there any way to findout the data that are excluded by this filter?