We are currently using a flat-file (CSV) source to aggregate HR data
into Identity Security Cloud. I am trying to understand how ISC
handles duplicate rows in the file — i.e., two or more rows with the
same correlation key (Employee ID) within a single aggregation.
In sandbox testing, I could not draw a consistent conclusion. For
example:
• Test 1: Two rows for the same Employee ID, with the first name on
row 2 changed to a different value. Result: one identity created,
but with mixed attributes — first name from row 2 and last name
from row 1. No warning or error in the aggregation log.
• Test 2: Two rows for the same Employee ID, where row 1 had EE
Status = Terminated and row 2 had EE Status = Active. Result: the
identity ended up disabled (matching Terminated), but the
termination workflow fired again even though the user was already
terminated.
In both cases, aggregation reported success. The total identity count
matched the unique Employee ID count (so duplicates were not created
as separate identities), but the merged attribute values on the final
identity were not predictable from the file alone.
A few questions:
1. Is there a documented behaviour for how ISC processes duplicate
rows in a flat-file source? Is it first-row-wins, last-row-wins,
or attribute-level merge?
2. Is there a way to configure the source so that ISC logs a
warning or rejects rows when duplicate correlation keys are
detected within an aggregation?
3. Is the recommended pattern to handle duplicates upstream (clean
the file before aggregation) or at the SailPoint layer (via
source correlation configuration or a custom correlation rule)?
Any guidance, documentation links, or examples from similar setups
would be appreciated.
Hi @SreeshaC93 I believe one of the behaviors you are seeing is reported in the Troubleshooting section of the delimited file connector guide. Recommendations are found within the VA-based connector or SaaS connector guides, as I am unsure which one you are using:
1.) For the Identity Security Cloud delimited file connector, use multiple rows for defining multiple values for an account. If an account has the same uid in multiple rows, this is all mapped to a single identity. I tend to use this technique for defining groups and entitlements - 1 entitlement per row, where all other data is exactly the same for the user account. See these 2 pages in the connector guide for more info:
If you change the name or an attribute of an identity after initial aggregation, but only on 1 row, then this could be considered a new identity and I would imagine it does break the merging or data into the original identity. Hopefully the merging file suggestions above resolve this for you.
2.) I don’t know of a way in the UI to configure the delimited file source with additional warning messages. However:
If you are using the VA-based delimited file connector, you can configure the VA cluster with increased logging.
You might be able to configure the File upload utility with custom messages, and
The delimited file connector can also be configured with abuild map rule.
3.) It is recommended to ensure clean data in the file before aggregating.
Is this an authoritative source? If so, the identity profile > mappings (vs. correlation) defines the identity. I am unsure if a transform would achieve this goals, but that is worth investigating.
A Correlation rule is worth investigating, although I have not used one on a delimited file authoritative source before, so cannot claim it is the definitive solution for this use case.