Partitioned rule task

13 Kudos

Summary
Motivation
Integrating

Summary

IIQ allows you to run any Rule as a Task via the TaskDefintion called “Run Rule”. This is fine for most use cases, but in our customization, we have found that we want to run a Rule on all of our Identities or ManagedAttributes. In these cases, a single partitioned solution is not practical even if it is multithreaded by convention means. To this end, I have developed a new TaskDefinition type “Partitioned Rule”. This TaskDefinition takes a rule, SailPointObject class name, and a few other parameters to partition your objects and get reasonable running time for rules that need to run on all of your objects.

Motivation

The specific motivation for making this TaskDefinition, and its infrastructure, was to fulfill a business requirement to trigger PolicyViolations on ManagedAttributes (AD Groups). As PolicyViolations are set up today, they only run on Identities. In my tinkering found how I could use the existing advance Policy template to take a ManagedAttribute and create the PolicyViolation but we lost the inbuilt partitioning granted to us by using an Identity refresh. This was simply unacceptable; It would take over 12 hours to chug through all of the objects. If we used the Rule, even if it were multi threaded, it would have taken over 8 hrs. The solution was to create this “Partitioned Rule” TaskDefinition. On implementation, all the objects are assigned to a partition and executed individually, similar to how Identity refresh works.

Integrating

We use the SSB (Services Standard Build) for our build and having an understanding of how to customize IIQ using the SSB will be helpful. The main reason is that I use several Java files to implement this infrastructure of “Partitioned Rule”. If anyone out there wants to make, a purely BeanShell version feel free to reply to this thread or message me so we can get the code out there.

Essentially you simply need to download the code, put it into your source, build, deploy, and import. Double check your package locations match between your Java files and BeanShell. Once imported you should be able to go to the Task window, click the “New Task” in the upper right, scroll down to “Partitioned Rule” and start filling out the fields.

“objectClassName” is the name of the SailPointObject that you want to iterate over. This is slightly technical, but in the Identity refresh case, you are iteration over “Identity”. In my example above I am iteration over “ManagedAttribute”. Technically you can iterate over any SailPointObject like TaskResult, Policy, Audit, and much more.
“rule” is the Rule that you want to run on each object. The thing to note here is that you are running this rule on every object one at a time, as such, your Rule should expect a single object to act on. You do not want your Rule to expect to be running on a list of objects.
“rConfig” is the config that is passed to the Rule that you supplied above. You don’t need to supply the SailPointObject or TaskDefinition, “Partitioned Rule” does this for you. But if your Rule has other configurations you wish to supply this is where you would do it. We use this field for maxRetries and timeout values. If you have no config for your Rule, you can leave this blank.
“debugMode” is a safety flag that I added for our use case. “Partitioned Rule” only passes this value along to your Rule. If you choose not to implement “debugMode” in your Rule, then you can leave this field blank. If you do implement it, you should supply the string “true” or “false”. 6.4p4 had a weird quirk with the boolean check make that did not behave as expected if you uncheck the box. I have not tried in 7.1 yet.
“partitionSize” is the size that you want each partition to be. I have a limiter in the code that if you go over 100 partitions, it instantly fails. This is to avoid accidentally supplying a small number and getting a huge number of partitions bring your task to a crawl.

If you have any questions feel free to reply to this thread, I’ll do my best to get back to you. And, if you are interested in how I finished solving the Policy problem above let me know, and I’ll work on a new article for an article addressing it.

jotorres · ‎May 29, 2018

Let me know if you have anymore questions. I'm happy to help.

jotorres · ‎Jun 28, 2018

I'm sorry I missed this before. Here is example of code that will delete all of the given type of sailpoint object. i use it in my sandbox all the the time. form the Tasks tab you just need to type the name of the class of the objects you want to delete. useful for sandbox if you want to delete 20,000 Identity or 60,000 ManagedAttribute.

TaskDefinition xml:

<?xml version='1.0' encoding='UTF-8'?>

<!DOCTYPE TaskDefinition PUBLIC "sailpoint.dtd" "sailpoint.dtd">

<Map>

<value>

<Map>

</Map>

</value>

</entry>

</Map>

</Attributes>

</Parent>

</TaskDefinition>

Rule xml (I had to edit it a little bit because mine is using some helper functions for deleting but this should be right.):

<?xml version='1.0' encoding='UTF-8'?>

<!DOCTYPE Rule PUBLIC "sailpoint.dtd" "sailpoint.dtd">

<Description>How to purge Links for specific Applications.</Description>

The log object associated with the SailPointContext.

</Description>

</Argument>

A sailpoint.api.SailPointContext object that can be used to query the database if necessary.

</Description>

</Argument>

If work should be done debugMode should be false.

</Description>

</Argument>

The object you are working with

</Description>

</Argument>

</Inputs>

</Signature>

import org.apache.log4j.Logger;

import java.util.Collection;

import java.util.HashSet;

import sailpoint.api.Terminator;

Logger logger = Logger.getLogger("DeleteIdentityRule");

Collection DO_NOT_INCLUDE = new HashSet();

DO_NOT_INCLUDE.add("spadmin");

logger.trace("TaskResult xml:\n"+taskResult.toXml());

logger.trace("debugMode: "+debugMode);

logger.trace("object: "+object);

logger.trace("object xml:\n"+object.toXml());

if (DO_NOT_INCLUDE.contains(object.getName())||DO_NOT_INCLUDE.contains(object.getId())) {

logger.info("Skipping - Do not include: " + object.getName());

return "ignored";

}

if (debugMode) {

logger.info("Skipping - DebugMode: " + object.getName());

return "ignored";

}

Terminator terminator = new Terminator(context);

terminator.deleteObject(object);

context.commitTransaction();

context.decache(object);

return "updated";

</Source>

</Rule>

enakshi_tarafdar · ‎Jul 19, 2018

Hi Tony,

We have implemented this on the number of records present in a file. So in our case, the number of partitions will be created based on no of records and partition size (No of partitions = No of records / Partition Size).

Now when we are mentioning partition size as 200, the request objects are getting created. But the executor class is not picking all the requests. When we are giving lesser partitioning size, the executor class is able to pick all the requests. So is there any limitation the number of records in a particular partition?

jotorres · ‎Jul 19, 2018

There should not be. As I have it written the task will fail immediately if you will create more then 100 partitions but that guard was only there to prevent you from accidentally making 1000's of partitions. We run ours with about 3000 objects per partition without issue. can you send me your xmls to see if I can find anything.

enakshi_tarafdar · ‎Jul 20, 2018

PartitionedRuleContainer class (Logic for reading from file and creating the partitions has been written in this class)

--------------------------------------------------

package custom.rule.task;

import java.io.BufferedReader;

import java.io.File;

import java.io.FileNotFoundException;

import java.io.FileReader;

import java.io.IOException;

import java.util.ArrayList;

import java.util.Iterator;

import java.util.List;

import java.util.Map;

import java.util.TreeMap;

import org.apache.log4j.Logger;

import com.google.common.collect.Lists;

import sailpoint.api.RequestManager;

import sailpoint.api.SailPointContext;

import sailpoint.object.Filter;

import sailpoint.object.Identity;

import sailpoint.object.Link;

import sailpoint.object.PolicyImpactAnalysis;

import sailpoint.object.QueryOptions;

import sailpoint.object.Request;

import sailpoint.object.RequestDefinition;

import sailpoint.object.Rule;

import sailpoint.object.SailPointObject;

import sailpoint.object.TaskItemDefinition;

import sailpoint.object.TaskResult;

import sailpoint.tools.GeneralException;

import sailpoint.tools.Message;

import ch.ethz.ssh2.Connection;

import ch.ethz.ssh2.ConnectionInfo;

import ch.ethz.ssh2.InteractiveCallback;

import ch.ethz.ssh2.SCPClient;

//import ch.ethz.ssh2.Session;

import java.nio.file.Path;

import java.nio.file.Paths;

import com.jcraft.jsch.Channel;

import com.jcraft.jsch.ChannelSftp;

import com.jcraft.jsch.JSch;

import com.jcraft.jsch.JSchException;

import com.jcraft.jsch.Session;

import com.jcraft.jsch.SftpException;

import sailpoint.object.Application;

public class PartitionedRuleContainer {

private static final int MAX_PARTITIONS = 100;

private static Logger logger = Logger.getLogger("PartitionedRuleContainer");

private int partitionSize = 1000;

public void setPartitionSize(int partitionSize) {

this.partitionSize = partitionSize;

}

public String runPartitionedReadyRule(String ruleIdentifier, String ruleConfig, String applicationName,

String readFilePath, boolean debugMode, SailPointContext context, TaskResult taskResult)

throws GeneralException, ClassNotFoundException, InterruptedException, IOException, JSchException, SftpException {

//System.out.println("applicatioName::::::::" + applicationName);

//System.out.println("ruleId: " + ruleIdentifier);

Rule rule = getRule(ruleIdentifier, context);

//System.out.println("ruleName: " + rule.getName());

PolicyImpactAnalysis policyImpactAnalysis = new PolicyImpactAnalysis();

policyImpactAnalysis.setPolicyName("placeHolder");

//System.out.println("111111");

taskResult.setAttribute("violationResults", policyImpactAnalysis);

taskResult.setPartitioned(true);

//System.out.println("222222222");

RequestDefinition definition = context.getObjectByName(RequestDefinition.class, "Partitioned Rule Definition");

List<Request> requests = Lists.newArrayList();

List objectList = null;

int numberOfObjects = 0;

String objectClassName = null;

//System.out.println("33333333333333");

BufferedReader br = null;

int numberOfRecords = -1;

//----------------------------------------sftp---------------------------------

int scpPort = 22;

String scpHost = "";

String scpUser = "";

String scpEncryptedPassword = "";

String inputFileName = "";

String current = "";

String appName = "LDAP";

Application appObject = context.getObjectByName(Application.class, appName);

if (appObject != null) {

scpHost = (String) appObject.getAttributeValue("host");

scpUser = (String) appObject.getAttributeValue("transportUser");

scpEncryptedPassword = (String) appObject.getAttributeValue("transportUserPassword");

}

JSch jsch = new JSch();

Session session = null;

session = jsch.getSession(scpUser, scpHost, scpPort);

session.setConfig("StrictHostKeyChecking","no");

session.setPassword(context.decrypt(scpEncryptedPassword));

session.connect();

Channel channel = session.openChannel("sftp");

channel.connect();

ChannelSftp sftpChannel = (ChannelSftp) channel;

current = new java.io.File( "." ).getCanonicalPath();

Path source = Paths.get(readFilePath);

inputFileName = source.getFileName().toString();

sftpChannel.get(readFilePath,current);

sftpChannel.exit();

String inputFilePath = current+"/"+inputFileName;

//----------------------------------------sftp---------------------------------

if (inputFilePath != null) {

String line = "";

try {

br = new BufferedReader(new FileReader(inputFilePath));

while ((line = br.readLine()) != null) {

numberOfRecords++;

}

} catch (FileNotFoundException e) {

e.printStackTrace();

}

//System.out.println("4444444444444444");

if (applicationName == null && inputFilePath != null) {

getRequestFromFile(partitionSize, numberOfRecords, inputFilePath, taskResult, ruleConfig, rule, definition,

requests, debugMode);

}

int numberOfPartitions = numberOfObjects / partitionSize;

//System.out.println("numberOfPartitions::::::::::" + numberOfPartitions);

if (numberOfPartitions > MAX_PARTITIONS) {

String key = "Too few items per partition. Please increase partitionSize.";

Message message = new Message();

message.setKey(key);

message.setType(Message.Type.Error);

taskResult.addMessage(message);

return "Fail";

}

//System.out.println("::::::::::out of first for loop:::::::::::");

for (Request request : requests) {

// System.out.println(":::::::::in for:::::::::::");

RequestManager.addRequest(context, request);

}

//System.out.println("::::::::::at last of method:::::::::::");

//--------------------------------------------sftp file delete---------------------------------------

boolean Success = (new File(current+"/"+inputFileName)).delete();

return "Success";

}

private Rule getRule(String ruleId, SailPointContext context) throws GeneralException {

Rule rule = context.getObjectById(Rule.class, ruleId);

if (rule == null) {

rule = context.getObjectByName(Rule.class, ruleId);

}

return rule;

}

private void getRequestFromFile(int partitionSize, int numberOfObjects, String inputFilePath, TaskResult taskResult,

String ruleConfig, Rule rule, RequestDefinition definition, List requests, boolean debugMode)

throws GeneralException, ClassNotFoundException, InterruptedException, IOException {

int numberOfPartitions = numberOfObjects / partitionSize;

BufferedReader br = null;

long partitionPointer = 1;

String line = null;

long lineNo = 0;

String[] header = null;

List<String> columnList = new ArrayList();

int i = 0;

br = new BufferedReader(new FileReader(inputFilePath));

while (br.ready()) {

if (lineNo < partitionPointer) {

//System.out.println("Inside skip");

if (null == line)

line = br.readLine();

if (null != line) {

header = line.split(",");

// System.out.println("header length: " + header.length);

for (int j = 0; j < header.length; ++j) {

//System.out.println("header: " + header[j]);

columnList.add(header[j]);

}

lineNo++;

// line = br.readLine();

line = null;

}

else {

// System.out.println("Inside process");

String[] column = null;

String[] val = null;

List<String> userList = new ArrayList();

Map<String, Map> userMap = new TreeMap();

TreeMap valueMap = new TreeMap();

TreeMap valueMapNew = new TreeMap();

int key = 0;

for (int counter = 0; counter < partitionSize; counter++) {

// System.out.println("counter: " + counter);

// System.out.println("partitionSize: " + partitionSize);

if (null == line)

line = br.readLine();

if (lineNo == 0) {

//System.out.println("This is header");

if (null != line) {

column = line.split(",");

//System.out.println("column length: " + column.length);

for (int j = 0; j < column.length; ++j) {

//System.out.println("column: " + column[j]);

}

else {

key++;

int newKey = 0;

if (null != line) {

val = line.split(",");

// System.out.println("val length: " + val.length);

valueMap.clear();

for (int j = 0; j < val.length; ++j) {

// System.out.println("val : " + val[j]);

newKey++;

String innerKey = "innerKey" + newKey;

//System.out.println("innerKey: " + innerKey);

valueMap.put(innerKey, val[j]);

}

// System.out.println("valueMap: " + valueMap);

valueMapNew = (TreeMap) valueMap.clone();

String outerKey = "outerKey" + key;

// System.out.println("outerKey: " + outerKey);

userMap.put(outerKey, valueMapNew);

// System.out.println("userMap in class file::::::::::::: " + userMap.size());

}

lineNo++;

line = br.readLine();

}

for (Map.Entry eOuter : userMap.entrySet()) {

String kOuter = (String) eOuter.getKey();

Map<String, String> vOuter = (Map) eOuter.getValue();

// System.out.println("-------------outer details----------------");

// System.out.println(kOuter + " : " + vOuter);

for (Map.Entry eInner : vOuter.entrySet()) {

String kInner = (String) eInner.getKey();

String vInner = (String) eInner.getValue();

//System.out.println("--------------inner details-------------");

//System.out.println(kInner + " : " + vInner);

if (kInner.equals("innerKey1")) {

userList.add(vInner);

break;

}

// System.out.println("userList:" + userList);

partitionPointer = partitionPointer + partitionSize;

//// System.out.println("lineNo:" + lineNo);

// System.out.println("partitionPointer: " + partitionPointer);

String partitionName = "partition " + (i + 1);

// System.out.println("partitionName: " + partitionName);

// System.out.println("1st check: " + (numberOfObjects % partitionSize));

if (i == (numberOfPartitions) && (numberOfObjects % partitionSize) == 0) {

logger.info("partition: '" + partitionName + "' is empty, skipping");

continue;

}

//System.out.println("2nd check: " + (i * partitionSize));

System.out.println("partitionName:::::::::::::::"+partitionName);

TaskResult partitionedTaskResult = new TaskResult();

partitionedTaskResult.setName(partitionName);

taskResult.getPartitionResults().add(partitionedTaskResult);

Request request = new Request();

request.setDependentPhase(1);

request.setPhase(2);

request.setTaskResult(taskResult);

request.setLauncher(taskResult.getLauncher());

request.setName(partitionName);

request.setType(TaskItemDefinition.Type.Partition);

request.setDefinition(definition);

request.setPartitionResult(partitionedTaskResult);

// request.addAttribute("firstRowObjectId", firstRowObjectId);

request.addAttribute("rule", rule);

request.addAttribute("ruleConfig", ruleConfig);

request.addAttribute("debugMode", debugMode);

request.addAttribute("partitionSize", partitionSize);

// request.addAttribute("objectClass", objectClass);

request.addAttribute("columnList", columnList);

request.addAttribute("userList", userList);

// request.addAttribute("userMapNew", userMapNew);

request.addAttribute("userMap", userMap);

System.out.println("Req in Class::::::::::::::::::::"+request);

System.out.println("Req in Class::::::::::::::::::::"+request.toXml());

requests.add(request);

i++;

}

Kindly let me know if anything else is needed.

Regards,

Enakshi

jotorres · ‎Jul 30, 2018

Sorry this reply is so late. We were in the home stretch for a big project the blew up on the last day. everything is looking good now.

I am looking though your code and I seems you have replaced PartitionedRuleContainer with your own implementation to ready from a file. I believe there are some incompatibilities between how your are splitting your partitions and how I was.

For context my original code only saved the first object to lookup and the number of records to read after that. so if partition 0 was firstRowObjectId=0 and partitionSize=100 then partition 2 would be firstRowObjectId=100 and partitionSize=100 and so on. This is done to save time when writing the requests to the database. in the case where you have 3000 records to run it takes a long time to save the ~80 requests with all 3000 object guid in each request. with the firstRowObjectId and partitionSize we just save 2 values and partition creation is fast.

The main thing i see is that it seems like you are adding all of your file rows into "columnList" but you never reset the value so your second request has all of the values from your 1st request + all the values from your second request and your last request all your whole file. this might look like the executor is "not be picked up" the request but rather as it starts requests that are larger the take longer to complete and then may appear to be hanging. doing this also negate the purpose for partitioning because you are running everything on your last partition anyway. and finally it is potentially harmful to your processing if you are expecting there to only be one run for row. This can be fixed by adding "columnList = new ArrayList();" when you do "i++;" at the end of your else in the the for loop.

Aside from that everything seems ok from here. if you are still having issues it might be in your rules implementation.

yubraj_budhathoki · ‎Aug 15, 2018

Hi Tony,

How do I put in filter if I want to run the rule for only certain number of identities? Is there a way to run this task for a population.

Thanks

jotorres · ‎Aug 15, 2018

That feature is not built in but you can add to the rule config a filtering a field. Then edit "PartitionedRuleContainer" to use that value to filter. Or you can allow the filter to pass down into the implementation, the rule that your wrote for your objects, and do a check to see if your object matches your filter, if it doesn't just return without doing anything.

If you go with the first suggestion you might want to make a new argument in TaskDefinition-PartitionedRule.xml to take this value. Then when you use it the two fields won't be confused in the same variable. But that's not really required.

atanu_ghosh · ‎Dec 04, 2018

Hi Tonny,

I tried you approach, but RuleRequestExecutor is not getting called from PartitionedRuleContainer.

Here is Request Definiation -

<Map>

</Map>

</Attributes>

</RequestDefinition>

Here is the PartitionedRuleContainer -

public class PartitionedRuleContainer {

private static final int MAX_PARTITIONS = 100;

private static Logger logger = Logger.getLogger("PartitionedRuleContainer");

private int partitionSize = 1000;

public void setPartitionSize(int partitionSize) {

this.partitionSize = partitionSize;

}

public <T extends SailPointObject> String runPartitionedReadyRule(String ruleIdentifier, String ruleConfig,

String objectClassName,int numberOfObjects, boolean debugMode, SailPointContext context, TaskResult taskResult, Set cssIDSet)

throws GeneralException, ClassNotFoundException, InterruptedException {

Class<T> objectClass = (Class<T>) Class.forName("sailpoint.object." + objectClassName);

//int numberOfObjects = context.countObjects(objectClass, new QueryOptions());

System.out.println("numberOfObjects: " + numberOfObjects);

System.out.println("partitionSize: " + partitionSize);

int numberOfPartitions = numberOfObjects / partitionSize;

System.out.println("numberOfPartitions: " + numberOfPartitions);

if (numberOfPartitions > MAX_PARTITIONS) {

String key = "Too few items per partition. Please increase partitionSize.";

System.out.println(key + " - partitionSize" + partitionSize + " - numberOfObjects: " + numberOfObjects

+ " - numberOfPartitions: " + numberOfPartitions);

Message message = new Message();

message.setKey(key);

message.setType(Message.Type.Error);

taskResult.addMessage(message);

return "Fail";

}

PolicyImpactAnalysis policyImpactAnalysis = new PolicyImpactAnalysis();

policyImpactAnalysis.setPolicyName("placeHolder");

taskResult.setAttribute("violationResults", policyImpactAnalysis);

taskResult.setPartitioned(true);

System.out.println("ruleId: " + ruleIdentifier);

Rule rule = getRule(ruleIdentifier, context);

System.out.println("ruleName: " + rule.getName());

RequestDefinition definition = context.getObjectByName(RequestDefinition.class, "Partitioned Rule Definition");

List<Request> requests = Lists.newArrayList();

List<List<String>> partitionCSSIDList = MyPartition.partition(new ArrayList(cssIDSet),partitionSize);

System.out.println("total partitionCSSIDList: " + partitionCSSIDList);

for (int i = 0; i <= numberOfPartitions; i++) {

String partitionName = "partition " + (i + 1);

if (i == (numberOfPartitions) && (numberOfObjects % partitionSize) == 0) {

System.out.println("partition: '" + partitionName + "' is empty, skipping");

continue;

}

System.out.println("partitioning " + (i + 1) + " out of " + (numberOfObjects / partitionSize + 1)

+ " partitionName: " + partitionName);

List cssIDList = partitionCSSIDList.get(i);

System.out.println("cssIDList in partition: " + cssIDList);

TaskResult partitionedTaskResult = new TaskResult();

partitionedTaskResult.setName(partitionName);

taskResult.getPartitionResults().add(partitionedTaskResult);

Request request = new Request();

request.setDependentPhase(1);

request.setPhase(2);

request.setTaskResult(taskResult);

request.setLauncher(taskResult.getLauncher());

request.setName(partitionName);

request.setType(TaskItemDefinition.Type.Partition);

request.setDefinition(definition);

request.setPartitionResult(partitionedTaskResult);

request.addAttribute("rule", rule);

request.addAttribute("ruleConfig", ruleConfig);

request.addAttribute("debugMode", debugMode);

request.addAttribute("partitionSize", partitionSize);

request.addAttribute("objectClass", objectClass);

request.addAttribute("cssIDListToProcess", cssIDList);

System.out.println("running rule ***************** " );

}

for (Request request : requests) {

RequestManager.addRequest(context, request);

}

return "Success";

}

private Rule getRule(String ruleId, SailPointContext context) throws GeneralException {

Rule rule = context.getObjectById(Rule.class, ruleId);

if (rule == null) {

rule = context.getObjectByName(Rule.class, ruleId);

}

return rule;

}

You help is much appreciated.

atanu_ghosh · ‎Dec 05, 2018

Hi Tony,

I tried with exactly your code with following Rule, however RuleRequestExecutor is not getting called after PartitionedRuleContainer.

Could you please help to resolve ? Let me know if you need additional details

Rule -

<?xml version='1.0' encoding='UTF-8'?>

<!DOCTYPE Rule PUBLIC "sailpoint.dtd" "sailpoint.dtd">

import sailpoint.object.*;

import java.util.*;

System.out.println("**********From Rule**************");

</Source>

</Rule>

Partitioned rule task