Partitioned rule task

[#1]

• Summary
• Motivation
• Integrating

Summary

IIQ allows you to run any Rule as a Task via the TaskDefintion called “Run Rule”. This is fine for most use cases, but in our customization, we have found that we want to run a Rule on all of our Identities or ManagedAttributes. In these cases, a single partitioned solution is not practical even if it is multithreaded by convention means. To this end, I have developed a new TaskDefinition type “Partitioned Rule”. This TaskDefinition takes a rule, SailPointObject class name, and a few other parameters to partition your objects and get reasonable running time for rules that need to run on all of your objects.

Motivation

The specific motivation for making this TaskDefinition, and its infrastructure, was to fulfill a business requirement to trigger PolicyViolations on ManagedAttributes (AD Groups). As PolicyViolations are set up today, they only run on Identities. In my tinkering found how I could use the existing advance Policy template to take a ManagedAttribute and create the PolicyViolation but we lost the inbuilt partitioning granted to us by using an Identity refresh. This was simply unacceptable; It would take over 12 hours to chug through all of the objects. If we used the Rule, even if it were multi threaded, it would have taken over 8 hrs. The solution was to create this “Partitioned Rule” TaskDefinition. On implementation, all the objects are assigned to a partition and executed individually, similar to how Identity refresh works.

Integrating

We use the SSB (Services Standard Build) for our build and having an understanding of how to customize IIQ using the SSB will be helpful. The main reason is that I use several Java files to implement this infrastructure of “Partitioned Rule”. If anyone out there wants to make, a purely BeanShell version feel free to reply to this thread or message me so we can get the code out there.

Essentially you simply need to download the code, put it into your source, build, deploy, and import. Double check your package locations match between your Java files and BeanShell. Once imported you should be able to go to the Task window, click the “New Task” in the upper right, scroll down to “Partitioned Rule” and start filling out the fields.

• “objectClassName” is the name of the SailPointObject that you want to iterate over. This is slightly technical, but in the Identity refresh case, you are iteration over “Identity”. In my example above I am iteration over “ManagedAttribute”. Technically you can iterate over any SailPointObject like TaskResult, Policy, Audit, and much more.

• “rule” is the Rule that you want to run on each object. The thing to note here is that you are running this rule on every object one at a time, as such, your Rule should expect a single object to act on. You do not want your Rule to expect to be running on a list of objects.

• “rConfig” is the config that is passed to the Rule that you supplied above. You don’t need to supply the SailPointObject or TaskDefinition, “Partitioned Rule” does this for you. But if your Rule has other configurations you wish to supply this is where you would do it. We use this field for maxRetries and timeout values. If you have no config for your Rule, you can leave this blank.

• “debugMode” is a safety flag that I added for our use case. “Partitioned Rule” only passes this value along to your Rule. If you choose not to implement “debugMode” in your Rule, then you can leave this field blank. If you do implement it, you should supply the string “true” or “false”. 6.4p4 had a weird quirk with the boolean check make that did not behave as expected if you uncheck the box. I have not tried in 7.1 yet.

• “partitionSize” is the size that you want each partition to be. I have a limiter in the code that if you go over 100 partitions, it instantly fails. This is to avoid accidentally supplying a small number and getting a huge number of partitions bring your task to a crawl.

If you have any questions feel free to reply to this thread, I’ll do my best to get back to you. And, if you are interested in how I finished solving the Policy problem above let me know, and I’ll work on a new article for an article addressing it.

[jotorres]

For this case where you are just testing it:

Technically "RuleRequestExecutor" will not be called *form* PartitionedRuleContainer but PartitionedRuleContainer will create a Request that RuleRequestExecutor will then pickup later. double check in the debug page that your *Request*s are being made in the first place.

Assuming that they are being made but never picked up. make sure that in RequestDefinition you are pointing to *your* src for the executor. I have it in "custom.rule.task.RuleRequestExecutor" for sharing but in our codebase it's realy in "com.expedia.custom.rule.task.RuleRequestExecutor"

<RequestDefinition executor="*custom.rule.task.RuleRequestExecutor*" name="Partitioned Rule Definition" retryMax="20">

For the above case check that you are not getting errors. I see that you have added some parameters if you didn't update the corisponding rule.xml then you will get errors saying that that no such method exsists.

Message let me know what you find.

[atanu_ghosh]

Hi Tonny,

Thank you for your response . Requests are being created. Here is one -

<?xml version='1.0' encoding='UTF-8'?>

<!DOCTYPE Request PUBLIC "sailpoint.dtd" "sailpoint.dtd">

<Request created="1544198829776" dependentPhase="1" id="2a8cab19678964680167896c36d00006" launcher="spadmin" name="partition 1" phase="2" type="Partition">

  <Attributes>

    <Map>

      <entry key="debugMode">

        <value>

          <Boolean></Boolean>

        </value>

      </entry>

      <entry key="firstRowObjectId" value="8aa080e353d31bed0153d31cfbbe0005"/>

      <entry key="objectClass">

        <value>

          <Class>sailpoint.object.ManagedAttribute</Class>

        </value>

      </entry>

      <entry key="partitionResult">

        <value>

          <TaskResult name="partition 1"/>

        </value>

      </entry>

      <entry key="partitionSize">

        <value>

          <Integer>10000</Integer>

        </value>

      </entry>

      <entry key="rule">

        <value>

          <Rule created="1541725052952" id="8aa080e366ef86750166f5f960170030" language="beanshell" modified="1544043164411" name="RJ Net Test">

            <Source>

import sailpoint.object.*;

    import java.util.*;

System.out.println("**********From Rule**************");

  </Source>

          </Rule>

        </value>

      </entry>

      <entry key="ruleConfig"/>

    </Map>

  </Attributes>

  <Definition>

    <Reference class="sailpoint.object.RequestDefinition" id="2a8cab196779edc8016779f095840004" name="Partitioned Rule Definition"/>

  </Definition>

  <TaskResultRef>

    <Reference class="sailpoint.object.TaskResult" id="2a8cab19678964680167896bdfe80004" name="Atanu Ad Partition"/>

  </TaskResultRef>

</Request>

Ideally it should print "**********From Rule**************". However it is not. Also the task keeps on running.

One important point to mention - Do I need to mention any Host while running the task? It is showing blank in the

Task result page. I am running it from my ocal m/c (localhost)

[atanu_ghosh]

Here is the screen shot from task result page -

[jotorres]

it's blank because no host has picked it up.  it waiting for the code in "RequestDefinition" - "Partitioned Rule Definition" to pick it up. is "custom.rule.task.RuleRequestExecutor" in your code and is it on that path. Maybe navigate to the file and screen shot the file path for me.

<RequestDefinition executor="custom.rule.task.RuleRequestExecutor" name="Partitioned Rule Definition" retryMax="20">

[atanu_ghosh]

Here is the RequestDefinition. I am using your one. The one that you mentioned is my custom one. I am trying with your version now.-

<?xml version='1.0' encoding='UTF-8'?>

<!DOCTYPE RequestDefinition PUBLIC "sailpoint.dtd" "sailpoint.dtd">

<RequestDefinition created="1543939069316" executor="custom.request.RuleRequestExecutor" id="2a8cab196779edc8016779f095840004" modified="1544031581934" name="Partitioned Rule Definition" retryMax="20">

  <Attributes>

    <Map>

      <entry key="maxThreads" value="%%PARTITIONED_RULE_MAX_THREADS%%"/>

    </Map>

  </Attributes>

</RequestDefinition>

Here is RuleRequestExecutor ->

Path is ->  custom.request.RuleRequestExecutor

I created Java Project and Put the jar inside C:\Project\apache-tomcat-7.0.78\webapps\identityiq\WEB-INF\lib

package custom.request;

import java.util.Iterator;

import java.util.List;

import java.util.Map;

import java.util.Map.Entry;

import org.apache.log4j.Logger;

import org.joda.time.DateTime;

import com.google.common.collect.Lists;

import com.google.common.collect.Maps;

import sailpoint.api.SailPointContext;

import sailpoint.object.Attributes;

import sailpoint.object.Filter;

import sailpoint.object.QueryOptions;

import sailpoint.object.Request;

import sailpoint.object.Rule;

import sailpoint.object.SailPointObject;

import sailpoint.object.TaskResult;

import sailpoint.request.AbstractRequestExecutor;

import sailpoint.request.RequestPermanentException;

import sailpoint.request.RequestTemporaryException;

import sailpoint.tools.GeneralException;

import sailpoint.tools.Message;

public class RuleRequestExecutor<T extends SailPointObject> extends AbstractRequestExecutor {

private static Logger logger = Logger.getLogger("RuleRequestExecutor");

@Override

public void execute(SailPointContext context, Request request, Attributes<String, Object> attributes)

throws RequestPermanentException, RequestTemporaryException {

DateTime start = new DateTime();

System.out.println("**********111111111111111***************");

try {

logger.trace("request xml:\n" + request.toXml());

} catch (GeneralException e) {

logger.error("Failed to print request xml.", e);

}

Rule rule = (Rule) request.getAttribute("rule");

String ruleConfig = (String) request.getAttribute("ruleConfig");

boolean debugMode = (boolean) request.getAttribute("debugMode");

Class<T> objectClass = (Class<T>) request.getAttribute("objectClass");

int partitionSize = (int) request.getAttribute("partitionSize");

String firstRowObjectId = (String) request.getAttribute("firstRowObjectId");

TaskResult partitionResults = request.getPartitionResult();

List<String> ids;

try {

ids = getObjectIds(context, objectClass, partitionSize, firstRowObjectId);

} catch (GeneralException e) {

logger.error("Failed to get objects.", e);

return;

}

DateTime ruleRunStart = new DateTime();

logger.info("setup time: " + (ruleRunStart.getMillis() - start.getMillis()));

Map<String, Object> partitionResultMap = runRule(rule, ruleConfig, ids, debugMode, objectClass,

partitionResults, context);

DateTime cleanUpStart = new DateTime();

logger.info("runRule time: " + (cleanUpStart.getMillis() - ruleRunStart.getMillis()));

updatePartitionedTaskResults(partitionResults, partitionResultMap);

DateTime end = new DateTime();

logger.info("cleanup time: " + (end.getMillis() - cleanUpStart.getMillis()));

logger.info("total time: " + (end.getMillis() - start.getMillis()));

}

private void updatePartitionedTaskResults(TaskResult partitionResults, Map<String, Object> partitionResultMap) {

for (Entry<String, Object> entry : partitionResultMap.entrySet()) {

if (entry.getValue() instanceof Integer && partitionResults.getAttribute(entry.getKey()) != null) {

partitionResults.setAttribute(entry.getKey(),

((int) partitionResults.getAttribute(entry.getKey()) + (int) entry.getValue()));

} else {

partitionResults.setAttribute(entry.getKey(), entry.getValue());

}

}

}

private List<String> getObjectIds(SailPointContext context, Class<T> objectClass, int partitionSize,

String firstObjectId) throws GeneralException {

QueryOptions queryOptions = new QueryOptions();

queryOptions.addFilter(Filter.ge("id", firstObjectId));

queryOptions.setResultLimit(partitionSize);

queryOptions.setOrderBy("id");

Iterator<Object[]> results = context.search(objectClass, queryOptions, "id");

List<String> ids = Lists.newArrayList();

while (results.hasNext()) {

Object[] result = results.next();

ids.add((String) result[0]);

}

if (ids.isEmpty()) {

logger.info("ids is empty");

} else {

logger.info("number of objects: " + ids.size());

logger.info("first id: " + ids.get(0));

}

return ids;

}

private Map<String, Object> runRule(Rule rule, String ruleInput, List<String> ids, boolean debugMode,

Class<T> objectClass, TaskResult partitionResult, SailPointContext context) {

Map<String, Object> partitionResultsMap = Maps.newHashMap();

try {

int i = 0;

for (String id : ids) {

SailPointObject object = context.getObjectById(objectClass, id);

logger.info("Working on ID: " + object.getId() + ", Name:" + object.getName());

try {

Map<String, Object> args = Maps.newHashMap();

args.put("ruleInput", ruleInput);

args.put("debugMode", debugMode);

args.put("object", object);

args.put("taskResult", new TaskResult());

logger.info("Running rule '" + rule + "' with following args:" + args);

Object ruleResults = context.runRule(rule, args);

logger.info("ruleResults: " + ruleResults);

try {

addResults(partitionResultsMap, ruleResults);

} catch (ClassCastException e) {

logger.warn("Type returned from rule '" + ruleResults.getClass().getName()

+ "' is not supported. Please return String or Map<String,Integer>");

}

} catch (Exception e) {

logger.error("Partitioned application rule caught exception", e);

logger.warn("Warning: problem running rule on object class: " + object.getClass().getName() + "id: "

+ object.getId());

if (partitionResultsMap.containsKey("exception")) {

partitionResultsMap.put("exception", ((int) partitionResultsMap.get("exception") + 1));

} else {

partitionResultsMap.put("exception", 1);

}

Message message = new Message(Message.Type.Error, e);

message.setKey("Error in running rule '" + rule.getName() + "' on object Id '" + object.getId()

+ "' of objectClass '" + objectClass.getSimpleName() + "'. - " + e.getMessage());

partitionResult.addMessage(message);

}

i++;

}

logger.info("Partition '" + partitionResult.getName() + "' runRule completed");

logger.info("Objects Acted on: " + i);

logger.info("Result Details:");

for (Entry<String, Object> partitionResultEntry : partitionResultsMap.entrySet()) {

logger.info(partitionResultEntry.getKey() + ": " + partitionResultEntry.getValue());

}

} catch (GeneralException e) {

throw new RuntimeException(e);

}

return partitionResultsMap;

}

private void addResults(Map<String, Object> partitionResults, Object ruleResults)

throws ClassCastException, GeneralException {

if (ruleResults instanceof Map) {

Map<String, Object> ruleReturnMap = (Map<String, Object>) ruleResults;

for (Entry<String, Object> result : ruleReturnMap.entrySet()) {

if (result.getValue() instanceof Integer) {

addResults(partitionResults, result.getKey(), (int) result.getValue());

} else {

partitionResults.put(result.getKey(), result.getValue());

}

}

} else if (ruleResults instanceof String) {

addResults(partitionResults, (String) ruleResults, 1);

} else {

throw new ClassCastException("Unsupported type");

}

}

private void addResults(Map<String, Object> partitionResults, String ruleReturnString, int value) {

if (partitionResults.containsKey(ruleReturnString)) {

partitionResults.put(ruleReturnString, ((int) partitionResults.get(ruleReturnString) + value));

} else {

partitionResults.put(ruleReturnString, value);

}

}

}

[atanu_ghosh]

One question -  Do you need to try in multiple host environment? I am trying in my local machine (localhost)?

[jotorres]

so long as the code is deployed and your server is set up to recive such calls (ie it is a task server) then you should be good.

how did you add this code to your exsisting code base? can you walk me though the process?

[jotorres]

"I created Java Project and Put the jar inside C:\Project\apache-tomcat-7.0.78\webapps\identityiq\WEB-INF\lib"

what jar?

[atanu_ghosh]

Here are the steps I followed-

1. In Eclipse I created a Java Project

2. Add PartitionedRuleContainer and RuleRequestExecutor in that project

3. Added required jar files (like -  google-collections-1.0-rc2.jar) in the project

4. Build locally using Eclipse

5. Crated jar file from the java project using Eclipse Export .

6. Put the jar file in C:\Project\apache-tomcat-7.0.78\webapps\identityiq\WEB-INF\lib

6. Restart Tomcat.

Here is my project structure -

[jotorres]

I'm not sure if what you did is equlivelent to SSB (Services Standard Build (SSB) v4). I sugest that you use the SSB to compile everything into a war file and then extract the war on your servre and run tomcat. If you do that then everything will be linked correctly and you should not need to add any new jars (like google-collections, joda-time, or log4j)