Pre-iterate rule for the DelimitedFileConnector to merge two files and append extra column

[#1]

Example pre-iterate rule for the DelimitedFileConnector

Here is an example of a Pre-Iterate rule for the DelimitedFileConnector that can check the stats of the file being processed against stats collected and is persisted during the last aggregation in the PostIterate Rule.

This example will merge two files and append an extra column to it. 

<?xml version='1.0' encoding='UTF-8'?>

<!DOCTYPE sailpoint PUBLIC "sailpoint.dtd" "sailpoint.dtd">

<sailpoint>

<Rule name="preIterate-AspenCedar">

  <Description>

     An example Preiterate rule for the DelimitedFileConnector that

     can check stats of the file being processed against stats

     collected and persisted during the last aggregation

     in the PostIterate rule.

  </Description>

  <Signature>

    <Inputs>

      <Argument name='application'>

        <Description>

          Application being iterated.

        </Description>

      </Argument>

      <Argument name='schema'>

        <Description>

          Schema representing the data being iterated

        </Description>

      </Argument>

      <Argument name='stats'>

        <Description>

          A map passed by the connector of the stats for the file about to be iterated.

          Contains keys:

            fileName : (String) filename of the file about to be processed

            absolutePath : (String) absolute filename

            length : (Long) length in bytes

            lastModified : (Long) last time the file was updated Java GMT

        </Description>

      </Argument>

      <Argument name='log'>

        <Description>

          log for debugging

        </Description>

      </Argument>

    </Inputs>

  </Signature>

  <Source>

    <![CDATA[

      import sailpoint.api.SailPointFactory;

      import sailpoint.api.SailPointContext;

      import sailpoint.tools.GeneralException;

      import sailpoint.tools.xml.XMLObjectFactory;

      import sailpoint.object.Configuration;

      import sailpoint.connector.Connector;

      import sailpoint.connector.DelimitedFileConnector;

      import sailpoint.object.Schema;

      import sailpoint.object.Permission;

      import sailpoint.tools.xml.XMLObjectFactory;

      import sailpoint.tools.RFC4180LineIterator;

      import sailpoint.tools.RFC4180LineParser;

      SailPointContext ctx = SailPointFactory.getCurrentContext();

      if ( ctx == null ) {

          throw new GeneralException("Unable to get sailpoint context.");

      }     

      HashMap appConfig = application.getAttributes();

      String DEVFile = (String)appConfig.get("DEVFile");

      String PRODFile = (String)appConfig.get("PRODFile");

      String outputfile = (String)appConfig.get("outputfile");

      //Construct the BufferedInputStream object

      DataInputStream DEVbis = new DataInputStream(new FileInputStream(new File(DEVFile)));

      DataInputStream PRODbis = new DataInputStream(new FileInputStream(new File(PRODFile)));

      // Constructing stream to return to the connector

      // Deleting old File so we can get the latest and greatest

      java.io.File mergedFile = new File(outputfile);     

      mergedFile.delete();

      // Now its time to write to the merge file

      BufferedWriter outputFile = new BufferedWriter(new FileWriter(mergedFile,true));

      String record;

      //Keep reading from the file while there is any content

      while ( ( record = PRODbis.readLine() ) != null) {

          outputFile.append(record + ",Prod");

          outputFile.newLine();

      }

      record = null;

      while ( ( record = DEVbis.readLine() ) != null) {

          outputFile.append(record + ",Dev/Test");

          outputFile.newLine();

      }

      record = null;

      outputFile.flush();

      outputFile.close();

      BufferedInputStream bis = new BufferedInputStream(new FileInputStream(new File(outputfile)));

      return bis;         

    ]]>

  </Source>

</Rule>

</sailpoint>

[#2]

In the statement String DEVFile = (String)appConfig.get("DEVFile");, can I include the path of DEVFile? In other words, can I have:

String DEVFile = (String)appConfig.get("/app/sailpoint/data/DEVFile.csv")

[shyamgarikapudi]

if that is the case you can specify direct value:

String DEVFIle = "/app/sailpoint/data/DEVFile.csv";

[joseph_hobbs]

You can hard code it in the rule, but I would recommend against it.  Future updates to the file location would require that you modify the rule (i.e. code change).  Pulling it from the application object means it's more of a 'configuration' asset.  This can be especially significant when you start talking change control processes/etc...

[gopi097]

I Used Above Code for Copy data of 2 files into another File... But it Will Copied Headers data Multiple times..

[Harish043]

where i can config the file names in application configuration for make this code dynamic?

[drosenbauer]

Is there a reason you wouldn't use the SQLLoader connector type for this?