Regression testing after making significant changes like installing a patch, performing a full version upgrade, or integrating with a new set of Applications as managed systems is a key component of the release process for an IdentityIQ deployment or upgrade project. Most projects have some form of written test-cases document that enumerates each functional case that should be tested and specifies various permutations of inputs and outputs that the system should produce during testing.
The list of test cases mirrors the list of functional requirements drawn up at the beginning of the IdentityIQ delivery project during the requirements gathering phase. Often requirements have numbered line items so that they can be signed off by both client and the services organization deploying IdentityIQ. Test case documents often have English descriptions of the test cases along with flowcharts describing the sequence of test steps and the expected behavior of the software system for each step in the process. A few test cases from a regression test document look something like this outline:
- Test Suite 110 - Discretionary Access Request, Approval and Rejection
- Test Case 110.1 - Approval of requested Access
- Manager "M1" logs into IdentityIQ and requests in a shopping cart Active Directory group "file share administrators" for their employee "E1".
- Approver "A1" receives a request to approve manger "M1"'s request via email and in their IdentityIQ inbox.
- Approver "A1" approves the request in the IdentityIQ user interface.
- Approver "A2" receives a request to approve manger "M1"'s request via email and in their IdentityIQ inbox.
- Approver "A2" approves the request in the IdentityIQ user interface.
- IdentityIQ then alter's "E1"'s active directory account
- Test Case 110.2 - Rejection of Access by first Approver
- Manager "M1" logs into IdentityIQ and requests in a shopping cart Active Directory group "file share administrators" for their employee "E1".
- Approver "A1" receives a request to approve manger "M1"'s request via email and in their IdentityIQ inbox.
- Approver "A1" rejects the request in the IdentityIQ user interface.
- Requester "M1" receives notification of the rejection via email.
- Test Case 110.2 - Rejection of Access by first Approver
- Manager "M1" logs into IdentityIQ and requests in a shopping cart Active Directory group "file share administrators" for their employee "E1".
- Approver "A1" receives a request to approve manger "M1"'s request via email and in their IdentityIQ inbox.
- Approver "A1" approves the request in the IdentityIQ user interface.
- Approver "A2" receives a request to approve manger "M1"'s request via email and in their IdentityIQ inbox.
- Approver "A2" rejects the request in the IdentityIQ user interface.
- Requester "M1" receives notification of the rejection via email.
Executing these 3 test cases takes considerable time and man power on a project. The steps require having an IdentityIQ system setup in a "Clean" state for testing, and then logging in to IdentityIQ repeatedly as user "M1", "A1", and "A2" to click on objects and carry out shopping cart populations and approvals or rejections. The entire process is very mechanical and testing personnel are looking for error messages or any part of the process that would be confusing or not work for an end user. The "often repeated" and mechanical nature of this testing process drives projects to deploy some kind of Test Automation (Test automation - Wikipedia, the free encyclopedia) for their projects. The time value of making the confirmation of functionality of the IdentityIQ becomes readily apparent for large installations.
Automated Testing Software (ATS) is any kind of software that can help automate regression testing of an IdentityIQ installation. The kind of Automated Test Software commonly used with IdentityIQ are test packages that drive IdentityIQ's User Interface (UI), usually by simulating the input of a human user through a web browser. Tools that performed API level integration are also used in testing, commonly to set up initial test state or initial test case data before a UI tool is invoked.
There are several ATS tools available that can be used with IdentityIQ to successfully automate regression testing and use-case verifications for an IdentityIQ project. Some tools are commercial and some tools are freely available from open-source software. Many deployments use a combination of free and open-sourced tooling to accomplish their testing. SailPoint does not endorse or recommend any particular automated testing software application over another as different software packages have various strengths and weaknesses. All test automation packages require administration and programming efforts to configure and execute tests. When considering an ATS package be certain to budget adequately for its setup, configuration, and execution in your project timelines.
The following tools have been used in the past by SailPoint customers to automate testing on their deployments.
- Apache JMeter (Apache JMeter - Apache JMeter™) is a scriptable tool for simulating user interface interaction. It can be used to execute single sequence test cases against IdentityIQ and to simulate large amounts of load against IdentityIQ with some script programming.
- HP LoadRunner (HP LoadRunner - Wikipedia, the free encyclopedia) is a commercial performance testing and test automation tool from HP. It has been used to automate test sequences and load testing at many larger IdentityIQ installs. The main disadvantage to using LoadRunner on your project is cost. If your organization has an existing license for this tool then you should consider using it.
- Onaware IDM (Onaware IDM | Onaware Identity Management Specialists) is a SailPoint delivery partner who have developed their own IdentityIQ regression testing suite. Their package has a unique interface for specifying test cases in an english-language like grammar among many other IdentityIQ-specific features. Projects looking to bring in services expertise to help with regression testing and test automation should consider contacting Onaware for a demonstration of their test automation technology.
- Oracle Application Testing Suite (OATS) (Oracle Application Testing Suite - Wikipedia, the free encyclopedia) is a testing suite available from Oracle. Installations integrating with Oracle Applications and that have an existing license to this tool should consider this ATS tool.
- Selenium (Selenium - Web Browser Automation) is a scriptable web browser automation tool that commonly integrates with the Firefox browser. Selenium is freely available at no license cost and is relatively easy to integrate with IdentityIQ. Selenium makes it easy to write on-off test cases that execute a sequence of steps in IdentityIQ's user interface. Writing multi-threaded, parallel test cases for testing high-load scenarios can be a challenge with Selenium. Older versions of IdentityIQ (6.1 and previous) provided challenges for integrating with Selenium. SailPoint has made efforts in recent releases to make IdentityIQ "more compatible" with Selenium as a test automation tool.
