How to increase the task scheduler thread pool size
How to increase the task scheduler thread pool size
IdentityIQ uses the Quartz Scheduler package to manage task scheduling. All Quartz Scheduler (JVM) threads pull task schedule objects from a shared queue for execution. At the time of this writing, IdentityIQ sets the number of scheduler-related threads to 5. The scheduler pool's thread count limits the number of concurrently running scheduled tasks. At startup, the (Quartz) scheduler starts the configured number of worker threads (filling the pool).
To detail the threads operation, browse to the "hidden" debug page "Threads" option. Depending upon the IIQ release, the displayed page lists the JVM threads or (later releases) gives a full stack trace:
As a configuration scenario, the "Sequential Task Launcher" task steps thru the task list, queueing a task schedule object at each entry and then waiting for a completion status before moving on to the next task list entry.
If multiple (or nested) "Sequential Task Launcher" tasks run, then the number of tasks waiting for completion could match or exceed the total number of scheduler threads. If so, then the scheduler threads deadlock, waiting forever for a task schedule completion while the task schedule object remain in the queue (awaiting a scheduler thread pickup).
I'm looking for a way to debug on my task and not have any other task kick off on the same server which would result in more debugging statements being logged. This causes debugging to take longer than it should to identify the root cause of issues.
I'm finding that in 8.1p1 (quartz version 2.2.3), the threadCount property only takes effect when the scheduler.quartzProperties.org.quartz.jobStore.isClustered=true is also set. Just discovered that, so I'm not sure what, if any, implications there are to that setting yet, so take this information with caution. I've tried the same in a 7.2p5 (with quartz 2.2.2-SNAPSHOT) environment, but the setting didn't make a difference for the threadCount, so possibly a version specific change.
If you are using MSSQL server make sure you uncomment the below lines from iiq.properties:
#scheduler.quartzProperties.org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.MSSQLDelegate
#scheduler.quartzProperties.org.quartz.jobStore.selectWithLockSQL=SELECT * FROM {0}LOCKS UPDLOCK WHERE LOCK_NAME = ?
In case this does not helps, please create a support ticket with your iiq.properties, App server (vendor and version) and JDK details.
Increasing the thread number as advised, does not always show an effect. Even for basically identical IIQ deployments, we have seen differences in evaluation of the the "scheduler.quartzProperties.org.quartz.threadPool.threadCount" setting, obviously in dependence of the underlying OS and/or JDK.
In fact, there different places, where the threadCount parameter can come from. Candidates are:
iiq.properties
quartz.properties (if provided)
iiqBeans.xml (embedded in identityiq.war)
Our analysis showed, that IIQ does not prioritize those candidates properly and eventually picks the final value from the wrong source. We've informed SailPoint Support about this in August 2020 with all the details, and finally received a solution now (April 2021!). In a nutshell:
This is a bug, tracked as IIQCB-6842.
It is supposedly fixed in 8.2, 7.3p7, 8.1p3, 8.0p5
There is a workaround for earlier versions by tweaking iiqBeans.xml.
.png "yatharth_singhal"){kind=avatar}
