cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

How to increase the task scheduler thread pool size

How to increase the task scheduler thread pool size

IdentityIQ uses the Quartz Scheduler package to manage task scheduling. All Quartz Scheduler (JVM) threads pull task schedule objects from a shared queue for execution. At the time of this writing, IdentityIQ sets the number of scheduler-related threads to 5. The scheduler pool's thread count limits the number of concurrently running scheduled tasks. At startup, the (Quartz) scheduler starts the configured number of worker threads (filling the pool).

 

To detail the threads operation, browse to the "hidden" debug page "Threads" option. Depending upon the IIQ release, the displayed page lists the JVM threads or (later releases) gives a full stack trace:

{IIQ URL}/debug/threads.jsf

 

To change the quartz threadpool, add the following line to iiq.properties (and restart IdentityIQ):

 

scheduler.quartzProperties.org.quartz.threadPool.threadCount=10

 

As a configuration scenario, the "Sequential Task Launcher" task steps thru the task list, queueing a task schedule object at each entry and then waiting for a completion status before moving on to the next task list entry.

 

If multiple (or nested) "Sequential Task Launcher" tasks run, then the number of tasks waiting for completion could match or exceed the total number of scheduler threads. If so, then the scheduler threads deadlock, waiting forever for a task schedule completion while the task schedule object remain in the queue (awaiting a scheduler thread pickup).

Labels (2)
Comments

Is it safe to assume you can use this to decrease the amount of threads ran at one time on a tasks server?

The use case for this is to be able to debug a specific task on one task server.

Can I set it to the following and then kick off the task on that specific server to get the results i'm looking for?

scheduler.quartzProperties.org.quartz.threadPool.threadCount=1

I'm looking for a way to debug on my task and not have any other task kick off on the same server which would result in more debugging statements being logged. This causes debugging to take longer than it should to identify the root cause of issues.

log4j should be printing out a thread number with its statements. Usually I just filter the log to help with debugging. Will that work?

Did anybody ever try to set this 

scheduler.quartzProperties.org.quartz.threadPool.threadCount=10

in iiq.properties ?

I did in IIQ 7.3.3 and it does not change anything at all.

 

I'm finding that in 8.1p1 (quartz version 2.2.3), the threadCount property only takes effect when the scheduler.quartzProperties.org.quartz.jobStore.isClustered=true is also set.    Just discovered that, so I'm not sure what, if any, implications there are to that setting yet, so take this information with caution.   I've tried the same in a 7.2p5 (with quartz 2.2.2-SNAPSHOT) environment, but the setting didn't make a difference for the threadCount, so possibly a version specific change.

@mike_klug ,

Setting below property on 7.3p5 does not increase the threads.

scheduler.quartzProperties.org.quartz.threadPool.threadCount=10

Can you suggest if there is additional config required on 7.3p5.

@yatharth_singhal ,

If you are using MSSQL server make sure you uncomment the below  lines from iiq.properties:

#scheduler.quartzProperties.org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.MSSQLDelegate
#scheduler.quartzProperties.org.quartz.jobStore.selectWithLockSQL=SELECT * FROM {0}LOCKS UPDLOCK WHERE LOCK_NAME = ?

In case this does not helps, please create a support ticket with your iiq.properties, App server (vendor and version) and JDK details.

 

 

@mohamadali_shaikh ,

Thanks. The two properties were already enabled. I will create a support case.

FYI:

Increasing the thread number as advised, does not always show an effect. Even for basically identical IIQ deployments, we have seen differences in evaluation of the the "scheduler.quartzProperties.org.quartz.threadPool.threadCount" setting, obviously in dependence of the underlying OS and/or JDK.

In fact, there different places, where the threadCount parameter can come from. Candidates are:

  • iiq.properties
  • quartz.properties (if provided)
  • iiqBeans.xml (embedded in identityiq.war)

Our analysis showed, that IIQ does not prioritize those candidates properly and eventually picks the final value from the wrong source. We've informed SailPoint Support about this in August 2020 with all the details, and finally received a solution now (April 2021!). In a nutshell:

  • This is a bug, tracked as IIQCB-6842.
  • It is supposedly fixed in 8.2, 7.3p7, 8.1p3, 8.0p5
  • There is a workaround for earlier versions by tweaking iiqBeans.xml.

whats the solution to this? can someone please tell me what they did?

I've implemented this in 8.4 and it's working as designed, @patelr3 @joergd @yatharth_singhal @darylclaude_medina 

scheduler.quartzProperties.org.quartz.threadPool.threadCount=10

 

IdentityIQ Threads

Host Name    lapd000912.server.nwie.net
Current Time    Monday, April 1, 2024 at 11:01:49 PM Eastern Daylight Time

Id    Name    Priority    State
     
48    QuartzScheduler_Worker-1    5    TIMED_WAITING
     
java.base@11.0.22/java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
    
49    QuartzScheduler_Worker-2    5    TIMED_WAITING
     
java.base@11.0.22/java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
50    QuartzScheduler_Worker-3    5    TIMED_WAITING
     
java.base@11.0.22/java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
51    QuartzScheduler_Worker-4    5    TIMED_WAITING
     
java.base@11.0.22/java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
52    QuartzScheduler_Worker-5    5    TIMED_WAITING
     
java.base@11.0.22/java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
53    QuartzScheduler_Worker-6    5    TIMED_WAITING
     
java.base@11.0.22/java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
54    QuartzScheduler_Worker-7    5    TIMED_WAITING
     
java.base@11.0.22/java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
55    QuartzScheduler_Worker-8    5    TIMED_WAITING
     
java.base@11.0.22/java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
56    QuartzScheduler_Worker-9    5    TIMED_WAITING
     
java.base@11.0.22/java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
57    QuartzScheduler_Worker-10    5    TIMED_WAITING
     
java.base@11.0.22/java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)

Version history
Revision #:
2 of 2
Last update:
‎Jul 17, 2023 11:11 PM
Updated by:
 
Contributors