0035329: scheduled processes are not executed in some cases (ie. after some OOM)

Description

If QuartzSchedulerThread throws an unhandled exception (it handles RuntimeException [1]), this thread gets terminated not being possible to schedule any new process till Tomcat is restarted.

This exceptional situation is difficult to reproduce but it has been observed in some real customers in OOM.

Exception in thread "DefaultQuartzScheduler_QuartzSchedulerThread" java.lang.OutOfMemoryError: Java heap space
        at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:62)
        at java.lang.StringBuffer.<init>(StringBuffer.java:96)
        at org.quartz.Trigger.getFullName(Trigger.java:454)
        at org.quartz.simpl.TriggerComparator.compare(RAMJobStore.java:1504)
        at java.util.TreeMap.getEntryUsingComparator(TreeMap.java:368)
        at java.util.TreeMap.getEntry(TreeMap.java:339)
        at java.util.TreeMap.remove(TreeMap.java:597)
        at java.util.TreeSet.remove(TreeSet.java:276)
        at org.quartz.simpl.RAMJobStore.acquireNextTrigger(RAMJobStore.java:1224)
        at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:266)

Note if the exception is thrown by a scheduled process, it is properly handled without further consequences.
---
[1] https://github.com/alostale/quartz/blob/quartz-1.6.2/src/java/org/quartz/core/QuartzSchedulerThread.java#L438 [^]

Steps To Reproduce

It's hard to reproduce because it is necessary to cause an unhandled exception in this concrete piece of code.

1. Replace quartz-1.6.2.jar with attached jar. This jar throws an OutOfMemoryError whenever Alert Process is executed.
2. Start tomcat
   -> OK: check with jstack there is a running thread named "DefaultQuartzSecheduler_QuartzSchedulerThread"
3. Schedule Alert Process
4. After it tries to run:
   -> FAIL: check with jstack there IS NOT a running thread named "DefaultQuartzSecheduler_QuartzSchedulerThread"
5. Schedule any other process
   -> FAIL: it does not get executed

Tags

No tags attached.

Attached Files

quartz-1.6.2.jar [^] (447,127 bytes) 2017-02-21 16:30

Relationships [ Relation Graph ] [ Dependency Graph ]

related to

defect

0029490

closed

egoitz

The quartz scheduler is interrupted and the background process stop working

Notes
(0094605) alostale (manager) 2017-02-24 10:38	Moving to design defect. It's not possible to safely recover from this situation as the overall system status is unknown, it would in any case require of restarting server. Possibly, it should be necessary to monitor and alert when this thread is not in execution.

Issue History
Date Modified	Username	Field	Change
2017-02-21 16:30	alostale	New Issue
2017-02-21 16:30	alostale	Assigned To	=> platform
2017-02-21 16:30	alostale	Modules	=> Core
2017-02-21 16:30	alostale	Triggers an Emergency Pack	=> No
2017-02-21 16:30	alostale	File Added: quartz-1.6.2.jar
2017-02-21 16:46	alostale	Summary	processes cannot be scheduled in some cases (ie. after some OOM) => scheduled processes are not executed in some cases (ie. after some OOM)
2017-02-21 17:01	alostale	Relationship added	related to 0029490
2017-02-22 09:24	egoitz	Issue Monitored: egoitz
2017-02-24 10:38	alostale	Note Added: 0094605
2017-02-24 10:38	alostale	Status	new => acknowledged
2017-02-24 10:38	alostale	Type	defect => design defect
2022-02-01 08:09	alostale	Assigned To	platform => Triage Platform Base