0040301: Very high memory usage for running junit tests from pi in several jobs having many modules

Description

Recently several jobs in ci running the pi junit tests started failing with OOM errors.

i.e. but-qa-multiserver
but also mod-functional-basic

However no error has been seen yet in normal pi or retail.

All those junit tests run since a long time (several years) with ANT_OPTS setting memory to 1280MB (long time ago we increased that from 1024MB)

The problem can be easily reproduce manually run by all tests i.e. in latest snapshot-but-qa-multiserver

Live view with i.e. visualvm shows that maybe in middle of time taken very high GC cpu activity is seen + nearly all heap is full.

Taking heap-dump at that moment show the probably culprit clearly:
Class Name | Shallow Heap | Retained Heap | Percentage
--------------------------------------------------------------------------------------
java.lang.Thread @ 0xb0014e18 main Thread| 104 | 567,198,520 | 54.67%
--------------------------------------------------------------------------------------------------------------------------
org.jboss.arquillian.container.impl.context.DeploymentContextImpl @ 0xc6fb6ca0| 24 | 553,691,616 | 53.37%
--------------------------------------------------------------------------------------------------------------------------
java.util.concurrent.ConcurrentHashMap @ 0xc6fb6cb8| 48 | 553,691,592 | 53.37%
-----------------------------------------------------------------------------------------------

So some part using weld via arquillian takes up >50% of the available heap which seems excessive. Dominator view of retained memory points to that concurrent HashMap called 'stores'.

Unclear what exactly is contained in there.

Note: The snapshot used has a change to beans.xml weld-config to remove the exclude for org.openbravo.erpCommon.utility.* That maybe could be a trigger for higher memory use (just an idea).

That would not explain the issue in the mod-functional-basic though.

Steps To Reproduce

clone Snapshot_BUT_QA_MULTISERVER with CodeSnapshot tool
ant install.source
ant run.all.test
- notice it failing after a longer runtime

Tags

No tags attached.

Attached Files

Relationships [ Relation Graph ] [ Dependency Graph ]

depends on

defect

0034821

3.0PR19Q2

closed

jarmendariz

Very high memory usage for running junit tests from pi in several jobs having many modules

Notes
(0110135) hgbot (developer) 2019-02-28 19:14	Repository: erp/backports/3.0PR19Q1 Changeset: b1710472ecf0bdfa7fccfec92536af8847c2006e Author: Javier Armendáriz <javier.armendariz <at> openbravo.com> Date: Thu Feb 28 19:09:58 2019 +0100 URL: http://code.openbravo.com/erp/backports/3.0PR19Q1/rev/b1710472ecf0bdfa7fccfec92536af8847c2006e [^] Fixed issue 40301: High memory usage on tests using arquillian Adding arquillian suite extension to avoid creating the same deployment for each test thus reducing memory impact on tests. --- M legal/Licensing.txt M modules/org.openbravo.base.weld/src-test/org/openbravo/base/weld/test/WeldBaseTest.java M src-test/.classpath.template A legal/WTFPL_license.txt A modules/org.openbravo.base.weld/lib/test/arquillian-suite-extension-1.1.2.jar A modules/org.openbravo.base.weld/lib/test/reflections-0.9.8.jar ---

(0110144) alostale (manager) 2019-03-01 08:57	reviewed

Issue History
Date Modified	Username	Field	Change
2019-02-28 19:07	jarmendariz	New Issue
2019-02-28 19:07	jarmendariz	Assigned To	=> jarmendariz
2019-02-28 19:07	jarmendariz	Modules	=> Core
2019-02-28 19:07	jarmendariz	Triggers an Emergency Pack	=> No
2019-02-28 19:07	jarmendariz	Issue generated from	0034821
2019-02-28 19:07	jarmendariz	Relationship added	depends on 0034821
2019-02-28 19:14	hgbot	Checkin
2019-02-28 19:14	hgbot	Note Added: 0110135
2019-02-28 19:14	hgbot	Status	new => resolved
2019-02-28 19:14	hgbot	Resolution	open => fixed
2019-02-28 19:14	hgbot	Fixed in SCM revision	=> http://code.openbravo.com/erp/backports/3.0PR19Q1/rev/b1710472ecf0bdfa7fccfec92536af8847c2006e [^]
2019-02-28 19:15	jarmendariz	Review Assigned To	=> alostale
2019-03-01 08:57	alostale	Note Added: 0110144
2019-03-01 08:57	alostale	Status	resolved => closed
2019-03-01 08:57	alostale	Fixed in Version	=> 3.0PR19Q1