Openbravo Issue Tracking System - Openbravo ERP
View Issue Details
IDProjectCategoryView StatusDate SubmittedLast Update
0034821Openbravo ERPA. Platformpublic2016-12-23 17:442019-03-28 14:56
Reportershuehner 
Assigned Tojarmendariz 
PriorityhighSeverityminorReproducibilityhave not tried
StatusclosedResolutionfixed 
PlatformOS5OS Version
Product Version 
Target Version3.0PR19Q2Fixed in Version3.0PR19Q2 
Merge Request Status
Review Assigned Toalostale
OBNetwork customer
Web browser
ModulesCore
Support ticket
Regression level
Regression date
Regression introduced in release
Regression introduced by commit
Triggers an Emergency PackNo
Summary0034821: Very high memory usage for running junit tests from pi in several jobs having many modules
DescriptionRecently several jobs in ci running the pi junit tests started failing with OOM errors.

i.e. but-qa-multiserver
but also mod-functional-basic

However no error has been seen yet in normal pi or retail.

All those junit tests run since a long time (several years) with ANT_OPTS setting memory to 1280MB (long time ago we increased that from 1024MB)

The problem can be easily reproduce manually run by all tests i.e. in latest snapshot-but-qa-multiserver

Live view with i.e. visualvm shows that maybe in middle of time taken very high GC cpu activity is seen + nearly all heap is full.

Taking heap-dump at that moment show the probably culprit clearly:
Class Name | Shallow Heap | Retained Heap | Percentage
--------------------------------------------------------------------------------------
java.lang.Thread @ 0xb0014e18 main Thread| 104 | 567,198,520 | 54.67%
--------------------------------------------------------------------------------------------------------------------------
org.jboss.arquillian.container.impl.context.DeploymentContextImpl @ 0xc6fb6ca0| 24 | 553,691,616 | 53.37%
--------------------------------------------------------------------------------------------------------------------------
java.util.concurrent.ConcurrentHashMap @ 0xc6fb6cb8| 48 | 553,691,592 | 53.37%
-----------------------------------------------------------------------------------------------

So some part using weld via arquillian takes up >50% of the available heap which seems excessive. Dominator view of retained memory points to that concurrent HashMap called 'stores'.

Unclear what exactly is contained in there.

Note: The snapshot used has a change to beans.xml weld-config to remove the exclude for org.openbravo.erpCommon.utility.* That maybe could be a trigger for higher memory use (just an idea).

That would not explain the issue in the mod-functional-basic though.

Steps To Reproduceclone Snapshot_BUT_QA_MULTISERVER with CodeSnapshot tool
ant install.source
ant run.all.test
- notice it failing after a longer runtime

Proposed Solution
Additional Information
TagsNo tags attached.
Relationships
blocks backport 00403013.0PR19Q1 closed jarmendariz Very high memory usage for running junit tests from pi in several jobs having many modules 
Attached Filespng Selection_001.png (23,125) 2016-12-23 17:44
https://issues.openbravo.com/file_download.php?file_id=10218&type=bug
png

png Selection_002.png (26,936) 2016-12-23 17:44
https://issues.openbravo.com/file_download.php?file_id=10219&type=bug
png

png Selection_003.png (53,240) 2016-12-23 17:46
https://issues.openbravo.com/file_download.php?file_id=10220&type=bug
png
Issue History
Date ModifiedUsernameFieldChange
2016-12-23 17:44shuehnerNew Issue
2016-12-23 17:44shuehnerAssigned To => platform
2016-12-23 17:44shuehnerModules => Core
2016-12-23 17:44shuehnerTriggers an Emergency Pack => No
2016-12-23 17:44shuehnerFile Added: Selection_001.png
2016-12-23 17:44shuehnerFile Added: Selection_002.png
2016-12-23 17:46shuehnerFile Added: Selection_003.png
2016-12-23 17:46shuehnerNote Added: 0092984
2016-12-23 19:00shuehnerNote Added: 0092986
2017-01-09 10:13alostalePrioritynormal => high
2017-01-09 10:13alostaleStatusnew => acknowledged
2019-01-29 13:58alostaleAssigned Toplatform => jarmendariz
2019-01-29 13:58alostaleStatusacknowledged => scheduled
2019-01-29 13:58alostaleTarget Version => 3.0PR19Q2
2019-02-11 17:15hgbotCheckin
2019-02-11 17:15hgbotNote Added: 0109678
2019-02-11 17:15hgbotStatusscheduled => resolved
2019-02-11 17:15hgbotResolutionopen => fixed
2019-02-11 17:15hgbotFixed in SCM revision => http://code.openbravo.com/erp/devel/pi/rev/9275bf6ed1ff3b58e8af24a0e8a2243fc1e75bb0 [^]
2019-02-12 12:45hgbotCheckin
2019-02-12 12:45hgbotNote Added: 0109707
2019-02-15 09:54alostaleReview Assigned To => alostale
2019-02-15 09:54alostaleNote Added: 0109852
2019-02-15 09:54alostaleStatusresolved => closed
2019-02-15 09:54alostaleFixed in Version => 3.0PR19Q2
2019-02-28 19:07jarmendarizIssue cloned0040301
2019-02-28 19:07jarmendarizRelationship addedblocks 0040301
2019-03-28 14:56hudsonbotCheckin
2019-03-28 14:56hudsonbotNote Added: 0110806
2019-03-28 14:56hudsonbotCheckin
2019-03-28 14:56hudsonbotNote Added: 0110807

Notes
(0092984)
shuehner   
2016-12-23 17:46   
Selection_003 is screenshot of visualvm view of gc cpu usage % and memory usage which clearly shows when the problem triggers.
(0092986)
shuehner   
2016-12-23 19:00   
Local testing (openjdk-7 64bit):
1280MB failing
1536MB working
1792MB working
2048MB working
(0109678)
hgbot   
2019-02-11 17:15   
Repository: erp/devel/pi
Changeset: 9275bf6ed1ff3b58e8af24a0e8a2243fc1e75bb0
Author: Javier Armendáriz <javier.armendariz <at> openbravo.com>
Date: Mon Feb 11 11:01:09 2019 +0100
URL: http://code.openbravo.com/erp/devel/pi/rev/9275bf6ed1ff3b58e8af24a0e8a2243fc1e75bb0 [^]

Fixed issue 34821: High memory usage on tests using arquillian.

---
M modules/org.openbravo.base.weld/src-test/org/openbravo/base/weld/test/WeldBaseTest.java
M src-test/.classpath.template
A modules/org.openbravo.base.weld/lib/test/arquillian-suite-extension-1.1.2.jar
A modules/org.openbravo.base.weld/lib/test/reflections-0.9.8.jar
---
(0109707)
hgbot   
2019-02-12 12:45   
Repository: erp/devel/pi
Changeset: 4d0df2352c164918d42fb91ab4b399097c6538cf
Author: Javier Armendáriz <javier.armendariz <at> openbravo.com>
Date: Tue Feb 12 12:42:45 2019 +0100
URL: http://code.openbravo.com/erp/devel/pi/rev/4d0df2352c164918d42fb91ab4b399097c6538cf [^]

Related to issue 34821: Adding license info for new libraries.

---
M legal/Licensing.txt
A legal/WTFPL_license.txt
---
(0109852)
alostale   
2019-02-15 09:54   
reviewed + tested
(0110806)
hudsonbot   
2019-03-28 14:56   
A changeset related to this issue has been promoted main and to the
Central Repository, after passing a series of tests.

Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/b2fbc1588df4 [^]
Maturity status: Test
(0110807)
hudsonbot   
2019-03-28 14:56   
A changeset related to this issue has been promoted main and to the
Central Repository, after passing a series of tests.

Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/b2fbc1588df4 [^]
Maturity status: Test