Anonymous | Login
Project:
RSS
  
News | My View | View Issues | Roadmap | Summary

View Issue Details[ Jump to Notes ] [ Issue History ] [ Print ]
ID
0034821
TypeCategorySeverityReproducibilityDate SubmittedLast Update
defect[Openbravo ERP] A. Platformminorhave not tried2016-12-23 17:442019-03-28 14:56
ReportershuehnerView Statuspublic 
Assigned Tojarmendariz 
PriorityhighResolutionfixedFixed in Version3.0PR19Q2
StatusclosedFix in branchFixed in SCM revision9275bf6ed1ff
ProjectionnoneETAnoneTarget Version3.0PR19Q2
OSAnyDatabaseAnyJava version
OS VersionDatabase versionAnt version
Product VersionSCM revision 
Merge Request Status
Review Assigned Toalostale
OBNetwork customerNo
Web browser
ModulesCore
Support ticket
Regression level
Regression date
Regression introduced in release
Regression introduced by commit
Triggers an Emergency PackNo
Summary

0034821: Very high memory usage for running junit tests from pi in several jobs having many modules

DescriptionRecently several jobs in ci running the pi junit tests started failing with OOM errors.

i.e. but-qa-multiserver
but also mod-functional-basic

However no error has been seen yet in normal pi or retail.

All those junit tests run since a long time (several years) with ANT_OPTS setting memory to 1280MB (long time ago we increased that from 1024MB)

The problem can be easily reproduce manually run by all tests i.e. in latest snapshot-but-qa-multiserver

Live view with i.e. visualvm shows that maybe in middle of time taken very high GC cpu activity is seen + nearly all heap is full.

Taking heap-dump at that moment show the probably culprit clearly:
Class Name | Shallow Heap | Retained Heap | Percentage
--------------------------------------------------------------------------------------
java.lang.Thread @ 0xb0014e18 main Thread| 104 | 567,198,520 | 54.67%
--------------------------------------------------------------------------------------------------------------------------
org.jboss.arquillian.container.impl.context.DeploymentContextImpl @ 0xc6fb6ca0| 24 | 553,691,616 | 53.37%
--------------------------------------------------------------------------------------------------------------------------
java.util.concurrent.ConcurrentHashMap @ 0xc6fb6cb8| 48 | 553,691,592 | 53.37%
-----------------------------------------------------------------------------------------------

So some part using weld via arquillian takes up >50% of the available heap which seems excessive. Dominator view of retained memory points to that concurrent HashMap called 'stores'.

Unclear what exactly is contained in there.

Note: The snapshot used has a change to beans.xml weld-config to remove the exclude for org.openbravo.erpCommon.utility.* That maybe could be a trigger for higher memory use (just an idea).

That would not explain the issue in the mod-functional-basic though.

Steps To Reproduceclone Snapshot_BUT_QA_MULTISERVER with CodeSnapshot tool
ant install.source
ant run.all.test
- notice it failing after a longer runtime

TagsNo tags attached.
Attached Filespng file icon Selection_001.png [^] (23,125 bytes) 2016-12-23 17:44


png file icon Selection_002.png [^] (26,936 bytes) 2016-12-23 17:44


png file icon Selection_003.png [^] (53,240 bytes) 2016-12-23 17:46

- Relationships [ Relation Graph ] [ Dependency Graph ]
blocks backport 00403013.0PR19Q1 closedjarmendariz Very high memory usage for running junit tests from pi in several jobs having many modules 

-  Notes
(0092984)
shuehner (administrator)
2016-12-23 17:46

 Selection_003 is screenshot of visualvm view of gc cpu usage % and memory usage which clearly shows when the problem triggers.
(0092986)
shuehner (administrator)
2016-12-23 19:00

 Local testing (openjdk-7 64bit):
1280MB failing
1536MB working
1792MB working
2048MB working
(0109678)
hgbot (developer)
2019-02-11 17:15

 Repository: erp/devel/pi
Changeset: 9275bf6ed1ff3b58e8af24a0e8a2243fc1e75bb0
Author: Javier Armendáriz <javier.armendariz <at> openbravo.com>
Date: Mon Feb 11 11:01:09 2019 +0100
URL: http://code.openbravo.com/erp/devel/pi/rev/9275bf6ed1ff3b58e8af24a0e8a2243fc1e75bb0 [^]

Fixed issue 34821: High memory usage on tests using arquillian.

---
M modules/org.openbravo.base.weld/src-test/org/openbravo/base/weld/test/WeldBaseTest.java
M src-test/.classpath.template
A modules/org.openbravo.base.weld/lib/test/arquillian-suite-extension-1.1.2.jar
A modules/org.openbravo.base.weld/lib/test/reflections-0.9.8.jar
---
(0109707)
hgbot (developer)
2019-02-12 12:45

 Repository: erp/devel/pi
Changeset: 4d0df2352c164918d42fb91ab4b399097c6538cf
Author: Javier Armendáriz <javier.armendariz <at> openbravo.com>
Date: Tue Feb 12 12:42:45 2019 +0100
URL: http://code.openbravo.com/erp/devel/pi/rev/4d0df2352c164918d42fb91ab4b399097c6538cf [^]

Related to issue 34821: Adding license info for new libraries.

---
M legal/Licensing.txt
A legal/WTFPL_license.txt
---
(0109852)
alostale (viewer)
2019-02-15 09:54

 reviewed + tested
(0110806)
hudsonbot (viewer)
2019-03-28 14:56

 A changeset related to this issue has been promoted main and to the
Central Repository, after passing a series of tests.

Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/b2fbc1588df4 [^]
Maturity status: Test
(0110807)
hudsonbot (viewer)
2019-03-28 14:56

 A changeset related to this issue has been promoted main and to the
Central Repository, after passing a series of tests.

Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/b2fbc1588df4 [^]
Maturity status: Test

- Issue History
Date Modified Username Field Change
2016-12-23 17:44 shuehner New Issue
2016-12-23 17:44 shuehner Assigned To => platform
2016-12-23 17:44 shuehner OBNetwork customer => No
2016-12-23 17:44 shuehner Modules => Core
2016-12-23 17:44 shuehner Triggers an Emergency Pack => No
2016-12-23 17:44 shuehner File Added: Selection_001.png
2016-12-23 17:44 shuehner File Added: Selection_002.png
2016-12-23 17:46 shuehner File Added: Selection_003.png
2016-12-23 17:46 shuehner Note Added: 0092984
2016-12-23 19:00 shuehner Note Added: 0092986
2017-01-09 10:13 alostale Priority normal => high
2017-01-09 10:13 alostale Status new => acknowledged
2019-01-29 13:58 alostale Assigned To platform => jarmendariz
2019-01-29 13:58 alostale Status acknowledged => scheduled
2019-01-29 13:58 alostale Target Version => 3.0PR19Q2
2019-02-11 17:15 hgbot Checkin
2019-02-11 17:15 hgbot Note Added: 0109678
2019-02-11 17:15 hgbot Status scheduled => resolved
2019-02-11 17:15 hgbot Resolution open => fixed
2019-02-11 17:15 hgbot Fixed in SCM revision => http://code.openbravo.com/erp/devel/pi/rev/9275bf6ed1ff3b58e8af24a0e8a2243fc1e75bb0 [^]
2019-02-12 12:45 hgbot Checkin
2019-02-12 12:45 hgbot Note Added: 0109707
2019-02-15 09:54 alostale Review Assigned To => alostale
2019-02-15 09:54 alostale Note Added: 0109852
2019-02-15 09:54 alostale Status resolved => closed
2019-02-15 09:54 alostale Fixed in Version => 3.0PR19Q2
2019-02-28 19:07 jarmendariz Issue cloned 0040301
2019-02-28 19:07 jarmendariz Relationship added blocks 0040301
2019-03-28 14:56 hudsonbot Checkin
2019-03-28 14:56 hudsonbot Note Added: 0110806
2019-03-28 14:56 hudsonbot Checkin
2019-03-28 14:56 hudsonbot Note Added: 0110807

Copyright © 2000 - 2009 MantisBT Group
Powered by Mantis Bugtracker