0040301: Very high memory usage for running junit tests from pi in several jobs having many modules

Openbravo Issue Tracking System - Openbravo ERP

View Issue Details

Project

Category

View Status

Date Submitted

Last Update

0040301

Openbravo ERP

A. Platform

public

2019-02-28 19:07

2019-03-01 08:57

Reporter

jarmendariz

Assigned To

jarmendariz

Priority

high

Severity

minor

Reproducibility

have not tried

Status

closed

Resolution

fixed

Platform

OS Version

Product Version

Target Version

3.0PR19Q1

Fixed in Version

3.0PR19Q1

Merge Request Status

Review Assigned To

alostale

OBNetwork customer

Web browser

Modules

Core

Support ticket

Regression level

Regression date

Regression introduced in release

Regression introduced by commit

Triggers an Emergency Pack

Summary

0040301: Very high memory usage for running junit tests from pi in several jobs having many modules

Description

Recently several jobs in ci running the pi junit tests started failing with OOM errors.

i.e. but-qa-multiserver
but also mod-functional-basic

However no error has been seen yet in normal pi or retail.

All those junit tests run since a long time (several years) with ANT_OPTS setting memory to 1280MB (long time ago we increased that from 1024MB)

The problem can be easily reproduce manually run by all tests i.e. in latest snapshot-but-qa-multiserver

Live view with i.e. visualvm shows that maybe in middle of time taken very high GC cpu activity is seen + nearly all heap is full.

Taking heap-dump at that moment show the probably culprit clearly:
Class Name | Shallow Heap | Retained Heap | Percentage
--------------------------------------------------------------------------------------
java.lang.Thread @ 0xb0014e18 main Thread| 104 | 567,198,520 | 54.67%
--------------------------------------------------------------------------------------------------------------------------
org.jboss.arquillian.container.impl.context.DeploymentContextImpl @ 0xc6fb6ca0| 24 | 553,691,616 | 53.37%
--------------------------------------------------------------------------------------------------------------------------
java.util.concurrent.ConcurrentHashMap @ 0xc6fb6cb8| 48 | 553,691,592 | 53.37%
-----------------------------------------------------------------------------------------------

So some part using weld via arquillian takes up >50% of the available heap which seems excessive. Dominator view of retained memory points to that concurrent HashMap called 'stores'.

Unclear what exactly is contained in there.

Note: The snapshot used has a change to beans.xml weld-config to remove the exclude for org.openbravo.erpCommon.utility.* That maybe could be a trigger for higher memory use (just an idea).

That would not explain the issue in the mod-functional-basic though.

Steps To Reproduce

clone Snapshot_BUT_QA_MULTISERVER with CodeSnapshot tool
ant install.source
ant run.all.test
- notice it failing after a longer runtime

Proposed Solution

Additional Information