0040754: OutOfMemory error on export.database in validation part with many modules and default heap assignment

Description

In case of having many modules and running export.database with most/all of them in development using default heap-size export.database fails sometimes with OutOfMemory or GC Overhead limit exceeded.

Example is ci job: mod-merged-old-stack failing around once a week

Errors have been seen in 2 different location at runtime
a.) flush in export.db code after look running 'validation' (which is using DAL)
b.) Also inside loop doing the validation as seen in following example stacktrace
     [java] 66510 [main] INFO org.openbravo.ddlutils.task.ExportDatabase - Exporting model of module: Carrier Integration Base
     [java] 66533 [main] INFO org.openbravo.ddlutils.task.ExportDatabase - Database [name=PostgreSql server; 1 tables; 0 sequences; 0 views; 0 functions; 1 triggers]
     [java] 66534 [main] INFO org.openbravo.ddlutils.task.ExportDatabase - Checking translation consistency
     [java] 66932 [main] INFO org.openbravo.ddlutils.task.ExportDatabase - Translation consistency check finished succesfully in 397 ms
     [java] 66981 [main] INFO org.openbravo.ddlutils.task.ExportDatabase - Validating Module...
     [java] Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
     [java] at java.util.LinkedHashMap.newNode(LinkedHashMap.java:256)
     [java] at java.util.HashMap.putVal(HashMap.java:631)
     [java] at java.util.HashMap.put(HashMap.java:612)
     [java] at org.hibernate.engine.spi.BatchFetchQueue.addBatchLoadableCollection(BatchFetchQueue.java:246)
     [java] at org.hibernate.engine.internal.StatefulPersistenceContext.addUninitializedCollection(StatefulPersistenceContext.java:828)
     [java] at org.hibernate.type.CollectionType.getCollection(CollectionType.java:794)
     [java] at org.hibernate.type.CollectionType.resolveKey(CollectionType.java:469)
     [java] at org.hibernate.type.CollectionType.resolve(CollectionType.java:462)
     [java] at org.hibernate.engine.internal.TwoPhaseLoad.doInitializeEntity(TwoPhaseLoad.java:172)
     [java] at org.hibernate.engine.internal.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:129)
     [java] at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:1151)
     [java] at org.hibernate.loader.Loader.processResultSet(Loader.java:1010)
     [java] at org.hibernate.loader.Loader.doQuery(Loader.java:948)
     [java] at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:340)
     [java] at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:310)
     [java] at org.hibernate.loader.Loader.loadCollectionBatch(Loader.java:2444)
     [java] at org.hibernate.loader.collection.plan.LegacyBatchingCollectionInitializerBuilder$LegacyBatchingCollectionInitializer.initialize(LegacyBatchingCollectionInitializerBuilder.java:83)
     [java] at org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:706)
     [java] at org.hibernate.event.internal.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:75)
     [java] at org.hibernate.internal.SessionImpl.initializeCollection(SessionImpl.java:2286)
     [java] at org.hibernate.collection.internal.AbstractPersistentCollection$4.doWork(AbstractPersistentCollection.java:585)
     [java] at org.hibernate.collection.internal.AbstractPersistentCollection.withTemporarySessionIfNeeded(AbstractPersistentCollection.java:263)
     [java] at org.hibernate.collection.internal.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:581)
     [java] at org.hibernate.collection.internal.AbstractPersistentCollection.read(AbstractPersistentCollection.java:148)
     [java] at org.hibernate.collection.internal.PersistentBag.iterator(PersistentBag.java:303)
     [java] at org.openbravo.service.system.DatabaseValidator.matchColumns(DatabaseValidator.java:480)
     [java] at org.openbravo.service.system.DatabaseValidator.validate(DatabaseValidator.java:223)
     [java] at org.openbravo.service.system.SystemService.validateDatabase(SystemService.java:132)
     [java] at org.openbravo.ddlutils.task.ExportDatabase.validateDatabaseForModule(ExportDatabase.java:320)
     [java] at org.openbravo.ddlutils.task.ExportDatabase.execute(ExportDatabase.java:185)
     [java] at org.openbravo.ddlutils.task.ExportDatabase.main(ExportDatabase.java:77)

Steps To Reproduce

i.e. as many module as current mod-merged job with default heap-assignment and enough repetitions of the export.database job

Proposed Solution

- DAL usage inside export.database is only for valiation part and nothing else
- While that is planned to be removed/refactored without dal in long term needs to work now
- Validation should only ever read data and not dirty any.

So .clear() just after the validate part inside the module loop should be possible and assuming DAL session size is the memory hog here fix the problem.

Tags

No tags attached.

Attached Files

Relationships [ Relation Graph ] [ Dependency Graph ]

related to

design defect

0040801

acknowledged

Triage Platform Base

do not use dal in module validation

Notes
(0111543) hgbot (developer) 2019-05-07 13:05	Repository: erp/devel/pi Changeset: 804489012532f36ec7fb9a7c2e9b30a5d19799ec Author: Asier Lostalé <asier.lostale <at> openbravo.com> Date: Tue May 07 12:58:29 2019 +0200 URL: http://code.openbravo.com/erp/devel/pi/rev/804489012532f36ec7fb9a7c2e9b30a5d19799ec [^] fixed issue 40754: OOM on export.database in validation part with many modules Validate model part of export database throws OOM error when there are many modules to check. This patch: - Clears DAL session after validating each module as this process never does any modification and there is no reutilization all loaded objects can be safely evited from 1st level cache. - Removes unused map retainig all AD tables. - Filters tables by datatype in query instead of in JVM. --- M src/org/openbravo/service/system/DatabaseValidator.java ---

(0112016) hudsonbot (developer) 2019-05-21 15:17	A changeset related to this issue has been promoted main and to the Central Repository, after passing a series of tests. Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/9b8f37d9d85e [^] Maturity status: Test

(0112074) shuehner (administrator) 2019-05-22 15:51	Code-review -> fine. - Removing unused list + moving manual if as filter in java to hql -> fine - .getSession().clear() - is at very end after which no DAL objects are retained by all callers - checked all validation code to only read data so no .flush() is needed before clear by definition - OOM / GC Overhead limit exceeded happening sporadicly in CI before has not been seen ever since.

Issue History
Date Modified	Username	Field	Change
2019-05-02 16:52	shuehner	New Issue
2019-05-02 16:52	shuehner	Assigned To	=> platform
2019-05-02 16:52	shuehner	Modules	=> Core
2019-05-02 16:52	shuehner	Triggers an Emergency Pack	=> No
2019-05-07 12:59	alostale	Assigned To	platform => alostale
2019-05-07 13:04	alostale	Relationship added	related to 0040801
2019-05-07 13:05	hgbot	Checkin
2019-05-07 13:05	hgbot	Note Added: 0111543
2019-05-07 13:05	hgbot	Status	new => resolved
2019-05-07 13:05	hgbot	Resolution	open => fixed
2019-05-07 13:05	hgbot	Fixed in SCM revision	=> http://code.openbravo.com/erp/devel/pi/rev/804489012532f36ec7fb9a7c2e9b30a5d19799ec [^]
2019-05-07 13:05	alostale	Review Assigned To	=> caristu
2019-05-07 13:06	alostale	Review Assigned To	caristu => shuehner
2019-05-21 15:17	hudsonbot	Checkin
2019-05-21 15:17	hudsonbot	Note Added: 0112016
2019-05-22 15:51	shuehner	Note Added: 0112074
2019-05-22 15:51	shuehner	Status	resolved => closed
2019-05-22 15:51	shuehner	Fixed in Version	=> 3.0PR19Q3