Openbravo Issue Tracking System - Openbravo ERP
View Issue Details
0040754Openbravo ERPY. DBSourceManagerpublic2019-05-02 16:522019-05-22 15:51
shuehner 
alostale 
normalminorhave not tried
closedfixed 
5
 
3.0PR19Q3 
shuehner
Core
No
0040754: OutOfMemory error on export.database in validation part with many modules and default heap assignment
In case of having many modules and running export.database with most/all of them in development using default heap-size export.database fails sometimes with OutOfMemory or GC Overhead limit exceeded.

Example is ci job: mod-merged-old-stack failing around once a week

Errors have been seen in 2 different location at runtime
a.) flush in export.db code after look running 'validation' (which is using DAL)
b.) Also inside loop doing the validation as seen in following example stacktrace
     [java] 66510 [main] INFO org.openbravo.ddlutils.task.ExportDatabase - Exporting model of module: Carrier Integration Base
     [java] 66533 [main] INFO org.openbravo.ddlutils.task.ExportDatabase - Database [name=PostgreSql server; 1 tables; 0 sequences; 0 views; 0 functions; 1 triggers]
     [java] 66534 [main] INFO org.openbravo.ddlutils.task.ExportDatabase - Checking translation consistency
     [java] 66932 [main] INFO org.openbravo.ddlutils.task.ExportDatabase - Translation consistency check finished succesfully in 397 ms
     [java] 66981 [main] INFO org.openbravo.ddlutils.task.ExportDatabase - Validating Module...
     [java] Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
     [java] at java.util.LinkedHashMap.newNode(LinkedHashMap.java:256)
     [java] at java.util.HashMap.putVal(HashMap.java:631)
     [java] at java.util.HashMap.put(HashMap.java:612)
     [java] at org.hibernate.engine.spi.BatchFetchQueue.addBatchLoadableCollection(BatchFetchQueue.java:246)
     [java] at org.hibernate.engine.internal.StatefulPersistenceContext.addUninitializedCollection(StatefulPersistenceContext.java:828)
     [java] at org.hibernate.type.CollectionType.getCollection(CollectionType.java:794)
     [java] at org.hibernate.type.CollectionType.resolveKey(CollectionType.java:469)
     [java] at org.hibernate.type.CollectionType.resolve(CollectionType.java:462)
     [java] at org.hibernate.engine.internal.TwoPhaseLoad.doInitializeEntity(TwoPhaseLoad.java:172)
     [java] at org.hibernate.engine.internal.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:129)
     [java] at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:1151)
     [java] at org.hibernate.loader.Loader.processResultSet(Loader.java:1010)
     [java] at org.hibernate.loader.Loader.doQuery(Loader.java:948)
     [java] at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:340)
     [java] at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:310)
     [java] at org.hibernate.loader.Loader.loadCollectionBatch(Loader.java:2444)
     [java] at org.hibernate.loader.collection.plan.LegacyBatchingCollectionInitializerBuilder$LegacyBatchingCollectionInitializer.initialize(LegacyBatchingCollectionInitializerBuilder.java:83)
     [java] at org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:706)
     [java] at org.hibernate.event.internal.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:75)
     [java] at org.hibernate.internal.SessionImpl.initializeCollection(SessionImpl.java:2286)
     [java] at org.hibernate.collection.internal.AbstractPersistentCollection$4.doWork(AbstractPersistentCollection.java:585)
     [java] at org.hibernate.collection.internal.AbstractPersistentCollection.withTemporarySessionIfNeeded(AbstractPersistentCollection.java:263)
     [java] at org.hibernate.collection.internal.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:581)
     [java] at org.hibernate.collection.internal.AbstractPersistentCollection.read(AbstractPersistentCollection.java:148)
     [java] at org.hibernate.collection.internal.PersistentBag.iterator(PersistentBag.java:303)
     [java] at org.openbravo.service.system.DatabaseValidator.matchColumns(DatabaseValidator.java:480)
     [java] at org.openbravo.service.system.DatabaseValidator.validate(DatabaseValidator.java:223)
     [java] at org.openbravo.service.system.SystemService.validateDatabase(SystemService.java:132)
     [java] at org.openbravo.ddlutils.task.ExportDatabase.validateDatabaseForModule(ExportDatabase.java:320)
     [java] at org.openbravo.ddlutils.task.ExportDatabase.execute(ExportDatabase.java:185)
     [java] at org.openbravo.ddlutils.task.ExportDatabase.main(ExportDatabase.java:77)
i.e. as many module as current mod-merged job with default heap-assignment and enough repetitions of the export.database job
- DAL usage inside export.database is only for valiation part and nothing else
- While that is planned to be removed/refactored without dal in long term needs to work now
- Validation should only ever read data and not dirty any.

So .clear() just after the validate part inside the module loop should be possible and assuming DAL session size is the memory hog here fix the problem.

No tags attached.
related to design defect 0040801 acknowledged Triage Platform Base do not use dal in module validation 
Issue History
2019-05-02 16:52shuehnerNew Issue
2019-05-02 16:52shuehnerAssigned To => platform
2019-05-02 16:52shuehnerModules => Core
2019-05-02 16:52shuehnerTriggers an Emergency Pack => No
2019-05-07 12:59alostaleAssigned Toplatform => alostale
2019-05-07 13:04alostaleRelationship addedrelated to 0040801
2019-05-07 13:05hgbotCheckin
2019-05-07 13:05hgbotNote Added: 0111543
2019-05-07 13:05hgbotStatusnew => resolved
2019-05-07 13:05hgbotResolutionopen => fixed
2019-05-07 13:05hgbotFixed in SCM revision => http://code.openbravo.com/erp/devel/pi/rev/804489012532f36ec7fb9a7c2e9b30a5d19799ec [^]
2019-05-07 13:05alostaleReview Assigned To => caristu
2019-05-07 13:06alostaleReview Assigned Tocaristu => shuehner
2019-05-21 15:17hudsonbotCheckin
2019-05-21 15:17hudsonbotNote Added: 0112016
2019-05-22 15:51shuehnerNote Added: 0112074
2019-05-22 15:51shuehnerStatusresolved => closed
2019-05-22 15:51shuehnerFixed in Version => 3.0PR19Q3

Notes
(0111543)
hgbot   
2019-05-07 13:05   
Repository: erp/devel/pi
Changeset: 804489012532f36ec7fb9a7c2e9b30a5d19799ec
Author: Asier Lostalé <asier.lostale <at> openbravo.com>
Date: Tue May 07 12:58:29 2019 +0200
URL: http://code.openbravo.com/erp/devel/pi/rev/804489012532f36ec7fb9a7c2e9b30a5d19799ec [^]

fixed issue 40754: OOM on export.database in validation part with many modules

  Validate model part of export database throws OOM error when there are many
  modules to check.

  This patch:
    - Clears DAL session after validating each module as this process never does
      any modification and there is no reutilization all loaded objects can be
      safely evited from 1st level cache.
    - Removes unused map retainig all AD tables.
    - Filters tables by datatype in query instead of in JVM.

---
M src/org/openbravo/service/system/DatabaseValidator.java
---
(0112016)
hudsonbot   
2019-05-21 15:17   
A changeset related to this issue has been promoted main and to the
Central Repository, after passing a series of tests.

Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/9b8f37d9d85e [^]
Maturity status: Test
(0112074)
shuehner   
2019-05-22 15:51   
Code-review -> fine.
- Removing unused list + moving manual if as filter in java to hql -> fine
- .getSession().clear()
  - is at very end after which no DAL objects are retained by all callers
  - checked all validation code to only read data so no .flush() is needed before clear by definition
- OOM / GC Overhead limit exceeded happening sporadicly in CI before has not been seen ever since.