Anonymous | Login
Project:
RSS
  
News | My View | View Issues | Roadmap | Summary

View Issue DetailsJump to Notes ] Issue History ] Print ]
ID
0040754
TypeCategorySeverityReproducibilityDate SubmittedLast Update
defect[Openbravo ERP] Y. DBSourceManagerminorhave not tried2019-05-02 16:522019-05-22 15:51
ReportershuehnerView Statuspublic 
Assigned Toalostale 
PrioritynormalResolutionfixedFixed in Version3.0PR19Q3
StatusclosedFix in branchFixed in SCM revision804489012532
ProjectionnoneETAnoneTarget Version
OSAnyDatabaseAnyJava version
OS VersionDatabase versionAnt version
Product VersionSCM revision 
Review Assigned Toshuehner
Web browser
ModulesCore
Regression level
Regression date
Regression introduced in release
Regression introduced by commit
Triggers an Emergency PackNo
Summary

0040754: OutOfMemory error on export.database in validation part with many modules and default heap assignment

DescriptionIn case of having many modules and running export.database with most/all of them in development using default heap-size export.database fails sometimes with OutOfMemory or GC Overhead limit exceeded.

Example is ci job: mod-merged-old-stack failing around once a week

Errors have been seen in 2 different location at runtime
a.) flush in export.db code after look running 'validation' (which is using DAL)
b.) Also inside loop doing the validation as seen in following example stacktrace
     [java] 66510 [main] INFO org.openbravo.ddlutils.task.ExportDatabase - Exporting model of module: Carrier Integration Base
     [java] 66533 [main] INFO org.openbravo.ddlutils.task.ExportDatabase - Database [name=PostgreSql server; 1 tables; 0 sequences; 0 views; 0 functions; 1 triggers]
     [java] 66534 [main] INFO org.openbravo.ddlutils.task.ExportDatabase - Checking translation consistency
     [java] 66932 [main] INFO org.openbravo.ddlutils.task.ExportDatabase - Translation consistency check finished succesfully in 397 ms
     [java] 66981 [main] INFO org.openbravo.ddlutils.task.ExportDatabase - Validating Module...
     [java] Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
     [java] at java.util.LinkedHashMap.newNode(LinkedHashMap.java:256)
     [java] at java.util.HashMap.putVal(HashMap.java:631)
     [java] at java.util.HashMap.put(HashMap.java:612)
     [java] at org.hibernate.engine.spi.BatchFetchQueue.addBatchLoadableCollection(BatchFetchQueue.java:246)
     [java] at org.hibernate.engine.internal.StatefulPersistenceContext.addUninitializedCollection(StatefulPersistenceContext.java:828)
     [java] at org.hibernate.type.CollectionType.getCollection(CollectionType.java:794)
     [java] at org.hibernate.type.CollectionType.resolveKey(CollectionType.java:469)
     [java] at org.hibernate.type.CollectionType.resolve(CollectionType.java:462)
     [java] at org.hibernate.engine.internal.TwoPhaseLoad.doInitializeEntity(TwoPhaseLoad.java:172)
     [java] at org.hibernate.engine.internal.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:129)
     [java] at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:1151)
     [java] at org.hibernate.loader.Loader.processResultSet(Loader.java:1010)
     [java] at org.hibernate.loader.Loader.doQuery(Loader.java:948)
     [java] at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:340)
     [java] at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:310)
     [java] at org.hibernate.loader.Loader.loadCollectionBatch(Loader.java:2444)
     [java] at org.hibernate.loader.collection.plan.LegacyBatchingCollectionInitializerBuilder$LegacyBatchingCollectionInitializer.initialize(LegacyBatchingCollectionInitializerBuilder.java:83)
     [java] at org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:706)
     [java] at org.hibernate.event.internal.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:75)
     [java] at org.hibernate.internal.SessionImpl.initializeCollection(SessionImpl.java:2286)
     [java] at org.hibernate.collection.internal.AbstractPersistentCollection$4.doWork(AbstractPersistentCollection.java:585)
     [java] at org.hibernate.collection.internal.AbstractPersistentCollection.withTemporarySessionIfNeeded(AbstractPersistentCollection.java:263)
     [java] at org.hibernate.collection.internal.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:581)
     [java] at org.hibernate.collection.internal.AbstractPersistentCollection.read(AbstractPersistentCollection.java:148)
     [java] at org.hibernate.collection.internal.PersistentBag.iterator(PersistentBag.java:303)
     [java] at org.openbravo.service.system.DatabaseValidator.matchColumns(DatabaseValidator.java:480)
     [java] at org.openbravo.service.system.DatabaseValidator.validate(DatabaseValidator.java:223)
     [java] at org.openbravo.service.system.SystemService.validateDatabase(SystemService.java:132)
     [java] at org.openbravo.ddlutils.task.ExportDatabase.validateDatabaseForModule(ExportDatabase.java:320)
     [java] at org.openbravo.ddlutils.task.ExportDatabase.execute(ExportDatabase.java:185)
     [java] at org.openbravo.ddlutils.task.ExportDatabase.main(ExportDatabase.java:77)
Steps To Reproducei.e. as many module as current mod-merged job with default heap-assignment and enough repetitions of the export.database job
Proposed Solution- DAL usage inside export.database is only for valiation part and nothing else
- While that is planned to be removed/refactored without dal in long term needs to work now
- Validation should only ever read data and not dirty any.

So .clear() just after the validate part inside the module loop should be possible and assuming DAL session size is the memory hog here fix the problem.

TagsNo tags attached.
Attached Files

- Relationships Relation Graph ] Dependency Graph ]
related to design defect 0040801 acknowledgedTriage Platform Base do not use dal in module validation 

-  Notes
(0111543)
hgbot (developer)
2019-05-07 13:05

Repository: erp/devel/pi
Changeset: 804489012532f36ec7fb9a7c2e9b30a5d19799ec
Author: Asier Lostalé <asier.lostale <at> openbravo.com>
Date: Tue May 07 12:58:29 2019 +0200
URL: http://code.openbravo.com/erp/devel/pi/rev/804489012532f36ec7fb9a7c2e9b30a5d19799ec [^]

fixed issue 40754: OOM on export.database in validation part with many modules

  Validate model part of export database throws OOM error when there are many
  modules to check.

  This patch:
    - Clears DAL session after validating each module as this process never does
      any modification and there is no reutilization all loaded objects can be
      safely evited from 1st level cache.
    - Removes unused map retainig all AD tables.
    - Filters tables by datatype in query instead of in JVM.

---
M src/org/openbravo/service/system/DatabaseValidator.java
---
(0112016)
hudsonbot (developer)
2019-05-21 15:17

A changeset related to this issue has been promoted main and to the
Central Repository, after passing a series of tests.

Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/9b8f37d9d85e [^]
Maturity status: Test
(0112074)
shuehner (administrator)
2019-05-22 15:51

Code-review -> fine.
- Removing unused list + moving manual if as filter in java to hql -> fine
- .getSession().clear()
  - is at very end after which no DAL objects are retained by all callers
  - checked all validation code to only read data so no .flush() is needed before clear by definition
- OOM / GC Overhead limit exceeded happening sporadicly in CI before has not been seen ever since.

- Issue History
Date Modified Username Field Change
2019-05-02 16:52 shuehner New Issue
2019-05-02 16:52 shuehner Assigned To => platform
2019-05-02 16:52 shuehner Modules => Core
2019-05-02 16:52 shuehner Triggers an Emergency Pack => No
2019-05-07 12:59 alostale Assigned To platform => alostale
2019-05-07 13:04 alostale Relationship added related to 0040801
2019-05-07 13:05 hgbot Checkin
2019-05-07 13:05 hgbot Note Added: 0111543
2019-05-07 13:05 hgbot Status new => resolved
2019-05-07 13:05 hgbot Resolution open => fixed
2019-05-07 13:05 hgbot Fixed in SCM revision => http://code.openbravo.com/erp/devel/pi/rev/804489012532f36ec7fb9a7c2e9b30a5d19799ec [^]
2019-05-07 13:05 alostale Review Assigned To => caristu
2019-05-07 13:06 alostale Review Assigned To caristu => shuehner
2019-05-21 15:17 hudsonbot Checkin
2019-05-21 15:17 hudsonbot Note Added: 0112016
2019-05-22 15:51 shuehner Note Added: 0112074
2019-05-22 15:51 shuehner Status resolved => closed
2019-05-22 15:51 shuehner Fixed in Version => 3.0PR19Q3


Copyright © 2000 - 2009 MantisBT Group
Powered by Mantis Bugtracker