Openbravo Issue Tracking System - Openbravo ERP
View Issue Details
IDProjectCategoryView StatusDate SubmittedLast Update
0024506Openbravo ERPA. Platformpublic2013-08-07 14:302013-08-16 20:31
Reportercaristu 
Assigned Toshuehner 
PriorityurgentSeveritymajorReproducibilitysometimes
StatusclosedResolutionfixed 
PlatformOS5OS Version
Product Version 
Target VersionFixed in Version 
Merge Request Status
Review Assigned Tocaristu
OBNetwork customer
Web browser
ModulesCore
Support ticket
Regression level
Regression date
Regression introduced in release
Regression introduced by commit
Triggers an Emergency PackNo
Summary0024506: Performance problem when importing bank statements
DescriptionThe import bank statement process is working very slow with some files. There is a performance problem with this process because a strange gc pattern appears in the cpu usage.

The flush operations in the FIN_BankStatementImport java class are where most part of the time is spent.
Steps To Reproduce1- Configure tomcat with the following parameters: -Xms384M -Xmx1024M -XX:MaxPermSize=256M
2- Go to the [Financial Account] window
3- Choose an account and click on "Import Bank Statement"
4- Use the csv importer algorithm and load a bank statement with about 250 lines.
Proposed SolutionThe root of the problem is that the process is loading too many business partners into the memory from the database.

Most of those business partners are not modified during the process.

So there is no need of keeping those unmodified business partners in the DAL session. They should be evicted directly to reduce size of the session and thus greatly reducing time spend on flush iterating over entities in the session.
Additional Information
TagsPerformance
Relationships
has duplicate defect 0024246 closed caristu Modules Error OutOfMemory when you try to import a CSV with 800 lines 
Attached Files
Issue History
Date ModifiedUsernameFieldChange
2013-08-07 14:30caristuNew Issue
2013-08-07 14:30caristuAssigned To => shuehner
2013-08-07 14:30caristuModules => Core
2013-08-07 14:30caristuTriggers an Emergency Pack => No
2013-08-07 14:30caristuIssue Monitored: networkb
2013-08-07 14:31caristuTag Attached: Performance
2013-08-07 14:35shuehnerProposed Solution updated
2013-08-07 15:44hgbotCheckin
2013-08-07 15:44hgbotNote Added: 0060533
2013-08-07 15:44hgbotStatusnew => resolved
2013-08-07 15:44hgbotResolutionopen => fixed
2013-08-07 15:44hgbotFixed in SCM revision => http://code.openbravo.com/erp/devel/pi/rev/fcf4d70d60194e7afdca63dcbf4725ffb3201f07 [^]
2013-08-08 11:54caristuReview Assigned To => caristu
2013-08-08 11:54caristuNote Added: 0060543
2013-08-08 11:54caristuStatusresolved => closed
2013-08-16 20:31hudsonbotCheckin
2013-08-16 20:31hudsonbotNote Added: 0060632
2013-09-10 19:19caristuRelationship addedhas duplicate 0024246

Notes
(0060533)
hgbot   
2013-08-07 15:44   
Repository: erp/devel/pi
Changeset: fcf4d70d60194e7afdca63dcbf4725ffb3201f07
Author: Stefan Hühner <stefan.huehner <at> openbravo.com>
Date: Wed Aug 07 15:43:49 2013 +0200
URL: http://code.openbravo.com/erp/devel/pi/rev/fcf4d70d60194e7afdca63dcbf4725ffb3201f07 [^]

Fixed 24506: Improve performance/reduce gc time of bankstatement import.
The code searching for matching business partners in the import of
bankstatementlines is loading a very high number of bp entities into
memory over the course of an import (example 10k bp's loaded for importing
250 lines).
That causes sub-sequent calls to flush() to be very slow (>=30 seconds) as those
need to iterate over all the objects in the hibernate session.

As all those bp's are not modified during the import they can be directly removed
(evicted) from the hibernate session as there's no need to keep them there and
check if they have been modified at flush() time.

---
M modules/org.openbravo.advpaymentmngt/src/org/openbravo/advpaymentmngt/utility/FIN_BankStatementImport.java
---
(0060543)
caristu   
2013-08-08 11:54   
Code Review + Testing OK
(0060632)
hudsonbot   
2013-08-16 20:31   
A changeset related to this issue has been promoted main and to the
Central Repository, after passing a series of tests.

Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/59a1180e7f4f [^]

Maturity status: Test