Anonymous | Login
Project:
RSS
  
News | My View | View Issues | Roadmap | Summary

View Issue DetailsJump to Notes ] Issue History ] Print ]
ID
0020658
TypeCategorySeverityReproducibilityDate SubmittedLast Update
defect[Openbravo ERP] A. Platformmajorhave not tried2012-05-31 18:402012-06-21 17:54
ReporteralostaleView Statuspublic 
Assigned Toalostale 
PrioritynormalResolutionfixedFixed in Version3.0MP13
StatusclosedFix in branchFixed in SCM revision8466c9b3c506
ProjectionnoneETAnoneTarget Version3.0MP13
OSAnyDatabaseAnyJava version
OS VersionDatabase versionAnt version
Product VersionSCM revision 
Review Assigned To
Web browser
ModulesCore
Regression level
Regression date
Regression introduced in release
Regression introduced by commit
Triggers an Emergency PackNo
Summary

0020658: Heap space problem when exporting big csv files

DescriptionTrying to export a grid to csv can stall the whole application if number of records being exported is big enough.

The problem is memory is not correctly cleaned up, so each loaded record keeps part of it in memory till the process finishes, reaching a point where 100% CPU is used trying to collect the garbage though it is not possible.
Steps To ReproduceIt hugely varies depending on the table being exported. Testing with 1.5GB of max heap space:
-200K Business Partners are exported in 1min (11MB) no memory issues
-130K Sales Invoices cannot be exported, after several minutes it exports ~45K (4.6MB) and 100%CPU
TagsPerformance
Attached Files? file icon issue-20658.export [^] (2,171 bytes) 2012-06-01 08:55

- Relationships Relation Graph ] Dependency Graph ]

-  Notes
(0049571)
alostale (developer)
2012-06-01 08:55

Attached patch since right now pi is closed to release 3.0MP12, fix will be included within 3.0MP13
(0049572)
alostale (developer)
2012-06-01 09:04
edited on: 2012-06-01 09:08

Executed test plan:
*Before applying the patch, export 100K BPs. Takes ~1min.
*After applying the patch, export 100K BPs. Takes ~1min.
*Compare both files: they are identical.
*Export 130K invoices. It takes ~10min. to export 13MB. While exporting one CPU is consuming 100%, but memory is properly cleaned so no much garbage collection consumption and in case of more CPUs available, the application is still usable. Before applying the patch, this case after 20min and 45K invoices exported made the application unusable with 100%CPU in garbage collection because of high memory consumption.

(0049600)
hgbot (developer)
2012-06-04 11:27

Repository: erp/devel/pi
Changeset: 8466c9b3c506ffa81b0569106d700df5dcd68c62
Author: Asier Lostalé <asier.lostale <at> openbravo.com>
Date: Fri Jun 01 08:53:46 2012 +0200
URL: http://code.openbravo.com/erp/devel/pi/rev/8466c9b3c506ffa81b0569106d700df5dcd68c62 [^]

fixed bug 20658: Heap space problem when exporting big csv files

In fetch for csv exportation, changed evict with clear. Evict removes current
record from memory, but it doesn't remove its references, this causes unnecesary
memory consumption.

---
M modules/org.openbravo.service.json/src/org/openbravo/service/json/DefaultJsonDataService.java
---
(0049997)
AugustoMauch (developer)
2012-06-21 17:54

Code reviewed and verified in pi@f06b2e72932e

- Issue History
Date Modified Username Field Change
2012-05-31 18:40 alostale New Issue
2012-05-31 18:40 alostale Assigned To => alostale
2012-05-31 18:40 alostale Modules => Core
2012-05-31 19:09 alostale Tag Attached: Performance
2012-05-31 19:24 alostale Steps to Reproduce Updated View Revisions
2012-06-01 08:51 alostale Steps to Reproduce Updated View Revisions
2012-06-01 08:55 alostale File Added: issue-20658.export
2012-06-01 08:55 alostale Note Added: 0049571
2012-06-01 08:58 alostale Steps to Reproduce Updated View Revisions
2012-06-01 09:04 alostale Note Added: 0049572
2012-06-01 09:08 alostale Note Edited: 0049572 View Revisions
2012-06-04 11:27 hgbot Checkin
2012-06-04 11:27 hgbot Note Added: 0049600
2012-06-04 11:27 hgbot Status new => resolved
2012-06-04 11:27 hgbot Resolution open => fixed
2012-06-04 11:27 hgbot Fixed in SCM revision => http://code.openbravo.com/erp/devel/pi/rev/8466c9b3c506ffa81b0569106d700df5dcd68c62 [^]
2012-06-21 17:54 AugustoMauch Note Added: 0049997
2012-06-21 17:54 AugustoMauch Status resolved => closed
2012-06-21 17:54 AugustoMauch Fixed in Version => 3.0MP13


Copyright © 2000 - 2009 MantisBT Group
Powered by Mantis Bugtracker