Openbravo Issue Tracking System - Openbravo ERP
View Issue Details
0020658Openbravo ERPA. Platformpublic2012-05-31 18:402012-06-21 17:54
alostale 
alostale 
normalmajorhave not tried
closedfixed 
5
 
3.0MP133.0MP13 
Core
No
0020658: Heap space problem when exporting big csv files
Trying to export a grid to csv can stall the whole application if number of records being exported is big enough.

The problem is memory is not correctly cleaned up, so each loaded record keeps part of it in memory till the process finishes, reaching a point where 100% CPU is used trying to collect the garbage though it is not possible.
It hugely varies depending on the table being exported. Testing with 1.5GB of max heap space:
-200K Business Partners are exported in 1min (11MB) no memory issues
-130K Sales Invoices cannot be exported, after several minutes it exports ~45K (4.6MB) and 100%CPU
Performance
? issue-20658.export (2,171) 2012-06-01 08:55
https://issues.openbravo.com/file_download.php?file_id=5271&type=bug
Issue History
2012-05-31 18:40alostaleNew Issue
2012-05-31 18:40alostaleAssigned To => alostale
2012-05-31 18:40alostaleModules => Core
2012-05-31 19:09alostaleTag Attached: Performance
2012-05-31 19:24alostaleSteps to Reproduce Updatedbug_revision_view_page.php?rev_id=3515#r3515
2012-06-01 08:51alostaleSteps to Reproduce Updatedbug_revision_view_page.php?rev_id=3518#r3518
2012-06-01 08:55alostaleFile Added: issue-20658.export
2012-06-01 08:55alostaleNote Added: 0049571
2012-06-01 08:58alostaleSteps to Reproduce Updatedbug_revision_view_page.php?rev_id=3519#r3519
2012-06-01 09:04alostaleNote Added: 0049572
2012-06-01 09:08alostaleNote Edited: 0049572bug_revision_view_page.php?bugnote_id=0049572#r3521
2012-06-04 11:27hgbotCheckin
2012-06-04 11:27hgbotNote Added: 0049600
2012-06-04 11:27hgbotStatusnew => resolved
2012-06-04 11:27hgbotResolutionopen => fixed
2012-06-04 11:27hgbotFixed in SCM revision => http://code.openbravo.com/erp/devel/pi/rev/8466c9b3c506ffa81b0569106d700df5dcd68c62 [^]
2012-06-21 17:54AugustoMauchNote Added: 0049997
2012-06-21 17:54AugustoMauchStatusresolved => closed
2012-06-21 17:54AugustoMauchFixed in Version => 3.0MP13

Notes
(0049571)
alostale   
2012-06-01 08:55   
Attached patch since right now pi is closed to release 3.0MP12, fix will be included within 3.0MP13
(0049572)
alostale   
2012-06-01 09:04   
(edited on: 2012-06-01 09:08)
Executed test plan:
*Before applying the patch, export 100K BPs. Takes ~1min.
*After applying the patch, export 100K BPs. Takes ~1min.
*Compare both files: they are identical.
*Export 130K invoices. It takes ~10min. to export 13MB. While exporting one CPU is consuming 100%, but memory is properly cleaned so no much garbage collection consumption and in case of more CPUs available, the application is still usable. Before applying the patch, this case after 20min and 45K invoices exported made the application unusable with 100%CPU in garbage collection because of high memory consumption.

(0049600)
hgbot   
2012-06-04 11:27   
Repository: erp/devel/pi
Changeset: 8466c9b3c506ffa81b0569106d700df5dcd68c62
Author: Asier Lostalé <asier.lostale <at> openbravo.com>
Date: Fri Jun 01 08:53:46 2012 +0200
URL: http://code.openbravo.com/erp/devel/pi/rev/8466c9b3c506ffa81b0569106d700df5dcd68c62 [^]

fixed bug 20658: Heap space problem when exporting big csv files

In fetch for csv exportation, changed evict with clear. Evict removes current
record from memory, but it doesn't remove its references, this causes unnecesary
memory consumption.

---
M modules/org.openbravo.service.json/src/org/openbravo/service/json/DefaultJsonDataService.java
---
(0049997)
AugustoMauch   
2012-06-21 17:54   
Code reviewed and verified in pi@f06b2e72932e