Openbravo Issue Tracking System - Openbravo ERP
View Issue Details
IDProjectCategoryView StatusDate SubmittedLast Update
0025461Openbravo ERPI. Performancepublic2014-01-10 17:082014-01-17 15:49
Reporteralostale 
Assigned Toalostale 
PrioritynormalSeveritymajorReproducibilityhave not tried
StatusclosedResolutionfixed 
PlatformOS5OS Version
Product Version 
Target VersionFixed in Version3.0PR14Q2 
Merge Request Status
Review Assigned ToAugustoMauch
OBNetwork customer
Web browser
ModulesCore
Support ticket
Regression level
Regression date
Regression introduced in release
Regression introduced by commit
Triggers an Emergency PackNo
Summary0025461: JSON WebServices don't support big volumes
DescriptionWhen using JSON standard WebServices it is not possible to execute queries returning big amount of data.
Steps To ReproduceTry to invoke a JSON standard WebService returning >100K records -> Server consumes all memory even before starting the response
Proposed Solution2 areas to improve:
-Use ScrollableResults instead of list
-Stream the response instead of generating it in memory before starting transmission
Additional Information
TagsPerformance
Relationships
related to defect 0037424 closed alostale Export To Csv too slow with high volumes 
causes defect 00282143.0PR15Q1 closed AugustoMauch The _selectedProperties parameter is not working on json rest webservices when not specifying a concrete record_id 
Attached Filespng before-fix.png (68,148) 2014-01-10 17:19
https://issues.openbravo.com/file_download.php?file_id=6629&type=bug
png

png after-fix.png (57,768) 2014-01-10 17:32
https://issues.openbravo.com/file_download.php?file_id=6630&type=bug
png
Issue History
Date ModifiedUsernameFieldChange
2014-01-10 17:08alostaleNew Issue
2014-01-10 17:08alostaleAssigned To => alostale
2014-01-10 17:08alostaleModules => Core
2014-01-10 17:08alostaleTriggers an Emergency Pack => No
2014-01-10 17:10alostaleSteps to Reproduce Updatedbug_revision_view_page.php?rev_id=5324#r5324
2014-01-10 17:18alostaleNote Added: 0063360
2014-01-10 17:18alostaleNote Edited: 0063360bug_revision_view_page.php?bugnote_id=0063360#r5326
2014-01-10 17:18alostaleNote Edited: 0063360bug_revision_view_page.php?bugnote_id=0063360#r5327
2014-01-10 17:19alostaleFile Added: before-fix.png
2014-01-10 17:32alostaleFile Added: after-fix.png
2014-01-10 17:33alostaleReview Assigned To => AugustoMauch
2014-01-10 17:36alostaleSummaryJSON WebServices doesn't support big volumes => JSON WebServices don't support big volumes
2014-01-10 17:37alostaleTag Attached: Performance
2014-01-10 17:38hgbotCheckin
2014-01-10 17:38hgbotNote Added: 0063361
2014-01-10 17:38hgbotStatusnew => resolved
2014-01-10 17:38hgbotResolutionopen => fixed
2014-01-10 17:38hgbotFixed in SCM revision => http://code.openbravo.com/erp/devel/pi/rev/c171ecac26732700b17deb174d4c0c309a1c7ed5 [^]
2014-01-13 16:43AugustoMauchNote Added: 0063387
2014-01-13 16:43AugustoMauchStatusresolved => closed
2014-01-13 16:43AugustoMauchFixed in Version => 3.0MP32
2014-01-17 15:49hudsonbotCheckin
2014-01-17 15:49hudsonbotNote Added: 0063490
2015-01-13 07:49alostaleRelationship addedcauses 0028214
2017-12-01 13:02alostaleRelationship addedblocks 0037424
2017-12-01 13:02alostaleRelationship replacedrelated to 0037424

Notes
(0063360)
alostale   
2014-01-10 17:18   
Executed test cases in an environment with 10K Order Lines and 360K Products.

* Retrieve all Order Lines (38MB response):
 -Checked output before and after the fix are identical (count parameters are now placed at the end of the json object and before they were at the beginning)
 -Checked memory consumption and CPU usage after and before the fix (see attached images)
 -Time goes down from 75s to 50s

* Retrieve all Products:
  -Before the fix the server crashed because of memory consumption, now it correctly finalizes after 10min
-Execute jmeter: https://code.openbravo.com/erp/devel/pi-performance/file/3a19a49b62fb/wsJSONOrderLine.jmx [^]

* Added test case to ci to check it: https://ci.openbravo.com/job/int-performance-pgsql/ [^]
(0063361)
hgbot   
2014-01-10 17:38   
Repository: erp/devel/pi
Changeset: c171ecac26732700b17deb174d4c0c309a1c7ed5
Author: Asier Lostalé <asier.lostale <at> openbravo.com>
Date: Fri Jan 10 17:35:45 2014 +0100
URL: http://code.openbravo.com/erp/devel/pi/rev/c171ecac26732700b17deb174d4c0c309a1c7ed5 [^]

fixed bug 25461: JSON WebServices don't support big volumes

-Using now ScrollableResults instead of list()
-Stream the result while iterating over the query instead of generating completely in memory

---
M modules/org.openbravo.service.json/src/org/openbravo/service/json/JsonRestServlet.java
---
(0063387)
AugustoMauch   
2014-01-13 16:43   
Code reviewed and verified in pi@c171ecac2673
(0063490)
hudsonbot   
2014-01-17 15:49   
A changeset related to this issue has been promoted main and to the
Central Repository, after passing a series of tests.

Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/9669102eb541 [^]
Maturity status: Test