Anonymous | Login
Project:
RSS
  
News | My View | View Issues | Roadmap | Summary

View Issue Details[ Jump to Notes ] [ Issue History ] [ Print ]
ID
0025461
TypeCategorySeverityReproducibilityDate SubmittedLast Update
defect[Openbravo ERP] I. Performancemajorhave not tried2014-01-10 17:082014-01-17 15:49
ReporteralostaleView Statuspublic 
Assigned Toalostale 
PrioritynormalResolutionfixedFixed in Version3.0PR14Q2
StatusclosedFix in branchFixed in SCM revisionc171ecac2673
ProjectionnoneETAnoneTarget Version
OSAnyDatabaseAnyJava version
OS VersionDatabase versionAnt version
Product VersionSCM revision 
Review Assigned ToAugustoMauch
Web browser
ModulesCore
Regression level
Regression date
Regression introduced in release
Regression introduced by commit
Triggers an Emergency PackNo
Summary

0025461: JSON WebServices don't support big volumes

DescriptionWhen using JSON standard WebServices it is not possible to execute queries returning big amount of data.
Steps To ReproduceTry to invoke a JSON standard WebService returning >100K records -> Server consumes all memory even before starting the response
Proposed Solution2 areas to improve:
-Use ScrollableResults instead of list
-Stream the response instead of generating it in memory before starting transmission
TagsPerformance
Attached Filespng file icon before-fix.png [^] (68,148 bytes) 2014-01-10 17:19


png file icon after-fix.png [^] (57,768 bytes) 2014-01-10 17:32

- Relationships [ Relation Graph ] [ Dependency Graph ]
related to defect 0037424 closedalostale Export To Csv too slow with high volumes 
causes defect 00282143.0PR15Q1 closedAugustoMauch The _selectedProperties parameter is not working on json rest webservices when not specifying a concrete record_id 

-  Notes
(0063360)
alostale (manager)
2014-01-10 17:18
edited on: 2014-01-10 17:18

 Executed test cases in an environment with 10K Order Lines and 360K Products.

* Retrieve all Order Lines (38MB response):
 -Checked output before and after the fix are identical (count parameters are now placed at the end of the json object and before they were at the beginning)
 -Checked memory consumption and CPU usage after and before the fix (see attached images)
 -Time goes down from 75s to 50s

* Retrieve all Products:
  -Before the fix the server crashed because of memory consumption, now it correctly finalizes after 10min
-Execute jmeter: https://code.openbravo.com/erp/devel/pi-performance/file/3a19a49b62fb/wsJSONOrderLine.jmx [^]

* Added test case to ci to check it: https://ci.openbravo.com/job/int-performance-pgsql/ [^]
(0063361)
hgbot (developer)
2014-01-10 17:38

 Repository: erp/devel/pi
Changeset: c171ecac26732700b17deb174d4c0c309a1c7ed5
Author: Asier Lostalé <asier.lostale <at> openbravo.com>
Date: Fri Jan 10 17:35:45 2014 +0100
URL: http://code.openbravo.com/erp/devel/pi/rev/c171ecac26732700b17deb174d4c0c309a1c7ed5 [^]

fixed bug 25461: JSON WebServices don't support big volumes

-Using now ScrollableResults instead of list()
-Stream the result while iterating over the query instead of generating completely in memory

---
M modules/org.openbravo.service.json/src/org/openbravo/service/json/JsonRestServlet.java
---
(0063387)
AugustoMauch (administrator)
2014-01-13 16:43

 Code reviewed and verified in pi@c171ecac2673
(0063490)
hudsonbot (developer)
2014-01-17 15:49

 A changeset related to this issue has been promoted main and to the
Central Repository, after passing a series of tests.

Promotion changeset: https://code.openbravo.com/erp/devel/main/rev/9669102eb541 [^]
Maturity status: Test

- Issue History
Date Modified Username Field Change
2014-01-10 17:08 alostale New Issue
2014-01-10 17:08 alostale Assigned To => alostale
2014-01-10 17:08 alostale Modules => Core
2014-01-10 17:08 alostale Triggers an Emergency Pack => No
2014-01-10 17:10 alostale Steps to Reproduce Updated View Revisions
2014-01-10 17:18 alostale Note Added: 0063360
2014-01-10 17:18 alostale Note Edited: 0063360 View Revisions
2014-01-10 17:18 alostale Note Edited: 0063360 View Revisions
2014-01-10 17:19 alostale File Added: before-fix.png
2014-01-10 17:32 alostale File Added: after-fix.png
2014-01-10 17:33 alostale Review Assigned To => AugustoMauch
2014-01-10 17:36 alostale Summary JSON WebServices doesn't support big volumes => JSON WebServices don't support big volumes
2014-01-10 17:37 alostale Tag Attached: Performance
2014-01-10 17:38 hgbot Checkin
2014-01-10 17:38 hgbot Note Added: 0063361
2014-01-10 17:38 hgbot Status new => resolved
2014-01-10 17:38 hgbot Resolution open => fixed
2014-01-10 17:38 hgbot Fixed in SCM revision => http://code.openbravo.com/erp/devel/pi/rev/c171ecac26732700b17deb174d4c0c309a1c7ed5 [^]
2014-01-13 16:43 AugustoMauch Note Added: 0063387
2014-01-13 16:43 AugustoMauch Status resolved => closed
2014-01-13 16:43 AugustoMauch Fixed in Version => 3.0MP32
2014-01-17 15:49 hudsonbot Checkin
2014-01-17 15:49 hudsonbot Note Added: 0063490
2015-01-13 07:49 alostale Relationship added causes 0028214
2017-12-01 13:02 alostale Relationship added blocks 0037424
2017-12-01 13:02 alostale Relationship replaced related to 0037424

Copyright © 2000 - 2009 MantisBT Group
Powered by Mantis Bugtracker