|
Notes |
|
|
(0100894)
|
|
alostale
|
2017-12-01 13:27
(edited on: 2017-12-15 11:31) |
|
Generated 20K orders executing twice this script [1].
Showing in grid the following fields: organization,documentNo,transactionDocument,orderDate,businessPartner,partnerAddress,grandTotalAmount,currency
Exporting all those sales orders (default filters, order by document number).
Took around 10s to export (from localhost, so no wire data transfer is taken into account), see attached before-fix image.
CSV file generation and data export starts as soon as first record is retrieved from db and converted to csv, so in this case it took 143ms to generate the query execute it, receive 1st field and process 1st record.
The rest 9.69secs are spent in fetching from DB and processing the rest of the records.
Even not all record properties are requested, they were processed to later discard them (even expensive computed columns and foreing keys), preventing those to be processed reduced time to 2.5secs.
Note in this case the biggest impact was caused because of computed columns (up to 70% of the total time) which were calculated even not needed. After the fix, the biggest improvement is only noticeable removing those fields from grid.
---
[1] https://github.com/alostale/ob-scripts/blob/master/sql/data-load/create-order-headers.sql [^]
|
|
|
|
(0100895)
|
|
hgbot
|
|
2017-12-01 13:45
|
|
Repository: erp/devel/pi
Changeset: 1c94691c47047068d24fad438ecb83846ba14be3
Author: Asier Lostalé <asier.lostale <at> openbravo.com>
Date: Fri Dec 01 13:39:02 2017 +0100
URL: http://code.openbravo.com/erp/devel/pi/rev/1c94691c47047068d24fad438ecb83846ba14be3 [^]
fixed bug 37424: export to CSV too slow with high volumes
Wehn exporting to CSV from grid, all the properties of the entity were calculated
even if they weren't included in the final file. This computation added overhead,
specially for computed columns.
Now only those properties that are going to be included in the CSV file are calulated.
---
M modules/org.openbravo.service.datasource/src/org/openbravo/service/datasource/DataSourceServlet.java
---
|
|
|
|
(0100896)
|
|
hgbot
|
|
2017-12-01 13:45
|
|
Repository: erp/devel/pi
Changeset: abaa47f3a421c0fd59f7254bca8c0e232f6f86cd
Author: Asier Lostalé <asier.lostale <at> openbravo.com>
Date: Fri Dec 01 13:45:14 2017 +0100
URL: http://code.openbravo.com/erp/devel/pi/rev/abaa47f3a421c0fd59f7254bca8c0e232f6f86cd [^]
related to bug 37424: export to CSV too slow with high volumes
Small optimization: selectedProperties field is used to only check whether a
propertie is selected. Use a HashSet that has an O(1) contains instead of
ArrayList, which is O(n).
---
M modules/org.openbravo.service.json/src/org/openbravo/service/json/DataToJsonConverter.java
---
|
|
|
|
|
|
|
|
|
|
before-fix.png (18,294) 2017-12-01 13:28