Anonymous | Login
Project:
RSS
  
News | My View | View Issues | Roadmap | Summary

View Issue Details[ Jump to Notes ] [ Issue History ] [ Print ]
ID
0034810
TypeCategorySeverityReproducibilityDate SubmittedLast Update
backport[Retail Modules] Web POScriticalhave not tried2016-12-01 16:372017-01-17 18:43
ReportermtaalView Statuspublic 
Assigned Tomarvintm 
PriorityimmediateResolutionfixedFixed in VersionRR16Q4.1
StatusclosedFix in branchFixed in SCM revision97f456dd2d47
ProjectionnoneETAnoneTarget VersionRR16Q4.1
OSAnyDatabaseAnyJava version
OS VersionDatabase versionAnt version
Product VersionSCM revision 
Review Assigned Tomarvintm
Regression levelProduction - Confirmed Stable
Regression date2015-10-28
Regression introduced in releaseRR16Q1
Regression introduced by commithttps://code.openbravo.com/erp/pmods/org.openbravo.mobile.core/rev/7f56ee2e32ec [^]
Triggers an Emergency PackNo
Summary

0034810: ProcessHQLQuery returns duplicates causing errors when inserting data and really slow login with many products

DescriptionThe ProcessHQLQuery class uses the offset and limit parameters to support paginated data [1]. This is needed in case of large datasets so that the client side is not flooded with too much data.

The ProcessHQLQuery class can execute multiple queries depending on the subclass implementation. It will change the offset/limit after each query [2]. For 2 examples see the Product [3] and Business Partner [4].

The problem is that if there are multiple queries then these queries will use different offset/limit parameters within one request. This means that the next request for the next page will get overlapping data with previous requests or can skip data.

Let's take an example:

Subclass of ProcessHQLQuery adds 2 queries. The first query would return 15000 records (unpaginated), the second query would return 8000 records (unpaginated).

Master data load:
Request 1:
offset: 0, limit: 10000
With the current code this would return only 10000 records of the first query, the second query won't be executed anymore
Request 2:
offset: 10000, limit: 10000
This would return 5000 records of the first query and the first 5000 records of the second query (the offset is set to zero if a query returns less than the limit).
Request 3:
offset: 20000, limit: 10000
This would return the first 8000 records of the second query, overlapping with request 2

[1]
https://code.openbravo.com/erp/pmods/org.openbravo.mobile.core/file/0d7bcfc60eb8/src/org/openbravo/mobile/core/process/ProcessHQLQuery.java#l165 [^]

[2]
https://code.openbravo.com/erp/pmods/org.openbravo.mobile.core/file/0d7bcfc60eb8/src/org/openbravo/mobile/core/process/ProcessHQLQuery.java#l229 [^]

[3]
https://code.openbravo.com/erp/pmods/org.openbravo.retail.posterminal/file/5754a065bc58/src/org/openbravo/retail/posterminal/master/Product.java#l247 [^]
https://code.openbravo.com/erp/pmods/org.openbravo.retail.posterminal/file/5754a065bc58/src/org/openbravo/retail/posterminal/master/Product.java#l271 [^]
https://code.openbravo.com/erp/pmods/org.openbravo.retail.posterminal/file/5754a065bc58/src/org/openbravo/retail/posterminal/master/Product.java#l301 [^]

[4]
https://code.openbravo.com/erp/pmods/org.openbravo.retail.posterminal/file/5754a065bc58/src/org/openbravo/retail/posterminal/master/BusinessPartner.java#l57 [^]
Steps To ReproduceCreate a database which would return products for all product queries in the product master

Set the limit in the request for products to something small
- Find this line:
handleIncrementalRequest(35000, 0, params, incremental);
and change it to:
handleIncrementalRequest(20, 0, params, incremental);

clear cache and log in --> duplicate errors
Proposed SolutionThe ProcessHQLQuery should not reset the limit or offset, they should be the same for each query to prevent duplicates and to prevent skipping data.

In BP/Product search you will get more results than the preference/default page size (each query will return max of that), but this is not too bad.

TagsNo tags attached.
Attached Files

- Relationships [ Relation Graph ] [ Dependency Graph ]
blocks defect 0034669RR17Q1 closedmigueldejuana ProcessHQLQuery returns duplicates causing errors when inserting data and really slow login with many products 

-  Notes
(0093599)
hgbot (developer)
2017-01-16 15:24

 Repository: retail/backports/3.0RR16Q4.1/org.openbravo.mobile.core
Changeset: 86d1c5f8100f924622be4a1f7b9daf748ceed0af
Author: Antonio Moreno <antonio.moreno <at> openbravo.com>
Date: Mon Jan 16 15:16:51 2017 +0100
URL: http://code.openbravo.com/retail/backports/3.0RR16Q4.1/org.openbravo.mobile.core/rev/86d1c5f8100f924622be4a1f7b9daf748ceed0af [^]

Fixed issue 34810. This changeset is not needed, offset is not needed to update in non-masterdata queries. In fact it breaks recalculation of limit using totalRows instead of queryRows

---
M src/org/openbravo/mobile/core/process/ProcessHQLQuery.java
---
(0093600)
hgbot (developer)
2017-01-16 15:24

 Repository: retail/backports/3.0RR16Q4.1/org.openbravo.mobile.core
Changeset: 97f456dd2d476b6d2e1c86232fc10cf165cf3ecb
Author: Antonio Moreno <antonio.moreno <at> openbravo.com>
Date: Mon Jan 16 15:23:45 2017 +0100
URL: http://code.openbravo.com/retail/backports/3.0RR16Q4.1/org.openbravo.mobile.core/rev/97f456dd2d476b6d2e1c86232fc10cf165cf3ecb [^]

Fixed issue 34810. ProcessHQLQuery returns duplicates causing errors when inserting data and really slow login with many products

- Change masterdata batch size from 35000 to 10000
- For masterdata requests send isMasterdata = true
- Request to backend, Parameters starting with '_' are not query params so we will set them in data object
- Define a reference to be able to change masterdata batch size with a preference
- In models with more than one query, do not recalculate limit for next queries when the request is a masterdata request, just recalculate it for remote requests

---
M src-db/database/sourcedata/AD_REF_LIST.xml
M src/org/openbravo/mobile/core/process/ProcessHQLQuery.java
M web/org.openbravo.mobile.core/source/data/ob-datasource.js
---

- Issue History
Date Modified Username Field Change
2016-12-22 12:18 marvintm Type defect => backport
2016-12-22 12:18 marvintm Target Version RR17Q1 => RR16Q4.1
2017-01-16 15:24 hgbot Checkin
2017-01-16 15:24 hgbot Note Added: 0093599
2017-01-16 15:24 hgbot Status scheduled => resolved
2017-01-16 15:24 hgbot Resolution open => fixed
2017-01-16 15:24 hgbot Fixed in SCM revision http://code.openbravo.com/erp/pmods/org.openbravo.mobile.core/rev/b08a151d8926837f4b917e77bad5f1be5c755565 [^] => http://code.openbravo.com/retail/backports/3.0RR16Q4.1/org.openbravo.mobile.core/rev/86d1c5f8100f924622be4a1f7b9daf748ceed0af [^]
2017-01-16 15:24 hgbot Checkin
2017-01-16 15:24 hgbot Note Added: 0093600
2017-01-16 15:24 hgbot Fixed in SCM revision http://code.openbravo.com/retail/backports/3.0RR16Q4.1/org.openbravo.mobile.core/rev/86d1c5f8100f924622be4a1f7b9daf748ceed0af [^] => http://code.openbravo.com/retail/backports/3.0RR16Q4.1/org.openbravo.mobile.core/rev/97f456dd2d476b6d2e1c86232fc10cf165cf3ecb [^]
2017-01-16 18:16 marvintm Assigned To migueldejuana => marvintm
2017-01-17 18:43 guilleaer Status resolved => closed
2017-01-17 18:43 guilleaer Fixed in Version => RR16Q4.1

Copyright © 2000 - 2009 MantisBT Group
Powered by Mantis Bugtracker