Openbravo Issue Tracking System - Retail Modules
View Issue Details
IDProjectCategoryView StatusDate SubmittedLast Update
0033825Retail ModulesStoreServerpublic2016-08-26 15:252016-09-08 07:42
ReporterAugustoMauch 
Assigned ToAugustoMauch 
PrioritynormalSeveritymajorReproducibilityhave not tried
StatusclosedResolutionfixed 
PlatformOS5OS Version
Product Version 
Target VersionFixed in VersionRR16Q4 
Merge Request Status
Review Assigned Tomtaal
OBNetwork customerNo
Support ticket
Regression level
Regression date
Regression introduced in release
Regression introduced by commit
Triggers an Emergency PackNo
Summary0033825: Provide a process to check if the data that would be exported to a store server is consistent
DescriptionThe data of a store server is exported to XML/CSV by using the export.store.data task, and imported using install.source. If the data is not consistent (the Store Server Creation Dataset is not properly set or because, records from the * org reference a child org, etc), the user will not know it until it has exported the data of the store and then has imported it. In that case, the install.source will fail because it will not be able to enable the foreign keys, and only the first inconsistency will be logged.

It would be better to have a process that validates the data of a store server without the need of actually exporting/importing it. This process would not log only the first inconsistency it founds, but all of them if possible.
Steps To Reproduce-
Proposed Solution
Additional Information
TagsNo tags attached.
Relationships
Attached Files
Issue History
Date ModifiedUsernameFieldChange
2016-08-26 15:25AugustoMauchNew Issue
2016-08-26 15:25AugustoMauchAssigned To => AugustoMauch
2016-08-26 15:25AugustoMauchOBNetwork customer => No
2016-08-26 15:25AugustoMauchTriggers an Emergency Pack => No
2016-08-26 15:33hgbotCheckin
2016-08-26 15:33hgbotNote Added: 0089466
2016-08-26 15:33hgbotStatusnew => resolved
2016-08-26 15:33hgbotResolutionopen => fixed
2016-08-26 15:33hgbotFixed in SCM revision => http://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/2826513b84442d1709824aab70e559b168d70ccc [^]
2016-08-26 15:35AugustoMauchReview Assigned To => mtaal
2016-08-26 15:56hgbotCheckin
2016-08-26 15:56hgbotNote Added: 0089468
2016-09-05 10:46mtaalNote Added: 0089729
2016-09-05 10:46mtaalStatusresolved => new
2016-09-05 10:46mtaalResolutionfixed => open
2016-09-05 11:22hgbotCheckin
2016-09-05 11:22hgbotNote Added: 0089733
2016-09-05 11:27hgbotCheckin
2016-09-05 11:27hgbotNote Added: 0089734
2016-09-05 11:27hgbotCheckin
2016-09-05 11:27hgbotNote Added: 0089735
2016-09-05 11:27hgbotStatusnew => resolved
2016-09-05 11:27hgbotResolutionopen => fixed
2016-09-05 11:27hgbotFixed in SCM revisionhttp://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/2826513b84442d1709824aab70e559b168d70ccc [^] => http://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/dcce46ff580af6c6e94ababecbe2fa95995a3588 [^]
2016-09-06 18:18hgbotCheckin
2016-09-06 18:18hgbotNote Added: 0089788
2016-09-08 07:42mtaalNote Added: 0089822
2016-09-08 07:42mtaalStatusresolved => closed
2016-09-08 07:42mtaalFixed in Version => RR16Q4

Notes
(0089466)
hgbot   
2016-08-26 15:33   
Repository: erp/pmods/org.openbravo.replication.symmetricds
Changeset: 2826513b84442d1709824aab70e559b168d70ccc
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Fri Aug 26 15:31:21 2016 +0200
URL: http://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/2826513b84442d1709824aab70e559b168d70ccc [^]

Fixes issue 33825: Adds a process to validate the consistency of a store data

The new process has been implemented as an ant task:

ant validate.store.data -DdataSet=#dataSetId# -DmobileServerKey=#mobileServerKey#

It will validate the consistency of the data that would be exported using the dataSet #dataSetId# (for instance 8930335BAE1D454CAB1CC5208B14B3F0, the Store Server Creation Datase
t) for a given mobile server.

After going through the data that would be exported, it logs all the errors (that is, all the rows that would be exported but that reference records that would not be exported).

---
M build.xml
A src/org/openbravo/replication/symmetricds/OrganizationNaturalTreeForMobileServerProvider.java
A src/org/openbravo/replication/symmetricds/task/StoreDataValidatorTask.java
A src/org/openbravo/replication/symmetricds/validation/DataValidationError.java
A src/org/openbravo/replication/symmetricds/validation/ExportedTableCache.java
A src/org/openbravo/replication/symmetricds/validation/SynchronizationStoreDataValidator.java
---
(0089468)
hgbot   
2016-08-26 15:56   
Repository: erp/pmods/org.openbravo.replication.symmetricds
Changeset: a41b88c940e12ac8ce4acdf1bfcfe0383fa63d1b
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Fri Aug 26 15:53:41 2016 +0200
URL: http://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/a41b88c940e12ac8ce4acdf1bfcfe0383fa63d1b [^]

Related with issue 33825: Removes unneeded session clears and debug code

---
M src/org/openbravo/replication/symmetricds/task/StoreDataValidatorTask.java
M src/org/openbravo/replication/symmetricds/validation/ExportedTableCache.java
---
(0089729)
mtaal   
2016-09-05 10:46   
Review comments:
[10:06:57] Martin Taal: for the data validation
[10:07:04] Martin Taal: the cached ids are cleaned
[10:07:06] Martin Taal: https://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/2826513b84442d1709824aab70e559b168d70ccc#l6.356 [^]
[10:07:11] Martin Taal: so rereading ids is quite fast?
[10:11:03] Augusto Mauch: quite fast, yes
[10:11:10] Augusto Mauch: these are some stats :https://docs.google.com/spreadsheets/d/1dyibbBbTcSsJyC7s0HGw_XzP4WaeDSObCKNlP0MLEQo/edit#gid=181785338 [^]
[10:11:21] Augusto Mauch: the Time per table cache shows how much it costs to load the caches
[10:11:43] Martin Taal: okay
[10:11:43] Augusto Mauch: the thing is, if I did not clean the cache, the RAM went all over the roof
[10:11:46] Martin Taal: yes
[10:11:52] Martin Taal: I see you clean the table cache
[10:11:53] Martin Taal: https://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/2826513b84442d1709824aab70e559b168d70ccc#l6.356 [^]
[10:11:58] Martin Taal: right away
[10:12:00] Martin Taal: so why
[10:12:11] Martin Taal: decreaseReferencesToTablesAndClean
[10:12:13] Martin Taal: do that?
[10:12:29] Martin Taal: as you clean right away anyway right?
[10:13:17] Augusto Mauch: you are right, with the current approach that is not needed
[10:13:28] Martin Taal: okay
[10:13:32] Augusto Mauch: I left it there just in case I found a better way to clean the caches
[10:13:34] Martin Taal: fine to keep the code
[10:13:36] Augusto Mauch: but I can get rid of that
[10:13:37] Augusto Mauch: ok
[10:13:39] Martin Taal: but maybe with a note
[10:13:48] Augusto Mauch: I will add a note, sure
[10:14:13] Martin Taal: other question
[10:14:14] Martin Taal: 1- Why is this sleep statement here:
https://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/2826513b84442d1709824aab70e559b168d70ccc#l3.40 [^]
[10:15:54] Augusto Mauch: it shouldn't be there, I put it for logging purposes
[10:16:05] Martin Taal: okay
[10:16:09] Augusto Mauch: just so I could stop the yourkit profiling before the validation ended
[10:16:40] Martin Taal: 2- Not sure, is complete Ad_User exported?
https://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/2826513b84442d1709824aab70e559b168d70ccc#l6.101 [^]
[10:16:43] Martin Taal: .
[10:19:04] Augusto Mauch: you are right, it is not
[10:19:16] Augusto Mauch: at first sight I thought the two ad_user dataset tables covered all the cases
[10:19:18] Augusto Mauch: like ad_org
[10:19:36] Augusto Mauch: but it is not the case, all the ad_users of business partners that belong to other stores are not exported
[10:19:39] Augusto Mauch: I will fix that too
(0089733)
hgbot   
2016-09-05 11:22   
Repository: erp/pmods/org.openbravo.replication.symmetricds
Changeset: 1434c88a8fc882d01ad942610a30d5cb51c6ea05
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Mon Sep 05 11:19:42 2016 +0200
URL: http://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/1434c88a8fc882d01ad942610a30d5cb51c6ea05 [^]

Related with issue 33825: References to AD_USER should be checked

It was incorrectly assumed that the table AD_USER was not segmentated, and that the two ad_dataset_table entries that referenced it covered all the rows. This was a wrong assumption, only the entries of ad_user whose c_bpartner field is empty or one of the business partners of the exported store will be exported.

---
M src/org/openbravo/replication/symmetricds/validation/SynchronizationStoreDataValidator.java
---
(0089734)
hgbot   
2016-09-05 11:27   
Repository: erp/pmods/org.openbravo.replication.symmetricds
Changeset: 047a7d1700ce51564b4cf8de206e324e16a2039c
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Mon Sep 05 11:22:55 2016 +0200
URL: http://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/047a7d1700ce51564b4cf8de206e324e16a2039c [^]

Related with issue 33825: Remove code that is not used

Removes some code that was intented to be used to know when all the references to a table have been validated, in order to clear the cache. Currently the caches are emptied after every validation and reinitialized if needed for RAM issues, so this code is no longer needed.

---
M src/org/openbravo/replication/symmetricds/validation/SynchronizationStoreDataValidator.java
---
(0089735)
hgbot   
2016-09-05 11:27   
Repository: erp/pmods/org.openbravo.replication.symmetricds
Changeset: dcce46ff580af6c6e94ababecbe2fa95995a3588
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Mon Sep 05 11:25:01 2016 +0200
URL: http://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/dcce46ff580af6c6e94ababecbe2fa95995a3588 [^]

Fixes issue 33825: Prevents org.hibernate.LazyInitializationException

Instead of getting the dataset columns from the previously initialized dataset tables, they are now calculated by querying them to the database. I tried to prevent this error by initializing the dataset tables fully with Hibernate.initialize() but the problem persisted.

---
M src/org/openbravo/replication/symmetricds/validation/ExportedTableCache.java
---
(0089788)
hgbot   
2016-09-06 18:18   
Repository: erp/pmods/org.openbravo.replication.symmetricds
Changeset: 547351556eb0564a5e94f8fb9d359ce1c38d9e65
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Tue Sep 06 18:15:39 2016 +0200
URL: http://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/547351556eb0564a5e94f8fb9d359ce1c38d9e65 [^]

Related with issue 33825: CreatedBy and UpdatedBy properties are excluded from validation

The CreatedBy and UpdatedBy properties have been excluded from the validation. All the entities have those properties, and including them would make the validation much slower. In any case, the kind of users that appear in those columns are users that do not have a business partner associated, and all those are exported in the default dataset

---
M src/org/openbravo/replication/symmetricds/validation/SynchronizationStoreDataValidator.java
---
(0089822)
mtaal   
2016-09-08 07:42   
Reviewed and tested