Anonymous | Login
Project:
RSS
  
News | My View | View Issues | Roadmap | Summary

View Issue Details[ Jump to Notes ] [ Issue History ] [ Print ]
ID
0033825
TypeCategorySeverityReproducibilityDate SubmittedLast Update
feature request[Retail Modules] StoreServermajorhave not tried2016-08-26 15:252016-09-08 07:42
ReporterAugustoMauchView Statuspublic 
Assigned ToAugustoMauch 
PrioritynormalResolutionfixedFixed in VersionRR16Q4
StatusclosedFix in branchFixed in SCM revisiondcce46ff580a
ProjectionnoneETAnoneTarget Version
OSAnyDatabaseAnyJava version
OS VersionDatabase versionAnt version
Product VersionSCM revision 
Review Assigned Tomtaal
Regression level
Regression date
Regression introduced in release
Regression introduced by commit
Triggers an Emergency PackNo
Summary

0033825: Provide a process to check if the data that would be exported to a store server is consistent

DescriptionThe data of a store server is exported to XML/CSV by using the export.store.data task, and imported using install.source. If the data is not consistent (the Store Server Creation Dataset is not properly set or because, records from the * org reference a child org, etc), the user will not know it until it has exported the data of the store and then has imported it. In that case, the install.source will fail because it will not be able to enable the foreign keys, and only the first inconsistency will be logged.

It would be better to have a process that validates the data of a store server without the need of actually exporting/importing it. This process would not log only the first inconsistency it founds, but all of them if possible.
Steps To Reproduce-
TagsNo tags attached.
Attached Files

- Relationships [ Relation Graph ] [ Dependency Graph ]

-  Notes
(0089466)
hgbot (developer)
2016-08-26 15:33

 Repository: erp/pmods/org.openbravo.replication.symmetricds
Changeset: 2826513b84442d1709824aab70e559b168d70ccc
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Fri Aug 26 15:31:21 2016 +0200
URL: http://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/2826513b84442d1709824aab70e559b168d70ccc [^]

Fixes issue 33825: Adds a process to validate the consistency of a store data

The new process has been implemented as an ant task:

ant validate.store.data -DdataSet=#dataSetId# -DmobileServerKey=#mobileServerKey#

It will validate the consistency of the data that would be exported using the dataSet #dataSetId# (for instance 8930335BAE1D454CAB1CC5208B14B3F0, the Store Server Creation Datase
t) for a given mobile server.

After going through the data that would be exported, it logs all the errors (that is, all the rows that would be exported but that reference records that would not be exported).

---
M build.xml
A src/org/openbravo/replication/symmetricds/OrganizationNaturalTreeForMobileServerProvider.java
A src/org/openbravo/replication/symmetricds/task/StoreDataValidatorTask.java
A src/org/openbravo/replication/symmetricds/validation/DataValidationError.java
A src/org/openbravo/replication/symmetricds/validation/ExportedTableCache.java
A src/org/openbravo/replication/symmetricds/validation/SynchronizationStoreDataValidator.java
---
(0089468)
hgbot (developer)
2016-08-26 15:56

 Repository: erp/pmods/org.openbravo.replication.symmetricds
Changeset: a41b88c940e12ac8ce4acdf1bfcfe0383fa63d1b
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Fri Aug 26 15:53:41 2016 +0200
URL: http://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/a41b88c940e12ac8ce4acdf1bfcfe0383fa63d1b [^]

Related with issue 33825: Removes unneeded session clears and debug code

---
M src/org/openbravo/replication/symmetricds/task/StoreDataValidatorTask.java
M src/org/openbravo/replication/symmetricds/validation/ExportedTableCache.java
---
(0089729)
mtaal (manager)
2016-09-05 10:46

 Review comments:
[10:06:57] Martin Taal: for the data validation
[10:07:04] Martin Taal: the cached ids are cleaned
[10:07:06] Martin Taal: https://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/2826513b84442d1709824aab70e559b168d70ccc#l6.356 [^]
[10:07:11] Martin Taal: so rereading ids is quite fast?
[10:11:03] Augusto Mauch: quite fast, yes
[10:11:10] Augusto Mauch: these are some stats :https://docs.google.com/spreadsheets/d/1dyibbBbTcSsJyC7s0HGw_XzP4WaeDSObCKNlP0MLEQo/edit#gid=181785338 [^]
[10:11:21] Augusto Mauch: the Time per table cache shows how much it costs to load the caches
[10:11:43] Martin Taal: okay
[10:11:43] Augusto Mauch: the thing is, if I did not clean the cache, the RAM went all over the roof
[10:11:46] Martin Taal: yes
[10:11:52] Martin Taal: I see you clean the table cache
[10:11:53] Martin Taal: https://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/2826513b84442d1709824aab70e559b168d70ccc#l6.356 [^]
[10:11:58] Martin Taal: right away
[10:12:00] Martin Taal: so why
[10:12:11] Martin Taal: decreaseReferencesToTablesAndClean
[10:12:13] Martin Taal: do that?
[10:12:29] Martin Taal: as you clean right away anyway right?
[10:13:17] Augusto Mauch: you are right, with the current approach that is not needed
[10:13:28] Martin Taal: okay
[10:13:32] Augusto Mauch: I left it there just in case I found a better way to clean the caches
[10:13:34] Martin Taal: fine to keep the code
[10:13:36] Augusto Mauch: but I can get rid of that
[10:13:37] Augusto Mauch: ok
[10:13:39] Martin Taal: but maybe with a note
[10:13:48] Augusto Mauch: I will add a note, sure
[10:14:13] Martin Taal: other question
[10:14:14] Martin Taal: 1- Why is this sleep statement here:
https://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/2826513b84442d1709824aab70e559b168d70ccc#l3.40 [^]
[10:15:54] Augusto Mauch: it shouldn't be there, I put it for logging purposes
[10:16:05] Martin Taal: okay
[10:16:09] Augusto Mauch: just so I could stop the yourkit profiling before the validation ended
[10:16:40] Martin Taal: 2- Not sure, is complete Ad_User exported?
https://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/2826513b84442d1709824aab70e559b168d70ccc#l6.101 [^]
[10:16:43] Martin Taal: .
[10:19:04] Augusto Mauch: you are right, it is not
[10:19:16] Augusto Mauch: at first sight I thought the two ad_user dataset tables covered all the cases
[10:19:18] Augusto Mauch: like ad_org
[10:19:36] Augusto Mauch: but it is not the case, all the ad_users of business partners that belong to other stores are not exported
[10:19:39] Augusto Mauch: I will fix that too
(0089733)
hgbot (developer)
2016-09-05 11:22

 Repository: erp/pmods/org.openbravo.replication.symmetricds
Changeset: 1434c88a8fc882d01ad942610a30d5cb51c6ea05
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Mon Sep 05 11:19:42 2016 +0200
URL: http://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/1434c88a8fc882d01ad942610a30d5cb51c6ea05 [^]

Related with issue 33825: References to AD_USER should be checked

It was incorrectly assumed that the table AD_USER was not segmentated, and that the two ad_dataset_table entries that referenced it covered all the rows. This was a wrong assumption, only the entries of ad_user whose c_bpartner field is empty or one of the business partners of the exported store will be exported.

---
M src/org/openbravo/replication/symmetricds/validation/SynchronizationStoreDataValidator.java
---
(0089734)
hgbot (developer)
2016-09-05 11:27

 Repository: erp/pmods/org.openbravo.replication.symmetricds
Changeset: 047a7d1700ce51564b4cf8de206e324e16a2039c
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Mon Sep 05 11:22:55 2016 +0200
URL: http://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/047a7d1700ce51564b4cf8de206e324e16a2039c [^]

Related with issue 33825: Remove code that is not used

Removes some code that was intented to be used to know when all the references to a table have been validated, in order to clear the cache. Currently the caches are emptied after every validation and reinitialized if needed for RAM issues, so this code is no longer needed.

---
M src/org/openbravo/replication/symmetricds/validation/SynchronizationStoreDataValidator.java
---
(0089735)
hgbot (developer)
2016-09-05 11:27

 Repository: erp/pmods/org.openbravo.replication.symmetricds
Changeset: dcce46ff580af6c6e94ababecbe2fa95995a3588
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Mon Sep 05 11:25:01 2016 +0200
URL: http://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/dcce46ff580af6c6e94ababecbe2fa95995a3588 [^]

Fixes issue 33825: Prevents org.hibernate.LazyInitializationException

Instead of getting the dataset columns from the previously initialized dataset tables, they are now calculated by querying them to the database. I tried to prevent this error by initializing the dataset tables fully with Hibernate.initialize() but the problem persisted.

---
M src/org/openbravo/replication/symmetricds/validation/ExportedTableCache.java
---
(0089788)
hgbot (developer)
2016-09-06 18:18

 Repository: erp/pmods/org.openbravo.replication.symmetricds
Changeset: 547351556eb0564a5e94f8fb9d359ce1c38d9e65
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Tue Sep 06 18:15:39 2016 +0200
URL: http://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/547351556eb0564a5e94f8fb9d359ce1c38d9e65 [^]

Related with issue 33825: CreatedBy and UpdatedBy properties are excluded from validation

The CreatedBy and UpdatedBy properties have been excluded from the validation. All the entities have those properties, and including them would make the validation much slower. In any case, the kind of users that appear in those columns are users that do not have a business partner associated, and all those are exported in the default dataset

---
M src/org/openbravo/replication/symmetricds/validation/SynchronizationStoreDataValidator.java
---
(0089822)
mtaal (manager)
2016-09-08 07:42

 Reviewed and tested

- Issue History
Date Modified Username Field Change
2016-08-26 15:25 AugustoMauch New Issue
2016-08-26 15:25 AugustoMauch Assigned To => AugustoMauch
2016-08-26 15:25 AugustoMauch Triggers an Emergency Pack => No
2016-08-26 15:33 hgbot Checkin
2016-08-26 15:33 hgbot Note Added: 0089466
2016-08-26 15:33 hgbot Status new => resolved
2016-08-26 15:33 hgbot Resolution open => fixed
2016-08-26 15:33 hgbot Fixed in SCM revision => http://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/2826513b84442d1709824aab70e559b168d70ccc [^]
2016-08-26 15:35 AugustoMauch Review Assigned To => mtaal
2016-08-26 15:56 hgbot Checkin
2016-08-26 15:56 hgbot Note Added: 0089468
2016-09-05 10:46 mtaal Note Added: 0089729
2016-09-05 10:46 mtaal Status resolved => new
2016-09-05 10:46 mtaal Resolution fixed => open
2016-09-05 11:22 hgbot Checkin
2016-09-05 11:22 hgbot Note Added: 0089733
2016-09-05 11:27 hgbot Checkin
2016-09-05 11:27 hgbot Note Added: 0089734
2016-09-05 11:27 hgbot Checkin
2016-09-05 11:27 hgbot Note Added: 0089735
2016-09-05 11:27 hgbot Status new => resolved
2016-09-05 11:27 hgbot Resolution open => fixed
2016-09-05 11:27 hgbot Fixed in SCM revision http://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/2826513b84442d1709824aab70e559b168d70ccc [^] => http://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/dcce46ff580af6c6e94ababecbe2fa95995a3588 [^]
2016-09-06 18:18 hgbot Checkin
2016-09-06 18:18 hgbot Note Added: 0089788
2016-09-08 07:42 mtaal Note Added: 0089822
2016-09-08 07:42 mtaal Status resolved => closed
2016-09-08 07:42 mtaal Fixed in Version => RR16Q4

Copyright © 2000 - 2009 MantisBT Group
Powered by Mantis Bugtracker