Openbravo Issue Tracking System - Retail Modules |
View Issue Details |
|
ID | Project | Category | View Status | Date Submitted | Last Update |
0034835 | Retail Modules | StoreServer | public | 2017-01-02 11:14 | 2017-03-09 16:49 |
|
Reporter | AugustoMauch | |
Assigned To | AugustoMauch | |
Priority | urgent | Severity | major | Reproducibility | have not tried |
Status | closed | Resolution | fixed | |
Platform | | OS | 5 | OS Version | |
Product Version | | |
Target Version | | Fixed in Version | RR17Q2 | |
Merge Request Status | |
Review Assigned To | mtaal |
OBNetwork customer | |
Support ticket | |
Regression level | |
Regression date | |
Regression introduced in release | |
Regression introduced by commit | |
Triggers an Emergency Pack | No |
|
Summary | 0034835: Use another approach for the Table Reload process |
Description | The job of the table reload process is to load data from the central to the store servers in the two conditions:
- (A) When a store server is started for the first time, to get all the records relevant to it that have been modified in the central server after the store was exported.
- (B) When a store server backup is started, to load all the updates that the backup might be missing
The current table reload approach executes a query in each of the synchronized tables, adding a 'where updated > reload_timestamp' clause to each of the queries. This has several problems:
- It is very bad in terms of performance if there is no index on the updated column
- There is a 1000 record limitation for some tables, where it is required to build a query like this: ... where id in (...)
- It requires to disable the foreign keys during the table reload, as the referencial integrity is not guaranteed until all the tables have been reloaded
The new approach will do the following for each of the two conditions mentioned above:
- (A) The store server will be registered in the central server just before it is exported. That way, the central server will keep track of the records that the store server should receive when it is first started.
- (B) The store server will ask the central server for the updates done after a given timestamp, and the central server will resend the batches (sym_outgoing_batch table) whose creation date is posterior to the given timestamp. |
Steps To Reproduce | - |
Proposed Solution | |
Additional Information | |
Tags | No tags attached. |
Relationships | depends on | design defect | 0035470 | RR17Q2 | closed | AugustoMauch | Code should not adapt the Map instance it gets from a method, but instead call the originating class to adapt the Map |
|
Attached Files | |
|
Issue History |
Date Modified | Username | Field | Change |
2017-01-02 11:14 | AugustoMauch | New Issue | |
2017-01-02 11:14 | AugustoMauch | Assigned To | => StoreServer |
2017-01-02 11:14 | AugustoMauch | Triggers an Emergency Pack | => No |
2017-01-02 11:35 | hgbot | Checkin | |
2017-01-02 11:35 | hgbot | Note Added: 0093086 | |
2017-01-03 11:02 | mtaal | Assigned To | StoreServer => AugustoMauch |
2017-01-03 17:23 | hgbot | Checkin | |
2017-01-03 17:23 | hgbot | Note Added: 0093143 | |
2017-01-03 18:04 | hgbot | Checkin | |
2017-01-03 18:04 | hgbot | Note Added: 0093144 | |
2017-01-09 18:17 | AugustoMauch | Review Assigned To | => mtaal |
2017-01-09 18:21 | hgbot | Checkin | |
2017-01-09 18:21 | hgbot | Note Added: 0093279 | |
2017-01-09 18:21 | hgbot | Status | new => resolved |
2017-01-09 18:21 | hgbot | Resolution | open => fixed |
2017-01-09 18:21 | hgbot | Fixed in SCM revision | => http://code.openbravo.com/erp/pmods/org.openbravo.retail.storeserver.synchronization/rev/a1c207e7a2d82889bac80dfd01150813c9f534dc [^] |
2017-01-10 10:34 | hgbot | Checkin | |
2017-01-10 10:34 | hgbot | Note Added: 0093292 | |
2017-01-16 13:29 | AugustoMauch | Note Added: 0093582 | |
2017-01-16 17:38 | AugustoMauch | Note Added: 0093604 | |
2017-02-02 10:43 | AugustoMauch | Status | resolved => new |
2017-02-02 10:43 | AugustoMauch | Resolution | fixed => open |
2017-02-02 11:20 | hgbot | Checkin | |
2017-02-02 11:20 | hgbot | Note Added: 0094012 | |
2017-02-02 11:20 | hgbot | Checkin | |
2017-02-02 11:20 | hgbot | Note Added: 0094013 | |
2017-02-02 11:36 | AugustoMauch | Note Added: 0094016 | |
2017-02-03 14:08 | hgbot | Checkin | |
2017-02-03 14:08 | hgbot | Note Added: 0094041 | |
2017-02-03 14:08 | hgbot | Checkin | |
2017-02-03 14:08 | hgbot | Note Added: 0094042 | |
2017-02-03 14:08 | hgbot | Status | new => resolved |
2017-02-03 14:08 | hgbot | Resolution | open => fixed |
2017-02-03 14:08 | hgbot | Fixed in SCM revision | http://code.openbravo.com/erp/pmods/org.openbravo.retail.storeserver.synchronization/rev/a1c207e7a2d82889bac80dfd01150813c9f534dc [^] => http://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/5f4547f93c1a9a8ba8add123b198ec084e8ff95e [^] |
2017-02-03 14:08 | hgbot | Checkin | |
2017-02-03 14:08 | hgbot | Note Added: 0094043 | |
2017-02-03 14:12 | AugustoMauch | Note Deleted: 0093582 | |
2017-02-06 09:53 | hgbot | Checkin | |
2017-02-06 09:53 | hgbot | Note Added: 0094062 | |
2017-03-09 16:49 | mtaal | Relationship added | depends on 0035470 |
2017-03-09 16:49 | mtaal | Note Added: 0094885 | |
2017-03-09 16:49 | mtaal | Status | resolved => closed |
2017-03-09 16:49 | mtaal | Fixed in Version | => RR17Q2 |
Notes |
|
(0093086)
|
hgbot
|
2017-01-02 11:35
|
|
Repository: erp/pmods/org.openbravo.replication.symmetricds
Changeset: 6419436c0df585b11b627815127e81ff960f6d1b
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Mon Jan 02 11:33:14 2017 +0100
URL: http://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/6419436c0df585b11b627815127e81ff960f6d1b [^]
Related with issue 34835: Table reload reimplementation
When a store server is started for the first time, it needs to load all the change that have occurred in the central server since the store server was exported.
In the previous approach, the store server sent a request to the central server with a timestamp, and the central server then queried each of the synchronized tables using that timestamp to obtain the records that were created/modified after the timestamp. This approach had several drawbacks:
- It was very bad in terms of performance
- It required dropping constraints and foreign keys during the table reload
- It did not synchronized deletions (as the query would only retrieve new and updated records).
The new approach works as follows:
- The store server must be registered in the central server before being exported. A new task called register.store.server has been created: ant register.server -Dstore.server.key=storeServerKey
- While the store server is offline, the central server will keep track of the changes relevant to it and will store them in the sym_outgoing_batch table. It will not try to actually send those records until the store server is online.
- Once the store server is deployed, it will contact the central server, which will start sending them the pending batches.
The SynchronizedTablesReloader was in charge of handling the previous table reload approach and is no longer needed.
---
M build.xml
M src/org/openbravo/replication/symmetricds/SetupSymmetricListener.java
M src/org/openbravo/replication/symmetricds/task/SymmetricDSTask_data.xsql
A src/org/openbravo/replication/symmetricds/task/RegisterServerTask.java
A src/org/openbravo/replication/symmetricds/task/UnregisterServerTask.java
R src/org/openbravo/replication/symmetricds/SynchronizedTablesReloader.java
---
|
|
|
(0093143)
|
hgbot
|
2017-01-03 17:23
|
|
|
|
(0093144)
|
hgbot
|
2017-01-03 18:04
|
|
|
|
(0093279)
|
hgbot
|
2017-01-09 18:21
|
|
Repository: erp/pmods/org.openbravo.retail.storeserver.synchronization
Changeset: a1c207e7a2d82889bac80dfd01150813c9f534dc
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Mon Jan 09 18:17:05 2017 +0100
URL: http://code.openbravo.com/erp/pmods/org.openbravo.retail.storeserver.synchronization/rev/a1c207e7a2d82889bac80dfd01150813c9f534dc [^]
Fixes issue 34835: Adds Batch Reload/Resend functionality
This changeset contains the Batch Reload functionality. It is used to, given a store server key and a timestamp, reload the batches that were sent from the central store to the given store server after the given timestamp. This is useful, for instance, to recover the batches that may have been lost after setting up a store server backup.
The OutgoingBatchReloader class contains the methods that provide the batch reload functionality. For a batch to be reloaded, its state needs to be set to 'RS' in the sym_outgoing_batch table of the central server and in the sym_incoming_batch table of the store server (if that batch exists).
Two process definitions have been created:
- Reload Batches in Store Server: To be executed from a Central Server. It has two parameters: A mobile server selector and a timestamp
- Request Batch Reload from Central Server. To be executed from a Store Server. It has one parameter: a timestamp
---
M src-db/database/sourcedata/AD_MENU.xml
M src-db/database/sourcedata/AD_TREENODE.xml
M src-db/database/sourcedata/OBUIAPP_PROCESS.xml
A src-db/database/sourcedata/OBUIAPP_PARAMETER.xml
A src/org/openbravo/retail/storeserver/synchronization/process/MarkIncomingBatchesToBeResent.java
A src/org/openbravo/retail/storeserver/synchronization/process/OutgoingBatchReloader.java
A src/org/openbravo/retail/storeserver/synchronization/process/ReloadBatchesInStoreServer.java
A src/org/openbravo/retail/storeserver/synchronization/process/RequestBatchReloadToCentralServer.java
A src/org/openbravo/retail/storeserver/synchronization/process/ResendOutgoingBatches.java
---
|
|
|
(0093292)
|
hgbot
|
2017-01-10 10:34
|
|
|
|
|
|
|
(0094012)
|
hgbot
|
2017-02-02 11:20
|
|
Repository: erp/pmods/org.openbravo.retail.storeserver.synchronization
Changeset: 823ab4a35541aca04e827bac7d074b037470f0d8
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Thu Feb 02 11:12:29 2017 +0100
URL: http://code.openbravo.com/erp/pmods/org.openbravo.retail.storeserver.synchronization/rev/823ab4a35541aca04e827bac7d074b037470f0d8 [^]
Related with issue 34835: Reverts most of the previous batch, plus one change
The way the table reload is going to be done has changed. It will be done automatically when the store server starts. It will send the central server the last batch it received (let's suppose it is X), and the central server will then reload all the outgoing batches sent to that store server whose batch id is higher than X.
There is no longer need to have UI processes to handle the table reload, so they are going to be removed. Also, the OutgoingBatchReloader no longer will resend batches based on a date, it will do it now based on a batch id.
---
M src-db/database/sourcedata/AD_MENU.xml
M src-db/database/sourcedata/AD_TREENODE.xml
M src-db/database/sourcedata/OBUIAPP_PROCESS.xml
M src/org/openbravo/retail/storeserver/synchronization/process/OutgoingBatchReloader.java
R src-db/database/sourcedata/OBUIAPP_PARAMETER.xml
R src/org/openbravo/retail/storeserver/synchronization/process/MarkIncomingBatchesToBeResent.java
R src/org/openbravo/retail/storeserver/synchronization/process/ReloadBatchesInStoreServer.java
R src/org/openbravo/retail/storeserver/synchronization/process/RequestBatchReloadToCentralServer.java
R src/org/openbravo/retail/storeserver/synchronization/process/ResendOutgoingBatches.java
---
|
|
|
(0094013)
|
hgbot
|
2017-02-02 11:20
|
|
Repository: erp/pmods/org.openbravo.retail.storeserver.synchronization
Changeset: 2d4eeae08abd5e4039cb143992fde56a9fc47154
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Thu Feb 02 11:19:32 2017 +0100
URL: http://code.openbravo.com/erp/pmods/org.openbravo.retail.storeserver.synchronization/rev/2d4eeae08abd5e4039cb143992fde56a9fc47154 [^]
Related with issue 34835: Moves OutgoingBatchReloader to replication module
The OutgoingBatchReloader is going to be used only by the SetupSymmetricListener, it makes sense that it is included in the same module.
---
R src/org/openbravo/retail/storeserver/synchronization/process/OutgoingBatchReloader.java
---
|
|
|
|
author Augusto Mauch <augusto.mauch@openbravo.com>
Thu Feb 02 11:32:50 2017 +0100 (3 minutes ago)
changeset 409 9a6f7c15b191
parent 408 6831c6401ef3
URL: https://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/9a6f7c15b191 [^]
Related with issue : Removes ServletDisablerDuringTableReloadFilter class
The ServletDisablerDuringTableReloadFilter was required by the no longer used table reload approach. That approach used to reload all synchronized tables one by one. There was not guarantee that the data would be consistent until the table reload had finished, that's why a servlet filter was put in place the reject all the requests done while the data was potentially in an inconsistent state.
In the new table reload approach the data is not inconsistent at any point, so the servlet filter can be removed. |
|
|
(0094041)
|
hgbot
|
2017-02-03 14:08
|
|
Repository: erp/pmods/org.openbravo.replication.symmetricds
Changeset: 53b3936ba518d0d043607045c28803e9dc70dbe0
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Fri Feb 03 13:12:40 2017 +0100
URL: http://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/53b3936ba518d0d043607045c28803e9dc70dbe0 [^]
Related with issue 34835: Moves OutgoingBatchReloader to replication module
The OutgoingBatchReloader, in charge of scheduling batches to be resent from the central server to a store server, is now part of the org.openbravo.replication.symmetricds module.
---
A src/org/openbravo/replication/symmetricds/util/OutgoingBatchReloader.java
---
|
|
|
(0094042)
|
hgbot
|
2017-02-03 14:08
|
|
Repository: erp/pmods/org.openbravo.replication.symmetricds
Changeset: 5f4547f93c1a9a8ba8add123b198ec084e8ff95e
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Fri Feb 03 14:00:40 2017 +0100
URL: http://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/5f4547f93c1a9a8ba8add123b198ec084e8ff95e [^]
Fixes issue 34835: Implement new Initial Data Load approach
The new Initial Data Load works as follows:
- When the store server starts, it obtains from sym_incoming_batch the ids of last batches received, grouped per channel
- It sends this information to the central server.
- The central server compares this information with the ids of the last batches sent to the store server, obtained from sym_outgoing_batch
- If it detects that some batches had already been sent but have not been received, they are resent
- The central server returns the list of batch ids sent per channel
- If the store servers is missing batches, it will store in SynchronizationEngineProvider the list of channels with missing data, along with the id of the batch that is the last one fro
m the initial data load.
- When a batch is loaded in the store server, the BatchDatabaseWriterControl filter will check if the Initial Data Load is now complete
This command can be used to know if the current server has an Initial Data Loadin progress: SynchronizationEngineProvider.getInstance().isInitialDataLoadInProgress().
As a consequence of the new Initial Data Load approach, the central server must be available when the store server is started. The store server will do five attempts to connect to the c
entral server, waiting 5 seconds between attempts. If the central server does not answer, Tomcat will be stopped.
---
M src/org/openbravo/replication/symmetricds/SetupSymmetricListener.java
M src/org/openbravo/replication/symmetricds/SynchronizationEngineProvider.java
M src/org/openbravo/replication/symmetricds/extensions/BatchDatabaseWriterControl.java
A src/org/openbravo/replication/symmetricds/LastOutgoingBatchIdProvider.java
---
|
|
|
(0094043)
|
hgbot
|
2017-02-03 14:08
|
|
Repository: erp/pmods/org.openbravo.replication.symmetricds
Changeset: bf9538fb58e0fe2a62e2d6275794f6467390f701
Author: Augusto Mauch <augusto.mauch <at> openbravo.com>
Date: Fri Feb 03 14:08:11 2017 +0100
URL: http://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/bf9538fb58e0fe2a62e2d6275794f6467390f701 [^]
Related with issue 34835: Archive batches when a server is unregistered
When a server is unregistered from its central server, the central server now replaces the mobileServerKey of the store server in the sym_outgoing_batch and sym_incoming_batch tables from <mobileServerKey> to <mobileServerKey>-archived.
This is needed because as part of the Initial Data Load process the central server must know what was the last batch sent to a store server per channel. If a server is unregistered and registered again, there should not be any outgoing batches for that server, but the central server would find the outgoing batches of the server before it was unregistered, if they have not been purged yet.
---
M src/org/openbravo/replication/symmetricds/process/UnregisterServer.java
---
|
|
|
(0094062)
|
hgbot
|
2017-02-06 09:53
|
|
|
|
(0094885)
|
mtaal
|
2017-03-09 16:49
|
|
|