Project:
| View Issue Details[ Jump to Notes ] | [ Issue History ] [ Print ] | |||||||
| ID | ||||||||
| 0034835 | ||||||||
| Type | Category | Severity | Reproducibility | Date Submitted | Last Update | |||
| feature request | [Retail Modules] StoreServer | major | have not tried | 2017-01-02 11:14 | 2017-03-09 16:49 | |||
| Reporter | AugustoMauch | View Status | public | |||||
| Assigned To | AugustoMauch | |||||||
| Priority | urgent | Resolution | fixed | Fixed in Version | RR17Q2 | |||
| Status | closed | Fix in branch | Fixed in SCM revision | 5f4547f93c1a | ||||
| Projection | none | ETA | none | Target Version | ||||
| OS | Any | Database | Any | Java version | ||||
| OS Version | Database version | Ant version | ||||||
| Product Version | SCM revision | |||||||
| Review Assigned To | mtaal | |||||||
| Regression level | ||||||||
| Regression date | ||||||||
| Regression introduced in release | ||||||||
| Regression introduced by commit | ||||||||
| Triggers an Emergency Pack | No | |||||||
| Summary | 0034835: Use another approach for the Table Reload process | |||||||
| Description | The job of the table reload process is to load data from the central to the store servers in the two conditions: - (A) When a store server is started for the first time, to get all the records relevant to it that have been modified in the central server after the store was exported. - (B) When a store server backup is started, to load all the updates that the backup might be missing The current table reload approach executes a query in each of the synchronized tables, adding a 'where updated > reload_timestamp' clause to each of the queries. This has several problems: - It is very bad in terms of performance if there is no index on the updated column - There is a 1000 record limitation for some tables, where it is required to build a query like this: ... where id in (...) - It requires to disable the foreign keys during the table reload, as the referencial integrity is not guaranteed until all the tables have been reloaded The new approach will do the following for each of the two conditions mentioned above: - (A) The store server will be registered in the central server just before it is exported. That way, the central server will keep track of the records that the store server should receive when it is first started. - (B) The store server will ask the central server for the updates done after a given timestamp, and the central server will resend the batches (sym_outgoing_batch table) whose creation date is posterior to the given timestamp. | |||||||
| Steps To Reproduce | - | |||||||
| Tags | No tags attached. | |||||||
| Attached Files | ||||||||
 Relationships [ Relation Graph ]
[ Dependency Graph ]
|
||||||||
|
||||||||
 Relationships |
|
Notes |
|
|
(0093086) hgbot (developer) 2017-01-02 11:35 |
Repository: erp/pmods/org.openbravo.replication.symmetricds Changeset: 6419436c0df585b11b627815127e81ff960f6d1b Author: Augusto Mauch <augusto.mauch <at> openbravo.com> Date: Mon Jan 02 11:33:14 2017 +0100 URL: http://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/6419436c0df585b11b627815127e81ff960f6d1b [^] Related with issue 34835: Table reload reimplementation When a store server is started for the first time, it needs to load all the change that have occurred in the central server since the store server was exported. In the previous approach, the store server sent a request to the central server with a timestamp, and the central server then queried each of the synchronized tables using that timestamp to obtain the records that were created/modified after the timestamp. This approach had several drawbacks: - It was very bad in terms of performance - It required dropping constraints and foreign keys during the table reload - It did not synchronized deletions (as the query would only retrieve new and updated records). The new approach works as follows: - The store server must be registered in the central server before being exported. A new task called register.store.server has been created: ant register.server -Dstore.server.key=storeServerKey - While the store server is offline, the central server will keep track of the changes relevant to it and will store them in the sym_outgoing_batch table. It will not try to actually send those records until the store server is online. - Once the store server is deployed, it will contact the central server, which will start sending them the pending batches. The SynchronizedTablesReloader was in charge of handling the previous table reload approach and is no longer needed. --- M build.xml M src/org/openbravo/replication/symmetricds/SetupSymmetricListener.java M src/org/openbravo/replication/symmetricds/task/SymmetricDSTask_data.xsql A src/org/openbravo/replication/symmetricds/task/RegisterServerTask.java A src/org/openbravo/replication/symmetricds/task/UnregisterServerTask.java R src/org/openbravo/replication/symmetricds/SynchronizedTablesReloader.java --- |
|
(0093143) hgbot (developer) 2017-01-03 17:23 |
Repository: erp/pmods/org.openbravo.replication.symmetricds Changeset: f8da777775694820a1612d8d3dad7b269c911f3b Author: Sandra Huguet <sandra.huguet <at> openbravo.com> Date: Tue Jan 03 17:22:53 2017 +0100 URL: http://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/f8da777775694820a1612d8d3dad7b269c911f3b [^] related to issue 34835 fixed failing build in oracle 8921 - storeserver oracle --- M src/org/openbravo/replication/symmetricds/task/SymmetricDSTask_data.xsql --- |
|
(0093144) hgbot (developer) 2017-01-03 18:04 |
Repository: erp/pmods/org.openbravo.replication.symmetricds Changeset: fa60204bf3541bf682c6d3e06e985dad79c0a858 Author: Sandra Huguet <sandra.huguet <at> openbravo.com> Date: Tue Jan 03 18:04:08 2017 +0100 URL: http://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/fa60204bf3541bf682c6d3e06e985dad79c0a858 [^] related to issue 34835 fixed failing build in oracle 8929 - storeserver oracle --- M src/org/openbravo/replication/symmetricds/task/SymmetricDSTask_data.xsql --- |
|
(0093279) hgbot (developer) 2017-01-09 18:21 |
Repository: erp/pmods/org.openbravo.retail.storeserver.synchronization Changeset: a1c207e7a2d82889bac80dfd01150813c9f534dc Author: Augusto Mauch <augusto.mauch <at> openbravo.com> Date: Mon Jan 09 18:17:05 2017 +0100 URL: http://code.openbravo.com/erp/pmods/org.openbravo.retail.storeserver.synchronization/rev/a1c207e7a2d82889bac80dfd01150813c9f534dc [^] Fixes issue 34835: Adds Batch Reload/Resend functionality This changeset contains the Batch Reload functionality. It is used to, given a store server key and a timestamp, reload the batches that were sent from the central store to the given store server after the given timestamp. This is useful, for instance, to recover the batches that may have been lost after setting up a store server backup. The OutgoingBatchReloader class contains the methods that provide the batch reload functionality. For a batch to be reloaded, its state needs to be set to 'RS' in the sym_outgoing_batch table of the central server and in the sym_incoming_batch table of the store server (if that batch exists). Two process definitions have been created: - Reload Batches in Store Server: To be executed from a Central Server. It has two parameters: A mobile server selector and a timestamp - Request Batch Reload from Central Server. To be executed from a Store Server. It has one parameter: a timestamp --- M src-db/database/sourcedata/AD_MENU.xml M src-db/database/sourcedata/AD_TREENODE.xml M src-db/database/sourcedata/OBUIAPP_PROCESS.xml A src-db/database/sourcedata/OBUIAPP_PARAMETER.xml A src/org/openbravo/retail/storeserver/synchronization/process/MarkIncomingBatchesToBeResent.java A src/org/openbravo/retail/storeserver/synchronization/process/OutgoingBatchReloader.java A src/org/openbravo/retail/storeserver/synchronization/process/ReloadBatchesInStoreServer.java A src/org/openbravo/retail/storeserver/synchronization/process/RequestBatchReloadToCentralServer.java A src/org/openbravo/retail/storeserver/synchronization/process/ResendOutgoingBatches.java --- |
|
(0093292) hgbot (developer) 2017-01-10 10:34 |
Repository: erp/pmods/org.openbravo.retail.storeserver.synchronization Changeset: 037a46d778adc854ea376832201b18598a0ec5e4 Author: Augusto Mauch <augusto.mauch <at> openbravo.com> Date: Tue Jan 10 10:33:24 2017 +0100 URL: http://code.openbravo.com/erp/pmods/org.openbravo.retail.storeserver.synchronization/rev/037a46d778adc854ea376832201b18598a0ec5e4 [^] Related with issue 34835: Adds elements to fix dbconsistency --- M src-db/database/sourcedata/AD_ELEMENT.xml M src-db/database/sourcedata/OBUIAPP_PARAMETER.xml --- |
|
(0093604) AugustoMauch (manager) 2017-01-16 17:38 |
Documentation about the registration of a store server before its data is exported: http://wiki.openbravo.com/wiki/How_to_create_a_Store_Server#Registering_a_store_in_the_central_server [^] |
|
(0094012) hgbot (developer) 2017-02-02 11:20 |
Repository: erp/pmods/org.openbravo.retail.storeserver.synchronization Changeset: 823ab4a35541aca04e827bac7d074b037470f0d8 Author: Augusto Mauch <augusto.mauch <at> openbravo.com> Date: Thu Feb 02 11:12:29 2017 +0100 URL: http://code.openbravo.com/erp/pmods/org.openbravo.retail.storeserver.synchronization/rev/823ab4a35541aca04e827bac7d074b037470f0d8 [^] Related with issue 34835: Reverts most of the previous batch, plus one change The way the table reload is going to be done has changed. It will be done automatically when the store server starts. It will send the central server the last batch it received (let's suppose it is X), and the central server will then reload all the outgoing batches sent to that store server whose batch id is higher than X. There is no longer need to have UI processes to handle the table reload, so they are going to be removed. Also, the OutgoingBatchReloader no longer will resend batches based on a date, it will do it now based on a batch id. --- M src-db/database/sourcedata/AD_MENU.xml M src-db/database/sourcedata/AD_TREENODE.xml M src-db/database/sourcedata/OBUIAPP_PROCESS.xml M src/org/openbravo/retail/storeserver/synchronization/process/OutgoingBatchReloader.java R src-db/database/sourcedata/OBUIAPP_PARAMETER.xml R src/org/openbravo/retail/storeserver/synchronization/process/MarkIncomingBatchesToBeResent.java R src/org/openbravo/retail/storeserver/synchronization/process/ReloadBatchesInStoreServer.java R src/org/openbravo/retail/storeserver/synchronization/process/RequestBatchReloadToCentralServer.java R src/org/openbravo/retail/storeserver/synchronization/process/ResendOutgoingBatches.java --- |
|
(0094013) hgbot (developer) 2017-02-02 11:20 |
Repository: erp/pmods/org.openbravo.retail.storeserver.synchronization Changeset: 2d4eeae08abd5e4039cb143992fde56a9fc47154 Author: Augusto Mauch <augusto.mauch <at> openbravo.com> Date: Thu Feb 02 11:19:32 2017 +0100 URL: http://code.openbravo.com/erp/pmods/org.openbravo.retail.storeserver.synchronization/rev/2d4eeae08abd5e4039cb143992fde56a9fc47154 [^] Related with issue 34835: Moves OutgoingBatchReloader to replication module The OutgoingBatchReloader is going to be used only by the SetupSymmetricListener, it makes sense that it is included in the same module. --- R src/org/openbravo/retail/storeserver/synchronization/process/OutgoingBatchReloader.java --- |
|
(0094016) AugustoMauch (manager) 2017-02-02 11:36 |
author Augusto Mauch <augusto.mauch@openbravo.com> Thu Feb 02 11:32:50 2017 +0100 (3 minutes ago) changeset 409 9a6f7c15b191 parent 408 6831c6401ef3 URL: https://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/9a6f7c15b191 [^] Related with issue : Removes ServletDisablerDuringTableReloadFilter class The ServletDisablerDuringTableReloadFilter was required by the no longer used table reload approach. That approach used to reload all synchronized tables one by one. There was not guarantee that the data would be consistent until the table reload had finished, that's why a servlet filter was put in place the reject all the requests done while the data was potentially in an inconsistent state. In the new table reload approach the data is not inconsistent at any point, so the servlet filter can be removed. |
|
(0094041) hgbot (developer) 2017-02-03 14:08 |
Repository: erp/pmods/org.openbravo.replication.symmetricds Changeset: 53b3936ba518d0d043607045c28803e9dc70dbe0 Author: Augusto Mauch <augusto.mauch <at> openbravo.com> Date: Fri Feb 03 13:12:40 2017 +0100 URL: http://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/53b3936ba518d0d043607045c28803e9dc70dbe0 [^] Related with issue 34835: Moves OutgoingBatchReloader to replication module The OutgoingBatchReloader, in charge of scheduling batches to be resent from the central server to a store server, is now part of the org.openbravo.replication.symmetricds module. --- A src/org/openbravo/replication/symmetricds/util/OutgoingBatchReloader.java --- |
|
(0094042) hgbot (developer) 2017-02-03 14:08 |
Repository: erp/pmods/org.openbravo.replication.symmetricds Changeset: 5f4547f93c1a9a8ba8add123b198ec084e8ff95e Author: Augusto Mauch <augusto.mauch <at> openbravo.com> Date: Fri Feb 03 14:00:40 2017 +0100 URL: http://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/5f4547f93c1a9a8ba8add123b198ec084e8ff95e [^] Fixes issue 34835: Implement new Initial Data Load approach The new Initial Data Load works as follows: - When the store server starts, it obtains from sym_incoming_batch the ids of last batches received, grouped per channel - It sends this information to the central server. - The central server compares this information with the ids of the last batches sent to the store server, obtained from sym_outgoing_batch - If it detects that some batches had already been sent but have not been received, they are resent - The central server returns the list of batch ids sent per channel - If the store servers is missing batches, it will store in SynchronizationEngineProvider the list of channels with missing data, along with the id of the batch that is the last one fro m the initial data load. - When a batch is loaded in the store server, the BatchDatabaseWriterControl filter will check if the Initial Data Load is now complete This command can be used to know if the current server has an Initial Data Loadin progress: SynchronizationEngineProvider.getInstance().isInitialDataLoadInProgress(). As a consequence of the new Initial Data Load approach, the central server must be available when the store server is started. The store server will do five attempts to connect to the c entral server, waiting 5 seconds between attempts. If the central server does not answer, Tomcat will be stopped. --- M src/org/openbravo/replication/symmetricds/SetupSymmetricListener.java M src/org/openbravo/replication/symmetricds/SynchronizationEngineProvider.java M src/org/openbravo/replication/symmetricds/extensions/BatchDatabaseWriterControl.java A src/org/openbravo/replication/symmetricds/LastOutgoingBatchIdProvider.java --- |
|
(0094043) hgbot (developer) 2017-02-03 14:08 |
Repository: erp/pmods/org.openbravo.replication.symmetricds Changeset: bf9538fb58e0fe2a62e2d6275794f6467390f701 Author: Augusto Mauch <augusto.mauch <at> openbravo.com> Date: Fri Feb 03 14:08:11 2017 +0100 URL: http://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/bf9538fb58e0fe2a62e2d6275794f6467390f701 [^] Related with issue 34835: Archive batches when a server is unregistered When a server is unregistered from its central server, the central server now replaces the mobileServerKey of the store server in the sym_outgoing_batch and sym_incoming_batch tables from <mobileServerKey> to <mobileServerKey>-archived. This is needed because as part of the Initial Data Load process the central server must know what was the last batch sent to a store server per channel. If a server is unregistered and registered again, there should not be any outgoing batches for that server, but the central server would find the outgoing batches of the server before it was unregistered, if they have not been purged yet. --- M src/org/openbravo/replication/symmetricds/process/UnregisterServer.java --- |
|
(0094062) hgbot (developer) 2017-02-06 09:53 |
Repository: erp/pmods/org.openbravo.retail.storeserver.synchronization Changeset: 772e8aa8e0c99d1c26d2a5fc5c7eac52fef5b83e Author: Augusto Mauch <augusto.mauch <at> openbravo.com> Date: Mon Feb 06 09:52:36 2017 +0100 URL: http://code.openbravo.com/erp/pmods/org.openbravo.retail.storeserver.synchronization/rev/772e8aa8e0c99d1c26d2a5fc5c7eac52fef5b83e [^] Related with issue 34835: Fixes database consistency Removes some entries from ad_element that are no longer used. --- M src-db/database/sourcedata/AD_ELEMENT.xml --- |
|
(0094885) mtaal (manager) 2017-03-09 16:49 |
Reviewed and tested |
|
Notes |
|
Issue History |
|||
| Date Modified | Username | Field | Change |
| 2017-01-02 11:14 | AugustoMauch | New Issue | |
| 2017-01-02 11:14 | AugustoMauch | Assigned To | => StoreServer |
| 2017-01-02 11:14 | AugustoMauch | Triggers an Emergency Pack | => No |
| 2017-01-02 11:35 | hgbot | Checkin | |
| 2017-01-02 11:35 | hgbot | Note Added: 0093086 | |
| 2017-01-03 11:02 | mtaal | Assigned To | StoreServer => AugustoMauch |
| 2017-01-03 17:23 | hgbot | Checkin | |
| 2017-01-03 17:23 | hgbot | Note Added: 0093143 | |
| 2017-01-03 18:04 | hgbot | Checkin | |
| 2017-01-03 18:04 | hgbot | Note Added: 0093144 | |
| 2017-01-09 18:17 | AugustoMauch | Review Assigned To | => mtaal |
| 2017-01-09 18:21 | hgbot | Checkin | |
| 2017-01-09 18:21 | hgbot | Note Added: 0093279 | |
| 2017-01-09 18:21 | hgbot | Status | new => resolved |
| 2017-01-09 18:21 | hgbot | Resolution | open => fixed |
| 2017-01-09 18:21 | hgbot | Fixed in SCM revision | => http://code.openbravo.com/erp/pmods/org.openbravo.retail.storeserver.synchronization/rev/a1c207e7a2d82889bac80dfd01150813c9f534dc [^] |
| 2017-01-10 10:34 | hgbot | Checkin | |
| 2017-01-10 10:34 | hgbot | Note Added: 0093292 | |
| 2017-01-16 13:29 | AugustoMauch | Note Added: 0093582 | |
| 2017-01-16 17:38 | AugustoMauch | Note Added: 0093604 | |
| 2017-02-02 10:43 | AugustoMauch | Status | resolved => new |
| 2017-02-02 10:43 | AugustoMauch | Resolution | fixed => open |
| 2017-02-02 11:20 | hgbot | Checkin | |
| 2017-02-02 11:20 | hgbot | Note Added: 0094012 | |
| 2017-02-02 11:20 | hgbot | Checkin | |
| 2017-02-02 11:20 | hgbot | Note Added: 0094013 | |
| 2017-02-02 11:36 | AugustoMauch | Note Added: 0094016 | |
| 2017-02-03 14:08 | hgbot | Checkin | |
| 2017-02-03 14:08 | hgbot | Note Added: 0094041 | |
| 2017-02-03 14:08 | hgbot | Checkin | |
| 2017-02-03 14:08 | hgbot | Note Added: 0094042 | |
| 2017-02-03 14:08 | hgbot | Status | new => resolved |
| 2017-02-03 14:08 | hgbot | Resolution | open => fixed |
| 2017-02-03 14:08 | hgbot | Fixed in SCM revision | http://code.openbravo.com/erp/pmods/org.openbravo.retail.storeserver.synchronization/rev/a1c207e7a2d82889bac80dfd01150813c9f534dc [^] => http://code.openbravo.com/erp/pmods/org.openbravo.replication.symmetricds/rev/5f4547f93c1a9a8ba8add123b198ec084e8ff95e [^] |
| 2017-02-03 14:08 | hgbot | Checkin | |
| 2017-02-03 14:08 | hgbot | Note Added: 0094043 | |
| 2017-02-03 14:12 | AugustoMauch | Note Deleted: 0093582 | |
| 2017-02-06 09:53 | hgbot | Checkin | |
| 2017-02-06 09:53 | hgbot | Note Added: 0094062 | |
| 2017-03-09 16:49 | mtaal | Relationship added | depends on 0035470 |
| 2017-03-09 16:49 | mtaal | Note Added: 0094885 | |
| 2017-03-09 16:49 | mtaal | Status | resolved => closed |
| 2017-03-09 16:49 | mtaal | Fixed in Version | => RR17Q2 |
|
Issue History |
| Copyright © 2000 - 2009 MantisBT Group |
 |