0036033: Prevent duplicate/parallel processing of the same message from webpos if it gets retried due to long processing times

Description

THE PROBLEM TODAY

It seems today in the customer environment, a process which creates records in a specific table continuously was run. This process impacted very significantly an EventHandler which the integrations team created and that is executed every time an order is created. The end result was that an order creation message suddenly took a very significant time to execute (more than 30 seconds in some cases).

In the Web POS, we are running synchronized mode. When an order is completed, it is sent to the backend, and the client waits for the backend to confirm that the order was correctly processed to close the popup and allow the cashier to continue working. There is also a mechanism that initiates a retry, and sends the message again in case it reaches the defined timeout.

This means that if the OrderLoader for some reason takes a very long time to execute, it is possible that we do a retry when it is still processing, and in this case we may receive a duplicated primary key error.

It seems this is exactly what was happening today in BUT. As the OrderLoader took very long, the Web POS did a retry, and this retry was executed while the initial OrderLoader was being executed, so finally it failed with a primary key error.

A POSSIBLE SOLUTION

Initially, I thought that the fix for the following issue:

https://issues.openbravo.com/view.php?id=35289 [^]

would solve the problem. However, now I think it would not help in this case. The reason is that we create the ImportEntryArchive only when we have confirmation that the order was correctly created. This makes sense, as we have the requirement from the Synchronized Mode that we should only reply to the client that everything is fine when everything is indeed completely fine, but in this particular case, it means that the second request arrives at a moment in time in which the ImportEntryArchive record still doesn't exist, which means that the second request will initiate a duplicated call to OrderLoader.

Steps To Reproduce

Install/run multi-server
Set a breakpoint in the OrderLoader
Submit a ticket from webpos
See that the breakpoint is hit multiple times

Proposed Solution

I think that to avoid this problem, we need to do a slight modification of the fix for 35289. Instead of creating the ImportEntryArchive only when the process is done, we should:
- Create it at the beginning of the process, with status "Initial"
- Mark it as "Completed" when the process finishes.
- If a request arrives with the same message Id, then we should check if there is already an ImportEntryArchive for it, and then do the following:
   * If it doesn't exist, create it and initiate the process.
   * If it exists already, and it is in status 'Initial', then we should reply to the client that the message is still being processed, and that it should retry in some time, but we should not initiate the process again <- this avoids the duplicated execution problem
   * If it exists and it's marked as "Completed", then we should reply to the client that everything is fine.
   * If it exists and it's marked as "Error" then we should return an error message to the client.

I think with this change, we are covering all the cases in which the processing of the request takes too long. @Martin, if I missed something or I'm mistaken in some of the conclusions I reached feel free to correct me. I'm also available tomorrow if you want for a call on this topic.

Tags

No tags attached.

Attached Files

Relationships [ Relation Graph ] [ Dependency Graph ]

related to

design defect

0036196

RR17Q3

closed

mtaal

Prevent same ticket from being send twice by the user from being processed

Notes
(0097219) hgbot (developer) 2017-06-08 13:46	Repository: erp/pmods/org.openbravo.mobile.core Changeset: 790a090a9bf56cb7a05ceb0ac3d2f2da95abbc49 Author: Martin Taal <martin.taal <at> openbravo.com> Date: Thu Jun 08 13:46:21 2017 +0200 URL: http://code.openbravo.com/erp/pmods/org.openbravo.mobile.core/rev/790a090a9bf56cb7a05ceb0ac3d2f2da95abbc49 [^] Fixes issue 36033: Prevent duplicate/parallel processing of the same message from webpos if it gets retried due to long processing times In multi-server-json-process store an import entry archive to track that a message is being processed. Then depending on the state of the import entry archive return response to the client for subsequent duplicate requests. In ob-request-router handle the case that a message is returned to show to the user. --- M src-db/database/sourcedata/AD_MESSAGE.xml M src/org/openbravo/mobile/core/servercontroller/MobileServerTransitionToOnlineHandler.java M src/org/openbravo/mobile/core/servercontroller/MultiServerJSONProcess.java M web/org.openbravo.mobile.core/source/data/ob-requestrouter.js ---

Issue History
Date Modified	Username	Field	Change
2017-05-18 10:07	mtaal	New Issue
2017-05-18 10:07	mtaal	Assigned To	=> mtaal
2017-05-18 10:07	mtaal	Triggers an Emergency Pack	=> No
2017-06-08 13:44	mtaal	Relationship added	related to 0036196
2017-06-08 13:46	hgbot	Checkin
2017-06-08 13:46	hgbot	Note Added: 0097219
2017-06-08 13:46	hgbot	Status	new => resolved
2017-06-08 13:46	hgbot	Resolution	open => fixed
2017-06-08 13:46	hgbot	Fixed in SCM revision	=> http://code.openbravo.com/erp/pmods/org.openbravo.mobile.core/rev/790a090a9bf56cb7a05ceb0ac3d2f2da95abbc49 [^]
2017-06-15 14:23	mtaal	Review Assigned To	=> marvintm
2017-06-15 14:23	mtaal	Description Updated	View Revisions
2017-06-16 15:46	marvintm	Status	resolved => closed
2017-06-16 15:46	marvintm	Fixed in Version	=> RR17Q3