0036033: Prevent duplicate/parallel processing of the same message from webpos if it gets retried due to long processing times

Openbravo Issue Tracking System - Retail Modules

View Issue Details

Project

Category

View Status

Date Submitted

Last Update

0036033

Retail Modules

StoreServer

public

2017-05-18 10:07

2017-06-16 15:46

Reporter

mtaal

Assigned To

mtaal

Priority

normal

Severity

major

Reproducibility

have not tried

Status

closed

Resolution

fixed

Platform

OS Version

Product Version

Target Version

RR17Q3

Fixed in Version

RR17Q3

Merge Request Status

Review Assigned To

marvintm

OBNetwork customer

Support ticket

Regression level

Regression date

Regression introduced in release

Regression introduced by commit

Triggers an Emergency Pack

Summary

0036033: Prevent duplicate/parallel processing of the same message from webpos if it gets retried due to long processing times

Description

THE PROBLEM TODAY

It seems today in the customer environment, a process which creates records in a specific table continuously was run. This process impacted very significantly an EventHandler which the integrations team created and that is executed every time an order is created. The end result was that an order creation message suddenly took a very significant time to execute (more than 30 seconds in some cases).

In the Web POS, we are running synchronized mode. When an order is completed, it is sent to the backend, and the client waits for the backend to confirm that the order was correctly processed to close the popup and allow the cashier to continue working. There is also a mechanism that initiates a retry, and sends the message again in case it reaches the defined timeout.

This means that if the OrderLoader for some reason takes a very long time to execute, it is possible that we do a retry when it is still processing, and in this case we may receive a duplicated primary key error.

It seems this is exactly what was happening today in BUT. As the OrderLoader took very long, the Web POS did a retry, and this retry was executed while the initial OrderLoader was being executed, so finally it failed with a primary key error.

A POSSIBLE SOLUTION

Initially, I thought that the fix for the following issue:

https://issues.openbravo.com/view.php?id=35289 [^]

would solve the problem. However, now I think it would not help in this case. The reason is that we create the ImportEntryArchive only when we have confirmation that the order was correctly created. This makes sense, as we have the requirement from the Synchronized Mode that we should only reply to the client that everything is fine when everything is indeed completely fine, but in this particular case, it means that the second request arrives at a moment in time in which the ImportEntryArchive record still doesn't exist, which means that the second request will initiate a duplicated call to OrderLoader.

Steps To Reproduce

Install/run multi-server
Set a breakpoint in the OrderLoader
Submit a ticket from webpos
See that the breakpoint is hit multiple times

Proposed Solution

I think that to avoid this problem, we need to do a slight modification of the fix for 35289. Instead of creating the ImportEntryArchive only when the process is done, we should:
- Create it at the beginning of the process, with status "Initial"
- Mark it as "Completed" when the process finishes.
- If a request arrives with the same message Id, then we should check if there is already an ImportEntryArchive for it, and then do the following:
   * If it doesn't exist, create it and initiate the process.
   * If it exists already, and it is in status 'Initial', then we should reply to the client that the message is still being processed, and that it should retry in some time, but we should not initiate the process again <- this avoids the duplicated execution problem
   * If it exists and it's marked as "Completed", then we should reply to the client that everything is fine.
   * If it exists and it's marked as "Error" then we should return an error message to the client.

I think with this change, we are covering all the cases in which the processing of the request takes too long. @Martin, if I missed something or I'm mistaken in some of the conclusions I reached feel free to correct me. I'm also available tomorrow if you want for a call on this topic.

Additional Information