0047139: Import entries can be permanently left unprocessed, in Initial status

Openbravo Issue Tracking System - Openbravo ERP

View Issue Details

Project

Category

View Status

Date Submitted

Last Update

0047139

Openbravo ERP

A. Platform

public

2021-06-15 12:35

2022-04-03 19:36

Reporter

AugustoMauch

Assigned To

AugustoMauch

Priority

normal

Severity

critical

Reproducibility

have not tried

Status

closed

Resolution

fixed

Platform

OS Version

Product Version

Target Version

Fixed in Version

PR21Q3

Merge Request Status

Review Assigned To

OBNetwork customer

Web browser

Modules

Core

Support ticket

Regression level

Regression date

Regression introduced in release

Regression introduced by commit

Triggers an Emergency Pack

Summary

0047139: Import entries can be permanently left unprocessed, in Initial status

Description

The ImportEntryManager uses a ThreadPoolExecutor to execute import entries concurrently. The ThreadPoolExecutor uses an ArrayBlockingQueue to store the tasks before they are executed. By default that queue has a maximum capacity of 1000, that value can be configured by setting a different value on the import.max.task.queue.size property.

If a new Runnable is submitted to the ThreadPoolExecutor once its queue has reached the maximum capacity, and exception will be thrown and the Runnable will not be added. The problem is that even though the Runnable is not actually submitted, ImportEntryProcessRunnable that summited that runnable is caching it, and as a consequence it determines that is already being executed, and is not trying to resubmit it. When that happens, the import entries that are part of the reject Runnables will be permanently left in Initial status, until the server is restarted.

For the issue to be reproduced, several factors must occur at the same time:
- The import entry processing throughput is not able to keep up with the demand, so the queue grows until reaching its maximum capacity
- The combination of (type of data, organization) is very high. Usually there is a runnable for each (type of data, organization) combination (each processor can define its own rules). If the combination was lower than the capacity of the queue the issue would not be reproduced.

Steps To Reproduce

This issue was reproduced by running a performance test on an environment with following characteristics:
- 711 organizations
- 15 different import entry type of data

The reproducibility was quite random until we reduced to 20 the number of available connections in the pool. This reduction resulted in the processing of the import entries being much slower, and as a consequence the import entry manager was not able to keep up with the demand, and the task queue reached its maximum capacity.

Once the performance test was run under these conditions, some import entries were left in Initial Status, and they were not processed until the server was restarted.

Proposed Solution

Additional Information

Tags

NOR

Relationships

depends on	backport	0047144	PR21Q2.1	closed	AugustoMauch	Openbravo ERP	Import entries can be permanently left unprocessed, in Initial status
depends on	backport	0047145	PR21Q1.4	closed	AugustoMauch	Openbravo ERP	Import entries can be permanently left unprocessed, in Initial status
depends on	defect	0047146		closed	platform	Openbravo ERP	API change: update visibility of ImportEntryManager.submitRunnable from public to package
related to	defect	0047219		closed	caristu	Modules	Use different runnables for EDL processes that should be executed in a specific thread