Project:
View Issue Details[ Jump to Notes ] | [ Issue History ] [ Print ] | |||||||
ID | ||||||||
0048427 | ||||||||
Type | Category | Severity | Reproducibility | Date Submitted | Last Update | |||
defect | [Openbravo ERP] A. Platform | major | have not tried | 2022-01-19 13:04 | 2022-11-28 23:54 | |||
Reporter | alostale | View Status | public | |||||
Assigned To | alostale | |||||||
Priority | normal | Resolution | fixed | Fixed in Version | PR22Q3 | |||
Status | closed | Fix in branch | Fixed in SCM revision | |||||
Projection | none | ETA | none | Target Version | ||||
OS | Any | Database | Any | Java version | ||||
OS Version | Database version | Ant version | ||||||
Product Version | SCM revision | |||||||
Review Assigned To | ||||||||
Web browser | ||||||||
Modules | Core | |||||||
Regression level | ||||||||
Regression date | ||||||||
Regression introduced in release | ||||||||
Regression introduced by commit | ||||||||
Triggers an Emergency Pack | No | |||||||
Summary | 0048427: high load of low priority import entries can cause higher priority entries not to be processed | |||||||
Description | Having the following import entry types: 1. OBUIELS_Data1 It has a higher priority (1) 2. OBUIELS_Data2 It has a lower priority (2) It takes long to process each entry Its entries span their keys to consume all the threads It has a high inflow of entries to process The import entry type with the lowest priority (OBUIELS_Data2) can take all the processing capacity till it finishes so the one with the highest priority (OBUIELS_Data1) has no throughput till all (or most) the entries from OBUIELS_Data2 are processed. See the attached graph which represents how many entries of each type are pending to be processed over the time. This issue is caused because: 1. 1st Import Entry manger loop starts: 1.1. It schedules 5000 (batch size) OBUIELS_Data2 entries, they are distributed by key (up to 10) consuming all the available threads. After they are scheduled Import Entry manager thread waits some time before resuming with 2nd loop 2. After wait time elapsed, 2nd loop starts: 2.1. It first submits to be scheduled OBUIELS_Data1 type entries (with a higher priority). As there are still OBUIELS_Data2 entries from the previous cycle pending to be processed OBUIELS_Data1 entries are queued to be processed once previously scheduled entries are done. 2.2. There are new OBUIELS_Data2 entries they are scheduled. The problem is they are assigned, based on their key, to import processor runnables that were created in the previous cycle and are still processing. Because of this, these runnables, after processing entries from the previous loop will process the ones that have just been scheduled, not allowing the OBUIELS_Data1 processors to start till they are done. | |||||||
Steps To Reproduce | This scenario can be simulated with Entry Load Simulator [1] and the following config:** Import Entry config import.batch.size 5000 import.number.of.threads 11 import.max.task.queue.size 1000 import.wait.time 600 import.processing.capacity.per.second 330 ** OBUIELS_Data1 Num of Entries 10 Period 200 Cycles 50 Delay 50 Exec Time 50 # of Keys 50 ** OBUIELS_Data2 Num of Entries 200 Period 100 Cycles 15 Delay 0 Exec Time 750 # of Keys 10 [1] https://gitlab.com/openbravo/tools/platform/org.openbravo.util.ieloadsim [^] | |||||||
Tags | NOR | |||||||
Attached Files | 48427- not-fixed.png [^] (18,079 bytes) 2022-01-19 13:05
| |||||||
Relationships [ Relation Graph ] [ Dependency Graph ] | |||||||||||||||||||||||||||||||||
|
Notes | |
(0134406) hgbot (developer) 2022-01-20 09:02 |
Merge Request created: https://gitlab.com/openbravo/product/openbravo/-/merge_requests/506 [^] |
(0136453) hgbot (developer) 2022-04-11 09:29 |
Directly closing issue as related merge request is already approved. Repository: https://gitlab.com/openbravo/product/openbravo [^] Changeset: f5b8202f8017ca4403c9965f8671e87e54535d69 Author: Asier Lostalé <asier.lostale@openbravo.com> Date: 11-04-2022 07:01:11 URL: https://gitlab.com/openbravo/product/openbravo/-/commit/f5b8202f8017ca4403c9965f8671e87e54535d69 [^] fixed BUG-48427: low prio entries can get precedence over high prio Having scheduled in a given ImportEntryProcessRunnable a set of import entries (all of them sharing same type of data and the same key). If the processing of those entries does not finalize when the next ImportEntry cycle looks for new entries to process and there are entries for the same type of data and key, they will be scheduled to be executed in the already running ImportEntryProcessRunnable. This makes these newly scheduled entries to take precedence over new entries of other types of data with a supposedly higher priority that require a new ImportEntryProcessRunnable. Now ImportEntryProcessRunnable instances are linked to the ImportEntryManager's execution cycle, so new entries for the same type and key are not scheduled if there are still entries with the same key and type scheduled from a previous cycle. --- M src/org/openbravo/service/importprocess/ImportEntryManager.java M src/org/openbravo/service/importprocess/ImportEntryProcessor.java --- |
(0136454) hgbot (developer) 2022-04-11 09:29 |
Merge request merged: https://gitlab.com/openbravo/product/openbravo/-/merge_requests/506 [^] |
Issue History | |||
Date Modified | Username | Field | Change |
2022-01-19 13:04 | alostale | New Issue | |
2022-01-19 13:04 | alostale | Assigned To | => Triage Platform Conn |
2022-01-19 13:04 | alostale | Modules | => Core |
2022-01-19 13:04 | alostale | Triggers an Emergency Pack | => No |
2022-01-19 13:05 | alostale | File Added: 48427- not-fixed.png | |
2022-01-19 13:06 | alostale | Relationship added | related to 0048418 |
2022-01-19 13:20 | alostale | Description Updated | View Revisions |
2022-01-20 08:59 | alostale | Description Updated | View Revisions |
2022-01-20 09:02 | hgbot | Note Added: 0134406 | |
2022-01-20 11:08 | alostale | Assigned To | Triage Platform Conn => alostale |
2022-02-01 07:24 | alostale | Assigned To | alostale => Triage Platform Base |
2022-03-22 15:24 | rafaroda | Tag Attached: NOR | |
2022-03-22 15:24 | rafaroda | Issue Monitored: rafaroda | |
2022-03-22 15:25 | rafaroda | Relationship added | related to 0045145 |
2022-03-28 16:22 | AugustoMauch | Assigned To | Triage Platform Base => alostale |
2022-03-28 16:22 | AugustoMauch | Status | new => acknowledged |
2022-04-04 14:17 | rafaroda | Relationship added | related to 0047219 |
2022-04-11 09:29 | hgbot | Resolution | open => fixed |
2022-04-11 09:29 | hgbot | Status | acknowledged => closed |
2022-04-11 09:29 | hgbot | Fixed in Version | => PR22Q3 |
2022-04-11 09:29 | hgbot | Note Added: 0136453 | |
2022-04-11 09:29 | hgbot | Note Added: 0136454 | |
2022-08-12 10:36 | alostale | Relationship added | related to 0049963 |
2022-11-28 23:54 | eugeni | Issue Monitored: eugeni |
Copyright © 2000 - 2009 MantisBT Group |